diff --git a/.agents/skills/reverse-engineer-anything/SKILL.md b/.agents/skills/reverse-engineer-anything/SKILL.md index cd50cff64..de9f6b7c1 100644 --- a/.agents/skills/reverse-engineer-anything/SKILL.md +++ b/.agents/skills/reverse-engineer-anything/SKILL.md @@ -2,7 +2,7 @@ name: reverse-engineer-anything description: Reverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. metadata: - version: "40" + version: "41" --- # REA @@ -61,12 +61,24 @@ explicit path or endpoint and do not need it. consult the NativeAOT workflow in `references/native-and-artifacts.md` for optional metadata recovery and its supported host/layout boundary. - User-owned browser page already open: `list_browser_targets`. -- Retained HAR/native mitmproxy capture: `inspect_web_network_capture`. Select +- Fiddler Everywhere traffic or a native `.saz` capture: read the + [Fiddler workflow](references/runtime-observation.md#fiddler-everywhere-traffic-debugging). + Inspect saved SAZ with `inspect_web_network_capture` and `format: "saz"`; + connect Everywhere's separate MCP server for live traffic. +- Retained HAR/SAZ/native mitmproxy/aioquic qlog capture: `inspect_web_network_capture`. Select original record ordinals when useful; inspect producer fields and byte/number sidecars without fetching recorded URLs or claiming live attribution. See [historical captures](https://github.com/morluto/rea/blob/main/docs/web-network-captures.md) for the exact upstream - profile and credential exclusions. + profile and credential exclusions. Qlog support starts with aioquic 1.5.0 + JSON 0.3; stream coverage and endpoint loss declarations do not recover bodies + or establish network loss. Prefer detail summary, then kind qlog trace/stream + pages; select one trace, stream ID and direction for contribution/range pages. + Facts retain all-recorded-events-before-selection scope. Other qlog layouts + and PCAP correlation are unsupported. - User-owned Electron runtime already open: `list_electron_targets`. +- Packet-field discovery: `inspect_packet_fields` searches installed protocols + and field names/types before choosing fields for `inspect_packet_capture`. + Query/page the relevant records; registrations do not establish observed traffic. - Recorded packet captures: `inspect_packet_capture` with selected TShark fields, display filters and decode-as rules. Requires Linux, util-linux prlimit and `REA_TSHARK_COMMAND`; see [capture guide](https://github.com/morluto/rea/blob/main/docs/packet-captures.md). @@ -74,6 +86,15 @@ explicit path or endpoint and do not need it. frames; use `match_offset` / `match_limit` for later pages, or explicit null limit for complete matching output. Preserve the source digest across pages. This does not perform live capture. + For address/DNS/SNI/protocol summaries use `inspect_pcap_capture` with MCP + detail summary, then retained kind pcap pages. Requires Wireshark 4.x tshark + and capinfos, with Linux prlimit or Windows Job Objects; see + [packet summaries](https://github.com/morluto/rea/blob/main/docs/pcap-analysis.md). + Select an exact DNS question for address pages; page exhaustion does not prove + network completeness. This reads recordings and does not acquire traffic. +- Decoder discovery: `inspect_signal_decoders` searches installed decoder IDs + or selected decoder help, including channels and options. Query and page the + relevant details before choosing settings; this does not scan hardware. - Recorded logic/signal traces: `decode_signal_recording` with explicit decoder, channel and protocol settings. On Linux with util-linux prlimit, supply sigrok-cli 0.7.2 through `REA_SIGROK_COMMAND`; see [recording guide](https://github.com/morluto/rea/blob/main/docs/signal-recordings.md). diff --git a/.agents/skills/reverse-engineer-anything/references/connection-and-recovery.md b/.agents/skills/reverse-engineer-anything/references/connection-and-recovery.md index ed8393bd9..0feae3ac4 100644 --- a/.agents/skills/reverse-engineer-anything/references/connection-and-recovery.md +++ b/.agents/skills/reverse-engineer-anything/references/connection-and-recovery.md @@ -46,7 +46,7 @@ rea setup --client codex --dry-run --json Use the current supported client's ID: `claude_code`, `claude_desktop`, `codex`, `cursor`, `gemini_cli`, `windsurf`, `devin`, `opencode`, `antigravity`, `copilot_cli`, `commandcode`, `qwen_code`, `vscode`, `trae`, `grok_build`, `omp`, `pi`, -`hermes`, `qoder`, `zcode`, `dsh`, or `grok_bot`. If unknown, inspect `doctor --json` before choosing a +`hermes`, `qoder`, `droid`, `zcode`, `dsh`, or `grok_bot`. If unknown, inspect `doctor --json` before choosing a scope. `grok_bot` uses a chat registration step rather than local `mcp.json`. For unsupported clients, use version-pinned manual stdio registration or CLI. diff --git a/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md b/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md index e43049e36..a4fc6d18f 100644 --- a/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md +++ b/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md @@ -83,6 +83,14 @@ names, the native API record, and value-flow summaries. Pseudocode offsets and l code units; use the returned `next_offset` for continuation. Provider limitations and unavailable facts remain explicit; the complete parent Evidence is retained. +`inspect_macho` on a large framework returns megabytes of imports and exports. +Select `detail: "summary"` for slice identity, UUID and per-collection counts, +then page rows with `view: {kind: "macho", facet: "exports", offset: 0, limit: 64}` +and `normalized_result.parent_evidence_id`. Facets are entrypoints, +architectures, build_metadata, load_commands, dependencies, imports, exports, +segments and provenance. `source_exhaustive: false` means unreturned symbols +may exist. + To see which functions or Objective-C methods a Mach-O actually calls in one run, use `observe_native_calls` with explicit breakpoints and a bounded `duration_ms`/`max_events`. It launches a new process under LLDB, so the target @@ -111,6 +119,8 @@ loaded bytes, file mappings, relocations and the entry against the immutable snapshot. Keep `mismatch` and `unsupported` explicit. Verified import covers the reported static image, not runtime DOS or PC-98 hardware behavior. +For Ghidra decompilation, `procedure_pseudo_code` also accepts `procedure: "file-offset:0x..."`; CLI `rea decompile --file-offset=0x... --provider ghidra` resolves the source byte offset internally. An unresolved 0x-prefixed numeric procedure selector also falls back to a unique file mapping. Existing symbols and analysis addresses win; multiple mappings fail as ambiguous. Preserve the returned file-offset and address locations and raw selection observations. ELF virtual addresses are distinct from file offsets; never assume an arbitrary offset plus image base is an analysis address. + Use `read_bytes` for initialized analysis memory and `address_to_file_offset` to anchor one observed address to original file bytes. Loader fixups can change a word; a source offset does not imply byte equality. Complete function body ranges @@ -194,8 +204,9 @@ every executable in the bundle. It also lists missing or weak loads and earlier `architecture`. System paths stay undetermined because the dyld shared cache provides them. -Use `extract_artifact` when materialized files are needed. It takes no arguments -and materializes all regular files into a fresh temporary directory chosen by +Use `extract_artifact` when materialized files are needed. It extracts the active +target and accepts optional `integrity_policy`; see the [MCP contracts](https://github.com/morluto/rea/blob/main/docs/mcp-contracts.md). +It materializes all regular files into a fresh temporary directory chosen by REA. Symlinks and encrypted entries are inventory facts, not extractable files. The result reports the materialized directory; callers do not choose the destination. No separate permission grant is needed. @@ -213,10 +224,13 @@ context. PSP/stack/device state remains unmodeled. See [DOS guide](https://githu For other headerless images (embedded firmware without ELF/Mach-O/PE/MZ headers), open with `format: "raw-binary"`, `architecture` (`x86`, `x86_64`, -`arm`, or `arm64`), and `entry_point` (`0x` hex). File offset 0 is mapped at +`arm`, `arm64`, or `6502`), and `entry_point` (`0x` hex). File offset 0 is mapped at that address. Hopper and IDA refuse this interpretation; Ghidra uses `BinaryLoader` with a little-endian generic language. Big-endian, Thumb-only, -and Cortex-M variants are unsupported. See [raw binary guide](https://github.com/morluto/rea/blob/main/docs/raw-binary.md). +and Cortex-M variants are unsupported. `6502` selects NMOS 6502; the entry and +complete file mapping must fit within `0x0000..0xffff`. 65C02, 65816, cartridge +containers, bank switching, and mapper semantics are unsupported. Windows +Ghidra P0 admits raw images only for `6502`. See [raw binary guide](https://github.com/morluto/rea/blob/main/docs/raw-binary.md). ### Function annotations in Ghidra diff --git a/.agents/skills/reverse-engineer-anything/references/runtime-observation.md b/.agents/skills/reverse-engineer-anything/references/runtime-observation.md index 3e16a3e19..2d660d675 100644 --- a/.agents/skills/reverse-engineer-anything/references/runtime-observation.md +++ b/.agents/skills/reverse-engineer-anything/references/runtime-observation.md @@ -26,3 +26,18 @@ passive browser/Electron Evidence exist. Prefer captured-source digest identity; report path/digest disagreement as a mismatch and preserve ambiguity. Explicit path mappings are inference inputs and never broaden filesystem or CDP authority. A bundle observed at runtime does not prove every contained module executed. + +## Fiddler Everywhere traffic debugging + +For large saved captures, request `detail: "summary"`, then page retained Evidence +with `inspect_analysis_view` and `view: {kind: "capture", detail: "preview", offset: 0, limit: 20}`. +Select complete record pages only for needed headers, metadata or payloads; views +read the historical parent without repeating capture decoding. + +Inspect native `.saz` snapshots with `inspect_web_network_capture` and +`format: "saz"`; encrypted archives are unsupported. For live +traffic analysis, connect Everywhere's separate MCP server and use its +advertised session tools with an explicit `sessionsSource` (`LiveTraffic` or +`AgentCalls`). Preserve existing sessions and filters when inspecting a +recorded run. See the [capture and MCP guide](https://github.com/morluto/rea/blob/main/docs/web-network-captures.md) +for the workflow and Telerik's public setup and traffic-debugging skills. diff --git a/.gitattributes b/.gitattributes index 1774c0113..b8247b27f 100644 --- a/.gitattributes +++ b/.gitattributes @@ -3,3 +3,6 @@ # Windows command fixtures retain their native line endings. *.cmd text eol=crlf + +# Preserve whitespace in original upstream licensing texts. +licenses/npm-runtime-notices.txt whitespace=-blank-at-eol diff --git a/.github/actions/portable-build-cache/action.yml b/.github/actions/portable-build-cache/action.yml index ad127ce3d..799b9476b 100644 --- a/.github/actions/portable-build-cache/action.yml +++ b/.github/actions/portable-build-cache/action.yml @@ -1,14 +1,24 @@ name: Portable build cache -description: Reuse Turbo build outputs without serializing verification jobs. +description: Restore portable Turbo build outputs from the current run or cross-run cache. inputs: save: description: Save newly built outputs after the job succeeds. default: "false" + build_cache_artifact: + description: Reuse portable Turbo outputs uploaded by the current workflow run. + default: "" runs: using: composite steps: + - name: Download portable build outputs from this workflow run + if: inputs.build_cache_artifact != '' + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 + with: + name: ${{ inputs.build_cache_artifact }} + path: .turbo/cache/ - name: Resolve portable build task hashes id: plan + if: inputs.build_cache_artifact == '' shell: bash env: BUILD_CACHE_PREFIX: rea-portable-build-v2-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('.npmrc', '.nvmrc', 'package-lock.json') }} @@ -28,15 +38,15 @@ runs: console.log(`prefix=${process.env.BUILD_CACHE_PREFIX}-`); ' >> "$GITHUB_OUTPUT" - name: Restore portable build outputs - if: inputs.save != 'true' - uses: actions/cache/restore@v4 + if: inputs.build_cache_artifact == '' && inputs.save != 'true' + uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 with: path: .turbo/cache/ key: ${{ steps.plan.outputs.key }} restore-keys: ${{ steps.plan.outputs.prefix }} - name: Restore and save portable build outputs - if: inputs.save == 'true' - uses: actions/cache@v4 + if: inputs.build_cache_artifact == '' && inputs.save == 'true' + uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 with: path: .turbo/cache/ key: ${{ steps.plan.outputs.key }} diff --git a/.github/workflows/ci-full-pr.yml b/.github/workflows/ci-full-pr.yml new file mode 100644 index 000000000..8f76a2ea2 --- /dev/null +++ b/.github/workflows/ci-full-pr.yml @@ -0,0 +1,18 @@ +name: Full PR qualification + +on: + pull_request: + branches: [main, "release/*"] + types: [labeled] + +permissions: + contents: read + actions: read + +jobs: + qualification: + name: Qualify labeled PR revision + if: github.event.label.name == 'ci:full' + uses: ./.github/workflows/ci.yml + with: + source_sha: ${{ github.sha }} diff --git a/.github/workflows/ci-timings.yml b/.github/workflows/ci-timings.yml new file mode 100644 index 000000000..73a9ba4b9 --- /dev/null +++ b/.github/workflows/ci-timings.yml @@ -0,0 +1,46 @@ +name: CI timings + +on: + schedule: + - cron: "17 * * * *" + workflow_dispatch: + +concurrency: + group: ci-timings-${{ github.repository }} + cancel-in-progress: false + +permissions: + contents: read + actions: read + +jobs: + report: + name: Collect recent completed workflow attempts + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + # Consume only API data and execute batch code from the trusted default branch. + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + with: + ref: ${{ github.event.repository.default_branch }} + persist-credentials: false + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 + with: + node-version-file: .nvmrc + - name: Record queue, execution, and matched step timing + continue-on-error: true + env: + GH_TOKEN: ${{ github.token }} + CI_REPOSITORY: ${{ github.repository }} + run: node scripts/ci/job-timings-batch.mjs + - name: Upload CI timing artifacts + if: always() && hashFiles('.cache/ci-job-timings/*') != '' + continue-on-error: true + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: ci-job-timings-batch + path: .cache/ci-job-timings/ + include-hidden-files: true + if-no-files-found: error + overwrite: true + retention-days: 14 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 86f1ba2e4..142e179bb 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -5,42 +5,44 @@ on: branches: [main] pull_request: branches: [main, "release/*"] + schedule: + - cron: "23 0-2,4-23 * * *" + - cron: "23 3 * * *" workflow_dispatch: + workflow_call: + inputs: + source_sha: + description: Exact qualification source; must match the caller revision. + required: true + type: string permissions: contents: read concurrency: - group: ci-${{ github.event.pull_request.number || github.ref }} - # The latest main push verifies the full tree, including earlier changes. - # Publication uses its own release workflow and concurrency group. - cancel-in-progress: true + # Complete qualification finishes even while main receives more pushes. + # Main feedback also finishes, coalescing newer pushes to one pending run. + # Scheduled/manual main runs coalesce to one active and the latest pending; + # release and label-triggered qualification use exact-revision groups. + group: ci-${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || inputs.source_sha != '' || contains(github.event.pull_request.labels.*.name, 'ci:full')) && 'qualification' || 'feedback' }}-${{ inputs.source_sha || github.event.pull_request.number || github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' && inputs.source_sha == '' && !contains(github.event.pull_request.labels.*.name, 'ci:full') }} env: HUSKY: "0" jobs: - pr-title: - name: Conventional PR title - if: github.event_name == 'pull_request' - runs-on: ubuntu-latest - steps: - - name: Validate title for Release Please - env: - PR_TITLE: ${{ github.event.pull_request.title }} - run: >- - node -e 'const title = process.env.PR_TITLE ?? ""; - const conventional = /^(?:feat|fix|docs|refactor|perf|test|build|ci|chore|revert)(?:\([a-z0-9._/-]+\))?!?: .+/u; - if (!conventional.test(title)) { - console.error("PR title must use Conventional Commits, for example: feat: add historical source import"); - process.exit(1); - }' - changes: name: Classify changes runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + contents: read + actions: read outputs: + tier: ${{ steps.plan.outputs.tier }} + qualification_run_id: ${{ steps.qualification.outputs.run_id }} code: ${{ steps.plan.outputs.code }} + sqlite: ${{ steps.plan.outputs.sqlite }} runtime: ${{ steps.plan.outputs.runtime }} package: ${{ steps.plan.outputs.package }} apple: ${{ steps.plan.outputs.apple }} @@ -59,37 +61,72 @@ jobs: nativeaot: ${{ steps.plan.outputs.nativeaot }} website: ${{ steps.plan.outputs.website }} matrix: ${{ steps.plan.outputs.matrix }} + test_selection: ${{ steps.plan.outputs.test_selection }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 with: + ref: ${{ inputs.source_sha || github.sha }} fetch-depth: 0 - - uses: actions/setup-node@v4 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc + - name: Bind qualification to the caller revision + if: inputs.source_sha != '' + env: + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + test "${SOURCE_SHA}" = "${GITHUB_SHA}" + test "${SOURCE_SHA}" = "$(git rev-parse HEAD)" + - name: Find a verified ancestor for push selection + id: baseline + if: github.event_name == 'push' && inputs.source_sha == '' + env: + GH_TOKEN: ${{ github.token }} + HEAD_SHA: ${{ github.sha }} + CI_REPOSITORY: ${{ github.repository }} + CI_DEFAULT_BRANCH: ${{ github.event.repository.default_branch }} + run: node scripts/ci/baseline.mjs + - name: Admit scheduled full qualification + id: qualification + if: github.event_name == 'schedule' && inputs.source_sha == '' + env: + GH_TOKEN: ${{ github.token }} + HEAD_SHA: ${{ github.sha }} + CI_REPOSITORY: ${{ github.repository }} + CI_REPOSITORY_ID: ${{ github.repository_id }} + CI_DEFAULT_BRANCH: ${{ github.event.repository.default_branch }} + CI_FORCE_FULL: ${{ github.event.schedule == '23 3 * * *' }} + run: node scripts/ci/qualification.mjs - name: Select verification lanes id: plan env: - BASE_SHA: ${{ github.event.pull_request.base.sha }} - HEAD_SHA: ${{ github.event.pull_request.head.sha }} - CI_FULL: ${{ contains(github.event.pull_request.labels.*.name, 'ci:full') }} + BASE_SHA: ${{ steps.baseline.outputs.base || github.event.pull_request.base.sha }} + BASELINE_REASON: ${{ steps.baseline.outputs.reason }} + HEAD_SHA: ${{ inputs.source_sha || github.event.pull_request.head.sha || github.sha }} + CI_FULL: ${{ inputs.source_sha != '' || contains(github.event.pull_request.labels.*.name, 'ci:full') }} + CI_SKIP_FULL: ${{ steps.qualification.outputs.run_full == 'false' }} run: node scripts/ci/plan.mjs static: name: Static checks needs: changes - if: ${{ !cancelled() }} + if: ${{ !cancelled() && needs.changes.outputs.tier != 'reused' }} runs-on: ubuntu-latest + timeout-minutes: 10 steps: - name: Require successful change classification env: CHANGES_RESULT: ${{ needs.changes.result }} run: test "${CHANGES_RESULT}" = "success" - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm - run: npm ci + - name: Validate workflow syntax and expressions + if: needs.changes.outputs.workflows == 'true' + run: go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.7 -shellcheck= -pyflakes= - if: needs.changes.outputs.code == 'true' run: npm run check:ci - if: needs.changes.outputs.code == 'true' @@ -101,7 +138,7 @@ jobs: run: npm run format:check - name: Upload complete static diagnostics if: always() && needs.changes.outputs.code == 'true' - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: name: static-check-reports path: .cache/static-checks/ @@ -114,25 +151,33 @@ jobs: needs: [changes, build] if: needs.changes.outputs.code == 'true' runs-on: ubuntu-latest + timeout-minutes: 30 strategy: fail-fast: false matrix: shard: [1/6, 2/6, 3/6, 4/6, 5/6, 6/6] steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm - run: npm ci - - uses: actions/download-artifact@v4 + - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 with: name: rea-test-runtime path: . - - run: npm run test:ci:shard:run -- --shard=${{ matrix.shard }} + - name: Run complete portable suite with coverage + if: needs.changes.outputs.tier == 'full' + run: npm run test:ci:shard:run -- --shard=${{ matrix.shard }} env: REA_CI_TEST_HISTORY: .cache/ci-test-history/test-results.json - - uses: actions/upload-artifact@v4 + - name: Run complete portable suite for fast feedback + if: needs.changes.outputs.tier == 'fast' + run: npm run test:run -- --reporter=default --reporter=blob --shard=${{ matrix.shard }} + env: + REA_CI_TEST_HISTORY: .cache/ci-test-history/test-results.json + - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 if: ${{ !cancelled() }} with: name: vitest-report-${{ strategy.job-index }} @@ -145,14 +190,26 @@ jobs: build: name: Build package needs: changes - if: needs.changes.outputs.code == 'true' + # Reusable provider workflows consume this run's portable build artifact, + # even when a workflow-only edit selects the lane without portable tests. + if: >- + needs.changes.outputs.code == 'true' || + needs.changes.outputs.browser == 'true' || + needs.changes.outputs.binary_layout == 'true' || + needs.changes.outputs.evm == 'true' || + needs.changes.outputs.javascript_recovery == 'true' || + needs.changes.outputs.recorded_crash == 'true' || + needs.changes.outputs.web_network == 'true' || + needs.changes.outputs.web_runtime == 'true' || + needs.changes.outputs.web_source_map == 'true' runs-on: ubuntu-latest + timeout-minutes: 10 permissions: contents: read actions: read steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm @@ -160,31 +217,27 @@ jobs: - name: Reuse portable build outputs across runs uses: ./.github/actions/portable-build-cache with: - save: "true" + save: ${{ github.event_name != 'pull_request' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }} - run: npm run test:prepare - - name: Find successful default-branch timing history + - name: Find available trusted full-suite timing history id: history continue-on-error: true env: GH_TOKEN: ${{ github.token }} CI_REPOSITORY: ${{ github.repository }} + CI_REPOSITORY_ID: ${{ github.repository_id }} CI_DEFAULT_BRANCH: ${{ github.event.repository.default_branch }} - run: | - run_id="$(gh api --method GET \ - "repos/${CI_REPOSITORY}/actions/workflows/ci.yml/runs" \ - -f branch="${CI_DEFAULT_BRANCH}" -f event=push -f status=success -f per_page=1 \ - --jq '.workflow_runs[0].id // empty')" - echo "run-id=${run_id}" >> "${GITHUB_OUTPUT}" + run: node scripts/ci/test-history.mjs - name: Download previous timing report if: steps.history.outputs.run-id != '' continue-on-error: true - uses: actions/download-artifact@v4 + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 with: name: test-timings run-id: ${{ steps.history.outputs.run-id }} github-token: ${{ github.token }} path: .cache/ci-test-history - - uses: actions/upload-artifact@v4 + - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: name: rea-test-runtime path: | @@ -201,7 +254,7 @@ jobs: # test:prepare caches only portable JS, skill, and catalog outputs. # Native package jobs still run their build and prepack checks; Turbo # restores matching outputs or rebuilds when their inputs differ. - - uses: actions/upload-artifact@v4 + - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: name: rea-build-cache path: .turbo/cache/ @@ -210,30 +263,33 @@ jobs: retention-days: 7 sqlite-runtime: - needs: [changes, build] - if: needs.changes.outputs.code == 'true' + # Preserve all selected coverage, but admit expensive host/provider work + # only after static validation. Portable tests still provide early feedback. + needs: [changes, static, build] + if: needs.changes.outputs.sqlite == 'true' uses: ./.github/workflows/sqlite-runtime.yml test: - name: Test (coverage + timing) + name: Portable tests needs: [changes, test-shard] - if: ${{ !cancelled() }} + if: ${{ !cancelled() && needs.changes.outputs.tier != 'reused' }} runs-on: ubuntu-latest + timeout-minutes: 10 steps: - name: Require successful change classification env: CHANGES_RESULT: ${{ needs.changes.result }} run: test "${CHANGES_RESULT}" = "success" - - uses: actions/checkout@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 if: needs.changes.outputs.code == 'true' && success() - - uses: actions/setup-node@v4 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 if: needs.changes.outputs.code == 'true' && success() with: node-version-file: .nvmrc cache: npm - if: needs.changes.outputs.code == 'true' && success() run: npm ci - - uses: actions/download-artifact@v4 + - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 if: needs.changes.outputs.code == 'true' && success() with: path: .vitest/blob @@ -241,7 +297,7 @@ jobs: merge-multiple: true - name: Merge coverage, JUnit, and timing reports id: merge - if: needs.changes.outputs.code == 'true' && success() + if: needs.changes.outputs.code == 'true' && needs.changes.outputs.tier == 'full' && success() run: >- npm run test:ci:merge -- --coverage.reportsDirectory=coverage @@ -251,27 +307,45 @@ jobs: --reporter=default --reporter=junit --reporter=json + --reporter=./scripts/ci/test-timing-reporter.mjs + --outputFile.junit=test-results.xml + --outputFile.json=test-results.json + - name: Merge fast-suite JUnit and timing reports + id: merge-fast + if: needs.changes.outputs.code == 'true' && needs.changes.outputs.tier == 'fast' && success() + run: >- + npm exec -- vitest --merge-reports + --reporter=default + --reporter=junit + --reporter=json + --reporter=./scripts/ci/test-timing-reporter.mjs --outputFile.junit=test-results.xml --outputFile.json=test-results.json - name: Summarize slowest test files - if: needs.changes.outputs.code == 'true' && !cancelled() && hashFiles('test-results.json') != '' + if: needs.changes.outputs.code == 'true' && !cancelled() && hashFiles('test-timings.json') != '' env: SHARD_RESULT: ${{ needs.test-shard.result }} - MERGE_RESULT: ${{ steps.merge.outcome }} + MERGE_RESULT: ${{ needs.changes.outputs.tier == 'full' && steps.merge.outcome || steps.merge-fast.outcome }} + CI_VERIFY_TEST_INVENTORY: "true" + CI_TEST_SELECTION: ${{ needs.changes.outputs.test_selection }} + CI_PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }} run: node scripts/ci/test-report.mjs - - uses: actions/upload-artifact@v4 + - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 if: needs.changes.outputs.code == 'true' && !cancelled() && hashFiles('test-results.json') != '' with: name: test-results path: | test-results.xml test-results.json + test-timings.json coverage/ + .cache/ci-test-selection/report.json + include-hidden-files: true if-no-files-found: error retention-days: 30 - name: Publish successful timing history - if: needs.changes.outputs.code == 'true' && success() && needs.test-shard.result == 'success' - uses: actions/upload-artifact@v4 + if: needs.changes.outputs.code == 'true' && needs.changes.outputs.tier == 'full' && success() && needs.test-shard.result == 'success' + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: name: test-timings path: .cache/ci-test-history/test-results.json @@ -287,24 +361,24 @@ jobs: conformance-fixtures: name: Source-built conformance fixtures - needs: changes + needs: [changes, static] if: needs.changes.outputs.fixtures == 'true' runs-on: ubuntu-latest timeout-minutes: 10 steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 - name: Verify fixtures from a clean checkout run: npm run verify:fixtures windows-native-build: name: Build bundled Windows x64 native controls - needs: changes + needs: [changes, static] if: needs.changes.outputs.runtime == 'true' runs-on: ubuntu-latest timeout-minutes: 10 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc - name: Prepare cross compiler @@ -321,12 +395,12 @@ jobs: - run: npm run build:windows-native - run: npm run verify:windows-native:artifact - run: node scripts/build-windows-native.mjs --inspection-fixture - - uses: actions/upload-artifact@v4 + - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: name: windows-native-inspection-fixture path: .cache/windows-native-inspection-fixture/ if-no-files-found: error - - uses: actions/upload-artifact@v4 + - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: name: windows-native-controls path: native/windows/build/ @@ -334,7 +408,7 @@ jobs: package-e2e: name: Native package baseline (${{ matrix.platform }} ${{ matrix.architecture }}) - needs: [changes, build, windows-native-build] + needs: [changes, static, build, windows-native-build] if: needs.changes.outputs.runtime == 'true' strategy: fail-fast: false @@ -349,8 +423,8 @@ jobs: NODE_OPTIONS: "--max-old-space-size=1024" UV_THREADPOOL_SIZE: "2" steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm @@ -364,15 +438,17 @@ jobs: assert.equal(process.arch, process.env.EXPECTED_CI_ARCHITECTURE); console.log(JSON.stringify({ platform: process.platform, architecture: process.arch, node: process.version }));" - run: npm ci - - uses: actions/download-artifact@v4 + - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 with: name: rea-build-cache path: .turbo/cache/ - - uses: actions/download-artifact@v4 + - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 with: name: windows-native-controls path: native/windows/build/ - run: npm run verify:windows-native:artifact + - name: Verify NativeAOT diagnostic applicability on this host + run: npm run test:focused -- tests/evaluation/package/nativeAotDiagnostics.test.ts - name: Resolve npm cache before verifier HOME isolation id: npm-cache run: | @@ -381,24 +457,35 @@ jobs: - run: npm run verify:package env: npm_config_cache: ${{ steps.npm-cache.outputs.path }} + REA_VERIFIED_PACKAGE_DIRECTORY: ${{ needs.changes.outputs.tier == 'full' && matrix.platform == 'linux' && matrix.architecture == 'x64' && '.cache/verified-package' || '' }} - name: Verify real Node Inspector discovery and observation timeout-minutes: 10 run: npm run verify:inspector + - name: Retain the exact qualified release package + if: needs.changes.outputs.tier == 'full' && matrix.platform == 'linux' && matrix.architecture == 'x64' + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: verified-release-package + path: .cache/verified-package/ + include-hidden-files: true + if-no-files-found: error + overwrite: true + retention-days: 7 apple-artifacts: name: Real Apple artifact CLI/MCP E2E - needs: [changes, build] + needs: [changes, static, build] if: needs.changes.outputs.apple == 'true' runs-on: macos-15 timeout-minutes: 15 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm - run: npm ci - - uses: actions/download-artifact@v4 + - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 with: name: rea-build-cache path: .turbo/cache/ @@ -435,13 +522,13 @@ jobs: reconstruction-readiness: name: Reconstruction readiness - needs: changes + needs: [changes, static] if: needs.changes.outputs.readiness == 'true' runs-on: ubuntu-latest timeout-minutes: 15 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm @@ -452,7 +539,7 @@ jobs: run: | set -o pipefail npm run verify:readiness | tee readiness-summary.json - - uses: actions/upload-artifact@v4 + - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 if: ${{ !cancelled() }} with: name: reconstruction-readiness @@ -464,23 +551,23 @@ jobs: windows-curated: name: Windows capability and package lane - needs: [changes, build, windows-native-build] + needs: [changes, static, build, windows-native-build] if: needs.changes.outputs.package == 'true' runs-on: windows-latest timeout-minutes: 25 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm - run: npm ci - - uses: actions/download-artifact@v4 + - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 with: name: rea-build-cache path: .turbo/cache/ - run: npm run build:cached - - uses: actions/download-artifact@v4 + - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 with: name: windows-native-controls path: native/windows/build/ @@ -509,6 +596,8 @@ jobs: tests/boundary/providers/native/artifactProvider.extractionCompleteness.test.ts tests/boundary/filesystem/setupUnavailableProvider.test.ts tests/boundary/filesystem/windowsNpxRegistration.test.ts + tests/acceptance/setup/homeDirectoryFallback.test.ts + tests/boundary/filesystem/javascriptApplicationWorkflows.test.ts tests/boundary/filesystem/webScriptManifestPaths.test.ts tests/boundary/process/ProcessEvidence.host.test.ts tests/acceptance/webModuleCli.test.ts @@ -534,7 +623,7 @@ jobs: tests/process-global/updateMaintenance.test.ts tests/conformance/providers/ghidra/realGhidraWindowsWorkflow.test.ts - run: npm run verify:windows-native - - uses: actions/download-artifact@v4 + - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 with: name: windows-native-inspection-fixture path: .cache/windows-native-inspection-fixture/ @@ -553,7 +642,7 @@ jobs: tests/boundary/filesystem/directoryArtifactReaderOpen.test.ts tests/boundary/filesystem/javascriptPackagePrecedence.test.ts tests/boundary/filesystem/javascriptArtifactContentIdentity.test.ts - - uses: actions/setup-node@v4 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version: 22.19.0 - name: Verify native path boundaries on Node 22 @@ -563,7 +652,7 @@ jobs: tests/boundary/filesystem/javascriptPackagePrecedence.test.ts tests/boundary/filesystem/javascriptArtifactContentIdentity.test.ts tests/boundary/process/sourceMapUncHosts.test.ts - - uses: actions/upload-artifact@v4 + - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 if: always() with: name: windows-capabilities @@ -575,9 +664,10 @@ jobs: needs: changes if: needs.changes.outputs.docs == 'true' runs-on: ubuntu-latest + timeout-minutes: 10 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm @@ -585,7 +675,7 @@ jobs: - run: npm run docs:check - name: Generation leaves authored files unchanged run: git diff --exit-code - - uses: actions/upload-artifact@v4 + - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: name: generated-docs path: | @@ -594,73 +684,78 @@ jobs: skills/reverse-engineer-anything if-no-files-found: error - workflows: - name: Validate workflows - needs: changes - if: needs.changes.outputs.workflows == 'true' - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - name: Validate workflow syntax and expressions - run: go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.7 -shellcheck= -pyflakes= - browser: - needs: changes + needs: [changes, static, build] if: needs.changes.outputs.browser == 'true' uses: ./.github/workflows/real-browser.yml + with: + build_cache_artifact: rea-build-cache binary_layout: - needs: changes + needs: [changes, static, build] if: needs.changes.outputs.binary_layout == 'true' uses: ./.github/workflows/real-binary-layout.yml + with: + build_cache_artifact: rea-build-cache evm: - needs: changes + needs: [changes, static, build] if: needs.changes.outputs.evm == 'true' uses: ./.github/workflows/real-evm-interface.yml + with: + build_cache_artifact: rea-build-cache javascript_recovery: - needs: changes + needs: [changes, static, build] if: needs.changes.outputs.javascript_recovery == 'true' uses: ./.github/workflows/real-javascript-recovery.yml + with: + build_cache_artifact: rea-build-cache recorded_crash: - needs: changes + needs: [changes, static, build] if: needs.changes.outputs.recorded_crash == 'true' uses: ./.github/workflows/real-recorded-crash.yml + with: + build_cache_artifact: rea-build-cache web_network: - needs: changes + needs: [changes, static, build] if: needs.changes.outputs.web_network == 'true' uses: ./.github/workflows/real-web-network-captures.yml + with: + build_cache_artifact: rea-build-cache web_runtime: - needs: changes + needs: [changes, static, build] if: needs.changes.outputs.web_runtime == 'true' uses: ./.github/workflows/real-web-runtime.yml + with: + build_cache_artifact: rea-build-cache web_source_map: - needs: changes + needs: [changes, static, build] if: needs.changes.outputs.web_source_map == 'true' uses: ./.github/workflows/real-web-source-map.yml + with: + build_cache_artifact: rea-build-cache nativeaot: - needs: changes + needs: [changes, static] if: needs.changes.outputs.nativeaot == 'true' uses: ./.github/workflows/nativeaot-fixtures.yml website: - needs: changes + needs: [changes, static] if: needs.changes.outputs.website == 'true' uses: ./.github/workflows/website-check.yml required: name: CI required - if: ${{ !cancelled() }} + if: ${{ always() }} needs: [ changes, - pr-title, static, build, test-shard, @@ -673,7 +768,6 @@ jobs: reconstruction-readiness, windows-curated, docs, - workflows, browser, binary_layout, evm, @@ -686,12 +780,24 @@ jobs: website, ] runs-on: ubuntu-latest + timeout-minutes: 5 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc - name: Require every selected lane to pass + id: gate env: CI_JOBS: ${{ toJSON(needs) }} + CI_QUALIFICATION_DIRECTORY: .cache/ci-qualification run: node scripts/ci/required.mjs + - name: Upload full qualification receipt artifact + if: needs.changes.outputs.tier == 'full' && success() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: ${{ steps.gate.outputs.receipt_name }} + path: ${{ steps.gate.outputs.receipt_path }} + if-no-files-found: error + overwrite: true + retention-days: 7 diff --git a/.github/workflows/nativeaot-fixtures.yml b/.github/workflows/nativeaot-fixtures.yml index e819ca5fb..b2cc5801f 100644 --- a/.github/workflows/nativeaot-fixtures.yml +++ b/.github/workflows/nativeaot-fixtures.yml @@ -12,22 +12,23 @@ jobs: name: Build Windows x64 NativeAOT fixture # .NET 8.0.22's bundled linker discovery supports VS 2019/2022. runs-on: windows-2022 - timeout-minutes: 15 + # Two bounded restore attempts and one compile can each take five minutes. + timeout-minutes: 20 env: DOTNET_CLI_TELEMETRY_OPTOUT: "1" DOTNET_PROCESSOR_COUNT: "1" COMPlus_GCHeapHardLimit: "30000000" steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc - - uses: actions/setup-dotnet@v4 + - uses: actions/setup-dotnet@67a3573c9a986a3f9c594539f4ab511d57bb3ce9 # v4.3.1 with: dotnet-version: "8.0.416" - name: Build benign fixture and independent byte oracle run: node scripts/generate-nativeaot-fixtures.mjs - - uses: actions/upload-artifact@v4 + - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: name: nativeaot-win-x64-${{ github.event.pull_request.head.sha || github.sha }} path: | diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml index e276a9f0d..6b8a05b46 100644 --- a/.github/workflows/pages.yml +++ b/.github/workflows/pages.yml @@ -19,8 +19,8 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 10 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm diff --git a/.github/workflows/pr-title.yml b/.github/workflows/pr-title.yml new file mode 100644 index 000000000..abe98bb48 --- /dev/null +++ b/.github/workflows/pr-title.yml @@ -0,0 +1,29 @@ +name: PR title + +on: + pull_request: + branches: [main, "release/*"] + types: [opened, synchronize, reopened, edited] + +permissions: {} + +concurrency: + group: pr-title-${{ github.event.pull_request.number }} + cancel-in-progress: true + +jobs: + pr-title: + name: Conventional PR title + runs-on: ubuntu-latest + timeout-minutes: 2 + steps: + - name: Validate title for Release Please + env: + PR_TITLE: ${{ github.event.pull_request.title }} + run: >- + node -e 'const title = process.env.PR_TITLE ?? ""; + const conventional = /^(?:feat|fix|docs|refactor|perf|test|build|ci|chore|revert)(?:\([a-z0-9._/-]+\))?!?: .+/u; + if (!conventional.test(title)) { + console.error("PR title must use Conventional Commits, for example: feat: add historical source import"); + process.exit(1); + }' diff --git a/.github/workflows/real-binary-layout.yml b/.github/workflows/real-binary-layout.yml index e8c244de6..0218543f0 100644 --- a/.github/workflows/real-binary-layout.yml +++ b/.github/workflows/real-binary-layout.yml @@ -8,14 +8,16 @@ on: type: boolean default: false workflow_call: + inputs: + build_cache_artifact: + description: Portable build cache artifact from the calling CI run. + required: false + type: string + default: "" permissions: contents: read -concurrency: - group: real-binary-layout-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: true - env: NODE_OPTIONS: --max-old-space-size=768 HUSKY: "0" @@ -26,13 +28,15 @@ jobs: runs-on: ubuntu-24.04 timeout-minutes: 15 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm - run: npm ci - uses: ./.github/actions/portable-build-cache + with: + build_cache_artifact: ${{ inputs.build_cache_artifact }} - name: Acquire exact upstream libraries in an isolated runner environment shell: bash run: | diff --git a/.github/workflows/real-browser.yml b/.github/workflows/real-browser.yml index 76ba2e4a6..3c3fe955a 100644 --- a/.github/workflows/real-browser.yml +++ b/.github/workflows/real-browser.yml @@ -2,12 +2,14 @@ name: Real browser verification on: workflow_call: + inputs: + build_cache_artifact: + description: Portable build cache artifact from the calling CI run. + required: false + type: string + default: "" workflow_dispatch: -concurrency: - group: real-browser-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: ${{ github.event_name == 'pull_request' }} - env: HUSKY: "0" @@ -19,13 +21,15 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 10 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm - run: npm ci - uses: ./.github/actions/portable-build-cache + with: + build_cache_artifact: ${{ inputs.build_cache_artifact }} - run: npm run verify:browser env: REA_BROWSER_EXECUTABLE: /usr/bin/google-chrome diff --git a/.github/workflows/real-evm-interface.yml b/.github/workflows/real-evm-interface.yml index 9e4cddeb4..6e510c973 100644 --- a/.github/workflows/real-evm-interface.yml +++ b/.github/workflows/real-evm-interface.yml @@ -3,14 +3,16 @@ name: Real offline EVM interface on: workflow_dispatch: workflow_call: + inputs: + build_cache_artifact: + description: Portable build cache artifact from the calling CI run. + required: false + type: string + default: "" permissions: contents: read -concurrency: - group: real-evm-interface-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: true - env: NODE_OPTIONS: --max-old-space-size=768 HUSKY: "0" @@ -20,13 +22,15 @@ jobs: runs-on: ubuntu-24.04 timeout-minutes: 15 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm - run: npm ci - uses: ./.github/actions/portable-build-cache + with: + build_cache_artifact: ${{ inputs.build_cache_artifact }} - name: Acquire unchanged fixture compiler in isolated runner storage shell: bash run: | diff --git a/.github/workflows/real-ghidra-windows.yml b/.github/workflows/real-ghidra-windows.yml index d6de7469f..db92cd0f8 100644 --- a/.github/workflows/real-ghidra-windows.yml +++ b/.github/workflows/real-ghidra-windows.yml @@ -16,8 +16,8 @@ jobs: if: github.ref == 'refs/heads/main' runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc - name: Prepare cross compiler @@ -31,7 +31,7 @@ jobs: sudo apt-get install --no-install-recommends -y g++-mingw-w64-x86-64 - run: npm run build:windows-native - run: npm run verify:windows-native:artifact - - uses: actions/upload-artifact@v4 + - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: name: real-ghidra-windows-native path: native/windows/build/ @@ -45,12 +45,12 @@ jobs: runs-on: [self-hosted, Windows, x64, ghidra-12-1-2] timeout-minutes: 20 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc - run: npm ci - - uses: actions/download-artifact@v4 + - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 with: name: real-ghidra-windows-native path: native/windows/build/ @@ -88,7 +88,7 @@ jobs: env: REA_ANALYSIS_PROVIDER: ghidra run: npm run verify:ghidra:windows:package -- --x86 --dll | Tee-Object -FilePath windows-ghidra-x86-dll-package-proof.log - - uses: actions/upload-artifact@v4 + - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 if: always() with: name: windows-ghidra-proof diff --git a/.github/workflows/real-go-binary.yml b/.github/workflows/real-go-binary.yml index b886bd70a..c7d74b001 100644 --- a/.github/workflows/real-go-binary.yml +++ b/.github/workflows/real-go-binary.yml @@ -44,12 +44,12 @@ jobs: runs-on: ubuntu-24.04 timeout-minutes: 20 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm - - uses: actions/setup-go@v5 + - uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5.6.0 with: go-version: "1.26.1" cache: false diff --git a/.github/workflows/real-hopper-linux.yml b/.github/workflows/real-hopper-linux.yml index 2355229de..d5e5f8c33 100644 --- a/.github/workflows/real-hopper-linux.yml +++ b/.github/workflows/real-hopper-linux.yml @@ -15,8 +15,8 @@ jobs: runs-on: ubuntu-24.04 timeout-minutes: 25 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm @@ -36,8 +36,8 @@ jobs: runs-on: [self-hosted, Linux, hopper] timeout-minutes: 20 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm diff --git a/.github/workflows/real-hopper.yml b/.github/workflows/real-hopper.yml index 51135f4b6..b3ceeb7d9 100644 --- a/.github/workflows/real-hopper.yml +++ b/.github/workflows/real-hopper.yml @@ -15,8 +15,8 @@ jobs: runs-on: [self-hosted, macOS, hopper] timeout-minutes: 20 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm diff --git a/.github/workflows/real-javascript-recovery.yml b/.github/workflows/real-javascript-recovery.yml index 4fee667d8..a5043b62e 100644 --- a/.github/workflows/real-javascript-recovery.yml +++ b/.github/workflows/real-javascript-recovery.yml @@ -2,15 +2,17 @@ name: Real JavaScript recovery on: workflow_call: + inputs: + build_cache_artifact: + description: Portable build cache artifact from the calling CI run. + required: false + type: string + default: "" workflow_dispatch: permissions: contents: read -concurrency: - group: javascript-recovery-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: ${{ github.event_name == 'pull_request' }} - env: HUSKY: "0" NODE_OPTIONS: "--max-old-space-size=768" @@ -21,13 +23,15 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 10 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm - run: npm ci - uses: ./.github/actions/portable-build-cache + with: + build_cache_artifact: ${{ inputs.build_cache_artifact }} - name: Fetch pinned Wakaru release into the job workspace shell: bash run: | diff --git a/.github/workflows/real-packet-capture.yml b/.github/workflows/real-packet-capture.yml index 8faf9ba99..8daebfd8a 100644 --- a/.github/workflows/real-packet-capture.yml +++ b/.github/workflows/real-packet-capture.yml @@ -7,6 +7,13 @@ on: - "tests/boundary/cli/captureCleanup.test.ts" - "tests/boundary/mcp/packetCaptureMcp.test.ts" - "src/captures/**" + - "src/pcap/**" + - "src/domain/pcap/**" + - "src/domain/analysisView/pcapCaptureView.ts" + - "src/application/pcap/**" + - "src/contracts/pcap/**" + - "src/server/registerPcapTools.ts" + - "tests/boundary/pcap/**" - "src/domain/captures/**" - "src/application/captures/**" - "src/contracts/captures/**" @@ -36,15 +43,16 @@ jobs: runs-on: ubuntu-24.04 timeout-minutes: 15 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm - run: sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y tshark util-linux python3 - run: npm ci - run: npm run build:cached - - run: npm run test:focused -- src/process/LinuxResourceLimits.test.ts tests/boundary/captures/captureProvider.test.ts tests/boundary/captures/captureDescriptorCleanup.test.ts tests/boundary/captures/resourceLimits.test.ts tests/boundary/captures/packetOutput.test.ts tests/boundary/mcp/packetCaptureMcp.test.ts tests/boundary/cli/captureCleanup.test.ts + - run: npm run test:focused -- tests/boundary/captures/fieldDiscovery.test.ts src/process/LinuxResourceLimits.test.ts tests/boundary/captures/captureProvider.test.ts tests/boundary/captures/captureDescriptorCleanup.test.ts tests/boundary/captures/resourceLimits.test.ts tests/boundary/captures/packetOutput.test.ts tests/boundary/mcp/packetCaptureMcp.test.ts tests/boundary/cli/captureCleanup.test.ts - run: node scripts/verify-packet-capture.mjs env: REA_TSHARK_COMMAND: /usr/bin/tshark + REA_CAPINFOS_COMMAND: /usr/bin/capinfos diff --git a/.github/workflows/real-recorded-crash.yml b/.github/workflows/real-recorded-crash.yml index d4410bb24..0463df27d 100644 --- a/.github/workflows/real-recorded-crash.yml +++ b/.github/workflows/real-recorded-crash.yml @@ -3,14 +3,16 @@ name: Real recorded crash evidence on: workflow_dispatch: workflow_call: + inputs: + build_cache_artifact: + description: Portable build cache artifact from the calling CI run. + required: false + type: string + default: "" permissions: contents: read -concurrency: - group: real-recorded-crash-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: true - env: NODE_OPTIONS: --max-old-space-size=768 HUSKY: "0" @@ -23,13 +25,15 @@ jobs: runs-on: ubuntu-24.04 timeout-minutes: 15 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm - run: npm ci - uses: ./.github/actions/portable-build-cache + with: + build_cache_artifact: ${{ inputs.build_cache_artifact }} - name: Install GDB on the disposable CI runner shell: bash run: | diff --git a/.github/workflows/real-signal-recording.yml b/.github/workflows/real-signal-recording.yml index 5ec413e72..5c1380e2d 100644 --- a/.github/workflows/real-signal-recording.yml +++ b/.github/workflows/real-signal-recording.yml @@ -36,15 +36,15 @@ jobs: runs-on: ubuntu-24.04 timeout-minutes: 15 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm - run: sudo apt-get update && sudo apt-get install -y sigrok-cli util-linux python3 - run: npm ci - run: npm run build:cached - - run: npm run test:focused -- src/process/LinuxResourceLimits.test.ts src/domain/signals/annotationTextWindow.test.ts tests/boundary/signals/recordingProvider.test.ts tests/boundary/signals/recordingDescriptorCleanup.test.ts tests/boundary/signals/resourceLimits.test.ts tests/boundary/mcp/signalRecordingMcp.test.ts tests/boundary/cli/signalCleanup.test.ts + - run: npm run test:focused -- tests/boundary/signals/decoderDiscovery.test.ts src/process/LinuxResourceLimits.test.ts src/domain/signals/annotationTextWindow.test.ts tests/boundary/signals/recordingProvider.test.ts tests/boundary/signals/recordingDescriptorCleanup.test.ts tests/boundary/signals/resourceLimits.test.ts tests/boundary/mcp/signalRecordingMcp.test.ts tests/boundary/cli/signalCleanup.test.ts - run: node scripts/verify-signal-recording.mjs env: REA_SIGROK_COMMAND: /usr/bin/sigrok-cli diff --git a/.github/workflows/real-termux.yml b/.github/workflows/real-termux.yml index 204acc1c6..e6f3890e1 100644 --- a/.github/workflows/real-termux.yml +++ b/.github/workflows/real-termux.yml @@ -18,7 +18,7 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 45 steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 - name: Download the pinned Termux app and verify its release checksum shell: bash run: | @@ -38,7 +38,7 @@ jobs: sudo udevadm control --reload-rules sudo udevadm trigger --name-match=kvm - name: Build and capture a page inside Termux - uses: reactivecircus/android-emulator-runner@4c44018e59b437e86cdfc41da381398f93ed8808 # v2 + uses: reactivecircus/android-emulator-runner@a421e43855164a8197daf9d8d40fe71c6996bb0d # v2.38.0 with: api-level: 30 arch: x86_64 @@ -50,7 +50,7 @@ jobs: script: bash scripts/verify/termux/emulator.sh - name: Upload Termux and Android diagnostics if: always() - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: name: termux-browser-diagnostics path: ${{ runner.temp }}/rea-termux/diagnostics diff --git a/.github/workflows/real-web-network-captures.yml b/.github/workflows/real-web-network-captures.yml index b888674f7..3ee59d2c8 100644 --- a/.github/workflows/real-web-network-captures.yml +++ b/.github/workflows/real-web-network-captures.yml @@ -3,14 +3,16 @@ name: Real historical web network captures on: workflow_dispatch: workflow_call: + inputs: + build_cache_artifact: + description: Portable build cache artifact from the calling CI run. + required: false + type: string + default: "" permissions: contents: read -concurrency: - group: real-web-network-captures-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: true - env: HUSKY: "0" NODE_OPTIONS: --max-old-space-size=768 @@ -20,8 +22,8 @@ jobs: runs-on: ubuntu-24.04 timeout-minutes: 15 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm @@ -29,6 +31,8 @@ jobs: run: npm install --global npm@11.16.0 - run: npm ci - uses: ./.github/actions/portable-build-cache + with: + build_cache_artifact: ${{ inputs.build_cache_artifact }} - name: Acquire fixed upstream native profile in runner temporary storage shell: bash run: | diff --git a/.github/workflows/real-web-runtime.yml b/.github/workflows/real-web-runtime.yml index c600e895c..f5b9f201a 100644 --- a/.github/workflows/real-web-runtime.yml +++ b/.github/workflows/real-web-runtime.yml @@ -1,12 +1,15 @@ name: Real website runtime attribution on: workflow_call: + inputs: + build_cache_artifact: + description: Portable build cache artifact from the calling CI run. + required: false + type: string + default: "" workflow_dispatch: permissions: contents: read -concurrency: - group: real-web-runtime-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: ${{ github.event_name == 'pull_request' }} env: HUSKY: "0" jobs: @@ -14,13 +17,15 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 10 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm - run: npm ci - uses: ./.github/actions/portable-build-cache + with: + build_cache_artifact: ${{ inputs.build_cache_artifact }} - run: npm run verify:browser:runtime env: REA_BROWSER_EXECUTABLE: /usr/bin/google-chrome diff --git a/.github/workflows/real-web-source-map.yml b/.github/workflows/real-web-source-map.yml index 483b2f712..2a1e65dc0 100644 --- a/.github/workflows/real-web-source-map.yml +++ b/.github/workflows/real-web-source-map.yml @@ -1,12 +1,15 @@ name: Real website source-map verification on: workflow_call: + inputs: + build_cache_artifact: + description: Portable build cache artifact from the calling CI run. + required: false + type: string + default: "" workflow_dispatch: permissions: contents: read -concurrency: - group: real-web-source-map-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: ${{ github.event_name == 'pull_request' }} env: HUSKY: "0" jobs: @@ -14,13 +17,15 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 10 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc cache: npm - run: npm ci - uses: ./.github/actions/portable-build-cache + with: + build_cache_artifact: ${{ inputs.build_cache_artifact }} - name: Install the pinned compiler into an isolated fixture directory run: npm install --prefix "$RUNNER_TEMP/rea-web-source-map-compiler" --no-audit --no-fund --package-lock=false esbuild@0.25.10 - run: npm run verify:browser:source-maps diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index c60f32a2c..61e39d029 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -48,7 +48,7 @@ jobs: echo "frozen=true" >> "$GITHUB_OUTPUT" fi - - uses: googleapis/release-please-action@v4 + - uses: googleapis/release-please-action@5c625bfb5d1ff62eadeeb3772007f7f66fdcf071 # v4.4.1 if: steps.editorial.outputs.frozen != 'true' with: token: ${{ secrets.RELEASE_PLEASE_TOKEN || secrets.GITHUB_TOKEN }} @@ -70,13 +70,13 @@ jobs: sha: ${{ steps.release.outputs.sha }} steps: - name: Check out reviewed release merge - uses: actions/checkout@v6 + uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0 with: ref: ${{ github.event.pull_request.merge_commit_sha }} fetch-depth: 0 persist-credentials: false - - uses: actions/setup-node@v6 + - uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6.5.0 with: node-version-file: .nvmrc package-manager-cache: false @@ -104,7 +104,7 @@ jobs: node scripts/release-notes.mjs publication --repository "$GH_REPO" \ --source "$SOURCE_SHA" --pr "$RELEASE_PR" - - uses: googleapis/release-please-action@v4 + - uses: googleapis/release-please-action@5c625bfb5d1ff62eadeeb3772007f7f66fdcf071 # v4.4.1 id: release with: token: ${{ secrets.RELEASE_PLEASE_TOKEN || secrets.GITHUB_TOKEN }} @@ -191,14 +191,14 @@ jobs: fi - name: Check out release controller and Git history - uses: actions/checkout@v6 + uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0 with: ref: ${{ github.sha }} fetch-depth: 0 persist-credentials: false - name: Set up Node.js for checkpoint validation - uses: actions/setup-node@v6 + uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6.5.0 with: node-version-file: .nvmrc package-manager-cache: false @@ -226,7 +226,7 @@ jobs: node scripts/release-notes.mjs publication --repository "$GH_REPO" \ --source "$SOURCE_SHA" - - uses: googleapis/release-please-action@v4 + - uses: googleapis/release-please-action@5c625bfb5d1ff62eadeeb3772007f7f66fdcf071 # v4.4.1 id: release with: # A dedicated credential starts PR CI automatically; the built-in @@ -255,7 +255,7 @@ jobs: - name: Check out release pull request if: inputs.phase == 'prepare' && steps.release.outputs.prs_created == 'true' - uses: actions/checkout@v6 + uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0 with: ref: ${{ fromJSON(steps.release.outputs.pr).headBranchName }} fetch-depth: 0 @@ -263,7 +263,7 @@ jobs: - name: Set up Node.js for generated documentation if: inputs.phase == 'prepare' && steps.release.outputs.prs_created == 'true' - uses: actions/setup-node@v6 + uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6.5.0 with: node-version-file: .nvmrc package-manager-cache: false @@ -290,24 +290,36 @@ jobs: if: inputs.phase == 'prepare' && steps.release.outputs.prs_created == 'true' run: npm run docs:check && git diff --exit-code - publish: - name: Publish to npm + qualification: + name: Qualify exact release revision needs: [merged-release, release-please] if: >- always() && !cancelled() && ((needs.merged-release.result == 'success' && needs.merged-release.outputs.release_created == 'true') || (needs.release-please.result == 'success' && inputs.phase == 'publish' && needs.release-please.outputs.release_created == 'true')) + permissions: + contents: read + actions: read + uses: ./.github/workflows/ci.yml + with: + source_sha: ${{ needs.merged-release.outputs.sha || needs.release-please.outputs.sha }} + + publish: + name: Publish to npm + needs: [merged-release, release-please, qualification] + if: >- + always() && !cancelled() && needs.qualification.result == 'success' runs-on: ubuntu-latest permissions: contents: read id-token: write steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0 with: ref: ${{ needs.merged-release.outputs.sha || needs.release-please.outputs.sha }} persist-credentials: false - - uses: actions/setup-node@v6 + - uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6.5.0 with: node-version-file: .nvmrc package-manager-cache: false @@ -315,17 +327,20 @@ jobs: - run: npm ci - - name: Prepare Windows native build toolchain - run: sudo apt-get update && sudo apt-get install --no-install-recommends -y g++-mingw-w64-x86-64 + - name: Download the exact qualified package + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 + with: + name: verified-release-package + path: .cache/verified-package/ - - name: Build and verify bundled Windows native controls - run: npm run build:windows-native && npm run verify:windows-native:artifact - - - name: Verify packaged release before publishing - run: npm run verify:package + - name: Verify qualified package provenance and digest + id: package + run: node scripts/ci/verified-package.mjs validate .cache/verified-package - name: Publish shell: bash + env: + VERIFIED_PACKAGE_TARBALL: ${{ steps.package.outputs.tarball }} run: | version="$(node --input-type=module -e 'import packageJson from "./package.json" with { type: "json" }; process.stdout.write(packageJson.version)')" published_version="$(npm view "rea-agents@${version}" version --json 2>/dev/null || true)" @@ -333,7 +348,7 @@ jobs: echo "rea-agents@${version} is already published; verifying the existing package." else tag="$(node scripts/release-npm-tag.mjs "${version}")" - npm publish --access public --tag "${tag}" + npm publish "${VERIFIED_PACKAGE_TARBALL}" --access public --tag "${tag}" fi - name: Verify published CLI from npm @@ -377,7 +392,7 @@ jobs: contents: read id-token: write steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0 with: ref: ${{ needs.merged-release.outputs.sha || needs.release-please.outputs.sha }} persist-credentials: false diff --git a/.github/workflows/sqlite-runtime.yml b/.github/workflows/sqlite-runtime.yml index 6e5bbcd10..fc379d1dc 100644 --- a/.github/workflows/sqlite-runtime.yml +++ b/.github/workflows/sqlite-runtime.yml @@ -22,14 +22,14 @@ jobs: # The repository-pinned Node 24 version also runs the complete test suite. node: ["22.x", "24.15.0", "24.x", "26.x"] steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version: ${{ matrix.node }} check-latest: true cache: npm - run: npm ci - - uses: actions/download-artifact@v4 + - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 with: name: rea-test-runtime path: . diff --git a/.github/workflows/website-check.yml b/.github/workflows/website-check.yml index 8931ba418..dc17b4b96 100644 --- a/.github/workflows/website-check.yml +++ b/.github/workflows/website-check.yml @@ -7,10 +7,6 @@ on: permissions: contents: read -concurrency: - group: website-check-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: true - jobs: check: name: Website checks @@ -18,9 +14,9 @@ jobs: timeout-minutes: 10 steps: - name: Check out the repository - uses: actions/checkout@v4 + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 - name: Set up Python for website assets - uses: actions/setup-python@v5 + uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 with: python-version: "3.12" cache: pip diff --git a/.github/workflows/website-pages.yml b/.github/workflows/website-pages.yml index 3c2251658..491b028a6 100644 --- a/.github/workflows/website-pages.yml +++ b/.github/workflows/website-pages.yml @@ -47,9 +47,9 @@ jobs: exit 1 fi - name: Check out the repository - uses: actions/checkout@v4 + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 - name: Set up Python for website assets - uses: actions/setup-python@v5 + uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 with: python-version: "3.12" cache: pip @@ -74,9 +74,9 @@ jobs: ) PY - name: Configure Pages - uses: actions/configure-pages@v5 + uses: actions/configure-pages@983d7736d9b0ae728b81ab479565c72886d7745b # v5.0.0 - name: Upload the website - uses: actions/upload-pages-artifact@v4 + uses: actions/upload-pages-artifact@7b1f4a764d45c48632c6b24a0339c27f5614fb0b # v4.0.0 with: path: website/_site @@ -95,15 +95,15 @@ jobs: steps: - name: Check out publishing configuration if: inputs.target == 'both' - uses: actions/checkout@v4 + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 - name: Set up Node.js for Wrangler if: inputs.target == 'both' - uses: actions/setup-node@v4 + uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: node-version-file: .nvmrc - name: Download the verified website if: inputs.target == 'both' - uses: actions/download-artifact@v4 + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 with: name: github-pages path: ${{ runner.temp }}/website-archive @@ -114,7 +114,7 @@ jobs: tar -xf "$RUNNER_TEMP/website-archive/artifact.tar" -C "$RUNNER_TEMP/rea-website-public" - name: Publish the website to Cloudflare if: inputs.target == 'both' - uses: cloudflare/wrangler-action@v4 + uses: cloudflare/wrangler-action@953926a2e2182532811c01a25e53647d93bf07c0 # v4.1.3 with: apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }} accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }} @@ -123,7 +123,7 @@ jobs: command: deploy --cwd website --env production --assets "${{ runner.temp }}/rea-website-public" - name: Publish to GitHub Pages id: deployment - uses: actions/deploy-pages@v4 + uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4.0.5 - name: Verify the selected published revisions env: PAGES_URL: ${{ steps.deployment.outputs.page_url }} diff --git a/.gitignore b/.gitignore index 90d431ec0..0d006e13f 100644 --- a/.gitignore +++ b/.gitignore @@ -36,6 +36,7 @@ coverage/ .vitest/ .vitest-reports/ test-results.xml +test-timings.json .turbo/ .cache/ docs/.vitepress/cache/ @@ -72,3 +73,5 @@ Thumbs.db # Complete static website output, including generated language editions. website/_site/ +# Temporary translation output prepared before replacing the complete site. +website/.i18n-*/ diff --git a/.jscpd.json b/.jscpd.json index c8e89d791..25608bfac 100644 --- a/.jscpd.json +++ b/.jscpd.json @@ -1,5 +1,5 @@ { - "threshold": 3.0, + "threshold": 3.5, "reporters": ["consoleFull"], "ignore": [ "node_modules/**", diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a8d5f3635..50674afa7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -142,6 +142,21 @@ caches deterministic builds and static checks across Git worktrees. After a pack Repository invariants live in [AGENTS.md](AGENTS.md). Keep provider-specific protocols in adapters and shared CLI/MCP workflows in the application layer. +### Windows source checkouts + +The published npm package includes the Windows x64 native addon and its +manifest. A source checkout does not: `native/windows/build/` is ignored by +Git, and neither `npm ci` nor `npm run build:cached` (or `npm run build`) +produces `rea-windows-x64.node` or `manifest.json` in that directory. +Windows workflows and tests that require native controls need these artifacts; +without them, those controls are unavailable even after a successful runtime build. + +Follow the [Windows native build guide](native/windows/README.md#build) before +running those workflows from source. The artifact build requires Linux, +MinGW-w64, and `tar`; the build script does not run on Windows. See that guide +for the build and artifact-verification commands and the separate real Windows +verification prerequisites. Keep the generated addon and manifest uncommitted. + ## Documentation website The VitePress site uses the Markdown files in `docs/`. Run `npm run docs:dev` diff --git a/README.md b/README.md index 1e14f3a2d..c4a89bcee 100644 --- a/README.md +++ b/README.md @@ -161,7 +161,7 @@ Additional tools and host support depend on the target: | Recorded Linux crashes | Raw notes, every recorded thread's registers/signals and optional mapping candidates | Caller-supplied pwntools on Linux x64; optional GDB/pwndbg; [recorded crashes](docs/recorded-crashes.md) | | JavaScript / Electron | Modules, imports, source maps, routes, IPC and native add-on relationships | Node.js and npm; [application analysis](https://rea.tools/guides/javascript/) | | Websites | Page structure, scripts, network observations and requested screenshots | A Chrome-family browser; [browser analysis](https://rea.tools/guides/browser/) | -| Saved network captures | Requests, responses, exposed payloads and source locations | HAR; mitmdump on Linux for native mitmproxy captures; [capture guide](docs/web-network-captures.md) | +| Saved network captures | Requests, responses, exposed payloads and source locations | HAR or Fiddler Everywhere SAZ; mitmdump on Linux for native mitmproxy; [capture guide](docs/web-network-captures.md) | | .NET assemblies | Metadata, CIL instructions, declared native dependencies and build comparisons | None (static only); [managed-code guide](docs/managed-code-analysis.md) | | Android APKs | Manifest declarations, classes, decompiled methods and references | Headless JADX and a full JDK on Linux/macOS/Windows x64; [Android guide](docs/android-analysis.md) | | Android devices | Devices, packages, processes, logs, dumps, screen captures, and APK/file transfers | Caller-supplied adb (emulator or hardware); [ADB guide](docs/adb-device-analysis.md) | @@ -170,6 +170,7 @@ Additional tools and host support depend on the target: | Flutter apps | Dart snapshot hash, per-ABI library digests and engine build provenance | None (static APK parsing); [Flutter guide](docs/flutter-analysis.md) | | Firmware | Regions, extraction results and native-analysis handoffs | Binwalk / Unblob on Linux; [firmware guide](docs/firmware-analysis.md) | | Recorded packets | Selected fields, repeated values, filters and decode-as rules | Linux: TShark 4.x + prlimit; [capture guide](docs/packet-captures.md) | +| Packet summaries | Endpoints, DNS, TLS SNI, protocol distribution and HTTP requests | Caller-supplied tshark and capinfos (Linux-verified); [pcap guide](docs/pcap-analysis.md) | | Recorded logic signals | Decoder annotations, sample ranges and diagnostics | Linux: sigrok-cli 0.7.2 + prlimit; [signal recordings](docs/signal-recordings.md) | | Packages and resources | File inventories, digests, plists, Apple bundle anatomy and extracted resources | [Artifact and JavaScript guide](docs/javascript-artifact-reconstruction.md), [Apple applications](docs/apple-application-analysis.md) | | Process behavior | Terminal output, interactions, exit and filesystem observations, and run comparisons | Linux/macOS with a native PTY; [process capture](docs/process-capture.md) | @@ -321,7 +322,7 @@ Report vulnerabilities through [SECURITY.md](SECURITY.md). ## Star history -🎉 **110,000 GitHub stars — thank you!** +🎉 **170,000 GitHub stars — thank you!** Thanks to everyone using REA, reporting bugs, requesting features, testing builds, and contributing fixes. diff --git a/README_ar.md b/README_ar.md index 0693fe67f..1c0bc367e 100644 --- a/README_ar.md +++ b/README_ar.md @@ -139,6 +139,7 @@ npx rea-agents@latest setup | JavaScript / Electron | الوحدات، وعمليات الاستيراد، وخرائط المصدر، والمسارات، وIPC، وعلاقات إضافات الشيفرة الأصلية | Node.js وnpm؛ [تحليل التطبيقات](https://rea.tools/ar/guides/javascript/) | | المواقع | بنية الصفحة، والسكربتات، وملاحظات الشبكة، ولقطات الشاشة المطلوبة | متصفح من عائلة Chrome؛ [تحليل المتصفح](https://rea.tools/ar/guides/browser/) | | تسجيلات الشبكة المحفوظة | الطلبات، والاستجابات، ومحتويات الحمولة المتاحة، ومواضعها في المصدر | HAR؛ وmitmdump على Linux لتسجيلات mitmproxy بصيغتها الأصلية؛ [دليل تسجيلات الشبكة](docs/web-network-captures.md) | +| التقاط الحزم | نقاط النهاية وDNS وTLS SNI وتوزيع البروتوكولات وطلبات HTTP | tshark وcapinfos يوفّرهما المُستدعي؛ [دليل pcap](docs/pcap-analysis.md) | | تجميعات .NET | البيانات الوصفية، وتعليمات CIL، واعتماديات الشيفرة الأصلية المعلنة، ومقارنات البناء | فحص ثابت؛ [دليل الشيفرة المُدارة](docs/managed-code-analysis.md) | | حزم Android APK | تصريحات ملف manifest، والأصناف، والدوال الناتجة عن فك الترجمة، والمراجع | JADX دون واجهة رسومية وJDK كامل على Linux/macOS/Windows x64؛ [دليل Android](docs/android-analysis.md) | | أجهزة Android | الأجهزة والحزم والعمليات والسجلات والدامبات ولقطات الشاشة ونقل APK/الملفات | adb يوفّره المُستدعي (محاكي أو جهاز)؛ [دليل ADB](docs/adb-device-analysis.md) | @@ -263,7 +264,7 @@ npx rea-agents@latest setup ## سجل النجوم -🎉 **110,000 نجمة على GitHub — شكرًا لكم!** +🎉 **170,000 نجمة على GitHub — شكرًا لكم!** شكرًا لكل من يستخدم REA، ويبلّغ عن الأخطاء، ويقترح ميزات، ويختبر إصدارات البناء، ويساهم بالإصلاحات. diff --git a/README_de.md b/README_de.md index 1e96cd46b..2c6d20290 100644 --- a/README_de.md +++ b/README_de.md @@ -139,6 +139,7 @@ REA benötigt Node.js 22.x (>=22.19), 24.x (>=24.11) oder 26+ sowie npm. Zusätz | JavaScript / Electron | Module, Imports, Source Maps, Routen, IPC und Beziehungen zu nativen Add-ons | Node.js und npm; [Anwendungsanalyse](https://rea.tools/de/guides/javascript/) | | Websites | Seitenstruktur, Skripte, Netzwerkbeobachtungen und angeforderte Screenshots | Ein Browser aus der Chrome-Familie; [Browseranalyse](https://rea.tools/de/guides/browser/) | | Gespeicherte Netzwerkaufzeichnungen | Anfragen, Antworten, zugängliche Nutzdaten und Quellpositionen | HAR; mitmdump unter Linux für Aufzeichnungen im nativen mitmproxy-Format; [Leitfaden zu Netzwerkaufzeichnungen](docs/web-network-captures.md) | +| Paketerfassungen | Endpunkte, DNS, TLS SNI, Protokollverteilung und HTTP-Anfragen | Vom Aufrufer bereitgestellte tshark- und capinfos-Binaries; [pcap-Leitfaden](docs/pcap-analysis.md) | | .NET-Assemblies | Metadaten, CIL-Anweisungen, deklarierte native Abhängigkeiten und Build-Vergleiche | Statische Untersuchung; [Leitfaden für verwalteten Code](docs/managed-code-analysis.md) | | Android-APKs | Manifest-Deklarationen, Klassen, dekompilierte Methoden und Verweise | JADX ohne grafische Oberfläche und ein vollständiges JDK unter Linux/macOS/Windows x64; [Android-Leitfaden](docs/android-analysis.md) | | Android-Geräte | Geräte, Pakete, Prozesse, Logs, Dumps, Bildschirmaufnahmen sowie APK- und Dateiübertragungen | Vom Aufrufer bereitgestelltes adb (Emulator oder Gerät); [ADB-Leitfaden](docs/adb-device-analysis.md) | @@ -261,7 +262,7 @@ Melde Sicherheitslücken gemäß [SECURITY.md](SECURITY.md). ## Sternverlauf -🎉 **110.000 GitHub-Sterne – vielen Dank!** +🎉 **170.000 GitHub-Sterne – vielen Dank!** Danke an alle, die REA nutzen, Fehler melden, Funktionswünsche äußern, Builds testen und Fehlerbehebungen beitragen. diff --git a/README_es.md b/README_es.md index 9a59a74d2..f955b5891 100644 --- a/README_es.md +++ b/README_es.md @@ -139,6 +139,7 @@ REA requiere Node.js 22.x (>=22.19), 24.x (>=24.11) o 26+, además de npm. Las h | JavaScript / Electron | Módulos, importaciones, mapas de código fuente, rutas, IPC y relaciones con complementos nativos | Node.js y npm; [análisis de aplicaciones](https://rea.tools/es/guides/javascript/) | | Sitios web | Estructura de la página, scripts, observaciones de red y capturas de pantalla solicitadas | Un navegador de la familia Chrome; [análisis del navegador](https://rea.tools/es/guides/browser/) | | Capturas de red guardadas | Solicitudes, respuestas, contenido accesible de las cargas útiles y ubicaciones de origen | HAR; mitmdump en Linux para capturas en formato nativo de mitmproxy; [guía de capturas](docs/web-network-captures.md) | +| Capturas de paquetes | Endpoints, DNS, TLS SNI, distribución de protocolos y solicitudes HTTP | tshark y capinfos suministrados por el autor de la llamada; [guía pcap](docs/pcap-analysis.md) | | Ensamblados .NET | Metadatos, instrucciones CIL, dependencias nativas declaradas y comparaciones de compilaciones | Inspección estática; [guía de código administrado](docs/managed-code-analysis.md) | | APK de Android | Declaraciones del manifiesto, clases, métodos descompilados y referencias | JADX sin interfaz gráfica y un JDK completo en Linux/macOS/Windows x64; [guía de Android](docs/android-analysis.md) | | Dispositivos Android | Dispositivos, paquetes, procesos, registros, volcados, capturas de pantalla y transferencias de APK/archivos | adb suministrado por el autor de la llamada (emulador o dispositivo); [guía de ADB](docs/adb-device-analysis.md) | @@ -261,7 +262,7 @@ Informa de vulnerabilidades siguiendo [SECURITY.md](SECURITY.md). ## Historial de estrellas -🎉 **110.000 estrellas en GitHub: ¡gracias!** +🎉 **170.000 estrellas en GitHub: ¡gracias!** Gracias a quienes usan REA, informan de errores, solicitan funciones, prueban compilaciones y contribuyen con correcciones. diff --git a/README_fa.md b/README_fa.md index e34788e7a..54af87c57 100644 --- a/README_fa.md +++ b/README_fa.md @@ -139,6 +139,7 @@ REA به Node.js 22.x (>=22.19)، 24.x (>=24.11) یا 26+، به‌همراه np | برنامه‌های فلاتر | هش اسنپ‌شات دارت، هش کتابخانه‌ها به تفکیک ABI و اصالت بیلد موتور | هیچ (تحلیل ایستا APK)؛ [راهنمای فلاتر](docs/flutter-analysis.md) | | Firmware | ناحیه‌ها، نتایج استخراج و تحویل به ابزارهای تحلیل بومی | Binwalk / Unblob در لینوکس؛ [راهنمای Firmware](docs/firmware-analysis.md) | | بسته‌های ضبط‌شده | فیلدهای منتخب، مقادیر تکراری، فیلترها و قواعد رمزگشایی | Linux: TShark 4.x + prlimit; [راهنمای ضبط بسته](docs/packet-captures.md) | +| ضبط بسته‌ها | نقاط پایانی، DNS، TLS SNI، توزیع پروتکل و درخواست‌های HTTP | tshark و capinfos تأمین‌شده توسط فراخوان‌کننده؛ [راهنمای pcap](docs/pcap-analysis.md) | | سیگنال‌های منطقی ضبط‌شده | توضیحات رمزگشا، بازه‌های نمونه و اطلاعات عیب‌یابی | Linux: sigrok-cli 0.7.2 + prlimit; [راهنمای سیگنال‌ها](docs/signal-recordings.md) | | بسته‌ها و منابع | فهرست فایل‌ها، digestها، فایل‌های plist، ساختار bundle اپل و منابع استخراج‌شده | [راهنمای آرتیفکت و JavaScript](docs/javascript-artifact-reconstruction.md)، [برنامه‌های اپل](docs/apple-application-analysis.md) | | رفتار فرایند | خروجی ترمینال، تعاملات، وضعیت خروج، مشاهدات سیستم فایل و مقایسهٔ اجراها | Linux/macOS با PTY بومی؛ [ضبط فرایند](docs/process-capture.md) | @@ -246,7 +247,7 @@ npx rea-agents@latest setup ## تاریخچهٔ ستاره‌ها -🎉 **۱۱۰٬۰۰۰ ستارهٔ GitHub — سپاسگزاریم!** +🎉 **۱۷۰٬۰۰۰ ستارهٔ GitHub — سپاسگزاریم!** از همهٔ کسانی که از REA استفاده می‌کنند، خطاها را گزارش می‌دهند، قابلیت‌های جدید پیشنهاد می‌کنند، buildها را آزمایش می‌کنند و در بهبود پروژه مشارکت دارند سپاسگزاریم. diff --git a/README_fr.md b/README_fr.md index f28e53f1a..346210525 100644 --- a/README_fr.md +++ b/README_fr.md @@ -143,6 +143,7 @@ REA nécessite Node.js 22.x (>=22.19), 24.x (>=24.11) ou 26+, ainsi que npm. Les | JavaScript / Electron | Modules, imports, source maps, routes, IPC et relations avec les modules natifs | Node.js et npm ; [analyse d'applications](https://rea.tools/fr/guides/javascript/) | | Sites web | Structure des pages, scripts, observations réseau et captures d'écran demandées | Un navigateur de la famille Chrome ; [analyse du navigateur](https://rea.tools/fr/guides/browser/) | | Captures réseau enregistrées | Requêtes, réponses, payloads exposés et emplacements dans le code source | HAR ; mitmdump sous Linux pour les captures mitmproxy natives ; [guide des captures](docs/web-network-captures.md) | +| Captures de paquets | Points de terminaison, DNS, TLS SNI, distribution des protocoles et requêtes HTTP | tshark et capinfos fournis par l'appelant ; [guide pcap](docs/pcap-analysis.md) | | Assemblys .NET | Métadonnées, instructions CIL, dépendances natives déclarées et comparaisons de builds | Inspection statique ; [guide du code managé](docs/managed-code-analysis.md) | | APK Android | Déclarations du manifeste, classes, méthodes décompilées et références | JADX headless et un JDK complet sous Linux/macOS/Windows x64 ; [guide Android](docs/android-analysis.md) | | Appareils Android | Appareils, paquets, processus, journaux, dumps, captures d'écran et transferts d'APK/fichiers | adb fourni par l'appelant (émulateur ou appareil) ; [guide ADB](docs/adb-device-analysis.md) | @@ -265,7 +266,7 @@ Signalez les vulnérabilités via [SECURITY.md](SECURITY.md). ## Historique des étoiles -🎉 **110 000 étoiles GitHub — merci !** +🎉 **170 000 étoiles GitHub — merci !** Merci à toutes les personnes qui utilisent REA, signalent des bugs, proposent des fonctionnalités, testent les builds et contribuent des correctifs. diff --git a/README_id.md b/README_id.md index 440a8f81b..7a564c434 100644 --- a/README_id.md +++ b/README_id.md @@ -146,6 +146,7 @@ Alat tambahan dan dukungan sistem host bergantung pada target yang dianalisis: | JavaScript / Electron | Modul, impor, source map, rute, IPC, dan hubungan add-on native | Node.js dan npm; [analisis aplikasi](https://rea.tools/id/guides/javascript/) | | Situs web | Struktur halaman, skrip, pengamatan jaringan, dan tangkapan layar yang diminta | Peramban berbasis Chrome; [analisis peramban](https://rea.tools/id/guides/browser/) | | Rekaman jaringan tersimpan | Permintaan, respons, payload yang terekspos, dan lokasi sumber | HAR; mitmdump di Linux untuk rekaman native mitmproxy; [panduan rekaman](docs/web-network-captures.md) | +| Capturan paket | Endpoint, DNS, TLS SNI, distribusi protokol, dan permintaan HTTP | tshark dan capinfos buatan pemanggil; [panduan pcap](docs/pcap-analysis.md) | | Assembly .NET | Metadata, instruksi CIL, dependensi native yang dideklarasikan, dan perbandingan build | Tidak ada (hanya analisis statis); [panduan kode terkelola](docs/managed-code-analysis.md) | | APK Android | Deklarasi manifest, kelas, metode hasil dekompilasi, dan referensi | JADX mode headless dan JDK lengkap di Linux/macOS/Windows x64; [panduan Android](docs/android-analysis.md) | | Perangkat Android | Perangkat, paket, proses, log, dump, tangkapan layar, serta transfer APK/berkas | adb buatan pemanggil (emulator atau perangkat); [panduan ADB](docs/adb-device-analysis.md) | @@ -286,7 +287,7 @@ Laporkan kerentanan melalui [SECURITY.md](SECURITY.md). ## Riwayat Bintang -🎉 **110.000 bintang GitHub, terima kasih!** +🎉 **170.000 bintang GitHub, terima kasih!** Terima kasih kepada semua orang yang menggunakan REA, melaporkan bug, meminta fitur, menguji build, dan berkontribusi dalam perbaikan. diff --git a/README_it.md b/README_it.md index 24c1d35bc..8b1c55afe 100644 --- a/README_it.md +++ b/README_it.md @@ -143,6 +143,7 @@ REA richiede Node.js 22.x (>=22.19), 24.x (>=24.11) o 26+, oltre a npm. Strument | JavaScript / Electron | Moduli, import, source map, route, IPC e relazioni con add-on nativi | Node.js e npm; [analisi delle applicazioni](https://rea.tools/it/guides/javascript/) | | Siti web | Struttura della pagina, script, osservazioni di rete e screenshot richiesti | Un browser della famiglia Chrome; [analisi browser](https://rea.tools/it/guides/browser/) | | Acquisizioni di rete salvate | Richieste, risposte, payload esposti e posizioni sorgente | HAR; mitmdump su Linux per acquisizioni mitmproxy native; [guida alle acquisizioni](docs/web-network-captures.md) | +| Captured di pacchetti | Endpoint, DNS, TLS SNI, distribuzione dei protocolli e richieste HTTP | tshark e capinfos forniti dal chiamante; [guida pcap](docs/pcap-analysis.md) | | Assembly .NET | Metadati, istruzioni CIL, dipendenze native dichiarate e confronti di build | Nessuno (solo statico); [guida managed-code](docs/managed-code-analysis.md) | | APK Android | Dichiarazioni manifest, classi, metodi decompilati e riferimenti | JADX headless e un JDK completo su Linux/macOS/Windows x64; [guida Android](docs/android-analysis.md) | | Dispositivi Android | Dispositivi, pacchetti, processi, log, dump, screenshot e trasferimenti APK/file | adb fornito dal chiamante (emulatore o hardware); [guida ADB](docs/adb-device-analysis.md) | @@ -265,7 +266,7 @@ Segnala vulnerabilità tramite [SECURITY.md](SECURITY.md). ## Cronologia delle stelle -🎉 **110.000 stelle su GitHub — grazie!** +🎉 **170.000 stelle su GitHub — grazie!** Grazie a tutti coloro che usano REA, segnalano bug, propongono funzionalità, testano le build e contribuiscono con correzioni. diff --git a/README_ja.md b/README_ja.md index 8b32338aa..5fb42d096 100644 --- a/README_ja.md +++ b/README_ja.md @@ -139,6 +139,7 @@ REA には Node.js 22.x(>=22.19)、24.x(>=24.11)、または 26+ と npm | JavaScript / Electron | モジュール、インポート、ソースマップ、ルート、IPC、ネイティブアドオンの関係 | Node.js と npm;[アプリケーション分析](https://rea.tools/ja/guides/javascript/) | | ウェブサイト | ページ構造、スクリプト、ネットワークの観測結果、依頼されたスクリーンショット | Chrome 系ブラウザー;[ブラウザー分析](https://rea.tools/ja/guides/browser/) | | 保存済みのネットワークキャプチャ | リクエスト、レスポンス、取得可能なペイロード、ソース内の位置 | HAR;mitmproxy ネイティブ形式のキャプチャには Linux 上の mitmdump;[キャプチャガイド](docs/web-network-captures.md) | +| パケットキャプチャ | エンドポイント、DNS、TLS SNI、プロトコル分布、HTTPリクエスト | 呼び出し側が用意した tshark と capinfos;[pcapガイド](docs/pcap-analysis.md) | | .NET アセンブリ | メタデータ、CIL 命令、宣言されたネイティブ依存関係、ビルドの比較 | 静的な検査;[マネージドコードのガイド](docs/managed-code-analysis.md) | | Android APK | マニフェストの宣言、クラス、逆コンパイルされたメソッド、参照 | Linux/macOS/Windows x64 上のヘッドレス JADX と完全な JDK;[Android ガイド](docs/android-analysis.md) | | Android デバイス | デバイス、パッケージ、プロセス、ログ、ダンプ、スクリーンキャプチャ、APK/ファイル転送 | 呼び出し側が用意した adb(エミュレーターまたは実機);[ADB ガイド](docs/adb-device-analysis.md) | @@ -263,7 +264,7 @@ npx rea-agents@latest setup ## スター履歴 -🎉 **GitHub スター 110,000 件、ありがとうございます!** +🎉 **GitHub スター 170,000 件、ありがとうございます!** REA の利用、バグ報告、機能の要望、ビルドのテスト、修正への貢献に感謝します。 diff --git a/README_ko.md b/README_ko.md index 656753d7b..5a9e7cd5d 100644 --- a/README_ko.md +++ b/README_ko.md @@ -139,6 +139,7 @@ REA에는 Node.js 22.x(>=22.19), 24.x(>=24.11) 또는 26+와 npm이 필요합니 | JavaScript / Electron | 모듈, 임포트, 소스 맵, 라우트, IPC, 네이티브 애드온 관계 | Node.js와 npm; [애플리케이션 분석](https://rea.tools/ko/guides/javascript/) | | 웹사이트 | 페이지 구조, 스크립트, 네트워크 관찰 결과, 요청한 스크린샷 | Chrome 계열 브라우저; [브라우저 분석](https://rea.tools/ko/guides/browser/) | | 저장된 네트워크 캡처 | 요청, 응답, 접근 가능한 페이로드, 소스 위치 | HAR; mitmproxy 네이티브 캡처에는 Linux의 mitmdump; [캡처 가이드](docs/web-network-captures.md) | +| 패킷 캡처 | 엔드포인트, DNS, TLS SNI, 프로토콜 분포, HTTP 요청 | 호출자 제공 tshark 및 capinfos; [pcap 가이드](docs/pcap-analysis.md) | | .NET 어셈블리 | 메타데이터, CIL 명령, 선언된 네이티브 의존성, 빌드 비교 | 정적 검사; [관리 코드 가이드](docs/managed-code-analysis.md) | | Android APK | 매니페스트 선언, 클래스, 디컴파일된 메서드, 참조 | Linux/macOS/Windows x64의 헤드리스 JADX와 전체 JDK; [Android 가이드](docs/android-analysis.md) | | Android 기기 | 기기, 패키지, 프로세스, 로그, 덤프, 화면 캡처, APK/파일 전송 | 호출자 제공 adb(에뮬레이터 또는 실제 기기); [ADB 가이드](docs/adb-device-analysis.md) | @@ -263,7 +264,7 @@ npx rea-agents@latest setup ## 스타 기록 -🎉 **GitHub 스타 110,000개, 감사합니다!** +🎉 **GitHub 스타 170,000개, 감사합니다!** REA를 사용하고, 버그를 보고하고, 기능을 요청하고, 빌드를 테스트하고, 수정에 기여해 주신 모든 분께 감사드립니다. diff --git a/README_pl.md b/README_pl.md index f29d5d2fe..c54af50fe 100644 --- a/README_pl.md +++ b/README_pl.md @@ -139,6 +139,7 @@ REA wymaga Node.js 22.x (>=22.19), 24.x (>=24.11) lub 26+ oraz npm. Dodatkowe na | JavaScript / Electron | Moduły, importy, mapy źródeł, trasy, IPC i powiązania z natywnymi dodatkami | Node.js i npm; [analiza aplikacji](https://rea.tools/pl/guides/javascript/) | | Strony internetowe | Struktura strony, skrypty, obserwacje sieciowe i zamówione zrzuty ekranu | Przeglądarka z rodziny Chrome; [analiza w przeglądarce](https://rea.tools/pl/guides/browser/) | | Zapisane przechwycenia ruchu sieciowego | Żądania, odpowiedzi, dostępne dane przesyłane i lokalizacje źródłowe | HAR; mitmdump w systemie Linux dla natywnych przechwyceń mitmproxy; [przewodnik przechwytywania](docs/web-network-captures.md) | +| Przechwytywanie pakietów | Punkty końcowe, DNS, TLS SNI, dystrybucja protokołów i żądania HTTP | Dostarczone przez wywołującego tshark i capinfos; [przewodnik pcap](docs/pcap-analysis.md) | | Zestawy .NET | Metadane, instrukcje CIL, zadeklarowane zależności natywne i porównania kompilacji | Analiza statyczna; [przewodnik kodu zarządzanego](docs/managed-code-analysis.md) | | Pakiety APK dla Androida | Deklaracje manifestu, klasy, zdekompilowane metody i odwołania | JADX bez interfejsu graficznego i pełny JDK w systemach Linux/macOS/Windows x64; [przewodnik Androida](docs/android-analysis.md) | | Urządzenia z Androidem | Urządzenia, pakiety, procesy, logi, zrzuty, zrzuty ekranu oraz transfery APK/plików | adb dostarczony przez wywołującego (emulator lub urządzenie); [przewodnik ADB](docs/adb-device-analysis.md) | @@ -261,7 +262,7 @@ Zgłaszaj podatności zgodnie z [SECURITY.md](SECURITY.md). ## Historia gwiazdek -🎉 **110 000 gwiazdek na GitHubie — dziękujemy!** +🎉 **170 000 gwiazdek na GitHubie — dziękujemy!** Dziękujemy wszystkim, którzy używają REA, zgłaszają błędy, proponują nowe funkcje, testują kompilacje i przesyłają poprawki. diff --git a/README_pt-BR.md b/README_pt-BR.md index ec18da8ff..dd138f7ff 100644 --- a/README_pt-BR.md +++ b/README_pt-BR.md @@ -139,6 +139,7 @@ O REA requer Node.js 22.x (>=22.19), 24.x (>=24.11) ou 26+, além do npm. As fer | JavaScript / Electron | Módulos, importações, source maps, rotas, IPC e relações com complementos nativos | Node.js e npm; [análise de aplicativos](https://rea.tools/pt-BR/guides/javascript/) | | Sites | Estrutura da página, scripts, observações de rede e capturas de tela solicitadas | Um navegador da família Chrome; [análise do navegador](https://rea.tools/pt-BR/guides/browser/) | | Capturas de rede salvas | Requisições, respostas, payloads acessíveis e localizações de origem | HAR; mitmdump no Linux para capturas no formato nativo do mitmproxy; [guia de capturas](docs/web-network-captures.md) | +| Capturas de pacotes | Endpoints, DNS, TLS SNI, distribuição de protocolos e requisições HTTP | tshark e capinfos fornecidos pelo chamador; [guia pcap](docs/pcap-analysis.md) | | Assemblies .NET | Metadados, instruções CIL, dependências nativas declaradas e comparações de builds | Inspeção estática; [guia de código gerenciado](docs/managed-code-analysis.md) | | APKs Android | Declarações do manifesto, classes, métodos descompilados e referências | JADX sem interface gráfica e um JDK completo no Linux/macOS/Windows x64; [guia de Android](docs/android-analysis.md) | | Dispositivos Android | Dispositivos, pacotes, processos, logs, despejos, capturas de tela e transferências de APK/arquivos | adb fornecido pelo chamador (emulador ou aparelho); [guia de ADB](docs/adb-device-analysis.md) | @@ -261,7 +262,7 @@ Relate vulnerabilidades conforme [SECURITY.md](SECURITY.md). ## Histórico de estrelas -🎉 **110.000 estrelas no GitHub — muito obrigado!** +🎉 **170.000 estrelas no GitHub — muito obrigado!** Obrigado a todos que usam o REA, relatam bugs, sugerem funcionalidades, testam builds e contribuem com correções. diff --git a/README_ru.md b/README_ru.md index 2d75af75e..34e7d3af2 100644 --- a/README_ru.md +++ b/README_ru.md @@ -147,6 +147,7 @@ REA требует Node.js 22.x (>=22.19), 24.x (>=24.11) или 26+, а так | Приложения Flutter | Хеш снапшота Dart, дайджесты библиотек по ABI и происхождение сборки движка | Не требуется (статический разбор APK); [руководство по Flutter](docs/flutter-analysis.md) | | Прошивки | Области, результаты извлечения и передача в нативный анализ | Binwalk / Unblob в Linux; [руководство по прошивкам](docs/firmware-analysis.md) | | Записанные пакеты | Выбранные поля, повторяющиеся значения, фильтры и правила декодирования | Linux: TShark 4.x + prlimit; [руководство по захватам](docs/packet-captures.md) | +| Захват пакетов | Конечные точки, DNS, TLS SNI, распределение протоколов и HTTP-запросы | Предоставленные вызывающей стороной tshark и capinfos; [руководство по pcap](docs/pcap-analysis.md) | | Записанные логические сигналы | Аннотации декодера, диапазоны отсчётов и диагностика | Linux: sigrok-cli 0.7.2 + prlimit; [руководство по сигналам](docs/signal-recordings.md) | | Пакеты и ресурсы | Списки файлов, хеши, plist, структура пакетов Apple и извлечённые ресурсы | [Руководство по артефактам и JavaScript](docs/javascript-artifact-reconstruction.md), [приложения Apple](docs/apple-application-analysis.md) | | Поведение процессов | Вывод терминала, взаимодействия, наблюдения за завершением и файловой системой, сравнения запусков | Linux/macOS с нативным PTY; [захват процессов](docs/process-capture.md) | @@ -261,7 +262,7 @@ npx rea-agents@latest setup ## История звёзд -🎉 **110 000 звёзд на GitHub — спасибо!** +🎉 **170 000 звёзд на GitHub — спасибо!** Спасибо всем, кто использует REA, сообщает об ошибках, предлагает новые функции, тестирует сборки и помогает с исправлениями. diff --git a/README_th.md b/README_th.md index 9e0c6831a..d07bbb6a3 100644 --- a/README_th.md +++ b/README_th.md @@ -139,6 +139,7 @@ REA ต้องใช้ Node.js 22.x (>=22.19), 24.x (>=24.11) หรือ 26 | JavaScript / Electron | โมดูล การนำเข้า ซอร์สแมป เส้นทาง IPC และความสัมพันธ์กับส่วนเสริมเนทีฟ | Node.js และ npm; [การวิเคราะห์แอปพลิเคชัน](https://rea.tools/th/guides/javascript/) | | เว็บไซต์ | โครงสร้างหน้า สคริปต์ ข้อมูลเครือข่ายที่สังเกตได้ และภาพหน้าจอที่ร้องขอ | เบราว์เซอร์ในตระกูล Chrome; [การวิเคราะห์ผ่านเบราว์เซอร์](https://rea.tools/th/guides/browser/) | | ข้อมูลเครือข่ายที่บันทึกไว้ | คำขอ คำตอบ เพย์โหลดที่ปรากฏ และตำแหน่งต้นทาง | HAR; mitmdump บน Linux สำหรับข้อมูลที่บันทึกในรูปแบบเนทีฟของ mitmproxy; [คู่มือการบันทึกข้อมูล](docs/web-network-captures.md) | +| การจับแพ็กเก็ต | จุดเชื่อมต่อ, DNS, TLS SNI, การกระจายโปรโตคอล และคำขอ HTTP | tshark และ capinfos ที่ผู้เรียกจัดเตรียม; [คู่มือ pcap](docs/pcap-analysis.md) | | แอสเซมบลี .NET | เมทาดาทา คำสั่ง CIL การพึ่งพาเนทีฟที่ประกาศไว้ และการเปรียบเทียบผลการบิลด์ | การตรวจสอบแบบสถิต; [คู่มือโค้ดแบบจัดการ](docs/managed-code-analysis.md) | | APK ของ Android | การประกาศใน manifest คลาส เมธอดที่ดีคอมไพล์ และการอ้างอิง | JADX แบบไม่มี GUI และ JDK แบบเต็มบน Linux/macOS/Windows x64; [คู่มือ Android](docs/android-analysis.md) | | อุปกรณ์ Android | อุปกรณ์ แพ็กเกจ โพรเซส บันทึก ดัมพ์ ภาพหน้าจอ และการถ่ายโอน APK/ไฟล์ | adb ที่ผู้เรียกจัดเตรียม (โปรแกรมจำลองหรืออุปกรณ์จริง); [คู่มือ ADB](docs/adb-device-analysis.md) | @@ -261,7 +262,7 @@ npx rea-agents@latest setup ## ประวัติดาว -🎉 **110,000 ดาวบน GitHub — ขอบคุณทุกคน!** +🎉 **170,000 ดาวบน GitHub — ขอบคุณทุกคน!** ขอบคุณทุกคนที่ใช้ REA รายงานบั๊ก เสนอฟีเจอร์ ทดสอบบิลด์ และช่วยแก้ไข diff --git a/README_tr.md b/README_tr.md index 3c6e3f3b6..1daaaf9d0 100644 --- a/README_tr.md +++ b/README_tr.md @@ -139,6 +139,7 @@ REA; Node.js 22.x (>=22.19), 24.x (>=24.11) veya 26+ ve npm gerektirir. Ek araç | JavaScript / Electron | Modüller, içe aktarmalar, kaynak haritaları, rotalar, IPC ve yerel eklenti ilişkileri | Node.js ve npm; [uygulama analizi](https://rea.tools/tr/guides/javascript/) | | Web siteleri | Sayfa yapısı, betikler, ağ gözlemleri ve istenen ekran görüntüleri | Chrome ailesinden bir tarayıcı; [tarayıcı analizi](https://rea.tools/tr/guides/browser/) | | Kaydedilmiş ağ yakalamaları | İstekler, yanıtlar, erişilebilir yük içerikleri ve kaynak konumları | HAR; mitmproxy'nin kendi biçimindeki yakalamalar için Linux üzerinde mitmdump; [ağ yakalama rehberi](docs/web-network-captures.md) | +| Paket yakalamaları | Uç noktalar, DNS, TLS SNI, protokol dağılımı ve HTTP istekleri | Çağıranın sağladığı tshark ve capinfos; [pcap rehberi](docs/pcap-analysis.md) | | .NET derlemeleri | Meta veriler, CIL talimatları, bildirilmiş yerel bağımlılıklar ve derleme karşılaştırmaları | Statik inceleme; [yönetilen kod rehberi](docs/managed-code-analysis.md) | | Android APK'leri | Manifest bildirimleri, sınıflar, tersine derlenmiş metotlar ve referanslar | Linux/macOS/Windows x64 üzerinde arayüzsüz JADX ve tam bir JDK; [Android rehberi](docs/android-analysis.md) | | Android cihazları | Cihazlar, paketler, süreçler, günlükler, dökümler, ekran görüntüleri ve APK/dosya aktarımları | Çağıranın sağladığı adb (emülatör veya cihaz); [ADB rehberi](docs/adb-device-analysis.md) | @@ -261,7 +262,7 @@ Güvenlik açıklarını [SECURITY.md](SECURITY.md) üzerinden bildirin. ## Yıldız geçmişi -🎉 **GitHub'da 110.000 yıldız — teşekkürler!** +🎉 **GitHub'da 170.000 yıldız — teşekkürler!** REA'yı kullanan, hata bildiren, özellik talep eden, derlemeleri test eden ve düzeltmelere katkıda bulunan herkese teşekkürler. diff --git a/README_uk.md b/README_uk.md index aea416718..14c3d4984 100644 --- a/README_uk.md +++ b/README_uk.md @@ -139,6 +139,7 @@ REA потребує Node.js 22.x (>=22.19), 24.x (>=24.11) або 26+, а та | JavaScript / Electron | Модулі, імпорти, карти вихідного коду, маршрути, IPC і зв'язки з нативними доповненнями | Node.js і npm; [аналіз застосунків](https://rea.tools/uk/guides/javascript/) | | Сайти | Структура сторінки, скрипти, мережеві спостереження й запитані знімки екрана | Браузер сімейства Chrome; [аналіз браузера](https://rea.tools/uk/guides/browser/) | | Збережені мережеві захоплення | Запити, відповіді, доступні корисні дані та місця в джерелі | HAR; mitmdump у Linux для захоплень у власному форматі mitmproxy; [посібник мережевих захоплень](docs/web-network-captures.md) | +| Захоплення пакетів | Кінцеві точки, DNS, TLS SNI, розподіл протоколів та HTTP-запити | Надані викликаючою стороною tshark та capinfos; [підручник pcap](docs/pcap-analysis.md) | | Збірки .NET | Метадані, інструкції CIL, оголошені нативні залежності й порівняння збірок | Статична перевірка; [посібник керованого коду](docs/managed-code-analysis.md) | | Android APK | Оголошення маніфесту, класи, декомпільовані методи й посилання | JADX без графічного інтерфейсу й повний JDK у Linux/macOS/Windows x64; [посібник Android](docs/android-analysis.md) | | Пристрої Android | Пристрої, пакети, процеси, журнали, дампи, знімки екрана та передавання APK/файлів | adb, наданий викликаючою стороною (емулятор або пристрій); [підручник ADB](docs/adb-device-analysis.md) | @@ -261,7 +262,7 @@ npx rea-agents@latest setup ## Історія зірок -🎉 **110 000 зірок на GitHub — дякуємо!** +🎉 **170 000 зірок на GitHub — дякуємо!** Дякуємо всім, хто використовує REA, повідомляє про помилки, пропонує нові функції, тестує збірки й допомагає з виправленнями. diff --git a/README_vi.md b/README_vi.md index ab3fd493d..86544a535 100644 --- a/README_vi.md +++ b/README_vi.md @@ -139,6 +139,7 @@ REA yêu cầu Node.js 22.x (>=22.19), 24.x (>=24.11) hoặc 26+, cùng với np | JavaScript / Electron | Mô-đun, các khai báo import, source map, tuyến, IPC và quan hệ với tiện ích mở rộng mã máy | Node.js và npm; [phân tích ứng dụng](https://rea.tools/vi/guides/javascript/) | | Trang web | Cấu trúc trang, script, quan sát mạng và ảnh chụp màn hình được yêu cầu | Trình duyệt thuộc họ Chrome; [phân tích trình duyệt](https://rea.tools/vi/guides/browser/) | | Bản ghi lưu lượng mạng đã lưu | Yêu cầu, phản hồi, payload có thể truy cập và vị trí nguồn | HAR; mitmdump trên Linux cho bản ghi định dạng gốc của mitmproxy; [hướng dẫn bản ghi mạng](docs/web-network-captures.md) | +| Bắt gói tin | Điểm cuối, DNS, TLS SNI, phân bố giao thức và yêu cầu HTTP | tshark và capinfos do nơi gọi cung cấp; [hướng dẫn pcap](docs/pcap-analysis.md) | | Assembly .NET | Siêu dữ liệu, lệnh CIL, các phụ thuộc mã máy được khai báo và so sánh bản dựng | Kiểm tra tĩnh; [hướng dẫn mã được quản lý](docs/managed-code-analysis.md) | | APK Android | Khai báo manifest, lớp, phương thức được dịch ngược và tham chiếu | JADX không giao diện đồ họa và JDK đầy đủ trên Linux/macOS/Windows x64; [hướng dẫn Android](docs/android-analysis.md) | | Thiết bị Android | Thiết bị, gói, tiến trình, nhật ký, dump, ảnh chụp màn hình và truyền APK/tệp | adb do nơi gọi cung cấp (trình giả lập hoặc thiết bị); [hướng dẫn ADB](docs/adb-device-analysis.md) | @@ -261,7 +262,7 @@ Báo cáo lỗ hổng theo [SECURITY.md](SECURITY.md). ## Lịch sử sao -🎉 **110.000 sao trên GitHub — xin cảm ơn!** +🎉 **170.000 sao trên GitHub — xin cảm ơn!** Cảm ơn mọi người đã sử dụng REA, báo lỗi, đề xuất tính năng, kiểm thử bản dựng và đóng góp bản sửa lỗi. diff --git a/README_zh-TW.md b/README_zh-TW.md index 81371e91d..62c9ab2f3 100644 --- a/README_zh-TW.md +++ b/README_zh-TW.md @@ -139,6 +139,7 @@ REA 需要 Node.js 22.x(>=22.19)、24.x(>=24.11)或 26+,以及 npm。 | JavaScript / Electron | 模組、匯入、source map、路由、IPC 和原生附加元件關係 | Node.js 和 npm;[應用分析](https://rea.tools/zh-TW/guides/javascript/) | | 網站 | 頁面結構、腳本、網路觀察結果和按請求獲取的螢幕擷取畫面 | Chrome 系瀏覽器;[瀏覽器分析](https://rea.tools/zh-TW/guides/browser/) | | 已保存的網路擷取 | 請求、回應、可訪問的酬載和來源位置 | HAR;原生 mitmproxy 擷取需要 Linux 上的 mitmdump;[擷取指南](docs/web-network-captures.md) | +| 封包擷取 | 端點、DNS、TLS SNI、通訊協定分佈和 HTTP 請求 | 呼叫端提供的 tshark 和 capinfos;[pcap 指南](docs/pcap-analysis.md) | | .NET 程式集 | 中繼資料、CIL 指令、聲明的原生依賴和建置對比 | 靜態檢查;[托管程式碼指南](docs/managed-code-analysis.md) | | Android APK | 清單聲明、類別、反編譯的方法和引用 | Linux/macOS/Windows x64 上的無界面 JADX 和完整 JDK;[Android 指南](docs/android-analysis.md) | | Android 裝置 | 裝置、套件、程序、日誌、傾印、螢幕擷取以及 APK/檔案傳輸 | 呼叫端提供的 adb(模擬器或實體裝置);[ADB 指南](docs/adb-device-analysis.md) | @@ -263,7 +264,7 @@ npx rea-agents@latest setup ## Star 歷史 -🎉 **GitHub Star 達到 110,000 個,感謝大家!** +🎉 **GitHub Star 達到 170,000 個,感謝大家!** 感謝每一位使用 REA、報告錯誤、提出功能需求、測試建置和貢獻修復的朋友。 diff --git a/README_zh.md b/README_zh.md index d802cad50..c8d66ce08 100644 --- a/README_zh.md +++ b/README_zh.md @@ -139,6 +139,7 @@ REA 需要 Node.js 22.x(>=22.19)、24.x(>=24.11)或 26+,以及 npm。 | JavaScript / Electron | 模块、导入、source map、路由、IPC 和原生扩展关系 | Node.js 和 npm;[应用分析](https://rea.tools/zh-CN/guides/javascript/) | | 网站 | 页面结构、脚本、网络观察结果和按请求获取的截图 | Chrome 系浏览器;[浏览器分析](https://rea.tools/zh-CN/guides/browser/) | | 已保存的网络捕获 | 请求、响应、可访问的载荷和来源位置 | HAR;原生 mitmproxy 捕获需要 Linux 上的 mitmdump;[捕获指南](docs/web-network-captures.md) | +| 数据包捕获 | 端点、DNS、TLS SNI、协议分布和 HTTP 请求 | 调用方提供的 tshark 和 capinfos;[pcap 指南](docs/pcap-analysis.md) | | .NET 程序集 | 元数据、CIL 指令、声明的原生依赖和构建对比 | 静态检查;[托管代码指南](docs/managed-code-analysis.md) | | Android APK | 清单声明、类、反编译的方法和引用 | Linux/macOS/Windows x64 上的无界面 JADX 和完整 JDK;[Android 指南](docs/android-analysis.md) | | Android 设备 | 设备、软件包、进程、日志、转储、屏幕截图以及 APK/文件传输 | 调用方提供的 adb(模拟器或实体设备);[ADB 指南](docs/adb-device-analysis.md) | @@ -263,7 +264,7 @@ npx rea-agents@latest setup ## Star 历史 -🎉 **GitHub Star 达到 110,000 个,感谢大家!** +🎉 **GitHub Star 达到 170,000 个,感谢大家!** 感谢每一位使用 REA、报告错误、提出功能需求、测试构建和贡献修复的朋友。 diff --git a/SECURITY.md b/SECURITY.md index 56d9ca784..1f3d73f8e 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -24,9 +24,12 @@ provider-dependent behavior to be reproduced safely. ## Security boundary -REA authenticates each local provider bridge session with a random capability -token and a current-user Unix socket. Tokens are transferred through private -session descriptors rather than process arguments or environment variables. +REA authenticates its Hopper and Ghidra bridges with per-session random +capability tokens. POSIX uses current-user Unix sockets; Windows Ghidra uses +authenticated loopback TCP with a private NTFS runtime whose DACL admits only +the current user and SYSTEM. See [Windows Ghidra P0](docs/windows-ghidra-p0.md#private-runtime-transport-and-lifecycle). +Tokens are transferred through private session descriptors rather than process +arguments or environment variables. This is not a sandbox and does not protect against malicious processes already running as the same operating-system user. Opening an untrusted binary delegates parsing and analysis to the selected local provider with that user's diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 3faad0dba..5fceaaa47 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -28,6 +28,13 @@ entries preserve the declared license and author metadata and include a clearly labeled standard license text. These are documented upstream notice gaps, not recovered copyright notices or a claim that attribution has been fully verified. +The optional `frida@17.23.2` npm archive and tagged Node binding source also +omit a standalone license/copyright document. Its declared +`LGPL-2.0 WITH WxWindows-exception-3.1` license is recorded with unchanged +[Frida licensing text](licenses/frida-17.23.2-COPYING.txt) and the +[GNU Library GPL v2](licenses/LGPL-2.0.txt). These are licensing supplements, +not recovered Node binding or prebuild-component copyright notices. + ## Open-source engine credits and licenses We thank the maintainers and contributors of the projects below. Their diff --git a/bridge/ghidra/ReaGhidraBridge.java b/bridge/ghidra/ReaGhidraBridge.java index e45f97172..c180c5866 100644 --- a/bridge/ghidra/ReaGhidraBridge.java +++ b/bridge/ghidra/ReaGhidraBridge.java @@ -13,6 +13,8 @@ import java.net.UnixDomainSocketAddress; import java.nio.channels.Channels; import java.nio.channels.ServerSocketChannel; import java.nio.channels.SocketChannel; +import java.nio.ByteBuffer; +import java.nio.charset.CharacterCodingException; import java.nio.charset.StandardCharsets; import java.nio.file.AtomicMoveNotSupportedException; import java.nio.file.Files; @@ -22,7 +24,9 @@ import java.nio.file.StandardCopyOption; import java.nio.file.StandardOpenOption; import java.nio.file.attribute.PosixFilePermissions; import java.security.MessageDigest; +import java.util.ArrayDeque; import java.util.ArrayList; +import java.util.Arrays; import java.util.Comparator; import java.util.HashSet; import java.util.IdentityHashMap; @@ -82,6 +86,7 @@ import ghidra.program.model.address.SegmentedAddress; import ghidra.program.model.reloc.Relocation; import java.io.ByteArrayOutputStream; import ghidra.program.model.symbol.Namespace; +import ghidra.program.model.symbol.FlowType; import ghidra.program.model.symbol.RefType; import ghidra.program.model.symbol.Reference; import ghidra.program.model.symbol.ReferenceIterator; @@ -100,12 +105,17 @@ import ghidra.program.model.pcode.PcodeBlock; import ghidra.program.model.pcode.PcodeBlockBasic; import ghidra.program.model.pcode.Varnode; import ghidra.program.model.pcode.VarnodeAST; +import ghidra.program.util.ContextEvaluatorAdapter; +import ghidra.program.util.SymbolicPropogator; +import ghidra.program.util.VarnodeContext; public final class ReaGhidraBridge extends HeadlessScript { private static final int MAX_HIGH_PCODE_OPS = 3000; private static final int MAX_HIGH_PCODE_INPUTS_PER_OP = 64; private static final int MAX_HIGH_PCODE_DEF_USE_EDGES = 12000; private static final Gson GSON = new GsonBuilder().serializeNulls().create(); + private final Map targetSourceIdentity = new IdentityHashMap<>(); + private String admittedTargetSha256; private static final Set DESCRIPTOR_KEYS = Set.of( "transport", "endpoint_path", @@ -207,6 +217,7 @@ public final class ReaGhidraBridge extends HeadlessScript { "Ghidra imported-byte digest does not match the admitted target" ); } + admittedTargetSha256 = descriptor.targetSha256; // GhidraScript wraps run() in a transaction. End it before serving so // each edit owns a real outer transaction and can roll back immediately. end(true); @@ -633,7 +644,7 @@ public final class ReaGhidraBridge extends HeadlessScript { return result; } - private JsonObject addressToFileOffset(JsonObject params) { + private JsonObject addressToFileOffset(JsonObject params) throws Exception { requireKeys(params, Set.of("document", "address")); requireDocument(params); Address address = requireAddress(params, "address"); @@ -643,14 +654,15 @@ public final class ReaGhidraBridge extends HeadlessScript { Long fileOffset = null; int mappings = 0; for (MemoryBlockSourceInfo source : block.getSourceInfos()) { - if (!source.contains(address) || source.getFileBytes().isEmpty()) continue; + if (!source.contains(address) || source.getFileBytes().isEmpty() || + !isTargetSource(source.getFileBytes().get())) continue; long offset = source.getFileBytesOffset(address); if (offset < 0) continue; fileOffset = offset; mappings++; } if (mappings != 1) - throw new RequestFailure("invalid_request", "Address has " + mappings + " authoritative source mappings: " + canonicalAddress(address)); + throw new RequestFailure("invalid_request", "Address has " + mappings + " authoritative target-file mappings: " + canonicalAddress(address)); JsonObject result = new JsonObject(); result.addProperty("address", canonicalAddress(address)); result.addProperty("file_offset", fileOffset); @@ -673,14 +685,96 @@ public final class ReaGhidraBridge extends HeadlessScript { private JsonElement procedurePseudocode(JsonObject params) throws Exception { requireKeys(params, Set.of("document", "procedure")); requireDocument(params); - Function function = resolveProcedure(requireString(params, "procedure")); - String value = decompiledText(decompile(function)); - return functionObservation( - value == null ? JsonNull.INSTANCE : GSON.toJsonTree(value), - noReturnCallLimitations( - currentProgram.getListing().getInstructions(function.getBody(), true) - ) - ); + String selector = requireString(params, "procedure"); + JsonObject selection = new JsonObject(); + Function function = resolveDecompileProcedure(selector, selection); + try { + String value = decompiledText(decompile(function)); + JsonObject result = functionObservation( + value == null ? JsonNull.INSTANCE : GSON.toJsonTree(value), + noReturnCallLimitations( + currentProgram.getListing().getInstructions(function.getBody(), true) + ) + ); + if (selection.size() > 0) result.add("selection", selection); + return result; + } catch (RequestFailure failure) { + if (selection.size() == 0) throw failure; + // Preserve completed resolution facts without changing the backend's + // failure code or suggesting that the successful mapping was invalid. + throw new RequestFailure(failure.code, "Selector " + selector + + " mapped to analysis address " + selection.get("mapped_address").getAsString() + + " and procedure " + canonicalAddress(function.getEntryPoint()) + + "; " + failure.getMessage()); + } + } + + private Function resolveDecompileProcedure(String selector, JsonObject selection) throws Exception { + String prefix = "file-offset:"; + boolean explicitOffset = selector.startsWith(prefix); + if (!explicitOffset) { + try { + return resolveProcedure(selector); + } catch (RequestFailure failure) { + // Existing names/addresses win. Only an unresolved numeric input + // can fall back to source coordinates; ambiguity never falls back. + if (!failure.code.equals("not_found") || !selector.matches("0x[0-9a-fA-F]+")) + throw failure; + } + } + String coordinate = explicitOffset ? selector.substring(prefix.length()) : selector; + if (!coordinate.matches("0x[0-9a-fA-F]+")) + throw new RequestFailure("invalid_request", "File offsets require hexadecimal file-offset:0x... syntax"); + BigInteger parsed = new BigInteger(coordinate.substring(2), 16); + if (parsed.compareTo(BigInteger.valueOf(9007199254740991L)) > 0) + throw new RequestFailure("invalid_request", "File offset exceeds the exact JSON integer range"); + long offset = parsed.longValueExact(); + monitor.checkCancelled(); + // Ghidra's convenience lookup spans every FileBytes and uses coordinates + // local to each object. Only a complete, digest-matched target source + // establishes that the caller's offset belongs to the admitted file. + Set
mappedAddresses = new java.util.LinkedHashSet<>(); + for (MemoryBlock block : currentProgram.getMemory().getBlocks()) { + for (MemoryBlockSourceInfo source : block.getSourceInfos()) { + monitor.checkCancelled(); + FileBytes backing = source.getFileBytes().orElse(null); + if (backing == null || !isTargetSource(backing)) continue; + Address mapped = source.locateAddressForFileOffset(offset); + if (mapped != null) mappedAddresses.add(mapped); + } + } + List
candidates = new ArrayList<>(mappedAddresses); + if (candidates.size() > 1) + throw new RequestFailure("ambiguous", "File offset " + coordinate + + " has multiple loaded mappings: " + candidates.stream().map(ReaGhidraBridge::canonicalAddress).toList() + + "; select an explicit analysis address"); + if (candidates.isEmpty()) + throw new RequestFailure("not_found", "File offset " + coordinate + + " has no loaded source mapping for the admitted target file; Ghidra image base: " + canonicalAddress(currentProgram.getImageBase()) + + ". Inspect mappings with rea inspect-native-load-image --provider ghidra --json or MCP inspect_native_load_image."); + Address mapped = candidates.get(0); + Function function = functionAtOrContaining(mapped); + if (function == null) + throw new RequestFailure("not_found", "File offset " + coordinate + + " maps to analysis address " + canonicalAddress(mapped) + + ", but no procedure starts at or contains that address in the current Ghidra analysis. " + + "Choose an observed procedure with CLI rea search . --kind procedures --mode regex --provider ghidra --json or MCP list_procedures."); + selection.addProperty("file_offset", offset); + selection.addProperty("mapped_address", canonicalAddress(mapped)); + selection.addProperty("procedure_address", canonicalAddress(function.getEntryPoint())); + return function; + } + + private boolean isTargetSource(FileBytes source) throws Exception { + Boolean observed = targetSourceIdentity.get(source); + if (observed != null) return observed; + // Original FileBytes are immutable even when analysis memory is edited. + // Partial/auxiliary sources remain inspectable but cannot establish an + // original-target offset from their object-local coordinates alone. + boolean matched = source.getFileOffset() == 0 && + constantTimeEquals(hashFileBytes(source, false), admittedTargetSha256); + targetSourceIdentity.put(source, matched); + return matched; } private JsonElement procedureAssembly(JsonObject params) throws Exception { @@ -718,7 +812,7 @@ public final class ReaGhidraBridge extends HeadlessScript { return functionObservation(result, boundaryLimitations); } - private JsonArray procedureCalls(JsonObject params, boolean callers) throws Exception { + private JsonObject procedureCalls(JsonObject params, boolean callers) throws Exception { requireKeys(params, Set.of("document", "procedure")); requireDocument(params); Function function = resolveProcedure(requireString(params, "procedure")); @@ -731,7 +825,12 @@ public final class ReaGhidraBridge extends HeadlessScript { for (Function item : ordered) { result.add(canonicalAddress(item.getEntryPoint())); } - return result; + return functionObservation( + result, + noReturnCallLimitations( + currentProgram.getListing().getInstructions(function.getBody(), true) + ) + ); } private JsonObject procedureInfo(JsonObject params) throws Exception { @@ -1233,6 +1332,13 @@ public final class ReaGhidraBridge extends HeadlessScript { result.add("outgoing_references", outgoing); result.add("referenced_strings", referencedStrings); result.add("referenced_names", referencedNames); + boolean swift = swiftProgram(); + boolean aarch64 = currentProgram.getLanguageID().getIdAsString().startsWith("AARCH64:"); + SwiftStringLiteralDecode swiftLiterals = swift && aarch64 ? swiftStringLiterals(function) : null; + if (swiftLiterals != null) { + result.add("swift_string_literals", swiftLiterals.literals()); + result.add("swift_string_literal_omissions", swiftLiterals.omissions()); + } result.add("basic_blocks", blocks); result.add( "native_api", @@ -1252,6 +1358,20 @@ public final class ReaGhidraBridge extends HeadlessScript { limitations.add( "Synthetic Ghidra entry-point references without actionable memory sources are omitted." ); + if (swift && aarch64) { + limitations.add( + "Swift String literals are decoded from constant register pairs at calls, returns and paired stores; literals stored one word at a time, chosen by conditional selects, passed through unresolved jumps, or built at runtime are not reported." + ); + if (swiftLiterals.omissions().size() > 0) { + limitations.add( + "Swift String literal records listed in swift_string_literal_omissions could not be retained or decoded within this dossier's remaining heap budget; later symbolic flows remain unexamined." + ); + } + } else if (swift) { + limitations.add( + "Swift String literal decoding is implemented for AArch64 register pairs only; no Swift literals were decoded for this architecture." + ); + } List boundaryLimitations = noReturnCallLimitations(scan.instructions); for (String limitation : boundaryLimitations) { limitations.add(limitation); @@ -1833,14 +1953,25 @@ public final class ReaGhidraBridge extends HeadlessScript { Address address = explicitAddress ? parseReaAddress(value) : tryParseAddress(value); if (address == null) throw new RequestFailure("not_found", "Unknown Ghidra procedure name or address: " + value); - Function function = currentProgram.getFunctionManager().getFunctionAt(address); - if (function == null && address.isMemoryAddress()) - function = currentProgram.getFunctionManager().getFunctionContaining(address); + Function function = functionAtOrContaining(address); if (function == null) - throw new RequestFailure("not_found", "No procedure exists at the requested address: " + value); + // Other procedure operations retain their analysis-address semantics. + throw new RequestFailure("not_found", "No procedure exists at the requested address: " + value + + "; Ghidra image base: " + canonicalAddress(currentProgram.getImageBase()) + + ". Choose an observed analysis address. CLI: rea search . --kind procedures --mode regex --provider ghidra --json; " + + "MCP: list_procedures. Inspect the loaded base and source mappings with CLI rea analyze --provider ghidra --json " + + "and rea inspect-native-load-image --provider ghidra --json, or MCP binary_overview and inspect_native_load_image. " + + "Repeat the original target options. For decompile, select source coordinates explicitly with --file-offset=0x... or MCP procedure file-offset:0x...."); return function; } + private Function functionAtOrContaining(Address address) { + Function function = currentProgram.getFunctionManager().getFunctionAt(address); + return function == null && address.isMemoryAddress() + ? currentProgram.getFunctionManager().getFunctionContaining(address) + : function; + } + private JsonObject nativeApiBoundary( Function function, DecompileResults decompilation @@ -3294,6 +3425,349 @@ public final class ReaGhidraBridge extends HeadlessScript { return result; } + private boolean swiftProgram() { + for (MemoryBlock block : currentProgram.getMemory().getBlocks()) { + String name = block.getName(); + if (name.startsWith("__swift5_") || name.startsWith("swift5_")) return true; + } + return false; + } + + /** + * Swift passes a String as two words: small literals carry their bytes in the words and + * immortal large literals carry their count and a biased pointer to the bytes. Ghidra keeps + * neither as a data reference, so decode constant pairs where Swift consumes them: argument + * and return registers at calls, tail calls and returns, and both registers of one STP. + * The canonical empty string is a small literal with a zero count. + */ + private SwiftStringLiteralDecode swiftStringLiterals(Function function) throws Exception { + Register[] registers = new Register[31]; + for (int index = 0; index < registers.length; index++) { + registers[index] = currentProgram.getRegister("x" + index); + } + TreeMap observed = new TreeMap<>(); + TreeMap omitted = new TreeMap<>(); + SwiftLiteralBudget budget = new SwiftLiteralBudget(); + Map selectedValues = new HashMap<>(); + SymbolicPropogator propagator = new SymbolicPropogator(currentProgram, false); + propagator.flowConstants(function.getEntryPoint(), function.getBody(), new ContextEvaluatorAdapter() { + @Override + public boolean evaluateContextBefore(VarnodeContext context, Instruction instruction) { + // Returning true ends only the current flow, so every queued flow checks the shared stop. + if (budget.stopped()) return true; + for (Register[] pair : swiftStringPairs(function, instruction, registers)) { + Long word = registerConstant(context, pair[0]); + Long object = registerConstant(context, pair[1]); + if (word == null || object == null) continue; + SwiftLiteralLayout layout = swiftLiteralLayout(word, object); + if (layout == null) continue; + // A selected word can combine with a path-dependent ordinary move. Cache the + // provenance question per site/register, charging retained cache state to the + // same dossier budget before allocating it. + boolean selected = false; + boolean provenanceOverCapacity = false; + for (Register register : pair) { + String provenanceKey = canonicalAddress(instruction.getAddress()) + "\u0000" + + register.getBaseRegister().getName(); + Boolean value = selectedValues.get(provenanceKey); + if (value == null) { + if (!budget.takeState()) { + provenanceOverCapacity = true; + break; + } + value = selectedValue(function, instruction, register); + selectedValues.put(provenanceKey, value); + } + if (value) { + selected = true; + break; + } + } + if (selected) continue; + // Forks can reach one site with different constants, so the words are part of the key. + String key = canonicalAddress(instruction.getAddress()) + "\u0000" + pair[0].getName() + + "\u0000" + pair[1].getName() + "\u0000" + Long.toUnsignedString(word, 16) + + "\u0000" + Long.toUnsignedString(object, 16); + if (observed.containsKey(key) || omitted.containsKey(key)) continue; + // Every record draws on the budget; the first one that does not fit is kept as an + // omission and decoding stops, so later sites are never examined. + String value = layout.smallValue(); + boolean overCapacity = provenanceOverCapacity || !budget.take(layout.byteLength()); + if (!overCapacity && value == null) { + try { + value = swiftLargeLiteralText(layout); + } catch (OutOfMemoryError exhausted) { + overCapacity = true; + } + } + if (value == null && !overCapacity) continue; + JsonObject item = overCapacity + ? swiftStringOmission(layout, word, object) + : swiftStringItem(value, layout, word, object); + item.addProperty("source_address", canonicalAddress(instruction.getAddress())); + JsonArray pairRegisters = new JsonArray(); + pairRegisters.add(pair[0].getName()); + pairRegisters.add(pair[1].getName()); + item.add("registers", pairRegisters); + (overCapacity ? omitted : observed).put(key, item); + if (overCapacity) { + budget.stop(); + return true; + } + } + return false; + } + }, false, monitor); + JsonArray literals = new JsonArray(); + observed.values().forEach(literals::add); + JsonArray omissions = new JsonArray(); + omitted.values().forEach(omissions::add); + return new SwiftStringLiteralDecode(literals, omissions); + } + + private static List swiftStringPairs( + Function function, + Instruction instruction, + Register[] registers + ) { + List pairs = new ArrayList<>(); + if (instruction.getMnemonicString().equals("stp")) { + Register first = instruction.getRegister(0); + Register second = instruction.getRegister(1); + if (first != null && second != null && first.getBitLength() == 64 && second.getBitLength() == 64) { + pairs.add(new Register[] {first, second}); + } + return pairs; + } + FlowType flow = instruction.getFlowType(); + boolean returns = false; + for (PcodeOp operation : instruction.getPcode()) { + if (operation.getOpcode() == PcodeOp.RETURN) returns = true; + } + // A jump is a proven tail call only when it has resolved targets and all leave the + // body. A targetless computed jump can still dispatch inside the function. + Address[] targets = instruction.getFlows(); + boolean tailCall = flow.isJump() && !flow.isConditional() && targets.length > 0 && + Arrays.stream(targets).allMatch(target -> !function.getBody().contains(target)); + if (!flow.isCall() && !returns && !tailCall) return pairs; + // A String occupies two consecutive registers among x0-x7, or x0 and x1 when returned. + int lastFirstRegister = returns ? 0 : 6; + for (int index = 0; index <= lastFirstRegister; index++) { + pairs.add(new Register[] {registers[index], registers[index + 1]}); + } + return pairs; + } + + private static Long registerConstant(VarnodeContext context, Register register) { + Varnode value = context.getRegisterVarnodeValue(register); + return value != null && context.isConstant(value) ? value.getOffset() : null; + } + + // Search every predecessor path for the reaching definitions, following register copies, + // since a select copied into an argument register or defined before a branch forks just as + // independently as one written there directly. + private boolean selectedValue(Function function, Instruction use, Register register) { + ArrayDeque instructions = new ArrayDeque<>(); + ArrayDeque targets = new ArrayDeque<>(); + Set visited = new HashSet<>(); + for (Instruction predecessor : predecessors(function, use)) { + instructions.push(predecessor); + targets.push(register.getBaseRegister()); + } + while (!instructions.isEmpty()) { + Instruction instruction = instructions.pop(); + Register target = targets.pop(); + if (!visited.add(canonicalAddress(instruction.getAddress()) + "\u0000" + target.getName())) continue; + boolean defines = false; + for (Object result : instruction.getResultObjects()) { + if (result instanceof Register written && written.getBaseRegister().equals(target)) defines = true; + } + Register next = target; + if (defines) { + PcodeOp[] operations = instruction.getPcode(); + for (PcodeOp operation : operations) { + if (operation.getOpcode() == PcodeOp.CBRANCH && operation.getInput(0).isConstant()) return true; + } + Register source = copiedRegister(operations); + if (source == null) continue; + next = source.getBaseRegister(); + } + for (Instruction predecessor : predecessors(function, instruction)) { + instructions.push(predecessor); + targets.push(next); + } + } + return false; + } + + private List predecessors(Function function, Instruction instruction) { + List result = new ArrayList<>(); + Address from = instruction.getFallFrom(); + if (from != null) { + Instruction previous = currentProgram.getListing().getInstructionAt(from); + if (previous != null) result.add(previous); + } + for (Reference reference : currentProgram.getReferenceManager().getReferencesTo(instruction.getAddress())) { + if (!reference.getReferenceType().isFlow() || reference.getReferenceType().isCall()) continue; + if (!function.getBody().contains(reference.getFromAddress())) continue; + Instruction source = currentProgram.getListing().getInstructionAt(reference.getFromAddress()); + if (source != null) result.add(source); + } + return result; + } + + private Register copiedRegister(PcodeOp[] operations) { + if (operations.length != 1) return null; + int opcode = operations[0].getOpcode(); + Varnode input = operations[0].getInput(0); + if ((opcode != PcodeOp.COPY && opcode != PcodeOp.INT_ZEXT) || input == null || !input.isRegister()) return null; + return currentProgram.getRegister(input); + } + + /** Validate the ABI layout of a pair; a small literal is decoded here, a large one only located. */ + private SwiftLiteralLayout swiftLiteralLayout(long word, long object) { + int discriminator = (int) (object >>> 60); + if (discriminator == 0xE || discriminator == 0xA) { + int count = (int) ((object >>> 56) & 0xF); + byte[] bytes = new byte[15]; + for (int index = 0; index < 8; index++) bytes[index] = (byte) (word >>> (8 * index)); + for (int index = 0; index < 7; index++) bytes[8 + index] = (byte) (object >>> (8 * index)); + for (int index = count; index < bytes.length; index++) { + if (bytes[index] != 0) return null; + } + // Count zero is the canonical empty string (object 0xE000000000000000, word 0). + String value = swiftStringText(bytes, count, discriminator == 0xE); + return value == null ? null : new SwiftLiteralLayout("small", null, count, true, value); + } + int flags = (int) (word >>> 60); + long count = word & 0xFFFFFFFFFFFFL; + // Immortal literals: ASCII and NFC or neither, tail allocated, with a pointer biased by 32. + if ((object >>> 56) != 0x80 || (flags != 0xD && flags != 0x1) || ((word >>> 48) & 0xFFF) != 0) { + return null; + } + try { + Address address = currentProgram.getAddressFactory().getDefaultAddressSpace() + .getAddress((object & 0x00FFFFFFFFFFFFFFL) + 0x20); + MemoryBlock block = currentProgram.getMemory().getBlock(address); + // The bytes and their terminator must lie in one initialized, non-executable block, + // and Java arrays bound one read. + if (block == null || !block.isInitialized() || block.isExecute() || + count >= block.getEnd().subtract(address) + 1 || count >= Integer.MAX_VALUE) { + return null; + } + return new SwiftLiteralLayout("large", address, count, flags == 0xD, null); + } catch (Exception unaddressable) { + return null; + } + } + + private String swiftLargeLiteralText(SwiftLiteralLayout layout) { + int count = (int) layout.byteLength(); + byte[] bytes = new byte[count + 1]; + try { + if (currentProgram.getMemory().getBytes(layout.address(), bytes) != bytes.length) return null; + } catch (Exception unreadable) { + return null; + } + return bytes[count] == 0 ? swiftStringText(bytes, count, layout.ascii()) : null; + } + + /** + * Every literal record of one dossier draws on half of the heap that was free when decoding + * began. Text is charged at the worst case per UTF-8 byte: two bytes of UTF-16 text, then up + * to six escaped JSON characters ("\\u0000") held as UTF-16 in Gson's growing writer buffer + * (up to twice its length), again in the returned String, and once more as UTF-8 on the + * socket. Each record also pays a fixed cost for its map key, JSON members, addresses, words + * and their serialized form, so many empty or small literals are bounded too. Retained + * provenance-cache entries draw on that same fixed-state allowance. + */ + private static final class SwiftLiteralBudget { + private static final long BYTES_PER_LITERAL_BYTE = 2 + 6 * 2 * 2 + 6 * 2 + 6; + private static final long BYTES_PER_RECORD = 4096; + private long remaining; + private boolean stopped; + + SwiftLiteralBudget() { + Runtime runtime = Runtime.getRuntime(); + remaining = Math.max(0, (runtime.maxMemory() - runtime.totalMemory() + runtime.freeMemory()) / 2); + } + + boolean takeState() { + if (remaining < BYTES_PER_RECORD) return false; + remaining -= BYTES_PER_RECORD; + return true; + } + + boolean take(long bytes) { + if (remaining < BYTES_PER_RECORD || + bytes > (remaining - BYTES_PER_RECORD) / BYTES_PER_LITERAL_BYTE) { + return false; + } + remaining -= BYTES_PER_RECORD + bytes * BYTES_PER_LITERAL_BYTE; + return true; + } + + void stop() { + stopped = true; + } + + boolean stopped() { + return stopped; + } + } + + // Swift String values may contain any scalar, including controls; JSON escapes them. + private static String swiftStringText(byte[] bytes, int length, boolean ascii) { + if (length < 0 || length > bytes.length) return null; + for (int index = 0; index < length; index++) { + if (ascii && bytes[index] < 0) return null; + } + try { + return StandardCharsets.UTF_8.newDecoder().decode(ByteBuffer.wrap(bytes, 0, length)).toString(); + } catch (CharacterCodingException invalid) { + return null; + } + } + + private static JsonObject swiftStringOmission(SwiftLiteralLayout layout, long word, long object) { + JsonObject item = new JsonObject(); + item.addProperty("representation", layout.representation()); + if (layout.address() == null) { + item.add("address", JsonNull.INSTANCE); + } else { + item.addProperty("address", canonicalAddress(layout.address())); + } + item.addProperty("byte_length", layout.byteLength()); + item.addProperty("reason", "capacity"); + item.add("words", swiftStringWords(word, object)); + return item; + } + + private static JsonObject swiftStringItem( + String value, + SwiftLiteralLayout layout, + long word, + long object + ) { + JsonObject item = new JsonObject(); + item.addProperty("value", value); + item.addProperty("representation", layout.representation()); + if (layout.address() == null) { + item.add("address", JsonNull.INSTANCE); + } else { + item.addProperty("address", canonicalAddress(layout.address())); + } + item.add("words", swiftStringWords(word, object)); + return item; + } + + private static JsonArray swiftStringWords(long word, long object) { + JsonArray words = new JsonArray(); + words.add("0x" + Long.toUnsignedString(word, 16)); + words.add("0x" + Long.toUnsignedString(object, 16)); + return words; + } + private static String exclusiveAddress(Address inclusive) { Address next = inclusive.next(); if (next != null && next.getAddressSpace().equals(inclusive.getAddressSpace())) { @@ -3833,6 +4307,14 @@ public final class ReaGhidraBridge extends HeadlessScript { private record AddressNameRequest(String inputAddress, Address address, String name) {} private record FunctionEntry(Function function, InventoryItem item) {} private record FunctionAnalysisObservation(JsonObject dossier, List limitations) {} + private record SwiftStringLiteralDecode(JsonArray literals, JsonArray omissions) {} + private record SwiftLiteralLayout( + String representation, + Address address, + long byteLength, + boolean ascii, + String smallValue + ) {} private record InstructionScan(List instructions, boolean truncated) {} private record SessionDescriptor( String transport, diff --git a/bridge/ghidra/ReaGhidraPrepareRaw.java b/bridge/ghidra/ReaGhidraPrepareRaw.java index 1a5db9627..143331390 100644 --- a/bridge/ghidra/ReaGhidraPrepareRaw.java +++ b/bridge/ghidra/ReaGhidraPrepareRaw.java @@ -1,6 +1,8 @@ import ghidra.app.script.GhidraScript; import ghidra.program.model.address.Address; import ghidra.program.model.listing.Program; +import ghidra.program.model.mem.MemoryBlock; +import ghidra.program.model.mem.MemoryBlockSourceInfo; /** Seed the caller-supplied raw-binary entry before default auto-analysis. */ public final class ReaGhidraPrepareRaw extends GhidraScript { @@ -21,8 +23,19 @@ public final class ReaGhidraPrepareRaw extends GhidraScript { ); } Address entry = currentProgram.getAddressFactory().getDefaultAddressSpace().getAddress(entryHex); - Address start = currentProgram.getMinAddress(); - if (entry == null || !entry.equals(start)) { + MemoryBlock block = entry == null ? null : currentProgram.getMemory().getBlock(entry); + boolean fileStart = false; + if (block != null) { + // Processor-defined zero page/stack blocks can precede the imported file. + for (MemoryBlockSourceInfo source : block.getSourceInfos()) { + if (source.getFileBytes().isPresent() && source.getFileBytesOffset() == 0 && + entry.equals(source.getMinAddress())) { + fileStart = true; + break; + } + } + } + if (!fileStart) { throw new IllegalArgumentException( "Raw binary import must map file offset 0 at the caller-supplied entry" ); diff --git a/bridge/mitmproxy/capture.py b/bridge/mitmproxy/capture.py index fa0535c02..dcd90e745 100644 --- a/bridge/mitmproxy/capture.py +++ b/bridge/mitmproxy/capture.py @@ -93,7 +93,7 @@ def project_record(state, secrets): safe_bytes, credential_url = transport_value(value, path, transport_url) content_base64 = base64.b64encode(value).decode("ascii") digest = hashlib.sha256(value).hexdigest() - hidden = any(secret in value or secret in safe_bytes for secret in byte_secrets) or any(secret in content_base64 or secret in digest for secret in secrets) + hidden = any(secret in value or secret in safe_bytes for secret in byte_secrets) or any(secret in content_base64 or secret in digest or secret in str(len(value)) for secret in secrets) safe_text = None try: safe_text = safe_bytes.decode("utf-8", errors="strict") diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts index 5a357c80f..001c07fbe 100644 --- a/docs/.vitepress/config.ts +++ b/docs/.vitepress/config.ts @@ -37,6 +37,10 @@ export default defineConfig({ link: "/javascript-application-workflows", }, { text: "Browser observation", link: "/browser-observation" }, + { + text: "Web network captures", + link: "/web-network-captures", + }, { text: "Electron observation", link: "/electron-observation" }, { text: "Process capture", link: "/process-capture" }, { text: ".NET assemblies", link: "/managed-code-analysis" }, diff --git a/docs/adb-device-analysis.md b/docs/adb-device-analysis.md index e5a95c536..12fd77cd2 100644 --- a/docs/adb-device-analysis.md +++ b/docs/adb-device-analysis.md @@ -136,8 +136,8 @@ existing analysis families directly, for example ## Resource and lifecycle limits - Caller values reach adb only as positional arguments in fixed vectors; - no tool accepts an arbitrary shell command line, and UI interaction - (`input`, `am`, `monkey`) is deliberately not exposed. + no tool accepts an arbitrary shell command line, `input`, or `monkey`. + The app/activity lifecycle tools use `am start -W` and `am force-stop`. - Per-invocation budgets: client/devices 10-20 s, shell reads 20-60 s with 64 KiB-8 MiB capture caps (logcat 4 MiB, screen 32 MiB), transfers and bugreports 600 s. Exceeded budgets report `resource_constraint`; stopped diff --git a/docs/android-analysis.md b/docs/android-analysis.md index 5a4a2f879..d98d9b5da 100644 --- a/docs/android-analysis.md +++ b/docs/android-analysis.md @@ -9,6 +9,27 @@ This family is present on repository main and in npm 4.1.0. Check the [released package boundary](installation.md#released-package-and-main) and the connected server's tool list before selecting it. +## Optional package delivery + +The MCP `inspect_android_package` request keeps complete output when `detail` is +omitted or `"complete"`. Select `detail: "summary"` explicitly to receive all +package, SDK and permission declarations, engine identity and class/resource +counts without decoded manifest text or duplicate raw provider replies. A +retaining session keeps the complete original Evidence and returns a distinct +linked `inspect_analysis_view` summary. This changes delivery, not JADX analysis. + +Use the returned `normalized_result.parent_evidence_id` with +`inspect_analysis_view`, source kind `retained-evidence`, and +`view: {kind: "android", facet: "manifest", offset: 0, limit: 4096}` to read the +exact decoded manifest without reopening the APK. Offsets and limits count UTF-16 +code units; follow `coverage.next_offset`. The page preserves `source_status` and +`reported_total_bytes`: exhausting retained text does not make a partial decoded +manifest complete. Complete source Evidence remains exportable under the existing +session lifetime. Servers without retention reject summary before starting JADX. +One-shot CLI output remains complete. Older servers require the existing complete +request after schema discovery. APK authenticity and runtime behavior remain +unverified. + ## Supply tools Use a **full JDK 17 or newer** (including its compiler module) and diff --git a/docs/apple-application-analysis.md b/docs/apple-application-analysis.md index e00454c3e..666717fc5 100644 --- a/docs/apple-application-analysis.md +++ b/docs/apple-application-analysis.md @@ -174,7 +174,7 @@ Resolution follows dyld: - `@rpath` tries each `LC_RPATH` of the loading image first, then those of the image that loaded it, up to the process root. - Dependencies load dependents-first in load-command order, and each image - loads once per process. A later request whose install name matches an + loads once per process. A later @rpath request whose install name matches an already loaded image reuses it. - Symlinks such as `Versions/Current` are followed inside the analyzed root. An escape through `..` or an absolute link is `escapes-target`. diff --git a/docs/binary-diagnostics.md b/docs/binary-diagnostics.md index 29e10a33c..0e6a3f196 100644 --- a/docs/binary-diagnostics.md +++ b/docs/binary-diagnostics.md @@ -98,8 +98,10 @@ section, symbol, mitigation or linkage views. The default `complete` detail and CLI output are unchanged. Complete results have a 32 MiB input, 64 MiB reply, 1 MiB combined diagnostics -and 30-second owned command deadline. The reply budget applies to the decoder -record; the CLI/MCP Evidence envelope adds copies and encoding overhead. The +and 30-second owned command deadline. `REA_PWNTOOLS_MAX_INPUT_BYTES` raises the +input bound alone, up to 1 GiB; the reply and deadline bounds are unchanged. The +reply budget applies to the decoder record; the CLI/MCP Evidence envelope adds +copies and encoding overhead. The pinned MCP SDK defaults to a 10 MiB receive buffer. Large complete results need a caller-configured `StdioClientTransport({ maxBufferSize: ... })` and a matching server `REA_MCP_MAX_RESPONSE_BYTES`. See [MCP delivery budgets](mcp-contracts.md#tool-results). diff --git a/docs/browser-observation.md b/docs/browser-observation.md index 2e3d8e37c..dca1f994b 100644 --- a/docs/browser-observation.md +++ b/docs/browser-observation.md @@ -8,6 +8,10 @@ exported script's native imports under an explicit URL/import-map context. REA can attach to a user-owned Chrome-family browser through the Chrome DevTools Protocol (CDP) and produce bounded Evidence about an existing page. This is a passive reverse-engineering capability, not a general browser automation or remote-control surface. +For request/response bodies and traffic debugging, see +[Fiddler Everywhere captures and MCP](web-network-captures.md#reading-fiddler-captures-saz). +Inspect native SAZ or HAR 1.2 offline with REA, or connect Everywhere's separate MCP server. + ## Shipped surfaces - `list_browser_targets` / `rea list-browser-targets` discovers all eligible page targets at the endpoint by default, with an optional exact-origin filter. @@ -76,6 +80,27 @@ google-chrome \ REA does not launch, own, or terminate this browser. Use a dedicated profile and stop it yourself when the investigation is complete. Do not expose the debugging port on a non-loopback interface. +Other Chromium-family browsers, such as Chromium or Brave, work the same way +with the same two flags. On macOS, run the executable inside the application +bundle, for example +`"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"` or +`"/Applications/Brave Browser.app/Contents/MacOS/Brave Browser"`. Fedora installs +Chromium as `chromium-browser`. + +REA needs the browser's HTTP discovery endpoint on a loopback port. A browser +driven over `--remote-debugging-pipe` or a private channel, for example by an +automation SDK, exposes no such endpoint, so `list_browser_targets` has nothing +to attach to. Connection failures report `endpoint_unreachable`; a reachable +HTTP service that returns a non-200 or non-JSON discovery response reports +`invalid_endpoint_response`. +Reproduce that flow in a Chromium-family browser started with +`--remote-debugging-port`. + +In a minimal Linux container, headed Chromium under Xvfb can keep running while +its DevTools port never opens. Install the GLES, EGL, and Mesa DRI libraries +(on Fedora: `libglvnd-gles`, `mesa-libEGL`, and `mesa-dri-drivers`), then +restart the browser. + On Linux, a reachable but locked desktop keyring can stall Chrome before its first navigation. For a dedicated investigation profile, add `--password-store=basic` to bypass that keyring. Empty or placeholder frame URLs @@ -203,7 +228,7 @@ commands fail closed. A correlated reply's session ID must match the command's selected session. An optional method is unavailable only when the browser rejects it as unknown; other command rejections retain their reported failure. -Network and console coverage starts only after REA attaches and enables the relevant CDP domains. `prior_activity_available` is always `false`; absence from these arrays is not evidence that an event never occurred. If CDP reuses a request ID without supplying redirect metadata, REA retains the predecessor observation and marks network coverage incomplete. +Network and console coverage starts only after REA attaches and enables the relevant CDP domains. `prior_activity_available` is always `false`; absence from these arrays is not evidence that an event never occurred. Scripts are different: scripts the page parsed before REA attached are still reported under `scripts`. A page that loaded before attach can therefore list its scripts while `network.requests` lacks the requests that fetched them. If CDP reuses a request ID without supplying redirect metadata, REA retains the predecessor observation and marks network coverage incomplete. Source maps are not fetched by inspection. Bundle analysis fetches them only when `fetch_source_maps` is selected, with exact-origin redirect checks and no credentials, cookies, or referrer. Each fetch operation has a 30-second deadline and a 64 MiB aggregate streamed-response budget across redirects. A deadline or byte-limit failure remains explicitly unavailable and makes bundle completeness partial. Source-map decoding also bounds raw resolved identities to 64 MiB and aggregate inline evidence to 32 MiB. Failed maps preserve their failure context when it fits; if another context cannot fit, collection stops with the full requested count, the returned processed count, and an explicit unknown-results limitation. Retained mappings and original sources are returned inline. See [source-location limits](web-source-location.md) for the structural decoder bounds. Original source artifacts with `.ts`, `.tsx`, `.mts`, or `.cts` suffixes carry `text/typescript`; other source names retain `text/javascript`. diff --git a/docs/cli.md b/docs/cli.md index dcd949e11..6d81ec7c1 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -174,7 +174,9 @@ rea analyze /absolute/path/to/program --provider ghidra --snapshot /absolute/pat rea analyze /absolute/path/to/program --provider ghidra --snapshot /absolute/path/to/analysis/program.json ``` -An exact CLI cache hit is read before starting a provider process. +An exact CLI cache hit is normally read before starting a provider process. +On Windows, Ghidra first admits the selected source through its native snapshot +owner, so replay can start that provider. In MCP, `open_binary` accepts `snapshot_path` to import a snapshot atomically for its matching target; an MCP provider may still start before returning a cached result. `close_binary` accepts `snapshot_path` and optional diff --git a/docs/client-configuration.md b/docs/client-configuration.md index c0e629d66..e27866f7d 100644 --- a/docs/client-configuration.md +++ b/docs/client-configuration.md @@ -6,6 +6,12 @@ for the reviewed setup transaction and [client IDs](installation.md#supported-ag The versioned verification notes below describe their stated runs; live service acceptance is verified only where explicitly recorded. +## Fiddler Everywhere alongside REA + +Connect Fiddler Everywhere's MCP server separately for HTTP traffic debugging, +preserving your REA client entry. See the +[setup instructions and public skills](web-network-captures.md#fiddler-everywhere-mcp-and-public-skills). + ## GitHub Copilot CLI For GitHub Copilot CLI 1.0.95, `gpt-4.1` model metadata can block ordinary @@ -95,6 +101,11 @@ Setup and doctor follow Qwen Code's tilde and working-directory-relative home overrides. Qwen Code also discovers personal skills in `~/.agents/skills`, where setup installs the bundled workflow. +## Antigravity + +Antigravity 2.0 discovers personal skills in `~/.gemini/config/skills`. +Setup installs its selected skill copy there. + ## Gemini CLI Gemini CLI uses `$GEMINI_CLI_HOME/.gemini/settings.json` when that home override @@ -154,6 +165,14 @@ For [Qoder](https://docs.qoder.com/cli/mcp-reference), setup writes from `~/.qoder/skills` (following `QODER_CONFIG_DIR` when set), so setup installs its skill copy there. +## Factory Droid + +For [Factory Droid](https://docs.factory.com/harness/mcp), setup writes +`mcpServers.rea` into the user-level `~/.factory/mcp.json`. Entries carry +`command`, `args`, and `env`. Droid discovers personal skills from +`~/.factory/skills` ([skills](https://docs.factory.com/harness/skills)), so +setup installs its skill copy there. + ## Trae For [Trae](https://docs.trae.ai/ide/add-mcp-servers), setup writes @@ -198,8 +217,15 @@ replaces) the REA entry in the home patch `$DSH_HOME/cordis.patch.yml` patches and comments. `rea doctor --client dsh` checks that same entry id. `rea uninstall` removes it by scanning supported clients; uninstall takes no `--client` flag. The home patch applies to every Harness profile, so the -registration is written once and no per-profile edit is needed. The written -entry is: +registration is written once and no per-profile edit is needed. + +Setup, doctor and uninstall resolve `DSH_HOME` and `DSH_AGENTS_HOME` as Harness +does: unset, empty or whitespace-only values use `~/.dsh` and `~/.agents`, +respectively. Bare `~`, `~/` and `~\` expand to the user home; other relative +values resolve against the command's working directory. Nonblank values retain +their original spaces, and the agents root is independent of the Harness home. +Run REA and Harness from the same working directory when using relative overrides. +The written entry is: ```yaml - insert: diff --git a/docs/client-verification.md b/docs/client-verification.md index a02c67edf..d9ece9aad 100644 --- a/docs/client-verification.md +++ b/docs/client-verification.md @@ -18,14 +18,26 @@ pass `-- chat` to check plain chat with REA enabled. Qwen also reads the full result that its client offloads, and Pi exercises default codemode execution. Use `REA_VERIFY_RUNTIME_ROOT` to select a production-only installed REA package. Receipts retain client versions, result digests, host coverage and owned-process -cleanup. These POSIX lanes use a deterministic loopback model, so they do not -prove live model-provider or native Windows compatibility. Qwen, Pi and Hermes +cleanup. These lanes use a deterministic loopback model and do not prove live +model-provider compatibility. Pi and Hermes remain POSIX-only. Qwen, Pi and Hermes bind both `HOME` and `USERPROFILE` to the disposable account. Qwen and Pi discover the shared personal skill through their default search paths without an explicit skill setting; Hermes discovers it through the selected `HERMES_HOME`. Set `REA_VERIFY_HERMES_STICKY_PROFILE=1` to exercise native Hermes selection of a named sticky profile with `HERMES_HOME` still pointing to its root. +The Qwen lane also supports native Windows (verified with Qwen Code 0.25.1 and +Node 24.19.0 for both `chat` and `call`). With Node 24.18.0, the same `call` +journey reached full Evidence readback but the client aborted during exit with +`UV_HANDLE_CLOSING`; that run remains a failure, not a compatibility claim. +Select the npm install's +`node_modules/@qwen-code/qwen-code/cli-entry.js` with `REA_VERIFY_QWEN_COMMAND`, +because `execFile` cannot launch the `qwen.cmd` shim directly. A selected `.js`, +`.mjs` or `.cjs` entry point runs under the verifier's Node. Windows path checks +use native separators; persisted output is read through Qwen's real shell tool +with a shell-independent encoded Node payload. Windows receipts report process +lineage as unavailable when the verifier has no retained native Job Object lease. + `verify:gemini-client` requires an installed Gemini CLI (verified with `@google/gemini-cli@0.63.0` on POSIX and native Windows); select it with `REA_VERIFY_GEMINI_COMMAND`. A selected `.js`, `.mjs` or `.cjs` entry point runs @@ -298,13 +310,42 @@ subsequent wire recovery and clean shutdown. Caller timeout does not interrupt Hopper's synchronous native operation. Socket boundary tests deterministically cover active and queued expiry, late replies and timer cleanup. -The Linux demo lane remains a separate `verify:hopper:linux` command. Its -lifecycle checks attempt competing CLI launches for the active target and a +The Linux demo lane remains a separate `verify:hopper:linux` command. Hopper +6.4.2's ELF loader did not recognize the original 4,119-byte literal or three +non-ASCII byte strings in the measured fixtures. The lane retains complete raw +byte checks for those literals through source-owned symbols. Recognition of the +original full-length literal is bound to its exact source address; its decoded +object coverage remains explicitly unverified on Linux even when recognized. +Absent non-ASCII recognition also reports decoded-object coverage as unverified. +Recognized non-ASCII and control-text objects must still match their source +bytes, encoding, filters and searches. +A separate 548-byte ASCII literal requires native fragmentation, shortened +provider display, complete decoded fragments and dossier provenance on Linux; +the macOS lane retains the original full-length recognition requirement. These +are observed fixture/build capabilities, not universal Hopper length limits. +Its lifecycle checks attempt competing CLI launches for the active target and a different target, requiring rejection before either can change the owning MCP session's documents or procedures. Lease boundary tests also preserve unresponsive or malformed live endpoints while allowing confirmed stale sockets to recover. +`verify:setup:linux` exercises installation on a clean supported Linux runner as +root. Use a disposable VM: this lane installs Hopper and its Linux runtime +dependencies through the system package manager. Set `REA_VERIFY_SERVER_COMMAND` +to the CLI entrypoint being verified. The command builds and checks source-owned +ELF fixtures before testing the read-only Hopper setup plan, approved +installation, scoped doctor, real CLI/MCP analysis, rejection of a modified +launcher, idempotent setup without requesting a reinstall, and owned child +cleanup. Before installation and after teardown, a clean-runner process scan +also rejects surviving Hopper demo helpers or Xvfb groups, including orphans +whose launcher exited. This scan observes processes without signaling them. +Hopper detection compares executable file identity through `/proc`, covering +alternate or configured launcher paths even after an invoked symlink is removed. +The dedicated runner must have no unrelated Hopper/Xvfb sessions. It requires +the repository's pinned development toolchain and an +existing host C compiler, `file`, `nm`, and `strings`. Use `verify:hopper:linux` +instead when Hopper is already installed. + `verify:hopper:fat` is a separate macOS lane requiring installed Hopper and the existing Xcode clang/lipo toolchain. It compiles arm64/x86-64 thin executables and one- and two-slice FAT32 containers, verifies exact/interior address mappings diff --git a/docs/flutter-analysis.md b/docs/flutter-analysis.md index 75c23dd66..ebf0ccd84 100644 --- a/docs/flutter-analysis.md +++ b/docs/flutter-analysis.md @@ -25,6 +25,26 @@ The equivalent MCP methods are `identify_flutter_build` (with `path`) and arm64-v8a, then x86_64, then x86, then armeabi-v7a, then the remaining ABIs alphabetically). +## Optional retained AOT delivery + +MCP `inspect_dart_aot` keeps complete output when `detail` is omitted or +`"complete"`. Explicit `detail: "summary"` returns the selected target/ABI/library +identity, all snapshot headers, structural coverage, string-pool coverage and +original lower-bound counts without the string lists. It retains complete source +Evidence and returns a distinct linked `inspect_analysis_view` summary. Analysis +and the caller's ABI choice are unchanged; one-shot CLI output stays complete. + +Use `normalized_result.parent_evidence_id` as the retained source for +`inspect_analysis_view`, with `view: {kind: "dart-aot", collection: "package_uris", +offset: 0, limit: 64}`. Collections are `package_uris`, `dart_uris` and +`class_like_tokens`. Pages preserve the original strings and order, selected ABI, +library identity and source coverage. `returned_counts` describe producer-returned +rows; counts ending in `_at_least` remain lower bounds. Exhausting a page sequence +does not make a partial pool complete or turn byte-pattern tokens into typed +classes/functions. Views do not reopen the APK. Original Evidence remains +exportable under its existing session lifetime. Summary requires a retaining +server; older servers keep the complete request after schema discovery. + ## What the AOT inspection returns `inspect_dart_aot` reads one ABI's `libapp.so` in full and reports: diff --git a/docs/ghidra-mips.md b/docs/ghidra-mips.md index 59d04a8b0..d5b58f00b 100644 --- a/docs/ghidra-mips.md +++ b/docs/ghidra-mips.md @@ -111,7 +111,7 @@ MIPS header and ABI record facts are committed in the analysis profile so interpretation changes invalidate profile-bound snapshots. The support-lane identity is versioned independently of file identity. Evidence, lifecycle outputs and saved snapshots preserve the `mips` family. Hopper and IDA adapters -do not implicitly gain MIPS support. Windows Ghidra P0 remains PE x86/x86-64 only. +do not implicitly gain MIPS support. Windows Ghidra P0 does not admit MIPS targets. ### Soft-float prototype diff --git a/docs/ghidra-provider.md b/docs/ghidra-provider.md index a79afecbd..5b83ee152 100644 --- a/docs/ghidra-provider.md +++ b/docs/ghidra-provider.md @@ -41,6 +41,80 @@ does not prove text is absent. Both list and search Evidence carry this limitation. When an address is known, use `read_bytes` and `xrefs` to inspect bytes and references. +## Loaded addresses and file offsets + +`rea analyze --provider ghidra --json` and MCP `binary_overview` report +the observed loaded `image_base` in `normalized_result`. They retain each +segment's reported `image_base` and `address_space`. A provider that does not +report a base, or whose segments disagree, omits the overview field, following +the existing contract on main. REA does not substitute zero or the first +segment's address. All reported bases and address spaces remain visible in the +complete segment inventory, including conflicting observations. + +`function` and most MCP procedure selectors use procedure names or +provider analysis addresses. Ghidra `decompile` / MCP `procedure_pseudo_code` +also accept source-file byte offsets. Existing symbols and analysis addresses +take precedence; a `0x`-prefixed number that resolves to no procedure falls back +to its observed source mapping. Use `--file-offset=0x...` on the CLI or +`procedure: "file-offset:0x..."` through MCP to select file coordinates explicitly. +Multiple mapping candidates fail as ambiguous rather than selecting a bank or +address space. A successful conversion retains the file offset, mapped address +and procedure entry in raw Evidence and `locations`. + +If an offset maps to loaded target bytes but no procedure starts at or contains +the mapped address, the failure preserves both the original file offset and +analysis address. This differs from an unmapped offset and does not recommend +retrying the same offset syntax. +If the function resolves but the backend fails or cancels decompilation, the +failure retains the selector, mapped address and procedure entry without +changing the backend failure code. + +Both mapping directions require a complete original `FileBytes` source whose +SHA-256 matches the admitted target. Auxiliary files and partial source objects +remain visible in the load-image inventory but do not establish offsets in the +original target. Ghidra's object-local offsets are not silently treated as +original-file coordinates. Original-source identity is checked once per source +object per session; analysis-memory edits do not change the original bytes. + +ELF virtual addresses and file offsets are +different coordinates. Ghidra can rebase a PIE ELF during import: in the +source-owned verification fixture, ELF entry `0x1013` becomes Ghidra procedure +`0x101013` with image base `0x100000`. That example is not a general conversion +rule. + +Use `list_procedures` to select an observed procedure address. To relate file +bytes to loaded memory, call MCP `inspect_native_load_image` or: + +```sh +rea search . --kind procedures --mode regex --provider ghidra --json +rea inspect-native-load-image --provider ghidra --json +rea decompile --provider ghidra --json +rea decompile --file-offset=0x1013 --provider ghidra --json +rea address-to-file-offset --provider ghidra --json +``` + +The load-image result preserves source-file indexes, file offsets, address +spaces, and inclusive mapping ranges. Follow the mapping for the selected +source file and region; adding `image_base` to an arbitrary file offset is not +valid. For a linear byte-addressed, file-backed mapping, a file offset `F` in +`[mapping.file_offset, mapping.file_offset + mapping.length)` maps to +`mapping.start + (F - mapping.file_offset)` in the reported `address_space`. +If several mappings cover the same file offset, keep those candidates distinct +and select the procedure from the observed inventory. Uninitialized regions may +have no file backing. ELF mappings are observed +without independent format-specific load-image verification, as the result's +status and limitations state. + +An unsuccessful numeric procedure lookup reports the current Ghidra image base +and gives the CLI commands above alongside the corresponding MCP operations. +Repeat any target-format, architecture, and entry-point options used to open the +target. Other procedure operations keep their analysis-address semantics; +file-offset fallback applies only to decompilation. + +Overview workflow revision 3 binds this output to snapshot identity. Older +serialized overview Evidence remains readable; it is not replayed as a current +overview merely because its source artifact is unchanged. + ## Admitted function-analysis semantics | Concern | Ghidra contract | @@ -57,9 +131,10 @@ bytes and references. Function extents and instruction completeness refer to Ghidra's analyzed body. An inferred no-return call can terminate that body before subsequent bytes. -Pseudocode, assembly, procedure info, instruction windows, and dossiers report -observed terminal call sites in Evidence limitations, including resolved callee -names, addresses, and FunctionManager flags when available. Instruction windows +Pseudocode, assembly, procedure info, instruction windows, callee and caller +reads, and dossiers report observed terminal call sites in Evidence limitations, +including resolved callee names, addresses, and FunctionManager flags when +available. Instruction windows and dossiers also retain these warnings in their result limitations. An unresolved callee or unestablished terminal-flow basis remains explicit. These observations qualify the analysis; they do not prove that a no-return flag is diff --git a/docs/ghidra-riscv.md b/docs/ghidra-riscv.md index e180122d7..afae01948 100644 --- a/docs/ghidra-riscv.md +++ b/docs/ghidra-riscv.md @@ -11,8 +11,8 @@ ELF `EM_RISCV` (243) is retained as the provider-neutral `riscv` family. The target also retains the raw ELF declarations that decide a lane: class, byte order, `e_type`, and `e_flags`. Recognition does not mean every provider or RISC-V variant is supported. Existing PE, managed and Mach-O architecture sets -are unchanged, and caller-supplied headerless raw-binary entry still accepts only -its existing four families. +are unchanged. Headerless inputs use the separate [raw-binary admission +boundary](raw-binary.md); raw RISC-V images are not supported. The Ghidra lane admits little-endian ELF32 and ELF64 `ET_EXEC` images that declare `EF_RISCV_FLOAT_ABI_DOUBLE` in `e_flags`, with no `EF_RISCV_RVE`, @@ -22,16 +22,16 @@ header declarations, not proof that every instruction conforms to that ISA. Provider selection returns the specific unsupported constraint rather than guessing a compatible profile: -| Declaration | Outcome | -| ----------------------------------------------- | --------------------------------------------------------------------------- | -| Little-endian ELF64 `ET_EXEC`, double-float ABI | admitted, `RISCV:LE:64:default` + `gcc` | -| Little-endian ELF32 `ET_EXEC`, double-float ABI | admitted, `RISCV:LE:32:default` + `gcc` | -| Big-endian | refused; Ghidra 12.1.4 ships no big-endian RISC-V language | -| `ET_REL`, `ET_DYN`, other `e_type` | refused with the declared value | -| `EF_RISCV_RVE` | refused; the embedded 15-register ABI needs its own processor specification | -| `EF_RISCV_TSO` | refused; the total-store-order memory model is unverified here | -| Soft, single, or quad float ABI | refused; the pinned generic compiler specification models double-float | -| Reserved or newer `e_flags` bits | refused with the offending bits named | +| Declaration | Outcome | +| ----------------------------------------------- | ---------------------------------------------------------------------------------------- | +| Little-endian ELF64 `ET_EXEC`, double-float ABI | admitted, `RISCV:LE:64:default` + `gcc` | +| Little-endian ELF32 `ET_EXEC`, double-float ABI | admitted, `RISCV:LE:32:default` + `gcc` | +| Big-endian | refused; Ghidra 12.1.4 ships no big-endian RISC-V language | +| `ET_REL`, `ET_DYN`, other `e_type` | refused with the declared value | +| `EF_RISCV_RVE` | refused; the embedded 16-register profile (x0–x15) needs its own processor specification | +| `EF_RISCV_TSO` | refused; the total-store-order memory model is unverified here | +| Soft, single, or quad float ABI | refused; the pinned generic compiler specification models double-float | +| Reserved or newer `e_flags` bits | refused with the offending bits named | `e_type` is the only load-shape declaration this lane reads. A fixed-address `ET_EXEC` declaration does **not** establish static linkage: `PT_INTERP` and diff --git a/docs/ida-provider.md b/docs/ida-provider.md index b22a35cce..fbf67c8a2 100644 --- a/docs/ida-provider.md +++ b/docs/ida-provider.md @@ -88,6 +88,8 @@ REA copies the admitted binary into a private temporary directory, checks its di One REA session owns one database. Stdio registrations default `IDA_MCP_MAX_WORKERS` to `1`; the explicit upstream `--max-workers 1` also bounds a supervisor. Database size and engine memory depend on the target and upstream; REA does not impose an engine memory limit. The original binary is never executed or overwritten. Optional `workspaceRoot` selects an existing absolute temporary-directory parent on the REA host. Ensure it has space for the input copy and IDA working files. +When a completed `idb_open` response reports success or is otherwise ambiguous, but no active owned private worker can be verified, REA retains the private workspace even if `idb_list` appears empty. A later close can retry verification if the original owned worker becomes discoverable. A completed explicit `idb_open` rejection retains the existing no-worker cleanup path. This is a local admission/ownership safeguard and **does not** provide a process-termination receipt or change the legacy reaped-worker protocol. + Headless cleanup uses the upstream database lifecycle. Abruptly terminating REA, its proxy, or the supervisor does not establish worker release. Prefer `close_binary` for MCP sessions; after an interrupted process, inspect upstream session inventory and retained workspace diagnostics before removing working files. This adapter does not claim Windows Job Object containment for the upstream detached workers. ## Connect an agent once @@ -126,6 +128,8 @@ For guided registration, set `REA_IDA_MCP_CONFIG` before `rea setup --client .recovery` copy of the current bytes. +Both snapshots remain available; a failed rollback reports the current recovery +copy's path and the restore failure. Setup also retains valid environment overrides in REA's existing server entry, such as `REA_MCP_INPUT_SCHEMA_PROFILE` and `REA_MCP_MAX_RESPONSE_BYTES`. Detected provider settings replace only matching environment keys. @@ -101,13 +105,11 @@ rea setup --yes --all-detected --install-hopper --json ### Skill destinations -The bundled skill is installed where each selected client discovers personal -skills: Claude Code uses `~/.claude/skills` (or -`$CLAUDE_CONFIG_DIR/skills` when configured). Gemini CLI uses `.agents/skills` -under its selected home; Antigravity 2.0 uses `~/.gemini/config/skills`. -Other integrations, including Qwen Code, use `~/.agents/skills`. -A mixed selection plans each distinct destination. Selecting the skill without a client uses the shared -directory. Setup leaves existing skill copies in other locations untouched. +The bundled skill is installed in each selected client's personal skill directory. +See [client configuration](client-configuration.md) for client-specific roots, +profiles, and environment overrides. A mixed selection plans each distinct +destination. Selecting the skill without a client uses `~/.agents/skills`. +Setup leaves existing skill copies in other locations untouched. `rea setup --skill-client claude_code --yes --json` refreshes and verifies the selected skill destination independently of MCP registration selection. The @@ -281,6 +283,7 @@ hosted connector; see [its configuration](client-configuration.md#other-client-s | Pi | `pi` | | Hermes | `hermes` | | Qoder | `qoder` | +| Factory Droid | `droid` | | Trae | `trae` | | ZCode | `zcode` | | DeepSeek Harness | `dsh` | @@ -319,8 +322,8 @@ The Hopper launcher action used by REA creates a document from an executable; its supported command-line interface does not attach to an already-open document. While another live REA session owns the same target and loader profile, a second session reports the owning run ID instead of opening a -duplicate document. Closing the owning REA session releases this guard; Hopper -keeps its document open. A later REA session can therefore open another +duplicate document. On macOS, closing the owning REA session releases this guard; +Hopper keeps its document open. A later REA session can therefore open another document for the same target. REA does not currently identify, focus, or reuse that existing GUI document, and the supported launcher exposes no attach or reuse action for it. @@ -343,8 +346,14 @@ Hopper application per user across cooperating REA processes. A competing CLI or MCP session receives the owning session ID before Hopper is launched, for both the same target and a different target. Switch targets through that MCP session, or close it before starting another session. Closing the owner -releases the reservation. This does not attach to or claim ownership of -manually opened Hopper applications. +releases the reservation and stops REA's owned demo application and private +display; the headless Linux demo does not retain documents after MCP shutdown. +Its helper watches a private lifetime pipe, so an owner crash or SIGKILL also +stops those children, without relying on a reusable PID or a process-name scan. +A later session recovers the dead owner's stale lease after child cleanup; +private runtime files can remain when REA itself was killed. This does not attach to +or claim ownership of manually opened Hopper applications. An application left +running by an older REA version must still be closed before starting a new session. Stale lease recovery is serialized. If an interrupted recovery leaves its reservation directory, the diagnostic identifies that exact path. Stop REA @@ -437,8 +446,9 @@ For cursor navigation, string decoding, symbol-name limits, and regex matching, see [Hopper analysis behavior](native-investigation.md#hopper-analysis-behavior). Closing or switching a target closes its bound Hopper document, shuts down REA's -bridge and removes its temporary socket directory while preserving the Hopper -application and unrelated documents. A `cleanup_incomplete` +bridge and removes its temporary socket directory. The macOS Hopper application +and unrelated documents remain open; the owned Linux demo application stops. +A `cleanup_incomplete` result identifies resources whose cleanup could not be verified. REA retains unresolved cleanup ownership and any confirmed shutdown phases. After addressing the reported failure, retry `close_binary` on the same @@ -503,7 +513,10 @@ and `set_addresses_names` name a function entry or any other mapped address session metadata edits leave executable bytes unchanged and are discarded on close. GUI controls require Hopper; Windows P0 remains read-only. -Windows P0 admits native x86 and x86-64 PE applications and DLLs on fixed local NTFS volumes. +Windows P0 admits native x86 and x86-64 PE applications and DLLs, plus explicitly +selected [raw NMOS 6502 images](raw-binary.md#nmos-6502) with a caller-supplied +entry at file offset 0, on fixed local NTFS volumes. Other raw CPU families remain +unsupported on Windows. The npm package bundles native Job Object ownership, protected private runtime DACLs, and handle-based path admission; no separate addon installation is needed. See the [Windows Ghidra P0 guide](windows-ghidra-p0.md) for verified scope and diff --git a/docs/javascript-application-graph.md b/docs/javascript-application-graph.md index 2290491a5..48e17bf44 100644 --- a/docs/javascript-application-graph.md +++ b/docs/javascript-application-graph.md @@ -104,6 +104,15 @@ static inferences. Dynamic or ambiguous IPC is retained without a pairing edge. Runtime entities retain passive-CDP authority, while a successful static/runtime correspondence is a separately identified cross-layer inference. +Source module links retain their resolution mode. Static imports in `.ts` and +`.tsx` are `compiler-dependent` when compiler options have not been retained. +REA compares Node import and require resolution: a shared outcome remains +available with a limitation; divergent outcomes remain unavailable with both +conditional branches and their limitations. This models source relationships, +not verified emitted code or execution. A declaration file found only through +an emitted-extension fallback is retained as a candidate, not a runtime target; +an explicitly selected path keeps its original identity. + ## Evidence and authority Every observation and edge carries all of the following: diff --git a/docs/javascript-application-workflows.md b/docs/javascript-application-workflows.md index af0b6ba94..e9fed813f 100644 --- a/docs/javascript-application-workflows.md +++ b/docs/javascript-application-workflows.md @@ -23,6 +23,14 @@ artifact digest and structural graph ID. Semantic tracing requires the semantic relation graph and reports when it is unavailable rather than treating missing data as an empty graph. +Explicit TypeScript `import type` and `export type` declarations remain source +facts and do not create runtime import or export relationships. Type export names +are reported separately from value export names. Inline `type` specifiers can +leave an empty import or re-export with side effects under +[`verbatimModuleSyntax`](https://www.typescriptlang.org/tsconfig/verbatimModuleSyntax.html); +REA retains those dependency candidates and reports the unavailable compiler +options rather than assuming the dependency is erased. + The semantic graph wire format stores shared non-location provenance once in its required `evidence_contexts` table. Each node, relation, fingerprint, and unknown retains its exact `evidence.location` and names the owning context with @@ -156,9 +164,12 @@ to the comparison Evidence in a live session. `compare_source_to_bundle` compares one cryptographically committed `HistoricalSourceGraph` with authenticated application-graph Evidence. The stable scoring model reports every admitted signal and weight: exact source -digest, source-map original path, exact current path, path suffix, basename, and -language extension. Exact digest wins over location inference; weak basename -or extension evidence never forces a mapping. +digest, source-map original path, exact current path, path suffix, Unicode- +equivalent path, basename, and language extension. Exact digest wins over +location inference; weak basename, extension, and Unicode-equivalence evidence +never forces a mapping. Unicode-equivalent spellings are candidate evidence +only because distinct filesystem paths can preserve those spellings as separate +files; multiple canonical-equivalent candidates remain unknown. Each historical source file is classified as `unchanged`, `modified`, `removed`, `split`, `merged`, `duplicated`, or `unknown`. Multiple equally @@ -197,6 +208,25 @@ arrays, and later explicit data properties and methods replace earlier object spread references. A known array prefix does not expose values from a following spread. Accessors, custom or uncertain iteration, unknown computed keys, and unknown spread lengths retain conservative uncertainty. +Getter-local aliases of `this`, including direct destructuring and assignments, +retain their receiver when a returned shared reference is written. These writes +invalidate the earlier literal instead of claiming the getter executed or the +new value is known. Straight-line alias reads retain their capture order; +uncertain control flow remains conservative. A binding outside the getter or +`this` in a nested ordinary function is not attributed to the getter's receiver. +Private reads keep receiver projections separate from the private value's path. +Receiver projections follow known data fields initialized from `this` or earlier +self-referencing fields; static fields retain their declaring class identity. +An alias cycle that repeatedly selects deeper children retains escape uncertainty +at its recovered roots rather than claiming a fixed number of loop iterations. + +Writes through an object getter's returned receiver retain that uncertainty +when the object is an array element, nested inside mixed object/array +containers, or assigned to a previously declared identifier. Saving the getter +result in an alias does not make its old field values immutable. Known array +indices include holes; an element after a spread has only a minimum possible +index. Unrelated properties and known array prefixes remain precise, and a +getter read without a write does not invalidate the receiver's fields. Nested callable returns are not assigned to their parent callable. Projected graph observations carry source ranges but never source text. diff --git a/docs/javascript-runtime-reconciliation.md b/docs/javascript-runtime-reconciliation.md index af59e825b..a0a600574 100644 --- a/docs/javascript-runtime-reconciliation.md +++ b/docs/javascript-runtime-reconciliation.md @@ -1,28 +1,31 @@ # JavaScript static/runtime reconciliation REA can combine an existing static JavaScript Application Graph with one or -more passive browser, Electron page, or Node/Electron V8 Inspector captures without collapsing their different -authorities. The `reconcile_javascript_runtime` MCP tool and +more browser, Electron, or Node/Electron V8 Inspector observations while +preserving their different authorities. The `reconcile_javascript_runtime` MCP tool and `rea reconcile-javascript-runtime` CLI command accept Evidence records and return derived Evidence containing a combined JavaScript Application Graph and explicit match, ambiguity, mismatch, and unknown classifications. The operation performs no filesystem, browser, or Electron I/O. It accepts only semantically verified Evidence produced by `analyze_javascript_application`, -`inspect_web_page`, `inspect_electron_page`, or -`observe_javascript_runtime`; matching operation names alone +`inspect_web_page`, `inspect_electron_page`, `observe_javascript_runtime`, or +`capture_electron_scenario`; matching operation names alone is insufficient. Provider identity, predicate type, subject digest, target parameters, and normalized-result schemas are checked before reconciliation. +`capture_electron_scenario` supplies target-only partial observations. It does +not establish loaded-script coverage; see [Electron observation](electron-observation.md). + ## Workflow 1. Run `analyze_javascript_application` against exactly one application layer. Optional cache or assets directories can be analyzed independently and supplied as additional layers. 2. Run `inspect_web_page`, `inspect_electron_page`, or the attach-only - `observe_javascript_runtime`. Capturing script sources - is optional, but a source capture provides the strongest byte - identity. + `observe_javascript_runtime`, or supply existing `capture_electron_scenario` + Evidence with the partial scope above. Capturing script sources is optional, + but a source capture provides the strongest byte identity. 3. Pass the resulting Evidence records to `reconcile_javascript_runtime`. Browser URLs normally need an explicit URL-prefix mapping. Extracted Electron and V8 Inspector file locations are mapped automatically when runtime files remain @@ -104,7 +107,7 @@ nodes. REA evaluates compatible static candidates conservatively: Every successful cross-layer `observed_as` edge has `authority: cross-layer-reconciliation` and `state: inferred`. The underlying -artifact/AST observations and passive-CDP observations remain unchanged and +artifact/AST observations and runtime observations remain unchanged and retain their own Evidence links. ## Loaded, resident, and not observed diff --git a/docs/managed-code-analysis.md b/docs/managed-code-analysis.md index 69da49e9f..d60603ce3 100644 --- a/docs/managed-code-analysis.md +++ b/docs/managed-code-analysis.md @@ -147,11 +147,33 @@ opposite observation's selection unknown, and application graphs report their selected scope as partial. Neither makes an absence claim about the whole assembly. Older member Evidence without `selection` remains readable. +### Edge target declaring types + +`member_refs[].parent_name`, `call_edges[].target_declaring_type`, and +`field_accesses[].field_declaring_type` name the type that declares each +target. Referenced spec signatures validate encoded type coordinates against +the metadata tables, including nested type arguments and custom modifiers. +TypeRef resolution scopes are checked transitively with cycle detection; +readable names survive malformed scopes as partial evidence. +TypeDef and TypeRef names use the namespace-qualified `full_name` +form, and nested types keep bare names. A generic-instantiation TypeSpec +parent names its generic type, such as ``System.Collections.Generic.List`1``. +Other well-formed TypeSpec forms and ModuleRef or MethodDef parents stay +`null`. A `method-spec` call edge reports the name and declaring type of the +generic method it instantiates. A field access is named only by a field or a +field-signature MemberRef. TypeSpec and MethodSpec blobs are decoded with the +member-signature grammar, and a MethodSpec must instantiate a generic method +with as many type arguments as it declares. Malformed or unsupported rows and +blobs, and arity mismatches, make coverage partial. A readable MethodSpec +method column that names a method, or a TypeRef whose resolution scope is +invalid, keeps its name. +Older member Evidence without these fields reads them as `null`: unknown. + ## Analysis objective Start by identifying the artifact with `inspect_managed_artifact`, then inspect its types and CIL with `inspect_managed_members`. Use [member comparison](#member-comparison) -for another build, and [managed/native boundary rules](#managednative-boundary-rules) +for another build, and [managed/native boundary rules](#managed-native-boundaries) when matching declared native calls to supplied native Evidence. Static inspection does not load or execute the assembly. Reconstructed structure and unresolved native/runtime questions retain their own authority and coverage. @@ -220,6 +242,8 @@ to native function addresses requires verified provider evidence; names or vtable similarity alone do not prove that a native function implements a specific managed method. +
+ ## Managed/native boundary rules `verify_managed_native_boundaries` compares declared P/Invoke module and entry diff --git a/docs/mcp-contracts.md b/docs/mcp-contracts.md index b6990b3c9..f3f7892f6 100644 --- a/docs/mcp-contracts.md +++ b/docs/mcp-contracts.md @@ -21,6 +21,12 @@ directly to `compare_functions`, and `inspect_artifact` Evidence to `compare_artifacts`. Use `get_evidence_bundle` when the task needs broader retained session history or an explicit bundle for transfer. +Evidence and bundle imports reject unsupported fields in their envelopes, +identities, manifests and other fixed metadata before authentication. Imports +leave retained session state unchanged when validation fails. Keep application +metadata in the supported JSON payloads (`parameters`, `raw_result`, +`normalized_result`, or analysis-profile parameters), where keys are preserved. + Tool execution failures return `isError: true` and the complete canonical `{ "error": ... }` diagnostic as JSON in text `content`. Read that diagnostic for the error code, target details, partial observations, and remediation. @@ -141,8 +147,8 @@ after target content was read. ## Retained application Evidence inputs -`analyze_javascript_application`, `inspect_binary_layout`, `inspect_artifact`, and -`inspect_wasm_artifact` accept `detail`: +`analyze_javascript_application`, `inspect_binary_layout`, `inspect_artifact`, +`inspect_wasm_artifact`, and `inspect_macho` accept `detail`: `complete` (default) returns the complete analysis Evidence; `summary` records that complete Evidence in the session and returns the derived `inspect_analysis_view` summary Evidence instead. Its @@ -212,6 +218,19 @@ attempting oversized delivery, and returns the same summary with an exact `parent_evidence_id` for later export or facet reads. Delivery detail does not reach WABT or change the original source Evidence ID. +Completed `inspect_macho` Evidence supports a summary and +`view: {kind: "macho", facet: "exports", offset: 0, limit: 64}`. Facets are +`entrypoints`, `architectures`, `build_metadata`, `load_commands`, +`dependencies`, `imports`, `exports`, `segments`, and `provenance`; pages return +the original rows. The summary keeps the selected slice, header fields and UUID, +and replaces each collection with its returned count, reported total and +`exhaustive` flag. Each page repeats the paged collection's `source_total` and +`source_exhaustive`; exhausting a non-exhaustive collection does not establish +that unreturned symbols are absent. Large frameworks return megabytes of +symbols (Electron Framework: 2.5 MB of imports and exports, 3.3 KB as a +summary), so select `detail: "summary"` on the first call. Delivery detail +never reaches the native tools or changes the source Evidence ID. + `trace_application_feature`, `trace_javascript_semantics`, `compare_application_versions`, `compare_source_to_bundle`, and `compare_javascript_export_shapes` accept complete inline application Evidence diff --git a/docs/native-investigation.md b/docs/native-investigation.md index 27381246b..5a2fcbd31 100644 --- a/docs/native-investigation.md +++ b/docs/native-investigation.md @@ -476,8 +476,11 @@ the tool fails with a `debugger-attach-denied` reason. Install a Ghidra 12.1.x release and the 64-bit full JDK it declares, then configure REA to use them. Ghidra analysis supports Linux x64/arm64 and macOS x64/arm64; each installation requires the matching native decompiler. Experimental Windows x64 P0 admits native -x86 and x86-64 PE applications and DLLs on local NTFS using bundled Job Object ownership, -protected runtime DACLs, and handle-based path admission. See [Windows Ghidra P0](windows-ghidra-p0.md) and +x86 and x86-64 PE applications and DLLs, plus explicitly selected +[raw NMOS 6502 images](raw-binary.md#nmos-6502) with a caller-supplied load address, +on fixed local NTFS volumes. Other raw CPU families remain unsupported on Windows. +The boundary uses bundled Job Object ownership, protected runtime DACLs, and +handle-based path admission. See [Windows Ghidra P0](windows-ghidra-p0.md) and [issue #527](https://github.com/morluto/rea/issues/527). On Linux and macOS, `annotate_native_function` atomically edits a function name and entry comments in the ephemeral database, returning refreshed analysis diff --git a/docs/packet-captures.md b/docs/packet-captures.md index d56a80037..32cedb11b 100644 --- a/docs/packet-captures.md +++ b/docs/packet-captures.md @@ -4,7 +4,7 @@ in an existing capture. Network, USB or Bluetooth recordings can use the same field/filter interface when the installed TShark supports their encapsulation and dissectors. This tool does not acquire traffic, open an interface or control -a device. HAR/mitmproxy web-capture workflows remain separate. +a device. HAR/SAZ/mitmproxy web-capture workflows remain separate. Supply an existing TShark 4.x installation **on Linux**, with util-linux `prlimit` available at `/usr/bin/prlimit`. Override its absolute path with @@ -128,3 +128,49 @@ Each provider admits one inspection at a time, including acquisition and cleanup An occupied provider returns a retryable capacity error before acquiring another file or worker. Closing the provider stops new admission and drains accepted work; failed cleanup remains owned and can be retried. + +## Discover protocols and field names + +`inspect_packet_fields` / `inspect-packet-fields` queries the installed TShark +registration glossary before selecting fields for capture inspection: + +```sh +rea inspect-packet-fields --query usb +rea inspect-packet-fields --kind fields --protocol usb --query idVendor +rea inspect-packet-fields --kind fields --protocol udp --all +``` + +`kind` defaults to `protocols`; `fields` selects field definitions. `protocol` +selects an exact protocol abbreviation (the protocol itself or its child fields). +`query` is a case-insensitive literal substring in name, abbreviation or field +description. `offset` skips matching records; `limit` defaults to 50. MCP +`limit: null` or CLI `--all` requests all remaining matches within the resource +budget. `--all` and `--limit` conflict. The window reports examined records of +the selected kind before filters, matching records, returned records and +`has_more`; it never counts capture packets. + +Returned records preserve field type, parent protocol, display base, bitmask, +description and original glossary line number. Empty producer strings remain +empty; protocol records have null field-only properties. Repeated abbreviations +remain separate registrations. A listed field is an installation capability, not +proof that the field appears in a selected recording. + +The adapter uses the documented `tshark -G fields` grammar under its existing +private empty profile and Linux process limits. TShark 4.6.9's full glossary was +measured at 22,303,197 bytes; discovery admits at most **32 MiB** combined command +output (capture inspection remains at **4 MiB**). Parsing walks lines once and +constructs objects only for the selected page. Returned strings are copied so +retained pages do not retain the complete glossary string. Before constructing a selected +record it charges six times the source UTF-8 bytes plus 512 bytes against an +**8 MiB** representation budget, covering JSON escaping and record bookkeeping. +If that budget would be exceeded, narrow the query or page size; results are not +silently truncated. This is a conservative admission budget, not measured heap +usage. Parent pipe buffers and producer per-process limits remain distinct. + +Tool/limiter identity, version, stderr and selected arguments are retained in +Evidence. No capture, interface or device is opened. Discovery shares capture +analysis's admission slot and cleanup owner. Each call rereads the glossary; +installed plugins, libraries and global configuration remain trusted and are not +fully fingerprinted. There is no cache or snapshot-consistent pagination claim. +The real Linux lane checks public CLI/MCP queries against the direct glossary, +including USB vendor-ID typing, negative queries and paging. diff --git a/docs/pcap-analysis.md b/docs/pcap-analysis.md new file mode 100644 index 000000000..1df3e5ccd --- /dev/null +++ b/docs/pcap-analysis.md @@ -0,0 +1,118 @@ +# Packet capture summaries + +REA summarizes already saved pcap/pcapng files through caller-selected Wireshark +4.x `tshark` and `capinfos`. Use `inspect_packet_capture` for selected fields, +filters and decode-as rules; use this family for address, DNS, SNI, protocol-stack +and URI inventories. It does not acquire traffic or fetch recorded targets. + +## Supply tools + +```sh +export REA_TSHARK_COMMAND=/absolute/path/tshark +export REA_CAPINFOS_COMMAND=/absolute/path/capinfos +# Linux only; default /usr/bin/prlimit +export REA_TSHARK_PRLIMIT_COMMAND=/absolute/path/prlimit +rea inspect-pcap-client +rea inspect-pcap-capture /captures/session.pcapng --json +``` + +Without explicit tool commands, this family resolves them through PATH and +records that selection source. Client inspection reads no capture file. Capture +inspection requires Linux with util-linux prlimit or Windows with owned Job +Objects. macOS capture inspection returns a capability gap because the new +native memory boundary is unverified; its version probes remain available. +Tools and installed libraries/plugins are caller-trusted. An empty private +personal Wireshark configuration is used; this is not an OS sandbox. + +## Inspect only the needed facts + +MCP `inspect_pcap_capture` accepts `path` and optional `detail: "summary"`. +Complete delivery remains the default. Summary commits and retains the same +complete Evidence, returning identity, capture metadata, observed counts, +retained counts, source coverage and diagnostic lengths without large lists. +Then use its `normalized_result.parent_evidence_id` with +`inspect_analysis_view` source kind `retained-evidence`: + +```json +{"kind":"pcap","facet":"endpoints","offset":0,"limit":20} +{"kind":"pcap","facet":"dns-queries","offset":0,"limit":20} +{"kind":"pcap","facet":"dns-addresses","query":"example.com","offset":0,"limit":20} +{"kind":"pcap","facet":"tls-sni","offset":0,"limit":20} +{"kind":"pcap","facet":"diagnostics","stream":"tshark","offset":0,"limit":4096} +``` + +Other row facets are `protocols`, `http-uris` and `unattributed-addresses`. +Defaults are offset 0 and limit 64. DNS question pages give address counts; +select one exact retained question to page its addresses. Diagnostic windows +count UTF-16 code units; other windows count rows. Missing/ambiguous selectors +fail. Unknown legacy diagnostics remain unknown, not an observed empty stream. +Portable CLI views use the existing `inspect-analysis-view` workflow with inline +complete Evidence. Views never reread the source or launch Wireshark and work +after the original file is deleted. Page exhaustion covers the retained facet; +it does not establish network completeness or recover missing traffic. + +## Interpret provenance and coverage + +`target.sha256` binds observations to an admitted stable capture snapshot. Both +tools read that copy, which is hashed again after analysis. Capture metadata and +literal field values are producer observations; address/protocol counts and DNS +associations are derivations. Capture Evidence has historical-reference authority. + +Addresses count once per packet, including loopback and nested IP occurrences. +DNS addresses are associated only with one question in the same response packet; +additional records may be included, so this does not prove final resolution. +Ambiguous addresses remain in a separate inventory. Protocol distribution counts +exact `frame.protocols` stack strings. TLS SNI is recorded handshake metadata, +not process identity or proof of the destination's authenticity. + +All examined inventories are retained within the resource budget; no fixed +200/100/50-item truncation is applied. HTTP transport credentials are excluded +without rewriting other URI bytes; ambiguous credential spans cause omission. +Counts preserve distinct observed source URIs before omission or coalescing. +`coverage: "partial"` records unresolved DNS attribution or URI item loss; +component credential exclusion is also counted explicitly. Diagnostics retain +both tool stderr streams. Recorded fields do not establish decrypted payloads, +causality, process identity, absence of unrecorded traffic or current deployments. + +## Resources and lifecycle + +One inspection is admitted at a time per provider. Concurrent calls return a +capacity error; close drains active work before releasing resources. Captures +are streamed into a read-only owned snapshot with a 2 GiB disk budget. Native +capture commands enforce Linux address-space ceilings of at most 1 GiB, +300 CPU seconds, 64 MiB per file and no core dumps, preserving stricter inherited +limits. Windows uses a 1 GiB committed-memory limit for each owned tool tree. +These are not aggregate machine or filesystem quotas. + +TShark output is bounded at 8 MiB; its known five-level selected-field shape and +128 MiB graph-expansion estimate are checked before allocation. Retained facts +have a separate 16 MiB estimate. Capinfos output is limited to 64 KiB. Command +wall deadlines are 300 s for dissection, 60 s for metadata and 20 s for probes. +Native ownership preparation has its own bounded lifecycle. + +Malformed/oversized output returns no partial success. Unvalidated partial JSON +is not returned as facts. Cancellation reaches probes, acquisition, commands and +projection. Cleanup failure returns a typed error, retains completed Evidence +when available, and retains owned resources for retry on close. + +## Verification scope + +Current follow-up verification uses Linux/Node 24.18.0 and Wireshark 4.0.17. +Private real recorded corpora are checked against independent raw-frame/IP +parsing and Python aggregation, public CLI/stdio MCP parity, SDK schemas, source +hashes and retained views. Synthetic native tools separately test malformed +output, cancellation, source/snapshot mutation, capacity and cleanup; they do +not establish real Wireshark or other-platform behavior. + +The original proposal reported Linux 4.6.6, macOS/Windows 4.6.9 verification on +its earlier implementation. Those reports do not verify the new final memory +boundary. Actual Windows/Bun Wireshark execution is unverified here; macOS +capture support needs a separate enforceable-bound decision. + +```sh +npm run verify:pcap -- --pcap /absolute/saved-capture.pcapng +``` + +This verifier exercises built CLI/stdio MCP, schemas, artifact identity and +cleanup. Without `--pcap`, capture inspection is reported as unverified. Private +captures, keys and provider runtime files are not committed. diff --git a/docs/raw-binary.md b/docs/raw-binary.md index 43fcd54f4..8d251d010 100644 --- a/docs/raw-binary.md +++ b/docs/raw-binary.md @@ -25,10 +25,12 @@ rea inspect-native-load-image /absolute/path/to/ecu.bin --target-format raw-bina } ``` -`architecture` is one of `x86`, `x86_64`, `arm`, or `arm64`. `entry_point` is a +`architecture` is one of `x86`, `x86_64`, `arm`, `arm64`, or `6502`. `entry_point` is a hexadecimal address of the form `0x…`. The complete file is mapped starting at that address, and Ghidra seeds an `entry` function there before auto-analysis. -ARM and AArch64 entries must be 4-byte aligned; x86 entries may be byte-aligned. +ARM and AArch64 entries must be 4-byte aligned; x86 and 6502 entries may be byte-aligned. +For 6502, the entry and complete mapped file must fit within `0x0000..0xffff`; +images that cross the 16-bit boundary are rejected before provider startup. `open_binary` and an open `binary_session` return the canonical `entry_point` for raw targets so callers can verify the selected mapping directly. Omitted `format` keeps header-based detection; architecture and entry without @@ -37,12 +39,13 @@ Omitted `format` keeps header-based detection; architecture and entry without Ghidra imports a private snapshot with `BinaryLoader` and these little-endian generic languages and compiler specifications: -| architecture | language_id | compiler_spec_id | -| ------------ | ------------------- | ---------------- | -| x86 | `x86:LE:32:default` | `gcc` | -| x86_64 | `x86:LE:64:default` | `gcc` | -| arm | `ARM:LE:32:v7` | `default` | -| arm64 | `AARCH64:LE:64:v8A` | `default` | +| architecture | language_id | compiler_spec_id | +| ------------ | -------------------- | ---------------- | +| x86 | `x86:LE:32:default` | `gcc` | +| x86_64 | `x86:LE:64:default` | `gcc` | +| arm | `ARM:LE:32:v7` | `default` | +| arm64 | `AARCH64:LE:64:v8A` | `default` | +| 6502 | `6502:LE:16:default` | `default` | Compiler specifications select analysis conventions; they do not identify the original compiler or ABI of a headerless target. Ghidra's x86 languages do not @@ -57,11 +60,51 @@ The mapped start is independent of Ghidra's program image base, which remains `0x0` for BinaryLoader. Ghidra may report additional language-defined entry points, including unmapped ARM vectors. Load-image verification requires the caller-supplied entry and preserves those other entries as observations. +For 6502, BinaryLoader maps the file before adding its language-defined zero-page +and stack blocks. Conflicting default blocks are skipped; low-address files stay +in the default address space rather than an overlay. Non-conflicting uninitialized +blocks remain observations, not file-backed bytes. Independent raw verification streams the complete private snapshot digest and extent with bounded memory and supports cancellation during the read. +It also requires the source-backed block to be initialized, loaded, and +non-overlay in the bound language's default address space. Contradictory mapping +observations are retained with `mismatch`, even when the source hashes agree. -Hopper and IDA refuse raw-binary targets. Windows Ghidra P0 continues to admit -PE applications only. +Hopper and IDA refuse raw-binary targets. Windows Ghidra P0 admits native PE +applications/DLLs and explicitly selected raw 6502 images. Other raw CPU families +remain outside the Windows boundary. Native path admission, private DACLs, Job +Object ownership, immutable snapshots, and the no-mutation policy still apply. + +## NMOS 6502 + +```bash +rea function /absolute/path/to/image.bin 0x8001 --target-format raw-binary --architecture 6502 --entry-point 0x8001 --provider ghidra --json +rea inspect-native-load-image /absolute/path/to/image.bin --target-format raw-binary --architecture 6502 --entry-point 0x8001 --provider ghidra --json +``` + +Select only headerless NMOS 6502 code with a known entry at file offset 0. +The built-in Ghidra processor supplies static instructions, functions, +pseudocode, and references. Evidence preserves `architecture: "6502"`, the +selected language, entry, file digest, and verified source mapping. Register and +calling conventions are analysis conventions, not a recovered source ABI. + +This lane does not interpret iNES headers, cartridge mappers, bank switching, +or other console-specific memory layouts. 65C02 and 65816 programs need different +processor semantics and are not admitted as those variants. REA does not infer +the CPU from the file suffix, guess the reset vector, or emulate the target. + +`npm run verify:ghidra:6502` assembles and links the source-owned fixture with +`ca65` and `ld65` (tested from cc65 tag `V2.19`, whose tools report `V2.18`). +Set `REA_CA65` and `REA_LD65` to explicit executable paths if needed. The lane +checks real CLI/MCP load-image parity, mapped bytes, decoded calls, decompilation, +invalid mappings, unchanged source bytes and runtime cleanup. +Low-memory checks include a file spanning the zero-page/stack boundary at `0xff` +and a complete 64-KiB image at `0x0`, with verified default-space mappings through +both CLI and MCP. On Windows, the lane also checks that a caller-selected +junction path is rejected at snapshot admission through CLI and MCP, with +unchanged source bytes and complete runtime cleanup. Windows requires the +matching native addon; the POSIX provider path has not been exercised by the +Windows run. ## Limitations diff --git a/docs/recorded-crashes.md b/docs/recorded-crashes.md index d68db74fd..8b1a51662 100644 --- a/docs/recorded-crashes.md +++ b/docs/recorded-crashes.md @@ -72,9 +72,13 @@ debugger context is `observed`. Initial Linux x64 limits: 32 MiB input, 64 MiB decoder reply, 1 MiB diagnostics per stage, 30 seconds owned wall deadline per stage, 3 GiB address-space soft ceiling -and 30 seconds CPU soft ceiling. Tighter inherited soft/hard limits are retained; -hard limits are not raised. Complete oversized results fail explicitly. These -process limits do not constitute a sandbox for caller-supplied engine code. +and 30 seconds CPU soft ceiling. `REA_PWNTOOLS_MAX_INPUT_BYTES` raises the input +bound alone, up to 1 GiB, and returns `provider_unavailable` when it is not a +decimal integer in range; the reply, diagnostic and deadline bounds stay fixed, so a +large core that outruns them still fails explicitly instead of partially. Tighter +inherited soft/hard limits are retained; hard limits are not raised. Complete +oversized results fail explicitly. These process limits do not constitute a sandbox +for caller-supplied engine code. Resource exit statuses need matching private failure-branch markers. Unmarked launcher failures retain their actual status/output and unknown resource cause. diff --git a/docs/releasing.md b/docs/releasing.md index 37a28d532..c532d0663 100644 --- a/docs/releasing.md +++ b/docs/releasing.md @@ -6,7 +6,10 @@ checks to pass. Merging that PR into main starts publication automatically. The Release workflow validates the merged release metadata and ancestry before creating a tag. npm and MCP Registry publication use the exact reviewed merge -SHA returned by Release Please. Later main commits cannot change that source. +SHA returned by Release Please. Before either registry is published, the release +run calls full CI qualification at that exact revision, including coverage, +portable providers, and every native package host. Later main commits cannot +change or cancel that proof. Ordinary main pushes and closed, unmerged PRs cannot publish packages. Check both publication jobs and the published-package canary; a merged PR or GitHub tag alone does not establish that publication finished. Retry failed jobs in @@ -196,8 +199,14 @@ both registry publishes. See [npm's provenance implementation](https://github.com/npm/cli/blob/v11.16.0/workspaces/libnpmpublish/lib/provenance.js) for the use of GitHub's workflow ref and commit SHA. -The workflow builds the bundled Windows controls and verifies the packaged -artifact before npm publication. It then verifies the published CLI, the +Full qualification builds the bundled Windows controls and verifies the packaged +artifact on its native hosts. The Linux x64 verifier retains the exact tarball +it installed and tested, after its workspace cleanup completes. The npm job +downloads that archive from the same workflow run and checks its source SHA, +run identity/attempt, package and Node versions, host identity, and SHA-256. +It publishes that archive without rebuilding or repacking it. Re-running failed +publisher jobs can reuse a prior successful attempt from the same run. +The workflow then verifies the published CLI, the capability-scoped MCP catalog, and the isolated package update path before publishing MCP Registry metadata. @@ -218,6 +227,15 @@ release baseline and can propose an already-published version again. After publication, open a PR from the release branch back to main. Preserve main's later implementation changes and regenerate ignored build outputs from the combined contracts with the released package version. +Before merging, check whether GitHub permits merge commits: + +```bash +gh api repos/morluto/rea --jq .allow_merge_commit +``` + +If disabled, a maintainer must arrange an approved ancestry-preserving merge +path before synchronization can finish. Squash and rebase merges do not put the +released tag in main's ancestry. Do not substitute them for this step. Review and test this synchronization PR, then use a merge commit so the release tag remains in main's ancestry. Keep the released tag unchanged. diff --git a/docs/signal-recordings.md b/docs/signal-recordings.md index 520d6e882..69b14d193 100644 --- a/docs/signal-recordings.md +++ b/docs/signal-recordings.md @@ -106,3 +106,37 @@ One decode is admitted at a time, including acquisition and cleanup. Excess requests receive a retryable capacity error before acquiring another file or worker. Shutdown closes admission and drains accepted work; failed cleanup remains owned and retryable. + +## Discover decoders before choosing settings + +`inspect_signal_decoders` / `inspect-signal-decoders` queries the **installed** +sigrok catalog without opening a recording or scanning devices: + +```sh +rea inspect-signal-decoders --query uart +rea inspect-signal-decoders --decoder uart --query baud +rea inspect-signal-decoders --decoder i2c --offset 0 --limit 50 +``` + +Omit `decoder` to search decoder IDs and names. Supply one exact decoder ID to +read its literal help, including required/optional channels, options, annotation +classes and documentation. `query` is a case-insensitive literal substring; +matching help retains original one-based line numbers. Empty query selects all. +`offset` skips matches; `limit` defaults to 50. MCP `limit: null` or CLI `--all` +returns all remaining matches; CLI `--all` and `--limit` conflict. +The window reports examined, matching and returned records and `has_more`. +A filtered empty page is not evidence that the installation has no decoders. + +Discovery refuses more than 65,535 line separators before constructing record +objects (65,536-line representation budget), in addition to the 4 MiB output cap. +Exceeding that budget is a resource failure with captured output. Returned text +is copied so a retained page does not keep the complete producer output alive. + +Discovery shares decoding's owned process, admission slot, resource limits, +cleanup retries and Evidence delivery. It records the executable and limiter +identities, version banner and warnings. Failed version checks and malformed +producer output preserve diagnostics. Help is producer text, not an independently +validated parameter schema. Each request probes again; it does not cache or claim +stable pages across installation changes. Libraries/decoder modules remain +trusted and are not fully fingerprinted. Catalog output includes a SHA-256 of +its executable, not a digest of the complete installation. diff --git a/docs/testing.md b/docs/testing.md index 6f0b80131..7aa80d22a 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -263,17 +263,62 @@ CI owns coverage; aggregate and domain/contract thresholds are maintained in Coverage does not replace named boundary or real-provider scenarios. CI runs six native Vitest shards without retries. Each shard emits a blob -report; the merge job produces aggregate coverage plus JUnit and JSON timing +report; the merge job produces JUnit and JSON diagnostics, a module timing +report, plus aggregate coverage during full qualification. It retains available reports even when a shard fails, uploads available diagnostics, and writes the slowest files to the workflow summary. Partial reports are marked diagnostic only. Failed, cancelled, or missing selected shards still fail the required gate, -and aggregate coverage thresholds remain enforced by the merge command. -Only successful complete merges publish the small `test-timings` artifact, with +and aggregate coverage thresholds remain enforced in full qualification. +Only successful complete full-tier merges publish the small `test-timings` artifact, with checkout-relative paths; the build job downloads that artifact for the sequencer -instead of the coverage/JUnit bundle. Missing history falls back to native +instead of the coverage/JUnit bundle. Timing weights come from Vitest module +diagnostics and include tests and hooks, worker preparation, environment, +imports, and setup. Import timings already include transform waits, which are +not added again. The weights describe file costs under the measured worker and +coverage settings; their sum is not workflow elapsed time. The diagnostic +artifact retains the phase breakdown, while shard history keeps only each +file's total. Legacy test-body-only history and missing history fall back to native sharding. Static checks, documentation, build, package verification, and real-system lanes remain separate jobs so one kind of evidence cannot stand in -for another. +for another. Workflow syntax and expression validation runs inside the required +static-check job when workflow changes select it, avoiding a separate runner. +Native, provider, fixture, SQLite compatibility, and website lanes wait for +static checks to succeed before allocating their runners. This avoids spending +those runner slots on revisions that already fail static validation. Portable +test shards and documentation checks remain independent of static results, so +they can still provide diagnostic feedback. The required gate rejects skipped +selected lanes when static checks fail; passing revisions retain all selected +coverage. This ordering reduces peak demand on invalid revisions but can add +static-check waiting to otherwise passing runs; it is not a measured reduction +in end-to-end latency. + +Core jobs have execution limits of five minutes for classification, ten minutes +for static checks, builds, docs, and report merging, and thirty minutes per +portable test shard. These limits leave headroom for cold setup and full coverage +while preventing a hung job from holding the active main run for hours. They do +not bound runner queue waiting. Timeouts still fail the required gate. + +The separate `CI timings` workflow batches recent completed CI and Release runs +hourly or on manual dispatch. It uploads `ci-job-timings-batch` with raw API pages and +a normalized point-in-time report for each collected workflow attempt, plus a +batch manifest describing examined coverage, failures, and limits. Overlapping +lookback windows may collect an attempt again; identify records by run ID and +attempt. Each batch examines at most 100 completed runs per workflow created in +the preceding 24 hours, with a four-minute API collection budget. It collects the +latest attempt of each selected run and does not claim an exhaustive archive. +It runs the reporter from the trusted default branch with read-only permissions, +treating workflow runs as API data. It does not allocate a new reporter job for +every completed, cancelled, or skipped workflow. Reports separate queue +waiting (`created_at` to `started_at`) from completed execution and report +percentiles plus matched `npm ci` and artifact-step durations. +Queue excludes dependency waiting before job creation and does not establish +whether account concurrency or runner availability caused the wait. Queued jobs with +placeholder start times, skipped jobs, and missing or invalid timestamps remain +unknown. The reporter targets each selected run and attempt explicitly. +Summed job elapsed time is parallel work, not workflow wall time, CPU time, +memory usage, or billing. Step names identify samples, not complete transfer +costs. Its queue, execution, and failures are independent of the required gate +and publication. CPU time and peak process-family memory are not measured by this API. The build job saves portable Turbo outputs across runs. Eight Linux real-system lanes restore this cache without waiting for the build job. The shared action @@ -637,7 +682,9 @@ use `--second-procedure rea_fixture_subtract --second-marker='- 13'`. The lane invokes the production CLI dispatcher and connects the pinned MCP client SDK to the production REA server. It verifies function Evidence and CLI/MCP parity, inventory/search, pseudocode, instructions, xrefs, malformed -input, original-input preservation, and lifecycle cleanup. For headless +input, original-input preservation, and lifecycle cleanup. The headless +profile also checks containing procedures, the admitted program as the +document, address context, and feature tracing. For headless analysis it confirms that the owned database IDs disappear from upstream discovery and private workspaces are removed. For attached analysis it confirms the existing GUI target remains reachable with the same input identity. @@ -687,6 +734,16 @@ truncation constraint, while leading-zero coordinates still resolve correctly. It has the same Ghidra/JDK prerequisites as the MZ lane. Neither lane claims DOS runtime or PC-98 device execution. +`npm run verify:ghidra:6502` uses `ca65` and `ld65` to build the source-owned +NMOS 6502 fixture from `tests/conformance/ghidra/6502.s` and `6502.cfg`. Set +`REA_CA65` and `REA_LD65` for non-PATH installations (tested with cc65 `V2.19`). +With an installed Ghidra 12.1.4 and full JDK, it verifies real CLI/stdio MCP +mapping parity, exact source bytes, functions, a direct call reference, +decompilation, 16-bit overflow rejection without replacing the active session, +and runtime cleanup. Windows additionally requires the matching native addon; +its verifier lineage remains explicitly unavailable outside the provider's +retained Job Object, as in the other Windows lanes. + `npm run verify:ghidra:raw` uses a generated ARM little-endian headerless image through the compiled CLI and stdio MCP server. With the same Ghidra/JDK prerequisites, it checks preparation and bridge startup, verified load mappings, @@ -697,6 +754,44 @@ specifications, independent mappings, unchanged bytes, and decompilation of a known return value. AArch64 remains covered by deterministic tests rather than this real-provider lane. See [raw binary analysis](raw-binary.md). +`npm run verify:ghidra:address-context` runs on Linux with a caller-installed +Ghidra, full JDK, and an x86-64 ELF GCC toolchain (`CC` selects another compatible +compiler). It builds +`tests/conformance/c/pie-address-context.c` as a PIE ELF without executing it. +The compiled CLI and real stdio MCP workflow must expose the measured image +base, retain segment address spaces, agree on the overview, resolve explicit and +numeric file offsets, and reject unmapped analysis addresses with actionable +guidance. It checks the reverse `address-to-file-offset` mapping, file-coordinate +Evidence locations, and the suggested CLI search and load-image commands. +The verifier compares mapped bytes with the source file and checks source +identity, session preservation after failed lookup, owned runtime cleanup, and +verified empty process lineage. This lane does not establish Windows or macOS +real-provider coverage. + +For source-identity boundaries that a normal single-file import cannot create, +`tests/conformance/ghidra/ReaFileOffsetSourceProbe.java` exercises the production +bridge against real Ghidra `FileBytes`. It rejects auxiliary and partial sources, +checks the original target's round trip, and retains ambiguity when duplicate +target bytes have distinct loaded mappings. Injected decompiler failures and +cancellation verify that completed source/function resolution facts and backend +failure codes survive; these injections do not establish natural engine failures. +On Windows, after creating the +source-owned PE fixture, run from the repository root: + +```powershell +node scripts/create-ghidra-windows-fixture.mjs +$proofRoot = Join-Path $env:TEMP 'rea-file-offset-source-proof' +New-Item -ItemType Directory -Path $proofRoot -Force | Out-Null +Copy-Item bridge/ghidra/ReaGhidraBridge.java, bridge/ghidra/ReaGhidraExtensions.java, tests/conformance/ghidra/ReaFileOffsetSourceProbe.java -Destination $proofRoot +& "$env:GHIDRA_INSTALL_DIR/support/analyzeHeadless.bat" $proofRoot source-proof -import "$PWD/build/fixtures/rea-ghidra-windows.exe" -scriptPath $proofRoot -postScript ReaFileOffsetSourceProbe.java -deleteProject +``` + +Require the `REA_FILE_OFFSET_SOURCE_OK` marker and no script errors; Ghidra may +exit successfully even when a post-script fails. The probe edits only its +temporary analysis database and does not execute the fixture. This producer +regression supplements public CLI/MCP coverage and does not establish Linux +host behavior or native process-supervision controls. + ## Apple Interface Builder archives `npm run verify:interface-builder` compiles the source-owned AppKit XIB into a @@ -717,11 +812,32 @@ or the GitHub issue and pull-request templates run formatting and generated-document validation, including the small onboarding example consistency suite, without the source-test shards or native package lanes. Classification compares the PR head with its merge base, so later -base-branch changes do not expand that scope. Static and coverage aggregate jobs +base-branch changes do not expand that scope. Static and portable-test aggregate jobs remain present and fail if classification fails. Every code change retains the complete six-shard Linux suite; individual test files are not selected by imports or filenames. +Ordinary PRs also produce an **observational** test-selection plan. The candidate +leaf-adapter ownership table lives in `scripts/ci/test-selection.mjs`, separately +from provider-job scopes. It proposes keeping affected owners plus every unowned +test, including all CLI/MCP, acceptance, composition, conformance, and process +tests. Shared inputs, unknown paths, missing owner tests, and removed tests fall +back to the complete inventory. Renames account for both paths. Main, scheduled, +manual, and `ci:full` runs always propose the full suite. + +All six shards still execute the complete suite. The existing report merge +compares the proposal with actual module outcomes and retains +`.cache/ci-test-selection/report.json` in `test-results`, including selected and +omitted paths, changed input paths, missed failures, unobserved files, scheduling weights, PR head and +executed merge SHA. This adds no test reruns or runner jobs. Missing, duplicate, +skipped, interrupted, or unhandled results cannot establish a complete comparison; +even a complete passing comparison is not proof of ownership completeness. +Before enabling omissions, review representative full-run comparisons and +cross-owner consumers, including failures, and measure completed CI feedback and +runner demand. There is no automatic promotion or switch to skip tests. The +initial proposal measures only leaf producer-test omissions; it does not promise +the savings of a broader affected-test planner. + `scripts/ci/plan.mjs` selects lanes using the ownership table in `scripts/ci/scopes.mjs`. Provider implementation changes run their real-provider verifiers and the Linux x64 installed-package/Inspector baseline. The `apple` @@ -736,9 +852,19 @@ lane. Edits to verifier entrypoints run only by manual, licensed or self-hosted workflows run static checks and the Linux suite. Website changes run website checks. Provider workflow changes validate the workflow and run that provider's verifier. Manual-provider/Android workflow changes run workflow validation only. +The four-version SQLite compatibility matrix has its own `sqlite` scope for +SQLite implementation, contracts, CLI/MCP registration, fixtures, and shared +runtime boundaries. SQLite and server-lifecycle test edits and the SQLite npm +scripts also select it. Unrelated code edits retain the complete portable suite +on the pinned Node version without starting four additional compatibility jobs. +Full qualification retains every SQLite version. Shared process/contract/CLI boundaries, dependencies, other packaging inputs, -unknown paths, and changes to the planner select the full baseline. Package -script-only edits select their owning verifier when its entrypoint is +unknown paths, and changes to the planner select the full baseline. +CI timing collection, timing history, and test-report helper edits retain static +checks and the complete portable suite without selecting unrelated host/provider +lanes. Classification, qualification, gate, release receipt, and unknown CI +scripts keep the broad fallback. +Package script-only edits select their owning verifier when its entrypoint is identifiable; unknown or shared script changes select everything. Android build-script edits retain the Linux suite and workflow validation; real Android verification stays manual. Renames account for both paths. @@ -746,38 +872,84 @@ verification stays manual. Renames account for both paths. This narrower selection trades PR-time coverage for runner capacity. A change to unowned shared runtime code that fails only on arm64, macOS, Windows, the Apple artifact lane, or a provider lane that does not own the file is not caught on -the PR; the main-push full baseline reports it after merge. Ownership globs do +the PR; scheduled full qualification reports it after merge. Ownership globs do not follow imports. Use the `ci:full` label when a change may have native-host or provider effects outside its owners. -Main pushes and manual dispatch retain the full baseline, including all portable -provider lanes and the native matrix below. A `ci:full` PR label selects the full -baseline on the next PR run; manual dispatch can request it immediately. Each +Ordinary main pushes use the same owner scopes as PRs. Their diff starts at a +successful ancestor push run of this CI workflow, rather than the preceding +push: changes from failed or cancelled intermediate runs remain in scope. +Missing, untrusted, or non-ancestor history selects every lane. The lookup is +bounded to the last 100 successful push runs; lookup failures broaden selection. +Every code change still runs all six portable shards, without coverage +instrumentation in this fast tier. This deliberately avoids relying on an +import graph for generated files, filesystem inputs, subprocesses, or bridges. + +Scheduled qualification is admitted hourly at minute 23. A forced full run at +03:23 UTC refreshes runner and dependency drift every day. Other hourly runs +can reuse a successful full-qualification receipt for the exact source SHA +from the same repository's trusted CI workflow on its default branch. Both the +receipt and original workflow run must be less than 24 hours old, so retrying an +old gate cannot renew stale proof. Missing, expired, failed, foreign, or uncertain proof runs full +validation; API lookup failure also runs full. Lookups examine at most 100 named +receipts and three candidate runs within a 20-second budget. Reused runs select +no validation lanes, identify the original run, and produce no new receipt, so +reuse cannot indefinitely extend proof freshness. They make no fresh provider +observation. Manual dispatch, `ci:full`, and exact release qualification always +run full validation. + +Full qualification includes aggregate coverage, +every portable provider lane, and all native hosts below. Adding `ci:full` +immediately starts the separate `Full PR qualification` workflow against the +label event's merge revision; subsequent PR revisions also select the full tier +in normal CI while that label remains. Unrelated labels do not start validation +jobs or replace the main CI checks. Manual dispatch also requests full validation. +Fast-tier broad fallbacks retain all owner lanes and the three-host +package matrix; Intel macOS and coverage remain full-tier obligations. Each classification job records its selection and broad-fallback reasons in the run summary. Representative Git-diff cases live in `tests/boundary/filesystem/ciChangeScope.test.ts`. -`CI required` checks classification and every selected lane, rejecting failed, -cancelled, or unexpectedly skipped work. Existing required check names are -preserved; changing branch protection to require the new aggregate is a separate -repository-settings operation. Newer commits cancel superseded CI for the same -PR or main branch. Release publication has its own concurrency group and retains -its package verification and public-registry canary. Licensed/self-hosted +`CI required` always runs and checks classification and every selected lane, +rejecting failed, cancelled, or unexpectedly skipped work. It rejects missing or +unknown tiers and requires every canonical scope to be explicitly `true` or +`false`; incomplete classification cannot silently omit verification lanes. +Branch protection +should require this aggregate, `Static checks`, `Portable tests`, and +`Conventional PR title`. Title validation runs separately on PR opening, +reopening, commits, and edits; correcting a title refreshes only this lightweight +check. It reads title text through an environment variable, executes no PR code, +and has no token permissions. Its separate concurrency group cannot cancel +test runs. Fast PR runs cancel superseded feedback for the same PR. Main push +runs keep one active run and the latest pending run for that branch. +Scheduled/manual qualification has a separate concurrency group: active work +finishes and only the latest pending run is retained. Release qualification uses +its exact source SHA and cannot be cancelled by newer main pushes. Provider and +website lanes inherit their caller's scheduling; reusable +workflows have no independent cancellation group. Direct manual lane dispatches +run independently on disposable hosted runners. Publication +requires that qualification to succeed and consumes its exact verified tarball, +checking source SHA, workflow run, run attempt, package/Node version and SHA-256 +before npm publication. Earlier attempts from the same workflow run are accepted +when rerunning failed publisher jobs; future attempts and other runs are rejected. +The public-registry canary still verifies the published package. Licensed/self-hosted provider workflows and the Android smoke test retain their explicit triggers. CI exercises the pinned Node.js runtime on native hosted runners: -| Host | Runner | Baseline checks | -| --------------------- | ------------------ | ---------------------------------------------------------------------------------------- | -| Linux x64 | `ubuntu-latest` | Installed package and real Node Inspector CLI/MCP | -| Linux arm64 (aarch64) | `ubuntu-24.04-arm` | Installed package and real Node Inspector CLI/MCP | -| macOS 15 arm64 | `macos-15` | Installed package and real Node Inspector CLI/MCP | -| macOS 15 x64 | `macos-15-intel` | Installed package and real Node Inspector CLI/MCP (main, dispatch and `ci:full` only) | -| Windows x64 | `windows-latest` | Curated capabilities, native controls, installed package and real Node Inspector CLI/MCP | +| Host | Runner | Baseline checks | +| --------------------- | ------------------ | ------------------------------------------------------------------------------------------- | +| Linux x64 | `ubuntu-latest` | Installed package and real Node Inspector CLI/MCP | +| Linux arm64 (aarch64) | `ubuntu-24.04-arm` | Installed package and real Node Inspector CLI/MCP | +| macOS 15 arm64 | `macos-15` | Installed package and real Node Inspector CLI/MCP | +| macOS 15 x64 | `macos-15-intel` | Installed package and real Node Inspector CLI/MCP (hourly, dispatch, release and `ci:full`) | +| Windows x64 | `windows-latest` | Curated capabilities, native controls, installed package and real Node Inspector CLI/MCP | -macOS x64 is the slowest package host. Pull requests without the `ci:full` label -omit it, so darwin x64 installed-package and Inspector regressions surface on -the main-push full baseline rather than on the PR. +macOS x64 is the slowest package host. Ordinary PRs and main pushes omit it, +so darwin x64 installed-package and Inspector regressions surface in full +qualification. Schedule admission can be delayed by GitHub; the hourly trigger +does not guarantee an hourly completed result. Release publication waits for +its own complete result at the exact released revision. Package lanes assert the actual Node platform/architecture before verification and record those values with the Node version. The package matrix runs up to @@ -803,17 +975,34 @@ The Linux source-test shards download a shared runtime and generated test metadata snapshot produced by the build job with `npm run test:prepare`. The snapshot includes `dist/`, generated skills, the MCP tool catalog, product catalog, and managed-conformance metadata. Each shard still installs the locked -dependencies and runs its assigned coverage-enabled tests; it no longer +dependencies and runs its assigned tests; full qualification enables coverage. It no longer regenerates the same metadata independently. -The build job also retrieves the merged timing report from the last successful -default-branch push and includes that same report in each shard's snapshot. +The build job also finds an available `test-timings` artifact from a successful +trusted default-branch scheduled or manual qualification and includes that same +report in each shard's snapshot. The bounded lookup verifies the artifact's +producer; newer schedules that reuse qualification without producing timings do +not hide the last full run's artifact. CI assigns discovered files by historical duration, with a deterministic fallback weight for new files, rather than dividing only by file count. Every -discovered file still belongs to exactly one shard; worker limits, project -ordering, coverage instrumentation, and aggregate thresholds remain unchanged. +discovered file still belongs to exactly one shard; worker limits and project +ordering remain unchanged. The merge job rejects successful reports with missing, +unexpected, or duplicate portable files. Coverage instrumentation and aggregate +thresholds remain enforced in full qualification. Missing or unusable history falls back to Vitest's default assignment. Only -timing data is reused; persistent compile caches remain disabled. +timing data is reused; live provider results are never cached. Turbo outputs are +restored by task hash, with trusted default-branch runs as cache writers and PRs +as restore-only consumers. npm caches hold downloads, not `node_modules`. + +The eight automatic browser, binary-layout, EVM, JavaScript-recovery, +recorded-crash, and web provider lanes download `rea-build-cache` from their own +run after the build and static jobs succeed. Their existing build commands still +validate task hashes and rebuild mismatched inputs; provider observations always +run. Provider-workflow-only edits also admit the build without enabling source +test shards. Direct provider dispatches retain the cross-run restore-only cache. +This removes repeated compilation on fresh source revisions, but adds the build +as a prerequisite: a delayed build can delay provider admission. Compare queue +and execution times before claiming an end-to-end speedup. These native baseline checks complement the Linux source-test shards and the separate Apple-artifact and real-provider lanes. Actual Hopper, Ghidra, IDA, @@ -849,6 +1038,14 @@ establish passing Android coverage; inspect its execution receipt. The `termux-browser-diagnostics` artifact retains the Termux command log, exit status, and Android logcat on success or failure. The emulator is disposable. +## Packet capture analysis + +`npm run verify:pcap -- --pcap PATH` exercises caller-supplied tshark and +capinfos against a saved capture. Linux verification uses Wireshark 4.6.6 +and captures containing DNS, TLS and HTTP traffic. Native Windows and macOS +engine behavior remains unverified; process fixtures do not establish it. +See [Packet capture analysis](pcap-analysis.md) for contracts and budgets. + ## Apple native metadata and UI `npm run verify:apple-dispatch` compiles the Objective-C fixture (classes, diff --git a/docs/web-network-captures.md b/docs/web-network-captures.md index d49f1dcc3..c447c4783 100644 --- a/docs/web-network-captures.md +++ b/docs/web-network-captures.md @@ -25,6 +25,36 @@ and original zero-based producer ordinals; duplicate or out-of-range selections are input errors. Every record is validated before selection, so selecting one entry does not hide a malformed entry elsewhere. +For agent discovery over MCP, request `detail: "summary"`. REA retains the +complete selected Evidence in the session and returns identity and counts. +Use `normalized_result.parent_evidence_id` with `inspect_analysis_view`: + +```json +{ + "source": { + "kind": "retained-evidence", + "evidence_id": "" + }, + "view": { "kind": "capture", "detail": "preview", "offset": 0, "limit": 20 } +} +``` + +Preview pages select recorded HTTP headline fields, event names/times when +present, original locations and byte descriptors; they omit headers, XML and +payload content. Choose `detail: "complete"` in that view for exact record +pages, including all sidecars. Offsets follow the parent's selected record +order; producer ordinals remain unchanged. `total_records` counts the original +capture while `retained_records` and page coverage count records in the parent. +An exhausted page does not establish absence from unselected records. + +Views do not reopen the file or rerun its decoder. They remain historical +Evidence if the source changes or disappears; inspect the capture again to +observe new bytes. The inspection tool's default `detail: "complete"` preserves +existing inline delivery. For CLI use, save complete inspection JSON and pass +it as inline Evidence to `inspect-analysis-view`; the same summary and capture +page contracts apply. Large complete records can still exceed MCP's response +budget; export the retained Evidence when a single record cannot fit. + ## Formats and upstream profiles HAR uses unchanged `har-schema@2.0.0` draft-06 schemas, `ajv@8.20.0`, @@ -57,11 +87,80 @@ tested upstream source is tag `v12.2.3`, commit an explicit unsupported-profile diagnostic. Non-Linux native decoding is not supported by this adapter. +## Reading Fiddler captures (`.saz`) + +Fiddler Everywhere saves traffic in **Session Archive Zip (`.saz`)** files. +Inspect native SAZ directly, without installing Fiddler or converting to HAR: + +```sh +rea inspect-web-network-capture /absolute/session.saz saz --json +rea inspect-web-network-capture /absolute/session.saz saz --record 0 --json +``` + +MCP uses the same `capture_path`, `format: "saz"`, and optional +`record_ordinals` fields as the examples above. REA pairs `raw/_c.txt` +requests, `_s.txt` responses, and `_m.xml` metadata by filename stem. Archive +member names identify the source; zero-based ordinals follow each session's +first member in ZIP directory order, not its numeric ID or capture time. + +HTTP fields and ordered duplicate headers are retained, with stored body bytes +in binary sidecars. Bodies are not dechunked or decompressed. Metadata is an +XML node tree; optional `_w.txt` and `_g.txt` files retain opaque WebSocket and +gRPC message bytes. Missing members remain null, distinct from empty bodies. +See Telerik's [SAZ format reference](https://www.telerik.com/fiddler/fiddler-everywhere/documentation/knowledge-base/fiddler-archives). + +The existing ZIP reader verifies read members; expanded archive contents are +limited to 32 MiB in addition to the input/output budgets below. Encrypted +archives and unrecognized raw session members are unsupported. SAZ requires an +enforced decoder memory bound: Node's fixed heap, or the Windows Job Object +commit limit. Bun on hosts without that Job Object limit returns capability +unavailable before starting the SAZ decoder. Non-session +members are inventoried without decoding their content. Original files remain +unchanged. Linux verification includes independently checked public recorded +Mattermost captures from [FOREST](https://github.com/dfrc-korea/FOREST/tree/405c9154658a128ea203a6c10040cec55111ed03/api-results/dataset) +and the Fiddler author's [encoding corpus](https://github.com/ericlaw1979/Clearinet/blob/b6faf78cde1e72042980364f180ceb34b77d1434/Test/data/Encodings.saz), +plus synthetic malformed/limit fixtures. Their exporting product versions are +not authenticated. An interactive export from a current Fiddler Everywhere +installation and its live MCP server remain unverified. Support here is the +bounded documented raw HTTP/XML SAZ layout, not every product/version profile. + +For visual inspection, use Everywhere's **Import Snapshot**, then **Inspectors** +and **Overview**. HAR remains an alternative: select sessions, choose +**Export > HTTPArchive**, and confirm `log.version` is `"1.2"`. Preserve the +original SAZ when exporting. See the [import/export instructions](https://www.telerik.com/fiddler/fiddler-everywhere/documentation/collaboration/exporting-and-importing). + +## Fiddler Everywhere MCP and public skills + +For agent-driven traffic debugging, connect Everywhere's MCP server separately +from REA. Run a current Everywhere release with MCP access (Trial, Pro, or +higher), internet access, and a compatible MCP client. Use +**Settings > MCP Server > Coding agent**, or **Manual setup** with the generated +API key. The default endpoint is `http://localhost:8868/mcp`. See Telerik's +[MCP setup guide](https://www.telerik.com/fiddler/fiddler-everywhere/documentation/agent-tools/fiddler-mcp-server) +and [agent-skills prerequisites](https://www.telerik.com/fiddler/fiddler-everywhere/documentation/agent-tools/agent-skills). + +Telerik's [public skills](https://github.com/telerik/fiddler-agent-tools) cover +installation (`fiddler-download-setup`), connection setup, and traffic analysis: + +```sh +npx skills add telerik/fiddler-agent-tools --skill fiddler-mcp-setup +npx skills add telerik/fiddler-agent-tools --skill fiddler-traffic-debugging +``` + +After setup, reload the agent client and use the connected server's advertised +tool names and schemas. Check status, list sessions, then inspect relevant +session IDs. Set `sessionsSource` to `LiveTraffic` for HTTP(S) traffic or +`AgentCalls` for enriched LLM calls; active filters affect the session list. +For an existing run, skip the skill's pre-capture steps that replace process +filters and clear sessions. The documented MCP session sources do not provide +a saved-snapshot selector; use the SAZ workflow above for saved files. REA does +not register or wrap the Fiddler MCP server. + ## Reading the evidence - `artifact` identifies the caller-selected file and stable snapshot digest. -- `records` preserve producer order/ordinals and original JSON pointers or byte - ranges. No live browser transaction or scenario IDs are invented. +- `records` preserve producer order/ordinals and original JSON pointers, byte + ranges, or SAZ member names. No live browser transaction or scenario IDs are invented. - `reported` retains the producer's fields. HAR container metadata is returned separately, with `log.entries` replaced by `null` and `records_pointer` identifying the record collection. This is an inspection representation, @@ -156,7 +255,7 @@ response fields, including HAR's `status: 0` and an empty `mimeType`, remain unknown, not invented statuses or MIME types. The view makes no handler-identity, schema or causality claim. It performs no -network activity. Native mitmproxy captures return a typed unsupported error. +network activity. Native mitmproxy, SAZ and qlog captures return a typed unsupported error. Parents inspected with `sensitive_values`, or whose Evidence does not record `sensitive_value_count`, return a typed unsupported error whose reason is the canonical token `sensitive_values_unretained`. The view would write path @@ -192,14 +291,15 @@ Complete evidence has a 32 MiB input, 96 MiB decoder reply and 64-level nesting budget. Exceeding a budget returns no partial success. Oversized selected captures return an input error identifying `capture_path` and the byte budget; owned reply/output failures remain output or provider errors. -HAR runs in an owned -192 MiB Node old-generation heap with one V8 worker. Native decoding applies +HAR, SAZ and qlog run in an owned +192 MiB Node old-generation heap with one V8 worker. Mitmproxy decoding applies Linux limits of 768 MiB address space, 30 seconds CPU and 96 MiB file output; both commands have a 30-second wall deadline and independently supervised process cleanup. Owned decoder diagnostics retain at most 1 MiB across stdout and stderr; the collector drops overflow as it arrives, reports the limit failure and verifies cleanup. Heap limits alone do not establish aggregate RSS limits. -When REA runs on Bun, the HAR decoder runs on Bun, which ignores these V8 heap +SAZ and qlog refuse decoding when neither a Node heap limit nor a Windows Job Object bound can be enforced. Existing HAR availability is preserved. +When REA runs on Bun, the HAR/SAZ/qlog decoder runs on Bun, which ignores these V8 heap flags; its limitations and resource failures then omit the heap sizes. On Windows, the decoder's owned Job Object instead limits its committed memory to 704 MiB, the 192 MiB budget plus a 512 MiB reserve for Bun itself. The @@ -207,7 +307,7 @@ limitations state that limit, and reaching it returns `resource_constraint` with `process_memory_limit_bytes`, whatever exit status Bun reports. Observed decoder memory exhaustion returns `resource_constraint`, with the -selected capture, applicable limits and retained process diagnostics. HAR's +selected capture, applicable limits and retained process diagnostics. HAR/SAZ/qlog's old-generation heap is fixed; inherited `NODE_OPTIONS` cannot raise it. Record selection happens after complete decoding and does not reduce that workload. Use a smaller capture exported by its producer or another decoder @@ -234,3 +334,108 @@ Command cleanup failures retain the original status and both diagnostic streams; the decoder's ownership preparation receives the same cancellation signal. MCP advertises the same strict named input contract used by the application. Historical inspection delegates argument validation to that shared application boundary so accepted explicit-sensitive declarations also cover invalid argument names and other correction details. The SDK receives raw arguments through its Standard Schema interface; malformed inputs still fail before decoder or filesystem effects and return the normal structured REA error. Protocol-envelope validation remains owned by MCP. + +## Retained aioquic qlog + +Select `format: "qlog"` through `inspect_web_network_capture`, or: + +```bash +rea inspect-web-network-capture /absolute/origin.qlog qlog --json +rea inspect-web-network-capture /absolute/origin.qlog qlog --record 95 --record 0 --json +``` + +The initial profile is **aioquic 1.5.0, JSON qlog 0.3**: one JSON object with +`qlog_format: "JSON"`, `qlog_version: "0.3"`, `traces` containing object events +with `name`, finite numeric `time` and object `data`. JSON-SEQ, JSONL, columnar +array events and other versions are unsupported. Duplicate JSON members, +malformed UTF-8, invalid interpreted QUIC integers and excessive nesting fail +without partial success, including in events outside the selected ordinals. +Well-formed event subsets and missing frame metadata remain inspectable. +Unknown extensions are retained; this is not validation of every qlog schema. + +Choosing the format selects the interpretation profile; the file does not +necessarily report or authenticate an aioquic version. The profile's clock +uses Unix-epoch milliseconds, verified in aioquic 1.5.0's `QuicLoggerTrace` +`log_event`/`encode_time`. The `qlog.clock.basis` and Evidence analysis profile +state that assumption explicitly. Numeric lexemes survive independently of +safe JSON-number projections; decimal spelling does not establish physical +clock precision or monotonicity. + +Every event has a global ordinal from trace/event source traversal and its +original `/traces/T/events/E` pointer. This order is not a global time sort. +Trace index and the selected artifact identify a connection observation; +reported ODCID is preserved without claiming global uniqueness. Streams are +scoped by trace, stream ID and sent/received direction. Control/QPACK streams +are retained, so counting streams does not count HTTP requests. + +`qlog` is a typed facet required exactly for this format. Its scope is +`all-recorded-events-before-selection`: selection changes returned event +records while facts cite the original ordinals across the entire recorded +file. The container retains trace metadata with each events array replaced +by null; there is no single events pointer shared by all traces. + +| Fact | Basis and limits | +| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Original event and loss declaration | Observed endpoint report; `recovery:packet_lost` does not prove actual network loss. Missing packet number/type remains unknown. | +| STREAM ranges and byte totals | Derived per trace and direction from reported offsets and lengths using exact integer arithmetic. `unique_bytes` unions intervals; `frame_length_sum` includes overlaps; `repeated_bytes` is their difference. Overlap alone does not prove retransmission or equal contents. | +| FIN final sizes | Reported offset plus length where FIN is true; multiple conflicting sizes remain visible. A final size does not establish complete coverage or a received body. | +| Incomplete fact records | Ordinals with omitted/redacted interpretable frame metadata; usable fields in other records survive. An empty list does not establish complete logging or capture. | +| HTTP bodies and causality | `body_bytes: "not-reported"`; lengths and headers are metadata, with no reconstructed body, hash, timing causality or live process attribution. | + +QUIC integer fields interpreted here accept exact nonnegative JSON integer +values through `2^62 - 1`, including equivalent exponent notation. Offsets plus +lengths must fit that range. Derived sums are decimal strings and are not +rounded to JavaScript numbers. A missing offset is unknown, never defaulted to +zero. Ranges can start above zero or contain gaps. Packet headers, HTTP events, +raw lengths and unknown extensions remain producer evidence rather than being +normalized into invented HAR requests. + +The adapter excludes authorization/proxy authorization/cookie/set-cookie +values in known HTTP frame events, reset tokens in transport parameters and +NEW_CONNECTION_ID frames, and NEW_TOKEN frame tokens. Known URL header userinfo +and explicit sensitive values follow the shared exclusions. Ordinary extension +fields and token-looking query text remain retained. Explicit declarations also +cover normalized numeric projections, derived totals and trace pointers; facts +that cannot be safely shown are omitted with incomplete-record coordinates. +No keylog, PCAP, live URL or recorded target is opened by the qlog decoder. + +To repeat the real-provider lane with **local caller-supplied evidence** from +the educational aioquic 1.5.0 experiment (one origin trace, known 32,803-byte +stream coverage and one 1,169-byte repeated range in the loss case): + +```bash +npm run verify:web:qlog -- /absolute/control/origin.qlog /absolute/loss/origin.qlog /absolute/new-report.json +``` + +This lane checks built CLI/stdio MCP parity, SDK-converted JSON Schemas, +artifact digests, original event count, selection scope, retained parent lookup, +credential exclusions and the known control/loss facts. The report is created +exclusively and contains no keys or captured credential values. It is a +fixture-specific verification, not a general loss detector. CI uses synthetic +format inputs for malformed cases and limits; those tests do not establish +producer provenance. PCAP parsing and correlation remain outside this stage. + +### Focused qlog views + +Use MCP `inspect_web_network_capture` with `format: "qlog", detail: "summary"` +for identity, profile, scope and retained-fact counts. Then pass its +`normalized_result.parent_evidence_id` to `inspect_analysis_view` as a +`retained-evidence` source. + +```json +{"kind":"qlog","facet":"traces","offset":0,"limit":10} +{"kind":"qlog","facet":"streams","trace_index":0,"offset":0,"limit":10} +{"kind":"qlog","facet":"contributions","trace_index":0,"stream_id":"0","direction":"sent","offset":0,"limit":20} +``` + +Trace pages omit nested lists and give their counts. Stream pages preserve +exact byte totals but omit contributions, ranges and final-size lists. For +those lists select `contributions`, `ranges` or `final-sizes` with all three +stream selectors. `loss-declarations` and `incomplete-records` need only +`trace_index`. Offsets count rows in that retained facet; defaults are 0 and 64. Missing or ambiguous selectors fail rather than choosing a connection. + +These views page already interpreted facts without decoding again or opening +the source, and work after its deletion. Facts still cover all recorded events +before parent record selection. Their exhaustion does not prove logging or +network completeness. Event preview/complete pages remain available through +kind `capture`, with original numeric lexemes and producer ordinals preserved. diff --git a/docs/windows-ghidra-p0.md b/docs/windows-ghidra-p0.md index d14add07c..5def0dae4 100644 --- a/docs/windows-ghidra-p0.md +++ b/docs/windows-ghidra-p0.md @@ -17,14 +17,16 @@ additional permission flag or degraded mode is required. - Node.js 22.x (>=22.19), 24.x (>=24.11), or 26+; - an operator-installed Ghidra 12.1.x distribution (verified with 12.1.4); - a 64-bit full JDK inside that installation's declared Java range (JDK 21 or newer, with no maximum, for current 12.1 releases); -- an explicit native, non-managed x86 or x86-64 PE application or DLL; and +- an explicit native, non-managed x86 or x86-64 PE application/DLL, or a + [raw NMOS 6502 image](raw-binary.md#nmos-6502) with a caller-supplied entry; and - the 25 read-only Ghidra inventory, memory, and function-analysis operations. Loaded memory reads and file offsets preserve Ghidra's observed source mappings. PE load-image inspection returns an explicit unsupported attestation status. Independent verification covers admitted DOS MZ, explicit COM, and explicit -[raw-binary imports](raw-binary.md) on their supported hosts. This does not -extend Windows P0 admission beyond native PE. For x64 PE +[raw-binary imports](raw-binary.md) on their supported hosts, including the +admitted raw NMOS 6502 mapping on Windows. Windows P0 admission remains limited +to native PE and raw NMOS 6502 images. For x64 PE NativeAOT RTR 9.1, the read-only snapshot parser can report metadata through `inspect_native_load_image` and `inspect_native_data_type` without a JAR or database mutation. A real Windows-host NativeAOT Ghidra run is still required @@ -35,6 +37,12 @@ state, annotations, other target architectures and formats, and Hopper-only operations remain unavailable through this provider. Managed PE/CLI inspection uses its separate execution-free provider. +Raw 6502 imports retain the same native filesystem and process ownership +requirements. The complete image must fit in the 16-bit address space; no +console-specific cartridge loader, CPU variant, or bank-switching model is +implied. Run `npm run verify:ghidra:6502` with caller-installed `ca65`/`ld65` +to check the source-built fixture through real CLI and MCP workflows. + REA does not install or upgrade Ghidra, Java, Python, npm, Node.js, Hopper, or a compiler. The adapter uses the packaged Java `HeadlessScript`; Python and PyGhidra are not prerequisites. Users do not build the native addon. @@ -153,8 +161,12 @@ the private runtime, isolating Ghidra preferences, logs, and caches. Process creation assigns a suspended child atomically to an owned Job Object. Membership and kill-on-owner-close policy are verified before resume. Breakaway is disabled. Only the child's default file owner is normalized; -the caller token stays unchanged. Caller CPU affinity and priority are -preserved. Close, cancellation, timeout, startup failure, and provider failure +the caller token stays unchanged. Caller priority is preserved, and caller CPU +affinity whenever the caller and the child share a processor group. A child that +Windows places in another group keeps that group's default when the caller asked +for no bound of its own; a caller pinned to fewer processors than its group +holds is refused there rather than widened. +Close, cancellation, timeout, startup failure, and provider failure terminate the job and verify all members have settled before runtime cleanup. Cleanup removes admitted objects by handle and unlinks reparse entries without following their targets. Abrupt owner death kills the job but can leave diff --git a/experiments/analysis-efficiency/README.md b/experiments/analysis-efficiency/README.md index 79bac97e0..5f656a02c 100644 --- a/experiments/analysis-efficiency/README.md +++ b/experiments/analysis-efficiency/README.md @@ -7,6 +7,18 @@ JSON measurements belong outside the checkout. This document keeps the measured findings, verification scope and revision references; the scripts and `corpus.json` keep the reproduction inputs. Generated `results/` is ignored. +Follow-up: [latest-main capability audit](latest-main-audit.md) records the +Flutter, SQLite, TShark and sigrok priorities and the first shipped Dart AOT +baseline, preserving existing controls before introducing another selector. + +Follow-up: [Dart AOT summary and retained string pools](flutter-aot-summary.md) +records real LocalSend delivery pairs and old Evidence continuation, preserving +ABI selection, snapshot identity, original lower bounds and partial coverage. + +Follow-up: [SQLite existing-control measurements](sqlite-existing-controls.md) +tests QGIS's shipped GeoPackage through the existing column selection, preserving +full selected-table schema and typed cells before considering another API. + Follow-up: [native PE/DOS entry views and real model verification](native-entry-views.md) records the post-merge native continuation, including a small-function case where four selected replies cost more bytes than the complete dossier. @@ -37,6 +49,11 @@ raw MCP replies from observed client execution/discovery and usage. Optional trial capture retains tool/usage items while excluding account and ambient state; existing ephemeral trials keep their default behavior. +Follow-up: [Android package delivery and retained manifest](android-package-summary.md) +records real NewPipe SDK pairs, unchanged old requests/Evidence, exact retained +text, and actual Codex outcomes including client errors and a failed discovery. +Reply bytes fall; model-token/time gains are not established. + Follow-up: [resource counters at payload completion](resource-counters.md) adds optional owned counter observations while preserving unknown whole-scope final counters, default receipt shape and the unchanged CPU/memory safeguards. diff --git a/experiments/analysis-efficiency/agent-trial.py b/experiments/analysis-efficiency/agent-trial.py index 682be27ec..dc13e8781 100644 --- a/experiments/analysis-efficiency/agent-trial.py +++ b/experiments/analysis-efficiency/agent-trial.py @@ -94,6 +94,9 @@ def main(): if undeclared: raise ValueError('Declare inherited provider settings explicitly in --providers: ' + ', '.join(undeclared) + '. Values are not logged.') + source_auth_home = (Path(os.environ['CODEX_HOME']) if 'CODEX_HOME' in os.environ + else Path.home() / '.codex') + source_auth = source_auth_home / 'auth.json' args.output.mkdir(parents=True, exist_ok=False, mode=0o700) runtime_files, runtime_digest = runtime_fingerprint(repository) (args.output / 'runtime-files.json').write_text(json.dumps(runtime_files)) @@ -103,7 +106,6 @@ def main(): workspace = profile / 'workspace' for directory in [trial_codex_home, account, workspace]: directory.mkdir(mode=0o700) - source_auth = Path(os.environ.get('CODEX_HOME', str(Path.home() / '.codex'))) / 'auth.json' auth = trial_codex_home / 'auth.json' started = time.monotonic() timeout = False @@ -178,6 +180,7 @@ def main(): 'inspect_android_package', 'search_android_classes', 'inspect_android_class', 'inspect_android_method', 'inspect_firmware_regions', 'inspect_managed_artifact', + 'inspect_sqlite_database', 'inspect_managed_members', 'inspect_managed_native_boundaries', 'inspect_artifact', 'inspect_wasm_artifact', 'inspect_evm_interface', 'get_evidence_bundle', 'export_evidence_bundle', diff --git a/experiments/analysis-efficiency/android-package-summary.md b/experiments/analysis-efficiency/android-package-summary.md new file mode 100644 index 000000000..448202d87 --- /dev/null +++ b/experiments/analysis-efficiency/android-package-summary.md @@ -0,0 +1,166 @@ +# Android package summary and exact retained manifest + +Continues [#2016](https://github.com/morluto/rea/issues/2016) and +[#2438](https://github.com/morluto/rea/pull/2438). The recurring NewPipe question +asks for APK/package/version identity, all case-sensitive MainActivity matches +and all reported onCreate entries, including explicit null DEX descriptors. +Complete package replies delivered decoded manifest text and duplicate upstream +calls unrelated to those selected facts. + +## Compatibility and ownership + +MCP `inspect_android_package` accepts explicit `detail: "summary"`; omitted and +`"complete"` requests preserve the complete output and provider request. The +one-shot CLI stays complete. Summary selection happens in the application +workflow, independently of the engine protocol, using the existing retained +analysis-view machinery. No third-party plugin API or persistent analysis cache +is introduced. Servers lacking retention reject summary before provider work. + +The complete original Evidence stays under its original ID and session lifetime. +A distinct linked derived view returns all package/SDK/permission declarations, +engine identity, class/resource counts and signature-verification status. +Manifest metadata retains the producer's partial/complete status, nullable +reported byte total, and observed UTF-16 length. Text and duplicate raw calls +remain in the original record, rather than being rewritten under its ID. + +Use `inspect_analysis_view` with the parent's ID and +`view: {kind: "android", facet: "manifest", offset: 0, limit: 4096}` to read exact +retained decoded text. Follow `coverage.next_offset`; surrogate boundaries use +the existing text-page implementation. Exhaustion describes the retained text, +not global completeness when the producer says partial. APK signatures and +runtime behavior remain unverified. Engine artifact SHA-256 identifies the JAR; +views require the authenticated APK subject and never substitute the engine +hash for a missing target identity. Older servers require schema discovery and +the original complete request; automatic fallback is not introduced. + +## Frozen runtime and real SDK pairs + +Source runtime: `28c84e78cf2fbfde4aceed345b773c59eb889f68`; later test-only schema +typing corrections do not relabel these receipts. Compiled dist/bridge file-table +SHA-256: `5332dda13981f043e146b3601ebec1f380d9f698280ca53d75786b638a912e11`. +The frozen old runtime is `8bc1601ca51e417ca472d90ea0367e9418a1a0f2`, digest +`9e7380f537301bf0cfe2926f5619011af848af7c220750e75189498e65d825e1`. + +Real NewPipe 0.29.1 bytes, acquisition and digest are pinned in `corpus.json`. +Actual configured JADX headless MCP 0.7.1/Java runs use the same selected +provider settings and search-path digests, one worker and 768 MiB Java heap. +Node is 24.18.0; actual SDK client/server are 2.3.1. Processes and engines are +fresh per trial; host page caches are not flushed. SDK schema mode is selected +in both treatments. Heavy runs are serial, with one CPU, unchanged 3 GiB early +stop/4 GiB hard memory, no swap, and 192 tasks. + +All four candidate workflows use three successful public MCP calls. Only the +first package delivery changes. Both pairs pass settings and selected-fact +comparison, including all declarations, manifest status/length, engine identity, +original limitations, matching class rows and onCreate entries. Original replies +and IDs remain separate. A fresh old-runtime complete workflow also passes +comparison against the candidate complete request; both complete package replies +are exactly 119,527 compact client-result JSON bytes. This is scoped compatibility +evidence, not a guarantee for every historical release/client/host. + +| Trial | Package reply bytes | SDK workflow seconds | Payload CPU seconds | Kernel peak bytes | +| -------------- | ------------------: | -------------------: | ------------------: | ----------------: | +| sdk-complete-1 | 119,527 | 24.059 | 25.259 | 1,225,076,736 | +| sdk-summary-1 | 8,659 | 23.694 | 24.894 | 1,226,870,784 | +| sdk-summary-2 | 8,659 | 24.407 | 25.611 | 1,228,296,192 | +| sdk-complete-2 | 119,527 | 24.509 | 25.740 | 1,229,844,480 | + +Summary replies are 8,659 bytes, about 92.8% smaller; all three replies together +fall from 149,577 to 38,709 bytes. This is a delivery result, not a model-token +or engine-work saving. The SDK timing differences are small. CPU and memory do +not establish a reduction. No provider setting or analysis scope changes. +Transport roundtrip includes server/client work and is not an engine-only timer. + +The complete catalog is 2,506,213 compact JSON bytes versus 2,497,635 on the old +runtime: about 0.34% growth, including intervening main changes. Discovery cost +remains visible alongside selected-delivery savings. + +## Authentic old Evidence continuation + +A real package record from old runtime 8bc160 imports into the candidate and +supports retained summary, exact manifest pages, inline portable summary and +export through actual public SDK tools. The new session explicitly selects an +absent JADX JAR; this workflow performs no package analysis. Eleven successful +calls preserve the original record, original ID, all declarations and all 25,222 +manifest UTF-16 units. Manifest UTF-8 SHA-256: +`fee192fe2a6d7892c22cba115184c74c7e2c42ce464e8bb37be6289f2c16881d`. +The exported original record is deeply equal to the imported record. + +## Actual Codex task outcomes + +Four serial trials run in complete/summary/summary/complete order on the same +frozen candidate. Requested configuration: Codex CLI 0.161.0, `gpt-5.6-terra` high, +three exact selected tool names and the same optional context/discovery capture +settings. The requested alias does not certify the server-resolved model. The +original analyst question is unchanged; equal treatment instructions explicitly +choose package detail and explain the summary carrier. Temporary configuration +points to the frozen modified REA MCP; no global configuration is edited. + +| Trial | Input | Cached | Uncached | Output | Seconds | MCP calls | Usage records | Facts | +| ---------------- | ------: | -----: | -------: | -----: | ------: | --------: | ------------: | ----- | +| model-complete-1 | 39,267 | 23,040 | 16,227 | 1,690 | 94.053 | 3 | 3 | Pass | +| model-summary-1 | 128,417 | 93,440 | 34,977 | 3,633 | 150.401 | 8 | 9 | Pass | +| model-summary-2 | 54,437 | 37,120 | 17,317 | 2,185 | 123.677 | 3 | 4 | Pass | +| model-complete-2 | 21,332 | 9,984 | 11,348 | 667 | 20.774 | 0 | 2 | Fail | + +Three answers pass the original selected-fact gate against the actual SDK +producer. Each trial preserves APK bytes and removes its isolated profile. +The first summary trial makes five successful calls and three rejected requests: +an invented `case_sensitive` argument, empty class-search input and empty +class-inspection input. It then repeats successful package/search work. These +errors and correction turns remain part of measured task cost; strict existing +schemas correctly reject them. The second summary trial uses the intended three +successful calls, but still does not establish improved total/uncached input or +latency against the factually acceptable first complete control. + +The final complete control's first client executor filters `ALL_TOOLS` for the +same three exact names and receives an empty array. It makes zero MCP calls, +reports all requested facts unknown and fails the factual gate. Identical name +filters worked in the other trials. Stderr contains no explanatory error; the +client visibility failure remains unexplained and is not attributed to REA or +summary delivery. Its short duration is not comparable completed work. The +original failed receipt is retained rather than replaced by a successful trial. + +These measurements demonstrate no real-model token or speed gain. Smaller raw +MCP envelopes do not establish smaller model context; observed usage also +includes discovery, execution context, corrections and extra responses. Usage +records count captured usage items rather than asserting exact remote request +contents or equating MCP calls with model rounds. The complete default and +opt-in choice remain unchanged; this is useful delivery/retention capability, +not a general performance recommendation. + +All model runs have verified payload-completion witnesses with only the observer +remaining and zero memory/task limit events at that phase. Kernel peaks are +602,472,448–1,444,945,920 bytes; the failed no-analysis control is included in that +range and is not evidence of lower analysis memory. Whole-scope final counters +remain unknown and the existing full resource grader stays false. No broader +CPU, memory, OS/provider or model claim is made. + +## Resource interpretation + +Every SDK run and portable continuation has a verified owned +payload-exited-observer-alive witness, with only the observer remaining and no +memory/task limit events at that phase. Kernel peak and CPU include that opt-in +observer. Whole-scope terminal counters remain unknown; the existing complete +physical resource grader is not changed or called successful. No OOM stress or +resource-limit increase is used. + +## Verification and scope + +Twenty-seven relevant converted-schema, SDK, domain and existing Android +failure/cancellation/cleanup checks pass. A three-case focused rerun passes after +a test-only schema typing correction. The 1 MiB manifest protocol fixture verifies +original default/full output, reconstruction after source removal, unchanged +exported Evidence and one engine launch. Other cases preserve nulls and partial +status, Unicode boundaries and tagged malformed/missing-identity failures, and +reject unavailable retention or cancelled requests before provider work. These +fixtures verify production boundaries; actual JADX claims use the real NewPipe +runs above. Full supported single-thread compiler and unchanged lint pass under +the same memory safeguards. No other Android OS/device/provider lane is inferred. + +Raw SDK/model replies, private profiles and resource receipts stay outside Git. +Reproduce SDK pairs using the existing `measure.mjs` with `--artifact newpipe.apk`, +`--schemas selected`, identical explicit provider configuration and frozen runtime, +and alternating `--delivery complete` / `--delivery summary`. Preserve failed +attempts and run `compare.mjs` on both pair orders. Model findings require their +own identical analyst facts and explicit delivery controls. diff --git a/experiments/analysis-efficiency/corpus.json b/experiments/analysis-efficiency/corpus.json index 540e88cdd..163b3b49a 100644 --- a/experiments/analysis-efficiency/corpus.json +++ b/experiments/analysis-efficiency/corpus.json @@ -235,6 +235,34 @@ }, "immutable_scope": "Published Hardhat artifact does not record an immutable-reference map; absence is unknown, not proof of no immutables. Runtime template bytes remain exact." } + }, + { + "id": "localsend-arm64.apk", + "file": "LocalSend-1.18.2-android-arm64v8.apk", + "bytes": 46558706, + "sha256": "82ec3568fba2aa5295b9aae8b76f701d7a4703d86b9f8bad749472038fbaeab3", + "provenance": { + "project": "LocalSend", + "version": "1.18.2", + "upstreamFile": "LocalSend-1.18.2-android-arm64v8.apk", + "url": "https://github.com/localsend/localsend/releases/download/v1.18.2/LocalSend-1.18.2-android-arm64v8.apk", + "transport_sha256": "82ec3568fba2aa5295b9aae8b76f701d7a4703d86b9f8bad749472038fbaeab3", + "transport_bytes": 46558706, + "derivation": "Official arm64 release APK, compared with GitHub release asset digest; byte-only analysis without execution" + } + }, + { + "id": "qgis-world-map.gpkg", + "file": "qgis-world-map.gpkg", + "bytes": 12914688, + "sha256": "01d293159d1b77bd9212b5252587853f61aa9e25b2387135f34afaba8d057190", + "provenance": { + "project": "QGIS bundled world map", + "source_commit": "e1223b8e9e71fc50ee995ed556f711c54ccc101b", + "git_blob_sha1": "c9ff80228dff0de93c9acdef16a97545ba360244", + "url": "https://raw.githubusercontent.com/qgis/QGIS/e1223b8e9e71fc50ee995ed556f711c54ccc101b/resources/data/world_map.gpkg", + "derivation": "unmodified official resource file; streaming Git blob and SHA-256 verification" + } } ] } diff --git a/experiments/analysis-efficiency/flutter-aot-summary.md b/experiments/analysis-efficiency/flutter-aot-summary.md new file mode 100644 index 000000000..8893cd917 --- /dev/null +++ b/experiments/analysis-efficiency/flutter-aot-summary.md @@ -0,0 +1,160 @@ +# Dart AOT summary and retained string pools + +Continues [#2016](https://github.com/morluto/rea/issues/2016) and +[#2455](https://github.com/morluto/rea/pull/2455). The selected analyst question +asks for shipped LocalSend target/ABI/library identity, every nullable snapshot +observation, string-pool lower bounds and returned counts, and original coverage +and limitations. It does not require individual observed strings. + +## Compatibility and ownership + +MCP `inspect_dart_aot` accepts explicit `detail: "summary"`. Omitted and complete +requests preserve the complete result and original provider inputs. The CLI +remains complete. ABI omission and explicit ABI selection retain their original +meaning; the provider never receives the delivery field. Missing retention, +invalid detail and pre-cancellation fail before provider work. + +The existing application workflow retains the complete original Evidence, then +returns a distinct derived summary linked to its parent ID. It preserves target +and libapp digests, every snapshot section and nullable fact, original pool and +overall coverage, lower-bound counts and limitations. `returned_counts` reports +the lengths of the producer-returned arrays; it does not upgrade lower bounds +to a complete snapshot inventory. The analysis and original source retention +are unchanged. No new engine, persistent cache or plugin SDK is introduced. + +Read exact retained strings using `inspect_analysis_view`, the parent ID, and +`view: {kind: "dart-aot", collection: "package_uris", offset: 0, limit: 64}`. +Other collections are `dart_uris` and `class_like_tokens`. Follow the returned +next offset. Pages preserve source order, ABI/library identity and partial pool +coverage without opening the APK again. Page exhaustion describes retained +strings, not the entire snapshot. Strings remain byte-pattern observations, +rather than recovered typed class/function semantics. Missing historical +subjects remain explicitly unknown; conflicting recorded target digests fail. + +## Frozen runtime and real SDK pairs + +The official LocalSend 1.18.2 arm64 APK acquisition, size and SHA-256 are pinned +in `corpus.json`. No APK installation, target execution or version-dataset +lookup occurred. Source runtime: `811034a7283c9751a3ecbff33e8b2f28bec55793`; +compiled dist/bridge file-table SHA-256: +`dabbb63536c5862be839ecdc0f3eaad05b29c2a3816f57041528d30f754311be`. +The previous runtime is `28c84e78cf2fbfde4aceed345b773c59eb889f68`, digest +`5332dda13981f043e146b3601ebec1f380d9f698280ca53d75786b638a912e11`. +Its production sources were verified byte-equal to main f254 at acquisition. +Subsequent test-only changes and main's SAZ addition do not relabel these pins. + +Node is 24.18.0 and the SDK client/server are 2.3.1. Each workflow starts a fresh +server and validates the selected live advertised schema. Artifact bytes, +provider environment/search-path fingerprints, driver and locked dependencies +stay equal. Host page caches are not flushed. Heavy tasks run serially under +the unchanged CPU, memory, task and host guards. + +| Trial | AOT reply bytes | SDK workflow seconds | Payload CPU seconds | Kernel peak bytes | +| -------------- | --------------: | -------------------: | ------------------: | ----------------: | +| sdk-complete-1 | 151,341 | 5.139 | 6.819 | 691,441,664 | +| sdk-summary-1 | 12,037 | 5.219 | 6.608 | 639,430,656 | +| sdk-summary-2 | 12,037 | 5.064 | 6.405 | 640,868,352 | +| sdk-complete-2 | 151,341 | 5.128 | 6.468 | 633,614,336 | + +Both pairs use one successful public tool call and pass settings/fact comparison, +including all snapshot fields, coverage, original limitations and all counts. +Reply bytes fall 92.05%. Whole SDK time stays approximately 5.1 seconds; these +trials do not establish engine speed, CPU or memory savings. Payload CPU includes +driver/catalog work beyond the SDK workflow timer. Catalog JSON grows from +2,506,213 bytes on the previous runtime to 2,514,896 (about 0.35%). + +A fresh old-runtime complete request passes comparison against candidate +complete, including identical 151,341-byte replies. This is scoped compatibility +evidence, not a guarantee for every historical release or third-party client. + +## Real old Evidence continuation + +Fifteen successful public MCP calls import genuine old-runtime AOT Evidence, +project retained summary, reconstruct all 500 package URIs, 216 Dart URIs and +2,000 class-like tokens, compare inline portable summary, and export the original +record. The APK is absent during all these operations. Original string order, +lower bounds, partial coverage, nullable snapshot facts, subject and parent ID +remain intact; the original exported record is deeply equal to its source. + +The first private verification attempt used a nonexistent receipt filename and +failed before import or artifact relocation. That driver error is retained; +the corrected attempt selects the fresh old-runtime record. It is not a REA +defect or a successful migration trial. + +## Actual Codex task outcomes + +Four serial trials use the same frozen candidate in complete/summary/summary/ +complete order. Requested configuration is Codex CLI 0.161.0, `gpt-5.6-terra` +high, one exact enabled tool (`inspect_dart_aot`), and identical optional owned +context capture/discovery guidance. This requested alias does not certify the +server-resolved model. Temporary configuration points to the modified frozen +REA MCP; all copied credential profiles are removed and target bytes remain +unchanged. The source question is fixed; equal treatment instructions choose +detail and explain the summary carrier and parent reference. + +| Trial | Input | Cached | Uncached | Output | Seconds | MCP attempts | Failed attempts | Facts | +| ---------------- | -----: | -----: | -------: | -----: | ------: | -----------: | --------------: | ----- | +| model-complete-1 | 74,641 | 50,176 | 24,465 | 2,765 | 116.898 | 3 | 1 | pass | +| model-summary-1 | 38,253 | 22,016 | 16,237 | 1,449 | 88.904 | 1 | 0 | pass | +| model-summary-2 | 38,363 | 22,016 | 16,347 | 1,515 | 55.190 | 1 | 0 | pass | +| model-complete-2 | 66,585 | 38,144 | 28,441 | 2,217 | 79.180 | 2 | 0 | pass | + +Both pairs pass checks for every selected producer fact, exact nullable +observations, target digest, observed response Evidence IDs, original limitations, +artifact preservation and profile cleanup. The grader compares the reported +facts with producer observations; it is not an independent Dart deserializer. +Total input falls 48.75% / 42.38%, uncached input 33.63% / 42.52%, reported output +47.59% / 31.66%, and whole-task time 23.95% / 30.30%. Usage is the final aggregate +reported by the actual CLI, one usage record per trial; cached tokens are not +subtracted from total input. Reported output includes reasoning output (1,262, +321, 383 and 709 tokens respectively). + +The first complete trial initially supplied `artifact_path` instead of the +advertised `path`, corrected it, then repeated a successful complete call. +The second complete trial made two successful calls. Both summary trials made +one successful call. All attempts, discovery and correction costs remain +included. The paired whole-task improvement therefore includes actual model +decisions and the convenience of explicit returned counts; it does not isolate +serialization as the sole cause or establish engine acceleration. Two pairs on +one app/question/model configuration support this scope, not a universal gain. + +Payload CPU seconds / kernel peak bytes are respectively 12.556 / 876,937,216; +9.213 / 785,272,832; 9.108 / 793,939,968; and 13.164 / 803,209,216. Model payload +observations verify owned cleanup and zero phase limit events, with no guard +aborts. Whole-scope final counters remain unknown, so every strict all-selected +resource grade remains false even though the separate factual grade passes. + +## Resource and verification scope + +SDK and migration payload-exit observations verify owned cleanup and zero +observed memory/pids limit events, with peaks below 692 MB. Missing whole-scope +terminal counters remain unknown; the strict whole-resource grader stays false. +One-CPU quotas are experiment controls rather than CPU-efficiency claims. + +Production-provider fixture tests cover explicit ABI, old complete defaults, +actual SDK-converted schemas, exact pages after APK removal, portable records, +nullable/partial facts, malformed inputs, target digest conflicts and early +refusals. CLI/MCP acceptance covers original report equality, duplicate-member +refusals and unsupported snapshot kinds. Fixture successes are distinguished +from the real LocalSend SDK/migration workflows. Other Flutter apps, architectures, +SDK layouts, typed class recovery and runtime behavior remain unverified here. +Raw replies, binaries, model traces and temporary credentials stay outside Git. + +## Integration with the newer main + +The rebased verification runtime is `63984ce0898bd7997e6d9b138e6d96d0a24f2b09` +on main `a2d8b94dd50dab8d966239dd63ab84718aa8b4b4`, with compiled file-table +SHA-256 `19c8ccdd3e99bb46a5c54f4edd8c6b2d2eb4f3a4870b3c129061d38145382b68`. +Targeted Flutter/SAZ/shared-view integration passes 35 tests; one opposite +runtime-capacity branch is skipped because this host enforces decoder memory. +Documentation checks and two README conformance tests pass. + +Fresh real complete/summary SDK checks on this runtime match the earlier selected +facts and settings, with unchanged 151,341 / 12,037-byte AOT replies. Fifteen +old-Evidence migration calls again preserve every original string and record +with the source APK absent. The real one-shot CLI complete result also matches +the previous-runtime normalized report, subject, parameters and original ID. +An initial CLI-only controller check lacked an owned resource receipt; it is +retained and excluded from resource/performance claims, then repeated in the +bounded owned scope. These checks establish integration, rather than relabeling +the earlier paired model/SDK measurements as the rebased runtime. diff --git a/experiments/analysis-efficiency/latest-main-audit.md b/experiments/analysis-efficiency/latest-main-audit.md new file mode 100644 index 000000000..8f767fb64 --- /dev/null +++ b/experiments/analysis-efficiency/latest-main-audit.md @@ -0,0 +1,75 @@ +# Latest-main efficiency continuation + +Continues [#2016](https://github.com/morluto/rea/issues/2016). The 2026-10-13 +fetch resolved main to `f254f5ac2b804b353a711264bb178f43fdf07867`, the same +revision as the preceding Android conclusion. Newer capabilities below were +added during that earlier cycle; they are additional measurement scopes, not +new commits after that conclusion. + +| Capability | Existing controls | Next evidence needed | +| ----------------------------- | ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | +| Flutter / Dart AOT (#1582) | Explicit ABI, bounded string observations, partial coverage | Real shipped snapshot/count question; opt-in delivery only if unrelated strings dominate | +| SQLite (#2271) | Compact inventory, selected table/columns, row limit | Real application database and model workflow before adding another selector | +| TShark (#2318) | Fields, filter, decode-as, pass, default 50 matching rows, paging, explicit null for all | Real public capture and repeated task cost; paging repeats dissection | +| sigrok (#2339) | Decoder/channels/annotation selection, default 50 display lines, paging | Real public capture; display-line count is not annotation count and paging repeats decoding | +| Retained unknown head results | Existing retained paging | Reuse the existing view rather than duplicate its implementation | + +Attached IDA, Frida and other live runtime workflows involve separate authority +and lifecycle questions and are outside this static parsing continuation. +Plugin SDKs and cross-session engine caches remain deferred until an observed +need and benefit justify them. + +## First selected scope: Dart AOT + +The official LocalSend 1.18.2 arm64 APK is pinned in `corpus.json`. The initial +real public SDK baseline inspected actual bytes without executing/installing +the APK. Its complete default AOT reply was 151,341 compact client-result JSON +bytes (70,487 normalized bytes), with 500 package URIs, 216 Dart URIs and 2,000 +class-like tokens. String-pool and overall coverage were partial. A second +explicit arm64 request returned the same normalized facts. + +These are exploratory first/warm calls, not an optimization pair. The frozen +runtime source is `28c84e78cf2fbfde4aceed345b773c59eb889f68`, with production +source bytes verified equal to main f254 at baseline acquisition. Its compiled +file-table digest is +`5332dda13981f043e146b3601ebec1f380d9f698280ca53d75786b638a912e11`; +the receipt is not relabeled as a newer build. + +The candidate adds explicit MCP `detail: "summary"` and retained pool pages. +Omitted/complete requests and the CLI preserve the old complete path. Analysis, +ABI selection and original Evidence stay intact. Summary retains target/libapp +identity, exact nullable snapshot observations, lower bounds, returned counts, +coverage and limitations. Pages preserve original string order and source +coverage without rescanning. No engine cache, provider protocol or third-party +plugin API is introduced. + +Follow-up measurements must compare identical selected facts, portable old +Evidence and actual model tasks. Reply bytes do not establish token or latency +improvements. Whole-scope final counters remain unknown if their cgroup has +disappeared, even when the narrower payload-exit observation is verified. +Raw records, binaries and model traces stay in the private lab outside Git. +Heavy work remains serial under the unchanged one-CPU, 3 GiB early-stop / 4 GiB +hard memory, no-swap and host-availability guards. + +## Main advanced during the experiment + +Main subsequently advanced to `a2d8b94dd50dab8d966239dd63ab84718aa8b4b4`, +adding native Fiddler SAZ parsing. Its web-network capture contract already +offers opt-in summary and retained capture pages, archive-member coordinates +and exact exposed encoded bytes. It decodes all records before ordinal +selection and rejects malformed/oversized inputs without partial success. +Measure real public SAZ artifacts using these controls before adding another +API or changing decoding or credential-redaction semantics. SAZ performance +remains unverified in this Flutter continuation. + +The Flutter cohorts keep their original frozen runtime; subsequent integration +onto this newer main does not relabel earlier measurement receipts. + +Main advanced again to `bb154d106` during CI, adding retained aioquic 1.5.0 +qlog 0.3 JSON parsing and qlog views. The existing capture summary/page path +already applies. Qlog's trace-scoped STREAM interval unions and endpoint loss +declarations have separate semantics: they do not recover bodies, prove network +loss or establish PCAP correlation. Measure genuine supported recordings and +existing retained views before adding delivery or caching. Other qlog layouts +and qlog performance remain unverified in the Flutter experiment. Shared-view +integration is checked after rebasing; earlier runtime receipts remain pinned. diff --git a/experiments/analysis-efficiency/measure.mjs b/experiments/analysis-efficiency/measure.mjs index 3e34d935e..160973aa7 100644 --- a/experiments/analysis-efficiency/measure.mjs +++ b/experiments/analysis-efficiency/measure.mjs @@ -28,6 +28,7 @@ const { values } = parseArgs({ "javascript-question": { type: "string", default: "inventory" }, "managed-question": { type: "string", default: "inventory" }, schemas: { type: "string", default: "default" }, + "sqlite-selection": { type: "string", default: "all" }, }, }); if ( @@ -43,6 +44,11 @@ if ( ) || !["inventory", "selected"].includes(values["javascript-question"]) || !["default", "selected"].includes(values.schemas) || + !["all", "columns"].includes(values["sqlite-selection"]) || + (values.artifact !== "qgis-world-map.gpkg" && + values["sqlite-selection"] !== "all") || + (values.artifact === "qgis-world-map.gpkg" && + values.delivery !== "complete") || !["inventory", "identity"].includes(values["managed-question"]) || (!["sharex.dll", "sharex.helpers.dll"].includes(values.artifact) && values["managed-question"] !== "inventory") || @@ -518,6 +524,54 @@ const inspectEvmTarget = async () => { }; const workflows = { + "qgis-world-map.gpkg": async () => { + const columns = ["fid", "NAME", "ISO_A3"]; + const input = { + path: values.path, + table: "countries", + row_limit: 1, + ...(values["sqlite-selection"] === "columns" ? { columns } : {}), + }; + const definition = catalogDefinitions.get("inspect_sqlite_database"); + if ( + definition === undefined || + !new Ajv2020({ strict: false, validateFormats: false }).validate( + definition.inputSchema, + input, + ) + ) + throw new Error( + "Selected SQLite request is not advertised by this runtime", + ); + const inspected = await call("inspect_sqlite_database", input); + const report = normalized(inspected); + if (report.rows === null || report.rows.table !== "countries") + throw new Error("Runtime returned no selected country sample"); + const indexes = columns.map((column) => { + const index = report.rows.columns.indexOf(column); + if (index < 0) + throw new Error(`Missing requested SQLite column ${column}`); + return index; + }); + return { + artifact_sha256: targetSha(inspected), + artifact: report.artifact, + wal: report.wal, + engine: report.engine, + database_encoding: report.database_encoding, + schema: report.schema, + rows: { + ...report.rows, + columns, + values: report.rows.values.map((row) => + indexes.map((index) => row[index]), + ), + }, + producer_limitations: report.limitations, + limitations: inspected.limitations, + evidence_ids: [inspected.evidence_id], + }; + }, "obsidian.asar": async () => { if (values["javascript-question"] === "selected") { const inspected = await call("analyze_javascript_application", { @@ -666,7 +720,40 @@ const workflows = { } }, "newpipe.apk": async () => { - const pkg = await call("inspect_android_package", { path: values.path }); + const packageInput = { + path: values.path, + ...(values.delivery === "summary" ? { detail: "summary" } : {}), + }; + const definition = catalogDefinitions.get("inspect_android_package"); + if ( + definition === undefined || + !new Ajv2020({ strict: false, validateFormats: false }).validate( + definition.inputSchema, + packageInput, + ) + ) + throw new Error( + "Selected Android delivery is not advertised by this runtime", + ); + const pkg = await call("inspect_android_package", packageInput); + const result = normalized(pkg); + const report = values.delivery === "summary" ? result.summary : result; + if ( + values.delivery === "summary" && + (result.kind !== "summary" || + result.parent_operation !== "inspect_android_package") + ) + throw new Error("Runtime returned no retained Android package summary"); + const { manifest, ...declarations } = report; + const manifestObservation = + values.delivery === "summary" + ? manifest + : { + status: manifest.status, + reported_total_bytes: manifest.reported_total_bytes, + text_code_units: manifest.text.length, + text_included: false, + }; const found = await call("search_android_classes", { path: values.path, query: "MainActivity", @@ -677,9 +764,12 @@ const workflows = { }); return { artifact_sha256: targetSha(pkg), - package_name: normalized(pkg).package_name, - version_name: normalized(pkg).version_name, - version_code: normalized(pkg).version_code, + package_name: report.package_name, + version_name: report.version_name, + version_code: report.version_code, + package_declarations: declarations, + manifest_observation: manifestObservation, + package_limitations: pkg.limitations, matches: normalized(found).matches, coverage: normalized(found).coverage, onCreate: normalized(cls).methods.filter( @@ -690,6 +780,54 @@ const workflows = { }, "sharex.dll": inspectManagedTarget, "sharex.helpers.dll": inspectManagedTarget, + "localsend-arm64.apk": async () => { + const input = { + path: values.path, + ...(values.delivery === "summary" ? { detail: "summary" } : {}), + }; + const definition = catalogDefinitions.get("inspect_dart_aot"); + if ( + definition === undefined || + !new Ajv2020({ strict: false, validateFormats: false }).validate( + definition.inputSchema, + input, + ) + ) + throw new Error( + "Selected Dart AOT delivery is not advertised by this runtime", + ); + const inspected = await call("inspect_dart_aot", input); + const result = normalized(inspected); + if ( + values.delivery === "summary" && + (result.kind !== "summary" || + result.parent_operation !== "inspect_dart_aot") + ) + throw new Error("Runtime returned no retained Dart AOT summary"); + const report = values.delivery === "summary" ? result.summary : result; + const { package_uris, dart_uris, class_like_tokens, ...pool } = + report.string_pool; + return { + artifact_sha256: targetSha(inspected), + target: report.target, + abi: report.abi, + libapp: report.libapp, + string_pool: + values.delivery === "summary" + ? pool + : { + ...pool, + returned_counts: { + package_uris: package_uris.length, + dart_uris: dart_uris.length, + class_like_tokens: class_like_tokens.length, + }, + }, + coverage: report.coverage, + limitations: inspected.limitations, + evidence_ids: [inspected.evidence_id], + }; + }, "openwrt.bin": async () => { const info = await call("inspect_firmware_regions", { path: values.path }); return { @@ -818,6 +956,10 @@ const report = { ) ? values["managed-question"] : null, + sqlite_selection: + values.artifact === "qgis-world-map.gpkg" + ? values["sqlite-selection"] + : null, provider_settings_sha256: sha(JSON.stringify(selectedProviders)), provider_search_path_sha256: sha( selectedProviders.PATH ?? process.env.PATH ?? "", diff --git a/experiments/analysis-efficiency/process-trace-validation.md b/experiments/analysis-efficiency/process-trace-validation.md new file mode 100644 index 000000000..a32848596 --- /dev/null +++ b/experiments/analysis-efficiency/process-trace-validation.md @@ -0,0 +1,113 @@ +# Process trace validation + +A 6,000-event directed cycle previously overflowed the JavaScript stack inside +the schema refinement. Cycle detection now uses iterative topological traversal. +Cyclic inputs retain independent declaration diagnostics without expanding all +event pairs. Unordered declarations retain group membership, with one diagnostic +per overlapping or conflicting group, instead of allocating a pair set and +repeating the same error for every pair. + +A unique topological order proves that every event pair is ordered, allowing +rank comparisons without retaining transitive closure. Other acyclic graphs +retain only one origin's forward and backward reachability sets at a time. +Complete pair validation preserves the missing count and first declaration-order +missing pair. General partial orders still require quadratic pair examination +and repeated graph traversal; they trade CPU time for bounded retained state. + +## Initial cycle fix comparison + +Flameox ran three randomized paired blocks (seed 917), each limited to 20 seconds +and 512 MiB. Node 24.18.0 used a 384-MiB old-space limit on macOS arm64. Each +process validated the same 2,500-event acyclic chain four times, then checked the +known 6,000-event cycle failure with per-event not-before constraints and one +unordered group containing every event. All six executions passed the embedded answer +digest and expected rejection checks. + +The baseline source is from `bb540826d`; the candidate is the initial cycle fix +in `10935b7b1`, which still memoized reachability sets for every queried origin. +Both modules had TypeScript annotations removed with Node's identical built-in +transform and used the same copied runtime dependencies. This isolates schema +validation rather than comparing complete packaged CLI/MCP processes. + +| Median measurement | Baseline | Candidate | +| -------------------------------------------- | -------: | --------: | +| First validation elapsed milliseconds | 335.80 | 177.25 | +| Warm validation elapsed milliseconds | 313.36 | 165.23 | +| First validation CPU milliseconds | 366.85 | 218.39 | +| Warm validation CPU milliseconds | 335.32 | 194.08 | +| Whole-process elapsed milliseconds | 1,344.13 | 787.00 | +| Process CPU milliseconds before final report | 1,457.67 | 941.05 | +| Peak process-family MiB | 328.36 | 395.52 | + +The input is identical: 233,095 JSON bytes, estimated at 77,010 tokens by tokenx +1.3.0. The valid parsed result digest is +`28bd99ce83f0da8e45600ef20f114cb52bb26b5f5e6bfc1601ca3e712eaf3147`. +Actual agent tokens, catalog injection, transport, model reasoning and provider +costs were not measured. This is a component measurement, not an end-to-end +investigation speed claim. + +First validation excludes module import and fixture preparation; warm runs reuse +the process but create fresh per-validation graph state. Whole-process elapsed +includes startup, imports, all four validations, the cyclic-input check, token +estimation and reporting. CPU includes startup through metrics publication; +peak RSS covers the complete single-process workload, with no children. Host +caches were not flushed. Peak memory increased in this cohort. Its retained +valid-order closure still required quadratic memory, prompting the follow-up. +Three pairs describe this fixture only. + +Preserved Flameox evidence: +`d6211639446e8b6ec2e82636913eaf471def96673e28f03e9c7f98bec07f8e13`; +analysis `e663b6bd8f5693975542d5b5e2b18437b3c167a0be939bb7386eb68549936231`. +An earlier cohort lost modules during another checkout build; its three failed +executions and timing results are excluded. The successful cohort uses an +isolated runtime copy. Source-module SHA-256 values are +`a7729faa2183b0e493b9ac4fc55f6feaee5b2e5294be13d1b29b57a44b899788` +(baseline) and `725409ce84902125941251037a8a214ccc3c1eec577e484c164c361ca580d693` +(candidate). + +## Bounded retention follow-up + +Flameox compared `10935b7b1` with the rank/streaming implementation in three +randomized paired blocks (seed 624), under the same host, Node, heap, memory and +20-second execution bounds. Each execution validated the chain above four times, +then a 1,200-event partial order with 600 unordered two-event layers and all four +edges between consecutive layers four times, then the same cyclic failure. +All six executions passed both parsed-answer digests and the cycle diagnostic. +Runtime dependencies were copied and isolated from concurrent checkout builds. + +| Median measurement | Memoized closure | Rank/streaming | +| ---------------------------------------- | ---------------: | -------------: | +| Chain first elapsed milliseconds | 166.11 | 11.37 | +| Chain warm elapsed milliseconds | 154.25 | 5.32 | +| Chain first CPU milliseconds | 210.13 | 24.38 | +| Chain warm CPU milliseconds | 184.76 | 16.84 | +| Partial order first elapsed milliseconds | 49.44 | 116.54 | +| Partial order warm elapsed milliseconds | 50.67 | 110.29 | +| Partial order first CPU milliseconds | 72.47 | 137.69 | +| Partial order warm CPU milliseconds | 70.65 | 121.77 | +| Capture process elapsed milliseconds | 901.24 | 557.30 | +| Process CPU milliseconds before report | 1,101.12 | 665.03 | +| Peak process-family MiB | 391.13 | 127.72 | + +First samples exclude module import and fixture preparation. Partial-order first +samples follow four chain validations, so they are first use of that fixture +within an already running process. Warm medians pool the three subsequent samples +from each of three processes. CPU and peak RSS cover the combined workload; +capture elapsed additionally includes Flameox capture startup and overhead. +Host caches were not flushed. The partial-order fixture is slower after this +change; these results establish a retention tradeoff, not universal acceleration. + +The partial-order input is 165,590 JSON bytes, estimated at 52,783 tokens by tokenx +1.3.0; its parsed-answer SHA-256 is +`41d06e593813e497221c36793bf00d5fece7055081895b5f6c123ba6ad9a8227`. +Chain bytes, token estimate and answer digest remain unchanged. Actual agent +tokens, transport, catalog injection, model reasoning and provider costs remain +unmeasured. Both cohorts are component measurements, not complete investigations. + +Preserved evidence: +`e29086af99afd8e67cbbd92e43d960ea40f73b4d318ea0cbca60f493575462b8`; +analysis `086133516f2acf031032d357a3ddb1ac611ce4f12556a24065a2becd567c5f7b`. +The baseline module SHA-256 is +`725409ce84902125941251037a8a214ccc3c1eec577e484c164c361ca580d693`; +the candidate module SHA-256 is +`2109f1096406e9ecac8bbf4f7ae8abb5113404790a9382487285d2831fcddf26`. diff --git a/experiments/analysis-efficiency/questions/flutter-aot.txt b/experiments/analysis-efficiency/questions/flutter-aot.txt new file mode 100644 index 000000000..c4c3c3897 --- /dev/null +++ b/experiments/analysis-efficiency/questions/flutter-aot.txt @@ -0,0 +1 @@ +Statically inspect the shipped Flutter APK at {{artifact_path}} using only the registered REA inspect_dart_aot tool. Use its default ABI selection. Report target identity, chosen ABI, every libapp ELF/snapshot field exactly as reported including null, original string-pool coverage, all three counts ending in _at_least, printable_run_count, and returned_counts for package_uris, dart_uris and class_like_tokens. Compute returned_counts from the returned arrays if needed. Preserve original overall coverage and producer limitations exactly, including order. Counts ending in _at_least are lower bounds and class-like tokens are byte-pattern observations, not recovered typed classes. Do not return individual strings or infer Dart SDK versions, class semantics, runtime behavior or complete inventory. Final answer must be JSON with artifact_sha256, target, abi, libapp, string_pool, coverage, limitations, evidence_ids. Use actual response Evidence IDs. Do not execute the APK, use shell or host file tools, access the network, or call other agents. Stop when these facts are supported. diff --git a/experiments/analysis-efficiency/questions/sqlite-qgis.txt b/experiments/analysis-efficiency/questions/sqlite-qgis.txt new file mode 100644 index 000000000..636c7a685 --- /dev/null +++ b/experiments/analysis-efficiency/questions/sqlite-qgis.txt @@ -0,0 +1 @@ +Statically inspect the supplied QGIS world-map GeoPackage at {{artifact_path}} using only the registered REA inspect_sqlite_database tool. Select the ordinary countries table with row_limit 1 and its default selected_table schema scope. Report the complete returned schema exactly as reported, including SQL, every declared column, indexes, views, triggers, completeness and explicit nulls. For the one returned sample, report only fid, NAME and ISO_A3 in that order, preserving the original tagged SQLite cells exactly: decimal integer strings, text values with bytes_hex and encoding, and SQL null. Preserve row_limit, returned_rows, truncated and order; row order is unspecified and this is not a complete country inventory. Preserve original artifact identity, wal including null, engine, database_encoding, both normalized-result producer limitations and outer Evidence limitations exactly including order. Do not return geometry blobs or unrequested sample cells, interpret boundaries/political labels as current facts, or infer missing values. Final JSON fields: artifact_sha256, artifact, wal, engine, database_encoding, schema, rows, producer_limitations, limitations, evidence_ids. Include real response Evidence IDs. Do not use shell or host file tools, network, other agents or execute QGIS. Stop once these facts are supported. diff --git a/experiments/analysis-efficiency/review-model.py b/experiments/analysis-efficiency/review-model.py index a61bb0984..0d6e1e8d7 100644 --- a/experiments/analysis-efficiency/review-model.py +++ b/experiments/analysis-efficiency/review-model.py @@ -69,7 +69,7 @@ def scope_with_verified_counts(value, expected_scope, returned_counts): def main(): parser = argparse.ArgumentParser() - parser.add_argument('--domain', choices=['wasm', 'evm', 'android', 'managed', 'managed-nonempty', 'firmware', 'inventory', 'javascript', 'native', 'browser'], required=True) + parser.add_argument('--domain', choices=['wasm', 'evm', 'android', 'flutter-aot', 'sqlite', 'managed', 'managed-nonempty', 'firmware', 'inventory', 'javascript', 'native', 'browser'], required=True) parser.add_argument('--trial', type=Path, required=True) parser.add_argument('--reference', type=Path) parser.add_argument('--resource-log', type=Path, required=True) @@ -133,6 +133,14 @@ def main(): for key, expected in {'artifact_bytes': n['artifact']['bytes'], 'bytecode_sha256': n['bytecode']['sha256'], 'bytecode_bytes': n['bytecode']['bytes'], 'functions': n['functions'], 'evidence_kind': n['evidence_kind'], 'discovery_completeness': n['discovery_completeness'], 'runtime_execution': n['runtime_execution'], 'bytecode_kind': n['bytecode']['kind'], 'hardfork': n['bytecode']['hardfork'], 'deployment_authenticity': n['bytecode']['deployment_authenticity']}.items(): check(key, answer.get(key), expected) check('evm_original_limitations', answer.get('limitations'), n['limitations']) + elif domain == 'sqlite': + for key in ['artifact', 'wal', 'engine', 'database_encoding', 'schema', 'rows', 'producer_limitations', 'limitations']: + # Missing is not SQL/JSON null; booleans are not numeric counters. + check(key, (key in answer, json.dumps(answer.get(key), sort_keys=True)), + (True, json.dumps(reference[key], sort_keys=True))) + elif domain == 'flutter-aot': + for key in ['target', 'abi', 'libapp', 'string_pool', 'coverage', 'limitations']: + check(key, answer.get(key), reference[key]) elif domain == 'android': for key in ['package_name', 'version_name', 'version_code', 'matches']: check(key, answer.get(key), reference[key]) diff --git a/experiments/analysis-efficiency/sqlite-existing-controls.md b/experiments/analysis-efficiency/sqlite-existing-controls.md new file mode 100644 index 000000000..9bac9f064 --- /dev/null +++ b/experiments/analysis-efficiency/sqlite-existing-controls.md @@ -0,0 +1,181 @@ +# SQLite existing-control measurements + +Continues [#2016](https://github.com/morluto/rea/issues/2016) after the Flutter +lane. The measured main runtime is pinned to +`172407787f1842dedbcd27b5385177595f786ce4`. Its only post-Flutter +commit repairs Windows transient runtime admission (#2467); this Linux round +does not establish Windows performance. No production/API change is proposed +merely because another summary or cache could be added. + +Before PR integration, main advanced to +`bb540826d65316c164d2de8ba4ef10f81a8ad073`. Its five intervening commits cover +MCP archive contracts/documentation, an advertised-schema inventory test, +pwntools cleanup identity, concurrent Mach-O slice inspection, and transient +DMG verification retries. None changes the SQLite selector. Rebase and current +main verification do not relabel the frozen measurements below. + +The committed post-format driver passes a real SDK smoke against the frozen +runtime. After rebase, a fresh build, real all/selected-column SDK pair and docs +check pass on the updated main. Selected facts match each other and the frozen +runtime; artifact bytes remain unchanged. These are integration checks, not +additional model performance pairs. + +## Real application data and question + +Use QGIS's bundled `world_map.gpkg`, unmodified from official immutable source +commit `e1223b8e9e71fc50ee995ed556f711c54ccc101b`. `corpus.json` pins its URL, +12,914,688-byte size, Git blob `c9ff80228dff0de93c9acdef16a97545ba360244`, and +SHA-256 `01d293159d1b77bd9212b5252587853f61aa9e25b2387135f34afaba8d057190`. +Streaming acquisition matches the upstream blob. The upstream resource install +list includes `data/*`; [its README](https://github.com/qgis/QGIS/blob/e1223b8e9e71fc50ee995ed556f711c54ccc101b/resources/data/README_world_map.md) +records data provenance. This is shipped application data, not a generated REA +fixture. QGIS is neither installed nor executed. + +The source question asks for the complete returned `countries` table schema +and one sample's `fid`, `NAME`, `ISO_A3` tagged cells, in that order. It preserves +database/WAL identity, engine/encoding, exact original SQL and columns, integer +strings, text bytes, nulls, row limit/count, unspecified order, truncation and +both producer/outer limitations. It does not require geometry blobs or a full +country inventory. Neither political labels nor the sampled ordering are +inferred to be current or globally exhaustive. + +SQLite already defaults to compact schema inventory without a table, detailed +selected-table schema with a table, and exposes exact `columns`. Selecting +columns avoids constructing unrelated cells without changing the requested +table schema. Thus compare existing all-visible columns against exact columns +on the same main runtime, rather than changing an old user's default or adding +a new delivery mechanism. Inventory scope cannot substitute for the detailed +schema requested by this question. + +## Public SDK observations + +Frozen runtime: main `172407787f1842dedbcd27b5385177595f786ce4`, compiled +dist/bridge file-table SHA-256 +`72a4ac6c6db491c455f3e4a798f8f52ed5850c08831e7cdc9f25c1525a5a1d11`. +Node 24.18.0, SDK client/server 2.3.1, observed SQLite 3.53.1, WAL null. Driver, +provider settings/search-path fingerprints, artifact and locked dependencies +remain equal. Each workflow starts a fresh server and validates the selected +actual advertised schema. Host page caches are not flushed. + +The initial default inventory returns 26 table entries, including explicit +virtual/shadow classifications, no rows and `not_requested` column details: +27,185 compact client-result JSON bytes. Selecting `countries` reveals eight +columns, including a geometry blob. Its one-row full reply is 485,277 bytes. +The selected request retains every requested declared-schema field and exact +tagged sample values while omitting unrelated sample cells. + +| Trial | Reply bytes | SDK workflow seconds | Payload CPU seconds | Kernel peak bytes | +| ------------- | ----------: | -------------------: | ------------------: | ----------------: | +| sdk-all-1 | 485,277 | 11.820 | 12.755 | 562,884,608 | +| sdk-columns-1 | 18,509 | 14.017 | 13.715 | 557,510,656 | +| sdk-columns-2 | 18,509 | 15.826 | 14.162 | 548,343,808 | +| sdk-all-2 | 485,277 | 5.679 | 7.310 | 561,704,960 | + +Both pairs make one successful call, pass settings/selected-fact comparison, +retain complete original schema and all requested typed cells, and preserve +source bytes. Reply bytes fall about 96.2%. These observations **do not show SDK +speed or CPU improvement**: selected workflows are slower in both pairs and +timing varies substantially. No cause is inferred from these data. Payload CPU +includes runner/catalog work outside the SDK workflow timer. Memory changes +are small and do not establish a general reduction. Raw replies and the exact +pre-format driver source are archived privately. + +Reproduce with `measure.mjs --artifact qgis-world-map.gpkg` and +`--sqlite-selection all|columns`, using the same `--runtime-root`, `--revision`, +artifact path and `--schemas selected`. `--delivery` stays `complete`; this is +an existing request selection, not a new summary API. Original caller arguments +and schemas remain in the private receipts. The source question is +`questions/sqlite-qgis.txt` and the selected factual grader is +`review-model.py --domain sqlite`. + +## Decision and scope + +The initial four actual Codex attempts never executed SQLite: the client rejected +the tool because its private snapshot writes require approval and the isolated +driver lacked this exact tool's approval override. Every factual grade fails; +these are failed client attempts, not completed-task performance comparisons. +All profiles are removed. The existing experiment-only per-tool approval list +now includes `inspect_sqlite_database`; each corrected trial exposes only that +exact enabled tool. REA annotations, target authority and global Codex configuration +are unchanged. The [official configuration reference](https://learn.chatgpt.com/docs/config-file/config-reference) +documents the per-tool override. Corrected cohorts use fresh output directories +and retain the original failures separately. + +Actual repeated Codex comparisons determine the task token/time benefit before +crediting smaller replies. An existing selector that answers the question is +evidence against adding another production mechanism. Keep negative outcomes +and unavailable coverage, including failure/retry costs and profile cleanup. + +Four corrected serial trials use Codex CLI 0.161.0, requested `gpt-5.6-terra` +high, one enabled tool, identical owned context/discovery capture, and the same +frozen main runtime. The requested alias is not a verified server-resolved model. +Equal treatment instructions select the existing all-visible or exact-column +request; the source question and requested schema/sample facts stay fixed. + +| Corrected trial | Input | Cached | Uncached | Output | Seconds | MCP calls | Facts | +| ------------------------ | -----: | -----: | -------: | -----: | ------: | --------: | ----- | +| approved-model-all-1 | 88,131 | 52,480 | 35,651 | 3,369 | 84.007 | 2 | pass | +| approved-model-columns-1 | 37,340 | 20,992 | 16,348 | 3,006 | 87.514 | 1 | pass | +| approved-model-columns-2 | 56,148 | 36,096 | 20,052 | 3,298 | 97.466 | 2 | pass | +| approved-model-all-2 | 88,340 | 52,480 | 35,860 | 3,084 | 113.847 | 2 | pass | + +Both pairs pass every selected producer fact, exact SQL/schema/null/tagged +cells, observed response Evidence IDs, target-byte preservation and profile +cleanup. The grader checks producer observations, not independent geometry +semantics. Actual events contain only the chosen MCP tool and agent messages; +no shell or other agent execution appears. Usage is the final aggregate actually +reported by the CLI, one usage record per trial; cached input remains included +in total input. Output includes 853 / 357 / 523 / 664 reasoning tokens. + +Total input falls **57.63% / 36.44%**, uncached input **54.14% / 44.08%**. +Time is mixed: columns are **4.17% slower** in pair 1 and **14.39% faster** in +pair 2. Output is 10.77% lower / 6.94% higher. Repeated calls remain counted; +pair 1 has two all-column calls versus one selected call, while pair 2 has two +calls on both sides. This supports scoped token savings from using the existing +selector, not a dependable speedup or a new API requirement. Client context and +usage are observed separately from raw SDK reply bytes. + +Model payload CPU seconds / kernel peaks are 7.596 / 763,838,464; +7.772 / 761,143,296; 8.409 / 762,597,376; and 11.759 / 774,025,216. CPU is mixed +and memory differences are small. Payload cleanup/limit observations pass; +missing terminal counters keep every strict whole-resource grade false. + +The original blocked attempts remain separate and never provide comparison +denominators. They each have one failed call, no SQLite result and failed facts: + +| Blocked trial | Input | Cached | Uncached | Output | Seconds | +| --------------- | -----: | -----: | -------: | -----: | ------: | +| model-all-1 | 35,290 | 29,952 | 5,338 | 798 | 59.881 | +| model-columns-1 | 35,356 | 22,016 | 13,340 | 670 | 44.764 | +| model-columns-2 | 35,034 | 20,992 | 14,042 | 639 | 30.007 | +| model-all-2 | 35,104 | 20,992 | 14,112 | 673 | 25.132 | + +**Decision: no production/API change.** Existing column selection materially +reduces unnecessary delivery and actual model input while retaining the complete +requested schema and exact cells. Existing documentation already recommends +table/column selection. No additional summary, schema-scope default change, +automatic field omission, plugin layer or cross-session cache is justified by +these results. Preserve the negative/mixed time, output and CPU findings. + +Real one-shot CLI inventory, all-column sample and exact-column sample match +the corresponding full MCP normalized results, subjects, parameters and IDs; +the source digest remains unchanged. An independent Python binding using SQLite +3.40.1 opens this immutable artifact read-only and agrees with the exact stored +table SQL, eight declared columns and selected tagged sample cells. This is a +second engine/binding observation of this file, not a general row-order or +geometry-semantics guarantee. The six existing context/driver regression tests +pass; Python compilation and JavaScript syntax checks pass. No new product test +or production implementation is added for this measurement-only finding. + +Manual review tightens the SQLite grader to distinguish missing fields from +explicit null and preserve JSON scalar types. Regrading all four stored model +answers still passes every selected fact; deliberate omission of `wal` and a +boolean substituted for numeric `row_limit` both fail. Original transcripts, +usage and previous review receipts stay intact; no model trial is rerun. + +Heavy tasks remain serial under one CPU, unchanged 3 GiB early stop / 4 GiB hard +memory, no swap, owned task scopes and host guards. SDK payload observations +verify cleanup and zero phase limit events; final whole-scope counters remain +unknown. This does not establish live/WAL atomicity, all SQLite applications, +other runtimes/platforms or geospatial geometry semantics. Other packet/signal +and capture lanes remain incremental follow-ups using their existing controls. diff --git a/experiments/analysis-efficiency/test_codex_context.py b/experiments/analysis-efficiency/test_codex_context.py index 37df83a7c..0b8e0247c 100644 --- a/experiments/analysis-efficiency/test_codex_context.py +++ b/experiments/analysis-efficiency/test_codex_context.py @@ -71,7 +71,9 @@ class ContextCaptureTests(unittest.TestCase): self.assertNotIn(marker, text) self.assertIn('caller-selected evidence and paths', text) self.assertIn('original-response', text) - self.assertEqual((output / 'codex-tool-context.jsonl').stat().st_mode & 0o777, 0o600) + # Windows mode bits do not describe access control; this is a POSIX assertion. + if os.name == 'posix': + self.assertEqual((output / 'codex-tool-context.jsonl').stat().st_mode & 0o777, 0o600) def test_missing_rollouts_are_unknown_and_malformed_lines_remain_partial(self): with tempfile.TemporaryDirectory() as directory: @@ -124,6 +126,7 @@ class ContextCaptureTests(unittest.TestCase): patch.object(agent.subprocess, 'check_output', return_value='v24.18.0\n'), \ patch.object(agent.subprocess, 'Popen', return_value=child), \ patch.object(agent, 'capture_tool_context', side_effect=OSError('fixture capture failure')), \ + patch.object(agent.Path, 'home', side_effect=AssertionError('Explicit CODEX_HOME must not resolve the default home')), \ patch('builtins.print'): with self.assertRaisesRegex(OSError, 'fixture capture failure'): agent.main() diff --git a/experiments/analysis-efficiency/windows-empty-profile.md b/experiments/analysis-efficiency/windows-empty-profile.md new file mode 100644 index 000000000..f8e5558e0 --- /dev/null +++ b/experiments/analysis-efficiency/windows-empty-profile.md @@ -0,0 +1,69 @@ +# Windows empty-profile startup verification + +Follow-up to [#2345](https://github.com/morluto/rea/pull/2345), including +@gefrezim's diagnosis of the import-time Windows home lookup. + +## Source and boundary + +Baseline: main `172407787f1842dedbcd27b5385177595f786ce4`. Verification used +the real Windows 11 10.0.22631.0 host, official Windows x64 Node 24.18.0, +locked production dependencies and REA's compiled public CLI / SDK 2.3.1 MCP +server. No IDA or Ghidra engine was used. Runtime copies and compact receipts +live outside the checkout; no account profile path or ambient environment is +recorded here. + +The tested dispatcher SHA-256 is +`526105418604cc6e5a202a6275b700814e27450bfaffd45e7510530ba669de8d`. +Only the dispatcher changed in the deployed runtime. It treats an empty Windows +`USERPROFILE` as unset before dependency imports. Nonempty selections continue +to reach the existing home resolver; no additional lookup or fallback owner is +introduced. + +## Real Windows observations + +With `HOME` empty in each case, the baseline produced: + +| USERPROFILE selection | `--version` | `--help` | MCP discovery | +| --------------------------- | ----------------------- | ----------------------- | ------------- | +| Unset | Exit 0 | Exit 0 | Healthy | +| Empty | Exit 1, `uv_os_homedir` | Exit 1, `uv_os_homedir` | Healthy | +| Nonempty test-owned profile | Exit 0 | Exit 0 | Healthy | + +The corrected CLI passed all six invocations with no stderr. Version output +remained 6 bytes and help output 15,497 bytes. Real setup dry runs with empty +and nonempty profile selections both exited 0, planned their configuration +under the expected home, preserved existing configuration bytes and created no +configuration under the working directory. No installation was performed. + +MCP was already healthy in the baseline; this change does not claim to repair +an MCP startup failure. Complete baseline discovery was 187 tools and 2,545,790 +serialized bytes, SHA-256 +`1e6bbca7f62852134eb703ea098f5a302d1fe333b8433475429f6fc1b1aba18f`. +All three corrected MCP controls also passed with no stderr and the same +catalogue bytes and digest. + +The existing `homeDirectoryFallback` acceptance test exercises the import-time +failure through the public setup dry run. Adding it to the Windows curated lane +makes that regression execute on the affected platform. No duplicate startup +test was retained. Nine related tests passed, including the existing MCP doctor +and curated-lane path checks; Linux passing alone does not prove Windows behavior. +Review caught a missing package-scope registration for the newly listed test; +the original CI shard failed that ownership invariant. The path is now also +registered in `scripts/ci/scopes.mjs`, and all 84 scope checks plus the two +curated-path checks pass. Runtime source and its tested digest are unchanged. + +## Efficiency and resource limits + +Heavy host probes ran serially at BelowNormal priority, with one CPU and a +512 MiB CLI/server heap setting (256 MiB for the probe parent). Heap settings do +not bound total process memory. An exit-time `process.resourceUsage()` observer +recorded individual corrected CLI invocations: CPU time ranged from 7.968 to +9.874 seconds and peak working set from 299,956 to 301,852 KiB. These figures +exclude the probe parent and any observer overhead outside the CLI process. +The six observed wall times ranged from 8.656 to 10.961 seconds. + +The baseline did not use that resource observer, and cold/warm caches were not +normalized. These observations establish functional behavior and bounded probe +scope; no CPU, memory or latency improvement is claimed. The fix adds one +constant-time environment check, no filesystem lookup, engine call, retry or +cache. Output bytes are not model tokens; actual model usage was not measured. diff --git a/experiments/analysis-efficiency/windows-posix-reconciliation.md b/experiments/analysis-efficiency/windows-posix-reconciliation.md new file mode 100644 index 000000000..0df06ac3e --- /dev/null +++ b/experiments/analysis-efficiency/windows-posix-reconciliation.md @@ -0,0 +1,128 @@ +# Windows reconciliation of recorded POSIX paths + +Follow-up to [#2422](https://github.com/morluto/rea/pull/2422), including +@Aurantium-47's diagnosis of the host-dependent relative-path operation. + +## Source pins and real data + +The initial reproduction and passive producer used main +fa4b8288655188426222773ead2ccbdaf76334fd. Before branching, main was updated +to 8b181716aca2cb467be499cba226abfb5ae19141. The owning matcher and its +affected reconciliation consumers were unchanged between those revisions. +The newer baseline again reproduced both errors on the real Windows host. + +Input: npm 11.16.0's published lib/utils/validate-lockfile.js, 1,023 bytes, +SHA-256 2295e2dac1c5816e1eae2d417804960ebbef17b64a0a44db4f3483ce6e6d9526. +The installed copy was independently checked against the exact module in the +[official registry package](https://registry.npmjs.org/npm/11.16.0). +The 2,945,850-byte tarball matched its published SHA-512 SRI before reading +that member; no archive extraction or installation occurred, and the tarball +was not retained. + +Two actual Linux filesystem directories, Runtime/App and Runtime/app, +held byte-identical module copies and had distinct directory identities. +A controlled Node 24.18.0 process loaded the public module and ran its real +function against 587 locked dependency records: compatible trees returned zero +errors, and one controlled version mismatch returned one error. A passive REA +Inspector capture recorded 114 loaded scripts, including the selected module. +The provider did not evaluate, pause, resume or read source. The owned process +exited normally. Function execution is established by the harness receipt; +REA's passive script metadata establishes loaded status. + +Production static analysis inspected both directories plus an independent +Static/public-module copy. The latter prevents implicit root matching from +hiding the explicit mapping result. Complete static/runtime Evidence was then +consumed by the actual **Windows** CLI and SDK 2.3.1 stdio MCP server, using +Windows 11 10.0.22631.0 and official Windows x64 Node 24.18.0. +No Electron GUI, IDA or Ghidra engine was used. + +## Correctness and provenance + +| Windows reconciliation | Baseline | Candidate | +| -------------------------------------- | ------------------------------ | ----------------------------------- | +| Exact POSIX input root | Matched, artifact path | Same complete normalized result | +| Case-mismatched POSIX input root | Incorrectly matched and loaded | Unknown; static load states unknown | +| Exact explicit POSIX mapping | Matched, operator file mapping | Same complete normalized result | +| Case-mismatched explicit POSIX mapping | Incorrectly matched and loaded | Unknown; static load states unknown | + +All four CLI and same-connection MCP normalized results agree. The input +Evidence, recorded runtime facts and caller-selected root spellings are +preserved. Rejected location membership does not establish that a module was +absent or unexecuted. The operation remains offline and performs no target or +filesystem inspection. + +The shared matcher now chooses posix.relative after identifying POSIX input; +Windows coordinates continue to use win32.relative. Both location matching +and active-layer scope use this owner. The obsolete host-default import is +removed; no downstream repair or new path abstraction is introduced. + +The related suite passed 102 tests: 15 workflow tests, 85 CI scope tests and +two curated-path checks. Four simulated Electron Evidence controls cover +POSIX mismatch/exact matching, Windows case-insensitive matching, the resulting +asset load state and retained mapping spelling. They are service regressions, +not real Electron-provider verification. The existing workflow test file is +registered in both the Windows execution lane and its package-change scope. + +## Delivery and resource observations + +Probes ran serially at BelowNormal priority with one CPU and 512 MiB Node heap +settings for CLI/server/client work. Heap settings do not bound aggregate memory. +Four corrected CLI exit-time observations reported 7.343–15.827 seconds CPU and +322,308–324,280 KiB peak working set, excluding the probe parent. Wall times +ranged from 7.785 to 17.601 seconds. There is no paired resource baseline. + +One complete MCP trial connected in 18,296 ms and discovered 189 tools in +875 ms (2,588,336 serialized bytes). The four subsequent same-connection calls +took 879, 320, 353 and 422 ms. Their serialized replies were 918,133 / 913,765 / +918,571 / 914,171 bytes; inputs totalled 681,312 bytes. Discovery plus these +inputs and replies totalled 6,934,288 bytes, excluding initialization, framing +and notifications. This is delivery volume, not model token usage. Actual +model context and total model task cost were not measured. + +A process-generation sampler recorded 233 observations over 25,671 ms: +15,797 ms aggregate observed CPU as a lower bound and a peak sampled working-set +sum of 491,597,824 bytes (about 469 MiB). Exited-process tails can be missed; +shared pages can be counted twice. This excludes the observer and is not an +exact physical-memory peak. The server wrote four structured completion records +to stderr; tool replies and normalized results passed the recorded checks. + +Cold/warm caches and observer differences were not normalized. No latency, +CPU, memory or model-token improvement is claimed. The fix adds no lookup I/O, +engine invocation, retry or cache. Complete default replies remain available; +only compact selected findings and hashes were sent to the investigating agent. + +## Main integration replay + +Runtime cb0d8a9fef2b907cff2ba27f1e017493e1173eb2, rebased onto main +73dd8d6ed, again passed the 101 related tests and all four real Windows CLI/MCP +cases using the unchanged older producer Evidence. Positive results and original +runtime/layer facts remain identical. Typecheck, lint, dependency and metadata +checks passed; all five changed files passed formatting independently. +The full format gate at that main revision failed on the unchanged upstream +GhidraDoctor.test.ts and GhidraSnapshotCache.test.ts files. The PR does not +repair those unrelated baseline files. Resource figures above refer to the +earlier candidate; the integration replay does not establish a speedup. + +Review found that owning only the Windows test path would still skip that lane +on a later implementation-only edit. The matcher implementation is now also +registered in package scope. A source-only classifier case verifies that it +selects both Windows package and JavaScript recovery lanes; 85 scope checks and +two curated-path checks pass. Runtime code and the real-data checks are unchanged. + +The original production fix merged as +[1a1b67c88](https://github.com/morluto/rea/commit/1a1b67c88) while this follow-up +was open. The follow-up was then rebased onto main fb28ec153 and contains only +the stronger workflow assertions, Windows lane ownership, scope classification +and this verification record. The updated 131 related tests, two representative +Ghidra boundary tests, seven IDA input-thinning tests, 24 semantic-structure +tests, 16 required-lane checks, 39 release-checkpoint tests and 11 dyld CLI +tests passed after the rebase (230 total). +Upstream a0317cf73 corrected the invalid FAT fixture and consolidated unrelated +duplicated helpers; e05d3399f aligned the older parameter-provenance assertion +with the upstream semantics. Those changes, the threshold change and the new +CI qualification tiers are part of main, outside this follow-up's diff. +Upstream 747e72991 restored release preflight coverage for reusable workflow +jobs. The independently reviewed, fully green #2755 fixture correction merged +as fb28ec153 and restored real x64 dyld CLI admission without altering the +original same-named arm64e assertions. +No production source is duplicated by the follow-up. diff --git a/experiments/analysis-efficiency/windows-runtime-retry.md b/experiments/analysis-efficiency/windows-runtime-retry.md new file mode 100644 index 000000000..2286727e8 --- /dev/null +++ b/experiments/analysis-efficiency/windows-runtime-retry.md @@ -0,0 +1,107 @@ +# Windows runtime admission retry verification + +Scope: [#2257](https://github.com/morluto/rea/issues/2257). Related initial +diagnosis: [#2266](https://github.com/morluto/rea/pull/2266). + +## Revisions and real input + +The baseline is `4f4f6cd7b5d7a400b90772a138d7b9f942c2a556`. +The candidate was developed from +`4773c152bc5f299fa682575bb8d1efea7587c7bc`; relevant runtime, native and +dependency-lock sources are identical between those two revisions. +These measurements precede integration of subsequent main changes. + +The host ran Windows 11 10.0.22631.0, official Windows x64 Node 24.18.0, +and locked production dependencies. The real application input was the official +npm 11.16.0 launcher distribution's `bin` directory. REA inspected that source; +it did not execute the application as an analysis target. Runtime copies, +raw replies and test executables remain outside the checkout. No IDA or Ghidra +engine was involved in this native-runtime verification. + +## Correctness and recovery + +A test-owned directory handle requesting `FILE_ADD_FILE` produced an actual +kernel sharing violation (Win32 32) during native runtime-parent admission. +Releasing that handle restored admission. Before the fix, public CLI analysis +reported a permanent `unsupported_provider` failure with `retryable: false`. +The corrected CLI preserves `capability_unavailable` and the reason, with +`category: unavailable` and `retryable: true`. + +Normal CLI analysis and post-conflict recovery produced the same canonical +normalized result before and after the correction: 535,733 compact JSON bytes, +SHA-256 `a24d3c2d9897cccc5ba70e9ee4dce03da88f033a939fe87457136fc5b3a3e2db`. +The complete successful CLI reply remained 537,124 bytes. + +A single real SDK 2.3.1 stdio MCP connection first completed analysis, then +encountered the same kernel conflict after native authority caching, then +recovered successfully when the handle was released. Its final receipt passed +and found no residual runtime directories. The native producer supplies +`runtimeAllocation: "not-started"` only when parent admission failed before +directory creation. The adapter requires that fact and exact numeric codes +32/33; permission failures, opaque error text and unknown allocation state +remain nonretryable. Incomplete cleanup also prevents retry. + +The native verifier additionally reproduces a real sharing conflict using an +owned runtime's DELETE lease, verifies the original error and allocation fact, +and confirms no child directory was allocated. This case runs unconditionally +in the Windows native verifier, without the optional process-boundary fixture. +The full verifier passed with 20 positive controls. Its optional independently +built process-boundary fixture was not supplied in this run. + +The related focused suite passed 99 tests across seven files, including worker +ownership, cancellation, cleanup, error projection, loader behavior and adapter +boundaries. Source-map and WABT acquisition-failure tests use injected failures; +they establish disposition propagation, not real engine execution. + +Thirty baseline native self-tests passed with no residual directories. +Win32 5 from #1701/#2196 was not reproduced on this host; these results do not +establish its cause or resolution. + +## Delivery and resource measurements + +One final same-connection MCP trial used one CPU per Node process, BelowNormal +priority, a 512 MiB client heap setting and a 768 MiB server heap setting. +Heap settings do not bound total process or family memory. + +| Stage | Observed wall time | Serialized reply bytes | +| ---------------------------------- | -----------------: | ---------------------: | +| Connect | 11,351 ms | Not recorded | +| Complete tool discovery, 187 tools | 744 ms | 2,498,686 | +| Normal analysis | 12,887 ms | 1,126,797 | +| Sharing failure | 265 ms | 878 | +| Recovery on the same connection | 5,578 ms | 1,126,797 | + +The earlier warm failure reply was 1,218 bytes. Successful reply sizes and the +catalogue digest were unchanged within the tested source comparison: +`7bdadddc909f4b2b093f0b3da9f79fafdac879c9d3c147938b8fbce1b52cd726`. +These are delivery bytes, not model tokens. Model-visible discovery handling, +actual model token usage and total model task cost were not measured. + +A targeted process-generation sampler collected 343 observations over +35,630 ms. Its aggregate observed CPU time was a **lower bound** of 26,766 ms: +process exits can hide the final CPU tail. Its peak sampled working-set sum was +727,244,800 bytes (about 694 MiB). This is a sampled sum, can double-count shared +pages and is not an exact physical-memory peak. Only the test-owned Node and +directory-holder processes were included; the observer itself was excluded. + +This is one controlled functional trial, with no paired CPU/memory baseline. +Cold/warm filesystem caches and host noise were not normalized. No speedup, +CPU reduction, memory reduction or model-token savings are claimed. The change +adds no analysis retry loop or cache and performs no additional engine calls. +Existing successful native authority caching remains intact; failed readiness +is retried by a later caller attempt. Retry becomes useful advice only after +the transient condition clears and cleanup permits another attempt. + +## Integration replay + +After rebasing onto main `a2d8b94dd`, runtime candidate +`2c7a09eaf3ffe26667b40081f1cf0fd2ce44a146` again passed the 99 related tests, +static and documentation gates, real Windows CLI and same-connection MCP +normal/conflict/recovery sequence. Both workflows found no residual directories; +the CLI normalized-result digest above still matched. The MCP success and error +reply sizes remained 1,126,797 and 878 bytes respectively. Upstream Android/SAZ +contract changes increased complete discovery to 2,517,904 bytes, digest +`d14588eab7404ad129907eb34a45a0db0d43233c4de829f76babd59723bed199`. +This later catalogue is not evidence of a delivery-size change caused by the +runtime-admission fix. Resource sampling above belongs to the earlier pinned +candidate; aggregate resources were not resampled for this integration replay. diff --git a/install.sh b/install.sh index 46260fe71..c18726611 100755 --- a/install.sh +++ b/install.sh @@ -167,7 +167,7 @@ main() { [ "$installed_version" = "$version" ] || fail "installed version $installed_version does not match $version. Reinstall the requested version, then retry." printf 'REA %s is installed.\n' "$version" - if [ "$start_setup" = true ] && [ -r /dev/tty ] && [ -w /dev/tty ] && [ -t 1 ]; then + if [ "$start_setup" = true ] && [ -r /dev/tty ] && [ -w /dev/tty ] && [ -t 1 ] && [ -t 2 ]; then printf 'Starting REA setup...\n' "$install_bin" setup /dev/tty else diff --git a/licenses/LGPL-2.0.txt b/licenses/LGPL-2.0.txt new file mode 100644 index 000000000..ede5d5441 --- /dev/null +++ b/licenses/LGPL-2.0.txt @@ -0,0 +1,480 @@ + GNU LIBRARY GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1991 Free Software Foundation, Inc. + + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + +[This is the first released version of the library GPL. It is + numbered 2 because it goes with version 2 of the ordinary GPL.] + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +Licenses are intended to guarantee your freedom to share and change +free software--to make sure the software is free for all its users. + + This license, the Library General Public License, applies to some +specially designated Free Software Foundation software, and to any +other libraries whose authors decide to use it. You can use it for +your libraries, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if +you distribute copies of the library, or if you modify it. + + For example, if you distribute copies of the library, whether gratis +or for a fee, you must give the recipients all the rights that we gave +you. You must make sure that they, too, receive or can get the source +code. If you link a program with the library, you must provide +complete object files to the recipients so that they can relink them +with the library, after making changes to the library and recompiling +it. And you must show them these terms so they know their rights. + + Our method of protecting your rights has two steps: (1) copyright +the library, and (2) offer you this license which gives you legal +permission to copy, distribute and/or modify the library. + + Also, for each distributor's protection, we want to make certain +that everyone understands that there is no warranty for this free +library. If the library is modified by someone else and passed on, we +want its recipients to know that what they have is not the original +version, so that any problems introduced by others will not reflect on +the original authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that companies distributing free +software will individually obtain patent licenses, thus in effect +transforming the program into proprietary software. To prevent this, +we have made it clear that any patent must be licensed for everyone's +free use or not licensed at all. + + Most GNU software, including some libraries, is covered by the ordinary +GNU General Public License, which was designed for utility programs. This +license, the GNU Library General Public License, applies to certain +designated libraries. This license is quite different from the ordinary +one; be sure to read it in full, and don't assume that anything in it is +the same as in the ordinary license. + + The reason we have a separate public license for some libraries is that +they blur the distinction we usually make between modifying or adding to a +program and simply using it. Linking a program with a library, without +changing the library, is in some sense simply using the library, and is +analogous to running a utility program or application program. However, in +a textual and legal sense, the linked executable is a combined work, a +derivative of the original library, and the ordinary General Public License +treats it as such. + + Because of this blurred distinction, using the ordinary General +Public License for libraries did not effectively promote software +sharing, because most developers did not use the libraries. We +concluded that weaker conditions might promote sharing better. + + However, unrestricted linking of non-free programs would deprive the +users of those programs of all benefit from the free status of the +libraries themselves. This Library General Public License is intended to +permit developers of non-free programs to use free libraries, while +preserving your freedom as a user of such programs to change the free +libraries that are incorporated in them. (We have not seen how to achieve +this as regards changes in header files, but we have achieved it as regards +changes in the actual functions of the Library.) The hope is that this +will lead to faster development of free libraries. + + The precise terms and conditions for copying, distribution and +modification follow. Pay close attention to the difference between a +"work based on the library" and a "work that uses the library". The +former contains code derived from the library, while the latter only +works together with the library. + + Note that it is possible for a library to be covered by the ordinary +General Public License rather than by this special one. + + GNU LIBRARY GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License Agreement applies to any software library which +contains a notice placed by the copyright holder or other authorized +party saying it may be distributed under the terms of this Library +General Public License (also called "this License"). Each licensee is +addressed as "you". + + A "library" means a collection of software functions and/or data +prepared so as to be conveniently linked with application programs +(which use some of those functions and data) to form executables. + + The "Library", below, refers to any such software library or work +which has been distributed under these terms. A "work based on the +Library" means either the Library or any derivative work under +copyright law: that is to say, a work containing the Library or a +portion of it, either verbatim or with modifications and/or translated +straightforwardly into another language. (Hereinafter, translation is +included without limitation in the term "modification".) + + "Source code" for a work means the preferred form of the work for +making modifications to it. For a library, complete source code means +all the source code for all modules it contains, plus any associated +interface definition files, plus the scripts used to control compilation +and installation of the library. + + Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running a program using the Library is not restricted, and output from +such a program is covered only if its contents constitute a work based +on the Library (independent of the use of the Library in a tool for +writing it). Whether that is true depends on what the Library does +and what the program that uses the Library does. + + 1. You may copy and distribute verbatim copies of the Library's +complete source code as you receive it, in any medium, provided that +you conspicuously and appropriately publish on each copy an +appropriate copyright notice and disclaimer of warranty; keep intact +all the notices that refer to this License and to the absence of any +warranty; and distribute a copy of this License along with the +Library. + + You may charge a fee for the physical act of transferring a copy, +and you may at your option offer warranty protection in exchange for a +fee. + + 2. You may modify your copy or copies of the Library or any portion +of it, thus forming a work based on the Library, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) The modified work must itself be a software library. + + b) You must cause the files modified to carry prominent notices + stating that you changed the files and the date of any change. + + c) You must cause the whole of the work to be licensed at no + charge to all third parties under the terms of this License. + + d) If a facility in the modified Library refers to a function or a + table of data to be supplied by an application program that uses + the facility, other than as an argument passed when the facility + is invoked, then you must make a good faith effort to ensure that, + in the event an application does not supply such function or + table, the facility still operates, and performs whatever part of + its purpose remains meaningful. + + (For example, a function in a library to compute square roots has + a purpose that is entirely well-defined independent of the + application. Therefore, Subsection 2d requires that any + application-supplied function or table used by this function must + be optional: if the application does not supply it, the square + root function must still compute square roots.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Library, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Library, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote +it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Library. + +In addition, mere aggregation of another work not based on the Library +with the Library (or with a work based on the Library) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may opt to apply the terms of the ordinary GNU General Public +License instead of this License to a given copy of the Library. To do +this, you must alter all the notices that refer to this License, so +that they refer to the ordinary GNU General Public License, version 2, +instead of to this License. (If a newer version than version 2 of the +ordinary GNU General Public License has appeared, then you can specify +that version instead if you wish.) Do not make any other change in +these notices. + + Once this change is made in a given copy, it is irreversible for +that copy, so the ordinary GNU General Public License applies to all +subsequent copies and derivative works made from that copy. + + This option is useful when you wish to copy part of the code of +the Library into a program that is not a library. + + 4. You may copy and distribute the Library (or a portion or +derivative of it, under Section 2) in object code or executable form +under the terms of Sections 1 and 2 above provided that you accompany +it with the complete corresponding machine-readable source code, which +must be distributed under the terms of Sections 1 and 2 above on a +medium customarily used for software interchange. + + If distribution of object code is made by offering access to copy +from a designated place, then offering equivalent access to copy the +source code from the same place satisfies the requirement to +distribute the source code, even though third parties are not +compelled to copy the source along with the object code. + + 5. A program that contains no derivative of any portion of the +Library, but is designed to work with the Library by being compiled or +linked with it, is called a "work that uses the Library". Such a +work, in isolation, is not a derivative work of the Library, and +therefore falls outside the scope of this License. + + However, linking a "work that uses the Library" with the Library +creates an executable that is a derivative of the Library (because it +contains portions of the Library), rather than a "work that uses the +library". The executable is therefore covered by this License. +Section 6 states terms for distribution of such executables. + + When a "work that uses the Library" uses material from a header file +that is part of the Library, the object code for the work may be a +derivative work of the Library even though the source code is not. +Whether this is true is especially significant if the work can be +linked without the Library, or if the work is itself a library. The +threshold for this to be true is not precisely defined by law. + + If such an object file uses only numerical parameters, data +structure layouts and accessors, and small macros and small inline +functions (ten lines or less in length), then the use of the object +file is unrestricted, regardless of whether it is legally a derivative +work. (Executables containing this object code plus portions of the +Library will still fall under Section 6.) + + Otherwise, if the work is a derivative of the Library, you may +distribute the object code for the work under the terms of Section 6. +Any executables containing that work also fall under Section 6, +whether or not they are linked directly with the Library itself. + + 6. As an exception to the Sections above, you may also compile or +link a "work that uses the Library" with the Library to produce a +work containing portions of the Library, and distribute that work +under terms of your choice, provided that the terms permit +modification of the work for the customer's own use and reverse +engineering for debugging such modifications. + + You must give prominent notice with each copy of the work that the +Library is used in it and that the Library and its use are covered by +this License. You must supply a copy of this License. If the work +during execution displays copyright notices, you must include the +copyright notice for the Library among them, as well as a reference +directing the user to the copy of this License. Also, you must do one +of these things: + + a) Accompany the work with the complete corresponding + machine-readable source code for the Library including whatever + changes were used in the work (which must be distributed under + Sections 1 and 2 above); and, if the work is an executable linked + with the Library, with the complete machine-readable "work that + uses the Library", as object code and/or source code, so that the + user can modify the Library and then relink to produce a modified + executable containing the modified Library. (It is understood + that the user who changes the contents of definitions files in the + Library will not necessarily be able to recompile the application + to use the modified definitions.) + + b) Accompany the work with a written offer, valid for at + least three years, to give the same user the materials + specified in Subsection 6a, above, for a charge no more + than the cost of performing this distribution. + + c) If distribution of the work is made by offering access to copy + from a designated place, offer equivalent access to copy the above + specified materials from the same place. + + d) Verify that the user has already received a copy of these + materials or that you have already sent this user a copy. + + For an executable, the required form of the "work that uses the +Library" must include any data and utility programs needed for +reproducing the executable from it. However, as a special exception, +the source code distributed need not include anything that is normally +distributed (in either source or binary form) with the major +components (compiler, kernel, and so on) of the operating system on +which the executable runs, unless that component itself accompanies +the executable. + + It may happen that this requirement contradicts the license +restrictions of other proprietary libraries that do not normally +accompany the operating system. Such a contradiction means you cannot +use both them and the Library together in an executable that you +distribute. + + 7. You may place library facilities that are a work based on the +Library side-by-side in a single library together with other library +facilities not covered by this License, and distribute such a combined +library, provided that the separate distribution of the work based on +the Library and of the other library facilities is otherwise +permitted, and provided that you do these two things: + + a) Accompany the combined library with a copy of the same work + based on the Library, uncombined with any other library + facilities. This must be distributed under the terms of the + Sections above. + + b) Give prominent notice with the combined library of the fact + that part of it is a work based on the Library, and explaining + where to find the accompanying uncombined form of the same work. + + 8. You may not copy, modify, sublicense, link with, or distribute +the Library except as expressly provided under this License. Any +attempt otherwise to copy, modify, sublicense, link with, or +distribute the Library is void, and will automatically terminate your +rights under this License. However, parties who have received copies, +or rights, from you under this License will not have their licenses +terminated so long as such parties remain in full compliance. + + 9. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Library or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Library (or any work based on the +Library), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Library or works based on it. + + 10. Each time you redistribute the Library (or any work based on the +Library), the recipient automatically receives a license from the +original licensor to copy, distribute, link with or modify the Library +subject to these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 11. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Library at all. For example, if a patent +license would not permit royalty-free redistribution of the Library by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Library. + +If any portion of this section is held invalid or unenforceable under any +particular circumstance, the balance of the section is intended to apply, +and the section as a whole is intended to apply in other circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 12. If the distribution and/or use of the Library is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Library under this License may add +an explicit geographical distribution limitation excluding those countries, +so that distribution is permitted only in or among countries not thus +excluded. In such case, this License incorporates the limitation as if +written in the body of this License. + + 13. The Free Software Foundation may publish revised and/or new +versions of the Library General Public License from time to time. +Such new versions will be similar in spirit to the present version, +but may differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Library +specifies a version number of this License which applies to it and +"any later version", you have the option of following the terms and +conditions either of that version or of any later version published by +the Free Software Foundation. If the Library does not specify a +license version number, you may choose any version ever published by +the Free Software Foundation. + + 14. If you wish to incorporate parts of the Library into other free +programs whose distribution conditions are incompatible with these, +write to the author to ask for permission. For software which is +copyrighted by the Free Software Foundation, write to the Free +Software Foundation; we sometimes make exceptions for this. Our +decision will be guided by the two goals of preserving the free status +of all derivatives of our free software and of promoting the sharing +and reuse of software generally. + + NO WARRANTY + + 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO +WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW. +EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR +OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY +KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE +LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME +THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN +WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY +AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU +FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR +CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE +LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING +RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A +FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF +SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH +DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Libraries + + If you develop a new library, and you want it to be of the greatest +possible use to the public, we recommend making it free software that +everyone can redistribute and change. You can do so by permitting +redistribution under these terms (or, alternatively, under the terms of the +ordinary General Public License). + + To apply these terms, attach the following notices to the library. It is +safest to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least the +"copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Library General Public + License as published by the Free Software Foundation; either + version 2 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Library General Public License for more details. + + You should have received a copy of the GNU Library General Public + License along with this library; if not, see . + +Also add information on how to contact you by electronic and paper mail. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the library, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the + library `Frob' (a library for tweaking knobs) written by James Random Hacker. + + , 1 April 1990 + Moe Ghoul, President of Vice + +That's all there is to it! diff --git a/licenses/README.md b/licenses/README.md index db0c9d05e..c811a4f76 100644 --- a/licenses/README.md +++ b/licenses/README.md @@ -4,19 +4,20 @@ Start with [THIRD_PARTY_NOTICES.md](../THIRD_PARTY_NOTICES.md) for the distincti between distributed material and separately obtained tools. These files retain upstream texts; they do not change [REA's MIT license](../LICENSE). -| Files | Provenance and scope | -| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `npm-dependencies.txt` | Every package path in the root lockfile, with its exact version, declared license, scope, download URL, and integrity. Includes development and optional platform entries. | -| `npm-runtime-notices.txt` | Original license, copyright, and NOTICE texts collected from integrity-verified locked runtime npm archives across all platforms; also the license of Rolldown's generated runtime helpers. The archive manifest's name and version are checked. | -| `nativeaot-parser-MIT.txt` | Original Washi notice for derived NativeAOT parser code; source revision is recorded in the parser and the root notices. | -| `dotnet-runtime-MIT.txt` | Original .NET Foundation MIT text from the NativeAOT reference revision `a3b2d40328be0be3f8dd43f0e5cc925b8827b044`. | -| `ghidra-12.1.4-{LICENSE,NOTICE,DISCLAIMER}.txt` | Unchanged top-level documents from `Ghidra_12.1.4_build`. Ghidra is not bundled; these reference copies do not replace its component and GPL support-program notices. The US-government-specific disclaimer applies to Ghidra, not to REA. | -| `cfworker-MIT.txt`, `xterm-MIT.txt` | Original notices missing from their published runtime npm archives, recovered from the releases' upstream Git revisions. They are included in the corresponding runtime entries. | -| `Apache-2.0.txt`, `MIT-template.txt` | Standard license texts for explicitly documented upstream notice gaps. The MIT placeholders are not an attribution to an inferred author or year. | -| `MinGW-w64-runtime.txt` | Upstream's combined runtime copyright and license notices, snapshot from v12.0.0, for the statically linked Windows addon. This is a notice snapshot, not a pin of the compiler used in release CI. | -| `GPL-3.0.txt`, `GCC-Runtime-Library-Exception-3.1.txt` | Unchanged GPLv3 and GCC runtime exception texts from GCC 14.2.0. The exception permits eligible compiled combinations under its conditions; the runtime keeps its own license. This snapshot does not assert that every release uses GCC 14.2.0. | -| `Node-22.19.0-LICENSE.txt` | The original Node 22.19.0 license and third-party notices. The Windows addon uses Node-API headers from that release; the complete Node runtime and all components listed in this reference document are not bundled. | -| `upstream-sources.json` | Source URLs and SHA-256 digests for the new unchanged reference and supplemental texts. These digests identify the preserved local texts. | +| Files | Provenance and scope | +| ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `npm-dependencies.txt` | Every package path in the root lockfile, with its exact version, declared license, scope, download URL, and integrity. Includes development and optional platform entries. | +| `npm-runtime-notices.txt` | Original license, copyright, and NOTICE texts collected from integrity-verified locked runtime npm archives across all platforms; also the license of Rolldown's generated runtime helpers. The archive manifest's name and version are checked. | +| `nativeaot-parser-MIT.txt` | Original Washi notice for derived NativeAOT parser code; source revision is recorded in the parser and the root notices. | +| `dotnet-runtime-MIT.txt` | Original .NET Foundation MIT text from the NativeAOT reference revision `a3b2d40328be0be3f8dd43f0e5cc925b8827b044`. | +| `ghidra-12.1.4-{LICENSE,NOTICE,DISCLAIMER}.txt` | Unchanged top-level documents from `Ghidra_12.1.4_build`. Ghidra is not bundled; these reference copies do not replace its component and GPL support-program notices. The US-government-specific disclaimer applies to Ghidra, not to REA. | +| `cfworker-MIT.txt`, `xterm-MIT.txt` | Original notices missing from their published runtime npm archives, recovered from the releases' upstream Git revisions. They are included in the corresponding runtime entries. | +| `Apache-2.0.txt`, `MIT-template.txt` | Standard license texts for explicitly documented upstream notice gaps. The MIT placeholders are not an attribution to an inferred author or year. | +| `frida-17.23.2-COPYING.txt`, `LGPL-2.0.txt` | Unchanged licensing supplements for the optional Frida dependency: tagged Frida wxWindows notice and GNU Library GPL v2 text. The npm archive and tagged Node binding omit a standalone license/copyright document; these texts do not recover binding or prebuild-component attribution. | +| `MinGW-w64-runtime.txt` | Upstream's combined runtime copyright and license notices, snapshot from v12.0.0, for the statically linked Windows addon. This is a notice snapshot, not a pin of the compiler used in release CI. | +| `GPL-3.0.txt`, `GCC-Runtime-Library-Exception-3.1.txt` | Unchanged GPLv3 and GCC runtime exception texts from GCC 14.2.0. The exception permits eligible compiled combinations under its conditions; the runtime keeps its own license. This snapshot does not assert that every release uses GCC 14.2.0. | +| `Node-22.19.0-LICENSE.txt` | The original Node 22.19.0 license and third-party notices. The Windows addon uses Node-API headers from that release; the complete Node runtime and all components listed in this reference document are not bundled. | +| `upstream-sources.json` | Source URLs and SHA-256 digests for the new unchanged reference and supplemental texts. These digests identify the preserved local texts. | Additional preserved originals live at [evmole/LICENSE](../third_party/evmole/LICENSE), @@ -40,6 +41,10 @@ only license and author metadata: their missing original copyright documents remain explicitly identified in the runtime texts. Do not describe those entries as complete recovered attribution. +`frida@17.23.2` also lacks an original standalone notice in its npm archive and +tagged Node source. Its declared license and the preserved licensing supplements +are recorded explicitly without inferring a copyright holder or year. + These records do not establish the legal accuracy of upstream metadata or prove that every upstream source-file attribution has been supplied in an archive. Review and refresh them when dependencies change; compare reference @@ -52,3 +57,12 @@ runtime/component notices with the retained snapshots and update them if they differ. Do not assume that a compiler change leaves redistribution requirements unchanged. Preserve applicable originals with any additionally bundled engine; top-level reference copies alone may not cover its entire distribution. + +Generate the exact locked inventory with `npm run licenses:generate`. After +refreshing verified runtime notices, update their header fingerprint to the +generator's reported `runtime_dependency_fingerprint`, then run +`npm run licenses:check`; `docs:check` also runs this check. It compares the +inventory, runtime package identities, +license declarations, archive URLs/integrity, runtime dependency fingerprint, +and preserved reference digests. It does not verify legal completeness or +replace inspection of original upstream notices. diff --git a/licenses/frida-17.23.2-COPYING.txt b/licenses/frida-17.23.2-COPYING.txt new file mode 100644 index 000000000..b01d49c67 --- /dev/null +++ b/licenses/frida-17.23.2-COPYING.txt @@ -0,0 +1,50 @@ + wxWindows Library Licence, Version 3.1 + ====================================== + +Copyright (c) 1998-2005 Julian Smart, Robert Roebling et al + +Everyone is permitted to copy and distribute verbatim copies +of this licence document, but changing it is not allowed. + + WXWINDOWS LIBRARY LICENCE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + +This library is free software; you can redistribute it and/or modify it +under the terms of the GNU Library General Public Licence as published by +the Free Software Foundation; either version 2 of the Licence, or (at your +option) any later version. + +This library is distributed in the hope that it will be useful, but WITHOUT +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU Library General Public +Licence for more details. + +You should have received a copy of the GNU Library General Public Licence +along with this software, usually in a file named COPYING.LIB. If not, +write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth +Floor, Boston, MA 02110-1301 USA. + +EXCEPTION NOTICE + +1. As a special exception, the copyright holders of this library give +permission for additional uses of the text contained in this release of the +library as licenced under the wxWindows Library Licence, applying either +version 3.1 of the Licence, or (at your option) any later version of the +Licence as published by the copyright holders of version 3.1 of the Licence +document. + +2. The exception is that you may use, copy, link, modify and distribute +under your own terms, binary object code versions of works based on the +Library. + +3. If you copy code from files distributed under the terms of the GNU +General Public Licence or the GNU Library General Public Licence into a +copy of this library, as this licence permits, the exception does not apply +to the code that you add in this way. To avoid misleading anyone as to the +status of such modified files, you must delete this exception notice from +such code and/or adjust the licensing conditions notice accordingly. + +4. If you write modifications of your own for this library, it is your +choice whether to permit this exception to apply to your modifications. If +you do not wish that, you must delete the exception notice from such code +and/or adjust the licensing conditions notice accordingly. diff --git a/licenses/npm-dependencies.txt b/licenses/npm-dependencies.txt index 7ae0aa0ad..ce45f7459 100644 --- a/licenses/npm-dependencies.txt +++ b/licenses/npm-dependencies.txt @@ -2,7 +2,7 @@ REA npm dependency inventory Prepared from the root package-lock.json; refresh this inventory when locked dependencies change. License identifiers are upstream lockfile declarations, not a legal assessment. Runtime includes transitive and optional platform packages; development includes build, test and website tooling. -Locked package entries: 561; runtime: 148. +Locked package entries: 586; runtime: 174. Full runtime copyright/license/NOTICE texts: npm-runtime-notices.txt. @algolia/abtesting@1.25.0 | development | MIT @@ -1375,11 +1375,31 @@ base64-js@1.5.1 | runtime | MIT Package: https://www.npmjs.com/package/base64-js/v/1.5.1 Archive: https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz Integrity: sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA== +bindings@1.5.0 | runtime, optional | MIT + Lockfile path: node_modules/bindings + Package: https://www.npmjs.com/package/bindings/v/1.5.0 + Archive: https://registry.npmjs.org/bindings/-/bindings-1.5.0.tgz + Integrity: sha512-p2q/t/mhvuOj/UeLlV6566GD/guowlr0hHxClI0W9m7MWYkL1F0hLo+0Aexs9HSPCtR1SXQ0TD3MMKrXZajbiQ== birpc@2.9.0 | development | MIT Lockfile path: node_modules/birpc Package: https://www.npmjs.com/package/birpc/v/2.9.0 Archive: https://registry.npmjs.org/birpc/-/birpc-2.9.0.tgz Integrity: sha512-KrayHS5pBi69Xi9JmvoqrIgYGDkD6mcSe/i6YKi3w5kekCLzrX4+nawcXqrj2tIp50Kw/mT/s3p+GVK0A0sKxw== +bl@4.1.0 | runtime, optional | MIT + Lockfile path: node_modules/bl + Package: https://www.npmjs.com/package/bl/v/4.1.0 + Archive: https://registry.npmjs.org/bl/-/bl-4.1.0.tgz + Integrity: sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w== +buffer@5.7.1 | runtime, optional | MIT + Lockfile path: node_modules/bl/node_modules/buffer + Package: https://www.npmjs.com/package/buffer/v/5.7.1 + Archive: https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz + Integrity: sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ== +readable-stream@3.6.2 | runtime, optional | MIT + Lockfile path: node_modules/bl/node_modules/readable-stream + Package: https://www.npmjs.com/package/readable-stream/v/3.6.2 + Archive: https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz + Integrity: sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA== brace-expansion@5.0.12 | runtime | MIT Lockfile path: node_modules/brace-expansion Package: https://www.npmjs.com/package/brace-expansion/v/5.0.12 @@ -1430,6 +1450,11 @@ character-entities-legacy@3.0.0 | development | MIT Package: https://www.npmjs.com/package/character-entities-legacy/v/3.0.0 Archive: https://registry.npmjs.org/character-entities-legacy/-/character-entities-legacy-3.0.0.tgz Integrity: sha512-RpPp0asT/6ufRm//AJVwpViZbGM/MkjQFxJccQRHmISF/22NBtsHqAWmL+/pmkPWoIUJdWyeVleTl1wydHATVQ== +chownr@1.1.4 | runtime, optional | ISC + Lockfile path: node_modules/chownr + Package: https://www.npmjs.com/package/chownr/v/1.1.4 + Archive: https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz + Integrity: sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg== clean-git-ref@2.0.1 | runtime | Apache-2.0 Lockfile path: node_modules/clean-git-ref Package: https://www.npmjs.com/package/clean-git-ref/v/2.0.1 @@ -1485,6 +1510,11 @@ decompress-response@6.0.0 | runtime | MIT Package: https://www.npmjs.com/package/decompress-response/v/6.0.0 Archive: https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz Integrity: sha512-aW35yZM6Bb/4oJlZncMH2LCoZtJXTRxES17vE3hoRiowU2kWHaJKFkSBDnDR+cm9J+9QhXmREyIfv0pji9ejCQ== +deep-extend@0.6.0 | runtime, optional | MIT + Lockfile path: node_modules/deep-extend + Package: https://www.npmjs.com/package/deep-extend/v/0.6.0 + Archive: https://registry.npmjs.org/deep-extend/-/deep-extend-0.6.0.tgz + Integrity: sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA== define-data-property@1.1.4 | runtime | MIT Lockfile path: node_modules/define-data-property Package: https://www.npmjs.com/package/define-data-property/v/1.1.4 @@ -1495,7 +1525,7 @@ dequal@2.0.3 | development | MIT Package: https://www.npmjs.com/package/dequal/v/2.0.3 Archive: https://registry.npmjs.org/dequal/-/dequal-2.0.3.tgz Integrity: sha512-0je+qPKHEMohvfRTCEo3CrPG6cAzAYgmzKyxRiYSSDkS6eGJdyVJm7WaYA5ECaAD9wLB2T4EEeymA5aFVcYXCA== -detect-libc@2.1.2 | development | Apache-2.0 +detect-libc@2.1.2 | runtime | Apache-2.0 Lockfile path: node_modules/detect-libc Package: https://www.npmjs.com/package/detect-libc/v/2.1.2 Archive: https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz @@ -1530,6 +1560,11 @@ emoji-regex-xs@1.0.0 | development | MIT Package: https://www.npmjs.com/package/emoji-regex-xs/v/1.0.0 Archive: https://registry.npmjs.org/emoji-regex-xs/-/emoji-regex-xs-1.0.0.tgz Integrity: sha512-LRlerrMYoIDrT6jgpeZ2YYl/L8EulRTt5hQcYjy5AInh7HWXKimpqx68aknBFpGL2+/IcogTcaydJEgaTmOpDg== +end-of-stream@1.4.5 | runtime, optional | MIT + Lockfile path: node_modules/end-of-stream + Package: https://www.npmjs.com/package/end-of-stream/v/1.4.5 + Archive: https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz + Integrity: sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg== entities@7.0.1 | development | BSD-2-Clause Lockfile path: node_modules/entities Package: https://www.npmjs.com/package/entities/v/7.0.1 @@ -1595,6 +1630,11 @@ evmole@0.9.3 | runtime | MIT Package: https://www.npmjs.com/package/evmole/v/0.9.3 Archive: https://registry.npmjs.org/evmole/-/evmole-0.9.3.tgz Integrity: sha512-UhO5HMAoEPCupSxvBJPlm/L0Qk/KVJ/OJo3r4EyhXADIEHqU5aTkAzsBfzk4lXGjqxVnaYbh1GPSmZtmSHFnQQ== +expand-template@2.0.3 | runtime, optional | (MIT OR WTFPL) + Lockfile path: node_modules/expand-template + Package: https://www.npmjs.com/package/expand-template/v/2.0.3 + Archive: https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz + Integrity: sha512-XYfuKMvj4O35f/pOXLObndIRvyQ+/+6AhODh+OKWj9S9498pHHn/IMszH+gt0fBCRWMNfk1ZSp5x3AifmnI2vg== expect-type@1.4.0 | development | Apache-2.0 Lockfile path: node_modules/expect-type Package: https://www.npmjs.com/package/expect-type/v/1.4.0 @@ -1640,6 +1680,11 @@ fdir@6.5.0 | development | MIT Package: https://www.npmjs.com/package/fdir/v/6.5.0 Archive: https://registry.npmjs.org/fdir/-/fdir-6.5.0.tgz Integrity: sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg== +file-uri-to-path@1.0.0 | runtime, optional | MIT + Lockfile path: node_modules/file-uri-to-path + Package: https://www.npmjs.com/package/file-uri-to-path/v/1.0.0 + Archive: https://registry.npmjs.org/file-uri-to-path/-/file-uri-to-path-1.0.0.tgz + Integrity: sha512-0Zt+s3L7Vf1biwWZ29aARiVYLx7iMGnEUl9x33fbB/j3jR81u/O2LbqK+Bm1CDSNDKVtJ/YjwY7TUd5SkeLQLw== focus-trap@7.8.0 | development | MIT Lockfile path: node_modules/focus-trap Package: https://www.npmjs.com/package/focus-trap/v/7.8.0 @@ -1655,6 +1700,16 @@ formatly@0.7.1 | development | MIT Package: https://www.npmjs.com/package/formatly/v/0.7.1 Archive: https://registry.npmjs.org/formatly/-/formatly-0.7.1.tgz Integrity: sha512-ESuCEJE0SGgabl9q+zABI8LhJDmF+8nhpuqDzuM48HfJ9QTD5o6PqFJDFZY7w6f8xsh5lixuj5nJsQNvYOWvKQ== +frida@17.23.2 | runtime, optional | LGPL-2.0 WITH WxWindows-exception-3.1 + Lockfile path: node_modules/frida + Package: https://www.npmjs.com/package/frida/v/17.23.2 + Archive: https://registry.npmjs.org/frida/-/frida-17.23.2.tgz + Integrity: sha512-gRCmmTUFdEILyHRN1b66erRoXafwsfWTqXyoBBqkSCFX0Ue7Jhvs/KhA3JO0XrgITNlPd91QAGsFvWzh0mhVgg== +fs-constants@1.0.0 | runtime, optional | MIT + Lockfile path: node_modules/fs-constants + Package: https://www.npmjs.com/package/fs-constants/v/1.0.0 + Archive: https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz + Integrity: sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow== fsevents@2.3.3 | development, optional | MIT Lockfile path: node_modules/fsevents Package: https://www.npmjs.com/package/fsevents/v/2.3.3 @@ -1685,6 +1740,11 @@ get-tsconfig@4.14.3 | development | MIT Package: https://www.npmjs.com/package/get-tsconfig/v/4.14.3 Archive: https://registry.npmjs.org/get-tsconfig/-/get-tsconfig-4.14.3.tgz Integrity: sha512-++QEw4DIY7WGoukz+/+A/8dGYPT9l9yIadnmSgZ8Rjr3YVSVDipQSO9CdnJo9ePqFqUUqh+wk9uIaoiAwsiPkA== +github-from-package@0.0.0 | runtime, optional | MIT + Lockfile path: node_modules/github-from-package + Package: https://www.npmjs.com/package/github-from-package/v/0.0.0 + Archive: https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz + Integrity: sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw== glob@13.0.6 | runtime | BlueOak-1.0.0 Lockfile path: node_modules/glob Package: https://www.npmjs.com/package/glob/v/13.0.6 @@ -1775,6 +1835,11 @@ inherits@2.0.4 | runtime | ISC Package: https://www.npmjs.com/package/inherits/v/2.0.4 Archive: https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz Integrity: sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ== +ini@1.3.8 | runtime, optional | ISC + Lockfile path: node_modules/ini + Package: https://www.npmjs.com/package/ini/v/1.3.8 + Archive: https://registry.npmjs.org/ini/-/ini-1.3.8.tgz + Integrity: sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew== is-callable@1.2.7 | runtime | MIT Lockfile path: node_modules/is-callable Package: https://www.npmjs.com/package/is-callable/v/1.2.7 @@ -2080,11 +2145,26 @@ mitt@3.0.1 | development | MIT Package: https://www.npmjs.com/package/mitt/v/3.0.1 Archive: https://registry.npmjs.org/mitt/-/mitt-3.0.1.tgz Integrity: sha512-vKivATfr97l2/QBCYAkXYDbrIWPM2IIKEl7YPhjCvKlG3kE2gm+uBo6nEXK3M5/Ffh/FLpKExzOQ3JJoJGFKBw== +mkdirp-classic@0.5.3 | runtime, optional | MIT + Lockfile path: node_modules/mkdirp-classic + Package: https://www.npmjs.com/package/mkdirp-classic/v/0.5.3 + Archive: https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz + Integrity: sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A== nanoid@3.3.20 | development | MIT Lockfile path: node_modules/nanoid Package: https://www.npmjs.com/package/nanoid/v/3.3.20 Archive: https://registry.npmjs.org/nanoid/-/nanoid-3.3.20.tgz Integrity: sha512-uKdg2G3GNCKQn9byYOpxbGqrT2fGO5KRt5J/8b3pok8rT6qxGWF6hxMyJiEYtAf+FVyYuD9hRaDqX5uPFYJ4ZQ== +napi-build-utils@2.0.0 | runtime, optional | MIT + Lockfile path: node_modules/napi-build-utils + Package: https://www.npmjs.com/package/napi-build-utils/v/2.0.0 + Archive: https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-2.0.0.tgz + Integrity: sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA== +node-abi@3.98.0 | runtime, optional | MIT + Lockfile path: node_modules/node-abi + Package: https://www.npmjs.com/package/node-abi/v/3.98.0 + Archive: https://registry.npmjs.org/node-abi/-/node-abi-3.98.0.tgz + Integrity: sha512-Llh8noNw7s15m+rmIrZnZQbe4gLyasL9loT74rC92PkmwWKeEedKvc7o6pjp8B3MhrYT/Cl846NOYZbz1wmddw== obug@2.2.1 | development | MIT Lockfile path: node_modules/obug Package: https://www.npmjs.com/package/obug/v/2.2.1 @@ -2230,6 +2310,11 @@ preact@10.29.8 | development | MIT Package: https://www.npmjs.com/package/preact/v/10.29.8 Archive: https://registry.npmjs.org/preact/-/preact-10.29.8.tgz Integrity: sha512-ej2aVZ+vZ8WO7tvlQWRM9N63A0KzF9q4mWJfDUHgYaIofWY9hu74QdnQrjoPMmZi2/nZ5gN0bJCQF49xQqx09Q== +prebuild-install@7.1.3 | runtime, optional | MIT + Lockfile path: node_modules/prebuild-install + Package: https://www.npmjs.com/package/prebuild-install/v/7.1.3 + Archive: https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz + Integrity: sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug== process@0.11.10 | runtime | MIT Lockfile path: node_modules/process Package: https://www.npmjs.com/package/process/v/0.11.10 @@ -2245,6 +2330,11 @@ property-information@7.2.0 | development | MIT Package: https://www.npmjs.com/package/property-information/v/7.2.0 Archive: https://registry.npmjs.org/property-information/-/property-information-7.2.0.tgz Integrity: sha512-IAtzIB6sUiWaJYrX9smp3V46pBGbBeLFRGdh25kg1334VcBlD8HzhPeNIWQH9zhGmo2itIe25EHt9dQP7G5hmg== +pump@3.0.4 | runtime, optional | MIT + Lockfile path: node_modules/pump + Package: https://www.npmjs.com/package/pump/v/3.0.4 + Archive: https://registry.npmjs.org/pump/-/pump-3.0.4.tgz + Integrity: sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA== pure-rand@8.4.2 | development | MIT Lockfile path: node_modules/pure-rand Package: https://www.npmjs.com/package/pure-rand/v/8.4.2 @@ -2255,6 +2345,16 @@ quick-format-unescaped@4.0.4 | runtime | MIT Package: https://www.npmjs.com/package/quick-format-unescaped/v/4.0.4 Archive: https://registry.npmjs.org/quick-format-unescaped/-/quick-format-unescaped-4.0.4.tgz Integrity: sha512-tYC1Q1hgyRuHgloV/YXs2w15unPVh8qfu/qCTfhTYamaw7fyhumKa2yGpdSo87vY32rIclj+4fWYQXUMs9EHvg== +rc@1.2.8 | runtime, optional | (BSD-2-Clause OR MIT OR Apache-2.0) + Lockfile path: node_modules/rc + Package: https://www.npmjs.com/package/rc/v/1.2.8 + Archive: https://registry.npmjs.org/rc/-/rc-1.2.8.tgz + Integrity: sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw== +strip-json-comments@2.0.1 | runtime, optional | MIT + Lockfile path: node_modules/rc/node_modules/strip-json-comments + Package: https://www.npmjs.com/package/strip-json-comments/v/2.0.1 + Archive: https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz + Integrity: sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ== readable-stream@4.7.0 | runtime | MIT Lockfile path: node_modules/readable-stream Package: https://www.npmjs.com/package/readable-stream/v/4.7.0 @@ -2465,6 +2565,21 @@ tabbable@6.5.0 | development | MIT Package: https://www.npmjs.com/package/tabbable/v/6.5.0 Archive: https://registry.npmjs.org/tabbable/-/tabbable-6.5.0.tgz Integrity: sha512-wieBHXygIm7OyQOu5hQlkk62/WyCFYGlWg7L6/ZCUZwx0o398Zkn4pVmMyfYhfMG8kGrj/Krt8eIk6UKC6VzwA== +tar-fs@2.1.5 | runtime, optional | MIT + Lockfile path: node_modules/tar-fs + Package: https://www.npmjs.com/package/tar-fs/v/2.1.5 + Archive: https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.5.tgz + Integrity: sha512-OboTd8mmMhZDNPV+UjQcK9yKAatXu2aJ+r1w4im1Otd4M4fl2hwvdoXUxIYHFTHWK/3y3FarBP70v3vwmGlOxw== +tar-stream@2.2.0 | runtime, optional | MIT + Lockfile path: node_modules/tar-stream + Package: https://www.npmjs.com/package/tar-stream/v/2.2.0 + Archive: https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz + Integrity: sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ== +readable-stream@3.6.2 | runtime, optional | MIT + Lockfile path: node_modules/tar-stream/node_modules/readable-stream + Package: https://www.npmjs.com/package/readable-stream/v/3.6.2 + Archive: https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz + Integrity: sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA== thread-stream@4.2.0 | runtime | MIT Lockfile path: node_modules/thread-stream Package: https://www.npmjs.com/package/thread-stream/v/4.2.0 @@ -2515,6 +2630,11 @@ tslib@2.8.1 | development, optional | 0BSD Package: https://www.npmjs.com/package/tslib/v/2.8.1 Archive: https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz Integrity: sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w== +tunnel-agent@0.6.0 | runtime, optional | Apache-2.0 + Lockfile path: node_modules/tunnel-agent + Package: https://www.npmjs.com/package/tunnel-agent/v/0.6.0 + Archive: https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz + Integrity: sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w== turbo@2.11.7 | development | MIT Lockfile path: node_modules/turbo Package: https://www.npmjs.com/package/turbo/v/2.11.7 @@ -2570,6 +2690,11 @@ unist-util-visit-parents@6.0.2 | development | MIT Package: https://www.npmjs.com/package/unist-util-visit-parents/v/6.0.2 Archive: https://registry.npmjs.org/unist-util-visit-parents/-/unist-util-visit-parents-6.0.2.tgz Integrity: sha512-goh1s1TBrqSqukSc8wrjwWhL0hiJxgA8m4kFxGlQ+8FYQ3C/m11FcTs4YYem7V664AhHVvgoQLk890Ssdsr2IQ== +util-deprecate@1.0.2 | runtime, optional | MIT + Lockfile path: node_modules/util-deprecate + Package: https://www.npmjs.com/package/util-deprecate/v/1.0.2 + Archive: https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz + Integrity: sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw== vfile@6.0.3 | development | MIT Lockfile path: node_modules/vfile Package: https://www.npmjs.com/package/vfile/v/6.0.3 diff --git a/licenses/npm-runtime-notices.txt b/licenses/npm-runtime-notices.txt index f34cd8ee4..d981aa68a 100644 --- a/licenses/npm-runtime-notices.txt +++ b/licenses/npm-runtime-notices.txt @@ -1,7 +1,7 @@ REA third-party npm runtime copyright, license and NOTICE texts Collected from integrity-verified locked npm archives. Release-specific supplements and upstream notice gaps are identified in their entries. External engines are separately obtained; see THIRD_PARTY_NOTICES.md. -Dependency fingerprint: 0b00d9530d00182e3f2b163bf27feea056c24bb6600806cab74061756ffdc36a +Runtime dependency fingerprint (SHA-256): 5de9cf97e7c327da1892efeff30b40507e4c7f8e496b5d305b8016c162bc15e3 ===== @babel/helper-string-parser@8.0.0 ===== Declared license: MIT @@ -6630,3 +6630,1590 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +===== bindings@1.5.0 ===== +Declared license: MIT +Archive: https://registry.npmjs.org/bindings/-/bindings-1.5.0.tgz +Integrity: sha512-p2q/t/mhvuOj/UeLlV6566GD/guowlr0hHxClI0W9m7MWYkL1F0hLo+0Aexs9HSPCtR1SXQ0TD3MMKrXZajbiQ== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE.md --- + +(The MIT License) + +Copyright (c) 2012 Nathan Rajlich <nathan@tootallnate.net> + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +'Software'), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, +TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE +SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + + +===== bl@4.1.0 ===== +Declared license: MIT +Archive: https://registry.npmjs.org/bl/-/bl-4.1.0.tgz +Integrity: sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE.md --- + +The MIT License (MIT) +===================== + +Copyright (c) 2013-2019 bl contributors +---------------------------------- + +*bl contributors listed at * + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + + +===== buffer@5.7.1 ===== +Declared license: MIT +Archive: https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz +Integrity: sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + +The MIT License (MIT) + +Copyright (c) Feross Aboukhadijeh, and other contributors. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + + +===== chownr@1.1.4 ===== +Declared license: ISC +Archive: https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz +Integrity: sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + +The ISC License + +Copyright (c) Isaac Z. Schlueter and Contributors + +Permission to use, copy, modify, and/or distribute this software for any +purpose with or without fee is hereby granted, provided that the above +copyright notice and this permission notice appear in all copies. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES +WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR +IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + + + +===== deep-extend@0.6.0 ===== +Declared license: MIT +Archive: https://registry.npmjs.org/deep-extend/-/deep-extend-0.6.0.tgz +Integrity: sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + +The MIT License (MIT) + +Copyright (c) 2013-2018, Viacheslav Lotsmanov + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + + +===== detect-libc@2.1.2 ===== +Declared license: Apache-2.0 +Archive: https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz +Integrity: sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright {yyyy} {name of copyright owner} + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + + +===== end-of-stream@1.4.5 ===== +Declared license: MIT +Archive: https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz +Integrity: sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + +The MIT License (MIT) + +Copyright (c) 2014 Mathias Buus + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +===== expand-template@2.0.3 ===== +Declared license: (MIT OR WTFPL) +Archive: https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz +Integrity: sha512-XYfuKMvj4O35f/pOXLObndIRvyQ+/+6AhODh+OKWj9S9498pHHn/IMszH+gt0fBCRWMNfk1ZSp5x3AifmnI2vg== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + +The MIT License (MIT) + +Copyright (c) 2018 Lars-Magnus Skog + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +===== file-uri-to-path@1.0.0 ===== +Declared license: MIT +Archive: https://registry.npmjs.org/file-uri-to-path/-/file-uri-to-path-1.0.0.tgz +Integrity: sha512-0Zt+s3L7Vf1biwWZ29aARiVYLx7iMGnEUl9x33fbB/j3jR81u/O2LbqK+Bm1CDSNDKVtJ/YjwY7TUd5SkeLQLw== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + +Copyright (c) 2014 Nathan Rajlich + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +'Software'), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, +TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE +SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + + +===== frida@17.23.2 ===== +Declared license: LGPL-2.0 WITH WxWindows-exception-3.1 +Archive: https://registry.npmjs.org/frida/-/frida-17.23.2.tgz +Integrity: sha512-gRCmmTUFdEILyHRN1b66erRoXafwsfWTqXyoBBqkSCFX0Ue7Jhvs/KhA3JO0XrgITNlPd91QAGsFvWzh0mhVgg== +Scope: npm runtime dependency, including optional platform packages. + +--- Declared metadata and supplemental licensing texts --- + +The integrity-verified npm archive and tagged frida-node source omit a standalone license/copyright document. The package declares LGPL-2.0 WITH WxWindows-exception-3.1. No Frida copyright holder or year is inferred. +The following unchanged Frida 17.23.2 COPYING and GNU Library GPL v2 texts are licensing supplements. The copyright in COPYING belongs to that license document; it is not recovered attribution for the Node binding or every component in its downloaded prebuild. + +Authors as declared by upstream package metadata: ["Frida Developers"] +Frida Node source: https://github.com/frida/frida-node/tree/11d56293686c5494f538247ccf89f08ab57912ef +License text provenance and SHA-256: licenses/upstream-sources.json + +--- licenses/frida-17.23.2-COPYING.txt --- + + wxWindows Library Licence, Version 3.1 + ====================================== + +Copyright (c) 1998-2005 Julian Smart, Robert Roebling et al + +Everyone is permitted to copy and distribute verbatim copies +of this licence document, but changing it is not allowed. + + WXWINDOWS LIBRARY LICENCE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + +This library is free software; you can redistribute it and/or modify it +under the terms of the GNU Library General Public Licence as published by +the Free Software Foundation; either version 2 of the Licence, or (at your +option) any later version. + +This library is distributed in the hope that it will be useful, but WITHOUT +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU Library General Public +Licence for more details. + +You should have received a copy of the GNU Library General Public Licence +along with this software, usually in a file named COPYING.LIB. If not, +write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth +Floor, Boston, MA 02110-1301 USA. + +EXCEPTION NOTICE + +1. As a special exception, the copyright holders of this library give +permission for additional uses of the text contained in this release of the +library as licenced under the wxWindows Library Licence, applying either +version 3.1 of the Licence, or (at your option) any later version of the +Licence as published by the copyright holders of version 3.1 of the Licence +document. + +2. The exception is that you may use, copy, link, modify and distribute +under your own terms, binary object code versions of works based on the +Library. + +3. If you copy code from files distributed under the terms of the GNU +General Public Licence or the GNU Library General Public Licence into a +copy of this library, as this licence permits, the exception does not apply +to the code that you add in this way. To avoid misleading anyone as to the +status of such modified files, you must delete this exception notice from +such code and/or adjust the licensing conditions notice accordingly. + +4. If you write modifications of your own for this library, it is your +choice whether to permit this exception to apply to your modifications. If +you do not wish that, you must delete the exception notice from such code +and/or adjust the licensing conditions notice accordingly. + + +--- licenses/LGPL-2.0.txt --- + + GNU LIBRARY GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1991 Free Software Foundation, Inc. + + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + +[This is the first released version of the library GPL. It is + numbered 2 because it goes with version 2 of the ordinary GPL.] + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +Licenses are intended to guarantee your freedom to share and change +free software--to make sure the software is free for all its users. + + This license, the Library General Public License, applies to some +specially designated Free Software Foundation software, and to any +other libraries whose authors decide to use it. You can use it for +your libraries, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if +you distribute copies of the library, or if you modify it. + + For example, if you distribute copies of the library, whether gratis +or for a fee, you must give the recipients all the rights that we gave +you. You must make sure that they, too, receive or can get the source +code. If you link a program with the library, you must provide +complete object files to the recipients so that they can relink them +with the library, after making changes to the library and recompiling +it. And you must show them these terms so they know their rights. + + Our method of protecting your rights has two steps: (1) copyright +the library, and (2) offer you this license which gives you legal +permission to copy, distribute and/or modify the library. + + Also, for each distributor's protection, we want to make certain +that everyone understands that there is no warranty for this free +library. If the library is modified by someone else and passed on, we +want its recipients to know that what they have is not the original +version, so that any problems introduced by others will not reflect on +the original authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that companies distributing free +software will individually obtain patent licenses, thus in effect +transforming the program into proprietary software. To prevent this, +we have made it clear that any patent must be licensed for everyone's +free use or not licensed at all. + + Most GNU software, including some libraries, is covered by the ordinary +GNU General Public License, which was designed for utility programs. This +license, the GNU Library General Public License, applies to certain +designated libraries. This license is quite different from the ordinary +one; be sure to read it in full, and don't assume that anything in it is +the same as in the ordinary license. + + The reason we have a separate public license for some libraries is that +they blur the distinction we usually make between modifying or adding to a +program and simply using it. Linking a program with a library, without +changing the library, is in some sense simply using the library, and is +analogous to running a utility program or application program. However, in +a textual and legal sense, the linked executable is a combined work, a +derivative of the original library, and the ordinary General Public License +treats it as such. + + Because of this blurred distinction, using the ordinary General +Public License for libraries did not effectively promote software +sharing, because most developers did not use the libraries. We +concluded that weaker conditions might promote sharing better. + + However, unrestricted linking of non-free programs would deprive the +users of those programs of all benefit from the free status of the +libraries themselves. This Library General Public License is intended to +permit developers of non-free programs to use free libraries, while +preserving your freedom as a user of such programs to change the free +libraries that are incorporated in them. (We have not seen how to achieve +this as regards changes in header files, but we have achieved it as regards +changes in the actual functions of the Library.) The hope is that this +will lead to faster development of free libraries. + + The precise terms and conditions for copying, distribution and +modification follow. Pay close attention to the difference between a +"work based on the library" and a "work that uses the library". The +former contains code derived from the library, while the latter only +works together with the library. + + Note that it is possible for a library to be covered by the ordinary +General Public License rather than by this special one. + + GNU LIBRARY GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License Agreement applies to any software library which +contains a notice placed by the copyright holder or other authorized +party saying it may be distributed under the terms of this Library +General Public License (also called "this License"). Each licensee is +addressed as "you". + + A "library" means a collection of software functions and/or data +prepared so as to be conveniently linked with application programs +(which use some of those functions and data) to form executables. + + The "Library", below, refers to any such software library or work +which has been distributed under these terms. A "work based on the +Library" means either the Library or any derivative work under +copyright law: that is to say, a work containing the Library or a +portion of it, either verbatim or with modifications and/or translated +straightforwardly into another language. (Hereinafter, translation is +included without limitation in the term "modification".) + + "Source code" for a work means the preferred form of the work for +making modifications to it. For a library, complete source code means +all the source code for all modules it contains, plus any associated +interface definition files, plus the scripts used to control compilation +and installation of the library. + + Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running a program using the Library is not restricted, and output from +such a program is covered only if its contents constitute a work based +on the Library (independent of the use of the Library in a tool for +writing it). Whether that is true depends on what the Library does +and what the program that uses the Library does. + + 1. You may copy and distribute verbatim copies of the Library's +complete source code as you receive it, in any medium, provided that +you conspicuously and appropriately publish on each copy an +appropriate copyright notice and disclaimer of warranty; keep intact +all the notices that refer to this License and to the absence of any +warranty; and distribute a copy of this License along with the +Library. + + You may charge a fee for the physical act of transferring a copy, +and you may at your option offer warranty protection in exchange for a +fee. + + 2. You may modify your copy or copies of the Library or any portion +of it, thus forming a work based on the Library, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) The modified work must itself be a software library. + + b) You must cause the files modified to carry prominent notices + stating that you changed the files and the date of any change. + + c) You must cause the whole of the work to be licensed at no + charge to all third parties under the terms of this License. + + d) If a facility in the modified Library refers to a function or a + table of data to be supplied by an application program that uses + the facility, other than as an argument passed when the facility + is invoked, then you must make a good faith effort to ensure that, + in the event an application does not supply such function or + table, the facility still operates, and performs whatever part of + its purpose remains meaningful. + + (For example, a function in a library to compute square roots has + a purpose that is entirely well-defined independent of the + application. Therefore, Subsection 2d requires that any + application-supplied function or table used by this function must + be optional: if the application does not supply it, the square + root function must still compute square roots.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Library, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Library, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote +it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Library. + +In addition, mere aggregation of another work not based on the Library +with the Library (or with a work based on the Library) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may opt to apply the terms of the ordinary GNU General Public +License instead of this License to a given copy of the Library. To do +this, you must alter all the notices that refer to this License, so +that they refer to the ordinary GNU General Public License, version 2, +instead of to this License. (If a newer version than version 2 of the +ordinary GNU General Public License has appeared, then you can specify +that version instead if you wish.) Do not make any other change in +these notices. + + Once this change is made in a given copy, it is irreversible for +that copy, so the ordinary GNU General Public License applies to all +subsequent copies and derivative works made from that copy. + + This option is useful when you wish to copy part of the code of +the Library into a program that is not a library. + + 4. You may copy and distribute the Library (or a portion or +derivative of it, under Section 2) in object code or executable form +under the terms of Sections 1 and 2 above provided that you accompany +it with the complete corresponding machine-readable source code, which +must be distributed under the terms of Sections 1 and 2 above on a +medium customarily used for software interchange. + + If distribution of object code is made by offering access to copy +from a designated place, then offering equivalent access to copy the +source code from the same place satisfies the requirement to +distribute the source code, even though third parties are not +compelled to copy the source along with the object code. + + 5. A program that contains no derivative of any portion of the +Library, but is designed to work with the Library by being compiled or +linked with it, is called a "work that uses the Library". Such a +work, in isolation, is not a derivative work of the Library, and +therefore falls outside the scope of this License. + + However, linking a "work that uses the Library" with the Library +creates an executable that is a derivative of the Library (because it +contains portions of the Library), rather than a "work that uses the +library". The executable is therefore covered by this License. +Section 6 states terms for distribution of such executables. + + When a "work that uses the Library" uses material from a header file +that is part of the Library, the object code for the work may be a +derivative work of the Library even though the source code is not. +Whether this is true is especially significant if the work can be +linked without the Library, or if the work is itself a library. The +threshold for this to be true is not precisely defined by law. + + If such an object file uses only numerical parameters, data +structure layouts and accessors, and small macros and small inline +functions (ten lines or less in length), then the use of the object +file is unrestricted, regardless of whether it is legally a derivative +work. (Executables containing this object code plus portions of the +Library will still fall under Section 6.) + + Otherwise, if the work is a derivative of the Library, you may +distribute the object code for the work under the terms of Section 6. +Any executables containing that work also fall under Section 6, +whether or not they are linked directly with the Library itself. + + 6. As an exception to the Sections above, you may also compile or +link a "work that uses the Library" with the Library to produce a +work containing portions of the Library, and distribute that work +under terms of your choice, provided that the terms permit +modification of the work for the customer's own use and reverse +engineering for debugging such modifications. + + You must give prominent notice with each copy of the work that the +Library is used in it and that the Library and its use are covered by +this License. You must supply a copy of this License. If the work +during execution displays copyright notices, you must include the +copyright notice for the Library among them, as well as a reference +directing the user to the copy of this License. Also, you must do one +of these things: + + a) Accompany the work with the complete corresponding + machine-readable source code for the Library including whatever + changes were used in the work (which must be distributed under + Sections 1 and 2 above); and, if the work is an executable linked + with the Library, with the complete machine-readable "work that + uses the Library", as object code and/or source code, so that the + user can modify the Library and then relink to produce a modified + executable containing the modified Library. (It is understood + that the user who changes the contents of definitions files in the + Library will not necessarily be able to recompile the application + to use the modified definitions.) + + b) Accompany the work with a written offer, valid for at + least three years, to give the same user the materials + specified in Subsection 6a, above, for a charge no more + than the cost of performing this distribution. + + c) If distribution of the work is made by offering access to copy + from a designated place, offer equivalent access to copy the above + specified materials from the same place. + + d) Verify that the user has already received a copy of these + materials or that you have already sent this user a copy. + + For an executable, the required form of the "work that uses the +Library" must include any data and utility programs needed for +reproducing the executable from it. However, as a special exception, +the source code distributed need not include anything that is normally +distributed (in either source or binary form) with the major +components (compiler, kernel, and so on) of the operating system on +which the executable runs, unless that component itself accompanies +the executable. + + It may happen that this requirement contradicts the license +restrictions of other proprietary libraries that do not normally +accompany the operating system. Such a contradiction means you cannot +use both them and the Library together in an executable that you +distribute. + + 7. You may place library facilities that are a work based on the +Library side-by-side in a single library together with other library +facilities not covered by this License, and distribute such a combined +library, provided that the separate distribution of the work based on +the Library and of the other library facilities is otherwise +permitted, and provided that you do these two things: + + a) Accompany the combined library with a copy of the same work + based on the Library, uncombined with any other library + facilities. This must be distributed under the terms of the + Sections above. + + b) Give prominent notice with the combined library of the fact + that part of it is a work based on the Library, and explaining + where to find the accompanying uncombined form of the same work. + + 8. You may not copy, modify, sublicense, link with, or distribute +the Library except as expressly provided under this License. Any +attempt otherwise to copy, modify, sublicense, link with, or +distribute the Library is void, and will automatically terminate your +rights under this License. However, parties who have received copies, +or rights, from you under this License will not have their licenses +terminated so long as such parties remain in full compliance. + + 9. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Library or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Library (or any work based on the +Library), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Library or works based on it. + + 10. Each time you redistribute the Library (or any work based on the +Library), the recipient automatically receives a license from the +original licensor to copy, distribute, link with or modify the Library +subject to these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 11. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Library at all. For example, if a patent +license would not permit royalty-free redistribution of the Library by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Library. + +If any portion of this section is held invalid or unenforceable under any +particular circumstance, the balance of the section is intended to apply, +and the section as a whole is intended to apply in other circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 12. If the distribution and/or use of the Library is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Library under this License may add +an explicit geographical distribution limitation excluding those countries, +so that distribution is permitted only in or among countries not thus +excluded. In such case, this License incorporates the limitation as if +written in the body of this License. + + 13. The Free Software Foundation may publish revised and/or new +versions of the Library General Public License from time to time. +Such new versions will be similar in spirit to the present version, +but may differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Library +specifies a version number of this License which applies to it and +"any later version", you have the option of following the terms and +conditions either of that version or of any later version published by +the Free Software Foundation. If the Library does not specify a +license version number, you may choose any version ever published by +the Free Software Foundation. + + 14. If you wish to incorporate parts of the Library into other free +programs whose distribution conditions are incompatible with these, +write to the author to ask for permission. For software which is +copyrighted by the Free Software Foundation, write to the Free +Software Foundation; we sometimes make exceptions for this. Our +decision will be guided by the two goals of preserving the free status +of all derivatives of our free software and of promoting the sharing +and reuse of software generally. + + NO WARRANTY + + 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO +WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW. +EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR +OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY +KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE +LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME +THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN +WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY +AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU +FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR +CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE +LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING +RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A +FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF +SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH +DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Libraries + + If you develop a new library, and you want it to be of the greatest +possible use to the public, we recommend making it free software that +everyone can redistribute and change. You can do so by permitting +redistribution under these terms (or, alternatively, under the terms of the +ordinary General Public License). + + To apply these terms, attach the following notices to the library. It is +safest to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least the +"copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Library General Public + License as published by the Free Software Foundation; either + version 2 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Library General Public License for more details. + + You should have received a copy of the GNU Library General Public + License along with this library; if not, see . + +Also add information on how to contact you by electronic and paper mail. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the library, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the + library `Frob' (a library for tweaking knobs) written by James Random Hacker. + + , 1 April 1990 + Moe Ghoul, President of Vice + +That's all there is to it! + + + +===== fs-constants@1.0.0 ===== +Declared license: MIT +Archive: https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz +Integrity: sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + +The MIT License (MIT) + +Copyright (c) 2018 Mathias Buus + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + + +===== github-from-package@0.0.0 ===== +Declared license: MIT +Archive: https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz +Integrity: sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + +This software is released under the MIT license: + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + + +===== ini@1.3.8 ===== +Declared license: ISC +Archive: https://registry.npmjs.org/ini/-/ini-1.3.8.tgz +Integrity: sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + +The ISC License + +Copyright (c) Isaac Z. Schlueter and Contributors + +Permission to use, copy, modify, and/or distribute this software for any +purpose with or without fee is hereby granted, provided that the above +copyright notice and this permission notice appear in all copies. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES +WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR +IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + + + +===== mkdirp-classic@0.5.3 ===== +Declared license: MIT +Archive: https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz +Integrity: sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + +The MIT License (MIT) + +Copyright (c) 2020 James Halliday (mail@substack.net) and Mathias Buus + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + + +===== napi-build-utils@2.0.0 ===== +Declared license: MIT +Archive: https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-2.0.0.tgz +Integrity: sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + +MIT License + +Copyright (c) 2018 inspiredware + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + + +===== node-abi@3.98.0 ===== +Declared license: MIT +Archive: https://registry.npmjs.org/node-abi/-/node-abi-3.98.0.tgz +Integrity: sha512-Llh8noNw7s15m+rmIrZnZQbe4gLyasL9loT74rC92PkmwWKeEedKvc7o6pjp8B3MhrYT/Cl846NOYZbz1wmddw== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + +MIT License + +Copyright (c) 2016 Lukas Geiger + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + + + +===== prebuild-install@7.1.3 ===== +Declared license: MIT +Archive: https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz +Integrity: sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + +The MIT License (MIT) + +Copyright (c) 2015 Mathias Buus + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + + +===== pump@3.0.4 ===== +Declared license: MIT +Archive: https://registry.npmjs.org/pump/-/pump-3.0.4.tgz +Integrity: sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + +The MIT License (MIT) + +Copyright (c) 2014 Mathias Buus + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +===== rc@1.2.8 ===== +Declared license: (BSD-2-Clause OR MIT OR Apache-2.0) +Archive: https://registry.npmjs.org/rc/-/rc-1.2.8.tgz +Integrity: sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE.APACHE2 --- + +Apache License, Version 2.0 + +Copyright (c) 2011 Dominic Tarr + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + + +--- package/LICENSE.BSD --- + +Copyright (c) 2013, Dominic Tarr +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +The views and conclusions contained in the software and documentation are those +of the authors and should not be interpreted as representing official policies, +either expressed or implied, of the FreeBSD Project. + + +--- package/LICENSE.MIT --- + +The MIT License + +Copyright (c) 2011 Dominic Tarr + +Permission is hereby granted, free of charge, +to any person obtaining a copy of this software and +associated documentation files (the "Software"), to +deal in the Software without restriction, including +without limitation the rights to use, copy, modify, +merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom +the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice +shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR +ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, +TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE +SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + + +===== readable-stream@3.6.2 ===== +Declared license: MIT +Archive: https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz +Integrity: sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + +Node.js is licensed for use as follows: + +""" +Copyright Node.js contributors. All rights reserved. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to +deal in the Software without restriction, including without limitation the +rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +sell copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. +""" + +This license applies to parts of Node.js originating from the +https://github.com/joyent/node repository: + +""" +Copyright Joyent, Inc. and other Node contributors. All rights reserved. +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to +deal in the Software without restriction, including without limitation the +rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +sell copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. +""" + + + +===== strip-json-comments@2.0.1 ===== +Declared license: MIT +Archive: https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz +Integrity: sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ== +Scope: npm runtime dependency, including optional platform packages. + +--- package/license --- + +The MIT License (MIT) + +Copyright (c) Sindre Sorhus (sindresorhus.com) + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + + +===== tar-fs@2.1.5 ===== +Declared license: MIT +Archive: https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.5.tgz +Integrity: sha512-OboTd8mmMhZDNPV+UjQcK9yKAatXu2aJ+r1w4im1Otd4M4fl2hwvdoXUxIYHFTHWK/3y3FarBP70v3vwmGlOxw== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + +The MIT License (MIT) + +Copyright (c) 2014 Mathias Buus + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +===== tar-stream@2.2.0 ===== +Declared license: MIT +Archive: https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz +Integrity: sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + +The MIT License (MIT) + +Copyright (c) 2014 Mathias Buus + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +===== tunnel-agent@0.6.0 ===== +Declared license: Apache-2.0 +Archive: https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz +Integrity: sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + +Apache License + +Version 2.0, January 2004 + +http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + +"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. + +"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. + +"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. + +"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. + +"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. + +"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. + +"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). + +"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. + +"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." + +"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. + +2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. + +3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. + +4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: + +You must give any other recipients of the Work or Derivative Works a copy of this License; and + +You must cause any modified files to carry prominent notices stating that You changed the files; and + +You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and + +If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. + +5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. + +6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. + +8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. + +9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. + +END OF TERMS AND CONDITIONS + + +===== util-deprecate@1.0.2 ===== +Declared license: MIT +Archive: https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz +Integrity: sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw== +Scope: npm runtime dependency, including optional platform packages. + +--- package/LICENSE --- + +(The MIT License) + +Copyright (c) 2014 Nathan Rajlich + +Permission is hereby granted, free of charge, to any person +obtaining a copy of this software and associated documentation +files (the "Software"), to deal in the Software without +restriction, including without limitation the rights to use, +copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the +Software is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +OTHER DEALINGS IN THE SOFTWARE. + + diff --git a/licenses/upstream-sources.json b/licenses/upstream-sources.json index 5853460b2..fb1dad975 100644 --- a/licenses/upstream-sources.json +++ b/licenses/upstream-sources.json @@ -58,5 +58,15 @@ "file": "licenses/Node-22.19.0-LICENSE.txt", "source": "https://raw.githubusercontent.com/nodejs/node/v22.19.0/LICENSE", "sha256": "e991d81497a85bb24fc6bffae0a3637a6accd6c6bc5ce1f2c5698bd555cf9d49" + }, + { + "file": "licenses/frida-17.23.2-COPYING.txt", + "source": "https://raw.githubusercontent.com/frida/frida/9adf8ed33ab6f1122b17399ee96dcbe101b941c9/COPYING", + "sha256": "5ea1544b51a28bc823b03159190d4108f9fb4f4ef912389f5137c6d295e175b2" + }, + { + "file": "licenses/LGPL-2.0.txt", + "source": "https://www.gnu.org/licenses/old-licenses/lgpl-2.0.txt", + "sha256": "cc535c21133c895b56b374c8a1dc1eb948d99003ed2b47372069456b62f42b24" } ] diff --git a/native/windows/README.md b/native/windows/README.md index 82dd5526b..e2543260f 100644 --- a/native/windows/README.md +++ b/native/windows/README.md @@ -69,10 +69,22 @@ without this lane reports Windows native controls as unavailable. limits only for its JavaScript helpers when running on Bun, which ignores their V8 heap flags; Node.js launches are unchanged. - Only the suspended child's default file owner is normalized. Caller token - privileges and ownership remain unchanged. Caller CPU affinity and process - priority are preserved in the child. Environment names are sorted and - deduplicated using Windows ordinal comparison, preserving their native Unicode - semantics and caller-selected values. + privileges and ownership remain unchanged. Caller process priority is + preserved in the child, and caller CPU affinity whenever both occupy the same + processor group. Windows chooses a new process's group itself and rejects a + mask belonging to another group, so a child placed elsewhere keeps that + group's default when the caller asked for no bound of its own. A caller + deliberately pinned to fewer processors than its group holds is a constraint + such a child cannot express, and is refused rather than widened to that + group's default. A mask covering its group's whole active set does not + distinguish a caller Windows placed in that group from one that selected the + group deliberately, so such a child can still be placed elsewhere: its + processor count stays bounded, its group placement is not preserved, and + neither that case nor the refusal above has a verification lane, because both + need a host whose processors span more than one group. + Environment names are sorted and deduplicated using Windows + ordinal comparison, preserving their native Unicode semantics and + caller-selected values. - Cleanup walks locked directory objects, removes files by handle, and unlinks reparse entries without traversing their targets. Failed removal retains the root and ancestor leases for retry; a partially removed runtime accepts only @@ -85,6 +97,17 @@ Windows failures retain their constraint, requested coordinate, Win32 code, and system message. Unsupported filesystems and reparse paths are distinct from OS access denial and missing packaged controls. +A failed runtime-parent admission additionally reports +`runtimeAllocation: "not-started"`: no runtime directory was created. The +Windows adapter uses that fact with numeric sharing/lock codes 32/33 to retain +a retryable private-runtime failure, including after successful native +authority caching. Missing allocation evidence, other codes, and opaque error +text do not establish a safe retry. Analysis errors propagate the producer's +retry disposition through JavaScript, source-map and WABT workflows; transient +failures report `category: unavailable` and retain the original reason. +Incomplete cleanup blocks an operation retry until its owned resources settle. +REA adds no automatic retry loop or new analysis cache for this behavior. + Ordinary drive-absolute paths accept backslashes, forward slashes, or mixed separators. File identity retains the original requested spelling. Separator translation does not resolve dot components, repeated separators, or extend diff --git a/native/windows/src/addon.cc b/native/windows/src/addon.cc index 0e39b84c0..bffd4cbd4 100644 --- a/native/windows/src/addon.cc +++ b/native/windows/src/addon.cc @@ -87,6 +87,8 @@ napi_value failureValue(napi_env env, const Failure& failure) { set(env, error, "requestedPath", string(env, failure.path)); set(env, error, "win32Code", number(env, failure.win32)); set(env, error, "win32Message", string(env, detail)); + if (failure.runtimeDirectoryUnallocated) + set(env, error, "runtimeAllocation", string(env, std::string("not-started"))); return error; } diff --git a/native/windows/src/authority.hpp b/native/windows/src/authority.hpp index cc713ce0b..13d7e4d60 100644 --- a/native/windows/src/authority.hpp +++ b/native/windows/src/authority.hpp @@ -33,6 +33,8 @@ struct Failure : std::runtime_error { DWORD win32; std::wstring path; std::string constraint; + // Affirmative proof for a failed runtime-parent admission; omission is unknown. + bool runtimeDirectoryUnallocated = false; Failure(std::string constraint, std::wstring path = L"", DWORD code = GetLastError()) : std::runtime_error(constraint), win32(code), path(std::move(path)), constraint(std::move(constraint)) {} diff --git a/native/windows/src/filesystem.cc b/native/windows/src/filesystem.cc index 6cfbc8e97..c8f66e5c1 100644 --- a/native/windows/src/filesystem.cc +++ b/native/windows/src/filesystem.cc @@ -212,7 +212,12 @@ static void verifyPrivate(HANDLE handle, const std::wstring& path, bool root) { } std::unique_ptr createRuntime(const std::wstring& parentPath, const std::wstring& prefix) { - auto parent = openFile(parentPath, 0, true); + std::unique_ptr parent; + try { parent = openFile(parentPath, 0, true); } + catch (Failure& failure) { + failure.runtimeDirectoryUnallocated = true; + throw; + } require(!prefix.empty() && prefix.find_first_of(L"\\/:\"<>|?*") == std::wstring::npos, "Runtime prefix must be one filename component", parentPath, ERROR_INVALID_PARAMETER); std::array random; diff --git a/native/windows/src/process.cc b/native/windows/src/process.cc index ede2b5622..80a4d0ac5 100644 --- a/native/windows/src/process.cc +++ b/native/windows/src/process.cc @@ -68,6 +68,18 @@ static ULONGLONG jobMemoryLimit(napi_env env, napi_value value) { return limit; } +// An affinity mask is meaningful only inside one processor group, so only a +// process occupying exactly one group has a group a child can inherit. A +// process whose threads span groups, or whose membership cannot be read, has no +// single group. +static bool singleProcessorGroup(HANDLE process, USHORT& group) { + USHORT count = 1; + std::array groups{}; + if (!GetProcessGroupAffinity(process, &count, groups.data()) || count != 1) return false; + group = groups[0]; + return true; +} + static void pipe(Handle& read, Handle& write, bool parentReads = true) { SECURITY_ATTRIBUTES attributes{sizeof(SECURITY_ATTRIBUTES), nullptr, TRUE}; HANDLE rawRead, rawWrite; @@ -168,11 +180,27 @@ std::unique_ptr spawnProcess(const std::wstring& command, const std::ws TOKEN_OWNER owner{reinterpret_cast(tokenUser.data())->User.Sid}; require(SetTokenInformation(token.get(), TokenOwner, &owner, sizeof(owner)), "Set suspended process file owner failed", command); + // A mask names processors inside one processor group, and Windows assigns a + // new process to a group of its own choosing, so a mask belonging to another + // group is rejected (ERROR_INVALID_PARAMETER). An unconstrained caller, whose + // mask is its group's whole active set, asks for no bound the child's own + // group cannot give, so such a child keeps that group's default and the spawn + // proceeds. A narrower mask is a deliberate CPU constraint that a child in + // another group cannot express, and silently widening it would hand the + // provider tree the whole group, so that spawn fails naming the constraint. DWORD_PTR affinity = 0, systemAffinity = 0; require(GetProcessAffinityMask(GetCurrentProcess(), &affinity, &systemAffinity), "Read caller process affinity failed", command); - if (affinity != 0) + USHORT callerGroup = 0, childGroup = 0; + const bool sharedGroup = singleProcessorGroup(GetCurrentProcess(), callerGroup) && + singleProcessorGroup(result->process.get(), childGroup) && + callerGroup == childGroup; + if (affinity != 0 && sharedGroup) require(SetProcessAffinityMask(result->process.get(), affinity), "Preserve caller process affinity failed", command); + else + require(affinity == 0 || affinity == systemAffinity, + "Caller CPU affinity cannot bound an owned process in another processor group", command, + ERROR_INVALID_PARAMETER); const DWORD priority = GetPriorityClass(GetCurrentProcess()); require(priority != 0 && SetPriorityClass(result->process.get(), priority), "Preserve caller process priority failed", command); diff --git a/package.json b/package.json index f803cc00c..7ee2b42ab 100644 --- a/package.json +++ b/package.json @@ -167,12 +167,14 @@ "verify:legacy-simulator": "npm run build:cached && node scripts/verify/apple/legacy-simulator.mjs", "verify:architecture-guards": "node scripts/verify/architecture-guards.mjs", "verify:apktool": "npm run build:cached && node scripts/verify-apktool.mjs", + "verify:pcap": "npm run build:cached && node scripts/verify-pcap.mjs", "verify:flutter": "npm run build:cached && node scripts/verify-flutter.mjs", "verify:adb": "npm run build:cached && node scripts/verify-adb.mjs", "verify:ghidra": "npm run build:cached && node scripts/verify-real-ghidra.mjs", "verify:ghidra:seeds": "npm run build:cached && node scripts/verify/ghidra/seeds.mjs", "verify:ghidra:mips": "npm run build:cached && node scripts/verify-real-ghidra-mips.mjs", "verify:ghidra:riscv": "npm run build:cached && node scripts/verify-real-ghidra-riscv.mjs", + "verify:ghidra:6502": "npm run build:cached && node scripts/verify-real-ghidra-6502.mjs", "verify:ghidra:mips-shared": "npm run build:cached && node scripts/verify-real-ghidra-mips-shared.mjs", "verify:ghidra:mips-relocatable": "npm run build:cached && node scripts/verify-real-ghidra-mips-relocatable.mjs", "verify:ghidra:swift": "npm run build:cached && node scripts/verify-real-ghidra-swift.mjs", @@ -181,6 +183,7 @@ "verify:ghidra:dos": "npm run build:cached && node scripts/verify-real-ghidra-dos.mjs", "verify:ghidra:com": "npm run build:cached && node scripts/verify-real-ghidra-com.mjs", "verify:ghidra:raw": "npm run build:cached && node scripts/verify-real-ghidra-raw.mjs", + "verify:ghidra:address-context": "npm run build:cached && node scripts/verify-real-ghidra-address-context.mjs", "verify:ghidra:aarch64-jump-table": "npm run build:cached && node scripts/verify-real-ghidra.mjs --aarch64-jump-table", "verify:interface-builder": "npm run build:cached && node scripts/verify/apple/interface-builder.mjs", "verify:ghidra:cross-format": "npm run build:cached && node scripts/verify-real-ghidra.mjs --cross-format", @@ -206,7 +209,7 @@ "fixtures:harmony": "node scripts/fixtures/harmony/fetch.mjs", "verify:agent": "npm run build:cached && node scripts/verify-agent-experience.mjs", "verify:hopper:linux": "npm run verify:fixtures && npm run build:cached && node scripts/verify-real-hopper-linux.mjs", - "verify:setup:linux": "npm run build:cached && node scripts/verify-linux-setup-e2e.mjs", + "verify:setup:linux": "npm run verify:fixtures && npm run build:cached && node scripts/verify-linux-setup-e2e.mjs", "build:fixtures": "node scripts/build-conformance-fixtures.mjs", "build:ghidra-windows-fixture": "node scripts/create-ghidra-windows-fixture.mjs", "verify:fixtures": "npm run build:fixtures && node scripts/verify-conformance-fixtures.mjs", @@ -236,7 +239,7 @@ "docs:preview": "vitepress preview docs", "docs:generate": "node scripts/check-dependency-install.mjs && node scripts/run-exclusive.mjs artifacts npm run docs:prepare", "docs:check": "node scripts/check-dependency-install.mjs && node scripts/run-exclusive.mjs artifacts turbo run docs:check:built && npm run test:local -- tests/conformance/readmeConsistency.test.ts", - "docs:check:built": "node scripts/generate-package-metadata.mjs --check && node scripts/generate-skill-metadata.mjs --check && node scripts/generate-product-catalog.mjs --check && node scripts/generate-completion-ledger.mjs --check && node scripts/check-doc-facts.mjs && node scripts/generate-error-schema.mjs --check", + "docs:check:built": "node scripts/generate-package-metadata.mjs --check && node scripts/generate-skill-metadata.mjs --check && node scripts/generate-product-catalog.mjs --check && node scripts/generate-completion-ledger.mjs --check && node scripts/check-doc-facts.mjs && node scripts/generate-error-schema.mjs --check && npm run licenses:check", "docs:error-schema": "node scripts/check-dependency-install.mjs && node scripts/run-exclusive.mjs artifacts turbo run artifacts:error-schema", "prepack": "node scripts/prepack.mjs", "prepare": "node scripts/prepare.mjs", @@ -278,7 +281,10 @@ "artifacts:error-schema": "node scripts/generate-error-schema.mjs", "test:prepare": "node scripts/check-dependency-install.mjs && node scripts/run-exclusive.mjs artifacts turbo run artifacts:mcp-catalog artifacts:product-catalog artifacts:managed-evidence", "evidence:check:built": "node scripts/generate-completion-ledger.mjs --check", - "docs:prepare": "node scripts/generate-package-metadata.mjs && turbo run artifacts:product-catalog artifacts:managed-evidence artifacts:error-schema" + "docs:prepare": "node scripts/generate-package-metadata.mjs && turbo run artifacts:product-catalog artifacts:managed-evidence artifacts:error-schema", + "verify:web:qlog": "npm run build:cached && node scripts/verify-qlog-captures.mjs", + "licenses:generate": "node scripts/generate-third-party-inventory.mjs", + "licenses:check": "node scripts/generate-third-party-inventory.mjs --check" }, "lint-staged": { "*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}": [ diff --git a/scripts/ci/baseline.mjs b/scripts/ci/baseline.mjs new file mode 100644 index 000000000..171bd7dd3 --- /dev/null +++ b/scripts/ci/baseline.mjs @@ -0,0 +1,76 @@ +import { execFileSync } from "node:child_process"; +import { appendFileSync } from "node:fs"; + +const repository = process.env.CI_REPOSITORY; +const branch = process.env.CI_DEFAULT_BRANCH; +const head = process.env.HEAD_SHA; +const shaPattern = /^[0-9a-f]{40}$/i; + +function ancestor(base) { + try { + execFileSync("git", ["merge-base", "--is-ancestor", base, head], { + stdio: "ignore", + }); + return true; + } catch { + return false; + } +} + +function write(base, reason) { + const output = { base, reason }; + if (process.env.GITHUB_OUTPUT) { + appendFileSync( + process.env.GITHUB_OUTPUT, + `base=${base}\nreason=${reason.replace(/[\r\n]/g, " ")}\n`, + ); + } + console.log(JSON.stringify(output)); +} + +if (!repository || !branch || !head || !shaPattern.test(head)) { + write("", "Repository, default branch, or checked-out head is invalid"); +} else { + try { + const response = execFileSync( + "gh", + [ + "api", + "--method", + "GET", + `repos/${repository}/actions/workflows/ci.yml/runs`, + "-f", + `branch=${branch}`, + "-f", + "event=push", + "-f", + "status=success", + "-F", + "per_page=100", + ], + { encoding: "utf8", timeout: 15_000, maxBuffer: 4 * 1024 * 1024 }, + ); + const runs = JSON.parse(response).workflow_runs; + if (!Array.isArray(runs)) + throw new Error("GitHub returned no workflow run list"); + const base = runs.find( + (run) => + run?.event === "push" && + run?.conclusion === "success" && + run?.head_branch === branch && + run?.head_repository?.full_name === repository && + typeof run?.head_sha === "string" && + shaPattern.test(run.head_sha) && + run.head_sha.toLowerCase() !== head.toLowerCase() && + ancestor(run.head_sha), + )?.head_sha; + if (base) write(base, ""); + else + write( + "", + "No trusted successful CI push baseline is an ancestor of this head", + ); + } catch { + write("", "Could not read a successful CI push baseline from GitHub"); + } +} diff --git a/scripts/ci/job-timings-batch.mjs b/scripts/ci/job-timings-batch.mjs new file mode 100644 index 000000000..842e884f0 --- /dev/null +++ b/scripts/ci/job-timings-batch.mjs @@ -0,0 +1,268 @@ +import { execFile } from "node:child_process"; +import { appendFile, mkdir, writeFile } from "node:fs/promises"; +import { join, resolve } from "node:path"; +import { promisify } from "node:util"; +import { fileURLToPath } from "node:url"; + +import { collectJobTimings } from "./job-timings.mjs"; + +const execFileAsync = promisify(execFile); +const OUTPUT_DIRECTORY = join(".cache", "ci-job-timings"); +const MAX_BUFFER = 16 * 1024 * 1024; +const API_TIMEOUT_MS = 30_000; +const WORKFLOWS = [ + { id: "ci.yml", name: "CI" }, + { id: "release.yml", name: "Release" }, +]; +const LOOKBACK_MS = 24 * 60 * 60 * 1_000; +const MAX_RUNS_PER_WORKFLOW = 100; +const TIME_BUDGET_MS = 240_000; + +/** Collect a bounded recent sample of completed CI and Release attempts. */ +export async function collectJobTimingsBatch({ + repository, + outputDirectory = OUTPUT_DIRECTORY, + observedAt = new Date().toISOString(), + env = process.env, + now = Date.now, + collect = collectJobTimings, +} = {}) { + const identity = validateRepository(repository); + const started = now(); + const cutoff = new Date(started - LOOKBACK_MS).toISOString(); + const manifest = { + schemaVersion: 1, + observedAt, + repository: identity, + selection: { + workflows: WORKFLOWS, + status: "completed", + createdAfter: cutoff, + lookbackHours: 24, + overlap: "Each hourly run re-reads the preceding 24 hours.", + pagination: + "Run listings use the first 100; jobs are paginated within the remaining time and 16 MiB response limits.", + maxRunsPerWorkflow: MAX_RUNS_PER_WORKFLOW, + timeBudgetMs: TIME_BUDGET_MS, + exhaustive: false, + }, + coverage: { + workflowsQueried: [], + runsListed: 0, + selectedRuns: 0, + collectedRuns: 0, + }, + truncation: [], + errors: [], + runs: [], + elapsedMs: 0, + }; + const budgetExpired = () => now() - started >= TIME_BUDGET_MS; + + const selected = []; + for (const workflow of WORKFLOWS) { + if (budgetExpired()) { + manifest.truncation.push("Time budget reached while listing workflows."); + break; + } + try { + const response = await apiJson( + `repos/${identity}/actions/workflows/${workflow.id}/runs?status=completed&created=%3E%3D${encodeURIComponent(cutoff)}&per_page=${String(MAX_RUNS_PER_WORKFLOW)}&page=1`, + env, + started, + now, + ); + if (!Array.isArray(response.workflow_runs)) + throw new Error("run listing must contain a workflow_runs array"); + manifest.coverage.workflowsQueried.push(workflow.name); + manifest.coverage.runsListed += response.workflow_runs.length; + if (response.workflow_runs.length >= MAX_RUNS_PER_WORKFLOW) + manifest.truncation.push( + `${workflow.name} run listing reached the 100-run first-page limit.`, + ); + for (const run of response.workflow_runs.slice( + 0, + MAX_RUNS_PER_WORKFLOW, + )) { + if ( + run.status !== "completed" || + !Number.isSafeInteger(run.id) || + !Number.isSafeInteger(run.run_attempt) + ) { + manifest.errors.push({ + scope: `workflow:${workflow.name}`, + runId: run.id ?? null, + message: + "Run record missing completed status, numeric id, or numeric latest attempt.", + }); + continue; + } + if (run.created_at && Date.parse(run.created_at) < Date.parse(cutoff)) + continue; + selected.push({ + id: run.id, + runAttempt: run.run_attempt, + workflow: workflow.name, + conclusion: run.conclusion ?? null, + createdAt: run.created_at ?? null, + }); + } + } catch (cause) { + manifest.errors.push({ + scope: `workflow:${workflow.name}`, + message: errorMessage(cause), + }); + } + } + + selected.sort( + (left, right) => + (Date.parse(right.createdAt ?? "") || 0) - + (Date.parse(left.createdAt ?? "") || 0), + ); + manifest.coverage.selectedRuns = selected.length; + for (const run of selected) { + if (budgetExpired()) { + manifest.truncation.push( + "Time budget reached before all selected runs were collected.", + ); + break; + } + const runRecord = { + ...run, + status: "pending", + rawPath: null, + reportPath: null, + error: null, + }; + manifest.runs.push(runRecord); + const runDirectory = join( + outputDirectory, + "runs", + `${String(run.id)}-${String(run.runAttempt)}`, + ); + try { + const result = await collect({ + repository: identity, + runId: String(run.id), + runAttempt: String(run.runAttempt), + outputDirectory: runDirectory, + env, + timeoutMs: Math.max( + 1, + Math.min(API_TIMEOUT_MS, TIME_BUDGET_MS - (now() - started)), + ), + }); + runRecord.status = "collected"; + runRecord.rawPath = result.rawPath; + runRecord.reportPath = result.reportPath; + manifest.coverage.collectedRuns += 1; + } catch (cause) { + runRecord.status = "error"; + runRecord.error = errorMessage(cause); + manifest.errors.push({ + scope: "run", + runId: run.id, + runAttempt: run.runAttempt, + message: runRecord.error, + }); + } + } + if (budgetExpired() && manifest.runs.length < manifest.coverage.selectedRuns) + manifest.truncation.push( + "Time budget reached; selected runs remain unattempted.", + ); + return await writeManifest(outputDirectory, manifest, started, now, env); +} + +function validateRepository(repository) { + if ( + typeof repository !== "string" || + !/^[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/u.test(repository) + ) + throw new Error( + "CI_REPOSITORY must be an owner/name repository identifier", + ); + return repository; +} + +async function apiJson(endpoint, env, started, now) { + const timeout = Math.max( + 1, + Math.min(API_TIMEOUT_MS, TIME_BUDGET_MS - (now() - started)), + ); + const { stdout } = await execFileAsync("gh", ["api", endpoint], { + env, + encoding: "utf8", + maxBuffer: MAX_BUFFER, + timeout, + }); + try { + return JSON.parse(stdout); + } catch (cause) { + throw new Error("GitHub Actions API returned malformed JSON", { cause }); + } +} + +async function writeManifest(outputDirectory, manifest, started, now, env) { + manifest.elapsedMs = Math.max(0, now() - started); + if ( + manifest.elapsedMs >= TIME_BUDGET_MS && + !manifest.truncation.some((item) => item.includes("Time budget")) + ) + manifest.truncation.push("Time budget reached."); + await mkdir(outputDirectory, { recursive: true }); + const manifestPath = join(outputDirectory, "batch-manifest.json"); + await writeFile( + manifestPath, + `${JSON.stringify(manifest, null, 2)}\n`, + "utf8", + ); + if (env.GITHUB_STEP_SUMMARY) + await appendBatchSummary(env.GITHUB_STEP_SUMMARY, manifest, manifestPath); + return { manifestPath, manifest }; +} + +async function appendBatchSummary(summaryPath, manifest, manifestPath) { + const collected = manifest.runs.filter( + ({ status }) => status === "collected", + ).length; + const failed = manifest.runs.filter( + ({ status }) => status === "error", + ).length; + const lines = [ + "### Batched CI timing snapshot", + "", + `Observed ${manifest.observedAt}; collected ${String(collected)} of ${String(manifest.coverage.selectedRuns)} selected recent completed runs (${String(failed)} run errors).`, + `Coverage uses the first ${String(MAX_RUNS_PER_WORKFLOW)} runs per workflow created in the preceding 24 hours and a ${String(TIME_BUDGET_MS / 1_000)} second total budget. This sample is bounded and is not exhaustive.`, + `Manifest: \`${manifestPath}\`.`, + ...manifest.truncation.map((message) => `- Truncated: ${message}`), + ...manifest.errors.map( + ({ scope, message }) => `- Error (${scope}): ${message}`, + ), + "", + ]; + await appendFile(summaryPath, lines.join("\n"), "utf8"); +} + +function errorMessage(cause) { + return cause instanceof Error ? cause.message : String(cause); +} + +if ( + process.argv[1] !== undefined && + resolve(process.argv[1]) === fileURLToPath(import.meta.url) +) { + try { + const result = await collectJobTimingsBatch({ + repository: process.env.CI_REPOSITORY, + outputDirectory: process.env.CI_JOB_TIMINGS_DIRECTORY ?? OUTPUT_DIRECTORY, + }); + process.stdout.write( + `Wrote batch manifest for ${String(result.manifest.coverage.collectedRuns)} collected runs to ${result.manifestPath}\n`, + ); + } catch (cause) { + process.stderr.write(`CI timing batch failed: ${errorMessage(cause)}\n`); + process.exitCode = 1; + } +} diff --git a/scripts/ci/job-timings.mjs b/scripts/ci/job-timings.mjs new file mode 100644 index 000000000..4d96f91b0 --- /dev/null +++ b/scripts/ci/job-timings.mjs @@ -0,0 +1,354 @@ +import { execFile } from "node:child_process"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { dirname, join, resolve } from "node:path"; +import { promisify } from "node:util"; +import { fileURLToPath } from "node:url"; + +const execFileAsync = promisify(execFile); +const MAX_BUFFER = 16 * 1024 * 1024; +const TIMEOUT_MS = 30_000; +const OUTPUT_DIRECTORY = join(".cache", "ci-job-timings"); +const STEP_COSTS = [ + { key: "npmCi", pattern: /\bnpm ci\b/iu }, + { key: "artifactUpload", pattern: /upload.*artifact/iu }, + { key: "artifactDownload", pattern: /download.*artifact/iu }, +]; + +/** Fetch and summarize one workflow attempt without turning telemetry into a gate. */ +export async function collectJobTimings({ + repository, + runId, + runAttempt, + outputDirectory = OUTPUT_DIRECTORY, + summaryPath, + observedAt = new Date().toISOString(), + env = process.env, + timeoutMs = TIMEOUT_MS, +}) { + const identity = validateIdentity({ repository, runId, runAttempt }); + const endpoint = `repos/${identity.repository}/actions/runs/${identity.runId}/attempts/${identity.runAttempt}/jobs?per_page=100`; + const { stdout } = await execFileAsync( + "gh", + ["api", "--paginate", "--slurp", endpoint], + { env, encoding: "utf8", maxBuffer: MAX_BUFFER, timeout: timeoutMs }, + ); + const rawPath = join(outputDirectory, "jobs.json"); + const reportPath = join(outputDirectory, "report.json"); + await mkdir(outputDirectory, { recursive: true }); + await writeFile(rawPath, stdout, "utf8"); + const pages = parsePages(stdout); + const jobs = pages.flatMap(({ jobs: pageJobs }, index) => + validateJobs(pageJobs, index), + ); + const report = normalizeReport({ identity, jobs, observedAt }); + + await writeFile(reportPath, `${JSON.stringify(report, null, 2)}\n`, "utf8"); + if (summaryPath !== undefined) + await appendSummary(summaryPath, report, rawPath, reportPath); + return { rawPath, reportPath, report }; +} + +function validateIdentity({ repository, runId, runAttempt }) { + if ( + typeof repository !== "string" || + !/^[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/u.test(repository) + ) + throw new Error( + "CI_REPOSITORY must be an owner/name repository identifier", + ); + return { + repository, + runId: parsePositiveInteger(runId, "CI_TIMING_RUN_ID"), + runAttempt: parsePositiveInteger(runAttempt, "CI_TIMING_RUN_ATTEMPT"), + }; +} + +function parsePositiveInteger(value, name) { + if (typeof value === "string" && /^[1-9]\d*$/u.test(value)) { + const parsed = Number(value); + if (Number.isSafeInteger(parsed)) return parsed; + } + throw new Error(`${name} must be a positive safe integer`); +} + +function parsePages(stdout) { + let pages; + try { + pages = JSON.parse(stdout); + } catch (cause) { + throw new Error("GitHub jobs API returned malformed JSON", { cause }); + } + if (!Array.isArray(pages) || pages.length === 0) + throw new Error("GitHub jobs API slurp must contain a nonempty page array"); + for (const page of pages) + if ( + typeof page !== "object" || + page === null || + Array.isArray(page) || + !Array.isArray(page.jobs) + ) + throw new Error("GitHub jobs API page must contain a jobs array"); + return pages; +} + +function validateJobs(jobs, pageIndex) { + return jobs.map((job, jobIndex) => { + if ( + typeof job !== "object" || + job === null || + Array.isArray(job) || + typeof job.name !== "string" || + job.name.length === 0 || + typeof job.status !== "string" + ) + throw new Error( + `GitHub jobs API page ${String(pageIndex + 1)} job ${String(jobIndex + 1)} is malformed`, + ); + return job; + }); +} + +function normalizeReport({ identity, jobs, observedAt }) { + const normalizedJobs = jobs.map(normalizeJob); + const queues = normalizedJobs + .map(({ queue }) => queue.seconds) + .filter((value) => value !== null); + const completedExecutions = normalizedJobs + .filter( + ({ status, conclusion }) => + status === "completed" && conclusion !== "skipped", + ) + .map(({ execution }) => execution.seconds) + .filter((value) => value !== null); + const completedDurationSumSeconds = completedExecutions.reduce( + (sum, seconds) => sum + seconds, + 0, + ); + const completedExecutionUnknown = normalizedJobs.filter( + ({ status, conclusion, execution }) => + status === "completed" && + conclusion !== "skipped" && + execution.seconds === null, + ).length; + const skipped = normalizedJobs.filter( + ({ conclusion }) => conclusion === "skipped", + ).length; + const costs = Object.fromEntries(STEP_COSTS.map(({ key }) => [key, []])); + for (const job of normalizedJobs) + for (const step of job.steps) + if (step.costKey !== null) + costs[step.costKey].push({ job: job.name, ...step }); + + return { + schemaVersion: 1, + run: { + ...identity, + observedAt, + snapshot: + normalizedJobs.length > 0 && + normalizedJobs.every(({ status }) => status === "completed") + ? "completed job status snapshot; unknown timestamps remain unmeasured" + : "partial; job statuses may still be queued or in progress", + }, + counts: { + jobs: normalizedJobs.length, + completed: normalizedJobs.filter(({ status }) => status === "completed") + .length, + skipped, + queueUnknown: normalizedJobs.filter(({ queue }) => queue.seconds === null) + .length, + executionUnknown: normalizedJobs.filter( + ({ execution }) => execution.seconds === null, + ).length, + completedExecutionUnknown, + }, + aggregates: { + queueSeconds: percentileSummary(queues), + completedExecutionSeconds: percentileSummary(completedExecutions), + summedCompletedJobElapsedSeconds: { + value: + completedExecutions.length === 0 + ? null + : round(completedDurationSumSeconds), + knownCount: completedExecutions.length, + unknownCount: completedExecutionUnknown, + label: + "sum of known completed job elapsed durations; incomplete when timestamps are unknown; not CPU time, billing, or workflow wall time", + }, + }, + jobs: normalizedJobs, + stepCostInterpretation: + "Matched install/artifact steps by step name; these are not total artifact transfer costs.", + stepCosts: costs, + }; +} + +function normalizeJob(job) { + const skipped = job.conclusion === "skipped"; + const hasStarted = ["in_progress", "completed"].includes(job.status); + const queue = skipped + ? unknownDuration("job was skipped") + : !hasStarted + ? unknownDuration("job start is not established by its status") + : duration(job.created_at, job.started_at); + const execution = skipped + ? unknownDuration("job was skipped") + : job.status === "completed" + ? duration(job.started_at, job.completed_at) + : unknownDuration("job is not completed"); + const steps = Array.isArray(job.steps) ? job.steps.map(normalizeStep) : []; + return { + id: Number.isSafeInteger(job.id) && job.id > 0 ? job.id : null, + url: stringOrNull(job.html_url), + name: job.name, + workflowName: stringOrNull(job.workflow_name), + status: job.status, + conclusion: stringOrNull(job.conclusion), + createdAt: stringOrNull(job.created_at), + startedAt: stringOrNull(job.started_at), + completedAt: stringOrNull(job.completed_at), + queue, + execution, + steps, + }; +} + +function normalizeStep(step) { + if ( + typeof step !== "object" || + step === null || + Array.isArray(step) || + typeof step.name !== "string" + ) + throw new Error("GitHub jobs API contains a malformed job step"); + const cost = STEP_COSTS.find(({ pattern }) => pattern.test(step.name)); + return { + name: step.name, + costKey: cost?.key ?? null, + duration: duration(step.started_at, step.completed_at), + }; +} + +function duration(startValue, endValue) { + if (typeof startValue !== "string" || typeof endValue !== "string") + return unknownDuration("timestamp unavailable"); + const start = Date.parse(startValue); + const end = Date.parse(endValue); + if (!Number.isFinite(start) || !Number.isFinite(end)) + return unknownDuration("invalid timestamp"); + const milliseconds = end - start; + if (!Number.isFinite(milliseconds) || milliseconds < 0) + return unknownDuration("timestamps out of order"); + return { seconds: round(milliseconds / 1_000), reason: null }; +} + +function unknownDuration(reason) { + return { seconds: null, reason }; +} + +function percentileSummary(values) { + return { + count: values.length, + p50: nearestRank(values, 0.5), + p95: nearestRank(values, 0.95), + unit: "seconds", + }; +} + +function nearestRank(values, percentile) { + if (values.length === 0) return null; + const sorted = [...values].sort((left, right) => left - right); + return round(sorted[Math.ceil(percentile * sorted.length) - 1]); +} + +function round(value) { + return Math.round(value * 1_000) / 1_000; +} + +function stringOrNull(value) { + return typeof value === "string" ? value : null; +} + +async function appendSummary(summaryPath, report, rawPath, reportPath) { + await mkdir(dirname(summaryPath), { recursive: true }); + const rows = [ + ["Queue", report.aggregates.queueSeconds], + ["Completed execution", report.aggregates.completedExecutionSeconds], + ] + .map( + ([label, item]) => + `| ${label} | ${display(item.p50)} | ${display(item.p95)} | ${String(item.count)} |`, + ) + .join("\n"); + const jobRows = report.jobs + .map( + (job) => + `| ${escapeCell(job.name)} | ${job.status} | ${job.conclusion ?? "unknown"} | ${display(job.queue.seconds)} | ${job.queue.seconds === null ? job.queue.reason : ""} | ${display(job.execution.seconds)} | ${job.execution.seconds === null ? job.execution.reason : ""} |`, + ) + .join("\n"); + const costRows = STEP_COSTS.map(({ key }) => { + const samples = report.stepCosts[key]; + const values = samples + .map(({ duration: item }) => item.seconds) + .filter((value) => value !== null); + return `| ${key} | ${String(samples.length)} | ${display(nearestRank(values, 0.5))} | ${samples.filter(({ duration: item }) => item.seconds === null).length} |`; + }).join("\n"); + const text = [ + "### CI job timing snapshot", + "", + `Observed at ${report.run.observedAt} for run ${String(report.run.runId)}, attempt ${String(report.run.runAttempt)}. This is a point-in-time snapshot of fetched jobs. Queue starts at job creation and excludes preceding dependency waiting. Missing, queued, in-progress, invalid, or out-of-order timestamps are reported as unknown. Elapsed durations are not CPU time or billing data.`, + "", + `Fetched ${String(report.counts.jobs)} jobs; ${String(report.counts.completed)} are completed (${String(report.counts.skipped)} skipped); ${String(report.counts.queueUnknown)} queue durations and ${String(report.counts.executionUnknown)} execution durations (${String(report.counts.completedExecutionUnknown)} for non-skipped completed jobs) are unknown.`, + "", + "| Metric | p50 (s) | p95 (s) | Samples |", + "| --- | ---: | ---: | ---: |", + rows, + "", + "| Job | Status | Conclusion | Queue (s) | Queue detail | Execution (s) | Execution detail |", + "| --- | --- | --- | ---: | --- | ---: | --- |", + jobRows, + "", + `Sum of known completed job elapsed durations: **${display(report.aggregates.summedCompletedJobElapsedSeconds.value)} s** across ${String(report.aggregates.summedCompletedJobElapsedSeconds.knownCount)} known and ${String(report.aggregates.summedCompletedJobElapsedSeconds.unknownCount)} unknown jobs; this is not CPU time, billing, or workflow wall time.`, + "", + "Matched install/artifact steps by name; these are not total artifact transfer costs.", + "", + "| Matched step group | Samples | p50 (s) | Unknown |", + "| --- | ---: | ---: | ---: |", + costRows, + "", + `Reports: \`${rawPath}\`, \`${reportPath}\`.`, + "", + ].join("\n"); + await writeFile(summaryPath, text, { encoding: "utf8", flag: "a" }); +} + +function display(value) { + return value === null ? "unknown" : String(value); +} + +function escapeCell(value) { + return value.replaceAll("|", "\\|").replaceAll("\n", " "); +} + +if ( + process.argv[1] !== undefined && + resolve(process.argv[1]) === fileURLToPath(import.meta.url) +) { + try { + const result = await collectJobTimings({ + repository: process.env.CI_REPOSITORY, + runId: process.env.CI_TIMING_RUN_ID, + runAttempt: process.env.CI_TIMING_RUN_ATTEMPT, + outputDirectory: process.env.CI_JOB_TIMINGS_DIRECTORY ?? OUTPUT_DIRECTORY, + summaryPath: process.env.GITHUB_STEP_SUMMARY, + }); + process.stdout.write( + `Wrote ${result.report.jobs.length} job timing records to ${result.reportPath}\n`, + ); + } catch (cause) { + process.stderr.write( + `CI timing telemetry failed: ${cause instanceof Error ? cause.message : String(cause)}\n`, + ); + process.exitCode = 1; + } +} diff --git a/scripts/ci/plan.mjs b/scripts/ci/plan.mjs index d52f99cb3..169d00854 100644 --- a/scripts/ci/plan.mjs +++ b/scripts/ci/plan.mjs @@ -1,30 +1,20 @@ import { execFileSync } from "node:child_process"; import { appendFileSync } from "node:fs"; import { matchesGlob } from "node:path"; -import { MANUAL_VERIFIERS, SCOPES } from "./scopes.mjs"; +import { CI_FLAGS as flags, MANUAL_VERIFIERS, SCOPES } from "./scopes.mjs"; +import { planTestSelection } from "./test-selection.mjs"; -const flags = [ - ...new Set([ - "code", - "runtime", - "package", - "apple", - "fixtures", - "readiness", - "docs", - "workflows", - ...Object.keys(SCOPES), - ]), -]; -// Test files select the native-host lanes that run them explicitly. Provider -// scopes remain implementation-owned, so test-only edits stay off those lanes. -const NATIVE_TEST_OWNERS = ["apple", "package"]; +// Test files select the host/runtime lanes that execute their focused suites. +const HOST_TEST_OWNERS = ["apple", "package", "sqlite"]; // Actionlint validates workflow syntax only. The semantic suite asserts these // workflows' release, publication and self-hosted runner safeguards. const GUARDED_WORKFLOWS = new Set([ + ".github/workflows/pr-title.yml", + ".github/workflows/ci-full-pr.yml", ".github/workflows/release.yml", ".github/workflows/real-ghidra-windows.yml", ".github/workflows/sqlite-runtime.yml", + ".github/workflows/ci-timings.yml", ]); // Sources compiled by build-conformance-fixtures.mjs. The readiness journey // analyzes the built C fixture as well as its own applications. @@ -36,6 +26,15 @@ const FIXTURE_SOURCES = [ "tests/conformance/versions/**", ]; const READINESS_SOURCES = ["tests/conformance/readiness/**"]; +// These reporters affect CI output only; they do not select a host or provider +// verification lane when changed on their own. +const PORTABLE_CI_REPORTERS = new Set([ + "scripts/ci/job-timings.mjs", + "scripts/ci/job-timings-batch.mjs", + "scripts/ci/test-report.mjs", + "scripts/ci/test-timing-reporter.mjs", + "scripts/ci/test-history.mjs", +]); const matchesAny = (file, patterns) => patterns.some((pattern) => matchesGlob(file, pattern)); const selected = Object.fromEntries(flags.map((flag) => [flag, false])); @@ -63,6 +62,10 @@ function classifyPackage(base, head) { ]), ].filter((key) => beforeScripts?.[key] !== afterScripts?.[key]); for (const script of changed) { + if (script === "test:sqlite" || script === "test:sqlite:run") { + selected.sqlite = true; + continue; + } if ( /^(test(?::|$)|check(?::|$)|lint(?::|$)|format(?::|$)|knip$|jscpd$)/.test( script, @@ -108,101 +111,137 @@ function classifyPackage(base, head) { } const event = process.env.GITHUB_EVENT_NAME; -let files = []; -if ( - event === "push" || +if (!["pull_request", "push", "schedule", "workflow_dispatch"].includes(event)) + throw new Error(`Unsupported CI event: ${event}`); +const requestedFull = event === "schedule" || event === "workflow_dispatch" || - process.env.CI_FULL === "true" -) { + process.env.CI_FULL === "true"; +const tier = + event === "schedule" && + process.env.CI_SKIP_FULL === "true" && + process.env.CI_FULL !== "true" + ? "reused" + : requestedFull + ? "full" + : "fast"; +let files = []; +if (tier === "reused") { + reasons.push( + "Recent exact-SHA full qualification reused; no fresh provider observation", + ); +} else if (tier === "full") { full("Full baseline requested by event or ci:full label"); -} else if (event === "pull_request") { - const base = git("merge-base", process.env.BASE_SHA, process.env.HEAD_SHA); +} else if (event === "pull_request" || event === "push") { const head = process.env.HEAD_SHA; - // Disable rename detection so both the removed and added paths participate. - files = git("diff", "--name-only", "--no-renames", "-z", base, head) - .split("\0") - .filter(Boolean); - for (const file of files) { - if ( - /^(README(?:_[^/]+)?\.md|AGENTS\.md|CONTRIBUTING\.md|CHANGELOG\.md|SECURITY\.md|native\/windows\/README\.md|\.github\/pull_request_template\.md)$/.test( - file, - ) || - file.startsWith("docs/") || - file.startsWith(".github/ISSUE_TEMPLATE/") - ) { - selected.docs = true; - selected.website ||= matchesAny(file, SCOPES.website); - continue; - } - if (/^(src|tests)\/.*\.(?:test|spec)\.[cm]?[jt]sx?$/.test(file)) { - selected.code = true; - for (const owner of NATIVE_TEST_OWNERS) - selected[owner] ||= matchesAny(file, SCOPES[owner]); - continue; - } - if (matchesAny(file, MANUAL_VERIFIERS)) { - selected.code = true; - continue; - } - const owners = Object.entries(SCOPES) - .filter(([, patterns]) => - patterns.some((pattern) => matchesGlob(file, pattern)), - ) - .map(([owner]) => owner); - for (const owner of owners) selected[owner] = true; - if ( - file.startsWith(".github/workflows/") || - file === ".github/actionlint.yaml" - ) { - selected.workflows = true; - if (GUARDED_WORKFLOWS.has(file)) selected.code = true; - if (file !== ".github/workflows/ci.yml") continue; - } - // The Notes example is also the source-owned agent evaluation fixture. - if (file.startsWith("website/public/examples/notes-electron/")) - selected.code = true; - if (owners.length === 1 && owners[0] === "website") continue; - if (/^src\/(contracts|domain)\//.test(file)) selected.docs = true; - if (file === "package.json") { - classifyPackage(base, head); - continue; - } - selected.code = true; - if (matchesAny(file, FIXTURE_SOURCES)) { - selected.fixtures = true; - selected.readiness = true; - } - if (matchesAny(file, READINESS_SOURCES)) selected.readiness = true; - if (file.startsWith("tests/")) continue; - if ( - /^(src\/process\/|src\/filesystem\/|src\/contracts\/tool|src\/server\/createServer|src\/cli\.ts|scripts\/ci\/)/.test( - file, - ) - ) { - full(`Shared boundary: ${file}`); - } else if (owners.length) { - if (file.startsWith("src/") || file.startsWith("bridge/")) - selected.runtime = true; - } else if (/^(src|bridge)\//.test(file)) { - // Unowned runtime sources keep the Linux x64 installed-package baseline; - // ci:full, main pushes and manual dispatch retain every native host. - selected.runtime = true; - reasons.push(`Unowned runtime source: ${file}`); - } else if (file.startsWith(".agents/skills/")) { - selected.runtime = true; - selected.docs = true; - } else { - full(`Unclassified input: ${file}`); + let base; + if (event === "pull_request") { + base = git("merge-base", process.env.BASE_SHA, head); + } else { + const candidate = process.env.BASE_SHA; + const baselineReason = process.env.BASELINE_REASON; + try { + if (!candidate || !/^[0-9a-f]{40}$/i.test(candidate)) + throw new Error( + baselineReason || "No valid successful push baseline was found", + ); + execFileSync("git", ["merge-base", "--is-ancestor", candidate, head], { + stdio: "ignore", + }); + base = candidate; + } catch { + full( + baselineReason || + "No valid successful push baseline ancestor; selecting all lanes", + ); } } -} else throw new Error(`Unsupported CI event: ${event}`); + if (base) { + // Disable rename detection so both the removed and added paths participate. + files = git("diff", "--name-only", "--no-renames", "-z", base, head) + .split("\0") + .filter(Boolean); + for (const file of files) { + if ( + /^(README(?:_[^/]+)?\.md|AGENTS\.md|CONTRIBUTING\.md|CHANGELOG\.md|SECURITY\.md|native\/windows\/README\.md|\.github\/pull_request_template\.md)$/.test( + file, + ) || + file.startsWith("docs/") || + file.startsWith(".github/ISSUE_TEMPLATE/") + ) { + selected.docs = true; + selected.website ||= matchesAny(file, SCOPES.website); + continue; + } + if (/^(src|tests)\/.*\.(?:test|spec)\.[cm]?[jt]sx?$/.test(file)) { + selected.code = true; + for (const owner of HOST_TEST_OWNERS) + selected[owner] ||= matchesAny(file, SCOPES[owner]); + continue; + } + if (matchesAny(file, MANUAL_VERIFIERS)) { + selected.code = true; + continue; + } + const owners = Object.entries(SCOPES) + .filter(([, patterns]) => + patterns.some((pattern) => matchesGlob(file, pattern)), + ) + .map(([owner]) => owner); + for (const owner of owners) selected[owner] = true; + if ( + file.startsWith(".github/workflows/") || + file === ".github/actionlint.yaml" + ) { + selected.workflows = true; + if (GUARDED_WORKFLOWS.has(file)) selected.code = true; + if (file !== ".github/workflows/ci.yml") continue; + } + // The Notes example is also the source-owned agent evaluation fixture. + if (file.startsWith("website/public/examples/notes-electron/")) + selected.code = true; + if (owners.length === 1 && owners[0] === "website") continue; + if (/^src\/(contracts|domain)\//.test(file)) selected.docs = true; + if (file === "package.json") { + classifyPackage(base, head); + continue; + } + selected.code = true; + if (matchesAny(file, FIXTURE_SOURCES)) { + selected.fixtures = true; + selected.readiness = true; + } + if (matchesAny(file, READINESS_SOURCES)) selected.readiness = true; + if (file.startsWith("tests/")) continue; + if (PORTABLE_CI_REPORTERS.has(file)) continue; + if ( + /^(src\/process\/|src\/filesystem\/|src\/contracts\/tool|src\/server\/createServer|src\/cli\.ts|scripts\/ci\/)/.test( + file, + ) + ) { + full(`Shared boundary: ${file}`); + } else if (owners.length) { + if (file.startsWith("src/") || file.startsWith("bridge/")) + selected.runtime = true; + } else if (/^(src|bridge)\//.test(file)) { + // Unowned runtime sources keep the Linux x64 installed-package baseline; + // full qualification retains every native host. + selected.runtime = true; + reasons.push(`Unowned runtime source: ${file}`); + } else if (file.startsWith(".agents/skills/")) { + selected.runtime = true; + selected.docs = true; + } else { + full(`Unclassified input: ${file}`); + } + } + } +} selected.runtime ||= selected.package; const linux = { os: "ubuntu-latest", platform: "linux", architecture: "x64" }; -// macOS x64 is the slowest package host. Pull requests run it only with ci:full; -// main pushes, schedules and manual dispatch always include it. +// macOS x64 is the slowest package host. Fast qualification omits it. const intel = - event !== "pull_request" || process.env.CI_FULL === "true" + tier === "full" ? [{ os: "macos-15-intel", platform: "darwin", architecture: "x64" }] : []; const matrix = { @@ -215,17 +254,23 @@ const matrix = { ] : [linux], }; -const plan = { ...selected, matrix, files, reasons }; +const testSelection = planTestSelection(files, event, tier); +const plan = { ...selected, tier, matrix, files, reasons, testSelection }; if (process.env.GITHUB_OUTPUT) appendFileSync( process.env.GITHUB_OUTPUT, - Object.entries({ ...selected, matrix: JSON.stringify(matrix) }) + Object.entries({ + ...selected, + tier, + matrix: JSON.stringify(matrix), + test_selection: JSON.stringify(testSelection), + }) .map(([key, value]) => `${key}=${value}\n`) .join(""), ); if (process.env.GITHUB_STEP_SUMMARY) appendFileSync( process.env.GITHUB_STEP_SUMMARY, - `### Selected CI lanes\n\n${flags.filter((flag) => selected[flag]).join(", ") || "Formatting only"}\n\n${reasons.join("\n\n")}\n`, + `### Selected CI lanes (${tier})\n\n${flags.filter((flag) => selected[flag]).join(", ") || "Formatting only"}\n\n${reasons.join("\n\n")}\n`, ); console.log(JSON.stringify(plan)); diff --git a/scripts/ci/qualification.mjs b/scripts/ci/qualification.mjs new file mode 100644 index 000000000..ab47c168f --- /dev/null +++ b/scripts/ci/qualification.mjs @@ -0,0 +1,176 @@ +import { execFileSync } from "node:child_process"; +import { appendFileSync, mkdirSync, writeFileSync } from "node:fs"; +import { resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +const ARTIFACT_NAME = "ci-full-qualification-v1"; +const MAX_AGE_MS = 24 * 60 * 60 * 1000; +const positiveId = (value) => + (typeof value === "string" || typeof value === "number") && + /^[1-9]\d*$/.test(String(value)) && + Number.isSafeInteger(Number(value)); +const commitSha = (value) => + typeof value === "string" && /^[0-9a-f]{40}$/.test(value); + +/** Record only after the required gate has verified every full-tier obligation. */ +export function recordQualification(directory) { + const sourceSha = process.env.GITHUB_SHA; + const runId = process.env.GITHUB_RUN_ID; + const runAttempt = process.env.GITHUB_RUN_ATTEMPT; + const repository = process.env.GITHUB_REPOSITORY; + if ( + !commitSha(sourceSha) || + !positiveId(runId) || + !positiveId(runAttempt) || + !repository + ) + throw new Error( + "Full qualification receipt requires source and workflow identity", + ); + const checkoutSha = execFileSync("git", ["rev-parse", "HEAD"], { + encoding: "utf8", + }).trim(); + if (checkoutSha !== sourceSha) + throw new Error("Qualification checkout differs from workflow source"); + mkdirSync(directory, { recursive: true }); + const path = resolve(directory, "receipt.json"); + writeFileSync( + path, + `${JSON.stringify( + { + schemaVersion: 1, + tier: "full", + sourceSha, + repository, + runId: Number(runId), + runAttempt: Number(runAttempt), + qualifiedAt: new Date().toISOString(), + }, + null, + 2, + )}\n`, + ); + if (process.env.GITHUB_OUTPUT) + appendFileSync( + process.env.GITHUB_OUTPUT, + `receipt_name=${ARTIFACT_NAME}\nreceipt_path=${path}\n`, + ); +} + +function admit() { + if (process.env.CI_FORCE_FULL === "true") + return { + run_full: true, + reason: "Forced daily runner and dependency refresh", + }; + const repository = process.env.CI_REPOSITORY; + const repositoryId = Number(process.env.CI_REPOSITORY_ID); + const branch = process.env.CI_DEFAULT_BRANCH; + const head = process.env.HEAD_SHA; + if (!repository || !positiveId(repositoryId) || !branch || !commitSha(head)) + return { + run_full: true, + reason: "Qualification lookup identity is incomplete", + }; + + const now = Date.now(); + const deadline = now + 20_000; + const api = (...arguments_) => { + const remaining = deadline - Date.now(); + if (remaining <= 0) + throw new Error("Qualification lookup budget exhausted"); + return JSON.parse( + execFileSync("gh", ["api", "--method", "GET", ...arguments_], { + encoding: "utf8", + timeout: Math.min(15_000, remaining), + maxBuffer: 8 * 1024 * 1024, + stdio: ["ignore", "pipe", "pipe"], + }), + ); + }; + try { + const { artifacts } = api( + `repos/${repository}/actions/artifacts`, + "-f", + `name=${ARTIFACT_NAME}`, + "-F", + "per_page=100", + ); + if (!Array.isArray(artifacts)) + throw new Error("Missing qualification artifact inventory"); + const candidates = artifacts.filter((artifact) => { + const lineage = artifact?.workflow_run; + const age = now - Date.parse(artifact?.created_at); + return ( + artifact?.name === ARTIFACT_NAME && + artifact?.expired === false && + typeof artifact?.created_at === "string" && + Number.isFinite(age) && + age >= 0 && + age < MAX_AGE_MS && + positiveId(lineage?.id) && + String(lineage.id) !== process.env.GITHUB_RUN_ID && + lineage?.repository_id === repositoryId && + lineage?.head_repository_id === repositoryId && + lineage?.head_branch === branch && + lineage?.head_sha === head + ); + }); + // Artifact metadata narrows the search before run lookups. Never scan an + // unbounded history or treat a missing receipt as a successful qualification. + for (const artifact of candidates.slice(0, 3)) { + const run = api( + `repos/${repository}/actions/runs/${artifact.workflow_run.id}`, + ); + // An old run can upload a new artifact when only its gate is rerun. + // Bound the original run age as well so retries cannot renew stale proof. + const runAge = Date.now() - Date.parse(run.created_at); + if ( + typeof run.created_at === "string" && + Number.isFinite(runAge) && + runAge >= 0 && + runAge < MAX_AGE_MS && + run.id === artifact.workflow_run.id && + run.status === "completed" && + run.conclusion === "success" && + run.path === ".github/workflows/ci.yml" && + ["schedule", "workflow_dispatch"].includes(run.event) && + run.head_sha === head && + run.head_branch === branch && + run.head_repository?.full_name === repository + ) { + return { + run_full: false, + run_id: run.id, + reason: `Reusing full qualification from run ${run.id}; no fresh provider observation`, + }; + } + } + return { + run_full: true, + reason: "No trusted full qualification for this SHA within 24 hours", + }; + } catch { + return { + run_full: true, + reason: "Qualification lookup unavailable; running full validation", + }; + } +} + +if (resolve(process.argv[1] ?? "") === fileURLToPath(import.meta.url)) { + const decision = admit(); + if (process.env.GITHUB_OUTPUT) + appendFileSync( + process.env.GITHUB_OUTPUT, + Object.entries(decision) + .map(([key, value]) => `${key}=${value}\n`) + .join(""), + ); + if (process.env.GITHUB_STEP_SUMMARY) + appendFileSync( + process.env.GITHUB_STEP_SUMMARY, + `### Scheduled qualification\n\n${decision.reason}.\n`, + ); + console.log(JSON.stringify(decision)); +} diff --git a/scripts/ci/required.mjs b/scripts/ci/required.mjs index 4f6586737..01907de37 100644 --- a/scripts/ci/required.mjs +++ b/scripts/ci/required.mjs @@ -1,18 +1,46 @@ +import { CI_FLAGS } from "./scopes.mjs"; +import { recordQualification } from "./qualification.mjs"; + const jobs = JSON.parse(process.env.CI_JOBS); -const outputs = jobs.changes.outputs; -const required = ["changes", "static", "test"]; +const outputs = jobs?.changes?.outputs; +if (!["fast", "full", "reused"].includes(outputs?.tier)) + throw new Error( + "CI classification requires a valid tier: fast, full, or reused", + ); +const invalidFlags = CI_FLAGS.filter( + (flag) => outputs[flag] !== "true" && outputs[flag] !== "false", +); +if (invalidFlags.length) + throw new Error( + `CI classification requires explicit true/false scope outputs: ${invalidFlags.join(", ")}`, + ); +const selectedScopes = CI_FLAGS.filter((flag) => outputs[flag] === "true"); +const reused = outputs.tier === "reused"; +if ( + reused && + (process.env.GITHUB_EVENT_NAME !== "schedule" || + !/^[1-9]\d*$/.test(outputs.qualification_run_id ?? "") || + !Number.isSafeInteger(Number(outputs.qualification_run_id)) || + selectedScopes.length !== 0) +) + throw new Error( + "Reused qualification requires a scheduled admission receipt and no selected lanes", + ); +if (outputs.tier === "full" && selectedScopes.length !== CI_FLAGS.length) + throw new Error("Full qualification requires every canonical CI scope"); +const required = reused ? ["changes"] : ["changes", "static", "test"]; const lanes = { - code: ["build", "test-shard", "sqlite-runtime"], + code: ["build", "test-shard"], runtime: ["windows-native-build", "package-e2e"], package: ["windows-curated"], + sqlite: ["sqlite-runtime"], apple: ["apple-artifacts"], fixtures: ["conformance-fixtures"], readiness: ["reconstruction-readiness"], docs: ["docs"], + workflows: ["static"], }; -for (const [scope, value] of Object.entries(outputs)) { - if (value === "true") required.push(...(lanes[scope] ?? [scope])); -} +for (const scope of selectedScopes) required.push(...(lanes[scope] ?? [scope])); const failed = [...new Set(required)].filter( (name) => jobs[name]?.result !== "success", ); @@ -23,3 +51,9 @@ for (const [name, job] of Object.entries(jobs)) { if (failed.length) throw new Error(`CI lanes did not pass: ${failed.join(", ")}`); console.log("Every selected CI lane passed"); +if (reused) + console.log( + `Qualification reused from run ${outputs.qualification_run_id}; no fresh provider observation`, + ); +if (outputs.tier === "full" && process.env.CI_QUALIFICATION_DIRECTORY) + recordQualification(process.env.CI_QUALIFICATION_DIRECTORY); diff --git a/scripts/ci/scopes.mjs b/scripts/ci/scopes.mjs index 497dfeed0..ffe15cb3d 100644 --- a/scripts/ci/scopes.mjs +++ b/scripts/ci/scopes.mjs @@ -55,6 +55,9 @@ export const SCOPES = { "tests/boundary/providers/native/artifactProvider.extractionCompleteness.test.ts", "tests/boundary/filesystem/setupUnavailableProvider.test.ts", "tests/boundary/filesystem/windowsNpxRegistration.test.ts", + "tests/acceptance/setup/homeDirectoryFallback.test.ts", + "tests/boundary/filesystem/javascriptApplicationWorkflows.test.ts", + "src/domain/javascript/javascriptRuntimeStaticCandidates.ts", "tests/boundary/filesystem/webScriptManifestPaths.test.ts", "tests/boundary/process/ProcessEvidence.host.test.ts", "tests/acceptance/webModuleCli.test.ts", @@ -84,6 +87,55 @@ export const SCOPES = { "tests/boundary/filesystem/javascriptArtifactContentIdentity.test.ts", "tests/boundary/process/sourceMapUncHosts.test.ts", ], + // Native SQLite runtime compatibility for the producer, public adapters, + // fixtures, shared lifecycle, and the package script that defines its suite. + sqlite: [ + "src/sqlite/**", + "src/domain/sqlite/**", + "src/contracts/sqlite/**", + "src/application/sqlite/**", + "src/composition/sqlite.ts", + "src/cli/sqliteCommands.ts", + "src/server/registerSqliteTools.ts", + "src/application/DirectAnalysisStatus*.ts", + "src/application/CapabilityInventory*.ts", + "src/application/AnalysisProvider*.ts", + "src/contracts/toolContracts.ts", + "src/contracts/toolContractTypes.ts", + "src/contracts/toolEffects*.ts", + "src/contracts/toolOutputSchemaPrimitives.ts", + "src/domain/analysisError*.ts", + "src/domain/evidence*.ts", + "src/domain/inputIssueProjection*.ts", + "src/domain/jsonSerialization*.ts", + "src/domain/jsonValue*.ts", + "src/domain/localPath*.ts", + "src/domain/providerAdapterError*.ts", + "src/domain/providerCleanupError*.ts", + "src/domain/result*.ts", + "src/process/OwnedCommand*.ts", + "src/process/ProviderProcess*.ts", + "src/process/PrivateRuntimeRoot*.ts", + "src/artifacts/ArtifactReader*.ts", + "src/artifacts/ArtifactResourceScope*.ts", + "src/artifacts/ArtifactPaths*.ts", + "src/artifacts/readStableArtifact*.ts", + "src/filesystem/OwnedFileHandle*.ts", + "src/server/EvidenceMcpServer*.ts", + "src/server/analysisAdmission*.ts", + "src/server/contractToolHandler*.ts", + "src/server/toolRegistrationOptions*.ts", + "src/server/createServer*.ts", + "src/cli.ts", + "src/cliCommandNames.ts", + "src/cliLogging.ts", + "src/cli/commandCancellation.ts", + "tests/boundary/sqlite/**", + "tests/fixtures/sqlite/**", + "tests/acceptance/sqliteDatabaseCli.test.ts", + "tests/boundary/mcp/serverLifecycleMcp.test.ts", + ".github/workflows/sqlite-runtime.yml", + ], browser: [ ".github/workflows/real-browser.yml", "scripts/verify-real-browser.mjs", @@ -299,6 +351,7 @@ export const SCOPES = { nativeaot: [ "bridge/nativeaot/**", "scripts/generate-nativeaot-fixtures.mjs", + "scripts/lib/nativeaot-build.*", "tests/fixtures/nativeaot/**", ".github/workflows/nativeaot-fixtures.yml", ], @@ -311,6 +364,20 @@ export const SCOPES = { ], }; +export const CI_FLAGS = [ + ...new Set([ + "code", + "runtime", + "package", + "apple", + "fixtures", + "readiness", + "docs", + "workflows", + ...Object.keys(SCOPES), + ]), +]; + // Verifier entrypoints that only manual, licensed, or self-hosted workflows // run. No selected CI lane executes them, so their edits need only the static // checks and the source-test suite. Shared helpers under scripts/lib remain diff --git a/scripts/ci/test-history.mjs b/scripts/ci/test-history.mjs new file mode 100644 index 000000000..4386f48af --- /dev/null +++ b/scripts/ci/test-history.mjs @@ -0,0 +1,114 @@ +import { execFileSync } from "node:child_process"; +import { appendFileSync } from "node:fs"; +import { resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +const ARTIFACT_NAME = "test-timings"; +const MAX_AGE_MS = 30 * 24 * 60 * 60 * 1000; +const MAX_CANDIDATES = 5; +const positiveId = (value) => + (typeof value === "string" || typeof value === "number") && + /^[1-9]\d*$/.test(String(value)) && + Number.isSafeInteger(Number(value)); + +function findTimingRun({ repository, branch, repositoryId, now = Date.now() }) { + if ( + !/^[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/.test(repository ?? "") || + !branch || + !positiveId(repositoryId) + ) + return ""; + const deadline = Date.now() + 20_000; + const api = (...arguments_) => { + const remaining = deadline - Date.now(); + if (remaining <= 0) + throw new Error("Timing history lookup budget exhausted"); + return JSON.parse( + execFileSync("gh", ["api", "--method", "GET", ...arguments_], { + encoding: "utf8", + timeout: Math.min(10_000, remaining), + maxBuffer: 4 * 1024 * 1024, + stdio: ["ignore", "pipe", "pipe"], + }), + ); + }; + + const { artifacts } = api( + `repos/${repository}/actions/artifacts`, + "-f", + `name=${ARTIFACT_NAME}`, + "-F", + "per_page=100", + ); + if (!Array.isArray(artifacts)) + throw new Error("Missing timing artifact inventory"); + const candidates = artifacts + .filter((artifact) => { + const lineage = artifact?.workflow_run; + const age = now - Date.parse(artifact?.created_at); + return ( + artifact?.name === ARTIFACT_NAME && + artifact?.expired === false && + typeof artifact?.created_at === "string" && + Number.isFinite(age) && + age >= 0 && + age < MAX_AGE_MS && + positiveId(lineage?.id) && + positiveId(lineage?.repository_id) && + positiveId(lineage?.head_repository_id) && + lineage.repository_id === lineage.head_repository_id && + lineage.repository_id === Number(repositoryId) && + lineage.head_branch === branch + ); + }) + .sort( + (left, right) => + Date.parse(right.created_at) - Date.parse(left.created_at), + ) + .slice(0, MAX_CANDIDATES); + + for (const artifact of candidates) { + const run = api( + `repos/${repository}/actions/runs/${artifact.workflow_run.id}`, + ); + const runAge = now - Date.parse(run?.created_at); + if ( + typeof run?.created_at === "string" && + Number.isFinite(runAge) && + runAge >= 0 && + runAge < MAX_AGE_MS && + run.id === artifact.workflow_run.id && + run.status === "completed" && + run.conclusion === "success" && + run.path === ".github/workflows/ci.yml" && + ["schedule", "workflow_dispatch"].includes(run.event) && + run.head_branch === branch && + run.head_repository?.full_name === repository + ) + return String(run.id); + } + return ""; +} + +function write(runId) { + if (process.env.GITHUB_OUTPUT) + appendFileSync(process.env.GITHUB_OUTPUT, `run-id=${runId}\n`); + console.log(JSON.stringify({ run_id: runId })); +} + +if ( + process.argv[1] && + fileURLToPath(import.meta.url) === resolve(process.argv[1]) +) { + let runId = ""; + try { + runId = findTimingRun({ + repository: process.env.CI_REPOSITORY, + branch: process.env.CI_DEFAULT_BRANCH, + repositoryId: process.env.CI_REPOSITORY_ID, + }); + } catch { + // History improves shard ordering but must never block CI. + } + write(runId); +} diff --git a/scripts/ci/test-report.mjs b/scripts/ci/test-report.mjs index a3686d6ed..035a01990 100644 --- a/scripts/ci/test-report.mjs +++ b/scripts/ci/test-report.mjs @@ -1,28 +1,36 @@ +import { execFileSync } from "node:child_process"; import { appendFile, mkdir, readFile, rm, writeFile } from "node:fs/promises"; import { isAbsolute, join, relative, resolve, sep } from "node:path"; import { z } from "zod"; +import { compareTestSelection } from "./test-selection.mjs"; + const root = process.cwd(); const historyDirectory = join(root, ".cache/ci-test-history"); const historyPath = join(historyDirectory, "test-results.json"); +const selectionDirectory = join(root, ".cache/ci-test-selection"); +const selectionPath = join(selectionDirectory, "report.json"); // Never let a failed or partial rerun leave a previous successful history file. await rm(historyPath, { force: true }); +await rm(selectionPath, { force: true }); const report = z .object({ + version: z.literal(1), success: z.boolean(), - numFailedTests: z.number().int().nonnegative(), - testResults: z.array( + reason: z.string().optional(), + unhandledErrors: z.number().int().nonnegative().optional(), + files: z.array( z.object({ name: z.string().min(1), - startTime: z.number().finite(), - endTime: z.number().finite(), + durationMs: z.number().finite().nonnegative(), + state: z.string().optional(), }), ), }) - .parse(JSON.parse(await readFile("test-results.json", "utf8"))); + .parse(JSON.parse(await readFile("test-timings.json", "utf8"))); -const files = report.testResults.map(({ name, startTime, endTime }) => { +const files = report.files.map(({ name, durationMs, state }) => { const path = relative(root, resolve(root, name)); if ( path === "" || @@ -31,32 +39,85 @@ const files = report.testResults.map(({ name, startTime, endTime }) => { isAbsolute(path) ) throw new Error(`Test report path is outside the checkout: ${name}`); - const duration = endTime - startTime; - if (!Number.isFinite(duration) || duration < 0) - throw new Error(`Test report has an invalid duration: ${name}`); - return { name: path.split(sep).join("/"), startTime: 0, endTime: duration }; + return { name: path.split(sep).join("/"), durationMs, state }; }); const complete = process.env.SHARD_RESULT === "success"; const passed = - complete && - process.env.MERGE_RESULT === "success" && - report.success && - report.numFailedTests === 0; + complete && process.env.MERGE_RESULT === "success" && report.success; +// Successful shards and merged reports must account for the complete portable +// inventory. Coverage thresholds alone cannot prove that every file ran. +const verifyInventory = process.env.CI_VERIFY_TEST_INVENTORY === "true"; +const expected = + verifyInventory || process.env.CI_TEST_SELECTION + ? execFileSync("git", ["ls-files", "-z", "--", "src", "tests"], { + encoding: "utf8", + maxBuffer: 8 * 1024 * 1024, + }) + .split("\0") + .filter((path) => path.endsWith(".test.ts")) + : []; +let selection; +if (process.env.CI_TEST_SELECTION) { + const comparison = compareTestSelection( + JSON.parse(process.env.CI_TEST_SELECTION), + expected, + files, + ); + selection = { + ...comparison, + complete: + comparison.complete && + report.unhandledErrors === 0 && + ["passed", "failed"].includes(report.reason) && + ["success", "failure"].includes(process.env.SHARD_RESULT) && + ["success", "failure"].includes(process.env.MERGE_RESULT), + executionSha: process.env.GITHUB_SHA ?? null, + prHeadSha: process.env.CI_PR_HEAD_SHA ?? null, + runId: process.env.GITHUB_RUN_ID ?? null, + runAttempt: process.env.GITHUB_RUN_ATTEMPT ?? null, + }; + await mkdir(selectionDirectory, { recursive: true }); + await writeFile(selectionPath, JSON.stringify(selection)); +} +if (passed && verifyInventory) { + const actual = files.map(({ name }) => name); + const expectedSet = new Set(expected); + const actualSet = new Set(actual); + const missing = expected.filter((path) => !actualSet.has(path)); + const unexpected = actual.filter((path) => !expectedSet.has(path)); + if (missing.length || unexpected.length || actualSet.size !== actual.length) + throw new Error( + `Incomplete portable test inventory: ${JSON.stringify({ missing, unexpected, duplicateFiles: actual.length - actualSet.size })}`, + ); +} const summary = [ complete ? "## Merged test diagnostics" : "## Partial test diagnostics", "", + ...(selection + ? [ + "### Observational PR test selection", + "", + `Plan: ${selection.plan.mode}. Proposed files: ${selection.selected.length}; omitted: ${selection.omitted.length}.`, + `Full-run comparison: ${selection.complete ? "complete" : "incomplete"}; failures outside proposed selection: ${selection.missedFailures.length}.`, + "The complete suite still ran. This report cannot enable omissions or qualify a revision. Skipped, missing, and interrupted observations do not establish parity.", + "", + ] + : []), `Shard result: ${process.env.SHARD_RESULT ?? "unknown"}; merge result: ${process.env.MERGE_RESULT ?? "unknown"}.`, passed ? "Successful timing history is available." : "This report is diagnostic only and is not a successful timing baseline.", "", + "Durations include tests and hooks, worker preparation, environment, imports, and setup. They are scheduling weights, not summed workflow wall time.", + "", "| Slowest file | Duration (ms) |", "| --- | ---: |", ...files - .toSorted((left, right) => right.endTime - left.endTime) + .toSorted((left, right) => right.durationMs - left.durationMs) .slice(0, 20) .map( - ({ name, endTime }) => `| ${name.replaceAll("|", "\\|")} | ${endTime} |`, + ({ name, durationMs }) => + `| ${name.replaceAll("|", "\\|")} | ${durationMs} |`, ), "", ].join("\n"); @@ -68,6 +129,10 @@ if (passed) { await mkdir(historyDirectory, { recursive: true }); await writeFile( historyPath, - JSON.stringify({ success: true, numFailedTests: 0, testResults: files }), + JSON.stringify({ + version: 1, + success: true, + files: files.map(({ name, durationMs }) => ({ name, durationMs })), + }), ); } diff --git a/scripts/ci/test-selection.d.mts b/scripts/ci/test-selection.d.mts new file mode 100644 index 000000000..c19414398 --- /dev/null +++ b/scripts/ci/test-selection.d.mts @@ -0,0 +1,35 @@ +export interface TestSelectionPlan { + readonly version: 1; + readonly mode: "full" | "candidate"; + readonly owners: readonly string[]; + readonly changedTests: readonly string[]; + readonly changedPaths: readonly string[]; + readonly reason: string; +} +export function planTestSelection( + files: readonly string[], + event: string, + tier: string, +): TestSelectionPlan; +export function compareTestSelection( + plan: unknown, + inventory: readonly string[], + results: readonly { + readonly name: string; + readonly durationMs: number; + readonly state?: string; + }[], +): { + readonly version: 1; + readonly observational: true; + readonly plan: TestSelectionPlan; + readonly selected: readonly string[]; + readonly omitted: readonly string[]; + readonly missing: readonly string[]; + readonly unexpected: readonly string[]; + readonly duplicateFiles: number; + readonly unobserved: readonly string[]; + readonly missedFailures: readonly string[]; + readonly complete: boolean; + readonly omittedDurationMs: number; +}; diff --git a/scripts/ci/test-selection.mjs b/scripts/ci/test-selection.mjs new file mode 100644 index 000000000..8165e3658 --- /dev/null +++ b/scripts/ci/test-selection.mjs @@ -0,0 +1,151 @@ +import { matchesGlob } from "node:path"; + +// Candidate ownership only: these leaf adapters and their producer tests can be +// compared with full-suite outcomes. This table never changes test execution. +// Shared application/domain code and dynamic CLI/MCP consumers are not leaf owners. +const OWNERS = { + adb: ["src/adb/**", "tests/boundary/providers/adb/**"], + apktool: ["src/apktool/**"], + dotnet: ["src/dotnet/**"], + evm: ["src/evm/**", "tests/boundary/evm/**"], + firmware: ["src/firmware/**", "tests/boundary/firmware/**"], + flutter: ["src/flutter/**"], + ghidra: ["src/ghidra/**", "tests/boundary/providers/ghidra/**"], + hopper: ["src/hopper/**", "tests/boundary/providers/hopper/**"], + ida: ["src/ida/**", "tests/boundary/providers/ida/**"], + jeb: ["src/jeb/**"], + pcap: ["src/pcap/**", "tests/boundary/pcap/**"], + sqlite: ["src/sqlite/**", "tests/boundary/sqlite/**"], + wasm: ["src/wasm/**"], +}; +const ownersOf = (path) => + Object.entries(OWNERS) + .filter(([, patterns]) => patterns.some((glob) => matchesGlob(path, glob))) + .map(([owner]) => owner); +const validPath = (path) => + typeof path === "string" && + /^(src|tests)\//.test(path) && + !/[\\\0\r\n]/.test(path) && + !path.split("/").some((part) => ["", ".", ".."].includes(part)); +const full = (reason, changedPaths = []) => ({ + version: 1, + mode: "full", + owners: [], + changedTests: [], + changedPaths, + reason, +}); + +/** Plan an observational PR subset from the classifier's existing Git diff. */ +export function planTestSelection(files, event, tier) { + if (event !== "pull_request" || tier !== "fast") + return full("Only ordinary PRs are candidates", files); + if (!files.length) return full("No changed paths available"); + const owners = new Set(); + const changedTests = []; + for (const path of files) { + if (!validPath(path)) + return full(`Shared or unknown input: ${path}`, files); + const matches = ownersOf(path); + if (!matches.length) return full(`Shared or unknown input: ${path}`, files); + for (const owner of matches) owners.add(owner); + if (path.endsWith(".test.ts")) changedTests.push(path); + } + return { + version: 1, + mode: "candidate", + owners: [...owners].sort(), + changedTests: [...new Set(changedTests)].sort(), + changedPaths: files, + reason: + "Observe affected leaf owners; retain integration and unowned tests", + }; +} + +/** Compare a proposal with the full run; missing observations never prove parity. */ +export function compareTestSelection(plan, inventory, results) { + if ( + plan?.version !== 1 || + !["full", "candidate"].includes(plan.mode) || + !Array.isArray(plan.owners) || + plan.owners.some((owner) => !Object.hasOwn(OWNERS, owner)) || + !Array.isArray(plan.changedTests) || + !Array.isArray(plan.changedPaths) || + plan.changedPaths.some((path) => typeof path !== "string") || + plan.changedTests.some( + (path) => !validPath(path) || !path.endsWith(".test.ts"), + ) || + typeof plan.reason !== "string" || + (plan.mode === "candidate" && !plan.owners.length) || + (plan.mode === "full" && (plan.owners.length || plan.changedTests.length)) + ) + throw new Error("Invalid observational test selection plan"); + const expected = new Set(inventory); + if (!expected.size || expected.size !== inventory.length) + throw new Error("Test selection requires a nonempty unique inventory"); + const ownersByFile = new Map( + plan.mode === "candidate" + ? inventory.map((path) => [path, ownersOf(path)]) + : [], + ); + // Removed/renamed tests have no current executable identity. Keep all tests. + const removed = plan.changedTests.filter((path) => !expected.has(path)); + const missingOwners = plan.owners.filter( + (owner) => + ![...ownersByFile.values()].some((owners) => owners.includes(owner)), + ); + const effective = removed.length + ? full( + `Changed tests absent from current inventory: ${removed.join(", ")}`, + plan.changedPaths, + ) + : missingOwners.length + ? full( + `Owners without executable tests: ${missingOwners.join(", ")}`, + plan.changedPaths, + ) + : plan; + const selected = []; + const omitted = []; + for (const path of inventory.toSorted()) { + const owners = ownersByFile.get(path) ?? []; + const keep = + effective.mode === "full" || + !owners.length || + effective.changedTests.includes(path) || + owners.some((owner) => effective.owners.includes(owner)); + (keep ? selected : omitted).push(path); + } + const actual = new Set(results.map((file) => file.name)); + const missing = inventory.filter((path) => !actual.has(path)); + const unexpected = [...actual].filter((path) => !expected.has(path)); + const duplicateFiles = results.length - actual.size; + const unobserved = results + .filter((file) => !["passed", "failed"].includes(file.state)) + .map((file) => file.name); + const omittedSet = new Set(omitted); + const missedFailures = results + .filter((file) => omittedSet.has(file.name) && file.state === "failed") + .map((file) => file.name); + return { + version: 1, + observational: true, + plan: effective, + selected, + omitted, + missing, + unexpected, + duplicateFiles, + unobserved, + missedFailures, + complete: + !missing.length && + !unexpected.length && + !duplicateFiles && + !unobserved.length, + // Phase totals are scheduling weights, not an estimate of wall-time savings. + omittedDurationMs: results + .filter((file) => omittedSet.has(file.name)) + .reduce((sum, file) => sum + file.durationMs, 0), + }; +} diff --git a/scripts/ci/test-timing-reporter.mjs b/scripts/ci/test-timing-reporter.mjs new file mode 100644 index 000000000..ce0b1c5dc --- /dev/null +++ b/scripts/ci/test-timing-reporter.mjs @@ -0,0 +1,43 @@ +import { writeFile } from "node:fs/promises"; + +/** @implements {import("vitest/node").Reporter} */ +export default class TestTimingReporter { + /** + * @param {ReadonlyArray} modules + * @param {ReadonlyArray} errors + * @param {import("vitest/node").TestRunEndReason} reason + */ + async onTestRunEnd(modules, errors, reason) { + const files = modules.map((module) => { + const diagnostic = module.diagnostic(); + // These disjoint phases match Vitest's per-file wall-time accounting. + // Import timings already include transform waits; do not add them again. + const phases = { + execution: diagnostic.duration, + preparation: diagnostic.prepareDuration, + environment: diagnostic.environmentSetupDuration, + imports: diagnostic.collectDuration, + setup: diagnostic.setupDuration, + }; + return { + name: module.moduleId, + state: module.state(), + durationMs: Object.values(phases).reduce( + (total, value) => total + value, + 0, + ), + phases, + }; + }); + await writeFile( + "test-timings.json", + JSON.stringify({ + version: 1, + success: reason === "passed" && errors.length === 0, + reason, + unhandledErrors: errors.length, + files, + }), + ); + } +} diff --git a/scripts/ci/verified-package.mjs b/scripts/ci/verified-package.mjs new file mode 100644 index 000000000..1053760b2 --- /dev/null +++ b/scripts/ci/verified-package.mjs @@ -0,0 +1,235 @@ +import { createHash } from "node:crypto"; +import { createReadStream } from "node:fs"; +import { execFile } from "node:child_process"; +import { copyFile, readFile, writeFile } from "node:fs/promises"; +import { join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { promisify } from "node:util"; + +const MANIFEST_NAME = "manifest.json"; +const TARBALL_NAME = "package.tgz"; +const MANIFEST_KEYS = [ + "schemaVersion", + "sourceSha", + "runId", + "runAttempt", + "platform", + "architecture", + "packageVersion", + "nodeVersion", + "sha256", +].sort(); +const execFileAsync = promisify(execFile); + +/** Normalize workflow identity and host facts into the manifest-owned fields. */ +export function createVerifiedPackageMetadata({ + sourceSha, + runId, + runAttempt, + platform, + architecture, + packageVersion, + nodeVersion, +}) { + if (!/^[a-f0-9]{40}$/u.test(sourceSha)) + throw new Error("Source SHA must be a full commit SHA"); + return { + sourceSha, + runId: parsePositiveInteger(runId, "GITHUB_RUN_ID"), + runAttempt: parsePositiveInteger(runAttempt, "GITHUB_RUN_ATTEMPT"), + platform, + architecture, + packageVersion, + nodeVersion, + }; +} + +/** Build the single supported manifest representation for a verified tarball. */ +function createVerifiedPackageManifest(metadata) { + const manifest = { schemaVersion: 1, ...metadata }; + validateManifestShape(manifest); + return manifest; +} + +/** Write a manifest after the caller has completed qualification and cleanup. */ +export async function writeVerifiedPackageManifest(directory, metadata) { + const manifest = createVerifiedPackageManifest(metadata); + await writeFile( + join(directory, MANIFEST_NAME), + `${JSON.stringify(manifest, null, 2)}\n`, + { encoding: "utf8" }, + ); + return manifest; +} + +/** Copy the qualified archive under its fixed name and prepare its manifest. */ +export async function retainVerifiedPackage({ + sourceTarball, + directory, + metadata, +}) { + const tarball = join(directory, TARBALL_NAME); + await copyFile(sourceTarball, tarball); + const sha256 = await sha256File(tarball); + return { + directory, + metadata: createVerifiedPackageManifest({ ...metadata, sha256 }), + }; +} + +/** Validate a retained package against the current checkout and workflow attempt. */ +async function validateVerifiedPackage({ + directory, + sourceSha, + runId, + runAttempt, + packageVersion, + nodeVersion, +}) { + if (!/^[a-f0-9]{40}$/u.test(sourceSha)) + throw new Error("Expected source SHA must be a full commit SHA"); + if (!Number.isSafeInteger(runId) || runId <= 0) + throw new Error("Expected workflow run ID must be a positive integer"); + if (!Number.isSafeInteger(runAttempt) || runAttempt <= 0) + throw new Error("Expected workflow run attempt must be a positive integer"); + const manifestPath = join(directory, MANIFEST_NAME); + let manifest; + try { + manifest = JSON.parse(await readFile(manifestPath, "utf8")); + } catch (cause) { + throw new Error(`Could not read verified package manifest: ${cause}`, { + cause, + }); + } + validateManifestShape(manifest); + assertEqual(manifest.sourceSha, sourceSha, "source SHA"); + assertEqual(manifest.runId, runId, "workflow run ID"); + // A successful producer may belong to an earlier attempt when only failed + // publisher jobs are rerun. The workflow still requires same-run qualification. + if (manifest.runAttempt > runAttempt) + throw new Error("Verified package workflow run attempt is from the future"); + assertEqual(manifest.platform, "linux", "verifier platform"); + assertEqual(manifest.architecture, "x64", "verifier architecture"); + assertEqual(manifest.packageVersion, packageVersion, "package version"); + assertEqual(manifest.nodeVersion, nodeVersion, "Node.js version"); + + // The artifact format deliberately fixes the name; no manifest value can + // redirect validation to another path. + const tarball = resolve(directory, TARBALL_NAME); + const digest = await sha256File(tarball); + assertEqual(manifest.sha256, digest, "tarball SHA-256"); + return tarball; +} + +function validateManifestShape(manifest) { + if ( + typeof manifest !== "object" || + manifest === null || + Array.isArray(manifest) + ) + throw new Error("Verified package manifest must be a JSON object"); + const keys = Object.keys(manifest).sort(); + if ( + keys.length !== MANIFEST_KEYS.length || + keys.some((key, index) => key !== MANIFEST_KEYS[index]) + ) + throw new Error("Verified package manifest has missing or unknown fields"); + if (manifest.schemaVersion !== 1) + throw new Error("Unsupported verified package manifest schema version"); + if ( + typeof manifest.sourceSha !== "string" || + !/^[a-f0-9]{40}$/u.test(manifest.sourceSha) + ) + throw new Error("Verified package manifest has an invalid source SHA"); + if (!Number.isSafeInteger(manifest.runId) || manifest.runId <= 0) + throw new Error("Verified package manifest has an invalid workflow run ID"); + if (!Number.isSafeInteger(manifest.runAttempt) || manifest.runAttempt <= 0) + throw new Error( + "Verified package manifest has an invalid workflow run attempt", + ); + for (const key of [ + "platform", + "architecture", + "packageVersion", + "nodeVersion", + ]) + if (typeof manifest[key] !== "string" || manifest[key].length === 0) + throw new Error(`Verified package manifest has an invalid ${key}`); + if ( + typeof manifest.sha256 !== "string" || + !/^[a-f0-9]{64}$/u.test(manifest.sha256) + ) + throw new Error("Verified package manifest has an invalid tarball SHA-256"); +} + +function assertEqual(actual, expected, label) { + if (actual !== expected) + throw new Error(`Verified package ${label} mismatch`); +} + +async function sha256File(path) { + const hash = createHash("sha256"); + for await (const chunk of createReadStream(path)) hash.update(chunk); + return hash.digest("hex"); +} + +async function validateFromCli(directory) { + const sourceSha = process.env.GITHUB_SHA; + const runId = parsePositiveInteger( + process.env.GITHUB_RUN_ID, + "GITHUB_RUN_ID", + ); + const runAttempt = parsePositiveInteger( + process.env.GITHUB_RUN_ATTEMPT, + "GITHUB_RUN_ATTEMPT", + ); + if (sourceSha === undefined || !/^[a-f0-9]{40}$/u.test(sourceSha)) + throw new Error("GITHUB_SHA must be a full commit SHA"); + const checkoutSha = ( + await execFileAsync("git", ["rev-parse", "HEAD"]) + ).stdout.trim(); + assertEqual(checkoutSha, sourceSha, "checked out source SHA"); + const packageJson = JSON.parse(await readFile("package.json", "utf8")); + const tarball = await validateVerifiedPackage({ + directory, + sourceSha, + runId, + runAttempt, + packageVersion: packageJson.version, + nodeVersion: process.version, + }); + if (process.env.GITHUB_OUTPUT === undefined) + throw new Error( + "GITHUB_OUTPUT is required to publish the validated tarball path", + ); + await writeFile(process.env.GITHUB_OUTPUT, `tarball=${tarball}\n`, { + encoding: "utf8", + flag: "a", + }); + process.stdout.write(`${tarball}\n`); +} + +function parsePositiveInteger(value, name) { + if (value === undefined || !/^[1-9]\d*$/u.test(value)) + throw new Error(`${name} must be a positive integer`); + const parsed = Number(value); + if (!Number.isSafeInteger(parsed)) + throw new Error(`${name} must be a safe positive integer`); + return parsed; +} + +const invokedPath = process.argv[1]; +if ( + invokedPath !== undefined && + resolve(invokedPath) === fileURLToPath(import.meta.url) +) { + const [command, directory, ...extra] = process.argv.slice(2); + if (command !== "validate" || directory === undefined || extra.length > 0) { + process.stderr.write( + "Usage: node scripts/ci/verified-package.mjs validate \n", + ); + process.exitCode = 2; + } else { + await validateFromCli(directory); + } +} diff --git a/scripts/fixtures/generate-mitmproxy-capture.py b/scripts/fixtures/generate-mitmproxy-capture.py index 4be3dd4a7..880c5e690 100644 --- a/scripts/fixtures/generate-mitmproxy-capture.py +++ b/scripts/fixtures/generate-mitmproxy-capture.py @@ -45,6 +45,20 @@ class Generator: assert "ows-secret" not in json.dumps(projected), projected if isinstance(value, bytes): assert projected["binary_fields"][-1]["state"] == "redacted", projected + body = b"A" * 12345 + state = {"response": {"content": body}} + control = adapter.project_record(state, []) + field = control["binary_fields"][0] + assert field["state"] == "retained" and field["bytes"] == len(body), control + for literal in ["12345", "2345"]: + # Only the generated byte count matches this exclusion. + assert literal not in control["reported"]["response"]["content"] + assert literal not in field["content_base64"] and literal not in field["sha256"] + projected = adapter.project_record(state, [literal]) + assert projected["reported"]["response"]["content"] is None, projected + assert projected["binary_fields"] == [{"pointer": "/response/content", "representation": "producer-bytes", "state": "redacted", "content_base64": None, "bytes": None, "sha256": None}], projected + assert {"pointer": "/response/content", "reason": "explicit-sensitive-value"} in projected["redactions"], projected + assert literal not in json.dumps(projected), projected (root / "native-error-classifications.json").write_text(json.dumps({"passed": len(classifications) + 2})) first = http.HTTPFlow(connection.Client(peername=("127.0.0.1", 1), sockname=("127.0.0.1", 2)), connection.Server(address=("example.test", 80))) first.id = "original-producer-id" diff --git a/scripts/generate-nativeaot-fixtures.mjs b/scripts/generate-nativeaot-fixtures.mjs index 84a8ac533..fa9127cf3 100644 --- a/scripts/generate-nativeaot-fixtures.mjs +++ b/scripts/generate-nativeaot-fixtures.mjs @@ -4,9 +4,14 @@ import { createHash } from "node:crypto"; import { copyFile, mkdir, readFile, writeFile } from "node:fs/promises"; import { dirname, join, resolve } from "node:path"; import { fileURLToPath } from "node:url"; +import { + buildNativeAotFixture, + nativeAotCommandError, + supportsNativeAotFixtureHost, +} from "./lib/nativeaot-build.mjs"; const root = resolve(dirname(fileURLToPath(import.meta.url)), ".."); -if (process.arch !== "x64" || !["linux", "win32"].includes(process.platform)) +if (!supportsNativeAotFixtureHost(process.platform, process.arch)) throw new Error( "NativeAOT fixture lane requires Linux or Windows x64; it does not cross-compile NativeAOT.", ); @@ -40,9 +45,7 @@ const run = (command, args) => { maxBuffer: 8 * 1024 * 1024, }); if (result.error || result.status !== 0) - throw new Error( - `NativeAOT fixture lane requires/action failed ${command}: ${result.error?.message ?? ""}\n${result.stdout ?? ""}${result.stderr ?? ""}`, - ); + throw nativeAotCommandError(command, args[0], result); return result.stdout; }; const dotnet = process.env.REA_NATIVEAOT_DOTNET ?? "dotnet"; @@ -57,20 +60,12 @@ if (process.platform === "linux") { run("clang", ["--version"]); } const map = join(output, "NativeAotFixture.map"); -run(dotnet, [ - "publish", - join(root, "tests/fixtures/nativeaot/NativeAotFixture.csproj"), - "-r", +await buildNativeAotFixture(run, { + dotnet, + project: join(root, "tests/fixtures/nativeaot/NativeAotFixture.csproj"), runtime, - "-c", - "Release", - "-o", - join(output, "symbols"), - "--maxcpucount:1", - `-p:BaseIntermediateOutputPath=${join(output, "obj")}/`, - `-p:BaseOutputPath=${join(output, "bin")}/`, - `-p:NativeAotMapPath=${map}`, -]); + output, +}); const name = process.platform === "win32" ? "NativeAotFixture.exe" : "NativeAotFixture"; const binary = join(output, "symbols", name); diff --git a/scripts/generate-third-party-inventory.mjs b/scripts/generate-third-party-inventory.mjs new file mode 100644 index 000000000..57934ebda --- /dev/null +++ b/scripts/generate-third-party-inventory.mjs @@ -0,0 +1,120 @@ +import { createHash } from "node:crypto"; +import { readFile } from "node:fs/promises"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +import { ensureGeneratedFile } from "./lib/generated-file.mjs"; + +const arguments_ = new Set(process.argv.slice(2)); +for (const argument of arguments_) + if (argument !== "--check") + throw new Error(`Unknown inventory option: ${argument}`); +const root = join(dirname(fileURLToPath(import.meta.url)), ".."); +const lock = JSON.parse( + await readFile(join(root, "package-lock.json"), "utf8"), +); +const entries = Object.entries(lock.packages) + .filter(([path]) => path !== "") + .map(([path, record]) => ({ + path, + name: record.name ?? path.slice(path.lastIndexOf("node_modules/") + 13), + version: record.version, + license: record.license ?? "UNDECLARED", + resolved: record.resolved, + integrity: record.integrity, + dev: record.dev === true, + optional: record.optional === true, + })) + .sort((left, right) => + left.path < right.path ? -1 : left.path > right.path ? 1 : 0, + ); +for (const entry of entries) + for (const field of ["name", "version", "resolved", "integrity"]) + if (typeof entry[field] !== "string" || entry[field].length === 0) + throw new Error(`Missing locked ${field}: ${entry.path}`); +const runtime = entries.filter(({ dev }) => !dev); +const runtimeFingerprint = createHash("sha256") + .update(JSON.stringify(runtime)) + .digest("hex"); +const source = [ + "REA npm dependency inventory", + "Prepared from the root package-lock.json; refresh this inventory when locked dependencies change.", + "License identifiers are upstream lockfile declarations, not a legal assessment.", + "Runtime includes transitive and optional platform packages; development includes build, test and website tooling.", + `Locked package entries: ${entries.length}; runtime: ${runtime.length}.`, + "Full runtime copyright/license/NOTICE texts: npm-runtime-notices.txt.", + "", + ...entries.flatMap((entry) => [ + `${entry.name}@${entry.version} | ${entry.dev ? "development" : "runtime"}${entry.optional ? ", optional" : ""} | ${entry.license}`, + ` Lockfile path: ${entry.path}`, + ` Package: https://www.npmjs.com/package/${entry.name}/v/${entry.version}`, + ` Archive: ${entry.resolved}`, + ` Integrity: ${entry.integrity}`, + ]), + "", +].join("\n"); +await ensureGeneratedFile({ + path: join(root, "licenses/npm-dependencies.txt"), + source, + check: arguments_.has("--check"), + generateCommand: "npm run licenses:generate", +}); + +if (arguments_.has("--check")) { + const notices = await readFile( + join(root, "licenses/npm-runtime-notices.txt"), + "utf8", + ); + const headings = [...notices.matchAll(/^===== (.+) =====$/gmu)]; + const sections = new Map(); + for (const [index, heading] of headings.entries()) { + if (sections.has(heading[1])) + throw new Error(`Duplicate runtime notice: ${heading[1]}`); + sections.set( + heading[1], + notices.slice( + heading.index + heading[0].length + 1, + headings[index + 1]?.index, + ), + ); + } + for (const entry of runtime) { + const identity = `${entry.name}@${entry.version}`; + const section = sections.get(identity); + if (section === undefined) + throw new Error(`Missing runtime notice: ${identity}`); + const metadata = [ + `Declared license: ${entry.license}`, + `Archive: ${entry.resolved}`, + `Integrity: ${entry.integrity}`, + "", + ].join("\n"); + if (!section.startsWith(metadata)) + throw new Error(`Runtime notice metadata differs for ${identity}`); + } + if ( + !notices + .slice(0, headings[0]?.index) + .includes( + `Runtime dependency fingerprint (SHA-256): ${runtimeFingerprint}\n`, + ) + ) + throw new Error( + `Runtime notice fingerprint differs; refresh verified notices for ${runtimeFingerprint}`, + ); + const references = JSON.parse( + await readFile(join(root, "licenses/upstream-sources.json"), "utf8"), + ); + for (const reference of references) { + const bytes = await readFile(join(root, reference.file)); + if (createHash("sha256").update(bytes).digest("hex") !== reference.sha256) + throw new Error(`Preserved notice digest differs: ${reference.file}`); + } +} +console.log( + JSON.stringify({ + locked_packages: entries.length, + runtime_packages: runtime.length, + runtime_dependency_fingerprint: runtimeFingerprint, + }), +); diff --git a/scripts/hopper-demo-x11.py b/scripts/hopper-demo-x11.py index 470c9fd28..da2fe884f 100644 --- a/scripts/hopper-demo-x11.py +++ b/scripts/hopper-demo-x11.py @@ -10,6 +10,7 @@ import json import os from pathlib import Path import secrets +import select import stat import subprocess import tempfile @@ -118,6 +119,7 @@ class XWindowAttributes(ctypes.Structure): def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser() parser.add_argument("--probe", action="store_true") + parser.add_argument("--owner-stdin", action="store_true") parser.add_argument("--hopper") parser.add_argument("--socket") parser.add_argument( @@ -133,6 +135,43 @@ def parse_args() -> argparse.Namespace: return parser.parse_args() +class OwnerLifetime: + """The launcher's private pipe is a lifetime capability, independent of PIDs.""" + + def __init__(self, enabled: bool) -> None: + self._poll = select.poll() if enabled else None + if self._poll is not None: + # Node/libuv can implement piped stdio with a Unix socket pair. + mode = os.fstat(0).st_mode + if not (stat.S_ISFIFO(mode) or stat.S_ISSOCK(mode)): + raise AdapterFailure( + INVALID_LAUNCH_COMMAND, + "invalid_launch_command", + "owner lifetime requires a private stdin pipe", + ) + self._poll.register(0, select.POLLIN | select.POLLHUP | select.POLLERR) + + def alive(self) -> bool: + if self._poll is None: + return True + for _, events in self._poll.poll(0): + if events & (select.POLLHUP | select.POLLERR | select.POLLNVAL): + return False + if events & select.POLLIN and not os.read(0, 4096): + return False + return True + + +def wait_for_hopper(child: subprocess.Popen[bytes], owner: OwnerLifetime) -> int: + while owner.alive(): + try: + return child.wait(timeout=0.1) + except subprocess.TimeoutExpired: + continue + # run_launch's finally owns cleanup of both direct, unreaped children. + return 0 + + def digest(path: Path) -> str: value = hashlib.sha256() with path.open("rb") as file: @@ -572,6 +611,7 @@ def start_xvfb( "-auth", str(authority), ], + stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, ) @@ -627,6 +667,9 @@ def run_launch(args: argparse.Namespace) -> int: "invalid_launch_command", "missing Hopper launch coordinates", ) + owner = OwnerLifetime(args.owner_stdin) + if not owner.alive(): + return 0 hopper = Path(args.hopper) socket = Path(args.socket) try: @@ -654,6 +697,8 @@ def run_launch(args: argparse.Namespace) -> int: child: subprocess.Popen[bytes] | None = None display: int | None = None try: + if not owner.alive(): + return 0 hopper_environment = dict(os.environ) for name in ("WAYLAND_DISPLAY", "WAYLAND_SOCKET"): hopper_environment.pop(name, None) @@ -666,7 +711,10 @@ def run_launch(args: argparse.Namespace) -> int: "XDG_SESSION_TYPE": "x11", } ) - child = subprocess.Popen(command, env=hopper_environment) + # Children must not consume or retain REA's lifetime pipe. + child = subprocess.Popen( + command, env=hopper_environment, stdin=subprocess.DEVNULL + ) leader = os.getpid() os.environ["XAUTHORITY"] = str(authority) @@ -702,10 +750,12 @@ def run_launch(args: argparse.Namespace) -> int: observed: list[tuple[int, int, int, int, int]] = [] dialog: tuple[int, int, int, int, int] | None = None while time.monotonic() < deadline: + if not owner.alive(): + return 0 if socket.exists(): x11.XCloseDisplay(display) display = None - return child.wait() + return wait_for_hopper(child, owner) if not hopper_is_owned(leader, hopper): if child.poll() is None: time.sleep(0.1) @@ -749,7 +799,7 @@ def run_launch(args: argparse.Namespace) -> int: x11.XFlush(display) x11.XCloseDisplay(display) display = None - return child.wait() + return wait_for_hopper(child, owner) except AdapterFailure as error: raise error.with_stderr(capture) finally: diff --git a/scripts/lib/catalog-builders.mjs b/scripts/lib/catalog-builders.mjs index 1658c08fa..32ec51ee3 100644 --- a/scripts/lib/catalog-builders.mjs +++ b/scripts/lib/catalog-builders.mjs @@ -52,6 +52,11 @@ export const toolFamilyCatalog = (sources) => { surface: "android-provider", contracts: sources.androidContracts.ANDROID_TOOL_CONTRACTS, }, + { + id: "pcap", + surface: "pcap-provider", + contracts: sources.pcapContracts.PCAP_TOOL_CONTRACTS, + }, { id: "flutter", surface: "flutter-provider", @@ -251,6 +256,10 @@ export const providerCatalog = (sources) => { identity: sources.androidProvider.JADX_PROVIDER_IDENTITY, contracts: sources.androidContracts.ANDROID_TOOL_CONTRACTS, }, + { + identity: sources.pcapProvider.PCAP_PROVIDER_IDENTITY, + contracts: sources.pcapContracts.PCAP_TOOL_CONTRACTS, + }, { identity: sources.flutterProvider.FLUTTER_PROVIDER_IDENTITY, contracts: sources.flutterContracts.FLUTTER_TOOL_CONTRACTS, @@ -272,6 +281,11 @@ export const providerCatalog = (sources) => { contracts: sources.webNetworkCaptureContracts.WEB_NETWORK_CAPTURE_TOOL_CONTRACTS, }, + { + identity: sources.webNetworkCaptureProvider.SAZ_CAPTURE_PROVIDER_IDENTITY, + contracts: + sources.webNetworkCaptureContracts.WEB_NETWORK_CAPTURE_TOOL_CONTRACTS, + }, { identity: sources.webNetworkCaptureProvider.MITMPROXY_CAPTURE_PROVIDER_IDENTITY, diff --git a/scripts/lib/catalog-core.mjs b/scripts/lib/catalog-core.mjs index 5a3b0e909..f5488a9e6 100644 --- a/scripts/lib/catalog-core.mjs +++ b/scripts/lib/catalog-core.mjs @@ -68,8 +68,8 @@ const SOURCE_PATHS = { flutterProvider: "dist/flutter/FlutterBuildProvider.js", androidProvider: "dist/android/JadxRelease.js", apktoolContracts: "dist/contracts/apktool/apktoolToolContracts.js", - flutterContracts: "dist/contracts/flutter/flutterToolContracts.js", - flutterProvider: "dist/flutter/FlutterBuildProvider.js", + pcapContracts: "dist/contracts/pcap/pcapToolContracts.js", + pcapProvider: "dist/pcap/PcapProvider.js", apktoolProvider: "dist/apktool/ApktoolProvider.js", jebContracts: "dist/contracts/jeb/jebToolContracts.js", jebProvider: "dist/jeb/JebProvider.js", diff --git a/scripts/lib/clean-linux-hopper-runtime.d.mts b/scripts/lib/clean-linux-hopper-runtime.d.mts new file mode 100644 index 000000000..dd1010aa1 --- /dev/null +++ b/scripts/lib/clean-linux-hopper-runtime.d.mts @@ -0,0 +1,3 @@ +export function assertCleanLinuxHopperRuntime( + launcherPaths?: readonly string[], +): Promise; diff --git a/scripts/lib/clean-linux-hopper-runtime.mjs b/scripts/lib/clean-linux-hopper-runtime.mjs new file mode 100644 index 000000000..27a306fce --- /dev/null +++ b/scripts/lib/clean-linux-hopper-runtime.mjs @@ -0,0 +1,64 @@ +import { execFile } from "node:child_process"; +import { promisify } from "node:util"; +import { readdir, stat } from "node:fs/promises"; + +const execFileAsync = promisify(execFile); + +/** + * Observe leftover demo processes on a dedicated clean Linux runner, including + * groups reparented after their launcher exits. This is not ownership proof and + * never authorizes signaling; unrelated runtime processes also block this lane. + */ +export const assertCleanLinuxHopperRuntime = async ( + launcherPaths = [ + "/opt/hopper/bin/Hopper", + ...(process.env.HOPPER_LAUNCHER_PATH === undefined + ? [] + : [process.env.HOPPER_LAUNCHER_PATH]), + ], +) => { + const { stdout } = await execFileAsync("ps", ["-ax", "-o", "command="]); + if (/\/opt\/hopper\/bin\/Hopper|\bXvfb\b|hopper-demo-x11\.py/u.test(stdout)) + throw new Error("Clean Linux runner retained a Hopper demo process"); + const launchers = []; + for (const path of launcherPaths) { + try { + launchers.push(await stat(path, { bigint: true })); + } catch (error) { + if (error?.code !== "ENOENT") throw error; + } + } + if (launchers.length === 0) return; + // /proc exposes the executed file even when argv[0] names a removed symlink. + // Compare file identity, not a display path or a process name. + let inspectionError; + for (const entry of await readdir("/proc", { withFileTypes: true })) { + if (!entry.isDirectory() || !/^\d+$/u.test(entry.name)) continue; + let executable; + try { + executable = await stat(`/proc/${entry.name}/exe`, { bigint: true }); + } catch (error) { + if (error?.code === "ENOENT" || error?.code === "ESRCH") continue; + // Other users' executables may be unreadable. The setup lane launches + // providers as the caller; unreadable caller-owned processes fail closed. + if (error?.code === "EACCES") { + const owner = await stat(`/proc/${entry.name}`).catch((cause) => { + if (cause?.code === "ENOENT") return undefined; + throw cause; + }); + if (owner === undefined || owner.uid !== process.getuid()) continue; + } + // Finish the inventory so an unrelated restricted process cannot mask a + // positively identified leak. Unknown caller processes still fail closed. + inspectionError ??= error; + continue; + } + if ( + launchers.some( + ({ dev, ino }) => dev === executable.dev && ino === executable.ino, + ) + ) + throw new Error("Clean Linux runner retained a Hopper demo process"); + } + if (inspectionError !== undefined) throw inspectionError; +}; diff --git a/scripts/lib/ida-feature-trace.d.mts b/scripts/lib/ida-feature-trace.d.mts new file mode 100644 index 000000000..13c84df27 --- /dev/null +++ b/scripts/lib/ida-feature-trace.d.mts @@ -0,0 +1,9 @@ +export function assertFeatureTrace( + trace: unknown, + procedure: { readonly address: string }, +): void; + +export function findMappedNonFunction( + strings: readonly { readonly address: string }[], + resolveAt: (address: string) => Promise, +): Promise; diff --git a/scripts/lib/ida-feature-trace.mjs b/scripts/lib/ida-feature-trace.mjs new file mode 100644 index 000000000..3c4778d25 --- /dev/null +++ b/scripts/lib/ida-feature-trace.mjs @@ -0,0 +1,52 @@ +import assert from "node:assert/strict"; + +const NEGATIVE_REASONS = new Set(["not_in_procedure", "outside_segments"]); + +/** + * Assert the `trace_feature` result of the real IDA lane for one selected + * procedure. The query is a substring search, so the trace may also match other + * procedures and strings; each reference is checked against its own matched + * target, and a source outside every function (a data reference) is valid. + */ +export function assertFeatureTrace(trace, procedure) { + assert.ok( + trace.matches.some( + ({ type, address }) => + type === "procedure" && address === procedure.address, + ), + "The trace omitted the selected procedure's match.", + ); + assert.equal(typeof trace.truncated, "boolean"); + const matched = new Set(trace.matches.map(({ address }) => address)); + for (const reference of trace.references) { + assert.ok( + matched.has(reference.target_address), + `A reference targets ${reference.target_address}, which is not a match.`, + ); + const { containing_procedure: containing } = reference; + if (containing.found) + assert.equal(typeof containing.procedure.address, "string"); + else assert.ok(NEGATIVE_REASONS.has(containing.reason)); + } +} + +/** + * Find a string that IDA maps outside every function. A string inside a + * function chunk (inline data in code) is valid and skipped, so the strings are + * tried in order. Any other answer must be "not in a procedure": IDA lists only + * mapped strings, so an unmapped answer is a containment defect and fails the + * lane. Returns the address, or undefined when every string is inside a function. + */ +export async function findMappedNonFunction(strings, resolveAt) { + for (const { address } of strings) { + const resolved = await resolveAt(address); + if (resolved.found) continue; + assert.equal( + resolved.reason, + "not_in_procedure", + `The listed string at ${address} resolved as ${resolved.reason}.`, + ); + return address; + } + return undefined; +} diff --git a/scripts/lib/nativeaot-build.d.mts b/scripts/lib/nativeaot-build.d.mts new file mode 100644 index 000000000..418111852 --- /dev/null +++ b/scripts/lib/nativeaot-build.d.mts @@ -0,0 +1,25 @@ +export function supportsNativeAotFixtureHost( + platform: string, + architecture: string, +): boolean; + +export function buildNativeAotFixture( + run: (command: string, args: string[]) => unknown, + options: { + dotnet: string; + project: string; + runtime: string; + output: string; + }, +): Promise; + +export function nativeAotCommandError( + command: string, + operation: string, + result: { + error?: { message: string }; + status: number | null; + stdout?: string | null; + stderr?: string | null; + }, +): Error; diff --git a/scripts/lib/nativeaot-build.mjs b/scripts/lib/nativeaot-build.mjs new file mode 100644 index 000000000..3b49ce839 --- /dev/null +++ b/scripts/lib/nativeaot-build.mjs @@ -0,0 +1,60 @@ +import { join } from "node:path"; +import { setTimeout as delay } from "node:timers/promises"; + +/** NativeAOT fixtures use native toolchains, not cross-compilation. */ +export function supportsNativeAotFixtureHost(platform, architecture) { + return architecture === "x64" && ["linux", "win32"].includes(platform); +} + +/** Keep network recovery separate from compilation and preserve restore failures. */ +export async function buildNativeAotFixture(run, options) { + const { dotnet, project, runtime, output } = options; + const common = [ + project, + "-r", + runtime, + "--maxcpucount:1", + "-p:Configuration=Release", + `-p:BaseIntermediateOutputPath=${join(output, "obj")}/`, + `-p:BaseOutputPath=${join(output, "bin")}/`, + `-p:NativeAotMapPath=${join(output, "NativeAotFixture.map")}`, + ]; + for (let attempt = 1; attempt <= 2; attempt++) { + console.error(`NativeAOT dependency restore: attempt ${attempt}/2`); + try { + run(dotnet, ["restore", ...common, "--verbosity", "normal"]); + break; + } catch (cause) { + if (attempt === 2) + throw new Error( + "NativeAOT dependency restore failed after two attempts", + { + cause, + }, + ); + // Child diagnostics can contain authenticated feed URLs; keep retry logs static. + console.error( + "NativeAOT dependency restore failed; retrying in 1 second", + ); + await delay(1_000); + } + } + console.error("NativeAOT compilation (dependencies already restored)"); + run(dotnet, [ + "publish", + ...common, + "--no-restore", + "-o", + join(output, "symbols"), + ]); +} + +/** Sanitize child diagnostics before any retry, cause chain, or uncaught output. */ +export function nativeAotCommandError(command, operation, result) { + const diagnostic = `NativeAOT fixture lane requires/action failed ${command} ${operation}: ${result.error?.message ?? `exit ${result.status}`}\n${result.stdout ?? ""}${result.stderr ?? ""}`; + // Only URL authority userinfo is authentication; preserve host, path, query, + // and unrelated diagnostic values. Never retain the raw result as a cause. + return new Error( + diagnostic.replace(/\b(https?:\/\/)[^\\\s/?#]*@/giu, "$1[redacted]@"), + ); +} diff --git a/scripts/lib/real-ghidra-file-offsets.mjs b/scripts/lib/real-ghidra-file-offsets.mjs new file mode 100644 index 000000000..ea881dd04 --- /dev/null +++ b/scripts/lib/real-ghidra-file-offsets.mjs @@ -0,0 +1,264 @@ +import assert from "node:assert/strict"; +import { execFile } from "node:child_process"; +import { + mkdir, + mkdtemp, + readFile, + readdir, + rm, + writeFile, +} from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { promisify } from "node:util"; +import { Client } from "@modelcontextprotocol/client"; +import { StdioClientTransport } from "@modelcontextprotocol/client/stdio"; +import { parseEvidence } from "../../dist/domain/evidence.js"; +import { + requireMcpOperationResult, + requireMcpToolError, + mcpTextValue, +} from "./mcp-verifier-results.mjs"; + +/** Exercise file coordinates through public CLI/MCP using the source-owned PE fixture. */ +export async function verifyGhidraFileOffsets( + targetPath, + entrypoint, + procedure, +) { + const original = await readFile(targetPath); + const workspace = await mkdtemp(join(tmpdir(), "rea-file-offsets-")); + const runtime = join(workspace, "runtime"); + await mkdir(runtime); + const env = { + ...process.env, + TEMP: runtime, + TMP: runtime, + TMPDIR: runtime, + REA_ANALYSIS_PROVIDER: "ghidra", + REA_LOG_LEVEL: "silent", + }; + const client = new Client({ name: "rea-file-offsets", version: "1" }); + const transport = new StdioClientTransport({ + command: process.execPath, + args: [entrypoint, "mcp"], + env, + stderr: "pipe", + }); + let connected = false; + let complete = false; + let primaryFailure; + const call = async (name, arguments_ = {}) => { + const result = await client.callTool( + { name, arguments: arguments_ }, + { timeout: 240_000 }, + ); + const projected = requireMcpOperationResult(result, name); + return ["open_binary", "binary_session", "close_binary"].includes(name) + ? projected + : (result.structuredContent ?? JSON.parse(mcpTextValue(result))); + }; + try { + await client.connect(transport); + connected = true; + await call("open_binary", { path: targetPath, provider_id: "ghidra" }); + const session = await call("binary_session"); + const mapping = parseEvidence( + await call("address_to_file_offset", { address: procedure }), + ); + const offset = mapping.normalized_result.file_offset; + const coordinate = `0x${offset.toString(16)}`; + const expected = parseEvidence( + await call("procedure_pseudo_code", { procedure }), + ); + const explicit = parseEvidence( + await call("procedure_pseudo_code", { + procedure: `file-offset:${coordinate}`, + }), + ); + const automatic = parseEvidence( + await call("procedure_pseudo_code", { procedure: coordinate }), + ); + for (const actual of [explicit, automatic]) { + assert.equal(actual.normalized_result, expected.normalized_result); + assert.equal(actual.raw_result.selection.file_offset, offset); + assert.equal(actual.raw_result.selection.mapped_address, procedure); + assert.ok( + actual.locations.some( + (location) => + location.kind === "file-offset" && location.offset === offset, + ), + ); + } + const invalid = await client.callTool({ + name: "procedure_pseudo_code", + arguments: { procedure: "file-offset:0x20000000000000" }, + }); + assert.equal(requireMcpToolError(invalid).code, "invalid_request"); + const unmapped = await client.callTool({ + name: "procedure_pseudo_code", + arguments: { procedure: "file-offset:0x100000" }, + }); + assert.ok( + JSON.stringify(requireMcpToolError(unmapped)).includes( + "no loaded source mapping", + ), + ); + assert.equal( + (await call("binary_session")).analysis_run.run_id, + session.analysis_run.run_id, + ); + const image = parseEvidence(await call("inspect_native_load_image")); + const headerMapping = image.normalized_result.observations.mappings.find( + (item) => item.initialized && item.file_offset === 0, + ); + assert.ok( + headerMapping, + "Fixture header must have an observed source mapping", + ); + const headerOffset = parseEvidence( + await call("address_to_file_offset", { address: headerMapping.start }), + ); + assert.equal(headerOffset.normalized_result.file_offset, 0); + for (const selector of ["file-offset:0x0", "0x0"]) { + const missing = requireMcpToolError( + await client.callTool({ + name: "procedure_pseudo_code", + arguments: { procedure: selector }, + }), + ); + assertMissingMappedProcedure(missing, headerMapping.start); + } + assert.equal( + (await call("binary_session")).analysis_run.run_id, + session.analysis_run.run_id, + ); + await call("close_binary"); + const cli = JSON.parse( + ( + await promisify(execFile)( + process.execPath, + [ + entrypoint, + "decompile", + targetPath, + `--file-offset=${coordinate}`, + "--provider", + "ghidra", + "--json", + ], + { env, timeout: 240_000, maxBuffer: 16 * 1024 * 1024 }, + ) + ).stdout, + ); + assert.equal(cli.normalized_result, explicit.normalized_result); + assert.deepEqual(cli.locations, explicit.locations); + const missingCli = await promisify(execFile)( + process.execPath, + [ + entrypoint, + "decompile", + targetPath, + "--file-offset=0x0", + "--provider", + "ghidra", + "--json", + ], + { env, timeout: 240_000, maxBuffer: 16 * 1024 * 1024 }, + ).then( + () => { + throw new Error( + "Mapped header bytes must not decompile as a procedure", + ); + }, + (cause) => JSON.parse(cause.stdout), + ); + assertMissingMappedProcedure(missingCli, headerMapping.start); + const alias = Buffer.from(original); + const pe = alias.readUInt32LE(0x3c); + if (alias.readUInt16LE(pe + 6) >= 2) { + const secondSection = pe + 24 + alias.readUInt16LE(pe + 20) + 40; + // Both sections now carry the same original bytes at distinct virtual addresses. + alias.writeUInt32LE(0x200, secondSection + 20); + const ambiguousPath = join(workspace, "ambiguous.exe"); + await writeFile(ambiguousPath, alias); + await call("open_binary", { path: ambiguousPath, provider_id: "ghidra" }); + const ambiguous = await client.callTool( + { + name: "procedure_pseudo_code", + arguments: { procedure: `file-offset:${coordinate}` }, + }, + { timeout: 240_000 }, + ); + assert.ok( + JSON.stringify(requireMcpToolError(ambiguous)).includes( + "multiple loaded mappings", + ), + ); + await call("close_binary"); + } + assert.deepEqual(await readFile(targetPath), original); + await client.close(); + connected = false; + assert.deepEqual(await readdir(runtime), []); + complete = true; + return { + cli_mcp_parity: true, + explicit_and_numeric_offsets: true, + invalid_and_unmapped_offsets: true, + mapped_offset_without_procedure: true, + ambiguous_mappings: + alias.readUInt16LE(pe + 6) >= 2 + ? "verified" + : "unavailable-single-section", + source_unchanged: true, + cleanup: "complete", + }; + } catch (cause) { + primaryFailure = cause; + throw cause; + } finally { + const cleanupFailures = []; + if (connected) { + try { + await call("close_binary"); + } catch (cause) { + cleanupFailures.push(cause); + } + } + try { + await client.close(); + } catch (cause) { + cleanupFailures.push(cause); + } + if (cleanupFailures.length > 0) { + process.stderr.write( + `Retained file-offset verifier workspace: ${workspace}\n`, + ); + throw new AggregateError( + primaryFailure === undefined + ? cleanupFailures + : [primaryFailure, ...cleanupFailures], + "File-offset verifier cleanup failed", + ); + } + if (complete) await rm(workspace, { recursive: true, force: true }); + else + process.stderr.write( + `Retained file-offset verifier workspace: ${workspace}\n`, + ); + } +} + +function assertMissingMappedProcedure(error, mappedAddress) { + assert.equal(error.code, "invalid_request"); + const message = error.details?.issues?.[0]?.message; + assert.equal(typeof message, "string"); + assert.ok( + message.includes("File offset 0x0 maps to analysis address"), + message, + ); + assert.ok(message.includes(mappedAddress), message); + assert.ok(message.includes("no procedure starts at or contains"), message); + assert.ok(!message.includes("--file-offset"), message); +} diff --git a/scripts/lib/real-hopper-owner-exit.mjs b/scripts/lib/real-hopper-owner-exit.mjs new file mode 100644 index 000000000..300b35ada --- /dev/null +++ b/scripts/lib/real-hopper-owner-exit.mjs @@ -0,0 +1,130 @@ +import assert from "node:assert/strict"; +import { execFile } from "node:child_process"; +import { setTimeout as delay } from "node:timers/promises"; +import { promisify } from "node:util"; + +import { Client } from "@modelcontextprotocol/client"; +import { StdioClientTransport } from "@modelcontextprotocol/client/stdio"; + +import { verifyNoTokenOwnedProcesses } from "../../dist/process/ProcessOwnership.js"; +import { + observeProcessStartIdentity, + signalProcessWithStartIdentity, +} from "../../dist/process/ProcessOwnershipObservation.js"; +import { startUnrelatedHopperSentinel } from "./unrelated-hopper-sentinel.mjs"; +import { + requireMcpEvidenceResult, + requireMcpLifecycleResult, +} from "./mcp-verifier-results.mjs"; + +const execFileAsync = promisify(execFile); + +/** Prove actual MCP owner exit releases the demo singleton for subsequent CLI/MCP sessions. */ +export async function verifyLinuxHopperOwnerExit({ + primary, + secondary, + serverCommand, + serverArgs, + environment, +}) { + const sentinel = await startUnrelatedHopperSentinel(); + const outcomes = []; + try { + for (const mode of ["graceful", "SIGKILL"]) { + const transport = new StdioClientTransport({ + command: serverCommand, + args: serverArgs, + env: environment, + stderr: "pipe", + }); + // Drain launcher diagnostics even on the deliberately interrupted owner. + transport.stderr?.resume(); + const client = new Client({ name: "hopper-owner-exit", version: "1" }); + const options = { timeout: 180_000 }; + try { + await client.connect(transport); + const ownerPid = transport.pid; + assert.ok(ownerPid !== null); + const owner = await observeProcessStartIdentity(ownerPid); + assert.equal(owner?.state, "readable"); + requireMcpLifecycleResult( + await client.callTool( + { + name: "open_binary", + arguments: { path: primary }, + }, + options, + ), + "open_binary", + ); + const procedures = requireMcpEvidenceResult( + await client.callTool( + { name: "list_procedures", arguments: {} }, + options, + ), + "list_procedures", + ); + assert.ok(procedures.length > 0); + const status = requireMcpLifecycleResult( + await client.callTool( + { name: "binary_session", arguments: {} }, + options, + ), + "binary_session", + ); + const runId = status.analysis_run?.run_id; + assert.equal(typeof runId, "string"); + assert.equal(status.analysis_provider_binding?.provider.id, "hopper"); + assert.equal((await verifyNoTokenOwnedProcesses(runId)).cleaned, false); + if (mode === "graceful") await client.close(); + else { + assert.equal( + await signalProcessWithStartIdentity( + ownerPid, + owner.identity, + "SIGKILL", + ), + "signaled", + ); + } + const deadline = Date.now() + 10_000; + let cleanup; + do { + cleanup = await verifyNoTokenOwnedProcesses(runId); + if (cleanup.cleaned) break; + await delay(100); + } while (Date.now() < deadline); + assert.equal(cleanup.cleaned, true, cleanup.reason); + // The SIGKILL case must recover the stale lease for both target identities. + for (const target of [primary, secondary]) { + const result = await execFileAsync( + process.execPath, + [ + "scripts/rea.mjs", + "inspect", + target, + "--provider", + "hopper", + "--format", + "json", + ], + { env: environment, timeout: 180_000, maxBuffer: 16 * 1024 * 1024 }, + ); + assert.ok(JSON.parse(result.stdout).normalized_result); + } + outcomes.push({ + mode, + ownedProcessesStopped: true, + sameTargetReopened: true, + differentTargetReopened: true, + }); + } finally { + await client.close(); + await transport.close(); + } + } + } finally { + assert.equal(await sentinel.verifyAndClose(), true); + } + return { outcomes, unrelatedHopperSurvived: true }; +} diff --git a/scripts/lib/real-hopper-search.d.mts b/scripts/lib/real-hopper-search.d.mts new file mode 100644 index 000000000..e64031652 --- /dev/null +++ b/scripts/lib/real-hopper-search.d.mts @@ -0,0 +1,30 @@ +type VerifierCall = ( + name: string, + arguments_?: Record, +) => Promise; + +interface StringExpectation { + readonly value: string; + readonly encoding: string; + readonly symbol?: string; + readonly recognitionOptional?: boolean; +} + +/** Verify source bytes, observed decoding, address filters and literal searches. */ +export function verifyHopperStringObjects( + call: VerifierCall, + expectations: readonly StringExpectation[], + inventory?: readonly z.output[] | null, +): Promise<{ + readonly sourceBytesVerified: true; + readonly recognition: readonly { + readonly value: string; + readonly address: string; + readonly state: "observed" | "not_observed"; + readonly decodedObject: "verified" | "unverified"; + }[]; + readonly displayEscapesExcludedFromMatching: true; +}>; +import type { z } from "zod"; + +import type { analysisStringSchema } from "../../src/domain/hopperValues.js"; diff --git a/scripts/lib/real-hopper-search.mjs b/scripts/lib/real-hopper-search.mjs index 0d796f3d2..3456bc6b2 100644 --- a/scripts/lib/real-hopper-search.mjs +++ b/scripts/lib/real-hopper-search.mjs @@ -1,13 +1,16 @@ import assert from "node:assert/strict"; import { mcpTextValue } from "./mcp-verifier-results.mjs"; +const fixtureSymbolPrefix = process.platform === "darwin" ? "_" : ""; + /** Exercise literal/regex search, Unicode identities and cache invalidation in Hopper. */ export async function verifyHopperSearch(call, invalid, procedure) { const address = procedure.address; const original = await call("address_name", { address }); + let stringCoverage; try { await verifySymbolNames(call, invalid, address); - await verifySearchModes(call, address); + stringCoverage = await verifySearchModes(call, address); } finally { await call("set_address_name", { address, name: original }); } @@ -26,7 +29,7 @@ export async function verifyHopperSearch(call, invalid, procedure) { symbolUtf16LimitValidatedBeforeMutation: true, regexAndCaseModes: true, annotationCacheInvalidation: true, - escapedUtf8AndByteStrings: true, + stringCoverage, }; } @@ -103,11 +106,26 @@ async function verifySearchModes(call, address) { [{ address, value: "REA_Search_Upper" }], ); const strings = await call("list_strings"); - const long = "REA_LONG_LITERAL_" + "x".repeat(4096) + "needle"; - const longItem = strings.find((item) => - item.value.startsWith("REA_LONG_LITERAL_"), + const linux = process.platform === "linux"; + const fullLiteral = "REA_LONG_LITERAL_" + "x".repeat(4096) + "needle"; + // Recognition is a provider/loader capability, not an adapter guarantee. + // Always retain the original size in raw-byte coverage; Linux additionally + // requires an extent observed to produce native fragmented string objects. + const fullBytes = Buffer.from(fullLiteral + "\0"); + const raw = await call("read_bytes", { + address: `${fixtureSymbolPrefix}rea_long_literal`, + length: fullBytes.length, + }); + assert.equal(raw.complete, true); + assert.equal(raw.bytes_hex, fullBytes.toString("hex")); + const long = linux + ? "REA_FRAGMENTED_LITERAL_" + "x".repeat(512) + "fragment_tail" + : fullLiteral; + const longItem = strings.find( + (item) => long.startsWith(item.value) && item.value.startsWith("REA_"), ); assert.ok(longItem, "fixture omitted its real long literal"); + if (!linux) assert.equal(longItem.address, raw.address); assert.ok(long.startsWith(longItem.value)); assert.ok(longItem.provider_value.endsWith("…")); assert.equal(longItem.string.termination, "missing"); @@ -158,7 +176,7 @@ async function verifySearchModes(call, address) { } assert.deepEqual( await call("search_strings", { - pattern: "needle", + pattern: linux ? "fragment_tail" : "needle", mode: "literal", case_sensitive: true, }), @@ -176,36 +194,122 @@ async function verifySearchModes(call, address) { }), expected, ); - await verifyHopperStringObjects(call, [ - { value: "REA_UTF8_é_😀", encoding: "utf-8" }, - { value: 'REA_ESCAPED_"\\line\nend\t\r', encoding: "utf-8" }, - { value: "REA_LITERAL_BACKSLASH_\\n", encoding: "utf-8" }, - { value: "REA_LITERAL_…", encoding: "utf-8" }, - { value: "REA_LATIN1_ÿ", encoding: "latin-1" }, - ]); + const objects = await verifyHopperStringObjects( + call, + [ + { + value: "REA_UTF8_é_😀", + encoding: "utf-8", + symbol: `${fixtureSymbolPrefix}rea_utf8_literal`, + recognitionOptional: linux, + }, + { value: 'REA_ESCAPED_"\\line\nend\t\r', encoding: "utf-8" }, + { value: "REA_LITERAL_BACKSLASH_\\n", encoding: "utf-8" }, + { + value: "REA_LITERAL_…", + encoding: "utf-8", + symbol: `${fixtureSymbolPrefix}rea_ellipsis_literal`, + recognitionOptional: linux, + }, + { + value: "REA_LATIN1_ÿ", + encoding: "latin-1", + symbol: `${fixtureSymbolPrefix}rea_latin1_literal`, + recognitionOptional: linux, + }, + ], + strings, + ); + return { + fullLiteralSourceBytes: fullBytes.length, + fragmentedLiteralSourceBytes: Buffer.byteLength(long), + fullLiteralRecognition: { + address: raw.address, + state: strings.some((item) => item.address === raw.address) + ? "observed" + : "not_observed", + // Linux validates the original through raw bytes; its required decoded + // fragment/dossier checks exercise the separate supported-size literal. + decodedObject: linux ? "unverified" : "verified", + }, + ...objects, + }; } /** Compare native string identities and values with the source fixture's bytes. */ -export async function verifyHopperStringObjects(call, expectations) { - const strings = await call("list_strings"); - for (const { value, encoding } of expectations) { - const item = strings.find((candidate) => candidate.value === value); - assert.ok( - item, - `Hopper omitted the decoded string ${JSON.stringify(value)}`, - ); - assert.equal(item.string.encoding, encoding); - assert.equal(item.string.encoding_status, "inferred"); - assert.equal(item.string.termination, "present_or_not_required"); +export async function verifyHopperStringObjects( + call, + expectations, + inventory = null, +) { + // Reuse only the caller's inventory for this unchanged document, never live + // state from an earlier session. Address-filter and search checks remain real. + const strings = inventory ?? (await call("list_strings")); + const recognition = []; + for (const { + value, + encoding, + symbol, + recognitionOptional = false, + } of expectations) { const expectedBytes = Buffer.from( value + "\0", encoding.replaceAll("-", ""), ); - assert.equal(item.string.byte_length, expectedBytes.length); - const bytes = await call("read_bytes", { + let sourceBytes; + if (symbol !== undefined) { + const raw = await call("read_bytes", { + address: symbol, + length: expectedBytes.length, + }); + sourceBytes = raw; + assert.equal(raw.complete, true); + assert.equal(raw.bytes_hex, expectedBytes.toString("hex")); + } + const item = strings.find((candidate) => + sourceBytes === undefined + ? candidate.value === value + : candidate.address === sourceBytes.address, + ); + if (item === undefined && recognitionOptional) { + assert.ok( + symbol, + "optional recognition requires independent source-byte coverage", + ); + recognition.push({ + value, + address: sourceBytes.address, + state: "not_observed", + decodedObject: "unverified", + }); + continue; + } + assert.ok( + item, + `Hopper omitted the decoded string ${JSON.stringify(value)}`, + ); + assert.equal( + item.value, + value, + "native string decoding changed the source value", + ); + recognition.push({ + value, address: item.address, - length: expectedBytes.length, + state: "observed", + decodedObject: "verified", }); + assert.equal(item.string.encoding, encoding); + assert.equal(item.string.encoding_status, "inferred"); + assert.equal(item.string.termination, "present_or_not_required"); + assert.equal(item.string.byte_length, expectedBytes.length); + const bytes = + sourceBytes ?? + (await call("read_bytes", { + address: item.address, + length: expectedBytes.length, + })); + assert.equal(bytes.complete, true); assert.equal(bytes.bytes_hex, expectedBytes.toString("hex")); assert.deepEqual(await call("list_strings", { address: item.address }), [ item, @@ -225,7 +329,7 @@ export async function verifyHopperStringObjects(call, expectations) { ); return { sourceBytesVerified: true, - inferredEncodingReported: true, + recognition, displayEscapesExcludedFromMatching: true, }; } diff --git a/scripts/rea.mjs b/scripts/rea.mjs index 78ffbdd21..75ee552c5 100755 --- a/scripts/rea.mjs +++ b/scripts/rea.mjs @@ -16,6 +16,12 @@ const isMcpMode = const isMcpDoctorMode = args[0] === "mcp" && args[1] === "doctor"; const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), ".."); +// Windows home lookup throws for an empty USERPROFILE. Dependencies resolve +// home during import, before REA's own resolver can apply its account fallback. +if (process.platform === "win32" && process.env.USERPROFILE === "") { + delete process.env.USERPROFILE; +} + // Playwright's registry assumes one of its three desktop host platforms when // it chooses the browser cache directory. Termux reports `android`, although // its Node runtime and filesystem are POSIX-compatible. Keep the cache path diff --git a/scripts/verify-linux-setup-e2e.mjs b/scripts/verify-linux-setup-e2e.mjs index 8b57fbacc..e08e8b160 100644 --- a/scripts/verify-linux-setup-e2e.mjs +++ b/scripts/verify-linux-setup-e2e.mjs @@ -12,6 +12,7 @@ import { tmpdir } from "node:os"; import { promisify } from "node:util"; import { completeVerifierRun, createVerifierRun } from "./lib/verifier-run.mjs"; +import { assertCleanLinuxHopperRuntime } from "./lib/clean-linux-hopper-runtime.mjs"; const execFileAsync = promisify(execFile); const verifierRun = createVerifierRun(); @@ -19,12 +20,12 @@ const command = process.env.REA_VERIFY_SERVER_COMMAND; if (process.platform !== "linux" || process.getuid?.() !== 0) throw new Error("Linux setup verification requires a root Linux runner"); if (!command) throw new Error("REA_VERIFY_SERVER_COMMAND is required"); +await assertCleanLinuxHopperRuntime(); try { await access("/opt/hopper/bin/Hopper"); throw new Error("Linux setup verification requires a clean runner"); } catch (error) { - if (error instanceof Error && error.message.includes("clean runner")) - throw error; + if (error?.code !== "ENOENT") throw error; } const runJson = async (args) => { @@ -35,14 +36,25 @@ const runJson = async (args) => { return JSON.parse(stdout); }; -const plan = await runJson(["setup", "--json"]); -if (plan.status !== "needs_confirmation") - throw new Error("setup did not produce a read-only confirmation plan"); +const plan = await runJson([ + "setup", + "--install-hopper", + "--dry-run", + "--json", +]); +if ( + plan.status !== "planned" || + !plan.plannedActions.some(({ kind }) => kind === "install_hopper") || + plan.appliedActions.length !== 0 +) + throw new Error("setup did not produce a read-only Hopper installation plan"); await access("/opt/hopper/bin/Hopper").then( () => { throw new Error("setup plan mutated the clean runner"); }, - () => undefined, + (error) => { + if (error?.code !== "ENOENT") throw error; + }, ); const installed = await runJson([ @@ -58,7 +70,7 @@ if ( throw new Error( `approved setup did not become ready: ${JSON.stringify(installed)}`, ); -const doctor = await runJson(["doctor", "--json"]); +const doctor = await runJson(["doctor", "--provider", "hopper", "--json"]); if (doctor.healthy !== true) throw new Error(`doctor is unhealthy: ${JSON.stringify(doctor)}`); @@ -67,8 +79,6 @@ await execFileAsync(command, ["analyze", "/usr/bin/true"], { timeout: 180_000, maxBuffer: 10 * 1024 * 1024, }); -process.env.HOPPER_TARGET_PATH = "/usr/bin/true"; -process.env.HOPPER_SECOND_TARGET_PATH = "/usr/bin/false"; process.env.REA_VERIFY_SERVER_ARGS_JSON = '["mcp"]'; await import("./verify-real-hopper-linux.mjs"); @@ -95,28 +105,24 @@ try { await rm(modifiedDirectory, { recursive: true, force: true }); } -const repeated = await runJson([ - "setup", - "--yes", - "--install-hopper", - "--json", -]); +const repeated = await runJson(["setup", "--yes", "--json"]); if ( repeated.status !== "ready" || - repeated.appliedActions.includes("installed_hopper") + repeated.plannedActions.length !== 0 || + repeated.appliedActions.length !== 0 ) throw new Error("a second setup was not idempotent"); -const { stdout: processes } = await execFileAsync("ps", [ - "-ax", - "-o", - "command=", -]); -if (/\/opt\/hopper\/bin\/Hopper|\bXvfb\b|hopper-demo-x11\.py/u.test(processes)) - throw new Error("Linux setup verification leaked a Hopper demo session"); +await assertCleanLinuxHopperRuntime(); +const completedVerifierRun = await completeVerifierRun(verifierRun); +if ( + completedVerifierRun.process_lineage.status !== "verified" || + completedVerifierRun.process_lineage.descendants.length !== 0 +) + throw new Error("Linux setup verification did not prove owned child cleanup"); console.log( JSON.stringify({ - verifier_run: await completeVerifierRun(verifierRun), + verifier_run: completedVerifierRun, status: "verified", setup: true, cli: true, diff --git a/scripts/verify-package.mjs b/scripts/verify-package.mjs index fff3a07c6..32bad0a62 100644 --- a/scripts/verify-package.mjs +++ b/scripts/verify-package.mjs @@ -1,6 +1,8 @@ -import { mkdtemp, rm } from "node:fs/promises"; +import { mkdir, mkdtemp, rm, unlink } from "node:fs/promises"; +import { execFile } from "node:child_process"; import { tmpdir } from "node:os"; import { join } from "node:path"; +import { promisify } from "node:util"; import { TOOL_CONTRACTS } from "../dist/contracts/toolContracts.js"; import { PRODUCT_IDENTITY } from "../dist/identity.js"; @@ -21,12 +23,19 @@ import { verifyRepositorySkill } from "./verify-repository-skill.mjs"; import { verifyManaged } from "./verify-package-managed.mjs"; import { verifyUnknownProvider } from "./verify-package-unknown-provider.mjs"; import { completeVerifierRun, createVerifierRun } from "./lib/verifier-run.mjs"; +import { + createVerifiedPackageMetadata, + retainVerifiedPackage, + writeVerifiedPackageManifest, +} from "./ci/verified-package.mjs"; import { measureMcpStartup, profileMcpModuleLoading, } from "./lib/mcp-startup-probe.mjs"; const root = process.cwd(); +const execFileAsync = promisify(execFile); +const retentionDirectory = process.env.REA_VERIFIED_PACKAGE_DIRECTORY; const verifierRun = createVerifierRun(); const workspace = await mkdtemp(join(tmpdir(), "rea-package-")); const evidenceRoot = join(workspace, "evidence"); @@ -37,6 +46,8 @@ let mcpModuleLoading; let update; let repositorySkill; let report; +let retainedPackage; +let retentionMetadata; const runStage = async (name, action) => { const started = performance.now(); @@ -49,6 +60,27 @@ const runStage = async (name, action) => { }; try { + if (retentionDirectory !== undefined && retentionDirectory !== "") { + const sourceSha = ( + await execFileAsync("git", ["rev-parse", "HEAD"], { cwd: root }) + ).stdout.trim(); + retentionMetadata = createVerifiedPackageMetadata({ + sourceSha, + runId: process.env.GITHUB_RUN_ID, + runAttempt: process.env.GITHUB_RUN_ATTEMPT, + platform: process.platform, + architecture: process.arch, + packageVersion: PRODUCT_IDENTITY.packageVersion, + nodeVersion: process.version, + }); + await mkdir(retentionDirectory, { recursive: true }); + await unlink(join(retentionDirectory, "manifest.json")).catch((cause) => { + if ( + !(cause instanceof Error && "code" in cause && cause.code === "ENOENT") + ) + throw cause; + }); + } repositorySkill = await runStage("repository skill", () => verifyRepositorySkill({ root, workspace }), ); @@ -215,6 +247,13 @@ try { mcpStartup, mcpModuleLoading, }; + if (retentionMetadata !== undefined) { + retainedPackage = await retainVerifiedPackage({ + sourceTarball: tarball, + directory: retentionDirectory, + metadata: retentionMetadata, + }); + } } finally { await runStage("workspace cleanup", () => rm(workspace, { recursive: true, force: true }), @@ -223,6 +262,11 @@ try { const verifierLineage = await runStage("verifier lineage", () => completeVerifierRun(verifierRun), ); +if (retainedPackage !== undefined) + await writeVerifiedPackageManifest( + retainedPackage.directory, + retainedPackage.metadata, + ); process.stdout.write( `${JSON.stringify({ verifier_run: verifierLineage, ...report })}\n`, ); diff --git a/scripts/verify-packet-capture.mjs b/scripts/verify-packet-capture.mjs index 371c72367..9993117c2 100644 --- a/scripts/verify-packet-capture.mjs +++ b/scripts/verify-packet-capture.mjs @@ -1,8 +1,9 @@ #!/usr/bin/env node import assert from "node:assert/strict"; +import { estimateTokenCount } from "tokenx"; import { execFile } from "node:child_process"; import { createHash } from "node:crypto"; -import { readFile, writeFile, readdir } from "node:fs/promises"; +import { readFile, writeFile, readdir, mkdir, rm } from "node:fs/promises"; import { isAbsolute, join } from "node:path"; import { fileURLToPath } from "node:url"; import { promisify } from "node:util"; @@ -91,6 +92,20 @@ const source = pcap(1, [ // LINKTYPE_BLUETOOTH_HCI_H4 = 187; H4 command packet, HCI Reset opcode 0x0c03, no parameters. const bluetoothSource = pcap(187, [Buffer.from("01030c00", "hex")]); let cases = 0; +const discoveryMeasurements = []; +const inheritedAddressSpace = (await readFile("/proc/self/limits", "utf8")) + .split("\n") + .find((line) => line.startsWith("Max address space ")) + ?.trim() + .match(/^Max address space\s+(\S+)\s+(\S+)\s+bytes$/u); +assert.ok(inheritedAddressSpace); +const expectedAddressSpace = Math.min( + 1073741824, + ...inheritedAddressSpace + .slice(1) + .map((value) => (value === "unlimited" ? Infinity : Number(value))), +); + try { await writeFile(path, source); await writeFile(bluetooth, bluetoothSource); @@ -121,14 +136,16 @@ try { .digest("hex"), ); assert.equal(report.hardware_acquisition, "not-performed"); - assert.equal(report.limits.address_space_bytes, 1073741824); + assert.equal(report.limits.address_space_bytes, expectedAddressSpace); assert.equal(report.tool.configured_path, process.env.REA_TSHARK_COMMAND); assert.equal( report.resource_limiter.configured_path, process.env.REA_TSHARK_PRLIMIT_COMMAND ?? "/usr/bin/prlimit", ); assert.ok( - report.resource_limiter.arguments.includes("--as=1073741824:1073741824"), + report.resource_limiter.arguments.includes( + `--as=${expectedAddressSpace}:${expectedAddressSpace}`, + ), ); assert.equal(report.configuration, "private-empty-personal-profile"); const flags = Object.entries(input) @@ -236,6 +253,52 @@ try { const hci = await inspect({ path: bluetooth, fields: ["bthci_cmd.opcode"] }); assert.deepEqual(hci.packets[0]["bthci_cmd.opcode"], ["0x0c03"]); assert.deepEqual(await readFile(path), source); + // Exercise the incremental summary workflow against the same owned bytes + // using real installed Wireshark tools; these inputs are format fixtures. + const pcapTool = (await client.listTools()).tools.find( + (t) => t.name === "inspect_pcap_capture", + ); + assert.ok(pcapTool?.outputSchema); + const pcapAjv = new Ajv2020({ strict: false, validateFormats: false }); + assert.ok(pcapAjv.validateSchema(pcapTool.inputSchema)); + assert.ok(pcapAjv.validateSchema(pcapTool.outputSchema)); + for (const capturePath of [path, bluetooth]) { + const response = await client.callTool({ + name: pcapTool.name, + arguments: { path: capturePath, detail: "summary" }, + }); + assert.notEqual(response.isError, true, JSON.stringify(response)); + assert.ok( + pcapAjv.validate(pcapTool.outputSchema, response.structuredContent), + pcapAjv.errorsText(), + ); + const summary = parseEvidence(response.structuredContent).normalized_result; + assert.equal( + summary.summary.target.sha256, + createHash("sha256") + .update(await readFile(capturePath)) + .digest("hex"), + ); + const source = { + kind: "retained-evidence", + evidence_id: summary.parent_evidence_id, + }; + const page = await client.callTool({ + name: "inspect_analysis_view", + arguments: { + source, + view: { kind: "pcap", facet: "protocols", limit: 10 }, + }, + }); + assert.notEqual(page.isError, true, JSON.stringify(page)); + assert.equal( + parseEvidence(page.structuredContent).normalized_result + .parent_evidence_id, + source.evidence_id, + ); + cases++; + } + assert.deepEqual(await readFile(bluetooth), bluetoothSource); for (const input of [ { path, fields: ["frame.number"], display_filter: "this is not a filter" }, @@ -267,6 +330,133 @@ try { true, ); cases++; + const discovery = (await client.listTools()).tools.find( + ({ name }) => name === "inspect_packet_fields", + ); + assert.ok(discovery?.outputSchema); + assert.ok(ajv.validateSchema(discovery.inputSchema)); + assert.ok(ajv.validateSchema(discovery.outputSchema)); + const nativeProfile = join(root.path, "native-profile"); + await mkdir(nativeProfile); + const native = ( + await execute(process.env.REA_TSHARK_COMMAND, ["-G", "fields"], { + env: { ...environment, WIRESHARK_CONFIG_DIR: nativeProfile }, + timeout: 30000, + maxBuffer: 32 * 1024 * 1024, + }) + ).stdout; + await rm(nativeProfile, { recursive: true }); + const nativeLines = native.split(/\r?\n/); + if (nativeLines.at(-1) === "") nativeLines.pop(); + for (const input of [ + { query: "usb" }, + { limit: 1 }, + { offset: 1, limit: 2 }, + { kind: "fields", protocol: "usb", query: "idVendor" }, + { kind: "fields", protocol: "udp", limit: null }, + { kind: "fields", query: "rea_no_such_field" }, + ]) { + const started = performance.now(); + const response = await client.callTool({ + name: discovery.name, + arguments: input, + }); + assert.notEqual(response.isError, true, JSON.stringify(response)); + assert.ok( + ajv.validate(discovery.outputSchema, response.structuredContent), + JSON.stringify(ajv.errors), + ); + const evidence = parseEvidence(response.structuredContent); + const report = evidence.normalized_result; + const kind = input.kind ?? "protocols"; + const query = (input.query ?? "").toLowerCase(); + const expectedMatches = nativeLines.flatMap((line, index) => { + const columns = line.split("\t"); + if ((kind === "protocols") !== (columns[0] === "P")) return []; + if ( + input.protocol !== undefined && + (columns[0] === "P" ? columns[2] : columns[4]) !== input.protocol + ) + return []; + if ( + ![ + columns[1], + columns[2], + ...(columns[0] === "F" ? [columns[7]] : []), + ].some((value) => value.toLowerCase().includes(query)) + ) + return []; + return [{ sourceLine: index + 1, columns }]; + }); + const offset = input.offset ?? 0; + const limit = input.limit === null ? undefined : (input.limit ?? 50); + const expectedPage = expectedMatches.slice( + offset, + limit === undefined ? undefined : offset + limit, + ); + assert.deepEqual( + report.records.map((record) => record.source_line), + expectedPage.map((record) => record.sourceLine), + ); + assert.deepEqual(report.output_window, { + examined: nativeLines.filter((line) => + line.startsWith(kind === "protocols" ? "P\t" : "F\t"), + ).length, + matching: expectedMatches.length, + returned: expectedPage.length, + has_more: + Math.max(0, expectedMatches.length - offset) > expectedPage.length, + }); + discoveryMeasurements.push({ + input, + elapsed_ms: +(performance.now() - started).toFixed(2), + response_bytes: Buffer.byteLength(JSON.stringify(response)), + estimated_tokens_one_tool_request_response: estimateTokenCount( + JSON.stringify(discovery) + + JSON.stringify(input) + + JSON.stringify(response), + ), + }); + for (const record of report.records) { + const columns = nativeLines[record.source_line - 1].split("\t"); + assert.equal(record.name, columns[1]); + assert.equal(record.abbreviation, columns[2]); + assert.equal(record.field_type, columns[0] === "F" ? columns[3] : null); + assert.equal( + record.parent_protocol, + columns[0] === "F" ? columns[4] : null, + ); + assert.equal(record.description, columns[0] === "F" ? columns[7] : null); + assert.equal(record.display_base, columns[0] === "F" ? columns[5] : null); + assert.equal(record.bitmask, columns[0] === "F" ? columns[6] : null); + } + if (input.query === "idVendor") + assert.ok( + report.records.some( + (row) => + row.abbreviation === "usb.idVendor" && + row.field_type === "FT_UINT16", + ), + ); + if (input.query === "usb") + assert.ok(report.records.some((row) => row.abbreviation === "usb")); + if (input.query === "rea_no_such_field") + assert.equal(report.output_window.matching, 0); + const flags = Object.entries(input).flatMap(([key, value]) => + value === null ? ["--all"] : [`--${key}`, String(value)], + ); + const cli = JSON.parse( + ( + await execute( + process.execPath, + [entrypoint, "inspect-packet-fields", ...flags, "--format", "json"], + { env: environment, timeout: 60000, maxBuffer: 16 * 1024 * 1024 }, + ) + ).stdout, + ); + assert.deepEqual(cli.normalized_result, report); + cases++; + } await client.close(); assert.deepEqual((await readdir(root.path)).sort(), [ "bluetooth.pcap", @@ -283,6 +473,9 @@ console.log( { status: "passed", cases, + discovery_measurements: discoveryMeasurements, + measurement_scope: + "Warm MCP server, fresh TShark processes; tokenx estimates include one advertised tool/request/response, not billed usage. Model-driven selection, CPU and peak process-family memory are unmeasured; no cache/speedup claim.", host: `${process.platform}/${process.arch}`, scope: "real TShark CLI/MCP; synthetic Ethernet/IPv4/UDP/DNS and Bluetooth HCI PCAP; no hardware acquisition", diff --git a/scripts/verify-pcap.mjs b/scripts/verify-pcap.mjs new file mode 100644 index 000000000..3adf14531 --- /dev/null +++ b/scripts/verify-pcap.mjs @@ -0,0 +1,131 @@ +import assert from "node:assert/strict"; +import { execFile } from "node:child_process"; +import { mkdtemp, readdir, rmdir } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { promisify } from "node:util"; +import { Client } from "@modelcontextprotocol/client"; +import { StdioClientTransport } from "@modelcontextprotocol/client/stdio"; +import { toolContract } from "../dist/contracts/toolContracts.js"; +import { hashStableFile } from "../dist/filesystem/StableFileHash.js"; +import { mcpTextValue } from "./lib/mcp-verifier-results.mjs"; + +const args = process.argv.slice(2); +if (args.includes("--help") || args.includes("-h")) { + console.log( + "Usage: npm run verify:pcap -- [--pcap PATH]\nRequires caller-supplied tshark/capinfos. Without --pcap, only client readiness is verified.", + ); + process.exit(0); +} +if (!(args.length === 0 || (args.length === 2 && args[0] === "--pcap"))) + throw new Error("Expected --pcap PATH"); +const path = args[1] === undefined ? null : resolve(args[1]); +// This parent is shared by independent CLI/MCP owners. A leased Windows +// runtime root would prevent their native admission of its directory handle. +const root = await mkdtemp(join(tmpdir(), "rea-pcap-verifier-")); +const environment = Object.fromEntries( + Object.entries(process.env).filter(([, value]) => typeof value === "string"), +); +Object.assign(environment, { + TMPDIR: root, + TMP: root, + TEMP: root, +}); +const entrypoint = resolve("scripts/rea.mjs"); +const execute = promisify(execFile); +const client = new Client({ name: "pcap-real-verifier", version: "1" }); +const transport = new StdioClientTransport({ + command: process.execPath, + args: [entrypoint, "mcp"], + env: environment, + stderr: "pipe", +}); +transport.stderr?.resume(); +try { + await client.connect(transport); + const readiness = await run("inspect_pcap_client", {}); + assert(readiness.client.tshark_version && readiness.client.capinfos_version); + if (path === null) + console.log("[unverified] Capture inspection: no --pcap was supplied"); + else { + const identity = await hashStableFile(path); + const result = await run("inspect_pcap_capture", { path }); + assert.deepEqual(result.target, { path, ...identity }); + assert(result.packet_count >= 0); + assert( + result.endpoints.every( + (entry) => entry.packet_count <= result.packet_count, + ), + ); + assert( + result.protocol_distribution.every( + (entry) => entry.packet_count <= result.packet_count, + ), + ); + const partial = + result.endpoint_count > result.endpoints.length || + result.dns_query_count > result.dns_queries.length || + result.tls_sni_count > result.tls_sni_values.length || + result.protocol_count > result.protocol_distribution.length || + result.http_request_uri_count > result.http_request_uris.length || + result.unattributed_dns_packet_count > 0 || + result.dns_queries.some( + (entry) => + entry.response_address_count > entry.response_addresses.length, + ); + assert.equal(result.coverage, partial ? "partial" : "complete"); + assert.deepEqual( + await hashStableFile(path), + identity, + "Input capture must remain unchanged", + ); + console.log( + `[ok] ${result.packet_count} packets; ${result.endpoint_count} addresses; coverage=${result.coverage}`, + ); + } +} finally { + try { + await client.close(); + } finally { + await transport.close(); + } + assert.deepEqual( + await readdir(root), + [], + `Owned capture resources remain under ${root}`, + ); + await rmdir(root); +} +console.log("All asserted pcap CLI/MCP checks passed"); + +async function run(name, input) { + const contract = toolContract(name); + const { stdout } = await execute( + process.execPath, + [ + entrypoint, + name.replaceAll("_", "-"), + ...(input.path === undefined ? [] : [input.path]), + "--json", + ], + { env: environment, timeout: 360_000, maxBuffer: 8 * 1024 * 1024 }, + ); + const cli = contract.outputSchema.parse(JSON.parse(stdout)); + const response = await client.callTool( + { name, arguments: input }, + { timeout: 360_000 }, + ); + assert.notEqual(response.isError, true, mcpTextValue(response)); + const mcp = contract.outputSchema.parse(JSON.parse(mcpTextValue(response))); + assert.deepEqual(mcp.normalized_result, cli.normalized_result); + assert.deepEqual(mcp.subject, cli.subject); + if (input.path !== undefined) { + assert.equal(cli.subject.local_path, input.path); + assert.equal( + cli.subject.digest.sha256, + cli.normalized_result.target.sha256, + ); + } + console.log(`[ok] ${name}: named output schema and CLI/MCP Evidence parity`); + return cli.normalized_result; +} diff --git a/scripts/verify-qlog-captures.mjs b/scripts/verify-qlog-captures.mjs new file mode 100644 index 000000000..8d1a16f6c --- /dev/null +++ b/scripts/verify-qlog-captures.mjs @@ -0,0 +1,215 @@ +#!/usr/bin/env node +import assert from "node:assert/strict"; +import { execFile } from "node:child_process"; +import { createHash } from "node:crypto"; +import { readFile, writeFile } from "node:fs/promises"; +import { isAbsolute } from "node:path"; +import { fileURLToPath } from "node:url"; +import { promisify } from "node:util"; +import { Client } from "@modelcontextprotocol/client"; +import { StdioClientTransport } from "@modelcontextprotocol/client/stdio"; +import { Ajv2020 } from "ajv/dist/2020.js"; +import { parseEvidence } from "../dist/domain/evidence.js"; + +// Caller-supplied producer evidence stays local; no fixture capture or secrets are bundled. +const [controlPath, lossPath, reportPath] = process.argv.slice(2); +if ( + ![controlPath, lossPath, reportPath].every( + (value) => typeof value === "string" && isAbsolute(value), + ) +) + throw new Error( + "Usage: node scripts/verify-qlog-captures.mjs /absolute/control/origin.qlog /absolute/loss/origin.qlog /absolute/new-report.json", + ); +const entrypoint = fileURLToPath(new URL("./rea.mjs", import.meta.url)); +const client = new Client({ name: "qlog-producer-verifier", version: "1" }); +const environment = { + PATH: process.env.PATH ?? "/usr/bin:/bin", + REA_LOG_LEVEL: "silent", + HOPPER_LAUNCHER_PATH: "/rea-unconfigured-deep-provider/hopper", +}; +const transport = new StdioClientTransport({ + command: process.execPath, + args: [entrypoint, "mcp"], + env: environment, + stderr: "pipe", +}); +let diagnostics = ""; +transport.stderr?.on("data", (bytes) => { + if (diagnostics.length < 65536) diagnostics += bytes.toString("utf8"); +}); +const ajv = new Ajv2020({ strict: false, validateFormats: false }); +const measurements = []; +try { + try { + await client.connect(transport); + } catch (cause) { + throw new Error(`MCP startup failed: ${diagnostics}`, { cause }); + } + const tool = (await client.listTools()).tools.find( + (tool) => tool.name === "inspect_web_network_capture", + ); + assert.ok(tool?.outputSchema); + for (const schema of [tool.inputSchema, tool.outputSchema]) + assert.ok(ajv.validateSchema(schema)); + const validateOutput = ajv.compile(tool.outputSchema); + for (const [lane, path] of [ + ["control", controlPath], + ["loss", lossPath], + ]) { + const source = await readFile(path); + const raw = JSON.parse(source.toString("utf8")); + const sha256 = createHash("sha256").update(source).digest("hex"); + assert.equal(raw.qlog_format, "JSON"); + assert.equal(raw.qlog_version, "0.3"); + const input = { capture_path: path, format: "qlog" }; + assert.ok(ajv.validate(tool.inputSchema, input)); + const response = await client.callTool({ + name: tool.name, + arguments: input, + }); + assert.notEqual(response.isError, true); + assert.ok( + validateOutput(response.structuredContent), + JSON.stringify(validateOutput.errors), + ); + const mcp = parseEvidence(response.structuredContent); + const completed = await promisify(execFile)( + process.execPath, + [entrypoint, "inspect-web-network-capture", path, "qlog", "--json"], + { env: environment, timeout: 40_000, maxBuffer: 96 * 1024 * 1024 }, + ); + const cli = parseEvidence(JSON.parse(completed.stdout)); + assert.equal(cli.evidence_id, mcp.evidence_id); + assert.deepEqual(cli.normalized_result, mcp.normalized_result); + const capture = mcp.normalized_result; + assert.equal(capture.artifact.sha256, sha256); + assert.equal( + capture.total_records, + raw.traces.reduce((sum, trace) => sum + trace.events.length, 0), + ); + for (const [ordinal, original] of raw.traces + .flatMap((trace) => trace.events) + .entries()) { + const record = capture.records[ordinal]; + assert.equal(record.ordinal, ordinal); + assert.equal(record.reported.name, original.name); + assert.equal(record.reported.time, original.time); + assert.deepEqual(record.binary_fields, []); + assert.ok( + record.numeric_literals.some( + (number) => + number.pointer === "/time" && + Number(number.literal) === original.time, + ), + ); + } + assert.equal(capture.qlog.body_bytes, "not-reported"); + assert.equal(mcp.authority, "historical-reference"); + assert.equal( + mcp.analysis_profile.parameters.qlog_profile, + "aioquic-1.5.0-json-0.3", + ); + const trace = capture.qlog.traces[0]; + assert.equal(trace.vantage_point, "server"); + const stream = trace.streams.find( + (stream) => stream.stream_id === "0" && stream.direction === "sent", + ); + assert.ok(stream); + assert.equal(stream.unique_bytes, "32803"); + assert.equal( + stream.frame_length_sum, + lane === "control" ? "32803" : "33972", + ); + assert.equal(stream.repeated_bytes, lane === "control" ? "0" : "1169"); + assert.equal(trace.loss_declarations.length, lane === "control" ? 0 : 1); + if (lane === "loss") + assert.deepEqual(trace.loss_declarations[0].packet_number, { + state: "reported", + value: "5", + }); + // Structurally-known secrets may not survive in Evidence, including sidecars. + const serialized = JSON.stringify(mcp); + for (const original of raw.traces.flatMap((trace) => trace.events)) { + if ( + original.name === "transport:parameters_set" && + typeof original.data.stateless_reset_token === "string" + ) + assert.ok(!serialized.includes(original.data.stateless_reset_token)); + for (const frame of original.data.frames ?? []) { + if ( + frame.frame_type === "new_connection_id" && + typeof frame.reset_token === "string" + ) + assert.ok(!serialized.includes(frame.reset_token)); + if (frame.frame_type === "new_token" && typeof frame.token === "string") + assert.ok(!serialized.includes(frame.token)); + } + } + const retained = await client.callTool({ + name: "inspect_analysis_view", + arguments: { + source: { kind: "retained-evidence", evidence_id: mcp.evidence_id }, + view: { kind: "api-surface" }, + }, + }); + assert.equal(retained.isError, true); + assert.equal( + JSON.parse(retained.content.find((item) => item.type === "text").text) + .error.code, + "unsupported_target", + ); + const selected = await client.callTool({ + name: tool.name, + arguments: { ...input, record_ordinals: [capture.total_records - 1, 0] }, + }); + const subset = parseEvidence(selected.structuredContent).normalized_result; + assert.deepEqual( + subset.records.map((record) => record.ordinal), + [capture.total_records - 1, 0], + ); + assert.deepEqual(subset.qlog, capture.qlog); + measurements.push({ + lane, + path, + sha256, + evidence_id: mcp.evidence_id, + profile_digest: mcp.analysis_profile.digest, + events: capture.total_records, + stream_id: stream.stream_id, + unique_bytes: stream.unique_bytes, + frame_length_sum: stream.frame_length_sum, + repeated_bytes: stream.repeated_bytes, + loss_declarations: trace.loss_declarations.length, + cli_mcp_parity: true, + retained_reference: true, + selected_scope_preserved: true, + auth_exclusions: true, + }); + } + await client.ping(); + const report = { + status: "passed", + profile: "aioquic-1.5.0-json-0.3", + measurements, + limitations: [ + "Caller supplies provenance; format selection does not authenticate the producer.", + "These known loopback cases do not establish WAN behavior, other qlog producers, platforms or general completeness.", + "No PCAP or keylog is read by this verifier; repeated STREAM coverage is not proof of retransmission or network loss.", + ], + }; + await writeFile(reportPath, JSON.stringify(report, null, 2) + "\n", { + flag: "wx", + mode: 0o600, + }); + console.log( + JSON.stringify({ + status: report.status, + cases: measurements.length, + report: reportPath, + }), + ); +} finally { + await client.close(); + await transport.close(); +} diff --git a/scripts/verify-qwen-client.mjs b/scripts/verify-qwen-client.mjs index ed60ccd4c..fff4338b7 100644 --- a/scripts/verify-qwen-client.mjs +++ b/scripts/verify-qwen-client.mjs @@ -8,7 +8,7 @@ import { realpath, } from "node:fs/promises"; import { tmpdir } from "node:os"; -import { join } from "node:path"; +import { join, resolve, sep } from "node:path"; import { fileURLToPath } from "node:url"; import { createHash } from "node:crypto"; import { exec } from "./lib/verify-package-core.mjs"; @@ -18,10 +18,6 @@ import { toolContract, } from "../dist/contracts/toolContracts.js"; -assert( - process.platform !== "win32", - "The Qwen real-client verifier currently requires a POSIX host; native Windows is unverified.", -); const repo = fileURLToPath(new URL("..", import.meta.url)); const runtimeRoot = process.env.REA_VERIFY_RUNTIME_ROOT ?? repo; const lab = await mkdtemp(join(tmpdir(), "rea-qwen-client-")); @@ -32,6 +28,13 @@ assert( "Usage: node scripts/verify-qwen-client.mjs [chat|call]", ); const command = process.env.REA_VERIFY_QWEN_COMMAND ?? "qwen"; +// npm exposes only qwen.cmd on Windows. Resolve a selected JavaScript entry +// before the verifier switches to its disposable workspace. +const entryPoint = /\.(c|m)?js$/iu.test(command) ? resolve(command) : undefined; +const launch = (args) => + entryPoint === undefined + ? [command, args] + : [process.execPath, [entryPoint, ...args]]; const verifier = createVerifierRun(); const profile = join(lab, "selected profile"); const account = join(lab, "account"); @@ -54,14 +57,18 @@ const baseEnvironment = { let clientVersion; try { clientVersion = ( - await exec(command, ["--version"], { + await exec(...launch(["--version"]), { env: baseEnvironment, timeout: 30_000, }) ).stdout.trim(); } catch (cause) { + const windowsHint = + process.platform === "win32" + ? " On Windows, select node_modules/@qwen-code/qwen-code/cli-entry.js." + : ""; throw new Error( - `Qwen client preflight failed for ${command}; install Qwen Code or set REA_VERIFY_QWEN_COMMAND. ${String(cause)}`, + `Qwen client preflight failed for ${command}; install Qwen Code or set REA_VERIFY_QWEN_COMMAND.${windowsHint} ${String(cause)}`, ); } const original = JSON.stringify( @@ -93,8 +100,8 @@ for (const step of [ assert( result.plannedActions.every( (a) => - a.target.startsWith(profile + "/") || - a.target.startsWith(sharedSkills + "/"), + a.target.startsWith(profile + sep) || + a.target.startsWith(sharedSkills + sep), ), ); await writeFile( @@ -163,7 +170,14 @@ const fixture = await createOpenAiModelFixture({ skillSeen ||= JSON.stringify(body.messages ?? []).includes( "reverse-engineer-anything", ); - selectedSkillLoaded ||= JSON.stringify(body.messages ?? []).includes( + const messageText = (body.messages ?? []) + .map((message) => + typeof message.content === "string" + ? message.content + : (message.content ?? []).map((part) => part.text ?? "").join("\n"), + ) + .join("\n"); + selectedSkillLoaded ||= messageText.includes( join(sharedSkills, "reverse-engineer-anything"), ); const toolText = JSON.stringify( @@ -243,9 +257,7 @@ const fixture = await createOpenAiModelFixture({ const invoke = tools.find((t) => t.name === "tool_call"); const skill = tools.find((t) => t.name === "skill"); const shell = tools.find((t) => t.name === "run_shell_command"); - const persisted = toolText.match( - /Full output saved to: ([^\\"\n]+)(?:\\n|\n)/u, - ); + const persisted = messageText.match(/Full output saved to: ([^\r\n]+)/u); const call = (name, args) => ({ role: "assistant", content: null, @@ -268,13 +280,14 @@ const fixture = await createOpenAiModelFixture({ artifactPath = persisted[1]; assert( (await realpath(artifactPath)).startsWith( - (await realpath(runtime)) + "/", + (await realpath(runtime)) + sep, ), ); const script = `const fs=require('node:fs');const e=JSON.parse(fs.readFileSync(${JSON.stringify(artifactPath)},'utf8'));console.log(JSON.stringify({REA_CLIENT_FIXTURE:JSON.stringify(e.normalized_result).includes('REA_CLIENT_FIXTURE'),evidence_id:e.evidence_id,graph_nodes:e.normalized_result.graph.nodes.length}));`; - const quote = (value) => "'" + value.replaceAll("'", "'\\''") + "'"; + // Keep paths and JSON quotes out of shell syntax on Windows and POSIX. + const encodedScript = Buffer.from(script).toString("base64"); message = call(shell.name, { - command: `node -e ${quote(script)}`, + command: `node -e "eval(Buffer.from('${encodedScript}','base64').toString('utf8'))"`, description: "Read the complete saved REA Evidence and verify the fixture facts.", }); @@ -329,7 +342,7 @@ const args = [ let response; let failure; try { - response = await exec(command, args, { + response = await exec(...launch(args), { env, cwd: workspace, timeout: 120_000, diff --git a/scripts/verify-real-ghidra-6502.mjs b/scripts/verify-real-ghidra-6502.mjs new file mode 100644 index 000000000..87fb01d41 --- /dev/null +++ b/scripts/verify-real-ghidra-6502.mjs @@ -0,0 +1,479 @@ +import assert from "node:assert/strict"; +import { execFile } from "node:child_process"; +import { createHash } from "node:crypto"; +import { + mkdir, + mkdtemp, + readFile, + readdir, + rm, + symlink, + unlink, + writeFile, +} from "node:fs/promises"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { promisify } from "node:util"; + +import { Client } from "@modelcontextprotocol/client"; +import { StdioClientTransport } from "@modelcontextprotocol/client/stdio"; +import { parseEvidence } from "../dist/domain/evidence.js"; +import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs"; +import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs"; + +if (process.argv.length !== 2) + throw new Error("Usage: node scripts/verify-real-ghidra-6502.mjs"); +if (process.env.GHIDRA_INSTALL_DIR === undefined) + throw new Error( + "Set GHIDRA_INSTALL_DIR to a bring-your-own Ghidra installation.", + ); +const exec = promisify(execFile); +const run = createVerifierRun(); +const root = fileURLToPath(new URL("..", import.meta.url)); +const ca65 = process.env.REA_CA65 ?? "ca65"; +const ld65 = process.env.REA_LD65 ?? "ld65"; +const toolchain = {}; +for (const [name, command] of Object.entries({ ca65, ld65 })) { + const version = await exec(command, ["--version"], { timeout: 10000 }); + toolchain[name] = (version.stdout + version.stderr).trim(); +} +await mkdir(join(root, "build"), { recursive: true }); +const workspace = await mkdtemp(join(root, "build", "rea-6502-proof-")); +const runtime = join(workspace, "runtime"); +await mkdir(runtime); +const path = join(workspace, "image.bin"); +const object = join(workspace, "image.o"); +const entrypoint = join(root, "scripts", "rea.mjs"); +const env = { + ...process.env, + TMPDIR: runtime, + TMP: runtime, + TEMP: runtime, + REA_LOG_LEVEL: "silent", + REA_ANALYSIS_PROVIDER: "ghidra", + REA_PROCESS_RUN_ID: run.run_id, +}; +const transport = new StdioClientTransport({ + command: process.execPath, + args: [entrypoint, "mcp"], + env, + stderr: "pipe", +}); +const client = new Client({ name: "rea-real-6502-proof", version: "1" }); +let stderr = ""; +transport.stderr?.on("data", (chunk) => { + stderr = (stderr + chunk.toString()).slice(-65536); +}); +let report; +try { + await exec( + ca65, + [join(root, "tests/conformance/ghidra/6502.s"), "-o", object], + { timeout: 10000 }, + ); + await exec( + ld65, + ["-C", join(root, "tests/conformance/ghidra/6502.cfg"), "-o", path, object], + { timeout: 10000 }, + ); + const bytes = await readFile(path); + // Independently known opcodes and link addresses from the source fixture. + assert.equal(bytes.toString("hex"), "a907201080c90ad003a92a60a9006018690360"); + const sha256 = createHash("sha256").update(bytes).digest("hex"); + await client.connect(transport); + const opened = await call("open_binary", { + path, + format: "raw-binary", + architecture: "6502", + entry_point: "0x008001", + provider_id: "ghidra", + }); + assert.equal(opened.architecture, "6502"); + assert.equal(opened.entry_point, "0x8001"); + const status = await call("binary_session"); + const image = evidence(await call("inspect_native_load_image"), sha256); + assert.equal( + image.normalized_result.status, + "verified", + JSON.stringify(image.normalized_result), + ); + assert.equal(image.normalized_result.architecture, "6502"); + assert.equal(image.normalized_result.base_address, "0x8001"); + assert.equal(image.normalized_result.language_id, "6502:LE:16:default"); + assert.equal(image.normalized_result.compiler_spec_id, "default"); + assert.equal(image.normalized_result.file_bytes, bytes.length); + const memory = evidence( + await call("read_bytes", { address: "0x8001", length: bytes.length }), + sha256, + ); + assert.equal(memory.normalized_result.bytes_hex, bytes.toString("hex")); + const procedures = evidence(await call("list_procedures"), sha256); + for (const address of ["0x8001", "0x8010"]) + assert.ok( + procedures.normalized_result.some( + (procedure) => procedure.address === address, + ), + ); + const instruction = evidence( + await call("inspect_native_instruction", { address: "0x8003" }), + sha256, + ); + assert.equal(instruction.normalized_result.mnemonic.toUpperCase(), "JSR"); + assert.ok( + instruction.normalized_result.references.some( + (reference) => reference.call && reference.target_address === "0x8010", + ), + ); + const pseudo = evidence( + await call("procedure_pseudo_code", { procedure: "0x8010" }), + sha256, + ); + assert.match(pseudo.normalized_result, /\+\s*(?:3|0x3|'\\x03')/u); + const filePseudo = evidence( + await call("procedure_pseudo_code", { procedure: "file-offset:0xf" }), + sha256, + ); + assert.equal(filePseudo.normalized_result, pseudo.normalized_result); + assert.ok( + filePseudo.locations.some( + (location) => location.kind === "file-offset" && location.offset === 15, + ), + ); + const rejected = await client.callTool({ + name: "open_binary", + arguments: { + path, + format: "raw-binary", + architecture: "6502", + entry_point: "0xffff", + }, + }); + assert.equal(rejected.isError, true); + assert.match(JSON.stringify(rejected.content), /6502 address width/u); + const preserved = await call("binary_session"); + assert.equal(preserved.analysis_run.run_id, status.analysis_run.run_id); + assert.equal(preserved.entry_point, "0x8001"); + await call("close_binary"); + await client.ping(); + const { stdout } = await exec( + process.execPath, + [ + entrypoint, + "inspect-native-load-image", + path, + "--target-format", + "raw-binary", + "--architecture", + "6502", + "--entry-point", + "0x8001", + "--provider", + "ghidra", + "--json", + ], + { env, timeout: 240000, maxBuffer: 16 * 1024 * 1024 }, + ); + const cli = parseEvidence(JSON.parse(stdout)); + assert.deepEqual(cli.normalized_result, image.normalized_result); + assert.equal(cli.analysis_profile.digest, image.analysis_profile.digest); + assert.equal(cli.subject.digest.sha256, sha256); + const fileCli = await exec( + process.execPath, + [ + entrypoint, + "decompile", + path, + "--file-offset=0xf", + "--target-format", + "raw-binary", + "--architecture", + "6502", + "--entry-point", + "0x8001", + "--provider", + "ghidra", + "--json", + ], + { env, timeout: 240000, maxBuffer: 16 * 1024 * 1024 }, + ); + const fileCliEvidence = parseEvidence(JSON.parse(fileCli.stdout)); + assert.equal(fileCliEvidence.normalized_result, filePseudo.normalized_result); + assert.deepEqual(fileCliEvidence.locations, filePseudo.locations); + assert.equal(fileCliEvidence.subject.digest.sha256, sha256); + assert.deepEqual(await readFile(path), bytes); + assert.deepEqual(await readdir(runtime), []); + const lowMemory = []; + // BinaryLoader maps file bytes before trying processor-defined blocks. Prove + // that overlapping zero-page/stack definitions never overlay the selected image. + for (const [entry, fullAddressSpace] of [ + [0xff, false], + [0, true], + ]) { + const lowPath = join(workspace, `low-${entry}.bin`); + await exec( + ld65, + [ + "-C", + join(root, "tests/conformance/ghidra/6502.cfg"), + "--start-addr", + String(entry), + "-o", + lowPath, + object, + ], + { timeout: 10000 }, + ); + const linked = await readFile(lowPath); + const expectedCode = Buffer.from( + "a907200000c90ad003a92a60a9006018690360", + "hex", + ); + expectedCode.writeUInt16LE(entry + 15, 3); + assert.deepEqual(linked, expectedCode); + const lowBytes = fullAddressSpace + ? Buffer.concat([linked, Buffer.alloc(65536 - linked.length)]) + : linked; + await writeFile(lowPath, lowBytes); + const lowSha256 = createHash("sha256").update(lowBytes).digest("hex"); + const lowEntry = `0x${entry.toString(16)}`; + const lowOpened = await call("open_binary", { + path: lowPath, + format: "raw-binary", + architecture: "6502", + entry_point: lowEntry, + provider_id: "ghidra", + }); + assert.equal(lowOpened.entry_point, lowEntry); + const lowImage = evidence( + await call("inspect_native_load_image"), + lowSha256, + ); + assert.equal( + lowImage.normalized_result.status, + "verified", + JSON.stringify(lowImage.normalized_result), + ); + assert.equal(lowImage.normalized_result.base_address, lowEntry); + assert.equal(lowImage.normalized_result.file_bytes, lowBytes.length); + const fileMappings = + lowImage.normalized_result.observations.mappings.filter( + (mapping) => mapping.source_file_index !== null, + ); + assert.equal(fileMappings.length, 1); + assert.equal( + fileMappings[0].address_space, + lowImage.normalized_result.observations.default_address_space, + ); + assert.equal(fileMappings[0].overlay, false); + assert.equal(fileMappings[0].start, lowEntry); + assert.equal(fileMappings[0].length, lowBytes.length); + const lowRead = evidence( + await call("read_bytes", { address: lowEntry, length: linked.length }), + lowSha256, + ); + assert.equal(lowRead.normalized_result.bytes_hex, linked.toString("hex")); + const lowCall = evidence( + await call("inspect_native_instruction", { + address: `0x${(entry + 2).toString(16)}`, + }), + lowSha256, + ); + assert.equal(lowCall.normalized_result.mnemonic.toUpperCase(), "JSR"); + assert.ok( + lowCall.normalized_result.references.some( + (reference) => + reference.call && + reference.target_address === `0x${(entry + 15).toString(16)}`, + ), + ); + await call("close_binary"); + const lowCli = await exec( + process.execPath, + [ + entrypoint, + "inspect-native-load-image", + lowPath, + "--target-format", + "raw-binary", + "--architecture", + "6502", + "--entry-point", + lowEntry, + "--provider", + "ghidra", + "--json", + ], + { env, timeout: 240000, maxBuffer: 16 * 1024 * 1024 }, + ); + const lowCliEvidence = evidence(JSON.parse(lowCli.stdout), lowSha256); + assert.deepEqual( + lowCliEvidence.normalized_result, + lowImage.normalized_result, + ); + assert.equal( + lowCliEvidence.analysis_profile.digest, + lowImage.analysis_profile.digest, + ); + assert.deepEqual(await readFile(lowPath), lowBytes); + assert.deepEqual(await readdir(runtime), []); + lowMemory.push({ + entry: lowEntry, + bytes: lowBytes.length, + sha256: lowSha256, + }); + } + let windowsReparseAdmission = "not-applicable"; + if (process.platform === "win32") { + const junction = join(workspace, "linked"); + await symlink(workspace, junction, "junction"); + try { + const linkedPath = join(junction, "image.bin"); + // Reopening the same bytes through a junction must replace the direct session. + await call("open_binary", { + path, + format: "raw-binary", + architecture: "6502", + entry_point: "0x8001", + provider_id: "ghidra", + }); + const directImage = evidence( + await call("inspect_native_load_image"), + sha256, + ); + assert.equal(directImage.normalized_result.status, "verified"); + // Canonical file identity must not bypass admission of the new path. + await call("open_binary", { + path: linkedPath, + format: "raw-binary", + architecture: "6502", + entry_point: "0x8001", + provider_id: "ghidra", + }); + // open_binary selects a lazy session; the first analysis admits the snapshot. + const linkedImage = await client.callTool( + { name: "inspect_native_load_image", arguments: {} }, + { timeout: 240000 }, + ); + assert.equal(linkedImage.isError, true); + assert.match( + JSON.stringify(linkedImage), + /Reparse path component is unsupported/u, + ); + await call("close_binary"); + const snapshotPath = join(workspace, "direct-snapshot.json"); + const cliArgs = (selectedPath) => [ + entrypoint, + "analyze", + selectedPath, + "--target-format", + "raw-binary", + "--architecture", + "6502", + "--entry-point", + "0x8001", + "--provider", + "ghidra", + "--snapshot", + snapshotPath, + "--json", + ]; + const saved = await exec(process.execPath, cliArgs(path), { + env, + timeout: 240000, + maxBuffer: 16 * 1024 * 1024, + }); + assert.equal( + evidence(JSON.parse(saved.stdout), sha256).operation, + "binary_overview", + ); + const savedSnapshot = JSON.parse(await readFile(snapshotPath, "utf8")); + assert.ok( + savedSnapshot.workflow_entries.some( + ({ operation }) => operation === "binary_overview", + ), + ); + assert.deepEqual(await readdir(runtime), []); + // Replaying a matching snapshot must still reach native path admission. + await assert.rejects( + exec(process.execPath, cliArgs(linkedPath), { + env, + timeout: 240000, + maxBuffer: 16 * 1024 * 1024, + }), + (cause) => { + assert.match( + cause.stdout + cause.stderr, + /Reparse path component is unsupported/u, + ); + return true; + }, + ); + assert.deepEqual(await readFile(path), bytes); + assert.deepEqual(await readdir(runtime), []); + windowsReparseAdmission = "cli-snapshot-mcp-rejected"; + } finally { + await unlink(junction); + } + } + report = { + ok: true, + mocked: false, + lane: "ghidra-6502", + host: process.platform, + toolchain, + provider: image.provider, + target: { + sha256, + bytes: bytes.length, + architecture: "6502", + entry: "0x8001", + }, + cli_mcp_load_image_parity: true, + instructions_and_call_reference: true, + decompilation: true, + file_offset_decompilation_cli_mcp: true, + invalid_mapping_preserves_session: true, + low_memory_cli_mcp_parity: lowMemory, + windows_reparse_admission: windowsReparseAdmission, + source_unchanged: true, + runtime_cleanup: "complete", + }; +} catch (cause) { + if (stderr) process.stderr.write(stderr); + throw cause; +} finally { + try { + await client.close(); + } finally { + await transport.close(); + } +} +const verifierRun = await completeVerifierRun(run); +if (process.platform !== "win32") { + assert.equal(verifierRun.process_lineage.status, "verified"); + assert.deepEqual(verifierRun.process_lineage.descendants, []); +} +// Keep the workspace for diagnosis unless verification and cleanup both pass. +await rm(workspace, { recursive: true, force: true }); +process.stdout.write( + JSON.stringify({ verifier_run: verifierRun, ...report }) + "\n", +); + +async function call(name, args = {}) { + const result = await client.callTool( + { name, arguments: args }, + { timeout: 240000 }, + ); + assert.notEqual(result.isError, true, JSON.stringify(result)); + if (["open_binary", "binary_session", "close_binary"].includes(name)) + return requireMcpOperationResult(result, name); + return result.structuredContent; +} + +function evidence(value, sha256) { + const result = parseEvidence(value); + assert.equal(result.subject.digest.sha256, sha256); + assert.equal(result.subject.architecture, "6502"); + assert.equal(result.subject.format, "raw-binary"); + return result; +} diff --git a/scripts/verify-real-ghidra-address-context.mjs b/scripts/verify-real-ghidra-address-context.mjs new file mode 100644 index 000000000..2b650df93 --- /dev/null +++ b/scripts/verify-real-ghidra-address-context.mjs @@ -0,0 +1,313 @@ +import assert from "node:assert/strict"; +import { execFile } from "node:child_process"; +import { createHash } from "node:crypto"; +import { mkdir, mkdtemp, readFile, readdir, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { promisify } from "node:util"; +import { Client } from "@modelcontextprotocol/client"; +import { StdioClientTransport } from "@modelcontextprotocol/client/stdio"; +import { parseEvidence } from "../dist/domain/evidence.js"; +import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs"; +import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs"; + +if (process.argv.length !== 2 || process.platform !== "linux") + throw new Error("Run this ELF verifier on Linux without arguments."); +if (!process.env.GHIDRA_INSTALL_DIR) + throw new Error("Set GHIDRA_INSTALL_DIR to a caller-installed Ghidra."); +const exec = promisify(execFile); +const run = createVerifierRun(); +const root = fileURLToPath(new URL("..", import.meta.url)); +const workspace = await mkdtemp(join(tmpdir(), "rea-address-context-")); +const runtime = join(workspace, "runtime"); +await mkdir(runtime); +const path = join(workspace, "pie.elf"); +const entrypoint = join(root, "scripts/rea.mjs"); +const env = { + ...process.env, + TMPDIR: runtime, + REA_ANALYSIS_PROVIDER: "ghidra", + REA_LOG_LEVEL: "silent", +}; +const transport = new StdioClientTransport({ + command: process.execPath, + args: [entrypoint, "mcp"], + env, + stderr: "pipe", +}); +const client = new Client({ name: "rea-real-address-context", version: "1" }); +let stderr = ""; +transport.stderr?.on("data", (chunk) => { + stderr = (stderr + chunk.toString()).slice(-65536); +}); +let report; +let connected = false; +let primaryFailure; +let verifierRun; +try { + const compiler = process.env.CC ?? "gcc"; + const version = await exec(compiler, ["--version"], { timeout: 10000 }); + await exec( + compiler, + [ + "-O0", + "-g", + "-fPIE", + "-pie", + "-nostdlib", + "-Wl,-e,rea_pie_entry", + "-Wl,--build-id=none", + join(root, "tests/conformance/c/pie-address-context.c"), + "-o", + path, + ], + { timeout: 30000 }, + ); + const bytes = await readFile(path); + assert.equal(bytes.subarray(0, 6).toString("hex"), "7f454c460201"); + assert.equal(bytes.readUInt16LE(16), 3); // ET_DYN: source-owned x86-64 PIE. + assert.equal(bytes.readUInt16LE(18), 62); + const declaredEntry = `0x${bytes.readBigUInt64LE(24).toString(16)}`; + const sha256 = createHash("sha256").update(bytes).digest("hex"); + await client.connect(transport); + connected = true; + await call("open_binary", { path, provider_id: "ghidra" }); + const overview = parseEvidence(await call("binary_overview")); + assert.equal(overview.subject.digest.sha256, sha256); + const base = overview.normalized_result.image_base; + assert.match(base, /^0x[0-9a-f]+$/u); + assert.notEqual(base, "0x0"); + const active = await call("binary_session"); + const image = parseEvidence(await call("inspect_native_load_image")); + assert.equal(base, image.normalized_result.observations.image_base); + assert.ok(overview.normalized_result.segments.length > 0); + assert.ok( + overview.normalized_result.segments.every( + (segment) => + segment.image_base === base && + typeof segment.address_space === "string", + ), + ); + const procedures = parseEvidence(await call("list_procedures")); + const entry = procedures.normalized_result.find( + (item) => item.value === "rea_pie_entry", + ); + const addition = procedures.normalized_result.find( + (item) => item.value === "rea_pie_add", + ); + assert.ok(entry && addition); + assert.notEqual(entry.address, declaredEntry); + assert.ok( + image.normalized_result.observations.entry_points.includes(entry.address), + ); + const mapping = image.normalized_result.observations.mappings.find( + (item) => + item.source_file_index === 0 && + item.file_offset !== null && + item.address_space === "ram" && + BigInt(item.start) <= BigInt(entry.address) && + BigInt(item.end) >= BigInt(entry.address), + ); + assert.ok(mapping); + const offset = + mapping.file_offset + Number(BigInt(entry.address) - BigInt(mapping.start)); + const reverseMapping = parseEvidence( + await call("address_to_file_offset", { address: entry.address }), + ); + assert.equal(reverseMapping.normalized_result.file_offset, offset); + const memory = parseEvidence( + await call("read_bytes", { address: entry.address, length: 8 }), + ); + assert.equal( + memory.normalized_result.bytes_hex, + bytes.subarray(offset, offset + 8).toString("hex"), + ); + const fromOffset = parseEvidence( + await call("procedure_pseudo_code", { + procedure: `file-offset:0x${offset.toString(16)}`, + }), + ); + assert.ok( + fromOffset.locations.some( + (location) => + location.kind === "file-offset" && location.offset === offset, + ), + ); + assert.equal(fromOffset.raw_result.selection.mapped_address, entry.address); + const fromNumericOffset = parseEvidence( + await call("procedure_pseudo_code", { + procedure: `0x${offset.toString(16)}`, + }), + ); + assert.equal( + fromNumericOffset.normalized_result, + fromOffset.normalized_result, + ); + const missingAddress = "ram:0xffffffff"; + const rejected = await client.callTool( + { name: "procedure_pseudo_code", arguments: { procedure: missingAddress } }, + { timeout: 240000 }, + ); + assert.equal(rejected.isError, true); + assert.ok( + JSON.stringify(rejected.content).includes(`Ghidra image base: ${base}`), + ); + assert.ok(JSON.stringify(rejected.content).includes("list_procedures")); + assert.ok( + JSON.stringify(rejected.content).includes("inspect_native_load_image"), + ); + const pseudo = parseEvidence( + await call("procedure_pseudo_code", { procedure: addition.address }), + ); + assert.match(pseudo.normalized_result, /\+\s*(?:7|0x7)\b/u); + const retained = await call("binary_session"); + assert.equal(retained.analysis_run.run_id, active.analysis_run.run_id); + await call("close_binary"); + const cli = async (args) => + exec( + process.execPath, + [entrypoint, ...args, "--provider", "ghidra", "--json"], + { env, timeout: 240000, maxBuffer: 16 * 1024 * 1024 }, + ); + const cliOverview = parseEvidence( + JSON.parse((await cli(["analyze", path])).stdout), + ); + assert.deepEqual(cliOverview.normalized_result, overview.normalized_result); + let cliFailure; + try { + await cli(["decompile", path, missingAddress]); + } catch (cause) { + cliFailure = cause; + } + assert.ok(cliFailure && typeof cliFailure.stdout === "string"); + const error = JSON.parse(cliFailure.stdout); + assert.equal(error.code, "invalid_request"); + assert.ok(JSON.stringify(error).includes(`Ghidra image base: ${base}`)); + for (const command of [ + "rea search . --kind procedures --mode regex", + "rea analyze ", + "rea inspect-native-load-image ", + ]) + assert.ok(JSON.stringify(error).includes(command)); + const cliProcedures = parseEvidence( + JSON.parse( + ( + await cli([ + "search", + path, + ".", + "--kind", + "procedures", + "--mode", + "regex", + ]) + ).stdout, + ), + ); + const cliEntry = cliProcedures.normalized_result.find( + (item) => item.value === "rea_pie_entry", + ); + assert.ok(cliEntry); + assert.equal(cliEntry.address, entry.address); + const cliImage = parseEvidence( + JSON.parse((await cli(["inspect-native-load-image", path])).stdout), + ); + assert.equal(cliImage.normalized_result.observations.image_base, base); + const cliPseudo = parseEvidence( + JSON.parse((await cli(["decompile", path, cliEntry.address])).stdout), + ); + const cliOffset = parseEvidence( + JSON.parse( + (await cli(["decompile", path, `--file-offset=0x${offset.toString(16)}`])) + .stdout, + ), + ); + assert.equal(cliOffset.normalized_result, fromOffset.normalized_result); + assert.deepEqual(cliOffset.locations, fromOffset.locations); + assert.match(cliPseudo.normalized_result, /rea_pie_add/u); + assert.deepEqual(await readFile(path), bytes); + assert.deepEqual(await readdir(runtime), []); + report = { + ok: true, + mocked: false, + provider: image.provider, + compiler: version.stdout.split("\n")[0], + sha256, + declared_elf_entry: declaredEntry, + loaded_entry: entry.address, + observed_image_base: base, + cli_mcp_overview_parity: true, + failed_lookup_guidance: true, + cli_recovery_commands: true, + loaded_decompilation: true, + file_offset_decompilation: true, + address_file_offset_round_trip: true, + source_unchanged: true, + runtime_cleanup: "complete", + }; +} catch (cause) { + primaryFailure = { cause }; + if (stderr) process.stderr.write(stderr); +} + +const cleanupFailures = []; +// Await provider-owned cleanup before the SDK can terminate its MCP parent. +if (connected) { + try { + await call("close_binary"); + } catch (cause) { + cleanupFailures.push(cause); + } +} +try { + await client.close(); +} catch (cause) { + cleanupFailures.push(cause); +} +try { + await transport.close(); +} catch (cause) { + cleanupFailures.push(cause); +} +try { + assert.deepEqual(await readdir(runtime), []); + verifierRun = await completeVerifierRun(run); + assert.equal(verifierRun.process_lineage.status, "verified"); + assert.deepEqual(verifierRun.process_lineage.descendants, []); +} catch (cause) { + cleanupFailures.push(cause); +} +if (cleanupFailures.length === 0) { + try { + await rm(workspace, { recursive: true, force: true }); + } catch (cause) { + cleanupFailures.push(cause); + } +} +if (cleanupFailures.length > 0) { + process.stderr.write( + `Retained verifier workspace after cleanup failure: ${workspace}\n`, + ); + throw new AggregateError( + primaryFailure === undefined + ? cleanupFailures + : [primaryFailure.cause, ...cleanupFailures], + "Ghidra address-context verifier cleanup failed", + ); +} +if (primaryFailure !== undefined) throw primaryFailure.cause; +process.stdout.write( + JSON.stringify({ verifier_run: verifierRun, ...report }) + "\n", +); + +async function call(name, args = {}) { + const result = await client.callTool( + { name, arguments: args }, + { timeout: 240000 }, + ); + assert.notEqual(result.isError, true, JSON.stringify(result)); + return ["open_binary", "binary_session", "close_binary"].includes(name) + ? requireMcpOperationResult(result, name) + : result.structuredContent; +} diff --git a/scripts/verify-real-ghidra-swift.mjs b/scripts/verify-real-ghidra-swift.mjs index 0593408b0..0aa58e2f0 100644 --- a/scripts/verify-real-ghidra-swift.mjs +++ b/scripts/verify-real-ghidra-swift.mjs @@ -45,8 +45,13 @@ const transport = new StdioClientTransport({ const source = fileURLToPath( new URL("../tests/conformance/swift/type-inventory.swift", import.meta.url), ); +const literalSource = fileURLToPath( + new URL("../tests/conformance/swift/string-literals.swift", import.meta.url), +); const binary = join(root, "swift-inventory"); +const literalBinary = join(root, "swift-literals"); const reports = []; +let literalReport; try { await execute("swiftc", ["-O", "-g", source, "-o", binary], { timeout: 60_000, @@ -54,6 +59,19 @@ try { await mkdir(join(root, "copied")); const copied = join(root, "copied", "swift-inventory"); await copyFile(binary, copied); + // Decoding covers AArch64 register pairs, so build that slice on any Mac host. + await execute( + "swiftc", + [ + "-O", + "-target", + "arm64-apple-macosx13.0", + literalSource, + "-o", + literalBinary, + ], + { timeout: 60_000 }, + ); await client.connect(transport); for (const [variant, path] of [ ["debug", binary], @@ -132,6 +150,7 @@ try { await call("close_binary", {}); } } + literalReport = await verifyStringLiterals(); } finally { await client.close(); await transport.close(); @@ -143,6 +162,7 @@ console.log( ok: true, provider: "ghidra", fixtures: reports, + swift_string_literals: literalReport, verifier_run: await completeVerifierRun(run), }, null, @@ -159,3 +179,51 @@ async function call(name, arguments_) { toolContract(name).outputSchema.parse(response.structuredContent); return requireMcpOperationResult(response, name); } + +// Literal text lives only in register constants, so it must come from the decoded pairs. +async function verifyStringLiterals() { + await call("open_binary", { path: literalBinary, provider_id: "ghidra" }); + try { + const procedures = await call("list_procedures", {}); + const expected = { + smallLiteral: [["SleepDisabled", "small"]], + largeLiteral: [["The power setting could not be changed.", "large"]], + unicodeLiterals: [ + ["café", "small"], + ["Résumé of the naïve décor, written twice", "large"], + ], + emptyAndControlLiterals: [ + ["", "small"], + ["tab\there", "small"], + ], + returnedLiteral: [["Holding awake", "small"]], + }; + const decoded = {}; + for (const [name, literals] of Object.entries(expected)) { + const procedure = procedures.find(({ value }) => value.includes(name)); + assert(procedure, `${name} is missing from the procedure inventory`); + const dossier = await call("analyze_function", { + procedure: procedure.address, + }); + const observed = dossier.swift_string_literals; + assert(Array.isArray(observed), `${name} recorded no Swift literals`); + assert.deepEqual(dossier.swift_string_literal_omissions, []); + assert.deepEqual( + observed.map(({ value, representation }) => [value, representation]), + literals, + ); + for (const literal of observed) { + assert.deepEqual(literal.registers, ["x0", "x1"]); + assert.equal( + literal.address === null, + literal.representation === "small", + `${literal.value} storage does not match its representation`, + ); + } + decoded[name] = observed.length; + } + return decoded; + } finally { + await call("close_binary", {}); + } +} diff --git a/scripts/verify-real-ghidra-windows.mjs b/scripts/verify-real-ghidra-windows.mjs index 5adf96009..bc822211f 100644 --- a/scripts/verify-real-ghidra-windows.mjs +++ b/scripts/verify-real-ghidra-windows.mjs @@ -28,6 +28,7 @@ import { import { parseBinaryTarget } from "../dist/application/BinaryTargetResolver.js"; import { completeVerifierRun, createVerifierRun } from "./lib/verifier-run.mjs"; import { verifyGhidraCompactReferences } from "./lib/real-ghidra-compact-references.mjs"; +import { verifyGhidraFileOffsets } from "./lib/real-ghidra-file-offsets.mjs"; const verifierRun = createVerifierRun(); const SESSION_CAPABILITIES = GHIDRA_SESSION_CAPABILITIES.filter( @@ -300,6 +301,11 @@ try { } if (report === undefined) throw new Error("Windows Ghidra report was not produced"); +report.file_offsets = await verifyGhidraFileOffsets( + targetPath, + resolve(root, "scripts", "rea.mjs"), + architecture === "x86" ? "0x401010" : "0x140001010", +); process.stdout.write( `${JSON.stringify({ verifier_run: await completeVerifierRun(verifierRun), ...report })}\n`, ); diff --git a/scripts/verify-real-hopper-linux.mjs b/scripts/verify-real-hopper-linux.mjs index 854a58ce0..951b71078 100644 --- a/scripts/verify-real-hopper-linux.mjs +++ b/scripts/verify-real-hopper-linux.mjs @@ -11,7 +11,8 @@ createVerifierRun(); if (process.platform !== "linux") throw new Error("Linux real-Hopper verification must run on Linux"); -const launcher = process.env.HOPPER_LAUNCHER_PATH ?? "/opt/hopper/bin/Hopper"; +const configuredLauncher = process.env.HOPPER_LAUNCHER_PATH; +const launcher = configuredLauncher ?? "/opt/hopper/bin/Hopper"; await access(launcher); await access("/usr/bin/Xvfb"); await access("/usr/bin/xauth"); @@ -28,5 +29,7 @@ try { process.env.HOPPER_LAUNCHER_PATH = alternateLauncher; await import("./verify-real-hopper.mjs"); } finally { + if (configuredLauncher === undefined) delete process.env.HOPPER_LAUNCHER_PATH; + else process.env.HOPPER_LAUNCHER_PATH = configuredLauncher; await rm(alternateDirectory, { recursive: true, force: true }); } diff --git a/scripts/verify-real-hopper.mjs b/scripts/verify-real-hopper.mjs index 12400cb94..8cad9a256 100644 --- a/scripts/verify-real-hopper.mjs +++ b/scripts/verify-real-hopper.mjs @@ -47,6 +47,7 @@ import { } from "./lib/real-hopper-boundaries.mjs"; import { startUnrelatedHopperSentinel } from "./lib/unrelated-hopper-sentinel.mjs"; import { snapshotHopperRuntime } from "./lib/real-hopper-cleanup.mjs"; +import { verifyLinuxHopperOwnerExit } from "./lib/real-hopper-owner-exit.mjs"; const execFileAsync = promisify(execFile); const verifierRun = createVerifierRun(); const timeout = 180_000; @@ -577,6 +578,14 @@ if (sessionsAfter.length > 0) { } if (summary === undefined) throw new Error("Real-Hopper verification did not produce a summary"); +if (process.platform === "linux") + summary.ownerExit = await verifyLinuxHopperOwnerExit({ + primary: targetA, + secondary: targetB, + serverCommand, + serverArgs, + environment: serverEnvironment, + }); const completedVerifierRun = await completeVerifierRun(verifierRun); if ( completedVerifierRun.process_lineage.status !== "verified" || diff --git a/scripts/verify-real-ida.mjs b/scripts/verify-real-ida.mjs index cedbf5076..bed0519a9 100644 --- a/scripts/verify-real-ida.mjs +++ b/scripts/verify-real-ida.mjs @@ -1,4 +1,8 @@ #!/usr/bin/env node +import { + assertFeatureTrace, + findMappedNonFunction, +} from "./lib/ida-feature-trace.mjs"; import { requireMcpToolError } from "./lib/mcp-verifier-results.mjs"; import assert from "node:assert/strict"; @@ -6,7 +10,7 @@ import { execFile } from "node:child_process"; import { createHash } from "node:crypto"; import { createReadStream } from "node:fs"; import { access, writeFile } from "node:fs/promises"; -import { dirname, join, resolve } from "node:path"; +import { basename, dirname, join, resolve } from "node:path"; import { fileURLToPath, pathToFileURL } from "node:url"; import { parseArgs, promisify } from "node:util"; import { Client } from "@modelcontextprotocol/client"; @@ -261,6 +265,61 @@ const mcp = async (cliDossier) => { break; } } + if (config.mode === "headless") { + const documents = observe(await call("list_documents", {})); + assert.equal(documents.length, 1); + assert.equal(basename(documents[0]), basename(target)); + assert.ok( + !workspaces.has(dirname(documents[0])), + "list_documents reported the private workspace copy.", + ); + const resolveAt = async (address) => + observe(await call("resolve_containing_procedure", { address })); + const instructionAddresses = dossier.assembly + .map((line) => /^(0x[0-9a-f]+)/iu.exec(line)?.[1]) + .filter((address) => address !== undefined); + const queries = [ + dossier.procedure.address, + ...instructionAddresses.slice(1, 2), + ]; + for (const queried of queries) { + const containing = await resolveAt(queried); + assert.equal(containing.found, true); + assert.equal(containing.procedure.address, dossier.procedure.address); + assert.equal(containing.procedure.name, dossier.procedure.name); + } + const unmapped = await resolveAt("0xffffffffffffff00"); + assert.equal(unmapped.found, false); + assert.equal(unmapped.reason, "outside_segments"); + const strings = observe(await call("list_strings", {})); + const mappedString = await findMappedNonFunction(strings, resolveAt); + if (mappedString === undefined) + report.unverified.push( + "Mapped non-function containment: no string of the target is mapped outside every function.", + ); + const context = await client.callTool({ + name: "inspect_address_context", + arguments: { address: dossier.procedure.address }, + }); + assert.notEqual(context.isError, true); + const contextResult = context.structuredContent.result; + assert.equal(contextResult.document, documents[0]); + assert.equal(contextResult.procedure.state, "available"); + assert.equal( + contextResult.procedure.value.procedure.address, + dossier.procedure.address, + ); + const traced = await client.callTool({ + name: "trace_feature", + arguments: { query: dossier.procedure.name }, + }); + assert.notEqual(traced.isError, true); + const trace = traced.structuredContent.normalized_result; + assertFeatureTrace(trace, dossier.procedure); + report.checks.push( + "headless containing procedures, the admitted program as the document, address context, and the procedure match and references of feature tracing", + ); + } if (values["expected-string"] === undefined) report.unverified.push( "String inventory/search correctness: no independent expected literal supplied.", diff --git a/scripts/verify-signal-recording.mjs b/scripts/verify-signal-recording.mjs index a095b97ff..f5baaf0b9 100644 --- a/scripts/verify-signal-recording.mjs +++ b/scripts/verify-signal-recording.mjs @@ -1,5 +1,6 @@ #!/usr/bin/env node import assert from "node:assert/strict"; +import { estimateTokenCount } from "tokenx"; import { execFile } from "node:child_process"; import { createHash } from "node:crypto"; import { readFile, writeFile, readdir } from "node:fs/promises"; @@ -68,6 +69,20 @@ for (const byte of Buffer.from("REA")) { lines.push(`#${tick + 1000}`, "1!"); const source = lines.join("\n") + "\n"; let cases = 0; +const discoveryMeasurements = []; +const inheritedAddressSpace = (await readFile("/proc/self/limits", "utf8")) + .split("\n") + .find((line) => line.startsWith("Max address space ")) + ?.trim() + .match(/^Max address space\s+(\S+)\s+(\S+)\s+bytes$/u); +assert.ok(inheritedAddressSpace); +const expectedAddressSpace = Math.min( + 1073741824, + ...inheritedAddressSpace + .slice(1) + .map((value) => (value === "unlimited" ? Infinity : Number(value))), +); + try { await writeFile(path, source); await client.connect(transport); @@ -104,14 +119,16 @@ try { createHash("sha256").update(source).digest("hex"), ); assert.equal(report.hardware_acquisition, "not-performed"); - assert.equal(report.limits.address_space_bytes, 1073741824); + assert.equal(report.limits.address_space_bytes, expectedAddressSpace); assert.equal(report.tool.configured_path, process.env.REA_SIGROK_COMMAND); assert.equal( report.resource_limiter.configured_path, process.env.REA_SIGROK_PRLIMIT_COMMAND ?? "/usr/bin/prlimit", ); assert.ok( - report.resource_limiter.arguments.includes("--as=1073741824:1073741824"), + report.resource_limiter.arguments.includes( + `--as=${expectedAddressSpace}:${expectedAddressSpace}`, + ), ); assert.equal(report.diagnostics, ""); const flags = Object.entries(input) @@ -245,6 +262,88 @@ try { }); assert.equal(invalid.isError, true); cases++; + const discoveryTool = (await client.listTools()).tools.find( + ({ name }) => name === "inspect_signal_decoders", + ); + assert.ok(discoveryTool?.outputSchema); + assert.ok(ajv.validateSchema(discoveryTool.inputSchema)); + assert.ok(ajv.validateSchema(discoveryTool.outputSchema)); + const nativeHelp = ( + await execute( + process.env.REA_SIGROK_COMMAND, + ["--dont-scan", "--protocol-decoders=uart", "--show"], + { env: environment, timeout: 30000, maxBuffer: 4 * 1024 * 1024 }, + ) + ).stdout.split(/\r?\n/); + if (nativeHelp.at(-1) === "") nativeHelp.pop(); + for (const input of [ + { query: "uart" }, + { limit: 1 }, + { offset: 1, limit: 2 }, + { decoder: "uart", query: "BaUd" }, + { decoder: "uart", offset: 50, limit: null }, + { query: "rea_no_such_decoder" }, + ]) { + const started = performance.now(); + const response = await client.callTool({ + name: discoveryTool.name, + arguments: input, + }); + assert.notEqual(response.isError, true, JSON.stringify(response)); + assert.ok( + ajv.validate(discoveryTool.outputSchema, response.structuredContent), + JSON.stringify(ajv.errors), + ); + const evidence = parseEvidence(response.structuredContent); + const report = evidence.normalized_result; + discoveryMeasurements.push({ + input, + elapsed_ms: +(performance.now() - started).toFixed(2), + response_bytes: Buffer.byteLength(JSON.stringify(response)), + estimated_tokens_one_tool_request_response: estimateTokenCount( + JSON.stringify(discoveryTool) + + JSON.stringify(input) + + JSON.stringify(response), + ), + }); + if (input.decoder) { + const matched = nativeHelp + .map((text, i) => ({ line_number: i + 1, text })) + .filter(({ text }) => + text.toLowerCase().includes((input.query ?? "").toLowerCase()), + ); + assert.deepEqual( + report.help_lines, + matched.slice( + input.offset ?? 0, + input.limit === null + ? undefined + : (input.offset ?? 0) + (input.limit ?? 50), + ), + ); + } else if (input.query === "uart") + assert.ok(report.decoders.some(({ id }) => id === "uart")); + const flags = Object.entries(input).flatMap(([key, value]) => + value === null ? ["--all"] : [`--${key}`, String(value)], + ); + const cli = JSON.parse( + ( + await execute( + process.execPath, + [entrypoint, "inspect-signal-decoders", ...flags, "--format", "json"], + { env: environment, timeout: 60000, maxBuffer: 8 * 1024 * 1024 }, + ) + ).stdout, + ); + assert.deepEqual(cli.normalized_result, report); + cases++; + } + const unavailable = await client.callTool({ + name: discoveryTool.name, + arguments: { decoder: "rea_no_such_decoder" }, + }); + assert.equal(unavailable.isError, true); + cases++; await client.close(); assert.deepEqual((await readdir(root.path)).sort(), [ "paged.vcd", @@ -260,6 +359,9 @@ console.log( { status: "passed", cases, + discovery_measurements: discoveryMeasurements, + measurement_scope: + "Warm MCP server, fresh sigrok processes; tokenx estimates include one advertised tool/request/response, not billed model usage. No model-driven selection benchmark, CPU or peak process-family memory measurement; no cache or speedup claim.", host: `${process.platform}/${process.arch}`, scope: "real sigrok UART VCD through compiled CLI/MCP; no physical acquisition", diff --git a/scripts/verify-windows-native-artifact.mjs b/scripts/verify-windows-native-artifact.mjs index 1a6777322..4db66e0ee 100644 --- a/scripts/verify-windows-native-artifact.mjs +++ b/scripts/verify-windows-native-artifact.mjs @@ -27,9 +27,9 @@ if ( ); const offset = bytes.length >= 64 ? bytes.readUInt32LE(0x3c) : bytes.length; if ( - bytes.toString("ascii", 0, 2) !== "MZ" || + bytes.toString("latin1", 0, 2) !== "MZ" || offset > bytes.length - 26 || - bytes.toString("ascii", offset, offset + 4) !== "PE\0\0" || + bytes.toString("latin1", offset, offset + 4) !== "PE\0\0" || bytes.readUInt16LE(offset + 4) !== 0x8664 || bytes.readUInt16LE(offset + 24) !== 0x20b ) diff --git a/scripts/verify-windows-native.mjs b/scripts/verify-windows-native.mjs index c63b6909f..8623081f9 100644 --- a/scripts/verify-windows-native.mjs +++ b/scripts/verify-windows-native.mjs @@ -11,6 +11,7 @@ import { mkdtemp, readFile, rename, + readdir, rm, symlink, writeFile, @@ -37,6 +38,14 @@ const { WindowsPrivateRuntime, windowsPrivateRuntime } = await import( const { ghidraSessionRoot } = await import( pathToFileURL(join(packageRoot, "dist/ghidra/GhidraSessionRoot.js")) ); +const { PrivateRuntimeRoot } = await import( + pathToFileURL(join(packageRoot, "dist/process/PrivateRuntimeRoot.js")) +); +const { PrivateRuntimeRootUnavailableError } = await import( + pathToFileURL( + join(packageRoot, "dist/process/PrivateRuntimeRootUnavailableError.js"), + ) +); const workspace = await mkdtemp(join(tmpdir(), "rea-native-conformance-")); const environment = Object.entries(process.env) .filter(([, value]) => value !== undefined) @@ -140,6 +149,30 @@ process.stdout.write(JSON.stringify({executable:basename(process.execPath),abiVe ]) assert.throws(() => native.call("open", [rejected])); report.controls.ordinaryDriveSeparatorsAndRequestedIdentity = true; + // An owned runtime's DELETE lease conflicts with a fresh parent admission. + // This produces a genuine sharing failure after successful authority caching, + // without another executable, and proves that no child runtime was created. + const sharingParent = WindowsPrivateRuntime.create( + workspace, + "rea-sharing-parent-", + ); + try { + await assert.rejects( + PrivateRuntimeRoot.create({ parent: sharingParent.observation.path }), + (failure) => { + assert.ok(failure instanceof PrivateRuntimeRootUnavailableError); + assert.equal(failure.retryable, true); + assert.ok(failure.cause instanceof Error); + assert.equal(failure.cause.win32Code, 32); + assert.equal(failure.cause.runtimeAllocation, "not-started"); + return true; + }, + ); + assert.deepEqual(await readdir(sharingParent.observation.path), []); + report.controls.transientRuntimeAdmissionBeforeAllocation = true; + } finally { + await sharingParent.close(); + } const driveRoot = parse(workspace).root; const runtimeParent = ghidraSessionRoot({ base: join(driveRoot, ".Tmp", "scratch"), @@ -629,6 +662,42 @@ setInterval(()=>{},1000);\n`, unicodeNamesPreserved: true, }); report.processes.windowsEnvironmentSemantics = true; + // Observed through this process's own id: a separate probe process would + // get its own processor group, so it cannot stand in for the process that + // spawns the child. + const callerAffinity = JSON.parse( + execFileSync(boundaryFixture, ["affinity", String(process.pid)], { + encoding: "utf8", + }), + ); + const affinityChild = native.call("process_spawn", [ + boundaryFixture, + `${quote(boundaryFixture)} affinity`, + workspace, + environment, + ]); + const observedAffinity = await wait(affinityChild); + native.call("process_close", [affinityChild.handle]); + assert.equal(observedAffinity.exitCode, 0); + const childAffinity = JSON.parse(observedAffinity.output); + // Masks cross as hexadecimal strings because a mask can exceed the + // safe-integer range. Sharing one processor group makes the caller's mask + // applicable, so it must be inherited exactly. Windows placing the child in + // another group is only permitted when the caller asked for no bound of its + // own, which its group's whole active set establishes; a narrower mask has + // to fail the spawn rather than widen to that group's default. + const sharedGroup = + callerAffinity.singleProcessorGroup && + childAffinity.singleProcessorGroup && + callerAffinity.processorGroup === childAffinity.processorGroup; + if (sharedGroup) + assert.equal(childAffinity.affinityMask, callerAffinity.affinityMask); + else assert.equal(callerAffinity.affinityMask, callerAffinity.systemMask); + report.processes.callerAffinityBoundsOwnedProcessOrRefusesToWiden = true; + report.callerAffinity = { + ...callerAffinity, + sharedGroupWithChild: sharedGroup, + }; } const ownerFixture = join(workspace, "owner.mjs"); const ownerPidFile = join(workspace, "crash-child.pid"); diff --git a/scripts/verify/firmware/analysis.mjs b/scripts/verify/firmware/analysis.mjs index 1f84681c9..804e8fffa 100644 --- a/scripts/verify/firmware/analysis.mjs +++ b/scripts/verify/firmware/analysis.mjs @@ -1,15 +1,8 @@ import assert from "node:assert/strict"; import { execFile } from "node:child_process"; -import { - access, - mkdir, - mkdtemp, - readFile, - rm, - symlink, -} from "node:fs/promises"; +import * as fs from "node:fs/promises"; import { tmpdir } from "node:os"; -import { dirname, join, resolve } from "node:path"; +import * as nodePath from "node:path"; import { fileURLToPath } from "node:url"; import { promisify } from "node:util"; import { Client } from "@modelcontextprotocol/client"; @@ -18,247 +11,348 @@ import { firmwareResultSchemas } from "../../../dist/domain/firmware/firmwareAna import { parseEvidence } from "../../../dist/domain/evidence.js"; import { hashFirmwareFile } from "../../../dist/firmware/FirmwareFiles.js"; -const exec = promisify(execFile); -const repository = resolve(dirname(fileURLToPath(import.meta.url)), "../../.."); -const entrypoint = resolve( +const execFileAsync = promisify(execFile); +const { join, resolve, dirname } = nodePath; + +const CALL_TIMEOUT_MS = 150_000; +const CONNECT_TIMEOUT_MS = 30_000; +const SEVENZIP_TIMEOUT_MS = 10_000; +const MAX_STDOUT_BYTES = 16 * 1024 * 1024; + +const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "../../.."); +const ENTRYPOINT = resolve( process.env.REA_FIRMWARE_TEST_ENTRYPOINT ?? - join(repository, "scripts/rea.mjs"), + join(REPO_ROOT, "scripts/rea.mjs"), ); -if (process.platform !== "linux") - throw new Error( - "verify:firmware requires Linux; other hosts are not verified", - ); -for (const variable of ["REA_BINWALK_COMMAND", "REA_UNBLOB_COMMAND"]) { - const path = process.env[variable]; - if (path === undefined) - throw new Error( - `verify:firmware requires ${variable}; see docs/firmware-analysis.md`, - ); - await access(path); -} -try { - await exec("7z", ["i"], { timeout: 10_000 }); -} catch (cause) { - throw new Error( - "verify:firmware prerequisite unavailable: Unblob's gzip/USTAR fixture extraction requires 7z on PATH", - { cause }, - ); -} -const fixtureRoot = resolve( +const FIXTURE_ROOT = resolve( process.env.REA_FIRMWARE_FIXTURE_ROOT ?? - join(repository, "_reference/firmware-integration/generated"), + join(REPO_ROOT, "_reference/firmware-integration/generated"), ); -await access(join(fixtureRoot, "oracle.json")).catch((cause) => { - throw new Error( - "verify:firmware requires generated fixtures; run npm run fixtures:firmware", - { cause }, - ); -}); + +// --------------------------------------------------------------------------- +// Prerequisite checks +// --------------------------------------------------------------------------- + +async function checkPrerequisites() { + if (process.platform !== "linux") { + throw new Error( + "verify:firmware requires Linux; other hosts are not verified", + ); + } + + for (const name of ["REA_BINWALK_COMMAND", "REA_UNBLOB_COMMAND"]) { + const value = process.env[name]; + if (value === undefined) { + throw new Error( + `verify:firmware requires ${name}; see docs/firmware-analysis.md`, + ); + } + await fs.access(value); + } + + try { + await execFileAsync("7z", ["i"], { timeout: SEVENZIP_TIMEOUT_MS }); + } catch (cause) { + throw new Error( + "verify:firmware prerequisite unavailable: Unblob's gzip/USTAR fixture extraction requires 7z on PATH", + { cause }, + ); + } + + try { + await fs.access(join(FIXTURE_ROOT, "oracle.json")); + } catch (cause) { + throw new Error( + "verify:firmware requires generated fixtures; run npm run fixtures:firmware", + { cause }, + ); + } +} + +await checkPrerequisites(); + +// --------------------------------------------------------------------------- +// Shared state +// --------------------------------------------------------------------------- + const oracle = JSON.parse( - await readFile(join(fixtureRoot, "oracle.json"), "utf8"), + await fs.readFile(join(FIXTURE_ROOT, "oracle.json"), "utf8"), ); -const path = join(fixtureRoot, "firmware.bin"); -assert.equal(await hashFirmwareFile(path), oracle.sha256); -const root = await mkdtemp(join(tmpdir(), "rea-real-firmware-")); -const environment = Object.fromEntries( - Object.entries(process.env).filter(([, value]) => value !== undefined), +const firmwarePath = join(FIXTURE_ROOT, "firmware.bin"); +assert.equal(await hashFirmwareFile(firmwarePath), oracle.sha256); + +const workDir = await fs.mkdtemp(join(tmpdir(), "rea-real-firmware-")); +const baseEnv = Object.fromEntries( + Object.entries(process.env).filter(([, v]) => v !== undefined), ); -const client = new Client({ name: "rea-real-firmware-verifier", version: "1" }); -const cli = async (args, env = environment) => { - const response = await exec( +const mcpClient = new Client({ + name: "rea-real-firmware-verifier", + version: "1", +}); + +// --------------------------------------------------------------------------- +// Invocation helpers +// --------------------------------------------------------------------------- + +async function runCli(args, env = baseEnv) { + const { stdout } = await execFileAsync( process.execPath, - [entrypoint, ...args, "--format", "json"], + [ENTRYPOINT, ...args, "--format", "json"], { env, - cwd: repository, - timeout: 150_000, - maxBuffer: 16 * 1024 * 1024, + cwd: REPO_ROOT, + timeout: CALL_TIMEOUT_MS, + maxBuffer: MAX_STDOUT_BYTES, }, ); - return parseEvidence(JSON.parse(response.stdout)); -}; -const call = async (name, args) => { - const response = await client.callTool( + return parseEvidence(JSON.parse(stdout)); +} + +async function callTool(name, args) { + const result = await mcpClient.callTool( { name, arguments: args }, - { timeout: 150_000 }, + { timeout: CALL_TIMEOUT_MS }, ); - assert.notEqual(response.isError, true, JSON.stringify(response)); - assert.ok(response.structuredContent !== undefined); - return response.structuredContent; -}; -const verifyRegions = (evidence) => { + assert.notEqual(result.isError, true, JSON.stringify(result)); + assert.ok(result.structuredContent !== undefined); + return result.structuredContent; +} + +// --------------------------------------------------------------------------- +// Result verifiers +// --------------------------------------------------------------------------- + +function verifyRegions(evidence) { assert.equal(evidence.subject.digest.sha256, oracle.sha256); - const result = firmwareResultSchemas.inspect_firmware_regions.parse( + + const parsed = firmwareResultSchemas.inspect_firmware_regions.parse( evidence.normalized_result, ); - const gzip = result.regions.find( - (region) => region.signature === "gzip" && region.offset === oracle.offset, + + const gzipRegion = parsed.regions.find( + (r) => r.signature === "gzip" && r.offset === oracle.offset, ); assert.ok( - gzip !== undefined, + gzipRegion !== undefined, "Binwalk must find the independently generated gzip offset", ); - assert.equal(gzip.reported_size, oracle.length); - assert.equal(gzip.size_basis, "provider_reported_validation_unknown"); - return result.regions.map( - ({ provider_id: _provider_id, ...region }) => region, - ); -}; -const verifyFiles = async (evidence) => { - const result = firmwareResultSchemas.extract_firmware.parse( + assert.equal(gzipRegion.reported_size, oracle.length); + assert.equal(gzipRegion.size_basis, "provider_reported_validation_unknown"); + + // Strip provider-specific ids so CLI and MCP results compare equal. + return parsed.regions.map((region) => { + const { provider_id: _ignored, ...rest } = region; + return rest; + }); +} + +async function verifyFiles(evidence) { + const parsed = firmwareResultSchemas.extract_firmware.parse( evidence.normalized_result, ); - assert.ok(result.chunks.some((chunk) => chunk.handler === "gzip")); + + assert.ok(parsed.chunks.some((c) => c.handler === "gzip")); + for (const [suffix, expected] of Object.entries(oracle.files)) { - const file = result.files.find((candidate) => - candidate.relative_path.endsWith(`/${suffix}`), + const entry = parsed.files.find((f) => + f.relative_path.endsWith(`/${suffix}`), ); - assert.ok(file !== undefined, `Missing extracted ${suffix}`); - assert.equal(file.sha256, expected.sha256); - assert.equal(file.size, expected.size); - assert.equal(await hashFirmwareFile(file.path), expected.sha256); - assert.equal(file.original_file_range, null); - assert.equal(file.runtime_address, null); + assert.ok(entry !== undefined, `Missing extracted ${suffix}`); + assert.equal(entry.sha256, expected.sha256); + assert.equal(entry.size, expected.size); + assert.equal(await hashFirmwareFile(entry.path), expected.sha256); + assert.equal(entry.original_file_range, null); + assert.equal(entry.runtime_address, null); } - return result; -}; -try { - const inspected = await cli(["inspect-firmware-regions", path]); - const cliRegions = verifyRegions(inspected); - const extracted = await cli([ - "extract-firmware", - path, - join(root, "cli-output"), + + return parsed; +} + +const toContentIdentity = (parsed) => + parsed.files.map(({ relative_path, sha256, size }) => ({ + relative_path, + sha256, + size, + })); + +// --------------------------------------------------------------------------- +// Verification stages +// --------------------------------------------------------------------------- + +async function stageCli() { + const regionsEvidence = await runCli([ + "inspect-firmware-regions", + firmwarePath, ]); - const cliFiles = await verifyFiles(extracted); - assert.equal(cliFiles.coverage, "partial"); + const regions = verifyRegions(regionsEvidence); + + const extractEvidence = await runCli([ + "extract-firmware", + firmwarePath, + join(workDir, "cli-output"), + ]); + const files = await verifyFiles(extractEvidence); + + assert.equal(files.coverage, "partial"); assert.ok( - cliFiles.chunks.some( - (chunk) => - chunk.handler === null && - chunk.range.offset === 0 && - chunk.range.length === 64, + files.chunks.some( + (c) => + c.handler === null && c.range.offset === 0 && c.range.length === 64, ), ); - await client.connect( - new StdioClientTransport({ - command: process.execPath, - args: [entrypoint, "mcp"], - cwd: repository, - env: environment, - stderr: "pipe", - }), - { timeout: 30_000 }, - ); - const inspectedMcp = await call("inspect_firmware_regions", { path }); - const mcpEvidence = parseEvidence(inspectedMcp); - assert.deepEqual(verifyRegions(mcpEvidence), cliRegions); - const extractedMcp = await call("extract_firmware", { - path, - output_directory: join(root, "mcp-output"), + + return { regions, files }; +} + +async function connectMcp() { + const transport = new StdioClientTransport({ + command: process.execPath, + args: [ENTRYPOINT, "mcp"], + cwd: REPO_ROOT, + env: baseEnv, + stderr: "pipe", }); - const mcpFiles = await verifyFiles(parseEvidence(extractedMcp)); - const contentIdentity = (result) => - result.files.map(({ relative_path, sha256, size }) => ({ - relative_path, - sha256, - size, - })); - assert.deepEqual(contentIdentity(mcpFiles), contentIdentity(cliFiles)); - const selected = await call("extract_firmware", { - path, - output_directory: join(root, "selected-output"), + await mcpClient.connect(transport, { timeout: CONNECT_TIMEOUT_MS }); +} + +async function stageMcp(cli) { + const regionsRaw = await callTool("inspect_firmware_regions", { + path: firmwarePath, + }); + assert.deepEqual(verifyRegions(parseEvidence(regionsRaw)), cli.regions); + + const extractRaw = await callTool("extract_firmware", { + path: firmwarePath, + output_directory: join(workDir, "mcp-output"), + }); + const mcpFiles = await verifyFiles(parseEvidence(extractRaw)); + assert.deepEqual(toContentIdentity(mcpFiles), toContentIdentity(cli.files)); + + const rangedRaw = await callTool("extract_firmware", { + path: firmwarePath, + output_directory: join(workDir, "selected-output"), range: { offset: oracle.offset, length: oracle.length }, }); - const selectedFiles = await verifyFiles(parseEvidence(selected)); - assert.equal(selectedFiles.selection.offset, oracle.offset); - assert.equal(selectedFiles.selection.length, oracle.length); - const depth = await call("extract_firmware", { - path, - output_directory: join(root, "depth-output"), + const ranged = await verifyFiles(parseEvidence(rangedRaw)); + assert.equal(ranged.selection.offset, oracle.offset); + assert.equal(ranged.selection.length, oracle.length); + + const depthRaw = await callTool("extract_firmware", { + path: firmwarePath, + output_directory: join(workDir, "depth-output"), max_depth: 1, }); - const depthResult = firmwareResultSchemas.extract_firmware.parse( - depth.normalized_result, + const depthParsed = firmwareResultSchemas.extract_firmware.parse( + depthRaw.normalized_result, ); - assert.equal(depthResult.coverage, "partial"); - assert.ok(depthResult.depth_limited_paths.length > 0); + assert.equal(depthParsed.coverage, "partial"); + assert.ok(depthParsed.depth_limited_paths.length > 0); + console.log( "PASS real Binwalk/Unblob: CLI, MCP, independent offsets/digests, range extraction, unknown chunks and depth limits", ); - if (process.env.REA_FIRMWARE_VERIFY_EXT4 === "1") { - const ext4 = join(fixtureRoot, "ext4.bin"); - await access(ext4); - const response = await call("extract_firmware", { - path: ext4, - output_directory: join(root, "ext4-output"), - }); - const result = firmwareResultSchemas.extract_firmware.parse( - response.normalized_result, + + return mcpFiles; +} + +async function stageExt4() { + const ext4Path = join(FIXTURE_ROOT, "ext4.bin"); + await fs.access(ext4Path); + + const raw = await callTool("extract_firmware", { + path: ext4Path, + output_directory: join(workDir, "ext4-output"), + }); + const parsed = firmwareResultSchemas.extract_firmware.parse( + raw.normalized_result, + ); + assert.ok(parsed.chunks.some((c) => c.handler === "extfs")); + assert.equal(parsed.coverage, "complete"); + + for (const [suffix, expected] of Object.entries(oracle.files)) { + const match = parsed.files.find((f) => + f.relative_path.endsWith(`/${suffix}`), ); - assert.ok(result.chunks.some((chunk) => chunk.handler === "extfs")); - assert.equal(result.coverage, "complete"); - for (const [suffix, expected] of Object.entries(oracle.files)) - assert.equal( - result.files.find((file) => file.relative_path.endsWith(`/${suffix}`)) - ?.sha256, - expected.sha256, - ); - console.log("PASS real ext4 extraction with debugfs; no mounts"); - // Preserve the process-ownership utility while withholding all extractors. - const withoutExtractors = join(root, "without-extractors"); - await access("/bin/ps"); - await mkdir(withoutExtractors); - await symlink("/bin/ps", join(withoutExtractors, "ps")); - const missing = await cli( - ["extract-firmware", ext4, join(root, "missing-extractor-output")], - { ...environment, PATH: withoutExtractors }, - ); - const partial = firmwareResultSchemas.extract_firmware.parse( - missing.normalized_result, - ); - assert.equal(partial.coverage, "partial"); - assert.ok(JSON.stringify(partial.diagnostics).includes("debugfs")); - console.log( - "PASS real missing-extractor diagnostics retained despite Unblob exit 1", - ); - } - if (process.env.REA_FIRMWARE_VERIFY_GHIDRA === "1") { - if (process.env.GHIDRA_INSTALL_DIR === undefined) - throw new Error( - "Firmware Ghidra handoff requires GHIDRA_INSTALL_DIR and Java; see docs/ghidra.md", - ); - const child = mcpFiles.files.find((file) => - file.relative_path.endsWith("/usr/bin/probe"), - ); - assert.ok(child !== undefined); - await call("open_binary", { path: child.path, provider_id: "ghidra" }); - const dossier = await call("analyze_function", { - procedure: "firmware_probe", - }); - const evidence = parseEvidence(dossier); - assert.equal( - evidence.subject.digest.sha256, - oracle.files["usr/bin/probe"].sha256, - ); - const pseudocode = await call("procedure_pseudo_code", { - procedure: "firmware_probe", - }); - assert.equal(parseEvidence(pseudocode).provider.id, "ghidra"); - assert.ok( - JSON.stringify(evidence.normalized_result).includes("firmware_probe"), - ); - await call("close_binary", {}); - console.log( - "PASS real Ghidra handoff of the selected extracted host ELF, bound to its digest", + assert.equal(match?.sha256, expected.sha256); + } + console.log("PASS real ext4 extraction with debugfs; no mounts"); + + // Keep the process-ownership utility available but hide every extractor. + const emptyBin = join(workDir, "without-extractors"); + await fs.access("/bin/ps"); + await fs.mkdir(emptyBin); + await fs.symlink("/bin/ps", join(emptyBin, "ps")); + + const degraded = await runCli( + ["extract-firmware", ext4Path, join(workDir, "missing-extractor-output")], + { ...baseEnv, PATH: emptyBin }, + ); + const degradedParsed = firmwareResultSchemas.extract_firmware.parse( + degraded.normalized_result, + ); + assert.equal(degradedParsed.coverage, "partial"); + assert.ok(JSON.stringify(degradedParsed.diagnostics).includes("debugfs")); + console.log( + "PASS real missing-extractor diagnostics retained despite Unblob exit 1", + ); +} + +async function stageGhidra(mcpFiles) { + if (process.env.GHIDRA_INSTALL_DIR === undefined) { + throw new Error( + "Firmware Ghidra handoff requires GHIDRA_INSTALL_DIR and Java; see docs/ghidra.md", ); } + + const probe = mcpFiles.files.find((f) => + f.relative_path.endsWith("/usr/bin/probe"), + ); + assert.ok(probe !== undefined); + + await callTool("open_binary", { path: probe.path, provider_id: "ghidra" }); + + const dossierRaw = await callTool("analyze_function", { + procedure: "firmware_probe", + }); + const dossier = parseEvidence(dossierRaw); assert.equal( - await hashFirmwareFile(path), + dossier.subject.digest.sha256, + oracle.files["usr/bin/probe"].sha256, + ); + + const pseudoRaw = await callTool("procedure_pseudo_code", { + procedure: "firmware_probe", + }); + assert.equal(parseEvidence(pseudoRaw).provider.id, "ghidra"); + assert.ok( + JSON.stringify(dossier.normalized_result).includes("firmware_probe"), + ); + + await callTool("close_binary", {}); + console.log( + "PASS real Ghidra handoff of the selected extracted host ELF, bound to its digest", + ); +} + +// --------------------------------------------------------------------------- +// Entry point +// --------------------------------------------------------------------------- + +try { + const cli = await stageCli(); + await connectMcp(); + const mcpFiles = await stageMcp(cli); + + if (process.env.REA_FIRMWARE_VERIFY_EXT4 === "1") await stageExt4(); + if (process.env.REA_FIRMWARE_VERIFY_GHIDRA === "1") + await stageGhidra(mcpFiles); + + assert.equal( + await hashFirmwareFile(firmwarePath), oracle.sha256, "Original firmware bytes must remain unchanged", ); } finally { - await client.close(); - await rm(root, { recursive: true, force: true }); + await mcpClient.close(); + await fs.rm(workDir, { recursive: true, force: true }); } diff --git a/scripts/verify/harmony/analysis.mjs b/scripts/verify/harmony/analysis.mjs index 3aa90c9f0..aaa401c75 100644 --- a/scripts/verify/harmony/analysis.mjs +++ b/scripts/verify/harmony/analysis.mjs @@ -153,10 +153,36 @@ try { }); try { await client.connect(transport, { timeout: 30_000 }); + await client.listTools(); + const opened = await client.callTool( + { name: "open_binary", arguments: { path } }, + { timeout: 150_000 }, + ); + assert.notEqual(opened.isError, true, JSON.stringify(opened)); + assert.equal(opened.structuredContent?.result?.format, "hap"); + assert.equal(opened.structuredContent?.result?.sha256, fixture.sha256); + const status = await client.callTool( + { name: "binary_session", arguments: {} }, + { timeout: 150_000 }, + ); + assert.notEqual(status.isError, true, JSON.stringify(status)); + assert.equal(status.structuredContent?.result?.open, true); + assert.equal(status.structuredContent?.result?.format, "hap"); + const inspected = await client.callTool( + { name: "inspect_artifact", arguments: {} }, + { timeout: 150_000 }, + ); + assert.notEqual(inspected.isError, true, JSON.stringify(inspected)); + const mcpInspection = parseEvidence(inspected.structuredContent); + assert.deepEqual( + mcpInspection.normalized_result, + inspection.normalized_result, + ); + console.log("PASS MCP open/status/inspect_artifact parity"); const response = await client.callTool( { name: "project_harmony_application_graph", - arguments: { inventory_evidence: [inspection] }, + arguments: { inventory_evidence: [mcpInspection] }, }, { timeout: 150_000 }, ); @@ -168,6 +194,23 @@ try { projectionEvidence.normalized_result, ); console.log("PASS MCP/CLI parity project_harmony_application_graph"); + const closed = await client.callTool( + { name: "close_binary", arguments: {} }, + { timeout: 150_000 }, + ); + assert.notEqual(closed.isError, true, JSON.stringify(closed)); + const empty = await client.callTool( + { name: "binary_session", arguments: {} }, + { timeout: 150_000 }, + ); + assert.notEqual(empty.isError, true, JSON.stringify(empty)); + assert.equal(empty.structuredContent?.result?.open, false); + assert.equal( + await hashFile(path), + fixture.sha256, + "MCP workflow must preserve source bytes", + ); + console.log("PASS MCP close and unchanged source bytes"); } catch (cause) { process.stderr.write(stderr); throw cause; diff --git a/scripts/website.py b/scripts/website.py index 1d6661fc1..22c0829b9 100644 --- a/scripts/website.py +++ b/scripts/website.py @@ -114,7 +114,7 @@ def shared_blocks(root, path, source): } from website_i18n import language_markup selector, alternates = language_markup(root, path.relative_to(site).as_posix()) - values["language_picker"] = selector if not metadata.noindex else "" + values["language_picker"] = indent(selector, " ") if not metadata.noindex else "" values["alternates"] = alternates if not metadata.noindex else "" templates = root / "website/templates" return { diff --git a/src/adb/AdbDeviceOutput.test.ts b/src/adb/AdbDeviceOutput.test.ts index fe11c2a26..12b82ab84 100644 --- a/src/adb/AdbDeviceOutput.test.ts +++ b/src/adb/AdbDeviceOutput.test.ts @@ -84,6 +84,24 @@ describe("parseAdbDevicesOutput", () => { ]); }); + it("accepts complete multiword states and rejects state prefixes", () => { + const output = [ + "serial-no-permission no permissions usb:1-1", + "serial-bad-device deviceXYZ ", + "serial-bad-offline offline-ish", + "serial-bad-auth unauthorizedXYZ", + ].join("\n"); + const { devices, unparsedLines } = parseAdbDevicesOutput(output); + expect(devices.map(({ serial, state }) => [serial, state])).toEqual([ + ["serial-no-permission", "no permissions"], + ]); + expect(unparsedLines).toEqual([ + "serial-bad-device deviceXYZ ", + "serial-bad-offline offline-ish", + "serial-bad-auth unauthorizedXYZ", + ]); + }); + it("returns unrecognized lines instead of dropping them", () => { const output = [ "List of devices attached", @@ -194,10 +212,10 @@ describe("parsePackageListOutput", () => { it("keeps unrecognized lines for partial coverage reporting", () => { const { packages, unparsedLines } = parsePackageListOutput( - "package:/data/app/x/base.apk=com.ok\ngarbage line\n", + "package:/data/app/x/base.apk=com.ok\n garbage line \n", ); expect(packages).toHaveLength(1); - expect(unparsedLines).toEqual(["garbage line"]); + expect(unparsedLines).toEqual([" garbage line "]); }); }); diff --git a/src/adb/AdbDeviceOutput.ts b/src/adb/AdbDeviceOutput.ts index 3380de608..8d3c74f5f 100644 --- a/src/adb/AdbDeviceOutput.ts +++ b/src/adb/AdbDeviceOutput.ts @@ -10,11 +10,15 @@ import { adbHostFileName, type AdbHostFileNameReason, } from "./AdbHostFileName.js"; +import { + ADB_DEVICE_STATES, + type AdbDeviceState, +} from "../domain/adb/adbDeviceAnalysis.js"; /** One device exactly as `adb devices -l` reports it. */ export interface AdbDeviceEntry { readonly serial: string; - readonly state: string; + readonly state: AdbDeviceState; readonly transport: "usb" | "tcp" | "unknown"; readonly product: string | null; readonly model: string | null; @@ -24,25 +28,6 @@ export interface AdbDeviceEntry { readonly kind_basis: "emulator_serial_prefix" | "usb_transport" | "none"; } -/** - * Device states adb prints after the serial, longest first so two-word - * states match before their single-word prefixes. A line whose remainder - * starts with none of them is not a device line. - */ -const DEVICE_STATES = [ - "no permissions", - "unauthorized", - "authorizing", - "offline", - "recovery", - "sideload", - "connecting", - "bootloader", - "device", - "unknown", - "host", -] as const; - const DEVICE_LINE = /^(\S+)\s+(.*)$/u; const metadataValue = (remainder: string, key: string): string | null => { @@ -73,11 +58,15 @@ export const parseAdbDevicesOutput = ( continue; } const match = DEVICE_LINE.exec(line); - const state = DEVICE_STATES.find((candidate) => - match?.[2]?.startsWith(candidate), - ); + const state = ADB_DEVICE_STATES.find((candidate) => { + const remainder = match?.[2]; + if (remainder === undefined || !remainder.startsWith(candidate)) + return false; + const next = remainder[candidate.length]; + return next === undefined || /\s/u.test(next); + }); if (match === null || state === undefined) { - if (line.trim() !== "") unparsedLines.push(line); + if (rawLine.trim() !== "") unparsedLines.push(rawLine); continue; } const serial = match[1]!; @@ -192,7 +181,7 @@ export const parsePackageListOutput = ( if (line === "") continue; const match = PACKAGE_LINE.exec(line); if (match === null) { - unparsedLines.push(line); + unparsedLines.push(rawLine); continue; } packages.push({ diff --git a/src/adb/AdbProvider.test.ts b/src/adb/AdbProvider.test.ts index d1a153848..23645e456 100644 --- a/src/adb/AdbProvider.test.ts +++ b/src/adb/AdbProvider.test.ts @@ -121,6 +121,52 @@ fi`, ]); }); +it("refuses device listings with an invalid state token", async () => { + const malformedLine = "serial deviceXYZ "; + const { binary } = await writeAdbStub( + () => `if [ "$1" = "version" ]; then ${VERSION_BLOCK}; exit 0; fi +if [ "$1" = "devices" ]; then + printf 'List of devices attached\\n${malformedLine}\\n' + exit 0 +fi`, + ); + const result = await providerOver(binary).execute({ + operation: "list_adb_devices", + input: {}, + }); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(result.error._tag).toBe("AnalysisProtocolError"); + expect(result.error.message).toContain(malformedLine); +}); + +it("lists rescue and detached transports alongside online devices", async () => { + const { binary } = await writeAdbStub( + () => `if [ "$1" = "version" ]; then ${VERSION_BLOCK}; exit 0; fi +if [ "$1" = "devices" ]; then + printf 'List of devices attached\\n' + printf 'serial-online device transport_id:1\\n' + printf 'serial-rescue rescue transport_id:2\\n' + printf 'serial-detached detached transport_id:3\\n' + exit 0 +fi`, + ); + const result = await evidenceResult(providerOver(binary), { + operation: "list_adb_devices", + input: {}, + }); + expect(result.devices).toEqual( + expect.arrayContaining([ + expect.objectContaining({ serial: "serial-online", state: "device" }), + expect.objectContaining({ serial: "serial-rescue", state: "rescue" }), + expect.objectContaining({ + serial: "serial-detached", + state: "detached", + }), + ]), + ); +}); + it("inspects a device through the getprop whitelist", async () => { const { binary } = await writeAdbStub( () => `if [ "$1" = "version" ]; then ${VERSION_BLOCK}; exit 0; fi @@ -580,6 +626,34 @@ it("projects package details from dumpsys package output", async () => { }); }); +// Android 10 and 11 append ": restricted=true" to hard- and soft-restricted +// entries of the requested-permissions list (Settings.dumpPackageLPr). +it("counts requested permissions that the device marks as restricted", async () => { + const { binary } = await writeAdbStub( + shellStub({ + "dumpsys:package": `Packages: + Package [com.example] (abcdef): + userId=10234 + versionName=1.2.3 + requested permissions: + android.permission.INTERNET + android.permission.WRITE_EXTERNAL_STORAGE: restricted=true + android.permission.CAMERA + install permissions: + android.permission.INTERNET: granted=true +`, + }), + ); + const result = await evidenceResult(providerOver(binary), { + operation: "inspect_adb_package", + input: { serial: "emulator-5554", package: "com.example" }, + }); + expect(result).toMatchObject({ + requested_permissions_count: 3, + coverage: "complete", + }); +}); + it("reports an uninstalled package from dumpsys refusal output", async () => { const { binary } = await writeAdbStub( shellStub({ "dumpsys:package": "Unable to find package: com.absent\n" }), @@ -736,6 +810,36 @@ exit 1`, expect(result.error.message).toContain("overwrite"); }); +it.each([ + ["offline", false, "error: device offline"], + ["unauthorized", false, "error: device unauthorized"], + ["offline", true, "error: device offline"], +] as const)( + "reports the %s device state when the push existence probe cannot run (overwrite %s)", + async (state, overwrite, stderr) => { + const { binary, root } = await writeAdbStub( + (stubRoot) => `if [ "$1" = "version" ]; then ${VERSION_BLOCK}; exit 0; fi +if [ "$1" = "-s" ]; then printf '%s\\n' "$3" >> "${stubRoot}/calls"; printf '%s\\n' "${stderr}" >&2; exit 1; fi`, + ); + const localFile = join(root, "payload.bin"); + await writeFile(localFile, "x"); + const projected = await refusedBy(providerOver(binary), { + operation: "push_adb_file", + input: { + serial: "emulator-5554", + local_path: localFile, + device_path: "/data/local/tmp/payload.bin", + overwrite, + }, + }); + // Existence is unknown, so nothing is claimed about the path and no push runs. + expect(projected).toMatchObject({ code: "unsupported_target" }); + expect(projected.message).toContain(state); + expect(projected.message).not.toContain("already exists"); + expect(await readFile(join(root, "calls"), "utf8")).toBe("shell\n"); + }, +); + it("reports a local push source that does not exist", async () => { const { binary } = await writeAdbStub( () => `if [ "$1" = "version" ]; then ${VERSION_BLOCK}; exit 0; fi diff --git a/src/adb/AdbProvider.ts b/src/adb/AdbProvider.ts index b9af6d14f..d26345eba 100644 --- a/src/adb/AdbProvider.ts +++ b/src/adb/AdbProvider.ts @@ -1623,17 +1623,15 @@ export class AdbProvider { { timeoutMs: SETTINGS_TIMEOUT_MS, maxOutputBytes: 1024 }, signal, ); - let deviceExists: boolean; - if (existsProbe.ok) deviceExists = true; - else if (existsProbe.error instanceof AnalysisProtocolError) - deviceExists = false; - else if (existsProbe.error instanceof AnalysisTimeoutError) + // `test -e` exits nonzero with no adb diagnosis when the path is absent. + // Any other probe failure leaves existence unknown, so it is reported as + // is rather than read as an existing path or followed by a push. + if ( + !existsProbe.ok && + !(existsProbe.error instanceof AnalysisProtocolError) + ) return existsProbe; - else if (existsProbe.error instanceof AnalysisCancelledError) - return existsProbe; - else if (existsProbe.error instanceof AnalysisResourceConstraintError) - return existsProbe; - else deviceExists = true; // authorization/refusal states surface on push instead + const deviceExists = existsProbe.ok; if (deviceExists && !input.overwrite) return err( new AnalysisOutputError( diff --git a/src/adb/AdbShellOutput.ts b/src/adb/AdbShellOutput.ts index f3323e7b2..726e6164d 100644 --- a/src/adb/AdbShellOutput.ts +++ b/src/adb/AdbShellOutput.ts @@ -236,7 +236,11 @@ export const parseDumpsysPackageOutput = ( inPermissions = true; continue; } - if (inPermissions && /^\s{2,}[A-Za-z0-9._]+$/.test(line)) { + // Android 10 and 11 mark restricted entries as "NAME: restricted=true". + if ( + inPermissions && + /^\s{2,}[A-Za-z0-9._]+(?:: restricted=true)?$/u.test(line) + ) { permissions += 1; continue; } diff --git a/src/android/JadxMcpTransport.ts b/src/android/JadxMcpTransport.ts index c648696fc..a40d517f5 100644 --- a/src/android/JadxMcpTransport.ts +++ b/src/android/JadxMcpTransport.ts @@ -199,9 +199,11 @@ export class JadxMcpTransport implements Transport { throw new Error(`JADX MCP frame exceeds ${MAX_FRAME_BYTES} bytes`); this.#appendFrame(fragment, frameBytes); if (newline === -1) return; - const line = this.#frameBuffer - .subarray(0, this.#frameLength) - .toString("utf8"); + const line = new TextDecoder("utf-8", { + fatal: true, + // Preserve a leading BOM so the JSON-RPC parser can reject it. + ignoreBOM: true, + }).decode(this.#frameBuffer.subarray(0, this.#frameLength)); this.#frameBuffer = Buffer.alloc(0); this.#frameLength = 0; this.onmessage?.(deserializeMessage(line)); diff --git a/src/apktool/ApktoolProvider.test.ts b/src/apktool/ApktoolProvider.test.ts index 5b2a32114..b6ee249e5 100644 --- a/src/apktool/ApktoolProvider.test.ts +++ b/src/apktool/ApktoolProvider.test.ts @@ -96,6 +96,7 @@ const decodeStub = ( readonly withStrings?: boolean; readonly withGerman?: boolean; readonly failDecode?: boolean; + readonly ymlFile?: string; readonly manifestFile?: string; readonly stringsFile?: string; readonly stringsXml?: string; @@ -104,6 +105,10 @@ const decodeStub = ( const withStrings = options.withStrings ?? true; const withGerman = options.withGerman ?? true; const stringsXml = options.stringsXml ?? REAL_STRINGS; + const ymlWrite = + options.ymlFile === undefined + ? `printf '%s\\n' '${REAL_YML.replace(/'/gu, "'\\''")}' > "$5/apktool.yml"` + : `cp '${options.ymlFile}' "$5/apktool.yml"`; const manifestWrite = options.manifestFile === undefined ? `printf '%s\\n' '${REAL_MANIFEST.replace(/'/gu, "'\\''")}' > "$5/AndroidManifest.xml"` @@ -117,7 +122,7 @@ if [ "$1" = "d" ]; then ${ options.failDecode === true ? `printf 'Exception in thread "main" brut.androlib.AndrolibException: Could not decode the APK\\n' >&2; exit 1;` - : `printf '%s\\n' '${REAL_YML.replace(/'/gu, "'\\''")}' > "$5/apktool.yml" + : `${ymlWrite} ${manifestWrite} mkdir -p "$5/res/values"${withGerman ? ' "$5/res/values-de"' : ""} ${withStrings ? stringsWrite : ""} @@ -192,6 +197,47 @@ it("decodes resources with target identity, metadata, and strings", async () => expect(result.coverage).toBe("complete"); }); +it.each([ + { + name: "malformed UTF-8", + bytes: Buffer.concat([ + Buffer.from("versionInfo:\n versionName: com."), + Buffer.from([0xff]), + Buffer.from(".app\n"), + ]), + valid: false, + }, + { + name: "a literal replacement character", + bytes: Buffer.from("versionInfo:\n versionName: com.�.app\n", "utf8"), + valid: true, + }, +])("handles $name in producer metadata", async ({ bytes, valid }) => { + const { command, root } = await writeApktoolStub((root) => + decodeStub({ ymlFile: join(root, "malformed-apktool.yml") })(root), + ); + await writeFile(join(root, "malformed-apktool.yml"), bytes); + const apk = await apkFixture(root); + if (valid) { + const result = await evidenceResult(providerOver(command), { + operation: "decode_android_resources", + input: { path: apk.path, include_strings: false }, + }); + expect(result.metadata).toMatchObject({ version_name: "com.�.app" }); + expect(result.coverage).toBe("complete"); + return; + } + const result = await providerOver(command).execute({ + operation: "decode_android_resources", + input: { path: apk.path, include_strings: false }, + }); + if (result.ok) throw new Error("expected invalid producer output"); + const projected = projectAnalysisError(result.error); + expect(projected).toMatchObject({ code: "unreadable_output" }); + expect(result.error.message).toContain("invalid UTF-8"); + expect(result.error.message).toContain("apktool.yml"); +}); + it("preserves empty resource strings and reports markup omissions as partial", async () => { const stringsXml = ' A & B Hello world'; diff --git a/src/apktool/ApktoolProvider.ts b/src/apktool/ApktoolProvider.ts index 682e2f2ea..1e1f187c0 100644 --- a/src/apktool/ApktoolProvider.ts +++ b/src/apktool/ApktoolProvider.ts @@ -117,13 +117,20 @@ const readBoundedText = async ( throwIfCancelled(operation, signal); const truncated = length > budget; // A prefix may end within a UTF-8 character; omit those trailing bytes. - const decoder = new TextDecoder("utf-8"); - return { - text: decoder.decode(bytes.subarray(0, Math.min(length, budget)), { - stream: truncated, - }), - truncated, - }; + try { + const decoder = new TextDecoder("utf-8", { fatal: true }); + return { + text: decoder.decode(bytes.subarray(0, Math.min(length, budget)), { + stream: truncated, + }), + truncated, + }; + } catch (cause: unknown) { + throw new AnalysisProtocolError( + `apktool produced invalid UTF-8 in ${path}`, + { cause }, + ); + } } finally { await file.close(); } diff --git a/src/application/AnalysisProvider.ts b/src/application/AnalysisProvider.ts index cc17cac82..376036fa1 100644 --- a/src/application/AnalysisProvider.ts +++ b/src/application/AnalysisProvider.ts @@ -50,10 +50,27 @@ export interface AnalysisExecution { readonly limitations: readonly string[]; /** Provider-owned completeness assessment for caller/callee observations. */ readonly relationshipCoverage?: RelationshipCoverage; + /** + * Whether the provider capped an `xrefs` read: true when it reported a + * truncation, false when it explicitly reported none. Absent when it reports + * nothing, which is not a completeness claim. Never part of Evidence or tool + * output. + */ + readonly referencesTruncated?: boolean; readonly locations: readonly EvidenceLocation[]; readonly subject: EvidenceSubjectTarget | null; } +interface CreateAnalysisExecutionOptions { + readonly rawResult?: unknown; + readonly analysisProfile?: AnalysisProfileCommitment; + readonly limitations?: readonly string[]; + readonly relationshipCoverage?: RelationshipCoverage; + readonly referencesTruncated?: boolean; + readonly locations?: readonly EvidenceLocation[]; + readonly subject?: EvidenceSubjectTarget; +} + /** * Build a validated atomic provider observation at an adapter boundary. * @@ -61,38 +78,46 @@ export interface AnalysisExecution { * `result`. Omitting it records `null` rather than a second complete copy of * the normalized result in every Evidence record and tool response. */ -export const createAnalysisExecution = ( +export function createAnalysisExecution( result: unknown, provider: ProviderIdentity, - options: { - readonly rawResult?: unknown; - readonly analysisProfile?: AnalysisProfileCommitment; - readonly limitations?: readonly string[]; - readonly relationshipCoverage?: RelationshipCoverage; - readonly locations?: readonly EvidenceLocation[]; - readonly subject?: EvidenceSubjectTarget; - } = {}, -): AnalysisExecution => ({ - result: jsonValueSchema.parse(result), - rawResult: - options.rawResult === undefined - ? null - : jsonValueSchema.parse(options.rawResult), - provider, - ...(options.analysisProfile === undefined - ? {} - : { analysisProfile: structuredClone(options.analysisProfile) }), - limitations: [...(options.limitations ?? [])], - ...(options.relationshipCoverage === undefined - ? {} - : { - relationshipCoverage: relationshipCoverageSchema.parse( - options.relationshipCoverage, - ), - }), - locations: [...(options.locations ?? [])], - subject: options.subject ?? null, -}); + options: CreateAnalysisExecutionOptions & { readonly subject: Subject }, +): Omit & { readonly subject: Subject }; +export function createAnalysisExecution( + result: unknown, + provider: ProviderIdentity, + options?: CreateAnalysisExecutionOptions, +): AnalysisExecution; +export function createAnalysisExecution( + result: unknown, + provider: ProviderIdentity, + options: CreateAnalysisExecutionOptions = {}, +): AnalysisExecution { + return { + result: jsonValueSchema.parse(result), + rawResult: + options.rawResult === undefined + ? null + : jsonValueSchema.parse(options.rawResult), + provider, + ...(options.analysisProfile === undefined + ? {} + : { analysisProfile: structuredClone(options.analysisProfile) }), + limitations: [...(options.limitations ?? [])], + ...(options.relationshipCoverage === undefined + ? {} + : { + relationshipCoverage: relationshipCoverageSchema.parse( + options.relationshipCoverage, + ), + }), + ...(options.referencesTruncated === undefined + ? {} + : { referencesTruncated: options.referencesTruncated }), + locations: [...(options.locations ?? [])], + subject: options.subject ?? null, + }; +} /** Provider-neutral application capability for one analysis operation. */ export interface AnalysisOperationPort { diff --git a/src/application/BinaryTargetResolver.ts b/src/application/BinaryTargetResolver.ts index ac691b55e..6f1bc4366 100644 --- a/src/application/BinaryTargetResolver.ts +++ b/src/application/BinaryTargetResolver.ts @@ -76,6 +76,8 @@ export const parseBinaryTarget = async ( await access(candidate, constants.R_OK); throwIfTargetResolutionCancelled(signal); const canonical = await realpath(candidate); + // Windows native admission must inspect the caller-selected reparse components. + const sourcePath = process.platform === "win32" ? candidate : canonical; throwIfTargetResolutionCancelled(signal); const resolved = await resolveAppBundle(canonical); throwIfTargetResolutionCancelled(signal); @@ -113,7 +115,7 @@ export const parseBinaryTarget = async ( ); return ok({ path, - sourcePath: canonical, + sourcePath, sha256: await sha256Handle(handle, path, metadata, signal), kind: "executable", format: "raw-binary", @@ -152,7 +154,7 @@ export const parseBinaryTarget = async ( ); return ok({ path, - sourcePath: canonical, + sourcePath, sha256: await sha256Handle(handle, path, metadata, signal), kind: "executable", format: "dos-com", @@ -166,7 +168,7 @@ export const parseBinaryTarget = async ( ) return ok({ path, - sourcePath: process.platform === "win32" ? candidate : canonical, + sourcePath, sha256: await sha256Handle(handle, path, metadata, signal), kind: "database", format: "analysis-database", @@ -175,7 +177,7 @@ export const parseBinaryTarget = async ( if (artifactFormat !== undefined) { const identity = { path, - sourcePath: process.platform === "win32" ? candidate : canonical, + sourcePath, sha256: await sha256Handle(handle, path, metadata, signal), }; return isArchiveFormat(artifactFormat) @@ -195,7 +197,7 @@ export const parseBinaryTarget = async ( ); return ok({ path, - sourcePath: process.platform === "win32" ? candidate : canonical, + sourcePath, ...bundle, sha256: await sha256Handle(handle, path, metadata, signal), kind: "executable", diff --git a/src/application/CapabilityInventory.test.ts b/src/application/CapabilityInventory.test.ts index 1b8ec6411..4286ff1d0 100644 --- a/src/application/CapabilityInventory.test.ts +++ b/src/application/CapabilityInventory.test.ts @@ -497,3 +497,71 @@ it("reports host denial before target mismatch but keeps composed workflow prere ).reason, ).toBe("target_required"); }); + +describe("trace_feature requirements", () => { + const capabilities = (operations: readonly string[]) => + operations.map((operation) => ({ + operation, + available: true, + reason: null, + })); + + it("needs the search operations the workflow actually calls", () => { + expect( + entry( + "trace_feature", + status({ + open: true, + kind: "executable", + capabilities: capabilities([ + "search_strings", + "search_procedures", + "xrefs", + "resolve_containing_procedure", + ]), + }), + ), + ).toMatchObject({ available: true, reason: "available" }); + }); + + it.each(["search_strings", "search_procedures"])( + "is unavailable without %s", + (missing) => { + const operations = [ + "search_strings", + "search_procedures", + "xrefs", + "resolve_containing_procedure", + ].filter((operation) => operation !== missing); + + expect( + entry( + "trace_feature", + status({ + open: true, + kind: "executable", + capabilities: capabilities(operations), + }), + ), + ).toMatchObject({ available: false, reason: "provider_missing" }); + }, + ); + + it("is unavailable when only the list operations exist", () => { + expect( + entry( + "trace_feature", + status({ + open: true, + kind: "executable", + capabilities: capabilities([ + "list_strings", + "list_procedures", + "xrefs", + "resolve_containing_procedure", + ]), + }), + ), + ).toMatchObject({ available: false, reason: "provider_missing" }); + }); +}); diff --git a/src/application/CapabilityInventory.ts b/src/application/CapabilityInventory.ts index c0f341506..f4584eb29 100644 --- a/src/application/CapabilityInventory.ts +++ b/src/application/CapabilityInventory.ts @@ -121,8 +121,8 @@ const ENHANCED_REQUIREMENTS: Readonly> = { analyze_function: ["analyze_function"], inspect_native_api: ["analyze_function"], trace_feature: [ - "list_strings", - "list_procedures", + "search_strings", + "search_procedures", "xrefs", "resolve_containing_procedure", ], @@ -141,6 +141,7 @@ const FRIDA_OPERATION_NAMES: ReadonlySet = new Set( /** Native contracts defined only for an active Mach-O executable. */ const MACHO_TARGET_OPERATIONS: ReadonlySet = new Set([ + "inspect_signature", "inspect_macho", "list_architectures", "observe_native_calls", @@ -397,7 +398,7 @@ const workflowAvailabilityFor = ({ remediation: "On Linux x64, set absolute REA_PWNTOOLS_PYTHON to caller-supplied Python with pwntools 4.15.0. No active binary target is required.", }; - if (name === "inspect_packet_capture") + if (name === "inspect_packet_capture" || name === "inspect_packet_fields") return policy.packetCaptureEnabled === true ? { reason: "available", remediation: null } : { @@ -405,7 +406,7 @@ const workflowAvailabilityFor = ({ remediation: "On Linux, supply TShark 4.x via absolute REA_TSHARK_COMMAND and util-linux prlimit (REA_TSHARK_PRLIMIT_COMMAND, default /usr/bin/prlimit). No active binary target or hardware is required; the tool is checked at execution.", }; - if (name === "decode_signal_recording") + if (name === "decode_signal_recording" || name === "inspect_signal_decoders") return policy.signalRecordingEnabled === true ? { reason: "available", remediation: null } : { @@ -481,7 +482,7 @@ const workflowAvailabilityFor = ({ remediation: "Run process capture on a supported Linux or macOS host.", }; if (kind === "application") return { reason: "available", remediation: null }; - if (kind === "flutter-provider") + if (kind === "pcap-provider" || kind === "flutter-provider") return { reason: "available", remediation: null }; const electron = electronProviderAvailability(name, kind, policy); if (electron !== null) return electron; diff --git a/src/application/ClientConfigurationDocument.ts b/src/application/ClientConfigurationDocument.ts index b32df3fd0..1b241a98b 100644 --- a/src/application/ClientConfigurationDocument.ts +++ b/src/application/ClientConfigurationDocument.ts @@ -1,9 +1,5 @@ import { parse as parseToml, stringify as stringifyToml } from "smol-toml"; -import { - parse as parseYaml, - parseDocument as parseYamlDocument, - stringify as stringifyYaml, -} from "yaml"; +import { stringify as stringifyYaml } from "yaml"; import { isDeepStrictEqual } from "node:util"; import { @@ -12,6 +8,10 @@ import { editDshDocument, dshClientRegistrationEntry, } from "./DshCordisPatch.js"; +import { + editHermesConfigurationYaml, + parseHermesConfigurationYaml, +} from "./HermesConfigurationYaml.js"; import { applyEdits, findNodeAtLocation, @@ -118,7 +118,10 @@ const parseDocument = ( if (isEmptyClientConfigurationText(text)) return {}; // Hermes stores MCP servers in `mcp_servers` inside a YAML document. if (format === "hermes") - return requireObject(parseYaml(text), "The configuration root"); + return requireObject( + parseHermesConfigurationYaml(text), + "The configuration root", + ); // ZCode loads nonempty native files with strict JSON.parse. if (format === "zcode") return requireObject(JSON.parse(text), "The configuration root"); @@ -1419,12 +1422,11 @@ const serializeHermesConfiguration = ( typeof servers === "object" && servers !== null ? (servers as Record)[PRODUCT_IDENTITY.mcpServerKey] : undefined; - const yamlDocument = parseYamlDocument(originalText); - if (entry === undefined) - yamlDocument.deleteIn([serversKey, PRODUCT_IDENTITY.mcpServerKey]); - else yamlDocument.setIn([serversKey, PRODUCT_IDENTITY.mcpServerKey], entry); - // Editing REA must not reflow unrelated user prompts or other YAML scalars. - return yamlDocument.toString({ lineWidth: 0 }); + return editHermesConfigurationYaml( + originalText, + [serversKey, PRODUCT_IDENTITY.mcpServerKey], + entry, + ); }; /** Build the stdio entry shape expected by one client's configuration dialect. */ diff --git a/src/application/ClientRegistrationStatus.ts b/src/application/ClientRegistrationStatus.ts index e2696e088..caeba4fb6 100644 --- a/src/application/ClientRegistrationStatus.ts +++ b/src/application/ClientRegistrationStatus.ts @@ -7,6 +7,7 @@ import { clientServerForcedEnabled, clientServerListedDisabled, effectiveClientServer, + type ClientConfigurationDocument, } from "./ClientConfigurationDocument.js"; import { access } from "node:fs/promises"; import { resolve } from "node:path"; @@ -74,24 +75,48 @@ const registrationSchema = z }) .passthrough(); -/** Inspect supported client registrations without reading their environment. */ +interface ClientRegistrationReadOptions { + readonly platform?: NodeJS.Platform; + readonly environment?: NodeJS.ProcessEnv; + readonly resources?: ArtifactResourceScope; +} + +/** Registration facts shared by diagnostics and update maintenance. */ +export interface ClientRegistrationInspection { + readonly status: ClientRegistrationStatus; + readonly activation: "enabled" | "disabled" | "unknown"; +} + +/** Project diagnostics from the same inspection used to select maintenance. */ export const readClientRegistrationStatuses = async ( home: string, currentCommandPath: string = resolve(process.argv[1] ?? "unknown"), - options: { - readonly platform?: NodeJS.Platform; - readonly environment?: NodeJS.ProcessEnv; - readonly resources?: ArtifactResourceScope; - } = {}, -): Promise => { - const statuses: ClientRegistrationStatus[] = []; + options: ClientRegistrationReadOptions = {}, +): Promise => + ( + await readClientRegistrationInspections(home, currentCommandPath, options) + ).map(({ status }) => status); + +/** Parse effective registrations and their client-specific activation once. */ +export const readClientRegistrationInspections = async ( + home: string, + currentCommandPath: string = resolve(process.argv[1] ?? "unknown"), + options: ClientRegistrationReadOptions = {}, +): Promise => { + const inspections: ClientRegistrationInspection[] = []; + const record = ( + status: ClientRegistrationStatus, + activation: ClientRegistrationInspection["activation"] = "unknown", + ): void => { + inspections.push({ status, activation }); + }; for (const client of supportedClients( home, options.platform, options.environment, )) { if (client.configPathError !== undefined) { - statuses.push({ + record({ client: client.name, config_path: client.configPath, command: [], @@ -104,7 +129,7 @@ export const readClientRegistrationStatuses = async ( const manualRemediation = manualRegistrationRemediation(client.name); if (client.format === "unsupported") { if (manualRemediation !== undefined) - statuses.push({ + record({ client: client.name, config_path: client.markerPath ?? client.configPath, command: [], @@ -119,7 +144,7 @@ export const readClientRegistrationStatuses = async ( options.resources, ); if (!files.ok) { - statuses.push({ + record({ client: client.name, config_path: client.configPath, command: [], @@ -130,9 +155,7 @@ export const readClientRegistrationStatuses = async ( } const parsed = effectiveClientConfiguration(files.value); if (parsed === undefined) { - statuses.push( - unavailableStatus(client.name, client.configPath, "missing"), - ); + record(unavailableStatus(client.name, client.configPath, "missing")); continue; } const policy = await readClientPolicyBlock( @@ -142,7 +165,7 @@ export const readClientRegistrationStatuses = async ( options.resources, ); if (!policy.ok) { - statuses.push({ + record({ client: client.name, config_path: client.configPath, command: [], @@ -159,7 +182,7 @@ export const readClientRegistrationStatuses = async ( ? dshDocument(parsed.document).editabilityError : undefined; if (editabilityError !== undefined) { - statuses.push({ + record({ client: client.name, config_path: client.configPath, command: [], @@ -168,9 +191,7 @@ export const readClientRegistrationStatuses = async ( }); continue; } - statuses.push( - unavailableStatus(client.name, client.configPath, "missing"), - ); + record(unavailableStatus(client.name, client.configPath, "missing")); continue; } const registration = parseRegistration(raw, client); @@ -178,21 +199,18 @@ export const readClientRegistrationStatuses = async ( registration.command, ...registration.args, ]; - statuses.push( + const activation = registrationActivation(registration, client, parsed); + record( configuredStatus( client.name, client.configPath, command, - registrationAligned( - registration, - client, - currentCommandPath, - options.platform ?? process.platform, - clientServerForcedEnabled(parsed, PRODUCT_IDENTITY.mcpServerKey), - ) && - !clientServerListedDisabled( - parsed, - PRODUCT_IDENTITY.mcpServerKey, + activation === "enabled" && + registrationAligned( + registration, + client, + currentCommandPath, + options.platform ?? process.platform, ) && (parsed.dialect !== "dsh" || dshDocument(parsed.document).ready) && policyBlock === undefined @@ -200,9 +218,10 @@ export const readClientRegistrationStatuses = async ( : "stale", policyBlock, ), + activation, ); } catch (cause: unknown) { - statuses.push( + record( unavailableStatus( client.name, client.configPath, @@ -211,26 +230,31 @@ export const readClientRegistrationStatuses = async ( ); } } - return statuses.sort((left, right) => - compareUnicodeCodePoints(left.client, right.client), + return inspections.sort((left, right) => + compareUnicodeCodePoints(left.status.client, right.status.client), ); }; +const registrationActivation = ( + registration: z.output, + client: SetupClient, + parsed: ClientConfigurationDocument, +): "enabled" | "disabled" => + (client.format !== "grok" && registration.disabled === true) || + (registration.enabled === false && + !clientServerForcedEnabled(parsed, PRODUCT_IDENTITY.mcpServerKey)) || + (client.format === "zcode" && registration.enable === false) || + clientServerListedDisabled(parsed, PRODUCT_IDENTITY.mcpServerKey) + ? "disabled" + : "enabled"; + const registrationAligned = ( registration: z.output, client: SetupClient, currentCommandPath: string, platform: NodeJS.Platform, - forcedEnabled = false, ): boolean => { const command = [registration.command, ...registration.args]; - if ( - (client.format !== "grok" && registration.disabled === true) || - (registration.enabled === false && !forcedEnabled) || - // ZCode's native flag is `enable`; other dialects leave that field opaque. - (client.format === "zcode" && registration.enable === false) - ) - return false; if (!isOwnedClientRegistrationCommand(command, currentCommandPath)) return false; if ( diff --git a/src/application/DirectAnalysis.ts b/src/application/DirectAnalysis.ts index c4d130f63..0bdb69864 100644 --- a/src/application/DirectAnalysis.ts +++ b/src/application/DirectAnalysis.ts @@ -23,6 +23,7 @@ import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; import type { AnalysisError } from "../domain/analysisErrorBase.js"; import { access } from "node:fs/promises"; import { readAnalysisSnapshot } from "./binary/AnalysisSnapshotFiles.js"; +import { requiresNativeSourceAdmissionBeforeReplay } from "./binary/AnalysisSnapshotCache.js"; import { parseBinaryTarget } from "./BinaryTargetResolver.js"; import type { BinaryTarget } from "../domain/binaryTargetTypes.js"; import { @@ -254,9 +255,14 @@ const runAnalysis = async ( const route = preview.value.route; const bindingProfile = route.profile ?? undefined; const evidenceProfile = analysisProfileForRoute(route, tool); + // A parsed Windows path does not prove Ghidra's native snapshot owner + // admitted the selected source; CLI replay must wait for that client. if ( bindingProfile !== undefined && evidenceProfile !== undefined && + !requiresNativeSourceAdmissionBeforeReplay( + route.binding?.identity.id, + ) && allowsSnapshotReplay(route, tool) ) { const cached = snapshotEvidenceForQuery(snapshot, { diff --git a/src/application/EnhancedLiteralTracing.ts b/src/application/EnhancedLiteralTracing.ts index 9c86781bc..79684ce0f 100644 --- a/src/application/EnhancedLiteralTracing.ts +++ b/src/application/EnhancedLiteralTracing.ts @@ -1,5 +1,8 @@ import { z } from "zod"; -import type { AnalysisOperation } from "./AnalysisProvider.js"; +import type { + AnalysisExecution, + AnalysisOperation, +} from "./AnalysisProvider.js"; import { AnalysisOutputError } from "../domain/analysisErrorCore.js"; import type { AnalysisError } from "../domain/analysisErrorBase.js"; import type { JsonValue } from "../domain/jsonValue.js"; @@ -16,6 +19,12 @@ type AnalysisCall = ( signal?: AbortSignal, ) => Promise>; +type ObserveCall = ( + name: AnalysisOperation, + arguments_: Readonly>, + signal?: AbortSignal, +) => Promise>; + type LiteralSearchSource = readonly [ "search_strings" | "search_procedures", "string" | "procedure", @@ -39,6 +48,7 @@ export interface LiteralTraceInput { /** Trace matching strings and procedures through provider xrefs. */ export const traceLiteralFeature = async ( call: AnalysisCall, + observe: ObserveCall, input: LiteralTraceInput, signal?: AbortSignal, ): EnhancedResult => { @@ -49,7 +59,7 @@ export const traceLiteralFeature = async ( signal, ); if (!searched.ok) return searched; - const traced = await traceReferences(call, { + const traced = await traceReferences(call, observe, { matches: searched.value.matches, ...(signal === undefined ? {} : { signal }), }); @@ -59,8 +69,8 @@ export const traceLiteralFeature = async ( search_mode: "literal", matches: searched.value.matches, references: traced.value.references, - truncated: false, - residual_unknowns: [], + truncated: traced.value.residual.length > 0, + residual_unknowns: traced.value.residual, }); }; @@ -126,24 +136,39 @@ const addPageMatches = (input: { const traceReferences = async ( call: AnalysisCall, + observe: ObserveCall, request: ReferenceTraceRequest, -): Promise> => { +): Promise< + Result<{ references: TraceReference[]; residual: string[] }, AnalysisError> +> => { const references: TraceReference[] = []; + const residual: string[] = []; + // A source address is resolved once however many matches reference it. + const containing = new Map(); for (const match of request.matches) { - const xrefs = await call( + const observed = await observe( "xrefs", { address: match.address }, request.signal, ); - if (!xrefs.ok) return xrefs; - if (!Array.isArray(xrefs.value)) + if (!observed.ok) return observed; + // Only an explicit producer report of truncation is a coverage gap; a + // provider that reports nothing keeps its previous, unqualified result. + const unknown = `References to ${match.address} were truncated by the provider; additional references may exist.`; + if ( + observed.value.referencesTruncated === true && + !residual.includes(unknown) + ) + residual.push(unknown); + const sources = observed.value.result; + if (!Array.isArray(sources)) return err( new AnalysisOutputError( "xrefs", "provider returned a non-array result", ), ); - for (const source of xrefs.value) { + for (const source of sources) { if (typeof source !== "string") return err( new AnalysisOutputError( @@ -151,18 +176,23 @@ const traceReferences = async ( "provider returned a non-address value", ), ); - const resolved = await call( - "resolve_containing_procedure", - { address: source }, - request.signal, - ); - if (!resolved.ok) return resolved; + let containingProcedure = containing.get(source); + if (containingProcedure === undefined) { + const resolved = await call( + "resolve_containing_procedure", + { address: source }, + request.signal, + ); + if (!resolved.ok) return resolved; + containingProcedure = resolved.value; + containing.set(source, containingProcedure); + } references.push({ target_address: match.address, source_address: source, - containing_procedure: resolved.value, + containing_procedure: containingProcedure, }); } } - return ok({ references }); + return ok({ references, residual }); }; diff --git a/src/application/EnhancedTools.ts b/src/application/EnhancedTools.ts index c8b80519f..207186ccc 100644 --- a/src/application/EnhancedTools.ts +++ b/src/application/EnhancedTools.ts @@ -21,8 +21,8 @@ import { parseFunctionDossier, parseListCount, parseRelatedAddresses, - parseSegmentImageBase, parseSegments, + parseSegmentImageBase, } from "../domain/hopperValues.js"; import { err, ok, type Result } from "../domain/result.js"; import { @@ -46,6 +46,7 @@ import { traceCallPath } from "./CallPathTracing.js"; import { resolveProcedureAddress } from "./ProcedureAddressResolution.js"; import { traceLiteralFeature } from "./EnhancedLiteralTracing.js"; import { projectNativeApiInspection } from "./native/NativeApiInspection.js"; + /** * Composes direct provider operations into bounded reverse-engineering tools. * Direct callers use `execute`; adapters that own validation use @@ -162,6 +163,8 @@ export class EnhancedTools { return traceLiteralFeature( (name, arguments_, operationSignal) => this.#call(name, arguments_, operationSignal), + (name, arguments_, operationSignal) => + this.#observe(name, arguments_, operationSignal), call.input, signal, ); @@ -528,24 +531,29 @@ export class EnhancedTools { const segments = parseSegments(segmentsResult.value); if (!segments.ok) return segments; - const imageBase = parseSegmentImageBase(segmentsResult.value); const procedureCount = parseListCount(proceduresResult.value, "procedures"); if (!procedureCount.ok) return procedureCount; const stringCount = parseListCount(stringsResult.value, "strings"); if (!stringCount.ok) return stringCount; + const imageBase = parseSegmentImageBase(segments.value); + return ok({ document: document.value, - segments: segments.value.map(({ name, start, end }) => ({ - name, - start, - end, - length: addressDistance(start, end), - })), + ...(imageBase === undefined ? {} : { image_base: imageBase }), + segments: segments.value.map( + ({ name, start, end, image_base, address_space }) => ({ + name, + start, + end, + length: addressDistance(start, end), + ...(image_base === undefined ? {} : { image_base }), + ...(address_space === undefined ? {} : { address_space }), + }), + ), segment_count: segments.value.length, procedure_count: procedureCount.value, string_count: stringCount.value, - ...(imageBase === undefined ? {} : { image_base: imageBase }), }); } diff --git a/src/application/HermesConfigurationYaml.test.ts b/src/application/HermesConfigurationYaml.test.ts new file mode 100644 index 000000000..41b74fcc9 --- /dev/null +++ b/src/application/HermesConfigurationYaml.test.ts @@ -0,0 +1,21 @@ +import { describe, expect, it } from "vitest"; + +import { + editHermesConfigurationYaml, + parseHermesConfigurationYaml, +} from "./HermesConfigurationYaml.js"; + +describe("Hermes configuration YAML", () => { + it("writes an edited number from its new value", () => { + expect( + editHermesConfigurationYaml("mode: 0600\nmask: 0x1F\n", ["mode"], 384), + ).toBe("mode: 384\nmask: 0x1F\n"); + }); + + it("accepts keys Hermes reads as distinct and rejects a repeated key", () => { + expect(() => + parseHermesConfigurationYaml("0600: octal\n600: decimal\n"), + ).not.toThrow(); + expect(() => parseHermesConfigurationYaml('a: 1\n"a": 2\n')).toThrow(); + }); +}); diff --git a/src/application/HermesConfigurationYaml.ts b/src/application/HermesConfigurationYaml.ts new file mode 100644 index 000000000..8d245da37 --- /dev/null +++ b/src/application/HermesConfigurationYaml.ts @@ -0,0 +1,74 @@ +import { + isScalar, + parse, + parseDocument, + type ParsedNode, + type ScalarTag, + type Tags, +} from "yaml"; + +const NUMBER_TAGS = new Set([ + "tag:yaml.org,2002:int", + "tag:yaml.org,2002:float", +]); + +/** + * Write a parsed number with its source spelling while that spelling still + * reads as the node's value. Hermes reads config.yaml as YAML 1.1 with exact + * integers, so re-printing a JavaScript number changes what it reads: a + * Discord channel ID loses its low digits, `0600` (octal) becomes `600`, and + * `0.0000001` becomes `1e-7`, which YAML 1.1 does not define as a float. + */ +const keepNumberSource = (tag: Tags[number]): Tags[number] => { + if (typeof tag === "string" || tag.collection !== undefined) return tag; + const { stringify } = tag; + if (!NUMBER_TAGS.has(tag.tag) || stringify === undefined) return tag; + return { + ...tag, + stringify: (item, context, ...rest) => { + const { source } = item; + if (source === undefined) return stringify(item, context, ...rest); + const reader = context.doc.schema.tags.find( + (candidate): candidate is ScalarTag => + candidate.default === true && candidate.test?.test(source) === true, + ); + const read = reader?.resolve( + source, + () => undefined, + context.doc.options, + ); + return Object.is(isScalar(read) ? read.value : read, item.value) + ? source + : stringify(item, context, ...rest); + }, + }; +}; + +const HERMES_YAML_OPTIONS = { + customTags: (tags: Tags) => tags.map(keepNumberSource), + // Hermes keeps integers exact, so IDs that round to one double, or `0600` + // and `600`, are distinct keys. + uniqueKeys: (a: ParsedNode, b: ParsedNode) => + a === b || + (isScalar(a) && + isScalar(b) && + a.value === b.value && + a.source === b.source), +}; + +/** Parse a Hermes `config.yaml`. */ +export const parseHermesConfigurationYaml = (text: string): unknown => + parse(text, HERMES_YAML_OPTIONS); + +/** Set or delete one value in a Hermes `config.yaml`, keeping the rest as written. */ +export const editHermesConfigurationYaml = ( + text: string, + path: readonly string[], + value: unknown, +): string => { + const document = parseDocument(text, HERMES_YAML_OPTIONS); + if (value === undefined) document.deleteIn(path); + else document.setIn(path, value); + // Editing REA must not reflow unrelated user prompts or other YAML scalars. + return document.toString({ lineWidth: 0 }); +}; diff --git a/src/application/InvestigationProviders.ts b/src/application/InvestigationProviders.ts index 883d3878c..af23463f4 100644 --- a/src/application/InvestigationProviders.ts +++ b/src/application/InvestigationProviders.ts @@ -18,11 +18,13 @@ export const workflowAnalysisProfile = ( ): AnalysisProfileCommitment => createAnalysisProfile(REA_WORKFLOW_PROVIDER, { upstream_analysis_profile: upstream, - ...(operation === "binary_overview" || - operation === "batch_decompile" || - operation === "get_call_graph" - ? { workflow_revision: 2 } - : {}), + ...(operation === "binary_overview" + ? { workflow_revision: 3 } + : operation === "batch_decompile" || + operation === "get_call_graph" || + operation === "trace_feature" + ? { workflow_revision: 2 } + : {}), }); /** Provider identity for deterministic artifact inventories. */ diff --git a/src/application/JsonInputFile.ts b/src/application/JsonInputFile.ts index a226cb0cc..25e5fcec0 100644 --- a/src/application/JsonInputFile.ts +++ b/src/application/JsonInputFile.ts @@ -78,7 +78,9 @@ export const readJsonObjectFile = ( value === null || Array.isArray(value) ) - throw new TypeError("JSON document root must be an object"); + return err( + invalidJson(operation, "JSON document root must be an object"), + ); for (const members of scans) for (const [key, member] of members) { if (!Object.hasOwn(value, key)) continue; @@ -177,7 +179,9 @@ export const readJsonObjectFile = ( signal === undefined ? {} : { signal }, ); if (!rootObject) - throw new TypeError("JSON document root must be an object"); + return err( + invalidJson(operation, "JSON document root must be an object"), + ); const digest = hash.digest("hex"); if (previousDigest !== undefined && digest !== previousDigest) throw new RegularFileChangedError(path); diff --git a/src/application/MacHopper.test.ts b/src/application/MacHopper.test.ts index d75959fd5..7896208c5 100644 --- a/src/application/MacHopper.test.ts +++ b/src/application/MacHopper.test.ts @@ -2,12 +2,16 @@ import { createHash } from "node:crypto"; import { describe, expect, it } from "vitest"; -import { installMacHopper, type MacHopperInstallHost } from "./MacHopper.js"; +import { + installMacHopper, + type MacHopperBundleVerification, + type MacHopperInstallHost, +} from "./MacHopper.js"; class FakeMacHopperHost implements MacHopperInstallHost { readonly archive = new TextEncoder().encode("hopper-dmg"); archiveOk = true; - valid = true; + verification: MacHopperBundleVerification = "verified"; existing = false; mountSucceeds = true; copySucceeds = true; @@ -15,6 +19,7 @@ class FakeMacHopperHost implements MacHopperInstallHost { badHash = false; unmounts = 0; cleanups = 0; + installs = 0; opens = 0; replaces = false; @@ -47,7 +52,7 @@ class FakeMacHopperHost implements MacHopperInstallHost { return Promise.resolve(); }; appBundle = () => Promise.resolve("/mounted/Hopper Disassembler.app"); - validBundle = () => Promise.resolve(this.valid); + validBundle = () => Promise.resolve(this.verification); destinationExists = () => Promise.resolve(this.existing); installBundle = ( _source: string, @@ -55,6 +60,7 @@ class FakeMacHopperHost implements MacHopperInstallHost { _stage: string, replaceExisting: boolean, ) => { + this.installs += 1; this.replaces = replaceExisting; return Promise.resolve(this.copySucceeds); }; @@ -115,7 +121,7 @@ describe("macOS Hopper installation", () => { it("cleans and detaches after bundle validation fails", async () => { const host = new FakeMacHopperHost(); - host.valid = false; + host.verification = "invalid_signature"; await expect(installMacHopper({}, host)).resolves.toEqual({ status: "failed", reason: "bundle", @@ -123,4 +129,17 @@ describe("macOS Hopper installation", () => { expect(host.unmounts).toBe(1); expect(host.cleanups).toBe(1); }); + + it("reports a publisher mismatch without copying or launching the bundle", async () => { + const host = new FakeMacHopperHost(); + host.verification = "unverified_publisher"; + await expect(installMacHopper({}, host)).resolves.toEqual({ + status: "failed", + reason: "publisher_verification_failed", + }); + expect(host.installs).toBe(0); + expect(host.opens).toBe(0); + expect(host.unmounts).toBe(1); + expect(host.cleanups).toBe(1); + }); }); diff --git a/src/application/MacHopper.ts b/src/application/MacHopper.ts index 31ca522f4..a211e9f2d 100644 --- a/src/application/MacHopper.ts +++ b/src/application/MacHopper.ts @@ -24,7 +24,35 @@ const execFileAsync = promisify(execFile); const RELEASES_URL = "https://www.hopperapp.com/include/files-api.php?request=releases&public=true"; const DOWNLOAD_PREFIX = "https://www.hopperapp.com:443/downloader/public/"; -const BUNDLE_IDENTIFIER = "com.cryptic-apps.hopper-web-4"; + +/** + * Apple publisher identity pinned for the official Hopper macOS bundle. The + * team identifier was captured from `Hopper-6.6.0-demo.dmg` at + * `https://www.hopperapp.com:443/downloader/public/Hopper-6.6.0-demo.dmg` + * (34846271 bytes, SHA-1 0be530f038249742590992199c7b042ea20ca955) with + * `codesign -dvvv`, which reported + * `Authority=Developer ID Application: Cryptic Apps (2AMA2753NF)`. + */ +export const HOPPER_PUBLISHER = Object.freeze({ + bundleIdentifier: "com.cryptic-apps.hopper-web-4", + teamIdentifier: "2AMA2753NF", +}); + +/** + * Code requirement enforcing the bundle identifier and signing team together. + * The identifier is read from the sealed CodeDirectory, so it cannot be changed + * by editing `Info.plist` the way the previous string comparison could. + */ +const HOPPER_PUBLISHER_REQUIREMENT = + `= identifier "${HOPPER_PUBLISHER.bundleIdentifier}"` + + ` and anchor apple generic` + + ` and certificate leaf[subject.OU] = "${HOPPER_PUBLISHER.teamIdentifier}"`; + +/** Outcome of reading a mounted bundle's signature and publisher identity. */ +export type MacHopperBundleVerification = + | "verified" + | "invalid_signature" + | "unverified_publisher"; /** Stable network and command effects disclosed before macOS Hopper setup. */ export const macHopperInstallDisclosure = { @@ -32,6 +60,7 @@ export const macHopperInstallDisclosure = { commands: [ "hdiutil attach -readonly -nobrowse -mountpoint ", "codesign --verify --deep --strict ", + `codesign --verify --strict '-R${HOPPER_PUBLISHER_REQUIREMENT}' `, "ditto ", "open ", ] as const, @@ -52,6 +81,7 @@ export type MacHopperInstallResult = | "integrity" | "mount" | "bundle" + | "publisher_verification_failed" | "destination_exists" | "copy" | "launcher_missing"; @@ -69,7 +99,7 @@ export interface MacHopperInstallHost { mount(packagePath: string, mountPath: string): Promise; unmount(mountPath: string): Promise; appBundle(mountPath: string): Promise; - validBundle(appPath: string): Promise; + validBundle(appPath: string): Promise; destinationExists(path: string): Promise; installBundle( source: string, @@ -124,8 +154,12 @@ export const installMacHopper = async ( return { status: "failed", reason: "mount" }; mounted = true; const source = await host.appBundle(mountPath); - if (source === undefined || !(await host.validBundle(source))) + if (source === undefined) return { status: "failed", reason: "bundle" }; + const verification = await host.validBundle(source); + if (verification === "invalid_signature") return { status: "failed", reason: "bundle" }; + if (verification === "unverified_publisher") + return { status: "failed", reason: "publisher_verification_failed" }; const stage = `${destination}.rea-stage`; if ( !(await host.installBundle( @@ -222,25 +256,21 @@ export const systemMacHopperInstallHost = ( environment, )) ) - return false; - try { - const identifier = ( - await execFileAsync( - "/usr/libexec/PlistBuddy", - [ - "-c", - "Print :CFBundleIdentifier", - join(appPath, "Contents/Info.plist"), - ], - { env: environment }, - ) - ).stdout.trim(); - return identifier === BUNDLE_IDENTIFIER; - } catch (cause: unknown) { - // best-effort cleanup: optional bundle probing; unreadable means invalid. - void cause; - return false; - } + return "invalid_signature"; + if ( + !(await commandSucceeds( + "codesign", + [ + "--verify", + "--strict", + `-R${HOPPER_PUBLISHER_REQUIREMENT}`, + appPath, + ], + environment, + )) + ) + return "unverified_publisher"; + return "verified"; }, async destinationExists(path) { try { diff --git a/src/application/ProgressReporter.ts b/src/application/ProgressReporter.ts index cf5ed7b7f..bb5a98f0d 100644 --- a/src/application/ProgressReporter.ts +++ b/src/application/ProgressReporter.ts @@ -23,7 +23,7 @@ export const createProgressReporter = ( } = {}, ): ProgressReporter => { const minimumIntervalMs = options.minimumIntervalMs ?? 100; - const now = options.now ?? Date.now; + const now = options.now ?? (() => performance.now()); let lastCompleted = -1; let lastSentAt = Number.NEGATIVE_INFINITY; let sequence = 0; diff --git a/src/application/ReferenceSourceImportEntries.ts b/src/application/ReferenceSourceImportEntries.ts index 3254d7cfb..49c6f82f5 100644 --- a/src/application/ReferenceSourceImportEntries.ts +++ b/src/application/ReferenceSourceImportEntries.ts @@ -12,12 +12,14 @@ import { detectReferenceSourceLanguage, } from "../domain/referenceSourceClassification.js"; import type { HistoricalSourceGraphInput } from "../domain/referenceSourceGraph.js"; -import { parseReferenceSourceImports } from "../domain/referenceSourceImportParsing.js"; +import { + parseReferenceSourceImports, + PARSEABLE_REFERENCE_SOURCE_LANGUAGES, +} from "../domain/referenceSourceImportParsing.js"; import type { ReferenceSourceEntry, ReferenceSourceRead, } from "../reference/ReferenceSourceReaderTypes.js"; -import { PARSEABLE_REFERENCE_SOURCE_LANGUAGES } from "./ReferenceSourceImportTypes.js"; import { typescriptEmittedSourceCandidates } from "./TypeScriptEmittedSourceCandidates.js"; export interface ParsedReferenceSourceEntries { @@ -95,14 +97,22 @@ const resolveInternalSpecifier = ( ); if (logicalPathEscapesRoot(normalized)) return { to: normalized, resolution: "unresolved" }; + // Preserve directory intent before normalization removes terminal dot segments. + const directoryReference = + specifier.endsWith("/") || [".", ".."].includes(posix.basename(specifier)); const suffixes = ["", ".ts", ".js", ".mjs", ".cjs", ".tsx", ".jsx"]; const candidates = suffixes.flatMap((suffix) => [ - `${normalized}${suffix}`, - `${normalized}/index${suffix}`, + ...(directoryReference ? [] : [`${normalized}${suffix}`]), + posix.join(normalized, `index${suffix}`), ]); // Retain present runtime files first; substitute only a missing counterpart. // Counterparts cover explicit filenames, not directory/index resolution. - candidates.push(...typescriptEmittedSourceCandidates(fromPath, normalized)); + if (!directoryReference) + candidates.push( + ...typescriptEmittedSourceCandidates(fromPath, normalized).map( + ({ path }) => path, + ), + ); const match = lookupPath.resolveCandidates(candidates); return match === undefined ? { to: normalized, resolution: "unresolved" } diff --git a/src/application/ReferenceSourceImportTypes.ts b/src/application/ReferenceSourceImportTypes.ts index 9395dc5c0..f8474315a 100644 --- a/src/application/ReferenceSourceImportTypes.ts +++ b/src/application/ReferenceSourceImportTypes.ts @@ -87,6 +87,3 @@ export const DEFAULT_REFERENCE_SOURCE_IGNORE_PATTERNS = [ ".coverage", "*.log", ] as const; - -export const PARSEABLE_REFERENCE_SOURCE_LANGUAGES: ReadonlySet = - new Set(["JavaScript", "TypeScript", "JSX", "TSX"]); diff --git a/src/application/RegularFileRead.ts b/src/application/RegularFileRead.ts index f09f4c192..ee4d6170a 100644 --- a/src/application/RegularFileRead.ts +++ b/src/application/RegularFileRead.ts @@ -19,6 +19,17 @@ export interface RegularFileReadOptions { readonly resources?: ArtifactResourceScope; } +export class InvalidUtf8FileError extends TypeError { + readonly _tag = "InvalidUtf8FileError"; + constructor( + readonly path: string, + cause: unknown, + ) { + super(`File is not valid UTF-8: ${path}`, { cause }); + this.name = this._tag; + } +} + export type RegularFileReadOutcome = | { readonly kind: "completed"; readonly value: Value } | { readonly kind: "failed"; readonly cause: unknown }; @@ -142,7 +153,34 @@ const withRegularFileOwned = async ( export const readRegularFile = ( path: string, options: RegularFileReadOptions = {}, -): Promise => +): Promise => readRegularFileContent(path, (bytes) => bytes, options); + +/** Read valid UTF-8 without replacing source bytes or stripping an initial BOM. */ +export const readRegularFileText = ( + path: string, + options: RegularFileReadOptions = {}, +): Promise => + readRegularFileContent( + path, + (bytes) => { + try { + return new TextDecoder("utf-8", { + fatal: true, + ignoreBOM: true, + }).decode(bytes); + } catch (cause: unknown) { + if (!(cause instanceof TypeError)) throw cause; + throw new InvalidUtf8FileError(path, cause); + } + }, + options, + ); + +const readRegularFileContent = ( + path: string, + decode: (bytes: Buffer) => Value, + options: RegularFileReadOptions, +): Promise => withRegularFile( path, async (handle, stats) => { @@ -153,30 +191,7 @@ export const readRegularFile = ( ); if (bytes === undefined || bytes.length !== stats.size) throw new RegularFileChangedError(path); - return bytes; + return decode(bytes); }, options, ); - -/** Read one admitted regular file as UTF-8 without waiting on a pipe. */ -export const readRegularFileText = async ( - path: string, - options: RegularFileReadOptions = {}, -): Promise => { - try { - return (await readRegularFile(path, options)).toString("utf8"); - } catch (cause: unknown) { - if ( - cause instanceof RegularFileCleanupFailure && - cause.outcome.kind === "completed" && - Buffer.isBuffer(cause.outcome.value) - ) - throw new RegularFileCleanupFailure( - cause.path, - cause.owner, - { kind: "completed", value: cause.outcome.value.toString("utf8") }, - cause.cleanupCause, - ); - throw cause; - } -}; diff --git a/src/application/Setup.recovery.test.ts b/src/application/Setup.recovery.test.ts index 951ef3a54..f29c9d375 100644 --- a/src/application/Setup.recovery.test.ts +++ b/src/application/Setup.recovery.test.ts @@ -54,6 +54,40 @@ describe("setup workflow", () => { expect(result.remediation).toContain("could not be updated"); }); + it("reports a failed config rollback with its destination and recovery backup", async () => { + const host = new FakeSetupHost(); + host.hopper = "/Applications/Hopper"; + host.clients = [{ name: "codex", configPath: "/fixture/codex.json" }]; + host.clientResults.set("codex", { + status: "failed", + reason: "rollback", + backupPath: "/fixture/codex.json.rea.backup", + failure: { + kind: "restore", + path: "/fixture/codex.json", + detail: "permission denied", + }, + }); + + const result = await runSetup( + { ...options(true), clientIds: ["codex"] }, + host, + ); + + expect(result.clients.codex).toEqual({ + status: "failed", + reason: "rollback", + backupPath: "/fixture/codex.json.rea.backup", + failure: { + kind: "restore", + path: "/fixture/codex.json", + detail: "permission denied", + }, + }); + expect(result.remediation).toContain("permission denied"); + expect(result.remediation).toContain("/fixture/codex.json.rea.backup"); + }); + it("explains skill installation recovery", async () => { const host = new FakeSetupHost(); host.hopper = "/Applications/Hopper"; diff --git a/src/application/SetupClientConfiguration.ts b/src/application/SetupClientConfiguration.ts index bfab80cbf..981b829a4 100644 --- a/src/application/SetupClientConfiguration.ts +++ b/src/application/SetupClientConfiguration.ts @@ -19,6 +19,7 @@ import { } from "./ClientConfigurationDocument.js"; import { dshDocument, DshCordisDocument } from "./DshCordisPatch.js"; import { constants as fsConstants } from "node:fs"; +import { createHash } from "node:crypto"; import { copyFile, mkdir, rm, writeFile } from "node:fs/promises"; import { dirname } from "node:path"; import writeFileAtomic from "write-file-atomic"; @@ -29,6 +30,7 @@ import { npxRegistrationCommand } from "./ClientRegistrationIdentity.js"; import { MCP_STARTUP_POLICY } from "../mcpStartupPolicy.js"; import { resolveClientConfigTransactionPath } from "./ClientConfigPath.js"; import { + readRegularFile, readRegularFileText, RegularFileCleanupFailure, } from "./RegularFileRead.js"; @@ -40,6 +42,26 @@ import type { } from "./SetupTypes.js"; import type { SetupClient } from "./SupportedClients.js"; +export class ClientConfigurationCleanupRollbackFailure extends RegularFileCleanupFailure { + constructor( + failure: RegularFileCleanupFailure, + readonly configurationFailure: Extract< + ClientConfigurationResult, + { readonly status: "failed"; readonly reason: "rollback" } + >, + ) { + super( + failure.path, + failure.owner, + failure.outcome, + new AggregateError( + [failure.cleanupCause, new Error(configurationFailure.failure.detail)], + `Configuration readback cleanup and rollback failed: ${configurationFailure.failure.path}`, + ), + ); + } +} + /** Configure one supported client's native stdio MCP registration shape. */ export const configureClientConfiguration = ( client: SetupClient, @@ -167,11 +189,23 @@ const persistClientConfiguration = async ({ }: ClientConfigurationUpdate): Promise => { const backupPath = original === undefined ? undefined : `${client.configPath}.rea.backup`; - if ( - backupPath !== undefined && - !(await preserveConfigBackup(transactionPath, backupPath)) - ) - return { status: "failed", reason: "backup" }; + let recoveryBackupPath: string | undefined; + if (backupPath !== undefined && original !== undefined) { + const selection = await selectCurrentRecoveryBackup(backupPath, original); + if (!selection.ok) return { status: "failed", reason: "backup" }; + if ( + !(await preserveConfigBackup(transactionPath, selection.value.backupPath)) + ) + return { status: "failed", reason: "backup" }; + const verified = await selectCurrentRecoveryBackup(backupPath, original); + if ( + !verified.ok || + verified.value.backupPath !== selection.value.backupPath || + !verified.value.present + ) + return { status: "failed", reason: "backup" }; + recoveryBackupPath = verified.value.backupPath; + } // A native OpenCode V2 entry replaces REA's legacy V1 entry, which would // otherwise conflict with it. const legacyPath = legacyClientServerPath( @@ -239,19 +273,39 @@ const persistClientConfiguration = async ({ !clientConfigurationValuesEqual(readback.servers[name], entry), ) ) { - await restoreConfig(transactionPath, original); - return { status: "failed", reason: "readback" }; + return rollbackOrReadbackFailure( + transactionPath, + original, + recoveryBackupPath, + ); } } catch (cause: unknown) { - if (cause instanceof RegularFileCleanupFailure) throw cause; + if (cause instanceof RegularFileCleanupFailure) { + const rollback = await restoreConfig(transactionPath, original); + if (!rollback.ok) + throw new ClientConfigurationCleanupRollbackFailure(cause, { + status: "failed", + reason: "rollback", + ...(recoveryBackupPath === undefined + ? {} + : { backupPath: recoveryBackupPath }), + failure: rollback.error, + }); + throw cause; + } // Readback failure restores the transaction before reporting. void cause; - await restoreConfig(transactionPath, original); - return { status: "failed", reason: "readback" }; + return rollbackOrReadbackFailure( + transactionPath, + original, + recoveryBackupPath, + ); } return { status: "configured", - ...(backupPath === undefined ? {} : { backupPath }), + ...(recoveryBackupPath === undefined + ? {} + : { backupPath: recoveryBackupPath }), }; }; @@ -406,9 +460,18 @@ export const inspectClientConfiguration = async ( "The existing configuration is malformed. Repair it before setup applies any other changes.", }; } + const backupSelection = await selectCurrentRecoveryBackup( + `${client.configPath}.rea.backup`, + original, + ); + if (!backupSelection.ok) + return { + status: "invalid", + remediation: backupSelection.error.detail, + }; return { status: "update", - backupPath: `${client.configPath}.rea.backup`, + backupPath: backupSelection.value.backupPath, }; }; @@ -473,19 +536,101 @@ const preserveConfigBackup = async ( } }; +interface RecoveryBackupSelection { + readonly backupPath: string; + readonly present: boolean; +} + +interface RecoveryBackupSelectionFailure { + readonly kind: "unreadable" | "occupied"; + readonly path: string; + readonly detail: string; +} + +const selectCurrentRecoveryBackup = async ( + firstBackupPath: string, + original: string, +): Promise> => { + const expected = Buffer.from(original, "utf8"); + let firstBackup: Buffer | undefined; + try { + firstBackup = await readRegularFile(firstBackupPath, { + symlinks: "reject", + }); + } catch (cause: unknown) { + if (cause instanceof RegularFileCleanupFailure) throw cause; + if (!isMissing(cause)) + return err({ + kind: "unreadable", + path: firstBackupPath, + detail: `The recovery backup could not be checked: ${errorDetail(cause)}`, + }); + } + if (firstBackup === undefined) + return ok({ backupPath: firstBackupPath, present: false }); + if (firstBackup.equals(expected)) + return ok({ backupPath: firstBackupPath, present: true }); + + const digest = createHash("sha256").update(original, "utf8").digest("hex"); + const recoveryPath = `${firstBackupPath}.${digest}.recovery`; + let recovery: Buffer | undefined; + try { + recovery = await readRegularFile(recoveryPath, { symlinks: "reject" }); + } catch (cause: unknown) { + if (cause instanceof RegularFileCleanupFailure) throw cause; + if (!isMissing(cause)) + return err({ + kind: "unreadable", + path: recoveryPath, + detail: `The recovery backup could not be checked: ${errorDetail(cause)}`, + }); + } + if (recovery !== undefined && !recovery.equals(expected)) + return err({ + kind: "occupied", + path: recoveryPath, + detail: `The recovery path already contains different data: ${recoveryPath}`, + }); + return ok({ backupPath: recoveryPath, present: recovery !== undefined }); +}; + const restoreConfig = async ( path: string, original: string | undefined, -): Promise => { +): Promise< + Result< + void, + { readonly kind: "restore"; readonly path: string; readonly detail: string } + > +> => { try { if (original === undefined) await rm(path, { force: true }); else await writeFile(path, original, { encoding: "utf8", mode: 0o600 }); + return ok(undefined); } catch (cause: unknown) { - // The backup remains available for the remediation reported by setup. - void cause; + return err({ kind: "restore", path, detail: errorDetail(cause) }); } }; +const rollbackOrReadbackFailure = async ( + targetPath: string, + original: string | undefined, + backupPath: string | undefined, +): Promise => { + const rollback = await restoreConfig(targetPath, original); + return rollback.ok + ? { status: "failed", reason: "readback" } + : { + status: "failed", + reason: "rollback", + ...(backupPath === undefined ? {} : { backupPath }), + failure: rollback.error, + }; +}; + +const errorDetail = (cause: unknown): string => + cause instanceof Error ? cause.message : String(cause); + /** Whether REA's entry matches, no legacy entry remains, and no client disable list suppresses it. */ const registrationCurrent = ( parsed: ClientConfigurationDocument, diff --git a/src/application/SetupClients.ts b/src/application/SetupClients.ts index bbeeedf3f..d86d04b9e 100644 --- a/src/application/SetupClients.ts +++ b/src/application/SetupClients.ts @@ -6,15 +6,19 @@ import type { } from "./SetupTypes.js"; import type { SetupClient } from "./SupportedClients.js"; -const failedConfigurationMessage = ( - reason: "path" | "backup" | "write" | "readback", +export const failedConfigurationMessage = ( + result: Extract, ): string => { - if (reason === "path") + if (result.reason === "path") return "Agent configuration path could not be safely verified. Check its permissions and, if it is a symbolic link, verify that the link resolves to a regular file owned by the current user, then rerun setup."; - if (reason === "backup") + if (result.reason === "backup") return "Agent configuration could not be backed up, so no change was made. Check file permissions, then rerun setup."; - if (reason === "write") + if (result.reason === "write") return "Agent configuration could not be updated. Check file permissions, then rerun setup."; + if (result.reason === "rollback") + return result.backupPath === undefined + ? `Agent configuration rollback failed at ${result.failure.path}: ${result.failure.detail}; the failed first-run file may remain. Remove or repair it before rerunning setup.` + : `Agent configuration rollback failed at ${result.failure.path}: ${result.failure.detail}. Inspect the preserved backup at ${result.backupPath} and restore the intended configuration, then rerun setup.`; return "Agent configuration could not be verified after writing. Repair the configuration file or restore its `.rea.backup`, then rerun setup."; }; @@ -40,12 +44,12 @@ export const configureDetectedClients = async (options: { ); options.clients[client.name] = result; if (result.status === "failed") { - failure ??= failedConfigurationMessage(result.reason); + failure ??= failedConfigurationMessage(result); options.onProgress?.({ actionId, label, state: "failed", - detail: failedConfigurationMessage(result.reason), + detail: failedConfigurationMessage(result), }); } else if (result.status === "configured") { options.appliedActions.push(`configured_${client.name}`); diff --git a/src/application/SetupHost.ts b/src/application/SetupHost.ts index 9e2ae95f1..7ac6828fd 100644 --- a/src/application/SetupHost.ts +++ b/src/application/SetupHost.ts @@ -33,9 +33,11 @@ import { } from "./SetupSkill.js"; import { clientConfigurationAligned, + ClientConfigurationCleanupRollbackFailure, configureClientConfiguration, inspectClientConfiguration, } from "./SetupClientConfiguration.js"; +import { failedConfigurationMessage } from "./SetupClients.js"; import { setupInstallFailure } from "./SetupInstallFailure.js"; import { RegularFileCleanupFailure, @@ -275,8 +277,12 @@ const runClientConfigurationOperation = async ( cleanup.outcome.kind === "failed" ? cleanup.outcome.cause : cause.cleanupCause; + const rollbackMessage = + cause instanceof ClientConfigurationCleanupRollbackFailure + ? ` ${failedConfigurationMessage(cause.configurationFailure)}` + : ""; throw configurationCleanupError( - `Could not safely access client configuration at ${cause.path}: ${errorMessage(primary)}`, + `Could not safely access client configuration at ${cause.path}: ${errorMessage(primary)}.${rollbackMessage}`, cause, cleanup.cleanup, ); diff --git a/src/application/SetupInstallFailure.test.ts b/src/application/SetupInstallFailure.test.ts new file mode 100644 index 000000000..980e22151 --- /dev/null +++ b/src/application/SetupInstallFailure.test.ts @@ -0,0 +1,23 @@ +import { describe, expect, it } from "vitest"; + +import { HOPPER_PUBLISHER } from "./MacHopper.js"; +import { setupInstallFailure } from "./SetupInstallFailure.js"; + +describe("macOS Hopper setup failure projection", () => { + it("projects a publisher mismatch onto its own code and recovery", () => { + expect(setupInstallFailure("publisher_verification_failed")).toEqual({ + status: "failed", + code: "publisher_verification_failed", + remediation: `The mounted Hopper application is not signed by the publisher REA pins (${HOPPER_PUBLISHER.bundleIdentifier}, Apple team ${HOPPER_PUBLISHER.teamIdentifier}); nothing was copied or launched. Install Hopper yourself and set HOPPER_LAUNCHER_PATH to its launcher, then rerun rea setup.`, + }); + }); + + it("keeps a broken bundle distinct from a publisher mismatch", () => { + expect(setupInstallFailure("bundle")).toEqual({ + status: "failed", + code: "bundle_invalid", + remediation: + "The mounted package did not contain the expected Hopper application bundle. Retry setup; if it repeats, update REA before installing.", + }); + }); +}); diff --git a/src/application/SetupInstallFailure.ts b/src/application/SetupInstallFailure.ts index fe2a2739e..0b332fa25 100644 --- a/src/application/SetupInstallFailure.ts +++ b/src/application/SetupInstallFailure.ts @@ -1,3 +1,5 @@ +import { HOPPER_PUBLISHER } from "./MacHopper.js"; + export type SetupFailureCode = | "unsupported_host" | "download_failed" @@ -10,6 +12,7 @@ export type SetupFailureCode = | "destination_exists" | "mount_failed" | "bundle_invalid" + | "publisher_verification_failed" | "copy_failed" | "skill_install_failed" | "cleanup_failed"; @@ -39,6 +42,7 @@ export type HopperInstallFailureReason = | "destination_exists" | "mount" | "bundle" + | "publisher_verification_failed" | "copy"; /** Map provider installer failures to stable setup codes and safe recovery. */ @@ -115,6 +119,12 @@ export const setupInstallFailure = ( remediation: "The mounted package did not contain the expected Hopper application bundle. Retry setup; if it repeats, update REA before installing.", }; + case "publisher_verification_failed": + return { + status: "failed", + code: "publisher_verification_failed", + remediation: `The mounted Hopper application is not signed by the publisher REA pins (${HOPPER_PUBLISHER.bundleIdentifier}, Apple team ${HOPPER_PUBLISHER.teamIdentifier}); nothing was copied or launched. Install Hopper yourself and set HOPPER_LAUNCHER_PATH to its launcher, then rerun rea setup.`, + }; case "copy": return { status: "failed", diff --git a/src/application/SetupPlan.ts b/src/application/SetupPlan.ts index 83402d1a4..a78697c80 100644 --- a/src/application/SetupPlan.ts +++ b/src/application/SetupPlan.ts @@ -320,5 +320,5 @@ const clientConfigurationDetail = ( ? " If this plan installs Hopper, REA will add HOPPER_LAUNCHER_PATH using the launcher path reported by that installation." : " The HOPPER_LAUNCHER_PATH value shown above is the current fallback; if this plan installs Hopper, REA will replace it with the launcher path reported by that installation." : ""; - return `Add the REA MCP registration for ${client}; preserve unrelated configuration. Command: ${command.map((part) => JSON.stringify(part)).join(" ")}.${environmentDetail}${hopperDetail}`; + return `Add the REA MCP registration for ${client}; preserve unrelated configuration. The backup path reflects the configuration inspected for this plan; if that file changes before execution, setup will select a recovery path for its current contents. Command: ${command.map((part) => JSON.stringify(part)).join(" ")}.${environmentDetail}${hopperDetail}`; }; diff --git a/src/application/SetupTypes.ts b/src/application/SetupTypes.ts index da58abbff..241e0f6dd 100644 --- a/src/application/SetupTypes.ts +++ b/src/application/SetupTypes.ts @@ -26,6 +26,16 @@ export type ClientConfigurationResult = | { readonly status: "failed"; readonly reason: "path" | "backup" | "write" | "readback"; + } + | { + readonly status: "failed"; + readonly reason: "rollback"; + readonly backupPath?: string; + readonly failure: { + readonly kind: "restore"; + readonly path: string; + readonly detail: string; + }; }; /** Read-only state used to block unsafe setup plans before any mutation. */ diff --git a/src/application/SupportedClients.ts b/src/application/SupportedClients.ts index 488dda49d..0a7352627 100644 --- a/src/application/SupportedClients.ts +++ b/src/application/SupportedClients.ts @@ -135,6 +135,9 @@ const codexDirectory = ({ home, env }: ClientPathContext): string => const qoderDirectory = ({ home, env }: ClientPathContext): string => env.QODER_CONFIG_DIR || join(home, ".qoder"); +const factoryDirectory = ({ home }: ClientPathContext): string => + join(home, ".factory"); + /** Match Qwen Code's home override, including tilde and cwd-relative paths. */ const qwenCodeDirectory = ({ home, env }: ClientPathContext): string => { const override = env.QWEN_HOME; @@ -157,8 +160,27 @@ const qwenCodeDirectory = ({ home, env }: ClientPathContext): string => { const grokDirectory = ({ home, env }: ClientPathContext): string => env.GROK_HOME || join(home, ".grok"); +/** Match Harness home-paths for both independent roots; trim only to detect blanks. */ +const dshDirectory = ( + home: string, + override: string | undefined, + fallback: string, +): string => { + const selected = + override !== undefined && override.trim().length > 0 + ? override + : join(home, fallback); + const expanded = + selected === "~" + ? home + : selected.startsWith("~/") || selected.startsWith("~\\") + ? join(home, selected.slice(2)) + : selected; + return resolve(expanded); +}; + const dshHomeDirectory = ({ home, env }: ClientPathContext): string => - env.DSH_HOME ?? join(home, ".dsh"); + dshDirectory(home, env.DSH_HOME, ".dsh"); const dshPatchDirectory = (context: ClientPathContext): string => join(dshHomeDirectory(context), "cordis.patch.yml"); @@ -539,6 +561,16 @@ export const SUPPORTED_CLIENT_DEFINITIONS = [ markerPath: qoderDirectory, format: "json", }, + { + name: "droid", + displayName: "Factory Droid", + skillPath: (context: ClientPathContext) => + join(factoryDirectory(context), "skills"), + configPath: (context: ClientPathContext) => + join(factoryDirectory(context), "mcp.json"), + markerPath: factoryDirectory, + format: "json", + }, { name: "zcode", zcodeFallbackPath: ({ home }: ClientPathContext) => @@ -555,7 +587,7 @@ export const SUPPORTED_CLIENT_DEFINITIONS = [ // The home Cordis patch applies to every profile; the skill directory is // shared agent storage Harness discovers. skillPath: ({ home, env }: ClientPathContext) => - join(env.DSH_AGENTS_HOME ?? join(home, ".agents"), "skills"), + join(dshDirectory(home, env.DSH_AGENTS_HOME, ".agents"), "skills"), configPath: dshPatchDirectory, markerPath: dshHomeDirectory, format: "dsh", diff --git a/src/application/TypeScriptEmittedSourceCandidates.ts b/src/application/TypeScriptEmittedSourceCandidates.ts index 663c7dada..0e7adf41c 100644 --- a/src/application/TypeScriptEmittedSourceCandidates.ts +++ b/src/application/TypeScriptEmittedSourceCandidates.ts @@ -8,6 +8,11 @@ const EMITTED_SUFFIXES: ReadonlyArray< [".jsx", [".tsx", ".d.ts"]], ]; +export interface TypeScriptEmittedSourceCandidate { + readonly path: string; + readonly kind: "implementation" | "declaration"; +} + /** * Return TypeScript source counterparts for an explicit emitted filename. * Callers retain exact runtime-file precedence and must not use these for @@ -16,7 +21,7 @@ const EMITTED_SUFFIXES: ReadonlyArray< export const typescriptEmittedSourceCandidates = ( sourcePath: string, candidate: string, -): readonly string[] => { +): readonly TypeScriptEmittedSourceCandidate[] => { if (!/\.(?:ts|tsx|mts|cts)$/u.test(sourcePath)) return []; const emitted = EMITTED_SUFFIXES.find(([suffix]) => candidate.endsWith(suffix), @@ -25,5 +30,8 @@ export const typescriptEmittedSourceCandidates = ( // Strip the matched suffix by length. posix.extname would miss dotfile // names such as ".js", whose whole basename is the emitted suffix itself. const stem = candidate.slice(0, candidate.length - emitted[0].length); - return emitted[1].map((suffix) => `${stem}${suffix}`); + return emitted[1].map((suffix) => ({ + path: `${stem}${suffix}`, + kind: suffix.startsWith(".d.") ? "declaration" : "implementation", + })); }; diff --git a/src/application/Uninstall.ts b/src/application/Uninstall.ts index c410280bc..a7f051a49 100644 --- a/src/application/Uninstall.ts +++ b/src/application/Uninstall.ts @@ -25,7 +25,11 @@ import { import { isOwnedClientRegistrationCommand } from "./ClientRegistrationIdentity.js"; import { dshDocument } from "./DshCordisPatch.js"; import { resolveClientConfigTransactionPath } from "./ClientConfigPath.js"; -import { readRegularFileText } from "./RegularFileRead.js"; +import { + InvalidUtf8FileError, + RegularFileCleanupFailure, + readRegularFileText, +} from "./RegularFileRead.js"; import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js"; import { manualRegistrationRemediation, @@ -269,13 +273,17 @@ const readClientConfiguration = async ( try { original = await fileSystem.readText(transactionPath); } catch (cause: unknown) { + const readFailure = + cause instanceof RegularFileCleanupFailure ? cause.cause : cause; return itemRead( isMissing(cause) ? item(client.name, "skipped", "Configuration does not exist.") : item( client.name, "failed", - "Configuration could not be read. Check file permissions, then rerun uninstall.", + readFailure instanceof InvalidUtf8FileError + ? "Configuration is not valid UTF-8 and was not changed. Repair it, then rerun uninstall." + : "Configuration could not be read. Check file permissions, then rerun uninstall.", ), ); } diff --git a/src/application/UpdateMaintenance.ts b/src/application/UpdateMaintenance.ts index 06ac1b512..fd55c8f23 100644 --- a/src/application/UpdateMaintenance.ts +++ b/src/application/UpdateMaintenance.ts @@ -1,19 +1,10 @@ import { z } from "zod"; -import { PRODUCT_IDENTITY } from "../identity.js"; import { isOwnedClientRegistrationCommand } from "./ClientRegistrationIdentity.js"; -import { readClientRegistrationStatuses } from "./ClientRegistrationStatus.js"; -import { - clientServerForcedEnabled, - clientServerListedDisabled, - effectiveClientServer, - parseClientConfiguration, -} from "./ClientConfigurationDocument.js"; -import { supportedClients } from "./SupportedClients.js"; +import { readClientRegistrationInspections } from "./ClientRegistrationStatus.js"; import { existingSkillDestinations } from "./SetupSkill.js"; import type { SetupAction } from "./SetupTypes.js"; import type { Result } from "../domain/result.js"; -import { readRegularFileText } from "./RegularFileRead.js"; import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js"; /** Existing REA integrations selected for maintenance, without discovering new targets. */ @@ -61,41 +52,23 @@ export const existingMaintenanceScope = async ( environment: Readonly, resources?: ArtifactResourceScope, ): Promise => { - const registrations = await readClientRegistrationStatuses(home, entryPoint, { - environment, - ...(resources === undefined ? {} : { resources }), - }); - const supported = supportedClients(home, process.platform, environment); + const registrations = await readClientRegistrationInspections( + home, + entryPoint, + { + environment, + ...(resources === undefined ? {} : { resources }), + }, + ); const clients: string[] = []; - for (const registration of registrations) { + for (const { status: registration, activation } of registrations) { if ( + activation !== "enabled" || (registration.state !== "aligned" && registration.state !== "stale") || !isOwnedClientRegistrationCommand(registration.command, entryPoint) ) continue; - const client = supported.find(({ name }) => name === registration.client); - if (client === undefined) continue; - const parsed = parseClientConfiguration( - await readRegularFileText( - client.configPath, - resources === undefined ? {} : { resources }, - ), - client.format, - ); - const enabled = z - .object({ - enabled: z.boolean().optional(), - disabled: z.boolean().optional(), - }) - .parse(effectiveClientServer(parsed, PRODUCT_IDENTITY.mcpServerKey)); - // Root disable lists override enabled entries and OMP's force-on allowlist. - if ( - (enabled.enabled !== false || - clientServerForcedEnabled(parsed, PRODUCT_IDENTITY.mcpServerKey)) && - enabled.disabled !== true && - !clientServerListedDisabled(parsed, PRODUCT_IDENTITY.mcpServerKey) - ) - clients.push(client.name); + clients.push(registration.client); } const skillDestinations = await existingSkillDestinations( home, diff --git a/src/application/WebModulePorts.ts b/src/application/WebModulePorts.ts index 81c94a161..2f03efce0 100644 --- a/src/application/WebModulePorts.ts +++ b/src/application/WebModulePorts.ts @@ -22,6 +22,7 @@ export interface WebModuleArtifacts extends SelectedWebScriptArtifacts { /** Read selected local artifacts without refetching website assets. */ export interface WebModuleArtifactPort { + close?(): Promise; load( input: WebModuleTraceInput, options?: ExecutionOptions, diff --git a/src/application/WebModuleTraceService.ts b/src/application/WebModuleTraceService.ts index f8aa0f8b9..be0936e1c 100644 --- a/src/application/WebModuleTraceService.ts +++ b/src/application/WebModuleTraceService.ts @@ -1,3 +1,4 @@ +import { closeOwnedWebResources } from "./closeOwnedWebResources.js"; import { isAbsolute } from "node:path"; import { z } from "zod"; import type { AnalysisError } from "../domain/analysisErrorBase.js"; @@ -70,7 +71,13 @@ export class WebModuleTraceService { ) {} close(): Promise { - return this.resolver.close?.() ?? Promise.resolve(); + return closeOwnedWebResources( + [ + this.artifacts.close?.bind(this.artifacts), + this.resolver.close?.bind(this.resolver), + ], + "Web module trace cleanup failed", + ); } async trace( diff --git a/src/application/WebNetworkCapturePort.ts b/src/application/WebNetworkCapturePort.ts index 5f9cd5545..2bd0f1968 100644 --- a/src/application/WebNetworkCapturePort.ts +++ b/src/application/WebNetworkCapturePort.ts @@ -12,4 +12,6 @@ export interface WebNetworkCapturePort { input: InspectWebNetworkCaptureInput, options?: ExecutionOptions, ): Promise>; + /** Retry retained reader/runtime cleanup when the owning service shuts down. */ + close?(): Promise; } diff --git a/src/application/WebNetworkCaptureService.ts b/src/application/WebNetworkCaptureService.ts index 326016a26..5a3d5cfea 100644 --- a/src/application/WebNetworkCaptureService.ts +++ b/src/application/WebNetworkCaptureService.ts @@ -10,12 +10,14 @@ import { AnalysisInputError, AnalysisOutputError, } from "../domain/analysisErrorCore.js"; +import { createAnalysisProfile } from "../domain/analysisProfile.js"; import { createEvidence, type Evidence } from "../domain/evidence.js"; import { analysisInputErrorFromIssues } from "../domain/inputIssueProjection.js"; import { jsonObjectSchema } from "../domain/jsonValue.js"; import { err, ok, type Result } from "../domain/result.js"; import { inspectWebNetworkCaptureInputSchema, + captureSensitiveValues, webNetworkCaptureSchema, } from "../domain/webNetworkCapture.js"; @@ -25,6 +27,11 @@ const OPERATION = "inspect_web_network_capture"; export class WebNetworkCaptureService { constructor(readonly port: WebNetworkCapturePort) {} + /** Release resources retained by the selected capture adapter. */ + close(): Promise { + return this.port.close?.() ?? Promise.resolve(); + } + /** Return selected complete records inline, without inventing live browser attribution. */ async inspect( rawInput: unknown, @@ -32,21 +39,8 @@ export class WebNetworkCaptureService { ): Promise> { if (options?.signal?.aborted) return err(new AnalysisCancelledError(OPERATION)); - const declarations = - inspectWebNetworkCaptureInputSchema.shape.sensitive_values.safeParse( - typeof rawInput === "object" && - rawInput !== null && - "sensitive_values" in rawInput - ? rawInput.sensitive_values - : undefined, - ); const failure = (error: AnalysisError): Result => - err( - redactExplicitFailure( - error, - declarations.success ? declarations.data : [], - ), - ); + err(redactExplicitFailure(error, captureSensitiveValues(rawInput))); const parsed = inspectWebNetworkCaptureInputSchema.safeParse(rawInput); if (!parsed.success) return failure( @@ -187,6 +181,15 @@ export class WebNetworkCaptureService { ? [...limitations, subjectUnavailableReason] : limitations, }, + ...(safeValue.format !== "qlog" + ? {} + : { + analysisProfile: createAnalysisProfile(value.decoder, { + qlog_profile: safeValue.qlog.profile, + clock: safeValue.qlog.clock, + scope: safeValue.qlog.scope, + }), + }), confidence: "observed", authority: "historical-reference", limitations, diff --git a/src/application/WebScriptArtifactPort.ts b/src/application/WebScriptArtifactPort.ts index 02214d006..77dcd9cfc 100644 --- a/src/application/WebScriptArtifactPort.ts +++ b/src/application/WebScriptArtifactPort.ts @@ -8,6 +8,7 @@ import type { ExecutionOptions } from "./AnalysisProvider.js"; /** Read the caller-selected capture export and verify the selected source identity. */ export interface WebScriptArtifactPort { + close?(): Promise; load( input: SelectedWebScriptInput, options?: ExecutionOptions, diff --git a/src/application/WebSourceLocationPorts.ts b/src/application/WebSourceLocationPorts.ts index 04fe1742c..1d21783cc 100644 --- a/src/application/WebSourceLocationPorts.ts +++ b/src/application/WebSourceLocationPorts.ts @@ -20,6 +20,7 @@ export interface WebSourceLocationArtifacts extends SelectedWebScriptArtifacts { /** Replaceable artifact acquisition boundary for a caller-selected pairing. */ export interface WebSourceLocationArtifactPort { + close?(): Promise; load( input: WebSourceLocationInput, options?: ExecutionOptions, diff --git a/src/application/WebSourceLocationService.ts b/src/application/WebSourceLocationService.ts index 7be901311..d084bff5a 100644 --- a/src/application/WebSourceLocationService.ts +++ b/src/application/WebSourceLocationService.ts @@ -1,3 +1,4 @@ +import { closeOwnedWebResources } from "./closeOwnedWebResources.js"; import { createHash } from "node:crypto"; import { isAbsolute } from "node:path"; import { z } from "zod"; @@ -71,7 +72,13 @@ export class WebSourceLocationService { ) {} close(): Promise { - return this.decoder.close?.() ?? Promise.resolve(); + return closeOwnedWebResources( + [ + this.artifacts.close?.bind(this.artifacts), + this.decoder.close?.bind(this.decoder), + ], + "Web source-location cleanup failed", + ); } /** Shared CLI/MCP workflow preserves raw declarations and every equal-position match. */ @@ -174,6 +181,7 @@ export class WebSourceLocationService { ); } } + const invalidOutput = (message: string): Result => err(new AnalysisOutputError(OPERATION, message)); const invalid = ( diff --git a/src/application/analysisView/AnalysisViewService.ts b/src/application/analysisView/AnalysisViewService.ts index a5b603a86..baaa6b7ca 100644 --- a/src/application/analysisView/AnalysisViewService.ts +++ b/src/application/analysisView/AnalysisViewService.ts @@ -47,14 +47,8 @@ export const inspectAnalysisViewValidated = ( input: InspectAnalysisViewInput, lookup?: EvidenceLookup, ): Result => { - const inline = input.source.kind === "inline" ? input.source.evidence : null; - const inlineCapture = - typeof inline === "object" && - inline !== null && - Object.getOwnPropertyDescriptor(inline, "operation")?.value === - "inspect_web_network_capture"; const accumulator = - input.view.kind === "api-surface" || inlineCapture + input.view.kind === "api-surface" ? new WebNetworkApiSurfaceAccumulator() : undefined; if (input.source.kind === "inline" && accumulator !== undefined) { diff --git a/src/application/android/AndroidPackageInspection.ts b/src/application/android/AndroidPackageInspection.ts new file mode 100644 index 000000000..3fbaa2f1b --- /dev/null +++ b/src/application/android/AndroidPackageInspection.ts @@ -0,0 +1,43 @@ +import type { ExecutionOptions } from "../AnalysisProvider.js"; +import type { AndroidAnalysisService } from "./AndroidAnalysisService.js"; +import { + requireAnalysisViewRetention, + summarizeRetainedAnalysis, + type AnalysisViewRetention, +} from "../analysisView/AnalysisViewService.js"; +import type { AnalysisError } from "../../domain/analysisErrorBase.js"; +import { AnalysisCancelledError } from "../../domain/analysisErrorCore.js"; +import { androidPackageInspectionInputSchema } from "../../domain/android/androidAnalysis.js"; +import type { Evidence } from "../../domain/evidence.js"; +import { analysisInputErrorFromIssues } from "../../domain/inputIssueProjection.js"; +import { err, type Result } from "../../domain/result.js"; + +/** Preserve the complete provider request; select retained delivery explicitly. */ +export const inspectAndroidPackage = async ( + service: AndroidAnalysisService, + rawInput: unknown, + retention: AnalysisViewRetention, + options?: ExecutionOptions, +): Promise> => { + const operation = "inspect_android_package"; + if (options?.signal?.aborted === true) + return err(new AnalysisCancelledError(operation)); + const parsed = androidPackageInspectionInputSchema.safeParse(rawInput); + if (!parsed.success) + return err( + analysisInputErrorFromIssues(operation, parsed.error.issues, rawInput, { + cause: parsed.error, + }), + ); + if (parsed.data.detail === "summary") { + const ready = requireAnalysisViewRetention(operation, retention); + if (!ready.ok) return ready; + } + const executed = await service.execute( + operation, + { path: parsed.data.path }, + options, + ); + if (!executed.ok || parsed.data.detail !== "summary") return executed; + return summarizeRetainedAnalysis(executed.value, retention); +}; diff --git a/src/application/binary/AnalysisSnapshotCache.test.ts b/src/application/binary/AnalysisSnapshotCache.test.ts index 48c2a54bd..91b9f8807 100644 --- a/src/application/binary/AnalysisSnapshotCache.test.ts +++ b/src/application/binary/AnalysisSnapshotCache.test.ts @@ -661,3 +661,52 @@ it("does not replay pre-envelope call graphs under the revised workflow profile" }).success, ).toBe(false); }); + +describe("executions that report truncated references", () => { + const record = (referencesTruncated: boolean | undefined) => { + const cache = new AnalysisSnapshotCache(); + const parameters = { address: "0x1000", document: "fixture" }; + cache.record({ + target: ANALYSIS_SNAPSHOT_TARGET, + profile: ANALYSIS_SNAPSHOT_PROFILE, + operation: "xrefs", + parameters, + execution: createAnalysisExecution( + ["0x2000"], + ANALYSIS_SNAPSHOT_PROVIDER, + { + subject: ANALYSIS_SNAPSHOT_TARGET, + analysisProfile: ANALYSIS_SNAPSHOT_PROFILE, + ...(referencesTruncated === undefined ? {} : { referencesTruncated }), + }, + ), + }); + return cache.lookup( + ANALYSIS_SNAPSHOT_TARGET, + ANALYSIS_SNAPSHOT_PROFILE, + "xrefs", + parameters, + ); + }; + + it("are not replayed, because a replay would lose the truncation", () => { + expect(record(true)).toBeUndefined(); + }); + + it("do not stop an execution that is complete or silent from being replayed", () => { + expect(record(false)?.result).toEqual(["0x2000"]); + expect(record(undefined)?.result).toEqual(["0x2000"]); + }); +}); + +it("revises the trace_feature workflow profile so earlier results are not replayed", () => { + const profile = ANALYSIS_SNAPSHOT_PROFILE; + + expect(workflowAnalysisProfile(profile, "trace_feature")).not.toEqual( + workflowAnalysisProfile(profile), + ); + expect( + workflowAnalysisProfile(profile, "trace_call_path"), + "other workflows keep their profile", + ).toEqual(workflowAnalysisProfile(profile)); +}); diff --git a/src/application/binary/AnalysisSnapshotCache.ts b/src/application/binary/AnalysisSnapshotCache.ts index bdaa9f68d..79e212cac 100644 --- a/src/application/binary/AnalysisSnapshotCache.ts +++ b/src/application/binary/AnalysisSnapshotCache.ts @@ -84,6 +84,11 @@ export const isSnapshotCacheable = ( descriptor.effects.mayWriteFilesystem === false) && descriptor.effects.changesPermissions === false; +/** Windows Ghidra must admit the selected source through its native snapshot owner. */ +export const requiresNativeSourceAdmissionBeforeReplay = ( + providerId: string | undefined, +): boolean => process.platform === "win32" && providerId === "ghidra"; + /** Bounded in-memory cache for one immutable binary identity. */ export class AnalysisSnapshotCache { readonly #entries = new Map(); @@ -337,6 +342,9 @@ export class AnalysisSnapshotCache { }): void { const { target, profile, operation, parameters, execution } = input; this.select(target, profile); + // The snapshot format does not carry reference truncation; a replay would + // present a capped read as complete, so it is not recorded. + if (execution.referencesTruncated === true) return; const queryId = analysisQueryId( snapshotTarget(target), snapshotBinding(profile), diff --git a/src/application/binary/BinarySession.ts b/src/application/binary/BinarySession.ts index f7b2d3529..3da571832 100644 --- a/src/application/binary/BinarySession.ts +++ b/src/application/binary/BinarySession.ts @@ -58,6 +58,7 @@ interface SessionBinding { readonly profile: AnalysisProfileCommitment | null; readonly route: SessionProviderRoute; readonly runId: string; + nativeSourceAdmitted: boolean; } /** @@ -323,6 +324,7 @@ export class BinarySession profile, route, runId, + nativeSourceAdmitted: false, }; this.#clearRuntimeAvailability(); this.resetSnapshotInvalidation(); @@ -609,6 +611,7 @@ export class BinarySession active: this.#active, operation: name, parameters: arguments_, + nativeSourceAdmitted: this.#active?.nativeSourceAdmitted === true, unboundOperationError: (operation, route) => this.#providerRouter.unboundOperationError(operation, route), lookupSnapshot: (target, profile, operation, parameters) => @@ -645,6 +648,14 @@ export class BinarySession }, }) : profiled; + // A successful Ghidra operation proves this client's native source snapshot + // was admitted; a composite health response and parsed path do not. + if ( + execution.ok && + capability.provider.id === "ghidra" && + this.#active === active + ) + this.#active.nativeSourceAdmitted = true; this.#observeRuntimeAvailability(name, execution); if ( execution.ok && @@ -838,6 +849,7 @@ export class BinarySession profile: previous.profile, route: previous.route, runId, + nativeSourceAdmitted: false, }; return ok(null); } diff --git a/src/application/binary/BinarySessionExecution.ts b/src/application/binary/BinarySessionExecution.ts index 996749aae..73dfe4eea 100644 --- a/src/application/binary/BinarySessionExecution.ts +++ b/src/application/binary/BinarySessionExecution.ts @@ -13,7 +13,10 @@ import type { AnalysisOperation, CapabilityDescriptor, } from "../AnalysisProvider.js"; -import { isSnapshotCacheable } from "./AnalysisSnapshotCache.js"; +import { + isSnapshotCacheable, + requiresNativeSourceAdmissionBeforeReplay, +} from "./AnalysisSnapshotCache.js"; import type { SessionProviderRoute } from "./SessionProviderRouter.js"; interface ActiveExecutionBinding { @@ -35,6 +38,7 @@ interface PrepareSessionExecutionInput { readonly active: ActiveExecutionBinding | undefined; readonly operation: AnalysisOperation; readonly parameters: Readonly>; + readonly nativeSourceAdmitted: boolean; readonly unboundOperationError: ( operation: AnalysisOperation, route: SessionProviderRoute, @@ -55,6 +59,7 @@ export const prepareSessionExecution = ( active, operation, parameters, + nativeSourceAdmitted, unboundOperationError, lookupSnapshot, } = input; @@ -79,9 +84,12 @@ export const prepareSessionExecution = ( const cacheable = profile !== undefined && isSnapshotCacheable(operation, capability, parameters); - const cached = cacheable - ? lookupSnapshot(active.target, profile, operation, parameters) - : undefined; + const cached = + cacheable && + (!requiresNativeSourceAdmissionBeforeReplay(capability.provider.id) || + nativeSourceAdmitted) + ? lookupSnapshot(active.target, profile, operation, parameters) + : undefined; return ok({ active, capability, profile, cacheable, cached }); }; diff --git a/src/application/binary/BinarySessionOpen.ts b/src/application/binary/BinarySessionOpen.ts index 8d810e671..cd5f6be4a 100644 --- a/src/application/binary/BinarySessionOpen.ts +++ b/src/application/binary/BinarySessionOpen.ts @@ -92,9 +92,7 @@ export const resolveSessionTarget = async ( input: ResolveSessionTargetInput, ): Promise> => { const { router, current, target, options } = input; - const sameTarget = - current?.target.path === target.path && - snapshotMatchesTarget(snapshotTarget(current.target), target); + const sameTarget = sameSessionTarget(current?.target, target); const resolvedRoute = sameTarget && options.providerId === undefined && current !== undefined ? ({ ok: true, value: current.route } as const) @@ -113,9 +111,7 @@ export const validateResolvedSessionTarget = ( }, ): Result => { const { current, target, route, options, stagedSnapshotMatches } = input; - const sameTarget = - current?.target.path === target.path && - snapshotMatchesTarget(snapshotTarget(current.target), target); + const sameTarget = sameSessionTarget(current?.target, target); const snapshotError = validateSnapshot({ snapshot: options.snapshot, target, @@ -127,6 +123,16 @@ export const validateResolvedSessionTarget = ( return { ok: true, value: { target, route, sameTarget } }; }; +const sameSessionTarget = ( + current: BinaryTarget | undefined, + target: BinaryTarget, +): boolean => + current !== undefined && + current.path === target.path && + // Native admission is bound to the requested path, not just its realpath. + (current.sourcePath ?? current.path) === (target.sourcePath ?? target.path) && + snapshotMatchesTarget(snapshotTarget(current), target); + interface ValidateSnapshotInput { readonly snapshot: AnalysisSnapshot | undefined; readonly target: BinaryTarget; diff --git a/src/application/binaryDiagnostics/PeResourcesPort.ts b/src/application/binaryDiagnostics/PeResourcesPort.ts index b1a7b53de..ee36d34f3 100644 --- a/src/application/binaryDiagnostics/PeResourcesPort.ts +++ b/src/application/binaryDiagnostics/PeResourcesPort.ts @@ -12,6 +12,7 @@ import type { Result } from "../../domain/result.js"; /** Stable read and PE decoding boundary, independent of an active disassembler session. */ export interface PeResourcesPort { readonly identity: ProviderIdentity; + close?(): Promise; inspect( input: InspectPeResourcesInput, options?: ExecutionOptions, diff --git a/src/application/binaryDiagnostics/PeResourcesService.ts b/src/application/binaryDiagnostics/PeResourcesService.ts index 3c8a53704..87de771a4 100644 --- a/src/application/binaryDiagnostics/PeResourcesService.ts +++ b/src/application/binaryDiagnostics/PeResourcesService.ts @@ -21,6 +21,10 @@ import { err, ok, type Result } from "../../domain/result.js"; export class PeResourcesService { constructor(readonly provider: PeResourcesPort) {} + close(): Promise { + return this.provider.close?.() ?? Promise.resolve(); + } + async inspect( rawInput: unknown, options?: ExecutionOptions, diff --git a/src/application/captures/PacketCaptureService.ts b/src/application/captures/PacketCaptureService.ts index f2098b49c..e2877fa3f 100644 --- a/src/application/captures/PacketCaptureService.ts +++ b/src/application/captures/PacketCaptureService.ts @@ -1,3 +1,9 @@ +import { + inspectPacketFieldsInputSchema, + packetFieldsCatalogSchema, + type InspectPacketFieldsInput, +} from "../../domain/captures/packetFields.js"; +import type { JsonValue } from "../../domain/jsonValue.js"; import { isAbsolute } from "node:path"; import type { AnalysisExecution, @@ -24,6 +30,10 @@ export interface PacketCapturePort { input: InspectPacketCaptureInput, options?: ExecutionOptions, ): Promise>; + inspectFields( + input: InspectPacketFieldsInput, + options?: ExecutionOptions, + ): Promise>; close(): Promise; } @@ -74,14 +84,58 @@ export class PacketCaptureService { "Capture analyzer returned malformed data or changed the selected artifact identity/options.", ), ); - const evidence = createEvidence(execution.subject, execution.provider, { - operation, - parameters: input.data, - result: report.data, - confidence: "observed", - locations: execution.locations, - limitations: execution.limitations, - }); + return this.#evidence(operation, input.data, execution, options); + } + async inspectFields( + raw: unknown, + options?: ExecutionOptions, + ): Promise> { + const operation = "inspect_packet_fields"; + if (options?.signal?.aborted) + return err(new AnalysisCancelledError(operation)); + const input = inspectPacketFieldsInputSchema.safeParse(raw); + if (!input.success) + return err( + analysisInputErrorFromIssues(operation, input.error.issues, raw, { + cause: input.error, + }), + ); + const decoded = await this.provider.inspectFields(input.data, options); + if (!decoded.ok) return decoded; + const execution = decoded.value; + const report = packetFieldsCatalogSchema.safeParse(execution.result); + if ( + !report.success || + JSON.stringify(report.data.selection) !== JSON.stringify(input.data) || + execution.subject?.path !== report.data.tool.path || + execution.subject.sha256 !== report.data.tool.sha256 + ) + return err( + new AnalysisOutputError( + operation, + "Field discovery returned malformed data or changed the selected installation/options.", + ), + ); + return this.#evidence(operation, input.data, execution, options); + } + #evidence( + operation: string, + parameters: Readonly>, + execution: AnalysisExecution, + options?: ExecutionOptions, + ): Result { + const evidence = createEvidence( + execution.subject ?? undefined, + execution.provider, + { + operation, + parameters, + result: execution.result, + confidence: "observed", + locations: execution.locations, + limitations: execution.limitations, + }, + ); return options?.signal?.aborted ? err( new AnalysisCancelledError(operation, { diff --git a/src/application/closeOwnedWebResources.ts b/src/application/closeOwnedWebResources.ts new file mode 100644 index 000000000..51b3e6f69 --- /dev/null +++ b/src/application/closeOwnedWebResources.ts @@ -0,0 +1,15 @@ +/** Attempt every web workflow cleanup owner and report all failures. */ +export const closeOwnedWebResources = async ( + close: readonly ((() => Promise) | undefined)[], + message: string, +): Promise => { + const results = await Promise.allSettled( + close.flatMap((owner) => + owner === undefined ? [] : [Promise.resolve().then(owner)], + ), + ); + const failures = results.flatMap((result) => + result.status === "rejected" ? [result.reason] : [], + ); + if (failures.length > 0) throw new AggregateError(failures, message); +}; diff --git a/src/application/evm/EvmInterfacePort.ts b/src/application/evm/EvmInterfacePort.ts index e9e29b756..a297e092c 100644 --- a/src/application/evm/EvmInterfacePort.ts +++ b/src/application/evm/EvmInterfacePort.ts @@ -4,13 +4,19 @@ import type { } from "../AnalysisProvider.js"; import type { AnalysisError } from "../../domain/analysisErrorBase.js"; import type { InspectEvmInterfaceInput } from "../../domain/evm/evmInterface.js"; +import type { EvidenceSubjectTarget } from "../../domain/evidence.js"; import type { Result } from "../../domain/result.js"; +/** Successful EVM interface recovery must identify the selected local file. */ +export type EvmInterfaceExecution = Omit & { + readonly subject: EvidenceSubjectTarget & { readonly format: "file" }; +}; + /** Replaceable offline interface-recovery engine, independent of wallets and chain clients. */ export interface EvmInterfacePort { close?(): Promise; inspect( input: InspectEvmInterfaceInput, options?: ExecutionOptions, - ): Promise>; + ): Promise>; } diff --git a/src/application/evm/EvmInterfaceService.test.ts b/src/application/evm/EvmInterfaceService.test.ts index 08cf15487..03f501742 100644 --- a/src/application/evm/EvmInterfaceService.test.ts +++ b/src/application/evm/EvmInterfaceService.test.ts @@ -1,5 +1,6 @@ import { expect, it } from "vitest"; import { EvmInterfaceService } from "./EvmInterfaceService.js"; +import type { EvmInterfaceExecution } from "./EvmInterfacePort.js"; import { evmInterfaceExecution, evmInterfaceFixture, @@ -43,7 +44,8 @@ it.each(["carrier", "encoding", "digest", "no-subject"])( }, }; const service = new EvmInterfaceService({ - inspect: () => Promise.resolve(ok(execution)), + inspect: () => + Promise.resolve(ok(execution as unknown as EvmInterfaceExecution)), }); expect( await service.inspect({ diff --git a/src/application/evm/EvmInterfaceService.ts b/src/application/evm/EvmInterfaceService.ts index eac1986d3..86ac77865 100644 --- a/src/application/evm/EvmInterfaceService.ts +++ b/src/application/evm/EvmInterfaceService.ts @@ -67,6 +67,7 @@ export class EvmInterfaceService { !report.success || report.data.artifact.path !== input.data.path || report.data.artifact.encoding !== input.data.encoding || + execution.subject?.format !== "file" || execution.subject?.path !== input.data.path || execution.subject.sha256 !== report.data.artifact.sha256 ) diff --git a/src/application/flutter/DartAotInspection.ts b/src/application/flutter/DartAotInspection.ts new file mode 100644 index 000000000..c11bbe1f4 --- /dev/null +++ b/src/application/flutter/DartAotInspection.ts @@ -0,0 +1,40 @@ +import type { ExecutionOptions } from "../AnalysisProvider.js"; +import type { FlutterBuildAnalysisService } from "./FlutterBuildAnalysisService.js"; +import { + requireAnalysisViewRetention, + summarizeRetainedAnalysis, + type AnalysisViewRetention, +} from "../analysisView/AnalysisViewService.js"; +import type { AnalysisError } from "../../domain/analysisErrorBase.js"; +import { AnalysisCancelledError } from "../../domain/analysisErrorCore.js"; +import { dartAotInspectionInputSchema } from "../../domain/flutter/flutterBuildAnalysis.js"; +import type { Evidence } from "../../domain/evidence.js"; +import { analysisInputErrorFromIssues } from "../../domain/inputIssueProjection.js"; +import { err, type Result } from "../../domain/result.js"; + +/** Keep the caller's ABI choice and complete analysis; select only its delivery. */ +export const inspectDartAot = async ( + service: FlutterBuildAnalysisService, + rawInput: unknown, + retention: AnalysisViewRetention, + options?: ExecutionOptions, +): Promise> => { + const operation = "inspect_dart_aot"; + if (options?.signal?.aborted === true) + return err(new AnalysisCancelledError(operation)); + const parsed = dartAotInspectionInputSchema.safeParse(rawInput); + if (!parsed.success) + return err( + analysisInputErrorFromIssues(operation, parsed.error.issues, rawInput, { + cause: parsed.error, + }), + ); + if (parsed.data.detail === "summary") { + const ready = requireAnalysisViewRetention(operation, retention); + if (!ready.ok) return ready; + } + const { detail, ...providerInput } = parsed.data; + const executed = await service.execute(operation, providerInput, options); + if (!executed.ok || detail !== "summary") return executed; + return summarizeRetainedAnalysis(executed.value, retention); +}; diff --git a/src/application/frida/FridaInstrumentationManager.test.ts b/src/application/frida/FridaInstrumentationManager.test.ts index 29ba97965..7a527a807 100644 --- a/src/application/frida/FridaInstrumentationManager.test.ts +++ b/src/application/frida/FridaInstrumentationManager.test.ts @@ -809,6 +809,42 @@ it("keeps an active script message bucket after its last message is evicted", as await manager.closeAll(); }); +it("retains the newest contiguous messages across repeated budget evictions", async () => { + const { bindings, manager, started } = await startLocalAttachedSession(); + expect(started.ok).toBe(true); + if (!started.ok) return; + bindings.device.session.nextPayload = { index: -1 }; + const loaded = await manager.loadScript(started.value.sessionId, { + sourceKind: "inline", + source: "send({index: -1});", + }); + expect(loaded.ok).toBe(true); + const script = bindings.device.session.scripts[0]; + expect(script).toBeDefined(); + if (script === undefined) return; + + // About 500 of these fit the 16 MiB budget, so 2,000 sends evict and + // compact the queue several times. + const filler = "x".repeat(32_000); + for (let index = 0; index < 2_000; index += 1) + script.message.emit({ type: "send", payload: { index, filler } }, null); + + const status = manager.status(started.value.sessionId); + expect(status?.messagesTruncated).toBe(true); + const indexes = (status?.messages ?? []).map( + (message) => (message as { payload: { index: number } }).payload.index, + ); + expect(indexes.length).toBeGreaterThan(400); + expect(indexes.at(-1)).toBe(1_999); + expect(indexes).toEqual( + Array.from( + { length: indexes.length }, + (_, offset) => 2_000 - indexes.length + offset, + ), + ); + await manager.closeAll(); +}); + it("serializes a final remote release against a new same-endpoint session", async () => { const bindings = new FakeDeviceManager(); let enteredRemoval: (() => void) | undefined; diff --git a/src/application/go/GoBinaryPort.ts b/src/application/go/GoBinaryPort.ts index 2094e9109..cef925921 100644 --- a/src/application/go/GoBinaryPort.ts +++ b/src/application/go/GoBinaryPort.ts @@ -12,6 +12,7 @@ import type { Result } from "../../domain/result.js"; /** Replaceable offline build-information reader with no target-execution authority. */ export interface GoBinaryPort { readonly identity: ProviderIdentity; + close?(): Promise; inspect( input: InspectGoBinaryInput, options?: ExecutionOptions, diff --git a/src/application/go/GoBinaryService.ts b/src/application/go/GoBinaryService.ts index f5642c96c..2b9928f82 100644 --- a/src/application/go/GoBinaryService.ts +++ b/src/application/go/GoBinaryService.ts @@ -34,6 +34,10 @@ const architectureForSubject = (architecture: string) => { export class GoBinaryService { constructor(readonly provider: GoBinaryPort) {} + close(): Promise { + return this.provider.close?.() ?? Promise.resolve(); + } + /** Inspect one explicitly selected file without acquiring a binary session. */ async inspect( rawInput: unknown, diff --git a/src/application/investigation/EvidenceLedger.residualUnknown.test.ts b/src/application/investigation/EvidenceLedger.residualUnknown.test.ts index bb81f97ce..2885c1906 100644 --- a/src/application/investigation/EvidenceLedger.residualUnknown.test.ts +++ b/src/application/investigation/EvidenceLedger.residualUnknown.test.ts @@ -78,6 +78,51 @@ const update = ( }); describe("residual unknown registry reads and identity", () => { + it("seals mutation Evidence without freezing caller-owned inputs", () => { + const store = ledger(); + const recordMutation = mutation("mutable-record-mutation"); + const recorded = store.recordUnknown( + input("Does record mutation evidence stay stable?"), + recordMutation, + ); + if (!recorded.ok) throw recorded.error; + + const updateMutation = mutation("mutable-update-mutation"); + const updated = store.updateUnknown(update(recorded.value), updateMutation); + if (!updated.ok) throw updated.error; + const atomicEvidence = evidence("mutable-atomic-evidence"); + const atomicMutation = mutation("mutable-atomic-mutation"); + const atomicallyRecorded = store.recordWithUnknown( + atomicEvidence, + input("Does atomic evidence stay stable?"), + atomicMutation, + ); + if (!atomicallyRecorded.ok) throw atomicallyRecorded.error; + + for (const callerEvidence of [ + recordMutation, + updateMutation, + atomicEvidence, + atomicMutation, + ]) { + expect(Object.isFrozen(callerEvidence)).toBe(false); + const retained = store.forAnalysis(callerEvidence.evidence_id); + if (retained === undefined) throw new Error("Missing retained Evidence"); + const originalResult = structuredClone(retained.normalized_result); + expect(Object.isFrozen(retained)).toBe(true); + Reflect.set(callerEvidence, "operation", "caller mutation"); + expect( + Reflect.set( + callerEvidence.normalized_result as object, + "label", + "caller nested mutation", + ), + ).toBe(true); + expect(retained.operation).not.toBe("caller mutation"); + expect(retained.normalized_result).toEqual(originalResult); + } + }); + it("merges record-only bundles without changing retained unknown history", () => { const store = ledger(); const created = store.recordUnknown( diff --git a/src/application/investigation/EvidenceLedger.ts b/src/application/investigation/EvidenceLedger.ts index ce806ad38..0cb1c5e29 100644 --- a/src/application/investigation/EvidenceLedger.ts +++ b/src/application/investigation/EvidenceLedger.ts @@ -11,11 +11,7 @@ import { import { AnalysisInputError } from "../../domain/analysisErrorCore.js"; import { EvidenceIntegrityError } from "../../domain/evidenceErrors.js"; import { UnknownRegistryError } from "../../domain/unknownRegistryError.js"; -import { - immutableEvidence, - parseEvidence, - type Evidence, -} from "../../domain/evidence.js"; +import { immutableEvidence, type Evidence } from "../../domain/evidence.js"; import { createResidualUnknown, updateResidualUnknown, @@ -220,7 +216,7 @@ export class EvidenceLedger { ): Result { let parsedMutation: Evidence; try { - parsedMutation = parseEvidence(mutationEvidence); + parsedMutation = immutableEvidence(mutationEvidence); } catch (cause: unknown) { return err( new EvidenceIntegrityError("Unknown mutation evidence is invalid", { @@ -255,8 +251,8 @@ export class EvidenceLedger { let evidence: Evidence; let mutation: Evidence; try { - evidence = parseEvidence(evidenceInput); - mutation = parseEvidence(mutationInput); + evidence = immutableEvidence(evidenceInput); + mutation = immutableEvidence(mutationInput); } catch (cause: unknown) { return err( new EvidenceIntegrityError("Atomic Evidence validation failed", { @@ -394,7 +390,7 @@ export class EvidenceLedger { ): Result { let mutationEvidence: Evidence; try { - mutationEvidence = parseEvidence(mutationEvidenceInput); + mutationEvidence = immutableEvidence(mutationEvidenceInput); } catch (cause: unknown) { return err( new EvidenceIntegrityError("Unknown mutation evidence is invalid", { diff --git a/src/application/javascript/JavaScriptArtifactAnalysis.sourceMaps.test.ts b/src/application/javascript/JavaScriptArtifactAnalysis.sourceMaps.test.ts new file mode 100644 index 000000000..cbd9c745c --- /dev/null +++ b/src/application/javascript/JavaScriptArtifactAnalysis.sourceMaps.test.ts @@ -0,0 +1,71 @@ +import { expect, it } from "vitest"; +import { TraceMap } from "@jridgewell/trace-mapping"; + +import { analyzeJavaScriptArtifactFiles } from "./JavaScriptArtifactAnalysis.js"; + +const HASH = "a".repeat(64); + +it("retains batched TraceMap-resolved paths and null source positions", () => { + const sourceMapText = JSON.stringify({ + version: 3, + file: "main.js", + sourceRoot: "../src/", + sources: ["cafe\u0301.ts", null, "other.ts"], + sourcesContent: ["export const value = 1;", null, "export {};"], + names: [], + mappings: "", + }); + const analysis = analyzeJavaScriptArtifactFiles({ + files: [ + { + path: "dist/main.js.map", + container_sha256: HASH, + sha256: HASH, + bytes: Buffer.byteLength(sourceMapText), + inventory_artifact_id: "artifact_test", + kind: "source-map", + unpacked: false, + text: { included: true, value: sourceMapText }, + }, + ], + containers: [], + text_bytes_read: Buffer.byteLength(sourceMapText), + invalid_utf8_files: 0, + }); + const originals = analysis.source_maps[0]?.sources; + const original = originals?.[0]; + const producerPath = new TraceMap( + { + version: 3, + file: "main.js", + sourceRoot: "../src/", + sources: ["cafe\u0301.ts", "", "other.ts"], + names: [], + mappings: "", + }, + "file:///artifact/dist/main.js.map", + ).resolvedSources[0]; + + if (producerPath === undefined) + throw new Error("TraceMap should resolve its single source declaration"); + expect(originals?.[1]?.reference).toBeNull(); + expect(originals?.[2]?.reference?.resolution).toEqual({ + kind: "artifact-relative", + path: "src/other.ts", + }); + expect(original?.reference).toMatchObject({ + source_name: "cafe\u0301.ts", + source_root: "../src/", + map_path: "dist/main.js.map", + resolution: { + kind: "artifact-relative", + path: "src/cafe\u0301.ts", + }, + }); + expect(original?.reference?.resolution).toEqual({ + kind: "artifact-relative", + path: decodeURIComponent( + new URL(producerPath).pathname.slice("/artifact/".length), + ), + }); +}); diff --git a/src/application/javascript/JavaScriptArtifactAnalysis.ts b/src/application/javascript/JavaScriptArtifactAnalysis.ts index e52ba04dd..e61d3ae94 100644 --- a/src/application/javascript/JavaScriptArtifactAnalysis.ts +++ b/src/application/javascript/JavaScriptArtifactAnalysis.ts @@ -1,9 +1,9 @@ import { createHash } from "node:crypto"; -import { moduleKindForJavaScriptReference } from "./JavaScriptArtifactPathResolution.js"; +import { moduleResolutionModeForJavaScriptReference } from "../../domain/javascript/javascriptModuleResolutionMode.js"; import { setImmediate } from "node:timers/promises"; -import { resolveJavaScriptSourceMapReference } from "../../domain/javascript/javascriptSourceMapPaths.js"; +import { resolveJavaScriptSourceMapReferences } from "../../domain/javascript/javascriptSourceMapPaths.js"; import { analyzeParsedJavaScriptStaticSourceSteps } from "../../domain/javascript/javascriptStaticAnalysis.js"; import { analyzeParsedJavaScriptSemanticsSteps, @@ -384,7 +384,10 @@ function* analyzeArtifactFileSteps< analysis.parse_status === "complete" || analysis.parse_status === "partial" ? yield* analyzeParsedJavaScriptSemanticsSteps( parsed, - moduleKindForJavaScriptReference(file.path, "static-import"), + moduleResolutionModeForJavaScriptReference( + file.path, + "static-import", + ), ) : null; const projected = @@ -561,21 +564,27 @@ const collectSourceMapOriginals = ( "Source map sourceRoot must be a string or null.", ); const root = typeof map.sourceRoot === "string" ? map.sourceRoot : null; - for (const [index, raw] of names.entries()) { + const sourceNames: (string | null)[] = []; + for (const raw of names) { if (raw !== null && typeof raw !== "string") return invalidSourceMap( file, "Source map contains a non-string source name.", ); + sourceNames.push(raw); + } + const references = resolveJavaScriptSourceMapReferences( + sourceNames, + root, + file.path, + ); + for (const [index, reference] of references.entries()) { const rawContent = Array.isArray(contents) ? contents[index] : undefined; const content = typeof rawContent === "string" ? rawContent : null; sources.push({ source_index: index, declaration_index: declarationIndex++, - reference: - typeof raw === "string" - ? resolveJavaScriptSourceMapReference(raw, root, file.path) - : null, + reference, content, content_sha256: content === null ? null : sha256(content), }); diff --git a/src/application/javascript/JavaScriptArtifactGraphContext.ts b/src/application/javascript/JavaScriptArtifactGraphContext.ts index 8b0fed5f1..1779bdaae 100644 --- a/src/application/javascript/JavaScriptArtifactGraphContext.ts +++ b/src/application/javascript/JavaScriptArtifactGraphContext.ts @@ -66,6 +66,7 @@ export interface AstEdgeInput { /** Inferred static relationship edge values. */ export interface InferenceEdgeInput extends AstEdgeInput { + readonly limitations?: readonly string[]; readonly relation: | "imports" | "loads" @@ -162,6 +163,9 @@ export const addStaticInferenceEdge = ( range: input.range, operation: "map-static-relationship", coverage: input.coverage, + ...(input.limitations === undefined + ? {} + : { limitations: input.limitations }), }), }); }; @@ -347,7 +351,7 @@ export const resolveArtifactPath = ( specifier: string, sourcePath: string, files: ReadonlyMap, - moduleKind?: "import" | "require", + moduleKind?: "import" | "require" | "compiler-dependent", ): string | null => resolveArtifactPathByContext({ declaredPath: specifier, diff --git a/src/application/javascript/JavaScriptArtifactGraphFindings.ts b/src/application/javascript/JavaScriptArtifactGraphFindings.ts index 3e92ff1a4..0020ef577 100644 --- a/src/application/javascript/JavaScriptArtifactGraphFindings.ts +++ b/src/application/javascript/JavaScriptArtifactGraphFindings.ts @@ -12,7 +12,6 @@ import { javascriptAnalysisCoverage, linkElectronRoleToAsset, moduleLookupKey, - resolveArtifactPath, selfReferenceOmissions, sourceNodeFor, type JavaScriptArtifactGraphContext, @@ -23,9 +22,11 @@ import { staticInferenceEvidence, } from "./JavaScriptArtifactGraphEvidence.js"; import { - moduleKindForJavaScriptReference, + artifactPathResolutionProperties, resolveArtifactPathByContext, + type ArtifactPathResolution, } from "./JavaScriptArtifactPathResolution.js"; +import { moduleResolutionModeForJavaScriptReference } from "../../domain/javascript/javascriptModuleResolutionMode.js"; type StaticReference = JavaScriptStaticAnalysis["references"][number]; type StaticEndpoint = JavaScriptStaticAnalysis["endpoints"][number]; @@ -41,9 +42,12 @@ interface FindingInput { } interface ResolvedReference { - readonly path: string; - readonly file: JavaScriptArtifactFile; + readonly path: string | null; + readonly file: JavaScriptArtifactFile | null; readonly node: ApplicationNode; + readonly limitations: readonly string[]; + readonly resolutionStatus?: string; + readonly resolution?: ArtifactPathResolution; } /** Project imports, workers, roles, endpoints, storage, and source-map links. */ @@ -90,7 +94,7 @@ const addReference = ( const resolved = input.value.specifier === null ? null - : resolveReference(context, input.file.path, input.value); + : resolveReference(context, input.file.path, input.value, input.coverage); if (input.value.kind === "worker" || input.value.kind === "service-worker") { const worker = context.accumulator.addNode({ kind: input.value.kind === "worker" ? "worker" : "service-worker", @@ -126,7 +130,7 @@ const addReference = ( relation: "loads", properties: { kind: input.value.kind }, }); - if (resolved !== null) + if (resolved !== null && resolved.path !== null) addStaticInferenceEdge(context, { source: worker, target: resolved.node, @@ -135,6 +139,7 @@ const addReference = ( coverage: input.coverage, relation: "maps_to", properties: { resolved_path: resolved.path }, + limitations: resolved.limitations, }); return; } @@ -154,13 +159,25 @@ const addReference = ( file: input.file, range: input.value.location, coverage: input.coverage, - relation: resolved?.file.kind === "native-addon" ? "loads" : "imports", + relation: resolved?.file?.kind === "native-addon" ? "loads" : "imports", properties: { kind: input.value.kind, specifier: input.value.specifier, expression: input.value.expression, + resolution_mode: + input.value.kind === "static-import" || + input.value.kind === "dynamic-import" || + input.value.kind === "require" + ? moduleResolutionModeForJavaScriptReference( + input.file.path, + input.value.kind, + ) + : null, resolved_path: resolved?.path ?? null, + resolution_status: resolved?.resolutionStatus ?? null, + ...artifactPathResolutionProperties(resolved?.resolution), }, + ...(resolved === null ? {} : { limitations: resolved.limitations }), }); }; @@ -373,6 +390,7 @@ const resolveReference = ( context: JavaScriptArtifactGraphContext, sourcePath: string, reference: StaticReference, + coverage: JavaScriptArtifactGraphCoverage, ): ResolvedReference | null => { const specifier = reference.specifier; if (specifier === null) return null; @@ -392,30 +410,70 @@ const resolveReference = ( const file = context.filesByPath.get(sourcePath); return file === undefined ? null - : { path: `${sourcePath}#module:${specifier}`, file, node: module }; + : { + path: `${sourcePath}#module:${specifier}`, + file, + node: module, + limitations: [], + }; } // Worker scripts are URLs relative to the script, not module specifiers. - const path = isWorker + const resolution = isWorker ? resolveArtifactPathByContext({ declaredPath: specifier, sourcePath, context: "url-reference", files: context.filesByPath, - }).resolved_path - : resolveArtifactPath( - specifier, + }) + : resolveArtifactPathByContext({ + declaredPath: specifier, sourcePath, - context.filesByPath, - reference.kind === "require" || - reference.kind === "static-import" || - reference.kind === "dynamic-import" - ? moduleKindForJavaScriptReference(sourcePath, reference.kind) - : undefined, - ); - if (path === null) return null; + context: "module-specifier", + files: context.filesByPath, + ...("require" === reference.kind || + "static-import" === reference.kind || + "dynamic-import" === reference.kind + ? { + moduleKind: moduleResolutionModeForJavaScriptReference( + sourcePath, + reference.kind, + ), + } + : {}), + }); + const path = resolution.resolved_path; + if (path === null) { + if (isWorker) return null; + const source = context.filesByPath.get(sourcePath); + return source === undefined + ? null + : { + path: null, + file: null, + node: unresolvedReferenceNode( + context, + source, + reference, + coverage, + resolution, + ), + limitations: resolution.limitations, + resolutionStatus: resolution.resolution_status, + resolution, + }; + } const file = context.filesByPath.get(path); const node = context.fileNodes.get(path); - return file === undefined || node === undefined ? null : { path, file, node }; + return file === undefined || node === undefined + ? null + : { + path, + file, + node, + limitations: resolution.limitations, + resolutionStatus: resolution.resolution_status, + resolution, + }; }; const resolveBundlerChunkReference = ( @@ -453,7 +511,7 @@ const resolveBundlerChunkReference = ( const node = unique[0]; return node === undefined ? null - : { path: `${sourcePath}#chunk:${key}`, file, node }; + : { path: `${sourcePath}#chunk:${key}`, file, node, limitations: [] }; }; const unresolvedReferenceNode = ( @@ -461,6 +519,7 @@ const unresolvedReferenceNode = ( file: JavaScriptArtifactFile, reference: StaticReference, coverage: JavaScriptArtifactGraphCoverage, + resolution?: ArtifactPathResolution, ): ApplicationNode => context.accumulator.addNode({ kind: "javascript-module", @@ -477,6 +536,10 @@ const unresolvedReferenceNode = ( specifier: reference.specifier, expression: reference.expression, resolution: "unresolved", + ...(resolution === undefined + ? {} + : { resolution_status: resolution.resolution_status }), + ...artifactPathResolutionProperties(resolution), }, evidence: staticInferenceEvidence({ sha256: file.sha256, @@ -487,6 +550,7 @@ const unresolvedReferenceNode = ( confidence: "low", limitations: [ "The static reference could not be resolved within the inventoried artifact.", + ...(resolution?.limitations ?? []), ], }), }, diff --git a/src/application/javascript/JavaScriptArtifactPathResolution.test.ts b/src/application/javascript/JavaScriptArtifactPathResolution.test.ts index 1a8713929..b746d7e97 100644 --- a/src/application/javascript/JavaScriptArtifactPathResolution.test.ts +++ b/src/application/javascript/JavaScriptArtifactPathResolution.test.ts @@ -1890,21 +1890,40 @@ describe("TypeScript NodeNext emitted-extension resolution", () => { ).toMatchObject({ resolution_status: "resolved", resolved_path: target }); }, ); +}); + +describe("TypeScript declaration fallback ownership", () => { + const container = "container"; it.each([ ["./dep.js", "src/dep.d.ts"], ["./cfg.mjs", "src/cfg.d.mts"], ["./legacy.cjs", "src/legacy.d.cts"], ] as const)( - "resolves %s to a declaration counterpart", + "retains %s as compile-time evidence without inventing a runtime target", (specifier, target) => { const files = fileMap([ file("src/main.ts", container), file(target, container, "export const value = 1;"), ]); + for (const moduleKind of ["import", "require"] as const) { + expect( + resolve({ + declaredPath: specifier, + sourcePath: "src/main.ts", + context: "module-specifier", + moduleKind, + files, + }), + ).toMatchObject({ + resolution_status: "unavailable", + resolved_path: null, + candidate_path: target, + }); + } expect( resolve({ - declaredPath: specifier, + declaredPath: "./" + target.slice(4), sourcePath: "src/main.ts", context: "module-specifier", moduleKind: "import", diff --git a/src/application/javascript/JavaScriptArtifactPathResolution.ts b/src/application/javascript/JavaScriptArtifactPathResolution.ts index b3d769af6..cb29ab2a9 100644 --- a/src/application/javascript/JavaScriptArtifactPathResolution.ts +++ b/src/application/javascript/JavaScriptArtifactPathResolution.ts @@ -19,7 +19,13 @@ import { classifyPackageImportTargetString, lookupExactPackageImport, } from "../../domain/javascript/javascriptPackageImportAliases.js"; -import { typescriptEmittedSourceCandidates } from "../TypeScriptEmittedSourceCandidates.js"; +import { + typescriptEmittedSourceCandidates, + type TypeScriptEmittedSourceCandidate, +} from "../TypeScriptEmittedSourceCandidates.js"; +import { isRelativeJavaScriptModuleSpecifier } from "../../domain/javascript/javascriptModuleSpecifier.js"; +import type { JavaScriptModuleResolutionMode } from "../../domain/javascript/javascriptModuleResolutionMode.js"; +import type { JsonValue } from "../../domain/jsonValue.js"; type ArtifactPathResolutionContext = | "package-entrypoint" @@ -38,6 +44,16 @@ interface ArtifactPathResolutionBase { readonly declared_path: string; readonly resolution_context: ArtifactPathResolutionContext; readonly limitations: readonly string[]; + readonly conditional_resolutions?: readonly ConditionalModuleResolution[]; + readonly candidate_path?: string; +} + +export interface ConditionalModuleResolution { + readonly mode: "import" | "require"; + readonly resolution_status: UnresolvedArtifactPathStatus | "resolved"; + readonly resolved_path: string | null; + readonly candidate_path?: string; + readonly limitations: readonly string[]; } /** Explicit outcome of artifact-confined path resolution. */ @@ -61,26 +77,13 @@ export interface ResolveArtifactPathInput { readonly files: ReadonlyMap; readonly htmlBaseHref?: string | null; /** Import/require semantics; runtime resource paths omit this provenance. */ - readonly moduleKind?: "import" | "require"; + readonly moduleKind?: JavaScriptModuleResolutionMode; } -/** Loader selected by a source-level JavaScript module reference. */ -export const moduleKindForJavaScriptReference = ( - sourcePath: string, - referenceKind: "static-import" | "dynamic-import" | "require", - resolutionMode?: "import" | "require", -): "import" | "require" => - resolutionMode ?? - (referenceKind === "require" || - (referenceKind === "static-import" && - posix.extname(sourcePath).toLowerCase() === ".cts") - ? "require" - : "import"); - /** Preserve an explicitly retained loader mode on graph relationships. */ export const moduleResolutionModeProperties = ( - mode: "import" | "require" | undefined, -): Readonly> => + mode: JavaScriptModuleResolutionMode | undefined, +): Readonly> => mode === undefined ? {} : { resolution_mode: mode }; const EXTENSIONS = [ @@ -109,43 +112,116 @@ interface PackageFileTarget { readonly url: URL; } -type CandidateResolution = - | { - readonly resolvedPath: string; - readonly status: "resolved"; - readonly limitations: readonly string[]; - } +type CandidateResolution = { + readonly limitations: readonly string[]; + readonly candidatePath?: string; + readonly conditionalResolutions?: readonly ConditionalModuleResolution[]; +} & ( + | { readonly resolvedPath: string; readonly status: "resolved" } | { readonly resolvedPath: null; readonly status: UnresolvedArtifactPathStatus; - readonly limitations: readonly string[]; /** A truthy package main failed, so Node must not try ancestor packages. */ readonly stopsAncestorLookup?: true; - }; + } +); /** Resolve one declaration while preserving its original input identity. */ export const resolveArtifactPathByContext = ( input: ResolveArtifactPathInput, ): ArtifactPathResolution => { const resolution = resolvePath(input); + const facts = { + declared_path: input.declaredPath, + resolution_context: input.context, + limitations: resolution.limitations, + ...(resolution.candidatePath === undefined + ? {} + : { candidate_path: resolution.candidatePath }), + ...(resolution.conditionalResolutions === undefined + ? {} + : { conditional_resolutions: resolution.conditionalResolutions }), + }; return resolution.status === "resolved" ? { - declared_path: input.declaredPath, - resolution_context: input.context, + ...facts, resolved_path: resolution.resolvedPath, resolution_status: resolution.status, - limitations: resolution.limitations, } - : { - declared_path: input.declaredPath, - resolution_context: input.context, - resolved_path: null, - resolution_status: resolution.status, - limitations: resolution.limitations, - }; + : { ...facts, resolved_path: null, resolution_status: resolution.status }; }; +/** Project retained conditional identities without flattening them into display text. */ +export const artifactPathResolutionProperties = ( + resolution: ArtifactPathResolution | undefined, +): Readonly> => ({ + ...(resolution?.candidate_path === undefined + ? {} + : { candidate_path: resolution.candidate_path }), + ...(resolution?.conditional_resolutions === undefined + ? {} + : { + conditional_resolutions: resolution.conditional_resolutions.map( + (branch) => ({ + mode: branch.mode, + resolution_status: branch.resolution_status, + resolved_path: branch.resolved_path, + limitations: [...branch.limitations], + ...(branch.candidate_path === undefined + ? {} + : { candidate_path: branch.candidate_path }), + }), + ), + }), +}); + +const conditionalResolution = ( + mode: "import" | "require", + result: CandidateResolution, +): ConditionalModuleResolution => ({ + mode, + resolution_status: result.status, + resolved_path: result.resolvedPath, + limitations: result.limitations, + ...(result.candidatePath === undefined + ? {} + : { candidate_path: result.candidatePath }), +}); + const resolvePath = (input: ResolveArtifactPathInput): CandidateResolution => { + if (input.moduleKind === "compiler-dependent") { + const imported = resolvePath({ ...input, moduleKind: "import" }); + const required = resolvePath({ ...input, moduleKind: "require" }); + const sameOutcome = + imported.status === required.status && + imported.resolvedPath === required.resolvedPath && + imported.candidatePath === required.candidatePath; + const limitation = + "Static TypeScript loader mode is compiler-dependent without retained compiler options. " + + (sameOutcome + ? "Node import and require resolution agree on this outcome." + : "Node import and require resolution disagree; neither branch is an established target."); + return sameOutcome + ? { + ...imported, + limitations: [ + ...new Set([ + ...imported.limitations, + ...required.limitations, + limitation, + ]), + ], + } + : { + resolvedPath: null, + status: "unavailable", + limitations: [limitation], + conditionalResolutions: [ + conditionalResolution("import", imported), + conditionalResolution("require", required), + ], + }; + } // Package import keys are literal lookup identities, not artifact paths. // The selected target is admitted separately by resolveExportTarget. if ( @@ -166,7 +242,7 @@ const resolvePath = (input: ResolveArtifactPathInput): CandidateResolution => { ); const isModuleUrlPath = input.context === "module-specifier" && - (input.declaredPath.startsWith(".") || + (isRelativeJavaScriptModuleSpecifier(input.declaredPath) || input.declaredPath.startsWith("/") || fileUrlPath(input.declaredPath) !== undefined); const emittedSourceCandidates = @@ -175,12 +251,47 @@ const resolvePath = (input: ResolveArtifactPathInput): CandidateResolution => { : []; if (isModuleUrlPath && input.moduleKind === "import") return ( - resolveFileCandidates(input, [confined, ...emittedSourceCandidates]) ?? - notFoundCandidate() + resolveRuntimeSourceCandidates( + input, + confined, + emittedSourceCandidates, + ) ?? notFoundCandidate() ); return resolveCandidate(input, confined, new Set(), emittedSourceCandidates); }; +const resolveRuntimeSourceCandidates = ( + input: ResolveArtifactPathInput, + exactPath: string, + sourceCandidates: readonly TypeScriptEmittedSourceCandidate[], +): CandidateResolution | null => { + const source = input.files.get(input.sourcePath); + const exactCandidate: TypeScriptEmittedSourceCandidate = { + path: exactPath, + kind: "implementation", + }; + for (const candidate of [exactCandidate, ...sourceCandidates]) { + const target = input.files.get(candidate.path); + if ( + target === undefined || + (source !== undefined && + target.container_sha256 !== source.container_sha256) + ) + continue; + return candidate.kind === "declaration" + ? { + resolvedPath: null, + status: "unavailable", + candidatePath: candidate.path, + limitations: [ + "Only a TypeScript declaration fallback was found; it does not establish the requested runtime module.", + ], + } + : { resolvedPath: candidate.path, status: "resolved", limitations: [] }; + } + return null; +}; + const rejectDeclaration = (declared: string): CandidateResolution | null => { if (declared.length === 0) return unresolvedCandidate("rejected", ["The declared path is empty."]); @@ -229,7 +340,10 @@ const contextualCandidate = ( return unresolvedCandidate("external", [ "URL and Node builtin schemes are outside static artifact module resolution.", ]); - if (!declared.startsWith(".") && !declared.startsWith("/")) + if ( + !isRelativeJavaScriptModuleSpecifier(declared) && + !declared.startsWith("/") + ) return bareModuleCandidate(input); if (input.moduleKind !== "require") { const decoded = decodeModuleUrlPath(stripQueryAndFragment(declared)); @@ -275,7 +389,10 @@ const contextualCandidate = ( "URL", ); } - return relativeCandidate(input.sourcePath, declared); + const candidate = relativeCandidate(input.sourcePath, declared); + return context === "module-specifier" && input.moduleKind === "import" + ? exactUrlPathCandidate(declared, candidate, "module URL") + : candidate; }; const decodeModuleUrlPath = ( @@ -310,9 +427,12 @@ const bareModuleCandidate = ( return unresolvedCandidate("external", [ "The bare specifier is a Node builtin or invalid package name.", ]); - if (input.moduleKind === "import" && request.packageName.includes("%")) + if ( + input.moduleKind === "import" && + (request.packageName.startsWith(".") || request.packageName.includes("%")) + ) return unresolvedCandidate("rejected", [ - "Percent signs are not admitted in ES module package names.", + "ES module package names cannot start with a dot or contain percent signs.", ]); const self = packageSelfCandidate(input, request); if (self !== null) return self; @@ -735,7 +855,7 @@ const htmlCandidate = ( const exactUrlPathCandidate = ( declared: string, candidate: string, - kind: "HTML" | "URL", + kind: "HTML" | "URL" | "module URL", ): string | CandidateResolution => { if (!urlPathIsDirectory(declared)) return candidate; const confined = confineCandidate(candidate); @@ -839,12 +959,10 @@ const resolveCandidate = ( input: ResolveArtifactPathInput, candidate: string, packageChain: ReadonlySet, - emittedSourceCandidates: readonly string[] = [], + emittedSourceCandidates: readonly TypeScriptEmittedSourceCandidate[] = [], ): CandidateResolution => - resolveFileCandidates(input, [ - ...fileCandidates(candidate, input.moduleKind), - ...emittedSourceCandidates, - ]) ?? + resolveFileCandidates(input, fileCandidates(candidate, input.moduleKind)) ?? + resolveRuntimeSourceCandidates(input, candidate, emittedSourceCandidates) ?? resolveDirectory( input, candidate, @@ -930,11 +1048,26 @@ const resolveFileCandidates = ( (source === undefined || target.container_sha256 === source.container_sha256) ) - return { resolvedPath: path, status: "resolved", limitations: [] }; + return isRuntimeDeclarationPath(input, path) + ? { + resolvedPath: null, + status: "unavailable", + candidatePath: path, + limitations: [ + `The TypeScript declaration ${path} is compile-time evidence and cannot identify a runtime module implementation.`, + ], + } + : { resolvedPath: path, status: "resolved", limitations: [] }; } return null; }; +const isRuntimeDeclarationPath = ( + input: ResolveArtifactPathInput, + path: string, +): boolean => + input.moduleKind !== undefined && /\.d\.(?:ts|mts|cts)$/iu.test(path); + const NODE_MODULE_EXTENSIONS = [".js", ".json", ".node"] as const; const fileCandidates = ( diff --git a/src/application/javascript/JavaScriptModuleRelationships.ts b/src/application/javascript/JavaScriptModuleRelationships.ts index 874377810..3b46e95ae 100644 --- a/src/application/javascript/JavaScriptModuleRelationships.ts +++ b/src/application/javascript/JavaScriptModuleRelationships.ts @@ -18,6 +18,7 @@ import type { JavaScriptModuleOrigin, JavaScriptSemanticModuleLink, } from "../../domain/javascript/javascriptSemanticIr.js"; +import { moduleResolutionModeForJavaScriptReference } from "../../domain/javascript/javascriptModuleResolutionMode.js"; import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js"; import { addAstContainsEdge, @@ -30,7 +31,7 @@ import { staticInferenceEvidence, } from "./JavaScriptArtifactGraphEvidence.js"; import { - moduleKindForJavaScriptReference, + artifactPathResolutionProperties, moduleResolutionModeProperties, resolveArtifactPathByContext, type ArtifactPathResolution, @@ -54,6 +55,7 @@ interface RelationshipInput { type ResolvedModuleTarget = { readonly node: ApplicationNode; readonly limitations: readonly string[]; + readonly resolution?: ArtifactPathResolution; } & ( | { readonly path: string; @@ -81,8 +83,13 @@ export const addJavaScriptSourceModules = ( if (asset === undefined || semantic === null || program === undefined) continue; const exports = semantic.ir.moduleLinks - .flatMap(({ exportedName }) => - exportedName === null ? [] : [exportedName], + .flatMap(({ exportedName, kind }) => + exportedName === null || kind === "type-import" ? [] : [exportedName], + ) + .sort(compareUnicodeCodePoints); + const typeExports = semantic.ir.moduleLinks + .flatMap(({ exportedName, kind }) => + exportedName === null || kind !== "type-import" ? [] : [exportedName], ) .sort(compareUnicodeCodePoints); const retainedExports = [...new Set(exports)]; @@ -105,8 +112,12 @@ export const addJavaScriptSourceModules = ( import_relationships: semantic.ir.moduleLinks.filter( ({ kind }) => kind === "import" || kind === "require", ).length, + type_relationships: semantic.ir.moduleLinks.filter( + ({ kind }) => kind === "type-import", + ).length, export_relationships: exports.length, export_names: retainedExports, + type_export_names: [...new Set(typeExports)], omitted_export_names: 0, }, evidence: astObservationEvidence({ @@ -152,6 +163,7 @@ export function* addJavaScriptModuleRelationshipsSteps( for (const link of semantic.ir.moduleLinks) { // One module can export thousands of bindings, each with return shapes. yield; + if (link.kind === "type-import") continue; const input = { context, file, semantic, source, link }; if (isExportLink(link)) addExportRelationship(input, omissions); else if (link.specifier !== null) @@ -277,7 +289,7 @@ const resolveModuleTarget = ( sourcePath: input.file.path, context: "module-specifier", files: input.context.filesByPath, - moduleKind: moduleKindForJavaScriptReference( + moduleKind: moduleResolutionModeForJavaScriptReference( input.file.path, input.link.kind === "require" || input.link.kind === "commonjs-export" ? "require" @@ -287,15 +299,12 @@ const resolveModuleTarget = ( }); if (resolution.resolution_status !== "resolved") return { - node: unresolvedModuleNode( - input, - specifier, - resolution.resolution_status, - ), + node: unresolvedModuleNode(input, specifier, resolution), path: null, file: null, status: resolution.resolution_status, limitations: resolution.limitations, + resolution, }; const path = resolution.resolved_path; const file = input.context.filesByPath.get(path); @@ -312,20 +321,24 @@ const resolveModuleTarget = ( file, status: "resolved", limitations: resolution.limitations, + resolution, }; }; const unresolvedModuleNode = ( input: RelationshipInput, specifier: string, - status: Exclude, + resolution: Extract< + ArtifactPathResolution, + { resolution_status: Exclude } + >, ): ApplicationNode => input.context.accumulator.addNode({ kind: "javascript-module", identity: artifactLocalIdentity( input.file.sha256, "unresolved-semantic-module", - `${status}:${specifier}`, + `${resolution.resolution_status}:${specifier}`, ), observations: [ { @@ -333,7 +346,8 @@ const unresolvedModuleNode = ( properties: { semantic_role: "module-reference", declared_specifier: specifier, - resolution_status: status, + resolution_status: resolution.resolution_status, + ...artifactPathResolutionProperties(resolution), }, evidence: staticInferenceEvidence({ sha256: input.file.sha256, @@ -354,6 +368,9 @@ const moduleOriginForExport = ( ): JavaScriptModuleOrigin | null => { if (link.specifier !== null) return { + ...(link.resolutionMode === undefined + ? {} + : { resolutionMode: link.resolutionMode }), specifier: link.specifier, importedPath: link.importedName === null ? [] : [link.importedName], }; @@ -383,6 +400,7 @@ const relationshipProperties = ( exported_name: link.exportedName, resolved_path: target?.path ?? null, resolution_status: target?.status ?? null, + ...artifactPathResolutionProperties(target?.resolution), target_file_kind: target?.file?.kind ?? null, target_json_status: target?.file?.kind === "json" || target?.file?.kind === "package-json" @@ -431,6 +449,7 @@ const moduleFormat = ( const extension = posix.extname(path).toLowerCase(); if (extension === ".mjs" || extension === ".mts") return "esm"; if (extension === ".cjs" || extension === ".cts") return "commonjs"; + if (extension === ".ts" || extension === ".tsx") return "unknown"; const esm = ir.moduleLinks.some(({ kind }) => ["import", "export", "re-export"].includes(kind), ); diff --git a/src/application/managed/ManagedReconstructionService.test.ts b/src/application/managed/ManagedReconstructionService.test.ts index 56b9d9442..793951ae9 100644 --- a/src/application/managed/ManagedReconstructionService.test.ts +++ b/src/application/managed/ManagedReconstructionService.test.ts @@ -22,6 +22,26 @@ const exampleMethod = () => { }; describe("managed decompiler reconstruction import", () => { + it("reports UTF-8 byte totals while preserving UTF-16 text lengths", () => { + const text = 'return "é"; // 😀'; + const method = exampleMethod(); + const result = importManagedReconstructionEvidence({ + ...MANAGED_RECONSTRUCTION_IMPORT_EXAMPLE, + methods: [ + { + ...method, + reconstruction: { ...method.reconstruction, text }, + }, + ], + }); + if (!result.ok) throw result.error; + + expect(result.value.normalized_result).toMatchObject({ + summary: { total_text_bytes: 20 }, + methods: [{ reconstruction: { text, text_length: 17 } }], + }); + }); + it("rejects reconstruction text hash drift", () => { expect(() => importManagedReconstruction({ diff --git a/src/application/pcap/PcapAnalysisPort.ts b/src/application/pcap/PcapAnalysisPort.ts new file mode 100644 index 000000000..c5acecd38 --- /dev/null +++ b/src/application/pcap/PcapAnalysisPort.ts @@ -0,0 +1,23 @@ +import type { + AnalysisExecution, + ExecutionOptions, + ProviderAvailability, +} from "../AnalysisProvider.js"; +import type { PcapRequest } from "../../domain/pcap/pcapAnalysis.js"; +import type { AnalysisError } from "../../domain/analysisErrorBase.js"; +import type { Result } from "../../domain/result.js"; + +/** + * Packet-capture analysis boundary over caller-supplied tshark/capinfos. + * Implementation details belong to the provider adapter. + */ +export interface PcapAnalysisPort { + /** Probe both tools without reading any capture file. */ + inspectAvailability(signal?: AbortSignal): Promise; + /** Retry cleanup of retained owned commands and capture snapshots. */ + close(): Promise; + execute( + request: PcapRequest, + options?: ExecutionOptions, + ): Promise>; +} diff --git a/src/application/pcap/PcapAnalysisService.ts b/src/application/pcap/PcapAnalysisService.ts new file mode 100644 index 000000000..23b968e58 --- /dev/null +++ b/src/application/pcap/PcapAnalysisService.ts @@ -0,0 +1,36 @@ +import type { PcapAnalysisPort } from "./PcapAnalysisPort.js"; +import type { ExecutionOptions } from "../AnalysisProvider.js"; +import { + pcapRequestSchema, + type PcapOperation, +} from "../../domain/pcap/pcapAnalysis.js"; +import { + AnalysisCancelledError, + AnalysisInputError, +} from "../../domain/analysisErrorCore.js"; +import type { AnalysisError } from "../../domain/analysisErrorBase.js"; +import type { Evidence } from "../../domain/evidence.js"; +import { createPcapEvidence } from "../../domain/pcap/pcapEvidence.js"; +import { err, ok, type Result } from "../../domain/result.js"; + +/** Shared CLI/MCP admission, execution, and Evidence composition for pcap. */ +export class PcapAnalysisService { + constructor(readonly provider: PcapAnalysisPort) {} + + /** Run one caller-selected pcap request and retain capture identity inline. */ + async execute( + operation: PcapOperation, + input: unknown, + options?: ExecutionOptions, + ): Promise> { + if (options?.signal?.aborted === true) + return err(new AnalysisCancelledError(operation)); + const request = pcapRequestSchema.safeParse({ operation, input }); + if (!request.success) + return err(new AnalysisInputError(operation, { cause: request.error })); + const executed = await this.provider.execute(request.data, options); + if (!executed.ok) return executed; + const execution = executed.value; + return ok(createPcapEvidence(request.data, execution)); + } +} diff --git a/src/application/signals/SignalRecordingService.ts b/src/application/signals/SignalRecordingService.ts index 921ac19c7..e5b99f1b7 100644 --- a/src/application/signals/SignalRecordingService.ts +++ b/src/application/signals/SignalRecordingService.ts @@ -1,3 +1,9 @@ +import { + inspectSignalDecodersInputSchema, + signalDecodersSchema, + type InspectSignalDecodersInput, +} from "../../domain/signals/signalDecoders.js"; +import type { JsonValue } from "../../domain/jsonValue.js"; import { isAbsolute } from "node:path"; import type { AnalysisExecution, @@ -24,6 +30,10 @@ export interface SignalRecordingPort { input: DecodeSignalRecordingInput, options?: ExecutionOptions, ): Promise>; + inspectDecoders( + input: InspectSignalDecodersInput, + options?: ExecutionOptions, + ): Promise>; close(): Promise; } @@ -74,14 +84,58 @@ export class SignalRecordingService { "Decoder returned malformed data or changed the selected recording identity/options.", ), ); - const evidence = createEvidence(execution.subject, execution.provider, { - operation, - parameters: input.data, - result: report.data, - confidence: "observed", - locations: execution.locations, - limitations: execution.limitations, - }); + return this.#evidence(operation, input.data, execution, options); + } + async inspectDecoders( + raw: unknown, + options?: ExecutionOptions, + ): Promise> { + const operation = "inspect_signal_decoders"; + if (options?.signal?.aborted) + return err(new AnalysisCancelledError(operation)); + const input = inspectSignalDecodersInputSchema.safeParse(raw); + if (!input.success) + return err( + analysisInputErrorFromIssues(operation, input.error.issues, raw, { + cause: input.error, + }), + ); + const decoded = await this.provider.inspectDecoders(input.data, options); + if (!decoded.ok) return decoded; + const execution = decoded.value; + const report = signalDecodersSchema.safeParse(execution.result); + if ( + !report.success || + JSON.stringify(report.data.selection) !== JSON.stringify(input.data) || + execution.subject?.path !== report.data.tool.path || + execution.subject.sha256 !== report.data.tool.sha256 + ) + return err( + new AnalysisOutputError( + operation, + "Decoder discovery returned malformed data or changed the selected installation/options.", + ), + ); + return this.#evidence(operation, input.data, execution, options); + } + #evidence( + operation: string, + parameters: Readonly>, + execution: AnalysisExecution, + options?: ExecutionOptions, + ): Result { + const evidence = createEvidence( + execution.subject ?? undefined, + execution.provider, + { + operation, + parameters, + result: execution.result, + confidence: "observed", + locations: execution.locations, + limitations: execution.limitations, + }, + ); return options?.signal?.aborted ? err( new AnalysisCancelledError(operation, { diff --git a/src/application/wasm/WasmArtifactPort.ts b/src/application/wasm/WasmArtifactPort.ts index 6cf6de352..90daab4b1 100644 --- a/src/application/wasm/WasmArtifactPort.ts +++ b/src/application/wasm/WasmArtifactPort.ts @@ -4,8 +4,14 @@ import type { } from "../AnalysisProvider.js"; import type { AnalysisError } from "../../domain/analysisErrorBase.js"; import type { InspectWasmArtifactInput } from "../../domain/wasm/wasmArtifact.js"; +import type { EvidenceSubjectTarget } from "../../domain/evidence.js"; import type { Result } from "../../domain/result.js"; +/** Successful WASM inspection must identify the selected local file. */ +export type WasmArtifactExecution = Omit & { + readonly subject: EvidenceSubjectTarget & { readonly format: "file" }; +}; + /** Replaceable offline WASM inspection engine, independent of target execution and network clients. */ export interface WasmArtifactPort { /** Retry removal of retained workspaces; failures preserve ownership for another attempt. */ @@ -13,5 +19,5 @@ export interface WasmArtifactPort { inspect( input: InspectWasmArtifactInput, options?: ExecutionOptions, - ): Promise>; + ): Promise>; } diff --git a/src/artifacts/ArtifactProvider.ts b/src/artifacts/ArtifactProvider.ts index c95cec6d5..47d5906ea 100644 --- a/src/artifacts/ArtifactProvider.ts +++ b/src/artifacts/ArtifactProvider.ts @@ -40,6 +40,7 @@ import { ArtifactResourceScope } from "./ArtifactResourceScope.js"; import { artifactCapabilities } from "./ArtifactProviderMetadata.js"; import { ARTIFACT_GRAPH_PROVIDER } from "../application/InvestigationProviders.js"; import { createEvidence } from "../domain/evidence.js"; +import { freezeJsonSnapshot } from "../domain/immutableJson.js"; import { createArtifactInspection } from "../domain/artifactInspection.js"; import { ProviderCleanupError } from "../domain/providerCleanupError.js"; @@ -341,7 +342,11 @@ class ArtifactClient implements AnalysisClient { total: 1, message: "inventory substep started", }); - const inventory = await this.inventory(inventoryParameters, options); + // The freshly parsed inventory is exclusively owned here. Sealing it lets + // the nested Evidence share it instead of cloning and re-digesting it. + const inventory = freezeJsonSnapshot( + await this.inventory(inventoryParameters, options), + ); const subject = subjectFor( this.target.sourcePath ?? this.target.path, inventory.manifest, @@ -355,7 +360,10 @@ class ArtifactClient implements AnalysisClient { // The nested inventory already names every occurrence's logical path. locations: [], }); - const result = createArtifactInspection(inventoryEvidence); + // The returned execution and its Evidence then reuse this owned report. + const result = freezeJsonSnapshot( + createArtifactInspection(inventoryEvidence), + ); await options?.progress?.report({ phase: "inspect_artifact.inventory", completed: 1, diff --git a/src/artifacts/ArtifactResourceScope.ts b/src/artifacts/ArtifactResourceScope.ts index 7c0bcf2c2..872899631 100644 --- a/src/artifacts/ArtifactResourceScope.ts +++ b/src/artifacts/ArtifactResourceScope.ts @@ -57,6 +57,10 @@ export class ArtifactResourceScope { #draining: Promise | undefined; #closed = false; + get hasPending(): boolean { + return this.#pending.size > 0; + } + /** Admit a resource operation while the scope remains open. */ async run(operation: () => Promise): Promise { while (true) { diff --git a/src/artifacts/MachOSliceArtifactReader.ts b/src/artifacts/MachOSliceArtifactReader.ts index 9420bef26..b57b0247e 100644 --- a/src/artifacts/MachOSliceArtifactReader.ts +++ b/src/artifacts/MachOSliceArtifactReader.ts @@ -8,11 +8,11 @@ import { type NativeCommandRunner, } from "../native/CommandRunner.js"; import { - lipoSubtypeBits, - parseLipoArchitectures, + parseLipoOutput, type LipoArchitecture, } from "../native/parsers/lipo.js"; import type { MachoSlice } from "../domain/apple/dylibResolution.js"; +import { machoCpuSubtypeBase } from "../domain/apple/machoCpuIdentity.js"; import { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js"; import { readMachoImage, type ReadAt } from "./apple/MachoLoadCommandReader.js"; import { @@ -81,7 +81,9 @@ export class MachOSliceArtifactReader implements ArtifactReader { if (!captured.ok) throw new ArtifactReaderFailure( captured.error.reason === "cancelled" ? "cancelled" : "unavailable", - "lipo could not enumerate universal Mach-O slices", + captured.error.detail === null + ? "lipo could not enumerate universal Mach-O slices" + : `lipo could not enumerate universal Mach-O slices: ${captured.error.detail}`, { cause: captured.error, ...(captured.error.cleanupFailure === null && @@ -112,7 +114,7 @@ export class MachOSliceArtifactReader implements ArtifactReader { }; await this.#verifySource(source); const fileSize = source.initial.size; - const architectures = parseLipoArchitectures(captured.value.stdout); + const { architectures } = parseLipoOutput(captured.value.stdout, this.path); const structural = await readStructuralSlices(source.handle, fileSize); await this.#verifySource(source); if (structural.status === "malformed") @@ -469,8 +471,10 @@ const lipoSubtypeMatches = ( ): boolean => { if (reported.cpu_subtype_code !== null) return reported.cpu_subtype_code === observedCpuSubtype; - const bits = lipoSubtypeBits(reported.cpu_subtype); - return bits === null || bits === (observedCpuSubtype & 0x00ffffff); + return ( + reported.cpu_subtype_base === null || + reported.cpu_subtype_base === machoCpuSubtypeBase(observedCpuSubtype) + ); }; /** diff --git a/src/artifacts/NativeDmgArtifactReader.ts b/src/artifacts/NativeDmgArtifactReader.ts index a3f3fb120..801ee4daf 100644 --- a/src/artifacts/NativeDmgArtifactReader.ts +++ b/src/artifacts/NativeDmgArtifactReader.ts @@ -22,6 +22,7 @@ import { type NativeDmgInfo, } from "./NativeDmgOwnership.js"; const HDIUTIL_COMMAND_TIMEOUT_MS = 120_000; +const VERIFY_SETTLE_DELAYS_MS = [250, 500] as const; // APFS ejects and busy reporting can lag a detach attempt. const DETACH_SETTLE_DELAYS_MS = [250, 500, 1000] as const; const delay = (milliseconds: number): Promise => @@ -44,22 +45,30 @@ export interface NativeDmgHost { class NativeDmgCommandFailure extends ArtifactReaderFailure { readonly processNotStarted: boolean; + readonly verifyDiagnostic: string | undefined; + readonly exitCode: number | null; constructor( - reason: ArtifactReaderFailure["reason"], + failure: { + readonly reason: ArtifactReaderFailure["reason"]; + readonly verifyDiagnostic?: string; + }, message: string, readonly stdout: string, options?: { readonly cause?: unknown; readonly processNotStarted?: boolean; + readonly exitCode?: number; }, ) { super( - reason, + failure.reason, message, options?.cause === undefined ? undefined : { cause: options.cause }, ); this.processNotStarted = options?.processNotStarted ?? false; + this.verifyDiagnostic = failure.verifyDiagnostic; + this.exitCode = options?.exitCode ?? null; } } @@ -84,7 +93,7 @@ const createSystemHost = ( const output = execFileOutputFailure(cause); if (cause instanceof Error && cause.name === "AbortError") throw new NativeDmgCommandFailure( - "cancelled", + { reason: "cancelled" }, "DMG operation cancelled", output?.stdout ?? "", { cause }, @@ -99,7 +108,7 @@ const createSystemHost = ( cause, }; throw new NativeDmgCommandFailure( - commandFailureReason(cause, arguments_[0]), + classifyCommandFailure(cause, arguments_[0]), `hdiutil ${arguments_[0] ?? "operation"} failed: ${describeCommandFailure(cause, arguments_)}`, output?.stdout ?? "", { cause, processNotStarted: processNotStarted(cause) }, @@ -348,8 +357,7 @@ export class NativeDmgArtifactReader implements ArtifactReader { } async #attach(signal?: AbortSignal): Promise { - await runChecked(this.host, ["verify", this.path], signal); - this.#provenance.push(command(["verify", this.path], ["read"])); + await this.#verify(signal); const baseline = await this.#readInfo(); this.#baselineDevices = new Set( baseline.images.flatMap((image) => @@ -431,6 +439,40 @@ export class NativeDmgArtifactReader implements ArtifactReader { } } + async #verify(signal?: AbortSignal): Promise { + const arguments_ = ["verify", this.path]; + const deadline = performance.now() + HDIUTIL_COMMAND_TIMEOUT_MS; + for (let attempt = 0; ; attempt += 1) { + if (signal?.aborted === true) + throw new NativeDmgCommandFailure( + { reason: "cancelled" }, + "DMG operation cancelled", + "", + { cause: signal.reason }, + ); + try { + await runChecked(this.host, arguments_, signal, { + timeoutMs: Math.max(1, Math.ceil(deadline - performance.now())), + }); + this.#provenance.push(command(arguments_, ["read"])); + return; + } catch (cause: unknown) { + if (!(cause instanceof NativeDmgCommandFailure)) throw cause; + this.#provenance.push(command(arguments_, ["read"], cause.exitCode)); + const delayMs = VERIFY_SETTLE_DELAYS_MS[attempt]; + // Retry only the producer's explicit transient diagnostic, before any mount. + if ( + cause.verifyDiagnostic !== "Resource temporarily unavailable" || + delayMs === undefined || + deadline - performance.now() <= delayMs + ) + throw cause; + await (this.host.delay ?? delay)(delayMs); + if (performance.now() >= deadline) throw cause; + } + } + } + async #readInfo(): Promise { const info = await runChecked(this.host, ["info", "-plist"], undefined, { timeoutMs: HDIUTIL_COMMAND_TIMEOUT_MS, @@ -501,14 +543,14 @@ const runChecked = async ( if (cause instanceof ArtifactReaderFailure) throw cause; if (cause instanceof Error && cause.name === "AbortError") throw new NativeDmgCommandFailure( - "cancelled", + { reason: "cancelled" }, "DMG operation cancelled", execFileOutputFailure(cause)?.stdout ?? "", { cause }, ); const output = execFileOutputFailure(cause); throw new NativeDmgCommandFailure( - commandFailureReason(cause, arguments_[0]), + classifyCommandFailure(cause, arguments_[0]), `hdiutil ${arguments_[0] ?? "operation"} failed: ${describeCommandFailure( cause, arguments_, @@ -527,17 +569,17 @@ const runChecked = async ( stdout: result.stdout, ...(result.stderr === undefined ? {} : { stderr: result.stderr }), }; - const reason = commandFailureReason(failure, arguments_[0]); + const classification = classifyCommandFailure(failure, arguments_[0]); const details = describeCommandFailure(failure, arguments_); const unknownVerifyFailure = - arguments_[0] === "verify" && reason === "unavailable"; + arguments_[0] === "verify" && classification.reason === "unavailable"; throw new NativeDmgCommandFailure( - reason, + classification, unknownVerifyFailure ? `hdiutil verify failed; captured diagnostics do not establish the failure cause: ${details}` : `hdiutil ${arguments_[0] ?? "operation"} failed: ${details}`, result.stdout, - { cause: result.cause ?? failure }, + { cause: result.cause ?? failure, exitCode: result.exitCode }, ); } return { stdout: result.stdout, exitCode: 0 }; @@ -587,20 +629,36 @@ const processOutput = (cause: unknown, field: "stdout" | "stderr"): string => { return typeof value === "string" ? value : ""; }; -const commandFailureReason = ( +const classifyCommandFailure = ( cause: unknown, operation: string | undefined, -): ArtifactReaderFailure["reason"] => { - if (typeof cause !== "object" || cause === null) return "unavailable"; +): { + readonly reason: ArtifactReaderFailure["reason"]; + readonly verifyDiagnostic?: string; +} => { + if (typeof cause !== "object" || cause === null) + return { reason: "unavailable" }; const code = Reflect.get(cause, "code"); const exitCode = Reflect.get(cause, "exitCode"); - if (typeof code === "number" || typeof exitCode === "number") - return operation === "verify" ? verifyFailureReason(cause) : "unavailable"; + if (typeof code === "number" || typeof exitCode === "number") { + if (operation !== "verify") return { reason: "unavailable" }; + const diagnostic = verifyFailureDiagnostic( + processOutput(cause, "stdout"), + processOutput(cause, "stderr"), + ); + return { + reason: verifyFailureReason(diagnostic), + ...(diagnostic === undefined ? {} : { verifyDiagnostic: diagnostic }), + }; + } if (code === "ENOENT") { const syscall = Reflect.get(cause, "syscall"); - return typeof syscall === "string" && syscall.startsWith("spawn") - ? "unavailable" - : "io"; + return { + reason: + typeof syscall === "string" && syscall.startsWith("spawn") + ? "unavailable" + : "io", + }; } if ( code === "EACCES" || @@ -613,17 +671,13 @@ const commandFailureReason = ( code === "EMFILE" || code === "ENFILE" ) - return "io"; - return "unavailable"; + return { reason: "io" }; + return { reason: "unavailable" }; }; const verifyFailureReason = ( - cause: unknown, + diagnostic: string | undefined, ): ArtifactReaderFailure["reason"] => { - const diagnostic = verifyFailureDiagnostic( - processOutput(cause, "stdout"), - processOutput(cause, "stderr"), - ); switch (diagnostic) { // hdiutil follows AppleLanguages, so en-GB and en-AU spell it "recognised". case "image not recognized": @@ -656,11 +710,12 @@ const verifyFailureDiagnostic = ( const command = ( arguments_: readonly string[], effects: ArtifactCommand["effects"], + exitCode: number | null = 0, ): ArtifactCommand => ({ tool: "/usr/bin/hdiutil", arguments: [...arguments_], tool_version: null, executable_sha256: null, - exit_code: 0, + exit_code: exitCode, effects, }); diff --git a/src/artifacts/SafeOutputTree.ts b/src/artifacts/SafeOutputTree.ts index 4bb4cfe1a..73885ad05 100644 --- a/src/artifacts/SafeOutputTree.ts +++ b/src/artifacts/SafeOutputTree.ts @@ -280,9 +280,11 @@ export class SafeOutputTree { } /** Sync the owned output tree and prevent further writes through this instance. */ - async commit(): Promise { + async commit(signal?: AbortSignal): Promise { this.#assertWritable(); + abortIfNeeded(signal); await this.#assertOwnedDirectories(); + abortIfNeeded(signal); // Windows has no directory fsync; file contents are already synced in write(). if (this.platform === "win32") { this.#published = true; @@ -315,6 +317,7 @@ export class SafeOutputTree { ); await output.handle.sync(); await parent.handle.sync(); + abortIfNeeded(signal); this.#published = true; } catch (cause: unknown) { sync = { kind: "failed", cause }; @@ -336,23 +339,26 @@ export class SafeOutputTree { closeFailure, this.#outputRoot, ); - if (sync.kind === "failed") - throw ArtifactReaderFailure.withCleanup( - new ArtifactReaderFailure( - "path", - "Could not durably sync extraction output", - { cause: sync.cause }, - ), - cleanup, - ); + if (sync.kind === "failed") { + const primary = isCancellationFailure(sync.cause) + ? sync.cause + : new ArtifactReaderFailure( + "path", + "Could not durably sync extraction output", + { cause: sync.cause }, + ); + throw ArtifactReaderFailure.withCleanup(primary, cleanup); + } throw ArtifactReaderFailure.withCleanup(closeFailure, cleanup); } - if (sync.kind === "failed") + if (sync.kind === "failed") { + if (isCancellationFailure(sync.cause)) throw sync.cause; throw new ArtifactReaderFailure( "path", "Could not durably sync extraction output", { cause: sync.cause }, ); + } } /** Remove only this operation's unsealed tree and verify absence. */ @@ -579,6 +585,11 @@ const normalizeDestinationPath = ( "path", `Windows destination cannot materialize artifact path as a regular file: ${path}; ':' denotes alternate data stream syntax.`, ); + if (platform === "win32" && /[\u0001-\u001f]/u.test(path)) + throw new ArtifactReaderFailure( + "path", + `Windows destination cannot materialize artifact path: ${JSON.stringify(path)}; Windows names cannot contain U+0001-U+001F.`, + ); return path; }; @@ -590,6 +601,11 @@ const abortIfNeeded = (signal?: AbortSignal): void => { ); }; +const isCancellationFailure = ( + cause: unknown, +): cause is ArtifactReaderFailure => + cause instanceof ArtifactReaderFailure && cause.reason === "cancelled"; + const isAlreadyExists = (cause: unknown): boolean => cause instanceof Error && "code" in cause && cause.code === "EEXIST"; diff --git a/src/artifacts/ZipArtifactReader.ts b/src/artifacts/ZipArtifactReader.ts index 11c68557b..066b005b8 100644 --- a/src/artifacts/ZipArtifactReader.ts +++ b/src/artifacts/ZipArtifactReader.ts @@ -2,7 +2,29 @@ import type { FileHandle } from "node:fs/promises"; import { PassThrough, type Readable } from "node:stream"; import { once } from "node:events"; -import { Reader, ZipReader, type Entry, type FileEntry } from "@zip.js/zip.js"; +import { + ERR_AMBIGUOUS_ARCHIVE, + ERR_BAD_FORMAT, + ERR_CENTRAL_DIRECTORY_NOT_FOUND, + ERR_ENCRYPTED_CENTRAL_DIRECTORY, + ERR_ENTRY_DATA_OUT_OF_BOUNDS, + ERR_EOCDR_LOCATOR_ZIP64_NOT_FOUND, + ERR_EOCDR_NOT_FOUND, + ERR_EXTRAFIELD_ZIP64_NOT_FOUND, + ERR_INVALID_COMPRESSED_DATA, + ERR_INVALID_CRC32, + ERR_INVALID_UNCOMPRESSED_SIZE, + ERR_LOCAL_FILE_HEADER_NOT_FOUND, + ERR_OVERLAPPING_ENTRY, + ERR_SPLIT_ZIP_FILE, + ERR_UNSUPPORTED_COMPRESSION, + ERR_UNSUPPORTED_ENCRYPTION, + ERR_UNSUPPORTED_UINT64, + Reader, + ZipReader, + type Entry, + type FileEntry, +} from "@zip.js/zip.js"; import { ArtifactReaderFailure, @@ -159,9 +181,9 @@ export class ZipArtifactReader implements ArtifactReader { let index = 0; // A traversal owns its accounting, including overlapping or repeated passes. let declaredFileBytes = 0; - for await (const entry of this.#reader.getEntriesGenerator({ - decodeText: decodeZipEntryText, - })) { + for await (const entry of typedZipFailures( + this.#reader.getEntriesGenerator({ decodeText: decodeZipEntryText }), + )) { abortIfNeeded(signal); // ZIP member identity is its directory occurrence, not its filename. // Logical path uniqueness belongs to the artifact inventory boundary. @@ -255,10 +277,29 @@ const decodeZipEntryText = ( encoding: string, type: "filename" | "comment", ): string | undefined => { - if (type !== "filename" || encoding !== "utf-8") return undefined; - return decodeStrictZipUtf8(bytes, "ZIP filename is not valid UTF-8"); + if (type !== "filename") return undefined; + if (encoding === "utf-8") + return decodeStrictZipUtf8(bytes, "ZIP filename is not valid UTF-8"); + return encoding === "cp437" ? decodeCp437(bytes) : undefined; }; +/** CP437 characters for bytes 0x80-0xFF. */ +const CP437_HIGH_HALF = + "ÇüéâäàåçêëèïîìÄÅÉæÆôöòûùÿÖÜ¢£¥₧ƒáíóúñѪº¿⌐¬½¼¡«»░▒▓│┤╡╢╖╕╣║╗╝╜╛┐└┴┬├─┼╞╟╚╔╩╦╠═╬╧╨╤╥╙╘╒╓╫╪┘┌█▄▌▐▀αßΓπΣσµτΦΘΩδ∞φε∩≡±≥≤⌠⌡÷≈°∙·√ⁿ²■\u00A0"; + +/** + * Decode a legacy ZIP name with the Unicode CP437 mapping, which Python's + * zipfile also uses: bytes below 0x80 are ASCII. zip.js shows 0x01-0x1F and + * 0x7F as IBM PC display glyphs, which turns the macOS folder icon file + * `Icon\r` into `Icon♪`. + */ +const decodeCp437 = (bytes: Uint8Array): string => + Array.from(bytes, (byte) => + byte < 0x80 + ? String.fromCharCode(byte) + : CP437_HIGH_HALF.charAt(byte - 0x80), + ).join(""); + const zipEntryPath = (entry: Entry): string => { const field = entry.extraFieldUnicodePath; if (field?.valid !== true) return entry.filename; @@ -308,7 +349,7 @@ const extractStream = (entry: FileEntry, signal?: AbortSignal): Readable => { output.end(); }, abort: (reason) => { - output.destroy(toError(reason)); + output.destroy(toError(typedZipFailure(reason))); }, }); void entry @@ -318,10 +359,61 @@ const extractStream = (entry: FileEntry, signal?: AbortSignal): Readable => { checkOverlappingEntry: true, onprogress: () => abortIfNeeded(signal), }) - .catch((cause: unknown) => output.destroy(toError(cause))); + .catch((cause: unknown) => output.destroy(toError(typedZipFailure(cause)))); return output; }; +/** zip.js reports archive faults as plain errors identified by message. */ +const ZIP_FAILURE_REASONS: ReadonlyMap = + new Map([ + ...[ + ERR_AMBIGUOUS_ARCHIVE, + ERR_BAD_FORMAT, + ERR_CENTRAL_DIRECTORY_NOT_FOUND, + ERR_ENTRY_DATA_OUT_OF_BOUNDS, + ERR_EOCDR_LOCATOR_ZIP64_NOT_FOUND, + ERR_EOCDR_NOT_FOUND, + ERR_EXTRAFIELD_ZIP64_NOT_FOUND, + ERR_INVALID_COMPRESSED_DATA, + ERR_INVALID_CRC32, + ERR_INVALID_UNCOMPRESSED_SIZE, + ERR_LOCAL_FILE_HEADER_NOT_FOUND, + ERR_OVERLAPPING_ENTRY, + ERR_UNSUPPORTED_UINT64, + ].map((message) => [message, "format"] as const), + ...[ + ERR_ENCRYPTED_CENTRAL_DIRECTORY, + ERR_SPLIT_ZIP_FILE, + ERR_UNSUPPORTED_COMPRESSION, + ERR_UNSUPPORTED_ENCRYPTION, + ].map((message) => [message, "unavailable"] as const), + ]); + +/** Type a zip.js archive fault so consumers do not report it as file I/O. */ +const typedZipFailure = (cause: unknown): unknown => { + if (!(cause instanceof Error) || cause instanceof ArtifactReaderFailure) + return cause; + const reason = ZIP_FAILURE_REASONS.get(cause.message); + if (reason === undefined) return cause; + return new ArtifactReaderFailure( + reason, + reason === "format" + ? `ZIP archive is malformed: ${cause.message}` + : `ZIP archive uses an unsupported feature: ${cause.message}`, + { cause }, + ); +}; + +async function* typedZipFailures( + source: AsyncIterable, +): AsyncIterable { + try { + yield* source; + } catch (cause: unknown) { + throw typedZipFailure(cause); + } +} + const isSymlink = (entry: Entry): boolean => entry.unixMode !== undefined && (entry.unixMode & 0o170000) === 0o120000; diff --git a/src/artifacts/apple/InterfaceBuilderAnalysis.ts b/src/artifacts/apple/InterfaceBuilderAnalysis.ts index fb90594b3..02ecd9821 100644 --- a/src/artifacts/apple/InterfaceBuilderAnalysis.ts +++ b/src/artifacts/apple/InterfaceBuilderAnalysis.ts @@ -56,6 +56,7 @@ export const analyzeInterfaceBuilderBundle = async (input: { const omittedArchives: Array<{ path: string; reason: string }> = []; let unreportedOmittedArchives = 0; const incompleteHierarchies = new Map(); + const unresolvedConnections = new Map(); const prototypeKeyOmissions = new Map(); let omitted = 0; let attempted = 0; @@ -136,6 +137,8 @@ export const analyzeInterfaceBuilderBundle = async (input: { : { value: nib.raw, omittedPrototypeKeys: 0 }; if (nib !== null && nib.omitted > 0) incompleteHierarchies.set(entry.path, nib.omitted); + if (nib !== null && nib.unresolvedConnections > 0) + unresolvedConnections.set(entry.path, nib.unresolvedConnections); if (omittedPrototypeKeys !== 0) prototypeKeyOmissions.set(entry.path, omittedPrototypeKeys); const documentHash = createHash("sha256").update(bytes).digest("hex"); @@ -170,6 +173,7 @@ export const analyzeInterfaceBuilderBundle = async (input: { }); return finalizeHierarchyCoverage(result, { incompleteHierarchies, + unresolvedConnections, prototypeKeyOmissions, invalid, omittedArchives, @@ -237,6 +241,7 @@ const finalizeHierarchyCoverage = ( result: ReturnType, counts: { incompleteHierarchies: ReadonlyMap; + unresolvedConnections: ReadonlyMap; prototypeKeyOmissions: ReadonlyMap; invalid: readonly string[]; omittedArchives: readonly { path: string; reason: string }[]; @@ -247,6 +252,7 @@ const finalizeHierarchyCoverage = ( ) => { const { incompleteHierarchies, + unresolvedConnections, prototypeKeyOmissions, invalid, omittedArchives, @@ -260,6 +266,13 @@ const finalizeHierarchyCoverage = ( ...result, documents: result.documents.map((document) => ({ ...document, + // The dictionary parser only sees connectors that were projected, so the + // omission recorded where an ownerless connector was dropped has to be + // added here for this producer to report the same document count as the + // keyed-archive producer for the same recognized connector. + connection_count: + document.connection_count + + (unresolvedConnections.get(document.relative_path) ?? 0), hierarchy_complete: document.hierarchy_complete && !incompleteHierarchies.has(document.relative_path), @@ -268,15 +281,32 @@ const finalizeHierarchyCoverage = ( ...result.graph, coverage: [ ...result.graph.coverage.map((facet) => { - const omitted = incompleteHierarchies.get( + const omittedHierarchy = incompleteHierarchies.get( facet.facet.replace(/^hierarchy:/u, ""), ); - return facet.facet.startsWith("hierarchy:") && omitted !== undefined + if ( + facet.facet.startsWith("hierarchy:") && + omittedHierarchy !== undefined + ) + return { + ...facet, + status: "partial" as const, + reason: "serialized_view_hierarchy_incomplete", + omitted: facet.omitted + omittedHierarchy, + }; + const omittedConnection = unresolvedConnections.get( + facet.facet.replace(/^connections:/u, ""), + ); + return facet.facet.startsWith("connections:") && + omittedConnection !== undefined ? { ...facet, status: "partial" as const, - reason: "serialized_view_hierarchy_incomplete", - omitted: facet.omitted + omitted, + // A connection dropped by the limit is recoverable by raising + // it, so that cause keeps the reason the graph already named. + // Only a facet with no other cause reports the omission. + reason: facet.reason ?? "connection_unresolved", + omitted: facet.omitted + omittedConnection, } : facet; }), @@ -306,7 +336,8 @@ const finalizeHierarchyCoverage = ( result.graph.truncated || counts.omitted > 0 || archiveDecodePartial || - incompleteHierarchies.size > 0, + incompleteHierarchies.size > 0 || + unresolvedConnections.size > 0, }, limitations: [ ...result.limitations, @@ -316,6 +347,11 @@ const finalizeHierarchyCoverage = ( : [ "Some serialized hierarchy links could not be projected within the bounded hierarchy; their parentage remains unknown.", ]), + ...(unresolvedConnections.size === 0 + ? [] + : [ + "Some recognized NIBArchive connectors declare no resolvable owner; they carry no connection evidence and are reported as omissions.", + ]), ...(invalid.length === 0 ? [] : [ @@ -496,6 +532,10 @@ const projectNibArchive = ( ); const connections: Record = {}; const hierarchy: NibHierarchyNode[] = []; + // A recognized connector whose owner reference cannot be resolved produces no + // connection entry, so the omission has to be counted here: the projected + // dictionary carries no trace of it for a downstream consumer to find. + let unresolvedConnections = 0; const reference = (value: JsonValue | undefined): number | null => { if (typeof value !== "object" || value === null || Array.isArray(value)) return null; @@ -512,7 +552,10 @@ const projectNibArchive = ( const values = object.values; const connectionFields = interfaceBuilderConnectionFieldNames(className); const source = reference(values[connectionFields.source]); - if (source === null) continue; + if (source === null) { + unresolvedConnections += 1; + continue; + } const destination = reference(values[connectionFields.destination]); const label = referencedString(values[connectionFields.label]); const type = className.includes("Outlet") @@ -608,6 +651,7 @@ const projectNibArchive = ( ); return { omitted: viewHierarchy.omitted + hierarchyOmitted + merged.omitted, + unresolvedConnections, raw: jsonValueSchema.parse({ "com.apple.ibtool.document.objects": objects, "com.apple.ibtool.document.connections": connections, diff --git a/src/artifacts/apple/NibArchive.ts b/src/artifacts/apple/NibArchive.ts index bd8ef12ca..c673184f4 100644 --- a/src/artifacts/apple/NibArchive.ts +++ b/src/artifacts/apple/NibArchive.ts @@ -134,7 +134,14 @@ export const decodeNibArchive = ( const value = values.get(index); if (value === undefined) continue; const key = keys[value.key_index]; - if (key !== undefined) fields[key] = value.value; + if (key !== undefined) + // Archive keys are data, including the legacy __proto__ setter name. + Object.defineProperty(fields, key, { + value: value.value, + enumerable: true, + configurable: true, + writable: true, + }); } return [{ id, class_name: className, values: fields }]; }); @@ -519,7 +526,8 @@ const decodeUtf8 = ( label: string, ): string => { try { - return new TextDecoder("utf-8", { fatal: true }).decode( + // These are length-prefixed names, not standalone text documents. + return new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }).decode( bytes.subarray(start, end), ); } catch (cause: unknown) { diff --git a/src/artifacts/extraction/ArtifactExtraction.ts b/src/artifacts/extraction/ArtifactExtraction.ts index 6306a08bf..8cad3543e 100644 --- a/src/artifacts/extraction/ArtifactExtraction.ts +++ b/src/artifacts/extraction/ArtifactExtraction.ts @@ -17,6 +17,7 @@ import { DirectoryArtifactReader } from "../DirectoryArtifactReader.js"; import { SafeOutputTree } from "../SafeOutputTree.js"; import { SafeOutputTreeCreationFailure } from "../SafeOutputTreeCreationFailure.js"; import { ZipArtifactReader } from "../ZipArtifactReader.js"; +import { isZipPackageFormat } from "../../domain/zipPackageFormat.js"; import { MachOSliceArtifactReader } from "../MachOSliceArtifactReader.js"; import { artifactExtractionResultSchema, @@ -248,7 +249,7 @@ const materializeSelection = async ({ compareUnicodeCodePoints(left.relative_path, right.relative_path), ); completed = createExtractionResult(input, inventory, selected, extracted); - await output.commit(); + await output.commit(signal); return completed; } catch (cause: unknown) { const cleanupFailures = await releaseExtractionOwners( @@ -400,13 +401,6 @@ interface ExtractionInventory { readonly integrityContradictions: readonly IntegrityContradiction[]; } -const ZIP_FORMATS = ["ipa", "apk", "msix", "appx", "zip"] as const; - -const isZipFormat = ( - format: BinaryTarget["format"], -): format is (typeof ZIP_FORMATS)[number] => - (ZIP_FORMATS as readonly string[]).includes(format); - /** * Refuse a format without an extraction reader before inventory work starts. * The target kind, not the host, is unsupported by extraction. The error keeps @@ -422,7 +416,7 @@ const requireExtractableFormat = async ( !directory && format !== "asar" && !isMachOFormat(format) && - !isZipFormat(format) + !isZipPackageFormat(format) ) throw new AnalysisUnsupportedTargetError( "extract_artifact", @@ -443,7 +437,7 @@ const createReader = async ( if (await requireExtractableFormat(path, input)) return new DirectoryArtifactReader(path); if (format === "asar") return new AsarArtifactReader(path); - if (isZipFormat(format)) return new ZipArtifactReader(path, format); + if (isZipPackageFormat(format)) return new ZipArtifactReader(path, format); return new MachOSliceArtifactReader(path, input.environment); }; diff --git a/src/artifacts/inventory/ArtifactGraphConstruction.test.ts b/src/artifacts/inventory/ArtifactGraphConstruction.test.ts new file mode 100644 index 000000000..925d3543a --- /dev/null +++ b/src/artifacts/inventory/ArtifactGraphConstruction.test.ts @@ -0,0 +1,103 @@ +import { describe, expect, it } from "vitest"; + +import { CPU, fatImage, machoImage } from "../apple/MachoImage.fixture.js"; +import { classifyArtifactContent } from "./ArtifactGraphConstruction.js"; + +const fat = (wide = false): Buffer => + Buffer.from(fatImage([{ cpu: CPU.arm64, bytes: machoImage({}) }], wide)); + +describe("bounded FAT inventory classification", () => { + it("requires the complete table but not slice bytes in the retained prefix", () => { + const bytes = fat(); + for (const length of [4, 7, 8, 27]) { + expect( + classifyArtifactContent( + "native", + bytes.subarray(0, length), + bytes.length, + ), + ).toEqual({ kind: "unknown", format: "unknown" }); + } + expect( + classifyArtifactContent( + "native.class", + bytes.subarray(0, 28), + bytes.length, + ), + ).toEqual({ kind: "executable", format: "mach-o-universal" }); + }); + + it.each([ + ["empty table", (bytes: Buffer) => bytes.writeUInt32BE(0, 4)], + ["slice overlapping table", (bytes: Buffer) => bytes.writeUInt32BE(0, 16)], + ["empty slice", (bytes: Buffer) => bytes.writeUInt32BE(0, 20)], + [ + "slice beyond file", + (bytes: Buffer) => bytes.writeUInt32BE(bytes.length, 20), + ], + ["unaligned slice", (bytes: Buffer) => bytes.writeUInt32BE(4097, 16)], + ["unsupported alignment", (bytes: Buffer) => bytes.writeUInt32BE(53, 24)], + ] as const)("keeps %s unknown", (_label, mutate) => { + const bytes = fat(); + mutate(bytes); + expect(classifyArtifactContent("native", bytes)).toEqual({ + kind: "unknown", + format: "unknown", + }); + }); + + it("does not infer Mach-O from an out-of-bounds architecture count", () => { + const bytes = fat(); + bytes.writeUInt32BE(0xffffffff, 4); + expect(classifyArtifactContent("native", bytes)).toEqual({ + kind: "resource", + format: "file", + }); + }); + + it("does not lose unsafe FAT64 range values to number conversion", () => { + const bytes = fat(true); + bytes.writeBigUInt64BE(BigInt(Number.MAX_SAFE_INTEGER) + 1n, 16); + expect(classifyArtifactContent("native", bytes)).toEqual({ + kind: "unknown", + format: "unknown", + }); + }); + + it("requires valid ranges for every declared slice", () => { + const bytes = Buffer.from( + fatImage([ + { cpu: CPU.arm64, bytes: machoImage({}) }, + { cpu: CPU.x86_64, bytes: machoImage({ cpu: CPU.x86_64 }) }, + ]), + ); + bytes.writeUInt32BE(bytes.length, 40); + expect(classifyArtifactContent("native", bytes)).toEqual({ + kind: "unknown", + format: "unknown", + }); + }); + + it("requires observed slice magic for an unknown CPU type", () => { + const cpu = { type: 0x12345678, subtype: 0 }; + const bytes = Buffer.from(fatImage([{ cpu, bytes: machoImage({ cpu }) }])); + expect( + classifyArtifactContent("native", bytes.subarray(0, 28), bytes.length), + ).toEqual({ kind: "unknown", format: "unknown" }); + expect(classifyArtifactContent("native", bytes)).toEqual({ + kind: "executable", + format: "mach-o-universal", + }); + }); + + it.each([NaN, Infinity, -1, 8.5, Number.MAX_SAFE_INTEGER + 1, 27])( + "keeps invalid or contradictory file size %s unknown", + (size) => { + const bytes = fat(); + expect(classifyArtifactContent("native", bytes, size)).toEqual({ + kind: "unknown", + format: "unknown", + }); + }, + ); +}); diff --git a/src/artifacts/javascript/JavaScriptArtifactFiles.ts b/src/artifacts/javascript/JavaScriptArtifactFiles.ts index e3849f814..78d8cf750 100644 --- a/src/artifacts/javascript/JavaScriptArtifactFiles.ts +++ b/src/artifacts/javascript/JavaScriptArtifactFiles.ts @@ -37,6 +37,7 @@ interface ExpectedFile { interface ExpectedContainer { readonly hash_status: ArtifactOccurrence["hash_status"]; + readonly format: ArtifactOccurrence["artifact_format"]; readonly inventory: JavaScriptArtifactContainer | null; } @@ -219,6 +220,9 @@ const visitReader = async ( "integrity", `Nested ASAR was not integrity-verified: ${path}`, ); + // Inventory expands an ASAR-named file only when its bytes are an + // ASAR. One it recorded as ordinary bytes has no members to read. + if (container.format !== "asar") continue; const inventory = container.inventory; if (inventory === null) throw new ArtifactReaderFailure( @@ -395,6 +399,7 @@ const expectedInventory = ( if (isNestedAsar) containers.set(occurrence.logical_path, { hash_status: occurrence.hash_status, + format: occurrence.artifact_format, inventory: null, }); continue; @@ -412,6 +417,7 @@ const expectedInventory = ( if (isNestedAsar) containers.set(occurrence.logical_path, { hash_status: occurrence.hash_status, + format: occurrence.artifact_format, inventory: { path: occurrence.logical_path, sha256: node.sha256, diff --git a/src/artifacts/readStableArtifact.ts b/src/artifacts/readStableArtifact.ts index e35a20783..808133d89 100644 --- a/src/artifacts/readStableArtifact.ts +++ b/src/artifacts/readStableArtifact.ts @@ -38,6 +38,13 @@ const afterValidation = async ( } }; +const assertReadActive = (signal: AbortSignal | undefined): void => { + if (signal?.aborted) + throw new ArtifactReaderFailure("cancelled", "Artifact read cancelled", { + cause: signal.reason, + }); +}; + /** Read one bounded regular file and reject replacement or in-place changes. */ export const readStableArtifact = async ( path: string, @@ -46,7 +53,7 @@ export const readStableArtifact = async ( fileSystem: StableArtifactFileSystem = FILE_SYSTEM, resourceScope?: ArtifactResourceScope, ): Promise<{ readonly bytes: Buffer; readonly sha256: string }> => { - signal?.throwIfAborted(); + assertReadActive(signal); const before = await fileSystem.lstat(path); if (!before.isFile() || before.isSymbolicLink()) throw new ArtifactReaderFailure( @@ -84,7 +91,7 @@ export const readStableArtifact = async ( autoClose: false, highWaterMark: 65536, })) { - signal?.throwIfAborted(); + assertReadActive(signal); const value: unknown = chunk; if (!Buffer.isBuffer(value)) throw new ArtifactReaderFailure( @@ -120,6 +127,7 @@ export const readStableArtifact = async ( "integrity", `Artifact changed during read: ${path}`, ); + assertReadActive(signal); const bytes = Buffer.concat(chunks, count); return { bytes, sha256: createHash("sha256").update(bytes).digest("hex") }; } catch (cause: unknown) { diff --git a/src/browser/BrowserScenarioPartialObservation.ts b/src/browser/BrowserScenarioPartialObservation.ts index ca0df4273..a573f600f 100644 --- a/src/browser/BrowserScenarioPartialObservation.ts +++ b/src/browser/BrowserScenarioPartialObservation.ts @@ -91,7 +91,7 @@ export const browserScenarioCaptureData = (input: { network_content: input.scenario.capture.network, event_families: [...new Set(input.scenario.capture.events)].sort(), }, - duration_ms: Date.now() - input.startedAt, + duration_ms: Math.max(0, Math.round(performance.now() - input.startedAt)), steps: [...input.steps], events, limitations: [ diff --git a/src/browser/CdpCaptureDocuments.baseUrl.test.ts b/src/browser/CdpCaptureDocuments.baseUrl.test.ts index a5481c161..e3201eade 100644 --- a/src/browser/CdpCaptureDocuments.baseUrl.test.ts +++ b/src/browser/CdpCaptureDocuments.baseUrl.test.ts @@ -2,6 +2,8 @@ import { describe, expect, it } from "vitest"; import { inspectWebPageInputSchema } from "../domain/browserObservation.js"; import { captureDom } from "./CdpCaptureDocuments.js"; +import { capturedSvgNodeKind } from "./CdpCaptureDomSvg.js"; +import { CdpCaptureCompleteness } from "./CdpCaptureCompleteness.js"; const origin = "https://app.example.test"; const documentUrl = `${origin}/screens/app`; @@ -50,8 +52,11 @@ const snapshot = ( ], }); -const capture = (value: ReturnType) => - captureDom(value, new Set([origin]), input); +const capture = (value: ReturnType) => { + const step = captureDom(value, new Set([origin]), input).next(); + if (!step.done) throw new Error("Unexpected SVG identity query"); + return step.value; +}; describe("DOM metadata document base URLs", () => { it("resolves links, actions and agent hints against the captured document base", () => { @@ -282,6 +287,7 @@ interface SvgSnapshotNode { readonly name: string; readonly parent: number; readonly attributes?: readonly (readonly [string, string])[]; + readonly svg?: boolean; } const svgDocument = ( @@ -300,7 +306,13 @@ const svgDocument = ( const nodeValue = [intern("")]; const parentIndex = [-1]; const attributes: number[][] = [[]]; + const backendNodeId = [1]; + const svgNodeKinds = new Map(); for (const element of elements) { + const id = backendNodeId.length + 1; + backendNodeId.push(id); + if (element.svg !== undefined) + svgNodeKinds.set(`${String(id)}:${element.name}`, element.svg); nodeType.push(1); nodeName.push(intern(element.name)); nodeValue.push(intern("")); @@ -316,14 +328,35 @@ const svgDocument = ( { documentURL: 0, baseURL: 1, - nodes: { nodeType, nodeName, nodeValue, parentIndex, attributes }, + nodes: { + nodeType, + nodeName, + nodeValue, + parentIndex, + attributes, + backendNodeId, + }, + svgNodeKinds, }, ], }; }; -const captureSvg = (value: ReturnType) => - captureDom(value, new Set([origin]), input); +const captureSvg = (value: ReturnType) => { + const kinds = new Map( + value.documents.flatMap((document) => [...document.svgNodeKinds]), + ); + const capture = captureDom(value, new Set([origin]), input); + let step = capture.next(); + while (!step.done) { + step = capture.next( + step.value.map((query) => + kinds.get(`${String(query.backendNodeId)}:${query.nodeName}`), + ), + ); + } + return step.value; +}; const urlFor = ( result: ReturnType, @@ -331,6 +364,136 @@ const urlFor = ( ): (typeof result.urls)[number] | undefined => result.urls.find((url) => url.node_index === nodeIndex); +describe("browser SVG identity", () => { + const query = { backendNodeId: 42, nodeName: "a" }; + const node = { ...query, nodeType: 1 }; + + it("uses only a matching described element's SVG flag", () => { + expect(capturedSvgNodeKind(query, { node: { ...node, isSVG: true } })).toBe( + true, + ); + expect( + capturedSvgNodeKind(query, { node: { ...node, isSVG: false } }), + ).toBe(false); + expect(capturedSvgNodeKind(query, { node })).toBe(false); + for (const invalid of [ + {}, + { node: { ...node, backendNodeId: 43, isSVG: true } }, + { node: { ...node, nodeName: "A", isSVG: true } }, + { node: { ...node, nodeType: 3, isSVG: true } }, + { node: { ...node, isSVG: "true" } }, + ]) + expect(capturedSvgNodeKind(query, invalid)).toBeUndefined(); + }); + + it("distinguishes identically named SVG and MathML elements regardless of parent", () => { + const result = captureSvg( + svgDocument([ + { name: "svg", parent: 0 }, + { name: "desc", parent: 1 }, + { + name: "a", + svg: true, + parent: 2, + attributes: [["xlink:href", "/svg"]], + }, + { + name: "a", + svg: false, + parent: 2, + attributes: [["xlink:href", "/math"]], + }, + { + name: "A", + svg: false, + parent: 1, + attributes: [["xlink:href", "/html"]], + }, + { + name: "a", + svg: true, + parent: 0, + attributes: [["xlink:href", "/orphan"]], + }, + ]), + ); + expect(result.urls.map(({ url }) => url)).toEqual([ + `${origin}/svg`, + `${origin}/orphan`, + ]); + }); + + it("records unresolved identity without inventing a destination or hiding attributes", () => { + const value = svgDocument([ + { name: "a", parent: 0, attributes: [["xlink:href", "/unknown"]] }, + ]); + const completeness = new CdpCaptureCompleteness(); + const capture = captureDom(value, new Set([origin]), input, completeness); + expect(capture.next()).toEqual({ + done: false, + value: [{ backendNodeId: 2, nodeName: "a" }], + }); + const step = capture.next([undefined]); + expect(step.done).toBe(true); + if (!step.done) throw new Error("Unexpected second query"); + expect(step.value.unresolvedSvgNodes).toBe(1); + expect(step.value.urls).toEqual([]); + expect(step.value.nodes[1]?.attribute_names).toEqual(["xlink:href"]); + expect(completeness.snapshot().excluded).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + section: "metadata", + reason: "provider_unavailable", + count: 1, + }), + ]), + ); + }); +}); + +it("keeps batched SVG identities aligned around missing IDs and ordinary links", () => { + const value = svgDocument([ + { name: "a", parent: 0, attributes: [["xlink:href", "/svg"]] }, + { name: "A", parent: 0, attributes: [["href", "/ordinary"]] }, + { name: "a", parent: 0, attributes: [["xlink:href", "/missing-id"]] }, + { name: "a", parent: 0, attributes: [["xlink:href", "/non-svg"]] }, + ]); + const completeness = new CdpCaptureCompleteness(); + const capture = captureDom( + { + ...value, + documents: value.documents.map((document) => ({ + ...document, + nodes: { ...document.nodes, backendNodeId: [1, 2, 3, null, 5] }, + })), + }, + new Set([origin]), + input, + completeness, + ); + expect(capture.next()).toEqual({ + done: false, + value: [ + { backendNodeId: 2, nodeName: "a" }, + { backendNodeId: 5, nodeName: "a" }, + ], + }); + const step = capture.next([true, false]); + if (!step.done) throw new Error("Unexpected second batch"); + expect( + step.value.urls.map(({ node_index, url }) => [node_index, url]), + ).toEqual([ + [1, `${origin}/svg`], + [2, `${origin}/ordinary`], + ]); + expect(step.value.unresolvedSvgNodes).toBe(1); + expect(completeness.snapshot().excluded).toContainEqual({ + section: "metadata", + reason: "provider_unavailable", + count: 1, + }); +}); + describe("SVG xlink:href destinations", () => { it("reports an SVG anchor that only has xlink:href, keeping the first value", () => { const result = captureSvg( @@ -339,6 +502,7 @@ describe("SVG xlink:href destinations", () => { { name: "g", parent: 1 }, { name: "a", + svg: true, parent: 2, attributes: [ ["xlink:href", "/xl-s2h"], @@ -371,6 +535,7 @@ describe("SVG xlink:href destinations", () => { { name: "svg", parent: 0 }, { name: "a", + svg: true, parent: 1, attributes: [ [first, valueFor(first)], @@ -395,18 +560,21 @@ describe("SVG xlink:href destinations", () => { svgDocument([ { name: "A", + svg: false, parent: 0, attributes: [["xlink:href", "/xl-html"]], }, { name: "math", parent: 0 }, { name: "mrow", + svg: false, parent: 2, attributes: [["xlink:href", "/xl-math"]], }, { name: "svg", parent: 0 }, { name: "a", + svg: true, parent: 4, attributes: [["xlink:href", "/xl-s2h"]], }, @@ -425,6 +593,7 @@ describe("SVG xlink:href destinations", () => { { name: "svg", parent: 0 }, { name: "a", + svg: true, parent: 1, attributes: [["xlink:href", "https://cdn.example.test/out"]], }, @@ -444,8 +613,18 @@ describe("SVG xlink:href destinations", () => { const result = captureSvg( svgDocument([ { name: "svg", parent: 0 }, - { name: "image", parent: 1, attributes: [["xlink:href", "guide"]] }, - { name: "use", parent: 1, attributes: [["xlink:href", "icon.svg"]] }, + { + name: "image", + svg: true, + parent: 1, + attributes: [["xlink:href", "guide"]], + }, + { + name: "use", + svg: true, + parent: 1, + attributes: [["xlink:href", "icon.svg"]], + }, ]), ); expect(result.urls).toEqual([ @@ -466,14 +645,24 @@ describe("SVG xlink:href destinations", () => { }); describe("SVG xlink:href context", () => { - it("keeps forward ancestor context local to each document", () => { + it("binds SVG identity to the selected element across documents", () => { const svg = svgDocument([ - { name: "a", parent: 2, attributes: [["xlink:href", "/svg-link"]] }, + { + name: "a", + svg: true, + parent: 2, + attributes: [["xlink:href", "/svg-link"]], + }, { name: "g", parent: 3 }, { name: "svg", parent: 0 }, ]); const html = svgDocument([ - { name: "A", parent: 2, attributes: [["xlink:href", "/html-link"]] }, + { + name: "A", + svg: false, + parent: 2, + attributes: [["xlink:href", "/html-link"]], + }, { name: "DIV", parent: 3 }, { name: "BODY", parent: 0 }, ]); @@ -507,32 +696,39 @@ describe("SVG xlink:href context", () => { ]); }); - it("stops SVG context at foreignObject and resumes it for a nested svg", () => { - const result = captureSvg( - svgDocument([ - { name: "svg", parent: 0 }, - { name: "foreignObject", parent: 1 }, - { - name: "A", - parent: 2, - attributes: [["xlink:href", "/xl-html-fo"]], - }, - { name: "svg", parent: 2 }, - { - name: "a", - parent: 4, - attributes: [["xlink:href", "/xl-nested"]], - }, - ]), - ); - expect(urlFor(result, 3)).toBeUndefined(); - expect(urlFor(result, 5)).toMatchObject({ - attribute: "href", - url: `${origin}/xl-nested`, - destination_scope: "approved", - }); - }); + it.each(["foreignObject", "desc", "title"])( + "preserves browser SVG identity across %s HTML integration points", + (integrationPoint) => { + const result = captureSvg( + svgDocument([ + { name: "svg", parent: 0 }, + { name: integrationPoint, parent: 1 }, + { + name: "A", + svg: false, + parent: 2, + attributes: [["xlink:href", "/xl-html-fo"]], + }, + { name: "svg", parent: 2 }, + { + name: "a", + svg: true, + parent: 4, + attributes: [["xlink:href", "/xl-nested"]], + }, + ]), + ); + expect(urlFor(result, 3)).toBeUndefined(); + expect(urlFor(result, 5)).toMatchObject({ + attribute: "href", + url: `${origin}/xl-nested`, + destination_scope: "approved", + }); + }, + ); +}); +describe("SVG XLink precedence and unknown identity", () => { it("keeps an empty href ahead of xlink:href", () => { const baseUrl = `${origin}/screens/assets/`; const result = captureSvg( @@ -541,6 +737,7 @@ describe("SVG xlink:href context", () => { { name: "svg", parent: 0 }, { name: "a", + svg: true, parent: 1, attributes: [ ["href", ""], @@ -567,6 +764,7 @@ describe("SVG xlink:href context", () => { { name: "svg", parent: 0 }, { name: "a", + svg: true, parent: 1, attributes: [["xlink:href", "javascript:void(0)"]], }, @@ -582,7 +780,7 @@ describe("SVG xlink:href context", () => { ]); }); - it("does not treat a cyclic or dangling parent chain as SVG", () => { + it("keeps an unavailable SVG identity unknown despite ancestor names", () => { const cyclic = captureSvg( svgDocument([ { name: "g", parent: 2 }, @@ -652,7 +850,12 @@ describe("XML SVG roots and namespace scope restoration", () => { ["xmlns:p", XLINK_NAMESPACE], ], }, - { name: "a", parent: 1, attributes: [["p:href", "/linked"]] }, + { + name: "a", + svg: true, + parent: 1, + attributes: [["p:href", "/linked"]], + }, ]), ); expect(result.urls).toEqual([approvedHref(2, `${origin}/linked`)]); @@ -764,6 +967,7 @@ describe("XHTML and XML SVG namespace prefixes", () => { { name: "svg:foreignObject", parent: 2 }, { name: "a", + svg: true, parent: 3, attributes: [["xlink:href", "/xl-html-fo"]], }, @@ -799,6 +1003,7 @@ describe("XHTML and XML SVG namespace prefixes", () => { }, { name: "a", + svg: true, parent: 2, attributes: [ ["xlink:href", "/not-xlink"], @@ -918,11 +1123,13 @@ describe("XHTML SVG exclusions, HTML documents, and prefix scope", () => { }, { name: "a", + svg: true, parent: 2, attributes: [["xlink:href", "/xl-literal"]], }, { name: "a", + svg: true, parent: 2, attributes: [["x:href", "/xl-custom-html"]], }, @@ -937,6 +1144,7 @@ describe("XHTML SVG exclusions, HTML documents, and prefix scope", () => { }, { name: "A", + svg: false, parent: 1, attributes: [["xlink:href", "/xl-html-anchor"]], }, @@ -944,7 +1152,9 @@ describe("XHTML SVG exclusions, HTML documents, and prefix scope", () => { ); expect(result.urls).toEqual([approvedHref(3, `${origin}/xl-literal`)]); }); +}); +describe("XML namespace resolution ordering and capacity", () => { it("applies an element's own xmlns declarations to its name and href", () => { const result = captureSvg( svgDocument([ diff --git a/src/browser/CdpCaptureDocuments.ts b/src/browser/CdpCaptureDocuments.ts index 3c7fb98ed..f7eb4fe31 100644 --- a/src/browser/CdpCaptureDocuments.ts +++ b/src/browser/CdpCaptureDocuments.ts @@ -11,7 +11,11 @@ import { } from "./CdpCaptureValues.js"; import { exclusionReasonForUrl } from "./CdpCaptureEventHelpers.js"; import type { CdpCaptureCompleteness } from "./CdpCaptureCompleteness.js"; -import { createDocumentSvgAncestry, domSvgLink } from "./CdpCaptureDomSvg.js"; +import { + createDocumentSvgAncestry, + domSvgLink, + type DomSvgNodeQuery, +} from "./CdpCaptureDomSvg.js"; import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js"; export type CapturedResource = Omit< @@ -135,18 +139,39 @@ export const walkFrameTrees = function* ( } }; -export const captureDom = ( +interface DomMetadataCapture { + readonly urls: WebPageInspection["metadata"]["dom_urls"]; + readonly agentHints: WebPageInspection["metadata"]["agent_hints"]; + readonly excludedUrls: number; + readonly unresolvedSvgNodes: number; +} + +export interface DomCapture extends DomMetadataCapture { + readonly total: number; + readonly nodes: WebPageInspection["dom"]["nodes"]; +} + +// Bound retained projections and outstanding CDP reads independently of page size. +// Eight leaves room for the session's other cancellation listeners. +const DOM_METADATA_BATCH_SIZE = 8; + +interface PendingDomMetadata { + readonly options: DomMetadataOptions; + // Undefined means no query is needed; null means the candidate has no valid ID. + readonly svgQuery: DomSvgNodeQuery | null | undefined; +} + +/** Project one snapshot, requesting bounded batches of SVG identity facts. */ +export function* captureDom( result: unknown, allowedOrigins: ReadonlySet, input: InspectWebPageInput, completeness?: CdpCaptureCompleteness, -): { - readonly total: number; - readonly nodes: WebPageInspection["dom"]["nodes"]; - readonly urls: WebPageInspection["metadata"]["dom_urls"]; - readonly agentHints: WebPageInspection["metadata"]["agent_hints"]; - readonly excludedUrls: number; -} => { +): Generator< + readonly DomSvgNodeQuery[], + DomCapture, + readonly (boolean | undefined)[] +> { const root = requiredRecord(result); const strings = Array.isArray(root.strings) ? root.strings.map((value) => cdpStringValue(value) ?? "") @@ -156,6 +181,7 @@ export const captureDom = ( const agentHints: WebPageInspection["metadata"]["agent_hints"] = []; let excludedUrls = 0; let total = 0; + let unresolvedSvgNodes = 0; for (const document of recordsValue(root.documents)) { const documentUrl = indexedString(strings, document.documentURL); if (allowedSanitizedUrl(documentUrl, allowedOrigins) === undefined) { @@ -172,6 +198,9 @@ export const captureDom = ( const attributes = Array.isArray(documentNodes.attributes) ? documentNodes.attributes : []; + const backendNodeIds = Array.isArray(documentNodes.backendNodeId) + ? documentNodes.backendNodeId + : []; const baseIndex = nodes.length; const ancestry = createDocumentSvgAncestry({ nodeTypes, @@ -181,12 +210,29 @@ export const captureDom = ( attributes, }); total += nodeTypes.length; + const pendingMetadata: PendingDomMetadata[] = []; for (let index = 0; index < nodeTypes.length; index += 1) { const attributeIndexes = numberArray(attributes[index]); const parent = Math.trunc(parents[index] ?? -1); const nodeIndex = nodes.length; const nodeName = indexedString(strings, nodeNames[index]); const attributeNames = uniqueAttributeNames(attributeIndexes, strings); + const svgLink = domSvgLink(ancestry, index); + let svgQuery: DomSvgNodeQuery | null | undefined; + if ( + !svgLink.xmlDocument && + nodeTypes[index] === 1 && + attributeNames.includes("xlink:href") && + !attributeNames.includes("href") + ) { + const backendNodeId = numberValue(backendNodeIds[index]); + svgQuery = + backendNodeId !== undefined && + Number.isSafeInteger(backendNodeId) && + backendNodeId > 0 + ? { backendNodeId, nodeName } + : null; + } nodes.push({ index: nodeIndex, parent_index: parent < 0 ? -1 : baseIndex + parent, @@ -195,30 +241,34 @@ export const captureDom = ( node_value_length: indexedString(strings, nodeValues[index]).length, attribute_names: attributeNames, }); - const metadata = domMetadata({ - strings, - attributes: attributeIndexes, - // The BASE element itself resolves against the document fallback URL. - baseUrl: nodeName.toLowerCase() === "base" ? documentUrl : baseUrl, - documentUrl, - nodeIndex, - nodeName, - allowedOrigins, - ...domSvgLink(ancestry, index), + pendingMetadata.push({ + svgQuery, + options: { + strings, + attributes: attributeIndexes, + // The BASE element itself resolves against the document fallback URL. + baseUrl: nodeName.toLowerCase() === "base" ? documentUrl : baseUrl, + documentUrl, + nodeIndex, + nodeName, + allowedOrigins, + ...svgLink, + }, }); - for (const url of metadata.urls) { - urls.push(url); - if (url.destination_scope !== "approved") { - excludedUrls += 1; - completeness?.exclude( - "metadata", - url.destination_scope === "outside_policy" - ? "disallowed_origin" - : "unsupported_url", - ); - } + if ( + pendingMetadata.length === DOM_METADATA_BATCH_SIZE || + index === nodeTypes.length - 1 + ) { + const metadata = yield* captureDomMetadata( + pendingMetadata, + completeness, + ); + urls.push(...metadata.urls); + agentHints.push(...metadata.agentHints); + excludedUrls += metadata.excludedUrls; + unresolvedSvgNodes += metadata.unresolvedSvgNodes; + pendingMetadata.length = 0; } - agentHints.push(...metadata.agentHints); } } return { @@ -227,8 +277,54 @@ export const captureDom = ( urls, agentHints, excludedUrls, + unresolvedSvgNodes, }; -}; +} + +function* captureDomMetadata( + pending: readonly PendingDomMetadata[], + completeness: CdpCaptureCompleteness | undefined, +): Generator< + readonly DomSvgNodeQuery[], + DomMetadataCapture, + readonly (boolean | undefined)[] +> { + const queries = pending.flatMap(({ svgQuery }) => + svgQuery == null ? [] : [svgQuery], + ); + const kinds = queries.length > 0 ? yield queries : []; + const urls: DomMetadataCapture["urls"] = []; + const agentHints: DomMetadataCapture["agentHints"] = []; + let queryIndex = 0; + let unresolvedSvgNodes = 0; + let excludedUrls = 0; + for (const { options, svgQuery } of pending) { + let svgElement = options.svgElement; + if (svgQuery !== undefined) { + const kind = svgQuery === null ? undefined : kinds[queryIndex++]; + svgElement = kind === true; + if (kind === undefined) { + unresolvedSvgNodes += 1; + completeness?.exclude("metadata", "provider_unavailable"); + } + } + const metadata = domMetadata({ ...options, svgElement }); + for (const url of metadata.urls) { + urls.push(url); + if (url.destination_scope !== "approved") { + excludedUrls += 1; + completeness?.exclude( + "metadata", + url.destination_scope === "outside_policy" + ? "disallowed_origin" + : "unsupported_url", + ); + } + } + agentHints.push(...metadata.agentHints); + } + return { urls, agentHints, unresolvedSvgNodes, excludedUrls }; +} export const captureAccessibility = ( results: readonly unknown[], diff --git a/src/browser/CdpCaptureDomSvg.ts b/src/browser/CdpCaptureDomSvg.ts index 3d1a5c977..4d021766c 100644 --- a/src/browser/CdpCaptureDomSvg.ts +++ b/src/browser/CdpCaptureDomSvg.ts @@ -1,4 +1,4 @@ -import { numberValue } from "./CdpCaptureValues.js"; +import { numberValue, recordValue } from "./CdpCaptureValues.js"; const SVG_NAMESPACE = "http://www.w3.org/2000/svg"; const XLINK_NAMESPACE = "http://www.w3.org/1999/xlink"; @@ -18,11 +18,34 @@ interface XmlSvgLink { interface DocumentSvgAncestry { readonly xml: boolean; - readonly nodes: DomSnapshotNodes; - readonly svgContexts: Map; readonly xmlLinks: Map; } +export interface DomSvgNodeQuery { + readonly backendNodeId: number; + readonly nodeName: string; +} + +/** Bind the browser's SVG flag to the exact snapshotted element. */ +export const capturedSvgNodeKind = ( + query: DomSvgNodeQuery, + result: unknown, +): boolean | undefined => { + const node = recordValue(recordValue(result)?.node); + if ( + node?.backendNodeId !== query.backendNodeId || + node.nodeName !== query.nodeName || + node.nodeType !== 1 + ) + return undefined; + // CDP omits isSVG for non-SVG nodes. Other values are malformed, not false. + return node.isSVG === undefined + ? false + : typeof node.isSVG === "boolean" + ? node.isSVG + : undefined; +}; + interface DomSvgLink { readonly svgElement: boolean; readonly xmlDocument: boolean; @@ -79,8 +102,6 @@ export const createDocumentSvgAncestry = ( const xml = xmlDocument(nodes); return { xml, - nodes, - svgContexts: new Map(), xmlLinks: xml ? resolveXmlElements(nodes) : new Map(), }; }; @@ -91,11 +112,7 @@ export const domSvgLink = ( ): DomSvgLink => { if (!ancestry.xml) { return { - svgElement: elementInSvgContext( - index, - ancestry.nodes, - ancestry.svgContexts, - ), + svgElement: false, xmlDocument: false, xlinkHref: undefined, }; @@ -108,43 +125,6 @@ export const domSvgLink = ( }; }; -// Chrome's HTML parser records SVG local names (`svg`, `foreignObject`). -// Descendants of foreignObject are HTML until a nested `svg`. -const elementInSvgContext = ( - index: number, - nodes: DomSnapshotNodes, - contexts: Map, -): boolean => { - let current = index; - const seen = new Set(); - let inSvg = false; - while ( - current >= 0 && - current < nodes.nodeNames.length && - !seen.has(current) - ) { - const cached = contexts.get(current); - if (cached !== undefined) { - inSvg = cached; - break; - } - seen.add(current); - if (stringAt(nodes.strings, nodes.nodeNames[current]) === "svg") { - inSvg = true; - break; - } - const parent = Math.trunc(nodes.parents[current] ?? -1); - if (parent < 0 || parent === current) break; - if (stringAt(nodes.strings, nodes.nodeNames[parent]) === "foreignObject") - break; - current = parent; - } - // Cache the entire examined chain, including forward references and cycles, - // so shared ancestors are examined once per document. - for (const node of seen) contexts.set(node, inSvg); - return inSvg; -}; - interface NamespaceRestore { readonly prefix: string; readonly previous: string | undefined; diff --git a/src/browser/CdpPageCapture.ts b/src/browser/CdpPageCapture.ts index 0d97576b5..f49d4e186 100644 --- a/src/browser/CdpPageCapture.ts +++ b/src/browser/CdpPageCapture.ts @@ -17,8 +17,10 @@ import { captureFrames, captureResources, type CapturedResource, + type DomCapture, mainFrameUrl, } from "./CdpCaptureDocuments.js"; +import { capturedSvgNodeKind } from "./CdpCaptureDomSvg.js"; import { allowedSanitizedUrl, delayWithCancellation, @@ -52,7 +54,7 @@ interface CaptureState { interface CapturedSections { readonly attachedUrl: string; readonly frames: WebPageInspection["frames"]; - readonly dom: ReturnType; + readonly dom: DomCapture; readonly accessibility: ReturnType; readonly scripts: WebPageInspection["scripts"]; readonly resources: WebPageInspection["resources"]; @@ -260,9 +262,7 @@ const authorizeObservationWindow = async ( throw new BrowserObservationError("inspect_web_page", "target_not_allowed"); }; -const capturePageDom = async ( - state: CaptureState, -): Promise> => { +const capturePageDom = async (state: CaptureState): Promise => { const { context, allowedOrigins } = state; const result = await context.connection.send( "DOMSnapshot.captureSnapshot", @@ -274,12 +274,56 @@ const capturePageDom = async ( context.sessionId, context.signal, ); - return captureDom( + const capture = captureDom( result, allowedOrigins, context.input, state.events.completeness, ); + let step = capture.next(); + let descriptionUnavailable = false; + const pendingController = new AbortController(); + const signal = + context.signal === undefined + ? pendingController.signal + : AbortSignal.any([context.signal, pendingController.signal]); + while (!step.done) { + const pending = step.value.map(async (query) => { + if (descriptionUnavailable) return undefined; + try { + return capturedSvgNodeKind( + query, + await context.connection.send( + "DOM.describeNode", + { backendNodeId: query.backendNodeId, depth: 0 }, + context.sessionId, + signal, + ), + ); + } catch (cause: unknown) { + // A detached node can disappear between snapshot and description. + // Preserve partial metadata, but propagate transport/cancellation errors. + if (!(cause instanceof CdpCommandRejection)) throw cause; + if (cause.code === -32_601) descriptionUnavailable = true; + else if (cause.code !== -32_000) throw cause; + return undefined; + } + }); + try { + // Promise.all preserves snapshot order even when CDP replies out of order. + step = capture.next(await Promise.all(pending)); + } catch (cause: unknown) { + // Stop and settle this bounded batch before the session can be detached. + pendingController.abort(); + await Promise.allSettled(pending); + throw cause; + } + } + if (step.value.unresolvedSvgNodes > 0) + state.limitations.push( + `SVG identity was unavailable for ${String(step.value.unresolvedSvgNodes)} snapshotted XLink candidate nodes; their URL destinations remain unknown.`, + ); + return step.value; }; const normalizedInspection = ( diff --git a/src/browser/PlaywrightBrowserScenarioProvider.test.ts b/src/browser/PlaywrightBrowserScenarioProvider.test.ts index aacf766ad..734e32426 100644 --- a/src/browser/PlaywrightBrowserScenarioProvider.test.ts +++ b/src/browser/PlaywrightBrowserScenarioProvider.test.ts @@ -303,6 +303,7 @@ describe("PlaywrightBrowserScenarioProvider", () => { }); expect(result.value.limitations).toContain(session.eventGaps[0]); }); + it("returns an initial state and every step beyond the former action ceiling", async () => { const session = new FakeSession("launch"); const provider = new PlaywrightBrowserScenarioProvider(() => @@ -376,6 +377,35 @@ describe("PlaywrightBrowserScenarioProvider", () => { }); }); +describe("PlaywrightBrowserScenarioProvider timing", () => { + it("keeps capture elapsed values valid across a backward wall-clock jump", async () => { + let wallClock = 10_000; + vi.spyOn(Date, "now").mockImplementation(() => wallClock); + const session = new FakeSession("launch"); + session.onPerform = () => { + wallClock = 1; + }; + const provider = new PlaywrightBrowserScenarioProvider(() => + Promise.resolve(session), + ); + + try { + const result = await provider.captureScenario(scenario()); + if (!result.ok) throw result.error; + expect(wallClock).toBe(1); + expect(Number.isInteger(result.value.duration_ms)).toBe(true); + expect(result.value.duration_ms).toBeGreaterThanOrEqual(0); + expect( + result.value.steps.every( + ({ elapsed_ms }) => Number.isInteger(elapsed_ms) && elapsed_ms >= 0, + ), + ).toBe(true); + } finally { + vi.restoreAllMocks(); + } + }); +}); + describe("PlaywrightBrowserScenarioProvider comparison coverage", () => { it("keeps quiet captures with different event families from comparing unchanged", async () => { const capture = async (events: readonly BrowserScenarioEventFamily[]) => { diff --git a/src/browser/PlaywrightBrowserScenarioProvider.ts b/src/browser/PlaywrightBrowserScenarioProvider.ts index a3c6d6a62..b0833e89c 100644 --- a/src/browser/PlaywrightBrowserScenarioProvider.ts +++ b/src/browser/PlaywrightBrowserScenarioProvider.ts @@ -172,7 +172,7 @@ const executeStep = async (input: { ), }); - const actionStartedAt = Date.now(); + const actionStartedAt = performance.now(); session.setStep(stepIndex); const requestCancelled = (): boolean => signal?.aborted === true; let outcome: BrowserScenarioStepOutcome = { @@ -200,7 +200,7 @@ const executeStep = async (input: { stepId: action.step_id, action: action.action, ...outcome, - elapsedMs: Date.now() - actionStartedAt, + elapsedMs: Math.max(0, Math.round(performance.now() - actionStartedAt)), beforeUrl, afterUrl: session.sanitizeUrl(session.currentUrl()), eventStart, @@ -233,7 +233,7 @@ const runScenario = async ( ): Promise => { if (options.signal?.aborted === true) throw new BrowserObservationError(OPERATION, "cancelled"); - const startedAt = Date.now(); + const startedAt = performance.now(); const session = await openSession(scenario, { ...(options.signal === undefined ? {} : { signal: options.signal }), retainCleanup, @@ -250,7 +250,7 @@ const runScenario = async ( await initialStep({ session, scenario, - elapsedMs: Date.now() - startedAt, + elapsedMs: Math.max(0, Math.round(performance.now() - startedAt)), signal: options.signal, }), ); diff --git a/src/browser/PlaywrightElectronActiveActions.ts b/src/browser/PlaywrightElectronActiveActions.ts index d59afb676..7bb9ec83b 100644 --- a/src/browser/PlaywrightElectronActiveActions.ts +++ b/src/browser/PlaywrightElectronActiveActions.ts @@ -72,7 +72,7 @@ export const runElectronActions = async ( ): Promise => { const actions: ElectronActions = []; for (const action of input.actions) { - const actionStartedAt = Date.now(); + const actionStartedAt = performance.now(); const windowIndex = actionWindowIndex(action); let selectedWindow: Page | undefined; try { @@ -107,7 +107,10 @@ export const runElectronActions = async ( target: selectedWindow === undefined ? null : `window:${String(windowIndex)}`, status: "completed", - elapsed_ms: Date.now() - actionStartedAt, + elapsed_ms: Math.max( + 0, + Math.round(performance.now() - actionStartedAt), + ), error: null, }); } catch (cause: unknown) { @@ -118,7 +121,10 @@ export const runElectronActions = async ( target: selectedWindow === undefined ? null : `window:${String(windowIndex)}`, status: options.signal?.aborted ? "cancelled" : "failed", - elapsed_ms: Date.now() - actionStartedAt, + elapsed_ms: Math.max( + 0, + Math.round(performance.now() - actionStartedAt), + ), error: safeActionErrorMessage(cause), }); break; @@ -366,7 +372,7 @@ export const runWithExecutionLimits = async ( new Promise((_, reject) => { timer = setTimeout( () => reject(new BrowserObservationError(OPERATION, "timeout")), - Math.max(1, deadline - Date.now()), + Math.max(1, deadline - performance.now()), ); }), ); diff --git a/src/browser/PlaywrightElectronActiveProvider.ts b/src/browser/PlaywrightElectronActiveProvider.ts index 4b6e6bb14..23379524d 100644 --- a/src/browser/PlaywrightElectronActiveProvider.ts +++ b/src/browser/PlaywrightElectronActiveProvider.ts @@ -86,7 +86,7 @@ export class PlaywrightElectronActiveProvider implements ElectronActiveObservati if (options.signal?.aborted === true) throw new BrowserObservationError(OPERATION, "cancelled"); await prepareProcessOwnershipInspection(options.signal); - const startupDeadline = Date.now() + STARTUP_TIMEOUT_MS; + const startupDeadline = performance.now() + STARTUP_TIMEOUT_MS; paths = { executable: input.executable_path, application: input.application_path, @@ -101,7 +101,7 @@ export class PlaywrightElectronActiveProvider implements ElectronActiveObservati cwd: paths.root, env: safeElectronEnvironment(this.environment, runId), args: ["-r", hookPath, paths.application, ...input.args], - timeout: Math.max(1, startupDeadline - Date.now()), + timeout: Math.max(1, startupDeadline - performance.now()), }); const leaderPid = application.process().pid; if ( @@ -128,7 +128,7 @@ export class PlaywrightElectronActiveProvider implements ElectronActiveObservati const state = await runWithExecutionLimits( readApplicationState(application), undefined, - options.signal?.aborted ? Date.now() + 5_000 : undefined, + options.signal?.aborted ? performance.now() + 5_000 : undefined, ); capture = projectElectronActiveCapture(paths, actions, state); outcome = options.signal?.aborted @@ -187,7 +187,7 @@ export class PlaywrightElectronActiveProvider implements ElectronActiveObservati await runWithExecutionLimits( application.close(), undefined, - Date.now() + 5_000, + performance.now() + 5_000, ); } catch (cause: unknown) { closeError = cause; diff --git a/src/browser/PlaywrightExecutionBoundary.test.ts b/src/browser/PlaywrightExecutionBoundary.test.ts index 323b4506c..51df81701 100644 --- a/src/browser/PlaywrightExecutionBoundary.test.ts +++ b/src/browser/PlaywrightExecutionBoundary.test.ts @@ -7,7 +7,10 @@ import { } from "../domain/analysisErrorCore.js"; describe("withPlaywrightExecutionBoundary", () => { - afterEach(() => vi.useRealTimers()); + afterEach(() => { + vi.useRealTimers(); + vi.restoreAllMocks(); + }); it("rejects work that exceeds its absolute remaining duration", async () => { vi.useFakeTimers(); @@ -31,4 +34,20 @@ describe("withPlaywrightExecutionBoundary", () => { controller.abort(); await expect(result).rejects.toBeInstanceOf(AnalysisCancelledError); }); + + it("keeps its timeout stable across a wall-clock jump", async () => { + vi.spyOn(Date, "now") + .mockReturnValueOnce(1_000) + .mockReturnValue(10_000_000); + + await expect( + withPlaywrightExecutionBoundary( + () => + new Promise((resolve) => + setTimeout(() => resolve("done"), 5), + ), + 1_000, + ), + ).resolves.toBe("done"); + }); }); diff --git a/src/browser/PlaywrightExecutionBoundary.ts b/src/browser/PlaywrightExecutionBoundary.ts index 9fd21fcf3..759fcd484 100644 --- a/src/browser/PlaywrightExecutionBoundary.ts +++ b/src/browser/PlaywrightExecutionBoundary.ts @@ -19,10 +19,10 @@ export const withPlaywrightExecutionBoundary = ( let settled = false; let timer: ReturnType | undefined; const deadline = - timeoutMs === undefined ? undefined : Date.now() + timeoutMs; + timeoutMs === undefined ? undefined : performance.now() + timeoutMs; const scheduleTimeout = (): void => { if (deadline === undefined) return; - const remaining = deadline - Date.now(); + const remaining = deadline - performance.now(); if (remaining <= 0) { fail(new AnalysisTimeoutError(OPERATION, timeoutMs ?? 0)); return; diff --git a/src/browser/PlaywrightScenarioSession.ts b/src/browser/PlaywrightScenarioSession.ts index 486f555ea..949193b7c 100644 --- a/src/browser/PlaywrightScenarioSession.ts +++ b/src/browser/PlaywrightScenarioSession.ts @@ -133,7 +133,7 @@ export class PlaywrightScenarioSession implements BrowserScenarioSessionPort { const secrets = BrowserScenarioSecrets.resolve(scenario, environment); if (secrets === undefined) throw new BrowserObservationError(OPERATION, "secret_unavailable"); - const startedAt = Date.now(); + const startedAt = performance.now(); const opening = openPlaywrightScenarioBrowser(scenario, environment, { signal: options.signal, ...(options.retainCleanup === undefined diff --git a/src/browser/WebSourceMapFetcher.ts b/src/browser/WebSourceMapFetcher.ts index 0b17df0cb..c4bae782a 100644 --- a/src/browser/WebSourceMapFetcher.ts +++ b/src/browser/WebSourceMapFetcher.ts @@ -2,7 +2,6 @@ import * as t from "@babel/types"; import { decodeValidatedSourceMapLeaves, isBeforeSourceMapLeafStop, - resolveSourceMapSource, } from "../javascript/sourceMaps/DecodedSourceMap.js"; import { sanitizeBrowserUrl } from "../domain/browserObservation.js"; @@ -21,6 +20,7 @@ import { webSourceMapsSchema } from "../domain/webBundleAnalysis.js"; import { createWebTextArtifact } from "../domain/webContentArtifact.js"; import { safeParseJson } from "../domain/safeJson.js"; import { sourceMapJsonText } from "../domain/sourceMapEnvelope.js"; +import { resolveJavaScriptSourceMapUrl } from "../domain/javascript/javascriptSourceMapPaths.js"; import { jsonParts } from "../domain/jsonSerialization.js"; import { inspectSourceMapValue, @@ -86,7 +86,7 @@ export const fetchWebSourceMaps = async ( signal?.addEventListener("abort", abortFromCaller, { once: true }); const operationSignal = operationController.signal; const deadlineAt = - Date.now() + (host.timeoutMs ?? SOURCE_MAP_FETCH_TIMEOUT_MS); + performance.now() + (host.timeoutMs ?? SOURCE_MAP_FETCH_TIMEOUT_MS); const budget = { retainedBytes: 0, deadlineAt, @@ -110,7 +110,7 @@ export const fetchWebSourceMaps = async ( } const expandedBytesBeforeItem = budget.decodedRecords.expandedBytes; budget.decodedRecords.expandedBytes += reservation; - if (operationSignal.aborted || Date.now() >= deadlineAt) { + if (operationSignal.aborted || performance.now() >= deadlineAt) { items.push( emptySourceMapItem( request, @@ -789,7 +789,7 @@ const checkOperation = ( ): void => { if (callerSignal?.aborted === true) throw callerSignal.reason; if (operationSignal?.aborted === true) throw operationSignal.reason; - if (Date.now() >= deadlineAt) throw new SourceMapDeadlineError(); + if (performance.now() >= deadlineAt) throw new SourceMapDeadlineError(); }; const approvedUrl = ( @@ -867,7 +867,11 @@ const preflightResolvedSourceBytes = ( const sourceKey = source; let resolved = sourcesForRoot.get(sourceKey); if (resolved === undefined) - resolved = resolveSourceMapSource(source, map.sourceRoot, mapUrl); + resolved = resolveJavaScriptSourceMapUrl( + source, + map.sourceRoot, + mapUrl, + ); const bytes = Buffer.byteLength(resolved, "utf8"); if (bytes > SOURCE_MAP_RESPONSE_BYTES - resolvedBytes) throw resolvedSourceLimitFailure(); diff --git a/src/browser/assets/LocalWebScriptArtifacts.ts b/src/browser/assets/LocalWebScriptArtifacts.ts index c5d6ea8ed..51fc14c45 100644 --- a/src/browser/assets/LocalWebScriptArtifacts.ts +++ b/src/browser/assets/LocalWebScriptArtifacts.ts @@ -1,6 +1,7 @@ import { lstat, realpath } from "node:fs/promises"; import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path"; import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js"; +import { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js"; import { readStableArtifact } from "../../artifacts/readStableArtifact.js"; import { normalizeArtifactPath } from "../../artifacts/ArtifactPaths.js"; import type { WebScriptArtifactPort } from "../../application/WebScriptArtifactPort.js"; @@ -19,10 +20,36 @@ import { /** Read one digest-verified captured source under the manifest's current location. */ export class LocalWebScriptArtifacts implements WebScriptArtifactPort { - constructor(readonly operation: WebArtifactReadContext["operation"]) {} + constructor( + readonly operation: WebArtifactReadContext["operation"], + readonly resources = new ArtifactResourceScope(), + ) {} + + close(): Promise { + return this.resources.close(); + } /** Read the selected source; relocated exports retain their reported original root. */ async load(input: SelectedWebScriptInput, options?: ExecutionOptions) { + try { + return await this.resources.run(() => this.#load(input, options)); + } catch (cause: unknown) { + return err( + webArtifactReadError( + cause, + { + operation: this.operation, + field: ["manifest_path"], + targetPath: input.manifest_path, + callerSelected: true, + }, + options?.signal, + ), + ); + } + } + + async #load(input: SelectedWebScriptInput, options?: ExecutionOptions) { let field: readonly (string | number)[] = ["manifest_path"]; let targetPath = input.manifest_path; let callerSelected = true; @@ -31,6 +58,7 @@ export class LocalWebScriptArtifacts implements WebScriptArtifactPort { input.manifest_path, 32 * 1024 * 1024, options?.signal, + this.resources, ); const manifest = webScriptExportManifestSchema.parse(manifestBytes.value); field = ["script_index"]; @@ -65,6 +93,8 @@ export class LocalWebScriptArtifacts implements WebScriptArtifactPort { sourcePath, 16 * 1024 * 1024, options?.signal, + undefined, + this.resources, ); await assertContained(root, sourcePath); if ( diff --git a/src/browser/assets/PublishWebScripts.ts b/src/browser/assets/PublishWebScripts.ts index 2db5d7dcf..ba886f00e 100644 --- a/src/browser/assets/PublishWebScripts.ts +++ b/src/browser/assets/PublishWebScripts.ts @@ -113,7 +113,7 @@ export const publishWebScripts = async ( options.signal, ); options.signal?.throwIfAborted(); - await tree.commit(); + await tree.commit(options.signal); return result; } catch (cause: unknown) { if (cause instanceof SafeOutputTreeCreationFailure) tree = cause.tree; diff --git a/src/browser/assets/WebArtifactReads.ts b/src/browser/assets/WebArtifactReads.ts index cd142f62f..e045e1575 100644 --- a/src/browser/assets/WebArtifactReads.ts +++ b/src/browser/assets/WebArtifactReads.ts @@ -1,5 +1,6 @@ import { z } from "zod"; import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js"; +import type { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js"; import { readStableArtifact } from "../../artifacts/readStableArtifact.js"; import type { AnalysisError } from "../../domain/analysisErrorBase.js"; import { @@ -9,6 +10,8 @@ import { } from "../../domain/analysisErrorCore.js"; import { ArtifactOperationError } from "../../domain/artifactOperationError.js"; import { ProviderAdapterError } from "../../domain/providerAdapterError.js"; +import { ProviderCleanupError } from "../../domain/providerCleanupError.js"; +import { analysisErrorWithCleanupFailure } from "../../domain/analysisErrorCleanup.js"; import { safeParseJson } from "../../domain/safeJson.js"; /** The selected artifact and input field owning a reader failure. */ @@ -42,8 +45,15 @@ export const readWebArtifactJson = async ( path: string, maximumBytes: number, signal?: AbortSignal, + resourceScope?: ArtifactResourceScope, ) => { - const data = await readStableArtifact(path, maximumBytes, signal); + const data = await readStableArtifact( + path, + maximumBytes, + signal, + undefined, + resourceScope, + ); const value = safeParseJson(decodeWebArtifact(data.bytes)); if (!value.ok) throw new WebArtifactFormatFailure(value.error); return { ...data, value: value.value }; @@ -55,8 +65,30 @@ export const webArtifactReadError = ( context: WebArtifactReadContext, signal?: AbortSignal, ): AnalysisError => { + if (cause instanceof ArtifactReaderFailure && cause.cleanup !== undefined) { + const primary = + signal?.aborted === true + ? new AnalysisCancelledError(context.operation, { cause }) + : new ArtifactOperationError( + context.operation, + cause.reason, + cause.details, + cause.message, + { cause }, + ); + return analysisErrorWithCleanupFailure( + primary, + new ProviderCleanupError( + "captured-web-artifacts", + cause.cleanup.resources, + { reason: cause.cleanup.reason }, + { cause, operation: context.operation }, + ), + context.operation, + ); + } if (signal?.aborted === true) - return new AnalysisCancelledError(context.operation); + return new AnalysisCancelledError(context.operation, { cause }); const selection = context.callerSelected ? selectedFileError(cause, context) : undefined; diff --git a/src/browser/assets/WebSourceLocationArtifacts.ts b/src/browser/assets/WebSourceLocationArtifacts.ts index 21c483bd8..733069469 100644 --- a/src/browser/assets/WebSourceLocationArtifacts.ts +++ b/src/browser/assets/WebSourceLocationArtifacts.ts @@ -1,6 +1,7 @@ import type { WebSourceLocationArtifactPort } from "../../application/WebSourceLocationPorts.js"; import type { WebScriptArtifactPort } from "../../application/WebScriptArtifactPort.js"; import type { ExecutionOptions } from "../../application/AnalysisProvider.js"; +import { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js"; import { readStableArtifact } from "../../artifacts/readStableArtifact.js"; import { WEB_SOURCE_MAP_LIMITS, @@ -12,14 +13,64 @@ import { decodeWebArtifact, webArtifactReadError } from "./WebArtifactReads.js"; /** Read one verified capture source and the caller-selected map, without fetching. */ export class LocalWebSourceLocationArtifacts implements WebSourceLocationArtifactPort { + readonly scripts: WebScriptArtifactPort; + readonly #resources: ArtifactResourceScope; + constructor( - readonly scripts: WebScriptArtifactPort = new LocalWebScriptArtifacts( - "trace_web_source_location", - ), - ) {} + scripts?: WebScriptArtifactPort, + resources = scripts instanceof LocalWebScriptArtifacts + ? scripts.resources + : new ArtifactResourceScope(), + ) { + this.#resources = resources; + this.scripts = + scripts ?? + new LocalWebScriptArtifacts("trace_web_source_location", resources); + } + + async close(): Promise { + const closeScript = + this.scripts instanceof LocalWebScriptArtifacts && + this.scripts.resources === this.#resources + ? undefined + : this.scripts.close?.bind(this.scripts); + const results = await Promise.allSettled([ + Promise.resolve().then(() => this.#resources.close()), + ...(closeScript === undefined + ? [] + : [Promise.resolve().then(closeScript)]), + ]); + const failures = results.flatMap((result) => + result.status === "rejected" ? [result.reason] : [], + ); + if (failures.length > 0) + throw new AggregateError( + failures, + "Web source-location artifact cleanup failed", + ); + } /** Preserve actual map identity separately from the selected deployment URL. */ async load(input: WebSourceLocationInput, options?: ExecutionOptions) { + try { + return await this.#resources.run(() => this.#load(input, options)); + } catch (cause: unknown) { + return err( + webArtifactReadError( + cause, + { + operation: "trace_web_source_location", + field: ["source_map", "path"], + targetPath: input.source_map.path, + callerSelected: true, + }, + options?.signal, + ), + ); + } + } + + async #load(input: WebSourceLocationInput, options?: ExecutionOptions) { const loaded = await this.scripts.load(input, options); if (!loaded.ok) return loaded; try { @@ -27,6 +78,8 @@ export class LocalWebSourceLocationArtifacts implements WebSourceLocationArtifac input.source_map.path, WEB_SOURCE_MAP_LIMITS.mapBytes, options?.signal, + undefined, + this.#resources, ); return ok({ ...loaded.value, diff --git a/src/browser/history/CaptureFailures.test.ts b/src/browser/history/CaptureFailures.test.ts index ff7be71a6..047722ac5 100644 --- a/src/browser/history/CaptureFailures.test.ts +++ b/src/browser/history/CaptureFailures.test.ts @@ -4,7 +4,7 @@ import { projectAnalysisError } from "../../domain/analysisErrorProjection.js"; import { analysisErrorProjectionSchema } from "../../contracts/errorSchemas.js"; import { inspectWebNetworkCaptureInputSchema } from "../../domain/webNetworkCapture.js"; import { historicalCaptureFailure } from "./CaptureFailures.js"; -import { HAR_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES } from "./CaptureRelease.js"; +import { NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES } from "./CaptureRelease.js"; import { redactExplicitFailure } from "../../domain/explicitSensitiveFailure.js"; import { OwnedCommandFailure } from "../../process/OwnedCommand.js"; @@ -118,7 +118,7 @@ it.each([ }, }); // The enforced limit is reported only where Bun runs on Windows. - if (HAR_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES === null) + if (NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES === null) expect(JSON.stringify(projected)).not.toContain( "process_memory_limit_bytes", ); @@ -126,12 +126,12 @@ it.each([ expect(projected).toMatchObject({ details: { reported_limits: { - process_memory_limit_bytes: HAR_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES, + process_memory_limit_bytes: NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES, }, }, remediation: { action: expect.stringContaining( - `${String(HAR_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES)} bytes`, + `${String(NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES)} bytes`, ), }, }); diff --git a/src/browser/history/CaptureFailures.ts b/src/browser/history/CaptureFailures.ts index 228541545..b525126a5 100644 --- a/src/browser/history/CaptureFailures.ts +++ b/src/browser/history/CaptureFailures.ts @@ -18,9 +18,9 @@ import { } from "../../domain/webNetworkCapture.js"; import { OwnedCommandFailure } from "../../process/OwnedCommand.js"; import { - HAR_CAPTURE_HEAP_ENFORCED, - HAR_CAPTURE_HEAP_LIMITS, - HAR_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES, + NODE_CAPTURE_HEAP_ENFORCED, + NODE_CAPTURE_HEAP_LIMITS, + NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES, } from "./CaptureRelease.js"; const OPERATION = "inspect_web_network_capture"; @@ -78,7 +78,7 @@ export const historicalCaptureFailure = ( ); if ( phase === "decoder" && - input.format === "har" && + input.format !== "mitmproxy" && cause.reason === "process" && (cause.snapshot?.signal === "SIGABRT" || (cause.snapshot?.signal === null && cause.snapshot.exitCode === 134)) && @@ -86,7 +86,7 @@ export const historicalCaptureFailure = ( ) return captureMemoryFailure( input, - "The HAR decoder exhausted its fixed V8 heap", + `The ${input.format} decoder exhausted its fixed V8 heap`, cause, ); } @@ -195,15 +195,17 @@ export const captureMemoryFailure = ( format: input.format, maximum_input_bytes: WEB_NETWORK_CAPTURE_LIMITS.inputBytes, maximum_reply_bytes: WEB_NETWORK_CAPTURE_LIMITS.outputBytes, - ...(input.format === "har" && HAR_CAPTURE_HEAP_ENFORCED + ...(input.format !== "mitmproxy" && NODE_CAPTURE_HEAP_ENFORCED ? { - old_generation_heap_mib: HAR_CAPTURE_HEAP_LIMITS.oldGenerationMiB, - semi_space_heap_mib: HAR_CAPTURE_HEAP_LIMITS.semiSpaceMiB, + old_generation_heap_mib: NODE_CAPTURE_HEAP_LIMITS.oldGenerationMiB, + semi_space_heap_mib: NODE_CAPTURE_HEAP_LIMITS.semiSpaceMiB, } : {}), - ...(input.format === "har" && - HAR_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES !== null - ? { process_memory_limit_bytes: HAR_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES } + ...(input.format !== "mitmproxy" && + NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES !== null + ? { + process_memory_limit_bytes: NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES, + } : {}), ...(cause instanceof OwnedCommandFailure ? { @@ -214,7 +216,7 @@ export const captureMemoryFailure = ( }, { cause, - remediationAction: `Use a smaller capture exported by its producer, or another decoder with sufficient capacity for the original capture. Preserve the original for provenance; a subset does not establish complete-capture coverage.${input.format === "har" && HAR_CAPTURE_HEAP_ENFORCED ? ` REA's HAR old-generation heap is fixed at ${HAR_CAPTURE_HEAP_LIMITS.oldGenerationMiB} MiB; inherited NODE_OPTIONS cannot raise it.` : ""}${input.format === "har" && HAR_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES !== null ? ` REA's HAR decoder process commit limit is fixed at ${String(HAR_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES)} bytes.` : ""}`, + remediationAction: `Use a smaller capture exported by its producer, or another decoder with sufficient capacity for the original capture. Preserve the original for provenance; a subset does not establish complete-capture coverage.${input.format !== "mitmproxy" && NODE_CAPTURE_HEAP_ENFORCED ? ` REA's Node capture old-generation heap is fixed at ${NODE_CAPTURE_HEAP_LIMITS.oldGenerationMiB} MiB; inherited NODE_OPTIONS cannot raise it.` : ""}${input.format !== "mitmproxy" && NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES !== null ? ` REA's Node capture decoder process commit limit is fixed at ${String(NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES)} bytes.` : ""}`, ...(cause instanceof OwnedCommandFailure && cause.snapshot !== null ? { capturedOutput: { diff --git a/src/browser/history/CaptureFormatError.ts b/src/browser/history/CaptureFormatError.ts index fd7a9272d..a31b3057d 100644 --- a/src/browser/history/CaptureFormatError.ts +++ b/src/browser/history/CaptureFormatError.ts @@ -1,7 +1,7 @@ /** Expected malformed, unsupported or oversized producer data at a historical decoder boundary. */ export class CaptureFormatError extends Error { constructor( - readonly reason: "format" | "unsupported" | "input-limit", + readonly reason: "format" | "unsupported" | "input-limit" | "limit", message: string, readonly pointer = "", ) { diff --git a/src/browser/history/HarJson.ts b/src/browser/history/CaptureJson.ts similarity index 83% rename from src/browser/history/HarJson.ts rename to src/browser/history/CaptureJson.ts index 257630932..59743788a 100644 --- a/src/browser/history/HarJson.ts +++ b/src/browser/history/CaptureJson.ts @@ -2,6 +2,7 @@ import { visit } from "jsonc-parser"; import { LosslessNumber } from "lossless-json"; import { WEB_NETWORK_CAPTURE_LIMITS } from "../../domain/webNetworkCapture.js"; import { CaptureFormatError } from "./CaptureFormatError.js"; +import { TextDecoder } from "node:util"; type Frame = | { @@ -13,7 +14,10 @@ type Frame = | { kind: "array"; pointer: string; items: unknown[] }; /** Materialize upstream JSON events without duplicate folding, prototype setters or numeric rounding. */ -export const parseHarJson = (text: string): unknown => { +export const parseCaptureJson = ( + text: string, + format: "HAR" | "qlog", +): unknown => { const frames: Frame[] = []; let result: unknown; const valuePointer = (): string => { @@ -29,7 +33,7 @@ export const parseHarJson = (text: string): unknown => { if (frames.length > WEB_NETWORK_CAPTURE_LIMITS.depth) throw new CaptureFormatError( "input-limit", - "HAR exceeds the 64-level complete-evidence nesting budget.", + `${format} exceeds the 64-level complete-evidence nesting budget.`, valuePointer(), ); }; @@ -69,7 +73,7 @@ export const parseHarJson = (text: string): unknown => { if (frame.members.has(key)) throw new CaptureFormatError( "format", - "HAR JSON contains duplicate object members.", + `${format} JSON contains duplicate object members.`, ); frame.pendingKey = key; }, @@ -103,7 +107,7 @@ export const parseHarJson = (text: string): unknown => { throw new TypeError("JSON visitor emitted an unsupported literal."); }, onError: () => { - throw new CaptureFormatError("format", "HAR is malformed JSON."); + throw new CaptureFormatError("format", `${format} is malformed JSON.`); }, }, { @@ -114,3 +118,18 @@ export const parseHarJson = (text: string): unknown => { ); return result; }; + +/** UTF-8 JSON decoding skips one permitted BOM and rejects malformed bytes. */ +export const decodeCaptureText = ( + bytes: Uint8Array, + format: "HAR" | "qlog", +): string => { + try { + return new TextDecoder("utf-8", { fatal: true }).decode(bytes); + } catch { + throw new CaptureFormatError( + "format", + `${format} is not valid UTF-8 JSON.`, + ); + } +}; diff --git a/src/browser/history/CaptureRelease.ts b/src/browser/history/CaptureRelease.ts index d8af643aa..77f29df51 100644 --- a/src/browser/history/CaptureRelease.ts +++ b/src/browser/history/CaptureRelease.ts @@ -8,19 +8,19 @@ export const HAR_CAPTURE_PROVIDER_IDENTITY = { version: "har-schema@2.0.0;jsonc-parser@3.3.1;lossless-json@4.3.1", } as const; -/** Fixed V8 heap settings used by the owned HAR decoder process. */ -export const HAR_CAPTURE_HEAP_LIMITS = { +/** Fixed V8 heap settings used by the owned Node capture decoder process. */ +export const NODE_CAPTURE_HEAP_LIMITS = { oldGenerationMiB: 192, semiSpaceMiB: 8, } as const; -/** The HAR decoder runs on this executable; Bun accepts but ignores V8 heap flags. */ -export const HAR_CAPTURE_HEAP_ENFORCED = +/** The Node capture decoder runs on this executable; Bun accepts but ignores V8 heap flags. */ +export const NODE_CAPTURE_HEAP_ENFORCED = javaScriptRuntime(process.versions).id === "node"; /** Windows Job Object commit limit replacing the ignored heap flags under Bun; otherwise null. */ -export const HAR_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES = - javaScriptWorkerMemoryLimitBytes(HAR_CAPTURE_HEAP_LIMITS.oldGenerationMiB); +export const NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES = + javaScriptWorkerMemoryLimitBytes(NODE_CAPTURE_HEAP_LIMITS.oldGenerationMiB); /** Native executable profile verified with unchanged mitmproxy raw-state decoding. */ export const MITMPROXY_CAPTURE_PROVIDER_IDENTITY = { @@ -28,3 +28,18 @@ export const MITMPROXY_CAPTURE_PROVIDER_IDENTITY = { name: "REA offline native mitmproxy adapter", version: "12.2.3", } as const; + +/** Native SAZ member decoding with the existing ZIP and XML package profiles. */ +export const SAZ_CAPTURE_PROVIDER_IDENTITY = { + id: "fiddler-saz", + name: "REA Fiddler SAZ capture adapter", + version: "1;zip.js@2.15.0;xmldom@0.9.12", +} as const; + +/** Fixed offline interpretation profile, not an authenticated producer claim. */ +export const QLOG_CAPTURE_PROVIDER_IDENTITY = { + id: "aioquic-qlog-json", + name: "REA aioquic qlog capture adapter", + version: + "aioquic-profile@1.5.0;qlog@0.3;jsonc-parser@3.3.1;lossless-json@4.3.1", +} as const; diff --git a/src/browser/history/HarCapture.test.ts b/src/browser/history/HarCapture.test.ts index 2eb1fe21a..b931fda06 100644 --- a/src/browser/history/HarCapture.test.ts +++ b/src/browser/history/HarCapture.test.ts @@ -1,7 +1,8 @@ import { createHash } from "node:crypto"; import { expect, it } from "vitest"; import { historicalHar } from "../../../tests/fixtures/historicalHar.js"; -import { decodeHarCapture, decodeHarText } from "./HarCapture.js"; +import { decodeCaptureText } from "./CaptureJson.js"; +import { decodeHarCapture } from "./HarCapture.js"; it.each([ { values: [] }, @@ -161,6 +162,19 @@ it("rejects identical duplicate core fields before schema validation", () => { expect(() => decodeHarCapture(text, [])).toThrow("duplicate"); }); +it("names the violated HAR schema constraint without restating capture values", () => { + expect(() => + decodeHarCapture( + JSON.stringify({ + log: { creator: { name: "x", version: "1" }, entries: [] }, + }), + [], + ), + ).toThrow( + "HAR schema validation failed: required (must have required property 'version').", + ); +}); + it("preserves prototype-named producer members through the capture boundary", () => { const text = JSON.stringify(historicalHar()).replace( '"cache":{}', @@ -563,18 +577,21 @@ it("ignores one leading byte-order mark, as HAR 1.2 requires of readers", () => const text = JSON.stringify(historicalHar()); const body = Buffer.from(text, "utf8"); const marked = Buffer.concat([Buffer.from([0xef, 0xbb, 0xbf]), body]); - expect(decodeHarText(marked)).toBe(text); - expect(decodeHarCapture(decodeHarText(marked), [])).toEqual( - decodeHarCapture(decodeHarText(body), []), + expect(decodeCaptureText(marked, "HAR")).toBe(text); + expect(decodeHarCapture(decodeCaptureText(marked, "HAR"), [])).toEqual( + decodeHarCapture(decodeCaptureText(body, "HAR"), []), ); // A second mark is content, not an encoding signature. expect(() => decodeHarCapture( - decodeHarText(Buffer.concat([Buffer.from([0xef, 0xbb, 0xbf]), marked])), + decodeCaptureText( + Buffer.concat([Buffer.from([0xef, 0xbb, 0xbf]), marked]), + "HAR", + ), [], ), ).toThrow("HAR is malformed JSON."); - expect(() => decodeHarText(Buffer.from([0x7b, 0xff, 0x7d]))).toThrow( - "HAR is not valid UTF-8 JSON.", - ); + expect(() => + decodeCaptureText(Buffer.from([0x7b, 0xff, 0x7d]), "HAR"), + ).toThrow("HAR is not valid UTF-8 JSON."); }); diff --git a/src/browser/history/HarCapture.ts b/src/browser/history/HarCapture.ts index f3a83af15..81c5ffb15 100644 --- a/src/browser/history/HarCapture.ts +++ b/src/browser/history/HarCapture.ts @@ -1,7 +1,6 @@ import { createHash } from "node:crypto"; -import { TextDecoder } from "node:util"; import { isSafeNumber, LosslessNumber } from "lossless-json"; -import { parseHarJson } from "./HarJson.js"; +import { parseCaptureJson } from "./CaptureJson.js"; import type { JsonValue } from "../../domain/jsonValue.js"; import { WEB_NETWORK_CAPTURE_LIMITS, @@ -26,25 +25,12 @@ type Projection = Pick< "reported" | "numeric_literals" | "binary_fields" | "redactions" >; -/** - * Decode retained HAR bytes as UTF-8. HAR 1.2 lets writers emit a byte-order - * mark and requires readers to ignore it, so one leading BOM is skipped. - */ -export const decodeHarText = (bytes: Uint8Array): string => { - try { - // ignoreBOM: false (the default) is what skips the mark; true keeps it. - return new TextDecoder("utf-8", { fatal: true }).decode(bytes); - } catch { - throw new CaptureFormatError("format", "HAR is not valid UTF-8 JSON."); - } -}; - /** Decode a retained HAR with exact numeric lexemes and unchanged upstream schema validation. */ export const decodeHarCapture = ( text: string, sensitiveValues: readonly string[], ): DecodedCapture => { - const raw = parseHarJson(text); + const raw = parseCaptureJson(text, "HAR"); const validation = projectHar(raw, [], false); createHarValidator()(validation.reported); if ( @@ -158,7 +144,8 @@ const projectHar = ( (literal) => item.includes(literal) || contentBase64.includes(literal) || - sha256.includes(literal), + sha256.includes(literal) || + String(bytes.length).includes(literal), ); binary_fields.push( hidden diff --git a/src/browser/history/HarCaptureAdapter.ts b/src/browser/history/HarCaptureAdapter.ts deleted file mode 100644 index 9a5204257..000000000 --- a/src/browser/history/HarCaptureAdapter.ts +++ /dev/null @@ -1,33 +0,0 @@ -import { fileURLToPath } from "node:url"; -import type { HistoricalCaptureFormatAdapter } from "./HistoricalCaptureFormatAdapter.js"; -import { - HAR_CAPTURE_PROVIDER_IDENTITY, - HAR_CAPTURE_HEAP_LIMITS, - HAR_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES, -} from "./CaptureRelease.js"; - -/** HAR's parser/schema packages run in one bounded Node heap, without network or target execution. */ -export class HarCaptureAdapter implements HistoricalCaptureFormatAdapter { - readonly format = "har"; - readonly identity = HAR_CAPTURE_PROVIDER_IDENTITY; - async command(requestPath: string): Promise<{ - command: string; - arguments: readonly string[]; - memoryLimitBytes?: number; - }> { - return { - command: process.execPath, - arguments: [ - `--max-old-space-size=${HAR_CAPTURE_HEAP_LIMITS.oldGenerationMiB}`, - `--max-semi-space-size=${HAR_CAPTURE_HEAP_LIMITS.semiSpaceMiB}`, - "--v8-pool-size=1", - "--single-threaded", - fileURLToPath(new URL("./HarCaptureProcess.js", import.meta.url)), - requestPath, - ], - ...(HAR_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES === null - ? {} - : { memoryLimitBytes: HAR_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES }), - }; - } -} diff --git a/src/browser/history/HarSchema.ts b/src/browser/history/HarSchema.ts index 069377f8a..add8c301e 100644 --- a/src/browser/history/HarSchema.ts +++ b/src/browser/history/HarSchema.ts @@ -23,9 +23,13 @@ export const createHarValidator = (): ((value: unknown) => void) => { allowMitmproxyAbsentPostData(value); if (!validate(value)) { const error = validate.errors?.[0]; + // AJV messages name the violated constraint, such as the missing + // property, without restating any captured value. throw new CaptureFormatError( "format", - `HAR schema validation failed: ${error?.keyword ?? "unknown constraint"}.`, + error === undefined + ? "HAR schema validation failed: unknown constraint." + : `HAR schema validation failed: ${error.keyword} (${error.message ?? "no detail"}).`, error?.instancePath ?? "", ); } diff --git a/src/browser/history/HistoricalCaptureDecoder.ts b/src/browser/history/HistoricalCaptureDecoder.ts index 155ba229b..85a91ef76 100644 --- a/src/browser/history/HistoricalCaptureDecoder.ts +++ b/src/browser/history/HistoricalCaptureDecoder.ts @@ -4,9 +4,9 @@ import { captureMemoryFailure, } from "./CaptureFailures.js"; import { - HAR_CAPTURE_HEAP_ENFORCED, - HAR_CAPTURE_HEAP_LIMITS, - HAR_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES, + NODE_CAPTURE_HEAP_ENFORCED, + NODE_CAPTURE_HEAP_LIMITS, + NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES, } from "./CaptureRelease.js"; import { projectAnalysisError } from "../../domain/analysisErrorProjection.js"; import type { HistoricalCaptureFormatAdapter } from "./HistoricalCaptureFormatAdapter.js"; @@ -17,6 +17,7 @@ import { z } from "zod"; import type { ExecutionOptions } from "../../application/AnalysisProvider.js"; import { readStableArtifact } from "../../artifacts/readStableArtifact.js"; import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js"; +import { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js"; import type { AnalysisError } from "../../domain/analysisErrorBase.js"; import { AnalysisCapabilityUnavailableError, @@ -26,6 +27,7 @@ import { } from "../../domain/analysisErrorCore.js"; import { ProviderAdapterError } from "../../domain/providerAdapterError.js"; import { ProviderCleanupError } from "../../domain/providerCleanupError.js"; +import { analysisErrorWithCleanupFailure } from "../../domain/analysisErrorCleanup.js"; import { err, ok, type Result } from "../../domain/result.js"; import { WEB_NETWORK_CAPTURE_LIMITS, @@ -60,18 +62,52 @@ const replySchema = z.discriminatedUnion("ok", [ /** One owned snapshot/process/root lifecycle for replaceable historical format decoders. */ export class HistoricalCaptureDecoder { + readonly #resources: ArtifactResourceScope; + constructor( readonly adapters: readonly HistoricalCaptureFormatAdapter[], readonly environment: Readonly, readonly createRuntime: () => Promise< Pick > = () => PrivateRuntimeRoot.create({ prefix: "rea-web-capture-" }), - ) {} + resources = new ArtifactResourceScope(), + ) { + this.#resources = resources; + } + + /** Retry retained readers and private roots at service shutdown. */ + close(): Promise { + return this.#resources.close(); + } /** Snapshot the selected artifact, decode it offline, and release private credentials and processes. */ async inspect( input: InspectWebNetworkCaptureInput, options?: ExecutionOptions, + ): Promise> { + try { + return await this.#resources.run(() => + this.inspectAdmitted(input, options), + ); + } catch (cause: unknown) { + const error = historicalCaptureFailure( + input, + cause, + "capture-read", + options, + ); + return err( + redactExplicitFailure( + this.withReaderCleanup(error, cause), + input.sensitive_values, + ), + ); + } + } + + private async inspectAdmitted( + input: InspectWebNetworkCaptureInput, + options?: ExecutionOptions, ): Promise> { let runtime: Pick | undefined; let result: Result; @@ -90,6 +126,8 @@ export class HistoricalCaptureDecoder { input.capture_path, WEB_NETWORK_CAPTURE_LIMITS.inputBytes, options?.signal, + undefined, + this.#resources, ); phase = "decoder"; runtime = await this.createRuntime(); @@ -139,9 +177,16 @@ export class HistoricalCaptureDecoder { replyPath, WEB_NETWORK_CAPTURE_LIMITS.outputBytes, options?.signal, + undefined, + this.#resources, ); reply = replySchema.parse(JSON.parse(replyFile.bytes.toString("utf8"))); } catch (cause: unknown) { + if ( + cause instanceof ArtifactReaderFailure && + cause.cleanup !== undefined + ) + throw cause; if (options?.signal?.aborted) throw new AnalysisCancelledError(OPERATION); throw new AnalysisOutputError( @@ -198,34 +243,53 @@ export class HistoricalCaptureDecoder { OPERATION, "Capture decoder changed the selected upstream profile identity.", ); - result = ok({ - ...reply.value, + const common = { artifact: { path: input.capture_path, sha256: snapshot.sha256, bytes: snapshot.bytes.length, }, - format: input.format, - runtime_attribution: "unknown", + + runtime_attribution: "unknown" as const, limitations: [ + ...(adapter.limitations ?? []), "This is retained producer evidence, without live browser transaction IDs, scenario provenance, execution attribution or deployment authenticity. Recorded URLs are never fetched.", "Every record is decoded before selection; no partial success is returned on malformed input or a resource limit.", "Known authentication headers/cookies and explicitly marked literal values are excluded. Unknown extension names do not establish sensitivity.", - `A 32 MiB input, 96 MiB reply and 64-level nesting budget bound complete evidence. ${harMemoryLimitation()}; native decoding uses a 768 MiB Linux address-space limit. Each owned command has a 30-second deadline with independent cleanup.`, + `A 32 MiB input, 96 MiB reply and 64-level nesting budget bound complete evidence. ${nodeMemoryLimitation()}; mitmproxy decoding uses a 768 MiB Linux address-space limit. Each owned command has a 30-second deadline with independent cleanup.`, ], - }); + }; + if (input.format === "qlog") { + if (!("qlog" in reply.value)) + throw new AnalysisOutputError( + OPERATION, + "Capture decoder omitted qlog transport facts.", + ); + result = ok({ ...reply.value, ...common, format: "qlog" }); + } else { + if ("qlog" in reply.value) + throw new AnalysisOutputError( + OPERATION, + "Capture decoder supplied inapplicable qlog transport facts.", + ); + result = ok({ ...reply.value, ...common, format: input.format }); + } } catch (cause: unknown) { + const error = historicalCaptureFailure(input, cause, phase, options); result = err( redactExplicitFailure( - historicalCaptureFailure(input, cause, phase, options), + this.withReaderCleanup(error, cause), input.sensitive_values, ), ); } if (runtime !== undefined) { - try { - await runtime.close(); - } catch (cause: unknown) { + const cleanup = await this.#resources.release({ + kind: "private-runtime-root", + resource: runtime.path, + root: runtime, + }); + if (cleanup.kind === "failed") { return err( redactExplicitFailure( new ProviderCleanupError( @@ -236,9 +300,12 @@ export class HistoricalCaptureDecoder { previous_error: result.ok ? null : projectAnalysisError(result.error), - reason: cause instanceof Error ? cause.message : String(cause), + reason: + cleanup.cause instanceof Error + ? cleanup.cause.message + : String(cleanup.cause), }, - { operation: OPERATION }, + { operation: OPERATION, cause: cleanup.cause }, ), input.sensitive_values, ), @@ -249,15 +316,43 @@ export class HistoricalCaptureDecoder { ? err(new AnalysisCancelledError(OPERATION)) : result; } + + private withReaderCleanup( + primary: AnalysisError, + cause: unknown, + ): AnalysisError { + if ( + !(cause instanceof ArtifactReaderFailure) || + cause.cleanup === undefined + ) + return primary; + if (cause.reason === "unavailable") + return new ProviderCleanupError( + "web-network-capture", + cause.cleanup.resources, + { reason: cause.cleanup.reason }, + { operation: OPERATION, cause }, + ); + const cleanup = new ProviderCleanupError( + "web-network-capture", + cause.cleanup.resources, + { + reason: cause.cleanup.reason, + previous_error: projectAnalysisError(primary), + }, + { operation: OPERATION, cause }, + ); + return analysisErrorWithCleanupFailure(primary, cleanup, OPERATION); + } } -/** The HAR decoder runs on this executable; only Node.js enforces its V8 heap flags. */ -const harMemoryLimitation = (): string => { - if (HAR_CAPTURE_HEAP_ENFORCED) - return `HAR uses a ${String(HAR_CAPTURE_HEAP_LIMITS.oldGenerationMiB)} MiB Node old-generation heap`; - return HAR_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES === null - ? "HAR runs on Bun, which does not enforce REA's V8 heap flags" - : `HAR runs on Bun, which does not enforce REA's V8 heap flags, under a ${String(HAR_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES)}-byte Windows Job Object commit limit`; +/** The Node capture decoder runs on this executable; only Node.js enforces its V8 heap flags. */ +const nodeMemoryLimitation = (): string => { + if (NODE_CAPTURE_HEAP_ENFORCED) + return `HAR/SAZ/qlog use a ${String(NODE_CAPTURE_HEAP_LIMITS.oldGenerationMiB)} MiB Node old-generation heap`; + return NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES === null + ? "HAR/SAZ/qlog run on Bun, which does not enforce REA's V8 heap flags" + : `HAR/SAZ/qlog run on Bun, which does not enforce REA's V8 heap flags, under a ${String(NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES)}-byte Windows Job Object commit limit`; }; const replyFailureReason = (cause: unknown): string => { diff --git a/src/browser/history/HistoricalCaptureFormatAdapter.ts b/src/browser/history/HistoricalCaptureFormatAdapter.ts index 811ea56e2..9660fd579 100644 --- a/src/browser/history/HistoricalCaptureFormatAdapter.ts +++ b/src/browser/history/HistoricalCaptureFormatAdapter.ts @@ -3,6 +3,7 @@ import type { InspectWebNetworkCaptureInput } from "../../domain/webNetworkCaptu /** Format-specific process profile keeps mature upstream code behind a replaceable adapter seam. */ export interface HistoricalCaptureFormatAdapter { readonly format: InspectWebNetworkCaptureInput["format"]; + readonly limitations?: readonly string[]; readonly identity: { readonly id: string; readonly name: string; diff --git a/src/browser/history/NodeCaptureAdapter.test.ts b/src/browser/history/NodeCaptureAdapter.test.ts new file mode 100644 index 000000000..bf2528f5b --- /dev/null +++ b/src/browser/history/NodeCaptureAdapter.test.ts @@ -0,0 +1,18 @@ +import { expect, it, vi } from "vitest"; +import { AnalysisCapabilityUnavailableError } from "../../domain/analysisErrorCore.js"; +import { NodeCaptureAdapter } from "./NodeCaptureAdapter.js"; + +vi.mock("./CaptureRelease.js", async (original) => ({ + ...(await original()), + NODE_CAPTURE_HEAP_ENFORCED: false, + NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES: null, +})); + +it.each(["saz", "qlog"] as const)( + "refuses a %s worker when the runtime has no enforceable memory bound", + async (format) => { + await expect( + new NodeCaptureAdapter(format).command("/owned/request.json"), + ).rejects.toBeInstanceOf(AnalysisCapabilityUnavailableError); + }, +); diff --git a/src/browser/history/NodeCaptureAdapter.ts b/src/browser/history/NodeCaptureAdapter.ts new file mode 100644 index 000000000..8f33430ca --- /dev/null +++ b/src/browser/history/NodeCaptureAdapter.ts @@ -0,0 +1,61 @@ +import { fileURLToPath } from "node:url"; +import type { HistoricalCaptureFormatAdapter } from "./HistoricalCaptureFormatAdapter.js"; +import { + NODE_CAPTURE_HEAP_ENFORCED, + HAR_CAPTURE_PROVIDER_IDENTITY, + SAZ_CAPTURE_PROVIDER_IDENTITY, + QLOG_CAPTURE_PROVIDER_IDENTITY, + NODE_CAPTURE_HEAP_LIMITS, + NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES, +} from "./CaptureRelease.js"; +import { AnalysisCapabilityUnavailableError } from "../../domain/analysisErrorCore.js"; + +/** Package-backed capture decoders share one bounded offline Node process profile. */ +export class NodeCaptureAdapter implements HistoricalCaptureFormatAdapter { + readonly identity; + readonly limitations; + constructor(readonly format: "har" | "saz" | "qlog") { + this.identity = + format === "har" + ? HAR_CAPTURE_PROVIDER_IDENTITY + : format === "saz" + ? SAZ_CAPTURE_PROVIDER_IDENTITY + : QLOG_CAPTURE_PROVIDER_IDENTITY; + this.limitations = + format === "saz" + ? [ + "SAZ ordinals follow first session-member occurrence in ZIP directory order, not chronology or numeric session ID.", + "HTTP fields use byte-preserving Latin-1 text; start-line separators and header delimiters are not projected. Bodies retain stored bytes without transfer/content decoding. Absent members are null, distinct from empty bodies.", + "XML preserves node order and attribute/text values, not its original byte serialization. WebSocket and gRPC members retain opaque message-file bytes, not decoded protocol events.", + "Non-session archive members are inventoried but their content is not decoded.", + ] + : []; + } + async command(requestPath: string) { + if ( + this.format !== "har" && + !NODE_CAPTURE_HEAP_ENFORCED && + NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES === null + ) + throw new AnalysisCapabilityUnavailableError( + this.identity.id, + "inspect_web_network_capture", + `${this.format} decoding requires an enforced decoder memory limit. Run REA with Node.js on this host.`, + ); + return { + command: process.execPath, + arguments: [ + `--max-old-space-size=${NODE_CAPTURE_HEAP_LIMITS.oldGenerationMiB}`, + `--max-semi-space-size=${NODE_CAPTURE_HEAP_LIMITS.semiSpaceMiB}`, + "--v8-pool-size=1", + "--single-threaded", + fileURLToPath(new URL("./NodeCaptureProcess.js", import.meta.url)), + requestPath, + this.format, + ], + ...(NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES === null + ? {} + : { memoryLimitBytes: NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES }), + }; + } +} diff --git a/src/browser/history/HarCaptureProcess.ts b/src/browser/history/NodeCaptureProcess.ts similarity index 53% rename from src/browser/history/HarCaptureProcess.ts rename to src/browser/history/NodeCaptureProcess.ts index 1e4c66c76..ff45a869a 100644 --- a/src/browser/history/HarCaptureProcess.ts +++ b/src/browser/history/NodeCaptureProcess.ts @@ -3,7 +3,6 @@ import { z } from "zod"; import { readStableArtifact } from "../../artifacts/readStableArtifact.js"; import { WEB_NETWORK_CAPTURE_LIMITS } from "../../domain/webNetworkCapture.js"; import { CaptureFormatError } from "./CaptureFormatError.js"; -import { decodeHarCapture, decodeHarText } from "./HarCapture.js"; const requestSchema = z.strictObject({ snapshot_path: z.string(), @@ -11,24 +10,37 @@ const requestSchema = z.strictObject({ sensitive_values: z.array(z.string().min(1)), }); -/** Decode untrusted HAR in an owned, bounded heap rather than the long-lived MCP process. */ +/** Parse untrusted captures in an owned heap and return one complete bounded reply. */ const main = async (): Promise => { + const format = z.enum(["har", "saz", "qlog"]).parse(process.argv[3]); const request = requestSchema.parse( JSON.parse(await readFile(process.argv[2] ?? "", "utf8")), ); let reply: unknown; try { - const snapshot = await readStableArtifact( - request.snapshot_path, - WEB_NETWORK_CAPTURE_LIMITS.inputBytes, - ); - reply = { - ok: true, - value: decodeHarCapture( - decodeHarText(snapshot.bytes), + let value; + if (format !== "saz") { + const { decodeCaptureText } = await import("./CaptureJson.js"); + const decoder = + format === "har" + ? (await import("./HarCapture.js")).decodeHarCapture + : (await import("./QlogCapture.js")).decodeQlogCapture; + const snapshot = await readStableArtifact( + request.snapshot_path, + WEB_NETWORK_CAPTURE_LIMITS.inputBytes, + ); + value = decoder( + decodeCaptureText(snapshot.bytes, format === "har" ? "HAR" : "qlog"), request.sensitive_values, - ), - }; + ); + } else { + const { decodeSazCapture } = await import("./SazCapture.js"); + value = await decodeSazCapture( + request.snapshot_path, + request.sensitive_values, + ); + } + reply = { ok: true, value }; } catch (cause: unknown) { reply = cause instanceof CaptureFormatError @@ -41,7 +53,7 @@ const main = async (): Promise => { : { ok: false, reason: "decoder", - message: `HAR decoder failed unexpectedly (${cause instanceof Error ? cause.name : "unknown exception"}).`, + message: `Capture decoder failed unexpectedly (${cause instanceof Error ? cause.name : "unknown exception"}).`, pointer: "", }; } @@ -50,7 +62,7 @@ const main = async (): Promise => { text = JSON.stringify({ ok: false, reason: "limit", - message: "HAR reply exceeds the complete-evidence output budget.", + message: "Capture reply exceeds the complete-evidence output budget.", pointer: "", }); await writeFile(request.reply_path, text, { flag: "wx", mode: 0o600 }); diff --git a/src/browser/history/QlogCapture.test.ts b/src/browser/history/QlogCapture.test.ts new file mode 100644 index 000000000..bd67d4e80 --- /dev/null +++ b/src/browser/history/QlogCapture.test.ts @@ -0,0 +1,259 @@ +import { expect, it } from "vitest"; +import { decodeQlogCapture } from "./QlogCapture.js"; +import { decodeCaptureText } from "./CaptureJson.js"; +import { CaptureFormatError } from "./CaptureFormatError.js"; + +// Synthetic format inputs exercise distinct parser semantics, not producer provenance. +const event = (frames: unknown[], name = "transport:packet_sent") => ({ + name, + time: 1791819895000.25, + data: { header: { packet_type: "1RTT", packet_number: 5 }, frames }, +}); +const frame = (offset: number, length: number, fin = false) => ({ + frame_type: "stream", + stream_id: 0, + offset, + length, + fin, +}); +const capture = (events: unknown[], extra: Record = {}) => + JSON.stringify({ + qlog_format: "JSON", + qlog_version: "0.3", + ...extra, + traces: [ + { + common_fields: { ODCID: "same" }, + vantage_point: { name: "aioquic", type: "server" }, + events, + }, + ], + }); +const facts = (text: string, sensitive: string[] = []) => { + const decoded = decodeQlogCapture(text, sensitive); + if (!("qlog" in decoded)) throw new Error("qlog facet required"); + return { decoded, traces: decoded.qlog.traces }; +}; + +it("derives interval unions per trace/direction while retaining event source order and extensions", () => { + const original = JSON.parse( + capture([ + event([frame(10, 10)]), + event([frame(0, 15), frame(30, 5, true)]), + event([frame(0, 4)], "transport:packet_received"), + { + name: "recovery:packet_lost", + time: 1, + data: { packet_number: 5, type: "1RTT" }, + }, + ]), + ); + original.traces.push({ + ...original.traces[0], + events: [event([frame(0, 3)])], + }); + Object.defineProperty(original, "__proto__", { + enumerable: true, + value: { ordinary: true }, + }); + original.extra = "https://example.test/path?token=ordinary"; + const { decoded, traces } = facts(JSON.stringify(original)); + expect(traces[0]?.streams[0]).toMatchObject({ + stream_id: "0", + direction: "sent", + unique_bytes: "25", + frame_length_sum: "30", + repeated_bytes: "5", + covered_ranges: [ + { start: "0", end: "20" }, + { start: "30", end: "35" }, + ], + reported_final_sizes: ["35"], + }); + expect(traces[0]?.streams[1]).toMatchObject({ + direction: "received", + unique_bytes: "4", + }); + expect(traces[1]?.streams[0]).toMatchObject({ unique_bytes: "3" }); + expect(traces[0]?.loss_declarations[0]).toMatchObject({ + confidence: "observed", + packet_number: { state: "reported", value: "5" }, + packet_number_space: "application", + }); + expect(decoded.records.at(-1)?.location).toEqual({ + kind: "json-pointer", + pointer: "/traces/1/events/0", + }); + expect(decoded.container.reported).toMatchObject({ + extra: original.extra, + ["__proto__"]: { ordinary: true }, + }); + expect(Reflect.get(Object.prototype, "ordinary")).toBeUndefined(); +}); + +it("keeps exact 62-bit integers and numeric lexemes without allocating exponent expansion", () => { + const text = capture([event([frame(0, 0, true)])]) + .replace('"stream_id":0', '"stream_id":4611686018427387903') + .replace('"offset":0', '"offset":9007199254740993') + .replace('"length":0', '"length":1e3'); + const { decoded, traces } = facts(text); + expect(traces[0]?.streams[0]).toMatchObject({ + stream_id: "4611686018427387903", + unique_bytes: "1000", + reported_final_sizes: ["9007199254741993"], + }); + expect(decoded.records[0]?.numeric_literals).toContainEqual({ + pointer: "/data/frames/0/length", + producer_type: "json-number", + literal: "1e3", + }); + expect(decoded.records[0]?.reported).toMatchObject({ + data: { frames: [{ stream_id: null, offset: null, length: 1000 }] }, + }); +}); + +it("keeps incomplete fields unknown and FIN without bytes as metadata", () => { + const { traces } = facts( + capture([ + event([{ frame_type: "stream", stream_id: 0, length: 5 }]), + event([frame(10, 0, true)]), + { name: "recovery:packet_lost", time: 1, data: {} }, + { + name: "extension:custom", + time: 0, + data: { body: "ordinary evidence" }, + }, + ]), + ); + expect(traces[0]?.incomplete_fact_records).toEqual([0]); + expect(traces[0]?.streams[0]).toMatchObject({ + unique_bytes: "0", + covered_ranges: [], + reported_final_sizes: ["10"], + }); + expect(traces[0]?.loss_declarations[0]?.packet_number.state).toBe("unknown"); +}); + +it("excludes structural auth material and explicit normalized values without guessing extension secrecy", () => { + const text = capture([ + { + name: "http:frame_parsed", + time: 1, + data: { + frame: { + frame_type: "headers", + headers: [ + { name: "authorization", value: "Bearer credential" }, + { name: ":path", value: "/?token=ordinary" }, + { name: "location", value: "https://user:pass@example.test/a" }, + ], + }, + }, + }, + { + name: "transport:parameters_set", + time: 2, + data: { stateless_reset_token: "reset-credential" }, + }, + event([ + { frame_type: "new_connection_id", reset_token: "reset-frame" }, + { frame_type: "new_token", token: "new-token" }, + frame(0, 100), + ]), + { + name: "extension:custom", + time: 3, + data: { + secret: "caller-selected ordinary value", + reset_token: "extension ordinary", + }, + }, + ]).replace('"length":100', '"length":1e2'); + const { decoded, traces } = facts(text, ["100"]); + const serialized = JSON.stringify(decoded); + for (const hidden of [ + "Bearer credential", + "reset-credential", + "reset-frame", + "new-token", + "100", + "user:pass", + ]) + expect(serialized).not.toContain(hidden); + expect(serialized).toContain("token=ordinary"); + expect(serialized).toContain("caller-selected ordinary value"); + expect(traces[0]?.streams).toEqual([]); + expect(traces[0]?.incomplete_fact_records).toContain(2); +}); + +it("hides marked ancestor keys and safe derived sums while still validating hidden subtrees", () => { + const text = capture([event([frame(0, 3), frame(3, 4)])]); + expect(facts(text, ["7"]).traces[0]?.streams).toEqual([]); + const { decoded, traces } = facts(text, ["events"]); + expect(decoded.records[0]?.reported).toBe(null); + expect(decoded.records[0]?.location.kind).toBe("unknown"); + expect(traces[0]?.location.kind).toBe("unknown"); + expect(traces[0]?.streams).toEqual([]); + expect(() => + facts(text.replace('"length":3', '"length":-3'), ["events"]), + ).toThrow(CaptureFormatError); +}); + +it.each([ + "-1", + "0.5", + "4611686018427387904", + "1e1000000", + "1e-1000000", + '"3"', + "true", +])("rejects malformed or out-of-range STREAM integer %s", (literal) => { + expect(() => + facts( + capture([event([frame(0, 3)])]).replace( + '"length":3', + `"length":${literal}`, + ), + ), + ).toThrow(CaptureFormatError); +}); + +it.each([ + '{"qlog_format":"JSON","qlog_format":"JSON","qlog_version":"0.3","traces":[]}', + capture([]).slice(0, -1), + capture([]) + " {}", + capture([{ name: "invalid", time: "1", data: {} }]), + capture([[1, "transport", "packet_sent", {}]]), + capture([event([{ ...frame(0, 1), fin: "true" }])]), + capture([event([{ ...frame(0, 1), offset: 4611686018427388000 }])]), +])( + "rejects malformed JSON/profile events with no partial success: %s", + (text) => { + expect(() => facts(text)).toThrow(CaptureFormatError); + }, +); + +it("rejects other qlog containers/versions and excessive nesting, accepts empty profiles and UTF-8 BOM", () => { + for (const extra of [{ qlog_format: "JSON-SEQ" }, { qlog_version: "0.4" }]) + expect(() => facts(capture([], extra))).toThrow("Only the aioquic"); + expect(() => + facts( + capture([], { + extension: JSON.parse("[".repeat(65) + "0" + "]".repeat(65)), + }), + ), + ).toThrow("nesting budget"); + expect( + facts('{"qlog_format":"JSON","qlog_version":"0.3","traces":[]}').traces, + ).toEqual([]); + const raw = Buffer.from(capture([])); + expect( + facts( + decodeCaptureText( + Buffer.concat([Buffer.from([0xef, 0xbb, 0xbf]), raw]), + "qlog", + ), + ).decoded.total_records, + ).toBe(0); + expect(() => decodeCaptureText(Buffer.from([0xff]), "qlog")).toThrow("UTF-8"); +}); diff --git a/src/browser/history/QlogCapture.ts b/src/browser/history/QlogCapture.ts new file mode 100644 index 000000000..2e8a1f790 --- /dev/null +++ b/src/browser/history/QlogCapture.ts @@ -0,0 +1,503 @@ +import { isSafeNumber, LosslessNumber } from "lossless-json"; +import type { JsonValue } from "../../domain/jsonValue.js"; +import type { QlogCaptureFacts } from "../../domain/qlogCapture.js"; +import { + webNetworkCapturePayloadSchema, + type WebNetworkCaptureRecord, +} from "../../domain/webNetworkCapture.js"; +import type { z } from "zod"; +import { parseCaptureJson } from "./CaptureJson.js"; +import { CaptureFormatError } from "./CaptureFormatError.js"; +import { + CaptureRedaction, + captureObject, + capturePointer, +} from "./CaptureRedaction.js"; +import { QLOG_CAPTURE_PROVIDER_IDENTITY } from "./CaptureRelease.js"; + +const MAX_VARINT = (1n << 62n) - 1n; +const credentials = new Set([ + "authorization", + "proxy-authorization", + "cookie", + "set-cookie", +]); +type Projection = Pick< + WebNetworkCaptureRecord, + "reported" | "numeric_literals" | "binary_fields" | "redactions" +>; +type Trace = QlogCaptureFacts["traces"][number]; +type Stream = Trace["streams"][number]; + +/** Inspect the selected aioquic JSON profile without authenticating its producer or interpreting other qlog layouts. */ +export const decodeQlogCapture = ( + text: string, + sensitiveValues: readonly string[], +): z.output => { + const raw = parseCaptureJson(text, "qlog"); + if (!object(raw) || raw.qlog_format !== "JSON" || raw.qlog_version !== "0.3") + throw new CaptureFormatError( + "unsupported", + "Only the aioquic 1.5.0 JSON qlog 0.3 profile is supported.", + ); + if (!Array.isArray(raw.traces)) + return invalid("qlog must contain a traces array.", "/traces"); + const traces = raw.traces.map(validateTrace); + const container = project( + { ...raw, traces: traces.map((trace) => ({ ...trace, events: null })) }, + sensitiveValues, + ); + const records: WebNetworkCaptureRecord[] = []; + const facts: Trace[] = []; + const excluded = (value: string): boolean => + sensitiveValues.some((literal) => value.includes(literal)); + const ancestorsHidden = excluded("traces") || excluded("events"); + for (const [traceIndex, trace] of traces.entries()) { + const tracePointer = `/traces/${traceIndex}`; + const fact = createTraceFact( + trace, + traceIndex, + sensitiveValues, + ancestorsHidden, + ); + const streams = new Map(); + for (const [eventIndex, event] of trace.events.entries()) { + const ordinal = records.length; + const projection: Projection = ancestorsHidden + ? { + reported: null, + numeric_literals: [], + binary_fields: [], + redactions: [{ pointer: "", reason: "explicit-sensitive-value" }], + } + : project(event, sensitiveValues); + records.push({ + ordinal, + location: ancestorsHidden + ? { kind: "unknown", reason: "explicit-sensitive-value" } + : { + kind: "json-pointer", + pointer: `${tracePointer}/events/${eventIndex}`, + }, + ...projection, + limitations: [ + "Record ordinal is derived from trace/event source traversal, not global time order. Qlog lengths and FIN do not expose original HTTP body bytes or establish network completeness.", + ], + }); + collectTransportFacts(projection, ordinal, { fact, streams, excluded }); + } + for (const stream of streams.values()) { + summarizeStream(stream); + const numericValues = [ + stream.frame_length_sum, + stream.unique_bytes, + stream.repeated_bytes, + ...stream.reported_final_sizes, + ...stream.covered_ranges.flatMap((r) => [r.start, r.end]), + ]; + if (numericValues.some(excluded)) + fact.incomplete_fact_records.push( + ...stream.contributions.map((c) => c.record_ordinal), + ); + else fact.streams.push(stream); + } + fact.incomplete_fact_records = [...new Set(fact.incomplete_fact_records)]; + facts.push(fact); + } + return { + decoder: { ...QLOG_CAPTURE_PROVIDER_IDENTITY }, + container: { + reported: container.reported, + numeric_literals: container.numeric_literals, + redactions: container.redactions, + records_pointer: null, + }, + total_records: records.length, + records, + qlog: { + profile: "aioquic-1.5.0-json-0.3", + scope: "all-recorded-events-before-selection", + body_bytes: "not-reported", + clock: { + unit: "milliseconds", + domain: "unix-epoch", + basis: "selected-aioquic-profile-not-producer-authentication", + }, + traces: facts, + }, + }; +}; + +const createTraceFact = ( + trace: Record & { events: unknown[] }, + traceIndex: number, + sensitiveValues: readonly string[], + ancestorsHidden: boolean, +): Trace => { + const tracePointer = `/traces/${traceIndex}`; + const metadata = project( + { ...trace, events: null }, + sensitiveValues, + ).reported; + const meta = !ancestorsHidden && object(metadata) ? metadata : null; + const common = + meta !== null && object(meta.common_fields) ? meta.common_fields : null; + const vantage = + meta !== null && object(meta.vantage_point) + ? meta.vantage_point.type + : null; + return { + trace_index: traceIndex, + location: ancestorsHidden + ? { kind: "unknown", reason: "explicit-sensitive-value" } + : { kind: "json-pointer", pointer: tracePointer }, + event_count: trace.events.length, + odcid: typeof common?.ODCID === "string" ? common.ODCID : null, + vantage_point: + vantage === "client" || vantage === "server" ? vantage : null, + loss_declarations: [], + streams: [], + incomplete_fact_records: [], + }; +}; + +const validateTrace = ( + trace: unknown, + index: number, +): Record & { events: unknown[] } => { + const pointer = `/traces/${index}`; + if (!object(trace) || !Array.isArray(trace.events)) + return invalid("qlog trace must contain an events array.", pointer); + for (const [eventIndex, event] of trace.events.entries()) { + const eventPointer = `${pointer}/events/${eventIndex}`; + if ( + !object(event) || + typeof event.name !== "string" || + !(event.time instanceof LosslessNumber) || + !Number.isFinite(Number(event.time.value)) || + !object(event.data) + ) + invalid( + "qlog events must be objects with name, finite numeric time and object data.", + eventPointer, + ); + // Validate interpretable fields on ORIGINAL values even when a declaration + // subsequently hides them. Missing fields remain meaningful unknowns. + if (event.name === "recovery:packet_lost") + unsigned(event.data.packet_number, `${eventPointer}/data/packet_number`); + if (packetEvent(event.name) && Array.isArray(event.data.frames)) { + for (const [frameIndex, frame] of event.data.frames.entries()) { + if (object(frame) && frame.frame_type === "stream") { + validateStreamFrame( + frame, + `${eventPointer}/data/frames/${frameIndex}`, + ); + } + } + } + } + return { ...trace, events: trace.events }; +}; + +const validateStreamFrame = ( + frame: Record, + p: string, +): void => { + unsigned(frame.stream_id, `${p}/stream_id`); + const offset = unsigned(frame.offset, `${p}/offset`); + const length = unsigned(frame.length, `${p}/length`); + if (offset !== null && length !== null && offset + length > MAX_VARINT) + invalid("qlog stream final offset exceeds the QUIC varint range.", p); + if ( + frame.fin !== undefined && + frame.fin !== null && + typeof frame.fin !== "boolean" + ) + invalid("qlog STREAM fin must be boolean when reported.", `${p}/fin`); +}; + +const collectTransportFacts = ( + projection: Projection, + ordinal: number, + state: { + fact: Trace; + streams: Map; + excluded: (value: string) => boolean; + }, +): void => { + const { fact, streams, excluded } = state; + if (!object(projection.reported) || !object(projection.reported.data)) { + fact.incomplete_fact_records.push(ordinal); + return; + } + const visible = projection.reported; + const data = visible.data; + if (!object(data)) return; + const literals = new Map( + projection.numeric_literals.map((n) => [n.pointer, n.literal]), + ); + const read = (value: unknown, pointer: string): bigint | null => { + const literal = literals.get(pointer); + const number = unsigned( + literal === undefined ? value : new LosslessNumber(literal), + pointer, + ); + return number !== null && excluded(number.toString()) ? null : number; + }; + if (visible.name === "recovery:packet_lost") { + fact.loss_declarations.push(lossDeclaration(data, ordinal, read)); + } + if (!packetEvent(visible.name)) return; + if (!Array.isArray(data.frames)) { + fact.incomplete_fact_records.push(ordinal); + return; + } + for (const [frameIndex, frame] of data.frames.entries()) { + if ( + !object(frame) || + frame.frame_type === undefined || + frame.frame_type === null + ) { + fact.incomplete_fact_records.push(ordinal); + continue; + } + if (frame.frame_type !== "stream") continue; + const p = `/data/frames/${frameIndex}`; + const id = read(frame.stream_id, `${p}/stream_id`), + offset = read(frame.offset, `${p}/offset`), + length = read(frame.length, `${p}/length`); + if (id === null || offset === null || length === null) { + fact.incomplete_fact_records.push(ordinal); + continue; + } + const direction = + visible.name === "transport:packet_sent" ? "sent" : "received"; + const key = `${direction}/${id}`; + let stream = streams.get(key); + if (stream === undefined) { + stream = { + stream_id: id.toString(), + direction, + confidence: "derived", + scope: "observed-stream-frame-ranges", + contributions: [], + covered_ranges: [], + unique_bytes: "0", + frame_length_sum: "0", + repeated_bytes: "0", + reported_final_sizes: [], + }; + streams.set(key, stream); + } + stream.contributions.push({ + record_ordinal: ordinal, + frame_index: frameIndex, + offset: offset.toString(), + length: length.toString(), + fin: typeof frame.fin === "boolean" ? frame.fin : null, + }); + } +}; + +const lossDeclaration = ( + data: Record, + ordinal: number, + read: (value: unknown, pointer: string) => bigint | null, +): Trace["loss_declarations"][number] => { + const pn = read(data.packet_number, "/data/packet_number"); + const type = typeof data.type === "string" ? data.type : null; + return { + record_ordinal: ordinal, + confidence: "observed", + packet_number: + pn === null + ? { + state: "unknown", + reason: "omitted-null-or-redacted", + } + : { state: "reported", value: pn.toString() }, + packet_type: type, + packet_number_space: + type === "initial" + ? "initial" + : type === "handshake" + ? "handshake" + : type === "0RTT" || type === "1RTT" + ? "application" + : null, + }; +}; + +const summarizeStream = (stream: Stream): void => { + const spans = stream.contributions + .map( + (c) => [BigInt(c.offset), BigInt(c.offset) + BigInt(c.length)] as const, + ) + .sort((a, b) => (a[0] < b[0] ? -1 : a[0] > b[0] ? 1 : 0)); + const merged: { start: bigint; end: bigint }[] = []; + for (const [start, end] of spans) { + if (start === end) continue; + const last = merged.at(-1); + if (last !== undefined && start <= last.end) + last.end = end > last.end ? end : last.end; + else merged.push({ start, end }); + } + const sum = stream.contributions.reduce((s, c) => s + BigInt(c.length), 0n); + const unique = merged.reduce((s, r) => s + r.end - r.start, 0n); + stream.covered_ranges = merged.map((r) => ({ + start: r.start.toString(), + end: r.end.toString(), + })); + stream.frame_length_sum = sum.toString(); + stream.unique_bytes = unique.toString(); + stream.repeated_bytes = (sum - unique).toString(); + stream.reported_final_sizes = [ + ...new Set( + stream.contributions + .filter((c) => c.fin === true) + .map((c) => (BigInt(c.offset) + BigInt(c.length)).toString()), + ), + ]; +}; + +const packetEvent = (name: unknown): boolean => + name === "transport:packet_sent" || name === "transport:packet_received"; +const object = (value: unknown): value is Record => + captureObject(value) && !(value instanceof LosslessNumber); +const invalid = (message: string, pointer: string): never => { + throw new CaptureFormatError("format", message, pointer); +}; + +/** Canonicalize JSON integer notation exactly, checking expansion before allocation. */ +const unsigned = (value: unknown, pointer: string): bigint | null => { + if (value === undefined || value === null) return null; + const literal = + value instanceof LosslessNumber + ? value.value + : typeof value === "number" + ? String(value) + : ""; + const m = /^(-?)([0-9]+)(?:\.([0-9]+))?(?:[eE]([+-]?[0-9]+))?$/u.exec( + literal, + ); + if (m === null) + return invalid("qlog QUIC integer must be a JSON number.", pointer); + const digits = `${m[2]}${m[3] ?? ""}`.replace(/^0+/u, ""); + if (digits === "") return 0n; + const exponent = m[4] === undefined ? 0 : Number(m[4]); + const shift = exponent - (m[3]?.length ?? 0); + const count = digits.length + shift; + if (m[1] === "-" || !Number.isSafeInteger(shift) || count > 19 || count <= 0) + return invalid( + "qlog QUIC integer is negative, fractional or outside the 62-bit range.", + pointer, + ); + if (shift < 0 && /[1-9]/u.test(digits.slice(count))) + return invalid("qlog QUIC integer is fractional.", pointer); + const canonical = + shift < 0 ? digits.slice(0, count) : digits + "0".repeat(shift); + const result = BigInt(canonical); + if (result > MAX_VARINT) + return invalid("qlog QUIC integer exceeds the 62-bit range.", pointer); + return result; +}; + +/** Preserve JSON values/lexemes and structurally exclude only transport credentials and declared values. */ +const project = ( + value: unknown, + sensitiveValues: readonly string[], +): Projection => { + const redactor = new CaptureRedaction(sensitiveValues); + const numeric_literals: Projection["numeric_literals"] = []; + const eventName = object(value) ? value.name : null; + const visit = ( + item: unknown, + pointer: string, + parent?: Record, + ): JsonValue => { + const header = + (eventName === "http:frame_created" || + eventName === "http:frame_parsed") && + /^\/data\/frame\/headers\/\d+\/value$/u.test(pointer) && + typeof parent?.name === "string"; + if ( + (header && credentials.has(String(parent?.name).toLowerCase())) || + qlogToken(eventName, pointer, parent) + ) + return redactor.credential(pointer); + if (item instanceof LosslessNumber) { + if ( + redactor.text(item.value, pointer) === null || + (isSafeNumber(item.value) && + redactor.text(String(Number(item.value)), pointer) === null) + ) + return null; + numeric_literals.push({ + pointer, + producer_type: "json-number", + literal: item.value, + }); + return isSafeNumber(item.value) ? Number(item.value) : null; + } + if (typeof item === "string") + return redactor.text( + item, + pointer, + header && + ["location", "referer", "origin"].includes( + String(parent?.name).toLowerCase(), + ), + ); + if (item === null || typeof item === "boolean") return item; + if (Array.isArray(item)) + return item.map((child, index) => + visit(child, capturePointer(pointer, String(index))), + ); + if (object(item)) + return Object.fromEntries( + Object.entries(item).flatMap(([key, child]) => { + if (sensitiveValues.some((literal) => key.includes(literal))) { + redactor.redactions.push({ + pointer, + reason: "explicit-sensitive-value", + scope: "property-name", + }); + return []; + } + return [[key, visit(child, capturePointer(pointer, key), item)]]; + }), + ); + return invalid("qlog parser returned a non-JSON value.", pointer); + }; + return { + reported: visit(value, ""), + numeric_literals, + binary_fields: [], + redactions: redactor.redactions, + }; +}; + +const qlogToken = ( + eventName: unknown, + pointer: string, + parent?: Record, +): boolean => { + if ( + packetEvent(eventName) && + /^\/data\/frames\/\d+\/reset_token$/u.test(pointer) && + parent?.frame_type === "new_connection_id" + ) + return true; + if ( + packetEvent(eventName) && + /^\/data\/frames\/\d+\/token$/u.test(pointer) && + parent?.frame_type === "new_token" + ) + return true; + if ( + eventName === "transport:parameters_set" && + pointer === "/data/stateless_reset_token" + ) + return true; + return false; +}; diff --git a/src/browser/history/SazCapture.ts b/src/browser/history/SazCapture.ts new file mode 100644 index 000000000..0d597eb5a --- /dev/null +++ b/src/browser/history/SazCapture.ts @@ -0,0 +1,174 @@ +import { ZipArtifactReader } from "../../artifacts/ZipArtifactReader.js"; +import { + ArtifactReaderFailure, + type ArtifactEntry, +} from "../../artifacts/ArtifactReader.js"; +import { + WEB_NETWORK_CAPTURE_LIMITS, + type WebNetworkCaptureRecord, +} from "../../domain/webNetworkCapture.js"; +import { SAZ_CAPTURE_PROVIDER_IDENTITY } from "./CaptureRelease.js"; +import { CaptureFormatError } from "./CaptureFormatError.js"; +import { decodeSazHttp } from "./SazHttp.js"; +import { decodeSazMetadata } from "./SazMetadata.js"; +import { projectSazRecord, type SazValue } from "./SazProjection.js"; + +/** Decode native SAZ sessions without extracting files, running Fiddler or converting to HAR. */ +export const decodeSazCapture = async ( + snapshotPath: string, + sensitiveValues: readonly string[], +) => { + const reader = new ZipArtifactReader( + snapshotPath, + "zip", + WEB_NETWORK_CAPTURE_LIMITS.inputBytes, + ); + try { + const groups = new Map(); + const inventory: SazValue[] = []; + const paths = new Set(); + let declaredBytes = 0; + for await (const entry of reader.entries()) { + if (paths.has(entry.path)) + throw new CaptureFormatError( + "format", + "SAZ contains duplicate archive member names.", + ); + paths.add(entry.path); + if (entry.encrypted) + throw new CaptureFormatError( + "unsupported", + "Encrypted SAZ archives are unsupported; export an unencrypted copy from Fiddler Everywhere.", + ); + if (entry.kind !== "file" && entry.kind !== "directory") + throw new CaptureFormatError( + "unsupported", + "SAZ archive links are unsupported.", + ); + if (entry.declaredSize !== null) { + if (!Number.isSafeInteger(entry.declaredSize) || entry.declaredSize < 0) + throw new CaptureFormatError("format", "SAZ member size is invalid."); + declaredBytes += entry.declaredSize; + } + if (declaredBytes > WEB_NETWORK_CAPTURE_LIMITS.inputBytes) + throw new CaptureFormatError( + "limit", + "SAZ expanded members exceed the 32 MiB complete-evidence budget.", + ); + inventory.push({ + path: entry.path, + kind: entry.kind, + bytes: entry.declaredSize, + }); + if (entry.kind !== "file") continue; + const match = /^raw\/(\d+)_(c\.txt|s\.txt|m\.xml|w\.txt|g\.txt)$/.exec( + entry.path, + ); + if (match?.[1] !== undefined) { + const group = groups.get(match[1]) ?? []; + group.push(entry); + groups.set(match[1], group); + } else if (entry.path.startsWith("raw/")) { + throw new CaptureFormatError( + "unsupported", + "SAZ contains an unrecognized raw session member.", + ); + } + } + if (groups.size === 0) + throw new CaptureFormatError( + "format", + "Archive contains no SAZ session members.", + ); + const records: WebNetworkCaptureRecord[] = []; + for (const [session_id, entries] of groups) { + const source: Record = { + session_id, + request: null, + response: null, + metadata: null, + websocket: null, + grpc: null, + }; + for (const entry of entries) { + const stream = await reader.open(entry); + const chunks: Buffer[] = []; + let length = 0; + for await (const chunk of stream) { + if (!Buffer.isBuffer(chunk)) + throw new CaptureFormatError( + "format", + "SAZ member stream returned non-byte data.", + ); + length += chunk.length; + if (length > (entry.declaredSize ?? 0)) + throw new CaptureFormatError( + "format", + "SAZ member exceeds its declared size.", + ); + chunks.push(chunk); + } + if (length !== entry.declaredSize) + throw new CaptureFormatError( + "format", + "SAZ member does not match its declared size.", + ); + const bytes = Buffer.concat(chunks, length); + if (entry.path.endsWith("_c.txt")) + source.request = { + member: entry.path, + ...decodeSazHttp(bytes, true), + }; + else if (entry.path.endsWith("_s.txt")) + source.response = { + member: entry.path, + ...decodeSazHttp(bytes, false), + }; + else if (entry.path.endsWith("_m.xml")) + source.metadata = { + member: entry.path, + document: decodeSazMetadata(bytes), + }; + else + source[entry.path.endsWith("_w.txt") ? "websocket" : "grpc"] = { + member: entry.path, + messages: bytes, + }; + } + records.push({ + ordinal: records.length, + location: { + kind: "archive-members", + members: entries.map((entry) => entry.path), + }, + ...projectSazRecord(source, sensitiveValues), + limitations: [], + }); + } + const container = projectSazRecord({ members: inventory }, sensitiveValues); + return { + decoder: { ...SAZ_CAPTURE_PROVIDER_IDENTITY }, + container: { + reported: container.reported, + numeric_literals: container.numeric_literals, + redactions: container.redactions, + records_pointer: null, + }, + total_records: records.length, + records, + }; + } catch (cause: unknown) { + if (cause instanceof CaptureFormatError) throw cause; + if (cause instanceof ArtifactReaderFailure && cause.reason === "limit") + throw new CaptureFormatError( + "limit", + "SAZ archive exceeds the ZIP reader's resource budget.", + ); + throw new CaptureFormatError( + "format", + "SAZ archive or session data is malformed or fails ZIP integrity validation.", + ); + } finally { + await reader.close(); + } +}; diff --git a/src/browser/history/SazHttp.ts b/src/browser/history/SazHttp.ts new file mode 100644 index 000000000..246dd6164 --- /dev/null +++ b/src/browser/history/SazHttp.ts @@ -0,0 +1,72 @@ +import { CaptureFormatError } from "./CaptureFormatError.js"; + +/** Preserve ordered HTTP fields and the stored body; never dechunk, decompress or fetch. */ +export const decodeSazHttp = (bytes: Buffer, request: boolean) => { + const crlfSplit = bytes.indexOf("\r\n\r\n"); + // Once a CRLF separator is found, only an earlier LF separator can win. + // Avoid searching an otherwise unrelated large body a second time. + const lfSplit = bytes + .subarray(0, crlfSplit < 0 ? bytes.length : crlfSplit) + .indexOf("\n\n"); + // The first separator ends the headers. Later separators belong to the + // stored body, which may use different line endings or contain binary data. + const split = + crlfSplit >= 0 && (lfSplit < 0 || crlfSplit < lfSplit) ? crlfSplit : -1; + const boundary = split < 0 ? lfSplit : split; + if (boundary < 0) + throw new CaptureFormatError( + "format", + "SAZ HTTP message has no header/body separator.", + ); + const lines = bytes + .subarray(0, boundary) + .toString("latin1") + .split(split < 0 ? "\n" : "\r\n"); + const first = lines.shift() ?? ""; + const fields = request + ? /^(\S+) +(\S+) +(HTTP\/\d+(?:\.\d+)?)$/.exec(first) + : /^(HTTP\/\d+(?:\.\d+)?) +(\d{3})(?: (.*))?$/.exec(first); + if (fields === null) + throw new CaptureFormatError("format", "SAZ HTTP start line is malformed."); + const headers: { name: string; value: string }[] = []; + for (const line of lines) { + if (/^[ \t]/.test(line)) { + const previous = headers.at(-1); + if (previous === undefined) + throw new CaptureFormatError( + "format", + "SAZ HTTP header continuation has no field.", + ); + previous.value += `${split < 0 ? "\n" : "\r\n"}${line}`; + continue; + } + const colon = line.indexOf(":"); + // HTTP/2 pseudo-headers use the second colon as the delimiter. + const delimiter = colon === 0 ? line.indexOf(":", 1) : colon; + if (delimiter < 1) + throw new CaptureFormatError("format", "SAZ HTTP header is malformed."); + const name = line.slice(0, delimiter); + if (!/^:?[!#$%&'*+.^_`|~0-9A-Za-z-]+$/.test(name)) + throw new CaptureFormatError( + "format", + "SAZ HTTP header name is malformed.", + ); + headers.push({ name, value: line.slice(delimiter + 1) }); + } + const start = request + ? { + method: fields[1] ?? "", + url: fields[2] ?? "", + http_version: fields[3] ?? "", + } + : { + http_version: fields[1] ?? "", + status: fields[2] ?? "", + reason: fields[3] ?? "", + }; + return { + ...start, + headers, + body: bytes.subarray(boundary + (split < 0 ? 2 : 4)), + }; +}; diff --git a/src/browser/history/SazMetadata.ts b/src/browser/history/SazMetadata.ts new file mode 100644 index 000000000..6e311f8e6 --- /dev/null +++ b/src/browser/history/SazMetadata.ts @@ -0,0 +1,83 @@ +import { DOMParser, ParseError, type Node } from "@xmldom/xmldom"; +import type { JsonValue } from "../../domain/jsonValue.js"; +import { WEB_NETWORK_CAPTURE_LIMITS } from "../../domain/webNetworkCapture.js"; +import { CaptureFormatError } from "./CaptureFormatError.js"; + +/** Preserve XML node order, attribute names and text without interpreting producer-specific flags. */ +export const decodeSazMetadata = (bytes: Buffer): JsonValue => { + let xml: string; + try { + const encoding = + bytes[0] === 0xff && bytes[1] === 0xfe + ? "utf-16le" + : bytes[0] === 0xfe && bytes[1] === 0xff + ? "utf-16be" + : "utf-8"; + xml = new TextDecoder(encoding, { fatal: true }).decode(bytes); + } catch { + throw new CaptureFormatError( + "unsupported", + "SAZ metadata must use UTF-8 or BOM-marked UTF-16.", + ); + } + let parseFailed = false; + let document: ReturnType; + try { + document = new DOMParser({ + normalizeLineEndings: (source) => source, + onError: () => { + parseFailed = true; + }, + }).parseFromString(xml, "application/xml"); + } catch (cause: unknown) { + if (!(cause instanceof ParseError)) throw cause; + throw new CaptureFormatError( + "format", + "SAZ metadata is not well-formed XML.", + ); + } + // The pinned parser does not resolve external or DTD-defined entities. + // Inspect its parsed node; text inside comments/CDATA is ordinary evidence. + if (document.doctype !== null) + throw new CaptureFormatError( + "unsupported", + "SAZ metadata DTDs are unsupported.", + ); + if (parseFailed) + throw new CaptureFormatError( + "format", + "SAZ metadata is not well-formed XML.", + ); + if (document.documentElement?.tagName !== "Session") + throw new CaptureFormatError( + "format", + "SAZ metadata must have a Session document element.", + ); + const visit = (node: Node, depth: number): JsonValue => { + if (depth > WEB_NETWORK_CAPTURE_LIMITS.depth) + throw new CaptureFormatError( + "input-limit", + "SAZ metadata exceeds the complete-evidence nesting budget.", + ); + const children: JsonValue[] = []; + for (let child = node.firstChild; child !== null; child = child.nextSibling) + children.push(visit(child, depth + 1)); + const attributes: JsonValue[] = []; + if (node.nodeType === 1) { + const element = node as import("@xmldom/xmldom").Element; + for (let i = 0; i < element.attributes.length; i++) { + const attribute = element.attributes.item(i); + if (attribute !== null) + attributes.push({ name: attribute.name, value: attribute.value }); + } + } + return { + name: node.nodeName, + type: node.nodeType, + value: node.nodeValue, + attributes, + children, + }; + }; + return visit(document, 0); +}; diff --git a/src/browser/history/SazProjection.ts b/src/browser/history/SazProjection.ts new file mode 100644 index 000000000..eba813628 --- /dev/null +++ b/src/browser/history/SazProjection.ts @@ -0,0 +1,131 @@ +import { createHash } from "node:crypto"; +import { isUtf8 } from "node:buffer"; +import type { JsonValue } from "../../domain/jsonValue.js"; +import { + WEB_NETWORK_CAPTURE_LIMITS, + type WebNetworkCaptureRecord, +} from "../../domain/webNetworkCapture.js"; +import { CaptureRedaction, capturePointer } from "./CaptureRedaction.js"; +import { CaptureFormatError } from "./CaptureFormatError.js"; + +export type SazValue = + | JsonValue + | Buffer + | { readonly [key: string]: SazValue } + | readonly SazValue[]; + +/** Project native byte fields with the same credential and explicit-literal exclusions as HAR. */ +export const projectSazRecord = ( + source: SazValue, + sensitiveValues: readonly string[], +) => { + const redactor = new CaptureRedaction(sensitiveValues); + const binary_fields: WebNetworkCaptureRecord["binary_fields"] = []; + const visit = ( + value: SazValue, + pointer: string, + depth: number, + parent?: Readonly>, + ): JsonValue => { + if (depth > WEB_NETWORK_CAPTURE_LIMITS.depth) + throw new CaptureFormatError( + "input-limit", + "SAZ projection exceeds the complete-evidence nesting budget.", + pointer, + ); + if (parent !== undefined && redactor.harCredential(pointer, parent)) + return redactor.credential(pointer); + if (Buffer.isBuffer(value)) { + const base64 = value.toString("base64"); + const sha256 = createHash("sha256").update(value).digest("hex"); + const hidden = + redactor.sensitiveBytes(value) || + sensitiveValues.some( + (literal) => + base64.includes(literal) || + sha256.includes(literal) || + String(value.length).includes(literal), + ); + if (hidden) { + binary_fields.push({ + pointer, + representation: "producer-bytes", + state: "redacted", + content_base64: null, + bytes: null, + sha256: null, + }); + redactor.redactions.push({ + pointer, + reason: "explicit-sensitive-value", + }); + return null; + } + binary_fields.push({ + pointer, + representation: "producer-bytes", + state: "retained", + content_base64: base64, + bytes: value.length, + sha256, + }); + return isUtf8(value) ? value.toString("utf8") : null; + } + if (typeof value === "string") { + // HTTP fields are Latin-1 views of bytes. A UTF-8 declaration must also + // exclude its byte spelling, even when its display is mojibake. + if ( + /^\/(request|response)\//.test(pointer) && + !pointer.endsWith("/member") && + redactor.sensitiveBytes(Buffer.from(value, "latin1")) + ) { + redactor.redactions.push({ + pointer, + reason: "explicit-sensitive-value", + }); + return null; + } + return redactor.text( + value, + pointer, + redactor.harTransportUrl(pointer, parent), + ); + } + if (value === null || typeof value === "boolean") return value; + if (typeof value === "number") + return redactor.text(String(value), pointer) === null ? null : value; + if (Array.isArray(value)) + return value.map((child: SazValue, i: number) => + visit(child, capturePointer(pointer, String(i)), depth + 1), + ); + return Object.fromEntries( + Object.entries(value).flatMap(([key, child]) => { + if (sensitiveValues.some((literal) => key.includes(literal))) { + redactor.redactions.push({ + pointer, + reason: "explicit-sensitive-value", + scope: "property-name", + }); + return []; + } + return [ + [ + key, + visit( + child, + capturePointer(pointer, key), + depth + 1, + value as Readonly>, + ), + ], + ]; + }), + ); + }; + return { + reported: visit(source, "", 0), + binary_fields, + numeric_literals: [], + redactions: redactor.redactions, + }; +}; diff --git a/src/browser/modules/WebModuleArtifacts.ts b/src/browser/modules/WebModuleArtifacts.ts index 287e6e8de..928025f09 100644 --- a/src/browser/modules/WebModuleArtifacts.ts +++ b/src/browser/modules/WebModuleArtifacts.ts @@ -4,6 +4,7 @@ import type { } from "../../application/WebModulePorts.js"; import type { WebScriptArtifactPort } from "../../application/WebScriptArtifactPort.js"; import type { ExecutionOptions } from "../../application/AnalysisProvider.js"; +import { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js"; import { jsonObjectSchema } from "../../domain/jsonValue.js"; import { err, ok } from "../../domain/result.js"; import type { WebModuleTraceInput } from "../../domain/webModuleTrace.js"; @@ -15,14 +16,64 @@ import { /** Add an explicitly selected import map to a verified captured-script source. */ export class LocalWebModuleArtifacts implements WebModuleArtifactPort { + readonly scripts: WebScriptArtifactPort; + readonly #resources: ArtifactResourceScope; + constructor( - readonly scripts: WebScriptArtifactPort = new LocalWebScriptArtifacts( - "trace_web_module_imports", - ), - ) {} + scripts?: WebScriptArtifactPort, + resources = scripts instanceof LocalWebScriptArtifacts + ? scripts.resources + : new ArtifactResourceScope(), + ) { + this.#resources = resources; + this.scripts = + scripts ?? + new LocalWebScriptArtifacts("trace_web_module_imports", resources); + } + + async close(): Promise { + const closeScript = + this.scripts instanceof LocalWebScriptArtifacts && + this.scripts.resources === this.#resources + ? undefined + : this.scripts.close?.bind(this.scripts); + const results = await Promise.allSettled([ + Promise.resolve().then(() => this.#resources.close()), + ...(closeScript === undefined + ? [] + : [Promise.resolve().then(closeScript)]), + ]); + const failures = results.flatMap((result) => + result.status === "rejected" ? [result.reason] : [], + ); + if (failures.length > 0) + throw new AggregateError(failures, "Web module artifact cleanup failed"); + } /** Read selected source and optional map without website requests. */ async load(input: WebModuleTraceInput, options?: ExecutionOptions) { + try { + return await this.#resources.run(() => this.#load(input, options)); + } catch (cause: unknown) { + return err( + webArtifactReadError( + cause, + { + operation: "trace_web_module_imports", + field: + input.import_map === undefined + ? ["manifest_path"] + : ["import_map", "path"], + targetPath: input.import_map?.path ?? input.manifest_path, + callerSelected: input.import_map !== undefined, + }, + options?.signal, + ), + ); + } + } + + async #load(input: WebModuleTraceInput, options?: ExecutionOptions) { const loaded = await this.scripts.load(input, options); if (!loaded.ok) return loaded; if (input.import_map === undefined) @@ -33,6 +84,7 @@ export class LocalWebModuleArtifacts implements WebModuleArtifactPort { selected.path, 4 * 1024 * 1024, options?.signal, + this.#resources, ); const importMap: WebModuleArtifacts["importMap"] = { file: { diff --git a/src/captures/TsharkCaptureProvider.ts b/src/captures/TsharkCaptureProvider.ts index 13cadd774..d22515929 100644 --- a/src/captures/TsharkCaptureProvider.ts +++ b/src/captures/TsharkCaptureProvider.ts @@ -1,3 +1,12 @@ +import { + type InspectPacketFieldsInput, + packetFieldsCatalogSchema, +} from "../domain/captures/packetFields.js"; +import { + parseFieldDiscovery, + FieldDiscoveryBudgetError, + FIELD_RECORD_BUDGET, +} from "./parseFieldDiscovery.js"; import { readLinuxResourceLimits, type LinuxResourceLimits, @@ -91,17 +100,31 @@ export class TsharkCaptureProvider implements PacketCapturePort { options?: ExecutionOptions, ): Promise> { return this.#resources.run( - () => this.#inspect(input, options), + () => this.#executeRequest(input, options), options?.signal, ); } + inspectFields( + input: InspectPacketFieldsInput, + options?: ExecutionOptions, + ): Promise> { + return this.#resources.run( + () => this.#executeRequest(input, options), + options?.signal, + "inspect_packet_fields", + ); + } close(): Promise { return this.#resources.close(); } - async #inspect( - input: InspectPacketCaptureInput, + async #executeRequest( + input: InspectPacketCaptureInput | InspectPacketFieldsInput, options?: ExecutionOptions, ): Promise> { + const operation = "path" in input ? OPERATION : "inspect_packet_fields"; + // TShark 4.6.9 emits a 22.3 MB glossary; keep capture output capped at 4 MiB. + const limits = + "path" in input ? LIMITS : { ...LIMITS, output_bytes: 32 * 1024 * 1024 }; let phase: | "configuration" | "artifact" @@ -109,12 +132,12 @@ export class TsharkCaptureProvider implements PacketCapturePort { | "producer" | "integrity" = "configuration"; let capturedOutput: AnalysisCapturedOutput | undefined; - let resourcePath = input.path; + let resourcePath = "path" in input ? input.path : "TShark field glossary"; let effectiveLimits: LinuxResourceLimits | undefined; let root: ArtifactAnalysisRuntime | undefined; let result: Result; try { - if (options?.signal?.aborted) throw new AnalysisCancelledError(OPERATION); + if (options?.signal?.aborted) throw new AnalysisCancelledError(operation); await this.#resources.release(); if (this.platform !== "linux") throw new Error( @@ -141,7 +164,7 @@ export class TsharkCaptureProvider implements PacketCapturePort { 64 * 1024 * 1024, options?.signal, ); - effectiveLimits = await this.#resources.readLimits(LIMITS); + effectiveLimits = await this.#resources.readLimits(limits); const limitArguments = [ `--as=${effectiveLimits.address_space_bytes}:${effectiveLimits.address_space_bytes}`, `--fsize=${effectiveLimits.file_bytes}:${effectiveLimits.file_bytes}`, @@ -152,7 +175,7 @@ export class TsharkCaptureProvider implements PacketCapturePort { ]; phase = "workspace"; root = await this.createRuntime(); - this.#resources.retain(root, undefined); + this.#resources.retain(root, undefined, operation); const snapshotPath = join(root.path, "capture"); const configurationPath = join(root.path, "configuration"); await mkdir(configurationPath, { mode: 0o700 }); @@ -170,8 +193,8 @@ export class TsharkCaptureProvider implements PacketCapturePort { }, }, { - timeoutMs: LIMITS.command_timeout_ms, - diagnosticBytes: LIMITS.output_bytes, + timeoutMs: limits.command_timeout_ms, + diagnosticBytes: limits.output_bytes, acceptNonZeroExit: true, }, { @@ -200,24 +223,24 @@ export class TsharkCaptureProvider implements PacketCapturePort { ); if (currentLimiter.sha256 !== limiterIdentity.sha256) throw new AnalysisArtifactChangedError( - OPERATION, + operation, limiter, "Configured resource limiter changed during analysis.", { capturedOutput }, ); if (current.sha256 !== tool.sha256) throw new AnalysisArtifactChangedError( - OPERATION, + operation, command, "Configured TShark executable changed during analysis.", { capturedOutput }, ); phase = previousPhase; - resourcePath = input.path; + resourcePath = "path" in input ? input.path : command; if (output.exitCode !== 0) throw new ProviderAdapterError( TSHARK_PROVIDER_IDENTITY.id, - OPERATION, + operation, { diagnostics: { exit_code: output.exitCode ?? null, @@ -238,86 +261,15 @@ export class TsharkCaptureProvider implements PacketCapturePort { throw new Error( "Expected TShark 4.x; inspect the selected installation.", ); - phase = "artifact"; - const selected = await this.#resources.read( - input.path, - LIMITS.input_bytes, - options?.signal, - ); - phase = "workspace"; - await writeFile(snapshotPath, selected.bytes, { - flag: "wx", - mode: 0o400, - }); - phase = "producer"; - const args = [ - "-n", - "-r", - "capture", - "-T", - "json", - "--no-duplicate-keys", - "--temp-dir=.", - ...(input.two_pass ? ["-2"] : []), - ...(input.packet_limit === null - ? [] - : ["-c", String(input.packet_limit)]), - ...(input.display_filter === null ? [] : ["-Y", input.display_filter]), - ...input.decode_as.flatMap((value) => ["-d", value]), - ...[...new Set(["frame.number", ...input.fields])].flatMap((field) => [ - "-e", - field, - ]), - ]; - const output = await execute(args); - phase = "integrity"; - resourcePath = snapshotPath; - if ( - (await this.#resources.read(snapshotPath, LIMITS.input_bytes)) - .sha256 !== selected.sha256 - ) - throw new AnalysisArtifactChangedError( - OPERATION, - snapshotPath, - "TShark changed the private capture copy.", - capturedOutput === undefined ? {} : { capturedOutput }, - ); - phase = "producer"; - resourcePath = input.path; - const matchingPackets = parsePacketOutput( - output.stdout.text, - input.fields, - ); - const start = Math.min(input.match_offset, matchingPackets.length); - const end = - input.match_limit === null - ? matchingPackets.length - : start + Math.min(input.match_limit, matchingPackets.length - start); - const packets = matchingPackets.slice(start, end); - const report = packetCaptureSchema.parse({ - artifact: { - path: input.path, - sha256: selected.sha256, - bytes: selected.bytes.length, - }, - selection: input, - packets, - output_window: { - matching_packets: matchingPackets.length, - returned_packets: packets.length, - has_more: end < matchingPackets.length, - }, - read_scope: - input.packet_limit === null ? "to-eof" : "up-to-input-packet-limit", - configuration: "private-empty-personal-profile", - diagnostics: output.stderr.text, + const metadata = (args: readonly string[], diagnostics: string) => ({ + diagnostics, tool: { configured_path: configured, path: command, sha256: tool.sha256, version_banner: version.stdout.text, version_diagnostics: version.stderr.text, - arguments: args, + arguments: [...args], }, resource_limiter: { configured_path: configuredLimiter, @@ -325,33 +277,136 @@ export class TsharkCaptureProvider implements PacketCapturePort { sha256: limiterIdentity.sha256, arguments: limitArguments, }, - limits: { ...LIMITS, ...effectiveLimits }, - hardware_acquisition: "not-performed", + limits: { ...limits, ...effectiveLimits }, + hardware_acquisition: "not-performed" as const, }); - result = ok( - createAnalysisExecution(report, TSHARK_PROVIDER_IDENTITY, { - subject: { + if (!("path" in input)) { + phase = "producer"; + const args = ["-G", "fields"]; + const output = await execute(args); + const report = packetFieldsCatalogSchema.parse({ + selection: input, + ...parseFieldDiscovery(output.stdout.text, input), + ...metadata(args, output.stderr.text), + limits: { + ...limits, + ...effectiveLimits, + record_budget_bytes: FIELD_RECORD_BUDGET, + }, + configuration: "private-empty-personal-profile", + }); + result = ok( + createAnalysisExecution(report, TSHARK_PROVIDER_IDENTITY, { + subject: { path: command, sha256: tool.sha256, format: "file" }, + limitations: [ + "Protocol and field registrations describe the selected installation, not traffic observed in a capture. Duplicate abbreviations remain distinct producer records with source line numbers.", + "Each request probes again without caching. The private empty personal profile does not fingerprint or isolate installed libraries, global configuration or plugins; they remain trusted.", + "No capture file, interface or device is opened. Resource ceilings do not constitute an OS sandbox. Glossary output is capped at 32 MiB and selected record representation at 8 MiB before expansion.", + ], + }), + ); + } else { + phase = "artifact"; + const selected = await this.#resources.read( + input.path, + LIMITS.input_bytes, + options?.signal, + ); + phase = "workspace"; + await writeFile(snapshotPath, selected.bytes, { + flag: "wx", + mode: 0o400, + }); + phase = "producer"; + const args = [ + "-n", + "-r", + "capture", + "-T", + "json", + "--no-duplicate-keys", + "--temp-dir=.", + ...(input.two_pass ? ["-2"] : []), + ...(input.packet_limit === null + ? [] + : ["-c", String(input.packet_limit)]), + ...(input.display_filter === null + ? [] + : ["-Y", input.display_filter]), + ...input.decode_as.flatMap((value) => ["-d", value]), + ...[...new Set(["frame.number", ...input.fields])].flatMap( + (field) => ["-e", field], + ), + ]; + const output = await execute(args); + phase = "integrity"; + resourcePath = snapshotPath; + if ( + (await this.#resources.read(snapshotPath, LIMITS.input_bytes)) + .sha256 !== selected.sha256 + ) + throw new AnalysisArtifactChangedError( + operation, + snapshotPath, + "TShark changed the private capture copy.", + capturedOutput === undefined ? {} : { capturedOutput }, + ); + phase = "producer"; + resourcePath = "path" in input ? input.path : command; + const matchingPackets = parsePacketOutput( + output.stdout.text, + input.fields, + ); + const start = Math.min(input.match_offset, matchingPackets.length); + const end = + input.match_limit === null + ? matchingPackets.length + : start + + Math.min(input.match_limit, matchingPackets.length - start); + const packets = matchingPackets.slice(start, end); + const report = packetCaptureSchema.parse({ + artifact: { path: input.path, sha256: selected.sha256, - format: "file", + bytes: selected.bytes.length, }, - locations: [{ kind: "artifact-path", path: input.path }], - limitations: [ - "Fields are TShark dissector observations under the selected filter/decode-as/pass settings, not independently verified protocol semantics. Missing fields are absent from the producer output, not known zero or empty values.", - "Packet limits count input packets before display filtering. Filtered results do not establish whole-capture or device traffic absence. Two-pass reassembly cannot recover bytes missing from the recording or beyond the read limit.", - "Repeated field values remain ordered string arrays; raw packet bytes are included only when explicitly selected as a supported field. Export index/score metadata are omitted. All stderr, including warnings on exit zero, is retained.", - "A private empty personal configuration is used. Installed libraries, global configuration and dissector plugins are trusted and not fully fingerprinted. File-only analysis and disabled name resolution do not constitute an OS sandbox.", - "Linux prlimit enforces soft and hard ceilings before execution, preserving stricter inherited limits: at most 1 GiB address space per process, 64 MiB per file, 30 CPU seconds and no core dumps. Effective limits are reported. Resident input is capped at 32 MiB, each command at 30 wall-clock seconds and combined output at 4 MiB. Output overflow fails. These are per-process/per-file limits, not aggregate memory or filesystem quotas; caller-supplied tools and the limiter are trusted.", - ], - }), - ); + selection: input, + packets, + output_window: { + matching_packets: matchingPackets.length, + returned_packets: packets.length, + has_more: end < matchingPackets.length, + }, + read_scope: + input.packet_limit === null ? "to-eof" : "up-to-input-packet-limit", + configuration: "private-empty-personal-profile", + ...metadata(args, output.stderr.text), + }); + result = ok( + createAnalysisExecution(report, TSHARK_PROVIDER_IDENTITY, { + subject: { + path: input.path, + sha256: selected.sha256, + format: "file", + }, + locations: [{ kind: "artifact-path", path: input.path }], + limitations: [ + "Fields are TShark dissector observations under the selected filter/decode-as/pass settings, not independently verified protocol semantics. Missing fields are absent from the producer output, not known zero or empty values.", + "Packet limits count input packets before display filtering. Filtered results do not establish whole-capture or device traffic absence. Two-pass reassembly cannot recover bytes missing from the recording or beyond the read limit.", + "Repeated field values remain ordered string arrays; raw packet bytes are included only when explicitly selected as a supported field. Export index/score metadata are omitted. All stderr, including warnings on exit zero, is retained.", + "A private empty personal configuration is used. Installed libraries, global configuration and dissector plugins are trusted and not fully fingerprinted. File-only analysis and disabled name resolution do not constitute an OS sandbox.", + "Linux prlimit enforces soft and hard ceilings before execution, preserving stricter inherited limits: at most 1 GiB address space per process, 64 MiB per file, 30 CPU seconds and no core dumps. Effective limits are reported. Resident input is capped at 32 MiB, each command at 30 wall-clock seconds and combined output at 4 MiB. Output overflow fails. These are per-process/per-file limits, not aggregate memory or filesystem quotas; caller-supplied tools and the limiter are trusted.", + ], + }), + ); + } } catch (cause) { if ( root !== undefined && cause instanceof OwnedCommandFailure && cause.cleanupOwner !== undefined ) - this.#resources.retain(root, cause.cleanupOwner); + this.#resources.retain(root, cause.cleanupOwner, operation); result = err( this.#failure( cause, @@ -360,18 +415,23 @@ export class TsharkCaptureProvider implements PacketCapturePort { root?.path ?? null, effectiveLimits, capturedOutput, + operation, + limits, ), ); } - return this.#resources.finish(result, (execution) => - createEvidence(execution.subject ?? undefined, execution.provider, { - operation: OPERATION, - parameters: input, - result: execution.result, - confidence: "observed", - locations: execution.locations, - limitations: execution.limitations, - }), + return this.#resources.finish( + result, + (execution) => + createEvidence(execution.subject ?? undefined, execution.provider, { + operation, + parameters: input, + result: execution.result, + confidence: "observed", + locations: execution.locations, + limitations: execution.limitations, + }), + operation, ); } #failure( @@ -386,6 +446,11 @@ export class TsharkCaptureProvider implements PacketCapturePort { workspacePath: string | null, effectiveLimits: LinuxResourceLimits | undefined, observedOutput: AnalysisCapturedOutput | undefined, + operation = OPERATION, + limits: { + readonly output_bytes: number; + readonly command_timeout_ms: number; + } = LIMITS, ): AnalysisError { if (cause instanceof AnalysisError) return cause; if (cause instanceof ArtifactReaderFailure && cause.cleanup !== undefined) @@ -393,15 +458,34 @@ export class TsharkCaptureProvider implements PacketCapturePort { cause.cleanup, cause, observedOutput, + operation, ); if ( (cause instanceof Error && cause.name === "AbortError") || (cause instanceof ArtifactReaderFailure && cause.reason === "cancelled") ) - return new AnalysisCancelledError(OPERATION, { cause }); + return new AnalysisCancelledError(operation, { cause }); + if (cause instanceof FieldDiscoveryBudgetError) + return new AnalysisResourceConstraintError( + operation, + "memory", + cause.message, + { record_budget_bytes: FIELD_RECORD_BUDGET }, + { + cause, + ...(observedOutput === undefined + ? {} + : { capturedOutput: observedOutput }), + }, + ); const reason = cause instanceof Error ? cause.message : String(cause); if (cause instanceof OwnedCommandFailure) - return this.#resources.commandFailure(cause, LIMITS, effectiveLimits); + return this.#resources.commandFailure( + cause, + limits, + effectiveLimits, + operation, + ); if (phase === "workspace") { const code = cause instanceof Error && @@ -411,13 +495,13 @@ export class TsharkCaptureProvider implements PacketCapturePort { : null; if (code === "ENOSPC" || code === "EDQUOT" || code === "EFBIG") return new AnalysisResourceConstraintError( - OPERATION, + operation, "file-size", `Private analysis workspace failed (${code}): ${reason}`, { workspace_path: workspacePath, system_code: code }, { cause }, ); - return new ProviderAdapterError(TSHARK_PROVIDER_IDENTITY.id, OPERATION, { + return new ProviderAdapterError(TSHARK_PROVIDER_IDENTITY.id, operation, { cause, userMessage: `Cannot prepare the private analysis workspace: ${reason}`, diagnostics: { @@ -440,12 +524,12 @@ export class TsharkCaptureProvider implements PacketCapturePort { ["limit", "integrity", "path"].includes(cause.reason) ) return new AnalysisArtifactChangedError( - OPERATION, + operation, path, `Post-run identity verification failed for ${path}: ${reason}`, detail, ); - return new ProviderAdapterError(TSHARK_PROVIDER_IDENTITY.id, OPERATION, { + return new ProviderAdapterError(TSHARK_PROVIDER_IDENTITY.id, operation, { ...detail, diagnostics: { stage: "integrity", resource_path: path, reason }, }); @@ -453,10 +537,13 @@ export class TsharkCaptureProvider implements PacketCapturePort { if (phase === "configuration") return new AnalysisCapabilityUnavailableError( TSHARK_PROVIDER_IDENTITY.id, - OPERATION, + operation, reason, { cause, + ...(observedOutput === undefined + ? {} + : { capturedOutput: observedOutput }), userMessage: `${reason} On Linux, configure TShark 4.x through REA_TSHARK_COMMAND and util-linux prlimit through REA_TSHARK_PRLIMIT_COMMAND (default /usr/bin/prlimit).`, }, ); @@ -466,14 +553,14 @@ export class TsharkCaptureProvider implements PacketCapturePort { cause.reason === "limit" ) return new AnalysisResourceConstraintError( - OPERATION, + operation, "file-size", reason, { input_bytes: LIMITS.input_bytes }, { cause }, ); if (cause instanceof ArtifactReaderFailure && cause.reason === "integrity") - return new AnalysisArtifactChangedError(OPERATION, path, reason, { + return new AnalysisArtifactChangedError(operation, path, reason, { cause, }); if ( @@ -482,13 +569,18 @@ export class TsharkCaptureProvider implements PacketCapturePort { "code" in cause && (cause.code === "EACCES" || cause.code === "EPERM") ) - return new AnalysisAccessDeniedError(OPERATION, path, cause.code, { + return new AnalysisAccessDeniedError(operation, path, cause.code, { cause, }); if (phase === "artifact") - return new AnalysisInputError(OPERATION, { cause }, [ + return new AnalysisInputError(operation, { cause }, [ { path: ["path"], reason: "invalid_value", message: reason }, ]); - return new AnalysisOutputError(OPERATION, reason, { cause }); + return new AnalysisOutputError(operation, reason, { + cause, + ...(observedOutput === undefined + ? {} + : { capturedOutput: observedOutput }), + }); } } diff --git a/src/captures/parseFieldDiscovery.ts b/src/captures/parseFieldDiscovery.ts new file mode 100644 index 000000000..6498e56dc --- /dev/null +++ b/src/captures/parseFieldDiscovery.ts @@ -0,0 +1,92 @@ +import type { + InspectPacketFieldsInput, + PacketFieldRecord, +} from "../domain/captures/packetFields.js"; + +export class FieldDiscoveryBudgetError extends Error {} +export const FIELD_RECORD_BUDGET = 8 * 1024 * 1024; + +/** TShark -G fields: P/name/abbreviation or F/name/abbreviation/type/parent/base/mask/blurb. */ +export const parseFieldDiscovery = ( + text: string, + input: InspectPacketFieldsInput, +) => { + const copy = (value: string) => + Buffer.from(value, "utf16le").toString("utf16le"); + const query = input.query.toLowerCase(); + const records: PacketFieldRecord[] = []; + let examined = 0, + matching = 0, + sourceLine = 0, + budget = 0; + // Walk the bounded text once; allocate record objects only for the requested page. + for (let start = 0; start < text.length;) { + const end = text.indexOf("\n", start); + let line = text.slice(start, end < 0 ? text.length : end); + start = end < 0 ? text.length : end + 1; + sourceLine++; + if (line.endsWith("\r")) line = line.slice(0, -1); + const columns = line.split("\t", 9); + const [tag, name, abbreviation] = columns; + if ( + !( + (tag === "P" && columns.length === 3) || + (tag === "F" && columns.length === 8) + ) || + name === undefined || + abbreviation === undefined || + abbreviation === "" + ) + throw new Error( + `Malformed TShark field glossary record at line ${sourceLine}`, + ); + if ((input.kind === "protocols") !== (tag === "P")) continue; + examined++; + if ( + input.protocol !== null && + (tag === "P" ? abbreviation : columns[4]) !== input.protocol + ) + continue; + if ( + ![name, abbreviation, ...(tag === "F" ? [columns[7] ?? ""] : [])].some( + (value) => value.toLowerCase().includes(query), + ) + ) + continue; + matching++; + if ( + matching <= input.offset || + records.length >= (input.limit ?? Number.MAX_SAFE_INTEGER) + ) + continue; + // Six bytes per source byte covers worst-case JSON escaping, plus record bookkeeping. + budget += Buffer.byteLength(line, "utf8") * 6 + 512; + if (budget > FIELD_RECORD_BUDGET) + throw new FieldDiscoveryBudgetError( + "Selected glossary records exceed the 8 MiB representation budget; narrow the query or page size", + ); + // Detach retained fields from the much larger producer string. + records.push({ + source_line: sourceLine, + kind: tag === "P" ? "protocol" : "field", + name: copy(name), + abbreviation: copy(abbreviation), + field_type: tag === "F" ? copy(columns[3] ?? "") : null, + parent_protocol: tag === "F" ? copy(columns[4] ?? "") : null, + display_base: tag === "F" ? copy(columns[5] ?? "") : null, + bitmask: tag === "F" ? copy(columns[6] ?? "") : null, + description: tag === "F" ? copy(columns[7] ?? "") : null, + }); + } + if (sourceLine === 0) + throw new Error("TShark returned an empty field glossary"); + return { + records, + output_window: { + examined, + matching, + returned: records.length, + has_more: Math.max(0, matching - input.offset) > records.length, + }, + }; +}; diff --git a/src/cli.ts b/src/cli.ts index 0b545c0ba..d1bdf2912 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -24,6 +24,7 @@ import { registerManagedCommands } from "./cli/managedCommands.js"; import { registerFirmwareCommands } from "./cli/firmwareCommands.js"; import { registerAndroidCommands } from "./cli/androidCommands.js"; import { registerApktoolCommands } from "./cli/apktoolCommands.js"; +import { registerPcapCommands } from "./cli/pcapCommands.js"; import { registerFlutterCommands } from "./cli/flutterCommands.js"; import { registerJebCommands } from "./cli/jebCommands.js"; import { registerAdbCommands } from "./cli/adbCommands.js"; @@ -105,6 +106,7 @@ export const createCli = ( registerManagedCommands(cli, logger, analysis.runProviderAnalysis); registerAndroidCommands(cli, logger, environment); registerApktoolCommands(cli, logger, environment); + registerPcapCommands(cli, logger, environment); registerFlutterCommands(cli, logger); registerJebCommands(cli, logger, environment); registerAdbCommands(cli, logger, environment); diff --git a/src/cli/apktoolCommands.ts b/src/cli/apktoolCommands.ts index d7b30a617..c7795f9d4 100644 --- a/src/cli/apktoolCommands.ts +++ b/src/cli/apktoolCommands.ts @@ -62,15 +62,17 @@ export const registerApktoolCommands = ( .describe( "Project strings from res/values- instead of the default", ), - "no-strings": z + strings: z .boolean() - .default(false) - .describe("Skip projecting string resources"), + .default(true) + .describe( + "Include string resources; use --no-strings to skip projecting them", + ), }), run: ({ args, options }) => execute(CLI_COMMANDS.decodeAndroidResources, "decode_android_resources", { path: args.path, - include_strings: !options["no-strings"], + include_strings: options.strings, ...(options.locale === undefined ? {} : { locale: options.locale }), }), }); diff --git a/src/cli/binaryDiagnosticsCommands.ts b/src/cli/binaryDiagnosticsCommands.ts index bddd16bed..5cb9577bd 100644 --- a/src/cli/binaryDiagnosticsCommands.ts +++ b/src/cli/binaryDiagnosticsCommands.ts @@ -7,11 +7,12 @@ import { } from "../composition/binaryDiagnostics.js"; import { CLI_COMMANDS } from "../cliCommandNames.js"; import { logCliCommand } from "../cliLogging.js"; -import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; import type { Logger } from "pino"; import type { CliInstance } from "./types.js"; import { withCommandCancellation } from "./commandCancellation.js"; import { inspectPeResourcesInputSchema } from "../domain/native/peResources.js"; +import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; +import { finishAnalysisCommand } from "./finishAnalysisCommand.js"; /** Inspect operator-selected files through the same shared workflow as MCP. */ export const registerBinaryDiagnosticsCommands = ( @@ -19,7 +20,6 @@ export const registerBinaryDiagnosticsCommands = ( logger: Logger, environment: Readonly, ): void => { - const pe = createPeResourcesService(); cli.command(CLI_COMMANDS.inspectPeResources, { description: "Inspect PE resource identities, payload ranges, hashes and icon-group relationships", @@ -29,19 +29,28 @@ export const registerBinaryDiagnosticsCommands = ( maxEntries: inspectPeResourcesInputSchema.shape.max_entries, }), run: ({ args, options }) => - withCommandCancellation((signal) => - logCliCommand(logger, CLI_COMMANDS.inspectPeResources, async () => { - const result = await pe.inspect( - { - path: resolve(args.path), - max_file_bytes: options.maxFileBytes, - max_entries: options.maxEntries, - }, - { signal }, - ); - return result.ok ? result.value : projectAnalysisError(result.error); - }), - ), + withCommandCancellation(async (signal) => { + const pe = createPeResourcesService(); + return await logCliCommand( + logger, + CLI_COMMANDS.inspectPeResources, + async () => + finishAnalysisCommand( + () => + pe.inspect( + { + path: resolve(args.path), + max_file_bytes: options.maxFileBytes, + max_entries: options.maxEntries, + }, + { signal }, + ), + pe, + "rea-pe-resources", + "inspect_pe_resources", + ), + ); + }), }); const service = createBinaryLayoutService(environment); cli.command(CLI_COMMANDS.inspectBinaryLayout, { diff --git a/src/cli/captureCommands.ts b/src/cli/captureCommands.ts index 281be7457..542361e90 100644 --- a/src/cli/captureCommands.ts +++ b/src/cli/captureCommands.ts @@ -1,8 +1,6 @@ +import { inspectPacketFieldsInputSchema } from "../domain/captures/packetFields.js"; import { z } from "incur"; import type { PacketCaptureService } from "../application/captures/PacketCaptureService.js"; -import { AnalysisError } from "../domain/analysisErrorBase.js"; -import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js"; -import { ProviderCleanupError } from "../domain/providerCleanupError.js"; import { resolve } from "node:path"; import { createPacketCaptureService } from "../composition/captures.js"; import { @@ -12,19 +10,60 @@ import { } from "../domain/captures/packetCapture.js"; import { CLI_COMMANDS } from "../cliCommandNames.js"; import { logCliCommand } from "../cliLogging.js"; -import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; import type { Logger } from "pino"; import type { CliInstance } from "./types.js"; import { withCommandCancellation } from "./commandCancellation.js"; +import { finishAnalysisCommand } from "./finishAnalysisCommand.js"; export const registerCaptureCommands = ( cli: CliInstance, logger: Logger, environment: Readonly, service: Pick< PacketCaptureService, - "inspect" | "close" + "inspect" | "inspectFields" | "close" > = createPacketCaptureService(environment), ): void => { + cli.command(CLI_COMMANDS.inspectPacketFields, { + description: "Search installed TShark protocol and field definitions", + options: z + .object({ + ...inspectPacketFieldsInputSchema.shape, + limit: z + .number() + .int() + .positive() + .max(Number.MAX_SAFE_INTEGER) + .optional() + .describe("Maximum matching records to return; defaults to 50"), + all: z + .boolean() + .default(false) + .describe("Return all remaining matches within resource limits"), + }) + .superRefine((options, context) => { + if (options.all && options.limit !== undefined) + context.addIssue({ + code: "custom", + path: ["all"], + message: "all and limit are mutually exclusive", + }); + }), + run: ({ options: { all, ...options } }) => + withCommandCancellation((signal) => + logCliCommand(logger, CLI_COMMANDS.inspectPacketFields, async () => + finishAnalysisCommand( + () => + service.inspectFields( + { ...options, ...(all ? { limit: null } : {}) }, + { signal }, + ), + service, + "tshark-capture", + "inspect_packet_fields", + ), + ), + ), + }); cli.command(CLI_COMMANDS.inspectPacketCapture, { description: "Inspect an existing packet capture with caller-supplied TShark", @@ -60,39 +99,20 @@ export const registerCaptureCommands = ( run: ({ args, options: { all_matches, ...options } }) => withCommandCancellation((signal) => logCliCommand(logger, CLI_COMMANDS.inspectPacketCapture, async () => { - const result = await service.inspect( - { - ...options, - ...(all_matches ? { match_limit: null } : {}), - path: resolve(args.path), - }, - { signal }, + return finishAnalysisCommand( + () => + service.inspect( + { + ...options, + ...(all_matches ? { match_limit: null } : {}), + path: resolve(args.path), + }, + { signal }, + ), + service, + "tshark-capture", + "inspect_packet_capture", ); - // A one-shot command must retry retained cleanup before discarding its owner. - try { - await service.close(); - } catch (cause) { - const cleanup = - cause instanceof AnalysisError - ? cause - : new ProviderCleanupError( - "tshark-capture", - [], - { reason: String(cause) }, - { cause, operation: "inspect_packet_capture" }, - ); - if (result.ok) cleanup.retainPartialObservation(result.value); - return projectAnalysisError( - result.ok - ? cleanup - : analysisErrorWithCleanupFailure( - result.error, - cleanup, - "inspect_packet_capture", - ), - ); - } - return result.ok ? result.value : projectAnalysisError(result.error); }), ), }); diff --git a/src/cli/cliResultOutput.ts b/src/cli/cliResultOutput.ts index 67e745b0b..5efde8b33 100644 --- a/src/cli/cliResultOutput.ts +++ b/src/cli/cliResultOutput.ts @@ -216,8 +216,9 @@ export const createCliResultOutput = ( : filtered === undefined ? { ok: true, meta } : { ok: true, data: filtered, meta } - : filtered === undefined || - filtered === null || + : // A selected null is a value; only a selection with no JSON value + // falls back to the empty document. + filtered === undefined || typeof filtered === "function" || typeof filtered === "symbol" ? {} diff --git a/src/cli/coreBinaryCommands.ts b/src/cli/coreBinaryCommands.ts index 70405302e..710b39181 100644 --- a/src/cli/coreBinaryCommands.ts +++ b/src/cli/coreBinaryCommands.ts @@ -137,6 +137,13 @@ const registerDecompileCommand = ( .describe( "Procedure name or address; use --procedure= for leading dashes", ), + "file-offset": z + .string() + .regex(/^0x[0-9a-fA-F]+$/u) + .optional() + .describe( + "Ghidra source-file byte offset; mapped internally to an analysis address", + ), snapshot: z .string() .min(1) @@ -152,8 +159,17 @@ const registerDecompileCommand = ( { procedure: resolveCliAnalysisSelector( args.address, - options.procedure, - "procedure", + options["file-offset"] === undefined + ? options.procedure + : resolveCliAnalysisSelector( + options.procedure, + `file-offset:${options["file-offset"]}`, + "file-offset", + "procedure_pseudo_code", + ), + options["file-offset"] === undefined + ? "procedure" + : "file-offset", "procedure_pseudo_code", ), }, diff --git a/src/cli/finishAnalysisCommand.ts b/src/cli/finishAnalysisCommand.ts new file mode 100644 index 000000000..9394ba828 --- /dev/null +++ b/src/cli/finishAnalysisCommand.ts @@ -0,0 +1,69 @@ +import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js"; +import { + AnalysisError, + type AnalysisPartialObservation, +} from "../domain/analysisErrorBase.js"; +import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js"; +import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; +import { ProviderAdapterError } from "../domain/providerAdapterError.js"; +import { ProviderCleanupError } from "../domain/providerCleanupError.js"; +import type { Result } from "../domain/result.js"; + +/** Finish a one-shot CLI operation without discarding its primary result on close failure. */ +export const finishAnalysisCommand = async < + Value extends AnalysisPartialObservation, +>( + run: () => Promise>, + service: { close(): Promise }, + providerId: string, + canonicalOperation: string, + fallbackResource?: string, +): Promise> => { + let result: Result; + try { + result = await run(); + } catch (cause) { + result = { + ok: false, + error: + cause instanceof AnalysisError + ? cause + : new ProviderAdapterError(providerId, canonicalOperation, { cause }), + }; + } + try { + await service.close(); + } catch (cause) { + let cleanup: AnalysisError; + if (cause instanceof AnalysisError) cleanup = cause; + else { + const observation = + cause instanceof ArtifactReaderFailure + ? ArtifactReaderFailure.cleanupObservation( + cause, + fallbackResource ?? `${canonicalOperation} resources`, + ) + : { + reason: cause instanceof Error ? cause.message : String(cause), + resources: [], + }; + cleanup = new ProviderCleanupError( + providerId, + observation.resources, + { reason: observation.reason }, + { cause, operation: canonicalOperation }, + ); + } + if (result.ok) cleanup.retainPartialObservation(result.value); + return projectAnalysisError( + result.ok + ? cleanup + : analysisErrorWithCleanupFailure( + result.error, + cleanup, + canonicalOperation, + ), + ); + } + return result.ok ? result.value : projectAnalysisError(result.error); +}; diff --git a/src/cli/goCommands.ts b/src/cli/goCommands.ts index d67a344e0..12b2b992a 100644 --- a/src/cli/goCommands.ts +++ b/src/cli/goCommands.ts @@ -3,14 +3,13 @@ import { resolve } from "node:path"; import { createGoBinaryService } from "../composition/go.js"; import { CLI_COMMANDS } from "../cliCommandNames.js"; import { logCliCommand } from "../cliLogging.js"; -import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; import type { Logger } from "pino"; import type { CliInstance } from "./types.js"; import { withCommandCancellation } from "./commandCancellation.js"; +import { finishAnalysisCommand } from "./finishAnalysisCommand.js"; /** Resolve caller paths and inspect static build metadata through the shared workflow. */ export const registerGoCommands = (cli: CliInstance, logger: Logger): void => { - const service = createGoBinaryService(); cli.command(CLI_COMMANDS.inspectGoBinary, { description: "Inspect embedded Go compiler, modules and build settings without target execution", @@ -18,14 +17,19 @@ export const registerGoCommands = (cli: CliInstance, logger: Logger): void => { path: z.string().describe("Local ELF, PE or thin Mach-O image"), }), run: ({ args }) => - withCommandCancellation((signal) => - logCliCommand(logger, CLI_COMMANDS.inspectGoBinary, async () => { - const result = await service.inspect( - { path: resolve(args.path) }, - { signal }, - ); - return result.ok ? result.value : projectAnalysisError(result.error); - }), - ), + withCommandCancellation(async (signal) => { + const service = createGoBinaryService(); + return await logCliCommand( + logger, + CLI_COMMANDS.inspectGoBinary, + async () => + finishAnalysisCommand( + () => service.inspect({ path: resolve(args.path) }, { signal }), + service, + "go-build-info", + "inspect_go_binary", + ), + ); + }), }); }; diff --git a/src/cli/pcapCommands.ts b/src/cli/pcapCommands.ts new file mode 100644 index 000000000..1b53e47cc --- /dev/null +++ b/src/cli/pcapCommands.ts @@ -0,0 +1,69 @@ +import { z } from "incur"; +import { AnalysisError } from "../domain/analysisErrorBase.js"; +import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js"; +import { PcapAnalysisService } from "../application/pcap/PcapAnalysisService.js"; +import type { PcapAnalysisPort } from "../application/pcap/PcapAnalysisPort.js"; +import { createPcapAnalysisProvider } from "../composition/pcap.js"; +import { CLI_COMMANDS } from "../cliCommandNames.js"; +import { logCliCommand } from "../cliLogging.js"; +import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; +import type { PcapOperation } from "../domain/pcap/pcapAnalysis.js"; +import type { Logger } from "pino"; +import type { CliInstance } from "./types.js"; +import { withCommandCancellation } from "./commandCancellation.js"; + +/** Expose the same explicit pcap requests through one-shot CLI commands. */ +export const registerPcapCommands = ( + cli: CliInstance, + logger: Logger, + environment: Readonly>, + provider: PcapAnalysisPort = createPcapAnalysisProvider(environment), +): void => { + const service = new PcapAnalysisService(provider); + const execute = (name: string, operation: PcapOperation, input: unknown) => + withCommandCancellation((signal) => + logCliCommand(logger, name, async () => { + const result = await service.execute(operation, input, { signal }); + try { + await provider.close(); + } catch (cause) { + if (cause instanceof AnalysisError) { + if (result.ok) cause.retainPartialObservation(result.value); + return projectAnalysisError( + result.ok + ? cause + : analysisErrorWithCleanupFailure( + result.error, + cause, + operation, + ), + ); + } + throw cause; + } + return result.ok ? result.value : projectAnalysisError(result.error); + }), + ); + + cli.command(CLI_COMMANDS.inspectPcapClient, { + description: + "Inspect the caller-selected tshark and capinfos binaries without reading a capture", + args: z.object({}), + run: () => + execute(CLI_COMMANDS.inspectPcapClient, "inspect_pcap_client", {}), + }); + cli.command(CLI_COMMANDS.inspectPcapCapture, { + description: + "Inspect one pcap/pcapng capture: endpoints, DNS, TLS SNI, and protocol distribution", + args: z.object({ + path: z + .string() + .min(1) + .describe("Local pcap or pcapng capture file to inspect"), + }), + run: ({ args }) => + execute(CLI_COMMANDS.inspectPcapCapture, "inspect_pcap_capture", { + path: args.path, + }), + }); +}; diff --git a/src/cli/signalCommands.ts b/src/cli/signalCommands.ts index 02e5555f5..63b7d9edf 100644 --- a/src/cli/signalCommands.ts +++ b/src/cli/signalCommands.ts @@ -1,8 +1,6 @@ +import { inspectSignalDecodersInputSchema } from "../domain/signals/signalDecoders.js"; import { z } from "incur"; import type { SignalRecordingService } from "../application/signals/SignalRecordingService.js"; -import { AnalysisError } from "../domain/analysisErrorBase.js"; -import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js"; -import { ProviderCleanupError } from "../domain/providerCleanupError.js"; import { resolve } from "node:path"; import { createSignalRecordingService } from "../composition/signals.js"; import { @@ -12,19 +10,62 @@ import { } from "../domain/signals/signalRecording.js"; import { CLI_COMMANDS } from "../cliCommandNames.js"; import { logCliCommand } from "../cliLogging.js"; -import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; import type { Logger } from "pino"; import type { CliInstance } from "./types.js"; import { withCommandCancellation } from "./commandCancellation.js"; +import { finishAnalysisCommand } from "./finishAnalysisCommand.js"; export const registerSignalCommands = ( cli: CliInstance, logger: Logger, environment: Readonly, service: Pick< SignalRecordingService, - "decode" | "close" + "decode" | "inspectDecoders" | "close" > = createSignalRecordingService(environment), ): void => { + cli.command(CLI_COMMANDS.inspectSignalDecoders, { + description: "Search installed sigrok decoders or selected decoder help", + options: z + .object({ + ...inspectSignalDecodersInputSchema.shape, + limit: z + .number() + .int() + .positive() + .max(Number.MAX_SAFE_INTEGER) + .optional() + .describe( + "Maximum matching decoders or help lines to return; defaults to 50", + ), + all: z + .boolean() + .default(false) + .describe("Return all remaining matches"), + }) + .superRefine((options, context) => { + if (options.all && options.limit !== undefined) + context.addIssue({ + code: "custom", + path: ["all"], + message: "all and limit are mutually exclusive", + }); + }), + run: ({ options: { all, ...options } }) => + withCommandCancellation((signal) => + logCliCommand(logger, CLI_COMMANDS.inspectSignalDecoders, async () => + finishAnalysisCommand( + () => + service.inspectDecoders( + { ...options, ...(all ? { limit: null } : {}) }, + { signal }, + ), + service, + "sigrok-recording", + "inspect_signal_decoders", + ), + ), + ), + }); cli.command(CLI_COMMANDS.decodeSignalRecording, { description: "Decode an existing signal recording with caller-supplied sigrok", @@ -60,39 +101,20 @@ export const registerSignalCommands = ( run: ({ args, options: { all_lines, ...options } }) => withCommandCancellation((signal) => logCliCommand(logger, CLI_COMMANDS.decodeSignalRecording, async () => { - const result = await service.decode( - { - ...options, - ...(all_lines ? { line_limit: null } : {}), - path: resolve(args.path), - }, - { signal }, + return finishAnalysisCommand( + () => + service.decode( + { + ...options, + ...(all_lines ? { line_limit: null } : {}), + path: resolve(args.path), + }, + { signal }, + ), + service, + "sigrok-recording", + "decode_signal_recording", ); - // A one-shot command must retry retained cleanup before discarding its owner. - try { - await service.close(); - } catch (cause) { - const cleanup = - cause instanceof AnalysisError - ? cause - : new ProviderCleanupError( - "sigrok-recording", - [], - { reason: String(cause) }, - { cause, operation: "decode_signal_recording" }, - ); - if (result.ok) cleanup.retainPartialObservation(result.value); - return projectAnalysisError( - result.ok - ? cleanup - : analysisErrorWithCleanupFailure( - result.error, - cleanup, - "decode_signal_recording", - ), - ); - } - return result.ok ? result.value : projectAnalysisError(result.error); }), ), }); diff --git a/src/cli/webNetworkCaptureCommands.ts b/src/cli/webNetworkCaptureCommands.ts index 1b902969c..d49e39b0e 100644 --- a/src/cli/webNetworkCaptureCommands.ts +++ b/src/cli/webNetworkCaptureCommands.ts @@ -1,11 +1,12 @@ import { z } from "incur"; +import { WEB_NETWORK_CAPTURE_FORMATS } from "../domain/webNetworkCapture.js"; import { createWebNetworkCaptureService } from "../composition/webNetworkCaptures.js"; import { CLI_COMMANDS } from "../cliCommandNames.js"; import { logCliCommand } from "../cliLogging.js"; -import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; import type { Logger } from "pino"; import type { CliInstance } from "./types.js"; import { withCommandCancellation } from "./commandCancellation.js"; +import { finishAnalysisCommand } from "./finishAnalysisCommand.js"; /** Expose offline historical capture inspection with the same selection contract as MCP. */ export const registerWebNetworkCaptureCommands = ( @@ -16,10 +17,12 @@ export const registerWebNetworkCaptureCommands = ( const service = createWebNetworkCaptureService(environment); cli.command(CLI_COMMANDS.inspectWebNetworkCapture, { description: - "Inspect retained HAR or native mitmproxy records without fetching recorded URLs", + "Inspect retained HAR, native mitmproxy, Fiddler SAZ or aioquic qlog records without fetching recorded URLs", args: z.object({ capturePath: z.string().describe("Absolute local capture path"), - format: z.enum(["har", "mitmproxy"]).describe("Producer capture format"), + format: z + .enum(WEB_NETWORK_CAPTURE_FORMATS) + .describe("Producer capture format"), }), options: z.object({ record: z @@ -39,20 +42,24 @@ export const registerWebNetworkCaptureCommands = ( logger, CLI_COMMANDS.inspectWebNetworkCapture, async () => { - const result = await service.inspect( - { - capture_path: args.capturePath, - format: args.format, - sensitive_values: options.sensitiveValue, - ...(options.record === undefined - ? {} - : { record_ordinals: options.record }), - }, - { signal }, + return finishAnalysisCommand( + () => + service.inspect( + { + capture_path: args.capturePath, + format: args.format, + sensitive_values: options.sensitiveValue, + ...(options.record === undefined + ? {} + : { record_ordinals: options.record }), + }, + { signal }, + ), + service, + "web-network-capture", + "inspect_web_network_capture", + "Web network capture resources", ); - return result.ok - ? result.value - : projectAnalysisError(result.error); }, ), ), diff --git a/src/cliCommandNames.ts b/src/cliCommandNames.ts index 1278cbea8..d63a9a2bc 100644 --- a/src/cliCommandNames.ts +++ b/src/cliCommandNames.ts @@ -1,6 +1,8 @@ /** Canonical names used by every one-shot CLI command registration. */ export const CLI_COMMANDS = Object.freeze({ + inspectPacketFields: "inspect-packet-fields", inspectPacketCapture: "inspect-packet-capture", + inspectSignalDecoders: "inspect-signal-decoders", decodeSignalRecording: "decode-signal-recording", inspectWasmArtifact: "inspect-wasm-artifact", inspectSqliteDatabase: "inspect-sqlite-database", @@ -55,6 +57,8 @@ export const CLI_COMMANDS = Object.freeze({ inspectFirmwareRegions: "inspect-firmware-regions", extractFirmware: "extract-firmware", inspectAndroidPackage: "inspect-android-package", + inspectPcapClient: "inspect-pcap-client", + inspectPcapCapture: "inspect-pcap-capture", identifyFlutterBuild: "identify-flutter-build", inspectDartAot: "inspect-dart-aot", inspectApktoolClient: "inspect-apktool-client", diff --git a/src/cliOutput.ts b/src/cliOutput.ts index 5119eebc3..ccfaf6b62 100644 --- a/src/cliOutput.ts +++ b/src/cliOutput.ts @@ -9,7 +9,7 @@ export type CliOutputArgumentValidation = | { readonly ok: false; readonly format: StructuredOutputFormat; - readonly code: "UNSUPPORTED_OUTPUT_COMBINATION"; + readonly code: "UNSUPPORTED_OUTPUT_COMBINATION" | "INVALID_OUTPUT_FILTER"; readonly message: string; }; @@ -89,13 +89,39 @@ export const parseCliOutputArguments = ( return parsed(false); }; -/** Reject text-window controls that would corrupt a structured document. */ +/** Reject slices that cannot advance within Incur's comma-delimited paths. */ +const hasUnclosedFilterSlice = (expression: string): boolean => { + let depth = 0; + let unclosed = false; + for (const character of expression) { + if (character === "[") { + depth += 1; + unclosed = true; + } else if (character === "]") { + depth -= 1; + // Incur consumes everything through the next closing bracket, including + // nested opening brackets. Do not impose a different filter grammar. + unclosed = false; + } else if (character === "," && depth === 0 && unclosed) return true; + } + return unclosed; +}; + +/** Reject output controls that cannot safely produce a complete result. */ export const validateCliOutputArguments = ( arguments_: readonly string[], ): CliOutputArgumentValidation => { - const { format, parseError, tokenWindow } = + const { filterOutput, format, parseError, tokenWindow } = parseCliOutputArguments(arguments_); if (parseError) return { ok: true }; + if (filterOutput !== undefined && hasUnclosedFilterSlice(filterOutput)) + return { + ok: false, + format: format === "yaml" || format === "jsonl" ? format : "json", + code: "INVALID_OUTPUT_FILTER", + message: + "The --filter-output expression has a slice without a closing ]. Close the slice or remove --filter-output.", + }; if ( tokenWindow && (format === "json" || format === "jsonl" || format === "yaml") diff --git a/src/composition/javascriptAnalysisWorkerProcess.ts b/src/composition/javascriptAnalysisWorkerProcess.ts index 09a9c9bc3..d1c3c134e 100644 --- a/src/composition/javascriptAnalysisWorkerProcess.ts +++ b/src/composition/javascriptAnalysisWorkerProcess.ts @@ -1,4 +1,4 @@ -import { moduleKindForJavaScriptReference } from "../application/javascript/JavaScriptArtifactPathResolution.js"; +import { moduleResolutionModeForJavaScriptReference } from "../domain/javascript/javascriptModuleResolutionMode.js"; import { parseJavaScriptSource } from "../domain/javascript/javascriptSourceParser.js"; import { analyzeParsedJavaScriptStaticSourceSteps } from "../domain/javascript/javascriptStaticAnalysis.js"; import { @@ -56,7 +56,7 @@ const analyze: JavaScriptSourceWorkerAnalyzer = async ( const ir = await completeJavaScriptAnalysisSteps( analyzeParsedJavaScriptSemanticsSteps( parsed, - moduleKindForJavaScriptReference(file.path, "static-import"), + moduleResolutionModeForJavaScriptReference(file.path, "static-import"), ), ); parsed = null; diff --git a/src/composition/pcap.ts b/src/composition/pcap.ts new file mode 100644 index 000000000..f3aaa3915 --- /dev/null +++ b/src/composition/pcap.ts @@ -0,0 +1,7 @@ +import type { PcapAnalysisPort } from "../application/pcap/PcapAnalysisPort.js"; +import { PcapProvider } from "../pcap/PcapProvider.js"; + +/** Construct one pcap provider bound to the caller-selected tools. */ +export const createPcapAnalysisProvider = ( + environment: Readonly> = process.env, +): PcapAnalysisPort => new PcapProvider({ environment }); diff --git a/src/composition/webNetworkCaptures.ts b/src/composition/webNetworkCaptures.ts index 6dc7668d4..9d9e8c6cc 100644 --- a/src/composition/webNetworkCaptures.ts +++ b/src/composition/webNetworkCaptures.ts @@ -1,6 +1,6 @@ import { WebNetworkCaptureService } from "../application/WebNetworkCaptureService.js"; import { HistoricalCaptureDecoder } from "../browser/history/HistoricalCaptureDecoder.js"; -import { HarCaptureAdapter } from "../browser/history/HarCaptureAdapter.js"; +import { NodeCaptureAdapter } from "../browser/history/NodeCaptureAdapter.js"; import { MitmproxyCaptureAdapter } from "../browser/history/MitmproxyCaptureAdapter.js"; /** Compose offline format adapters without import-time reads, installation or process startup. */ @@ -9,7 +9,12 @@ export const createWebNetworkCaptureService = ( ): WebNetworkCaptureService => new WebNetworkCaptureService( new HistoricalCaptureDecoder( - [new HarCaptureAdapter(), new MitmproxyCaptureAdapter(environment)], + [ + new NodeCaptureAdapter("har"), + new NodeCaptureAdapter("saz"), + new NodeCaptureAdapter("qlog"), + new MitmproxyCaptureAdapter(environment), + ], environment, ), ); diff --git a/src/config.test.ts b/src/config.test.ts index eb784802f..67db7e111 100644 --- a/src/config.test.ts +++ b/src/config.test.ts @@ -3,6 +3,18 @@ import { describe, expect, it } from "vitest"; import { parseConfig } from "./config/parseConfig.js"; import { projectAnalysisError } from "./domain/analysisErrorProjection.js"; +it("reports a configuration failure for a NUL in Hopper loader arguments", () => { + const result = parseConfig({ + HOPPER_LOADER_ARGS_JSON: JSON.stringify(["--option", "value\0suffix"]), + }); + if (result.ok) + throw new Error("NUL argv cannot be passed to a child process"); + expect(result.error).toMatchObject({ + settings: [{ setting: "HOPPER_LOADER_ARGS_JSON" }], + }); + expect(result.error.message).toContain("NUL"); +}); + describe("runtime configuration", () => { it("allows target-free startup and applies runtime defaults", () => { const empty = parseConfig({}); diff --git a/src/config/parsers.ts b/src/config/parsers.ts index fc842f884..52feae622 100644 --- a/src/config/parsers.ts +++ b/src/config/parsers.ts @@ -1,6 +1,7 @@ import { z } from "zod"; import { ConfigurationError } from "../domain/configurationErrors.js"; +import { nulFreeStringSchema } from "../domain/localPath.js"; import { err, ok, type Result } from "../domain/result.js"; import { safeParseJson } from "../domain/safeJson.js"; @@ -20,7 +21,8 @@ export const parseLoaderArgs = ( return parseConfiguredStringArray( encoded, "HOPPER_LOADER_ARGS_JSON", - z.string(), + nulFreeStringSchema, + "must encode an array of strings without NUL bytes", ); }; @@ -28,6 +30,7 @@ const parseConfiguredStringArray = ( encoded: string, setting: string, itemSchema: z.ZodString, + requirement = "must encode an array of strings", ): Result => { const decoded = safeParseJson(encoded); if (!decoded.ok) @@ -37,13 +40,7 @@ const parseConfiguredStringArray = ( const parsed = z.array(itemSchema).safeParse(decoded.value); return parsed.success ? ok(parsed.data) - : err( - configuredArrayError( - setting, - "must encode an array of strings", - parsed.error, - ), - ); + : err(configuredArrayError(setting, requirement, parsed.error)); }; const configuredArrayError = ( diff --git a/src/contracts/analysisViewToolContracts.ts b/src/contracts/analysisViewToolContracts.ts index 682c49e6b..7d023752d 100644 --- a/src/contracts/analysisViewToolContracts.ts +++ b/src/contracts/analysisViewToolContracts.ts @@ -9,14 +9,14 @@ import { evidenceResultOf } from "./toolOutputSchemaPrimitives.js"; const RETAINED_EXAMPLE_ID = `ev_${"a".repeat(64)}`; -/** Selected views of completed layout, JavaScript, native, inventory, WASM and web network Evidence. */ +/** Selected views of completed layout, JavaScript, native, inventory, WASM, Mach-O and web network Evidence. */ export const ANALYSIS_VIEW_TOOL_CONTRACTS = [ { name: "inspect_analysis_view", ...toolContractMetadata("inspect_analysis_view"), kind: "application", description: - 'Inspect selected views of completed binary-layout, JavaScript application, native analyze_function, inspect_artifact, inventory_artifact, inspect_wasm_artifact, inspect_evm_interface, inspect_managed_artifact or inspect_web_network_capture Evidence without repeating analysis. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. For a retained record use source: {kind: "retained-evidence", evidence_id: ""}. Use normalized_result.parent_evidence_id from a summary receipt. Source kind is required. Portable source uses {kind: "inline", evidence: }. For layout, JavaScript, native, inventory, WASM, EVM and managed artifact sources, request a summary with view: {kind: "summary"}; summary is a view kind and has no facet. The WASM summary gives validation, counts, WAT digest/length, tool profile and execution/network states without reading WAT text. Select a layout mitigations/linkage facet, one section/symbol/module, or a stable page. Module pages include assets, bundled and source modules; use node_id for ambiguous or unavailable paths. Inventory uses view: {kind: "inventory", collection: "occurrences", logical_paths: ["package.json"], offset: 0, limit: 64}. Inventory views page nodes, occurrences, edges or integrity contradictions; occurrence logical_paths matches exact recorded strings (omitted means all available, [] means none). source_complete distinguishes complete inventories from partial source records; page exhaustion does not prove absence in a partial source. WASM views use kind wasm and facet sections/imports/exports/candidates/glue/wat/headers/details/tool_profile; text facets use UTF-16 code units and source forms remain exact. EVM summary returns all inferred interface candidates, carrier/decoded identities and explicit unknowns without hex or duplicate stdout. EVM views use kind evm and facet functions/bytecode_hex/stdout/stderr; function pages preserve the original candidate rows, and text offsets/limits count UTF-16 code units (hex has two characters per decoded byte). Exhausting candidate pages does not establish global interface completeness. Managed artifact summaries preserve complete identity/classification, metadata status, module/MVID, frameworks, coverage and limitations without unrelated inventories. Managed pages use kind managed and collection references/resources/attributes, retaining source coverage and build-local token scope; page exhaustion does not establish complete metadata. Native views select procedure, pseudocode, assembly, basic blocks, comments, callers/callees, references, unresolved calls, referenced strings/names, native API and high-pcode. Use nonnegative offset and a positive caller-selected limit (defaults 0 and 64); pseudocode uses UTF-16 code units, other native pages use rows. Returns artifact identity, parent Evidence ID, a distinct view digest, coverage, limitations and unknowns. Actual serialized size determines MCP transport admission; choose a smaller page or export the complete retained Evidence if needed.', + 'Inspect selected views of completed binary-layout, JavaScript application, native analyze_function, inspect_artifact, inventory_artifact, inspect_wasm_artifact, inspect_macho, inspect_evm_interface, inspect_managed_artifact, inspect_android_package, inspect_dart_aot or inspect_web_network_capture Evidence without repeating analysis. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. For a retained record use source: {kind: "retained-evidence", evidence_id: ""}. Use normalized_result.parent_evidence_id from a summary receipt. Source kind is required. Portable source uses {kind: "inline", evidence: }. For layout, JavaScript, inventory, WASM, Mach-O, EVM, managed artifact, Android package and Dart AOT sources, request a summary with view: {kind: "summary"}; summary is a view kind and has no facet. The WASM summary gives validation, counts, WAT digest/length, tool profile and execution/network states without reading WAT text. Select a layout mitigations/linkage facet, one section/symbol/module, or a stable page. Module pages include assets, bundled and source modules; use node_id for ambiguous or unavailable paths. Inventory uses view: {kind: "inventory", collection: "occurrences", logical_paths: ["package.json"], offset: 0, limit: 64}. Inventory views page nodes, occurrences, edges or integrity contradictions; occurrence logical_paths matches exact recorded strings (omitted means all available, [] means none). source_complete distinguishes complete inventories from partial source records; page exhaustion does not prove absence in a partial source. WASM views use kind wasm and facet sections/imports/exports/candidates/glue/wat/headers/details/tool_profile; text facets use UTF-16 code units and source forms remain exact. Mach-O views use kind macho and facet entrypoints/architectures/build_metadata/load_commands/dependencies/imports/exports/segments/provenance; source_exhaustive false means unreturned rows may exist. EVM summary returns all inferred interface candidates, carrier/decoded identities and explicit unknowns without hex or duplicate stdout. EVM views use kind evm and facet functions/bytecode_hex/stdout/stderr; function pages preserve the original candidate rows, and text offsets/limits count UTF-16 code units (hex has two characters per decoded byte). Exhausting candidate pages does not establish global interface completeness. Managed artifact summaries preserve complete identity/classification, metadata status, module/MVID, frameworks, coverage and limitations without unrelated inventories. Managed pages use kind managed and collection references/resources/attributes, retaining source coverage and build-local token scope; page exhaustion does not establish complete metadata. Android package summaries retain all declarations, engine identity and counts without decoded manifest text. Android manifest pages use kind android, facet manifest, offset and limit in UTF-16 code units; source_status preserves partial decoded coverage even when the retained text page is exhausted. Dart AOT summary preserves target/ABI/library identity, snapshot headers and pool lower bounds. String pages use kind dart-aot, collection package_uris/dart_uris/class_like_tokens; source coverage remains partial when the producer says partial. Native function Evidence requires kind native: view: {kind: "native", facet: "procedure"}. Facets select procedure, pseudocode, assembly, basic blocks, comments, callers/callees, references, unresolved calls, referenced strings/names, native API and high-pcode. Use nonnegative offset and a positive caller-selected limit (defaults 0 and 64); pseudocode uses UTF-16 code units, other native pages use rows. Returns artifact identity, parent Evidence ID, a distinct view digest, coverage, limitations and unknowns. Actual serialized size determines MCP transport admission; choose a smaller page or export the complete retained Evidence if needed.', inputSchema: inspectAnalysisViewInputSchema, outputSchema: evidenceResultOf(analysisViewResultSchema), examples: [ diff --git a/src/contracts/android/androidToolContracts.ts b/src/contracts/android/androidToolContracts.ts index df2cccf59..b555fcbea 100644 --- a/src/contracts/android/androidToolContracts.ts +++ b/src/contracts/android/androidToolContracts.ts @@ -1,7 +1,10 @@ +import { z } from "zod"; import { androidInputSchemas, + androidPackageInspectionInputSchema, androidResultSchemas, } from "../../domain/android/androidAnalysis.js"; +import { androidPackageSummaryViewResultSchema } from "../../domain/analysisView/analysisView.js"; import type { ToolContract } from "../toolContractTypes.js"; import { evidenceResultOf } from "../toolOutputSchemaPrimitives.js"; import { toolContractMetadata } from "../toolEffects.js"; @@ -12,14 +15,21 @@ export const ANDROID_TOOL_CONTRACTS = [ name: "inspect_android_package", ...toolContractMetadata("inspect_android_package"), description: - "Inspect a local APK's package, SDK and permission declarations, decoded manifest and class/resource counts. Does not execute the APK or verify signatures. Requires a caller-supplied JADX MCP JAR and JDK; uses one worker in an isolated temporary workspace.", + 'Inspect a local APK\'s package, SDK and permission declarations, decoded manifest and class/resource counts. Omitted or complete detail preserves the complete result. Select detail: "summary" for all declarations, engine identity and counts without decoded manifest text or duplicate raw calls; retains the complete source and returns a distinct linked inspect_analysis_view summary. Read the exact retained manifest with view kind android, facet manifest, offset and limit (UTF-16 code units), using normalized_result.parent_evidence_id. Does not execute the APK or verify signatures. Requires a caller-supplied JADX MCP JAR and JDK; uses one worker in an isolated temporary workspace.', kind: "android-provider", - inputSchema: androidInputSchemas.inspect_android_package, + inputSchema: androidPackageInspectionInputSchema, outputSchema: evidenceResultOf( - androidResultSchemas.inspect_android_package, + z.union([ + androidResultSchemas.inspect_android_package, + androidPackageSummaryViewResultSchema, + ]), ), examples: [ { title: "Inspect an APK", input: { path: "/tmp/Example.apk" } }, + { + title: "Inspect APK declarations with retained manifest", + input: { path: "/tmp/Example.apk", detail: "summary" }, + }, ], }, { diff --git a/src/contracts/applicationToolContracts.ts b/src/contracts/applicationToolContracts.ts index 9a592b536..ade29e960 100644 --- a/src/contracts/applicationToolContracts.ts +++ b/src/contracts/applicationToolContracts.ts @@ -218,7 +218,7 @@ export const APPLICATION_TOOL_CONTRACTS = [ name: "evaluate_reconstruction_coverage", ...toolContractMetadata("evaluate_reconstruction_coverage"), description: - "Evaluate inline evidence-backed reconstruction coverage against one named boundary. Missing ownership or inventory is partial; stale, weak, truncated, skipped, or unresolved proof is unknown; contradictions, failed proof, missing owners, and authority routing fail closed.", + "Evaluate inline evidence-backed reconstruction coverage against one named boundary. Missing ownership or inventory is partial; stale, weak, truncated, skipped, unresolved, or conflicting tied verifier proof is unknown; contradictions, failed proof, missing owners, and authority routing fail closed.", kind: "application", inputSchema: reconstructionCoverageEvaluationInputSchema, outputSchema: reconstructionClosureResultSchema, diff --git a/src/contracts/captures/captureToolContracts.ts b/src/contracts/captures/captureToolContracts.ts index 3e67b3b19..dca211511 100644 --- a/src/contracts/captures/captureToolContracts.ts +++ b/src/contracts/captures/captureToolContracts.ts @@ -1,3 +1,7 @@ +import { + inspectPacketFieldsInputSchema, + packetFieldsCatalogSchema, +} from "../../domain/captures/packetFields.js"; import { inspectPacketCaptureInputSchema, packetCaptureSchema, @@ -6,6 +10,22 @@ import type { ToolContract } from "../toolContractTypes.js"; import { toolContractMetadata } from "../toolEffects.js"; import { evidenceResultOf } from "../toolOutputSchemaPrimitives.js"; export const CAPTURE_TOOL_CONTRACTS = [ + { + name: "inspect_packet_fields", + ...toolContractMetadata("inspect_packet_fields"), + kind: "artifact-provider", + description: + "Discover protocols or field names/types registered in the installed TShark 4.x. kind defaults to protocols; use fields and optional exact protocol abbreviation to select field definitions. query searches name/abbreviation/description by case-insensitive literal substring. Return 50 matches by default with offset/limit paging; null requests all remaining within resource limits. Preserve type/base/bitmask/description, duplicate registrations, source line numbers, coverage, tool identity and diagnostics. Linux, REA_TSHARK_COMMAND and prlimit required. No capture file, interface or binary target needed. Registrations describe tool capabilities, not observed traffic. Each call rereads the glossary; use selected fields with inspect_packet_capture.", + inputSchema: inspectPacketFieldsInputSchema, + outputSchema: evidenceResultOf(packetFieldsCatalogSchema), + examples: [ + { title: "Find USB protocol registrations", input: { query: "usb" } }, + { + title: "Find the installed USB vendor-ID field", + input: { kind: "fields", protocol: "usb", query: "idVendor" }, + }, + ], + }, { name: "inspect_packet_capture", ...toolContractMetadata("inspect_packet_capture"), diff --git a/src/contracts/flutter/flutterToolContracts.ts b/src/contracts/flutter/flutterToolContracts.ts index 9a663d492..9ade1057b 100644 --- a/src/contracts/flutter/flutterToolContracts.ts +++ b/src/contracts/flutter/flutterToolContracts.ts @@ -1,7 +1,10 @@ +import { z } from "zod"; import { flutterInputSchemas, + dartAotInspectionInputSchema, flutterResultSchemas, } from "../../domain/flutter/flutterBuildAnalysis.js"; +import { dartAotSummaryViewResultSchema } from "../../domain/analysisView/analysisView.js"; import type { ToolContract } from "../toolContractTypes.js"; import { evidenceResultOf } from "../toolOutputSchemaPrimitives.js"; import { toolContractMetadata } from "../toolEffects.js"; @@ -27,10 +30,15 @@ export const FLUTTER_TOOL_CONTRACTS = [ name: "inspect_dart_aot", ...toolContractMetadata("inspect_dart_aot"), description: - "Inspect one ABI's Dart AOT snapshot inside an APK: the four snapshot symbols located through the ELF dynamic symbol table with their offsets, sizes, and section headers (magic, kind, length), the snapshot hash, and a bounded string-pool projection from the isolate data section (package: source URIs, dart: SDK URIs, identifier-like tokens) exposing the app's dependency structure. String-pool entries are byte-pattern observations, not deserialized cluster semantics; typed class and function recovery needs an SDK-specific parser. No engine, no execution.", + "Inspect one ABI's Dart AOT snapshot inside an APK: the four snapshot symbols located through the ELF dynamic symbol table with their offsets, sizes, and section headers (magic, kind, length), the snapshot hash, and a bounded string-pool projection from the isolate data section (package: source URIs, dart: SDK URIs, identifier-like tokens) exposing the app's dependency structure. String-pool entries are byte-pattern observations, not deserialized cluster semantics; typed class and function recovery needs an SDK-specific parser. Omitted or complete detail returns the full observation. Select detail: \"summary\" for target/ABI/library identity, all snapshot headers, coverage and original lower-bound counts without string lists. The complete source stays retained; read returned strings via inspect_analysis_view with parent_evidence_id and kind dart-aot, collection package_uris/dart_uris/class_like_tokens, offset and limit. Exhausting pages does not imply complete pool coverage. No engine, no execution.", kind: "flutter-provider", - inputSchema: flutterInputSchemas.inspect_dart_aot, - outputSchema: evidenceResultOf(flutterResultSchemas.inspect_dart_aot), + inputSchema: dartAotInspectionInputSchema, + outputSchema: evidenceResultOf( + z.union([ + flutterResultSchemas.inspect_dart_aot, + dartAotSummaryViewResultSchema, + ]), + ), examples: [ { title: "Inspect the preferred ABI's AOT snapshot", @@ -40,6 +48,10 @@ export const FLUTTER_TOOL_CONTRACTS = [ title: "Inspect one ABI explicitly", input: { path: "/targets/Example.apk", abi: "arm64-v8a" }, }, + { + title: "Inspect snapshot identity with retained string lists", + input: { path: "/targets/Example.apk", detail: "summary" }, + }, ], }, ] as const satisfies readonly ToolContract[]; diff --git a/src/contracts/native/nativeToolContracts.ts b/src/contracts/native/nativeToolContracts.ts index 093add61e..d45b62cc6 100644 --- a/src/contracts/native/nativeToolContracts.ts +++ b/src/contracts/native/nativeToolContracts.ts @@ -11,6 +11,7 @@ import { jsonValueSchema } from "../../domain/jsonValue.js"; import { localPathStringSchema } from "../../domain/localPath.js"; import { toolContractMetadata } from "../toolEffects.js"; import { requireOutputSchema } from "../toolOutputSchemaPrimitives.js"; +import { analysisDetailSchema } from "../analysisDetail.js"; const examples: Readonly>>> = { inspect_macho: {}, @@ -94,12 +95,19 @@ export const NATIVE_TOOL_CONTRACTS = [ ), native( "inspect_macho", - "Inspect Mach-O slices, load commands, imports, exports, dependencies, build metadata, segments, sections, permissions, and exact command provenance without launching Hopper.", - z.object({}), + "Inspect Mach-O slices, load commands, imports, exports, dependencies, build metadata, segments, sections, permissions, and exact command provenance without launching Hopper. Large frameworks return megabytes of symbols; detail summary retains the complete Evidence and returns slice identity with per-collection counts, then page rows with inspect_analysis_view kind macho.", + z.object({ + detail: analysisDetailSchema + .unwrap() + .optional() + .describe( + "Omitted or complete returns the complete inspection Evidence. summary retains that complete Evidence in this session and returns its inspect_analysis_view summary; use normalized_result.parent_evidence_id for macho pages.", + ), + }), ), native( "inspect_signature", - "Inspect the active artifact's code-signing identity, hashes, authorities, requirements, entitlements, hardened-runtime state, and exact command provenance.", + "Inspect the active Mach-O executable's code-signing identity, hashes, authorities, requirements, entitlements, hardened-runtime state, and exact command provenance.", z.object({}), ), native( diff --git a/src/contracts/officialToolContracts.ts b/src/contracts/officialToolContracts.ts index f9c024fe5..4750528ad 100644 --- a/src/contracts/officialToolContracts.ts +++ b/src/contracts/officialToolContracts.ts @@ -149,12 +149,12 @@ export const OFFICIAL_TOOL_CONTRACTS = [ ), official( "procedure_callees", - "Return the provider's resolved direct callees for one procedure identified by symbol or address. Unresolved indirect calls may be absent; use analyze_function and typed references to preserve available edge uncertainty.", + "Return the provider's resolved direct callees for one procedure identified by symbol or address. Unresolved indirect calls may be absent; use analyze_function and typed references to preserve available edge uncertainty. Ghidra terminal-call assumptions that may cut the analyzed body are reported in Evidence limitations.", z.object({ procedure, document }), ), official( "procedure_callers", - "Return the provider's resolved direct callers for one procedure identified by symbol or address. Results reflect completed static analysis and may omit unresolved indirect references.", + "Return the provider's resolved direct callers for one procedure identified by symbol or address. Results reflect completed static analysis and may omit unresolved indirect references. Ghidra terminal-call assumptions that may cut the analyzed body are reported in Evidence limitations.", z.object({ procedure, document }), ), official( @@ -181,7 +181,7 @@ export const OFFICIAL_TOOL_CONTRACTS = [ ), official( "procedure_pseudo_code", - "Decompile one analyzed procedure by symbol name or provider-normalized address. Returns provider-specific pseudocode, never original source or cross-provider text equivalence, and may return null; request procedure_assembly when instruction precision matters. Ghidra no-return assumptions may cut the analyzed body; Evidence limitations report observed terminal calls and available callee identities.", + "Decompile one analyzed procedure by symbol name or provider-normalized address. Ghidra also accepts file-offset:0x... selectors and falls back to a unique source-file mapping when a 0x-prefixed number resolves to no procedure; existing analysis addresses take precedence and ambiguous mappings fail explicitly. Returns provider-specific pseudocode, never original source or cross-provider text equivalence, and may return null; request procedure_assembly when instruction precision matters. Ghidra no-return assumptions may cut the analyzed body; Evidence limitations report observed terminal calls and available callee identities.", z.object({ procedure, document }), ), official( diff --git a/src/contracts/pcap/pcapToolContracts.ts b/src/contracts/pcap/pcapToolContracts.ts new file mode 100644 index 000000000..66f281b4e --- /dev/null +++ b/src/contracts/pcap/pcapToolContracts.ts @@ -0,0 +1,46 @@ +import { + pcapInputSchemas, + pcapResultSchemas, +} from "../../domain/pcap/pcapAnalysis.js"; +import type { ToolContract } from "../toolContractTypes.js"; +import { evidenceResultOf } from "../toolOutputSchemaPrimitives.js"; +import { toolContractMetadata } from "../toolEffects.js"; +import { + analysisDetailSchema, + analysisDetailOutputSchemaOf, +} from "../analysisDetail.js"; +import { pcapCaptureSummaryViewResultSchema } from "../../domain/analysisView/analysisView.js"; + +/** Packet-capture analysis served by caller-supplied tshark and capinfos. */ +export const PCAP_TOOL_CONTRACTS = [ + { + name: "inspect_pcap_client", + ...toolContractMetadata("inspect_pcap_client"), + description: + "Inspect the caller-selected tshark and capinfos binaries: selected commands, selection sources, and versions. Both tools must be usable. Reads no capture file. REA never installs the Wireshark suite; select binaries with REA_TSHARK_COMMAND and REA_CAPINFOS_COMMAND (absolute) or from PATH.", + kind: "pcap-provider", + inputSchema: pcapInputSchemas.inspect_pcap_client, + outputSchema: evidenceResultOf(pcapResultSchemas.inspect_pcap_client), + examples: [{ title: "Check the analysis tools", input: {} }], + }, + { + name: "inspect_pcap_capture", + ...toolContractMetadata("inspect_pcap_capture"), + description: + "Request detail summary for capture identity/counts, then page retained facts with inspect_analysis_view kind pcap. Default detail complete preserves inline delivery. Inspect one saved pcap or pcapng capture through an owned snapshot: metadata, exact packet count, SHA-256, IPv4/IPv6 address counts, DNS questions and address values from the same single-question response, TLS SNI, protocol distribution and HTTP URIs. All derived inventories are retained within declared resource budgets; unknown DNS attribution and excluded credentials remain explicit. Linux requires util-linux prlimit; Windows uses owned Job Objects. Requires caller-supplied capinfos and tshark; REA never captures traffic.", + kind: "pcap-provider", + inputSchema: pcapInputSchemas.inspect_pcap_capture.extend({ + detail: analysisDetailSchema, + }), + outputSchema: analysisDetailOutputSchemaOf( + pcapResultSchemas.inspect_pcap_capture, + pcapCaptureSummaryViewResultSchema, + ), + examples: [ + { + title: "Inspect a capture file", + input: { path: "/captures/session.pcap" }, + }, + ], + }, +] as const satisfies readonly ToolContract[]; diff --git a/src/contracts/signals/signalToolContracts.ts b/src/contracts/signals/signalToolContracts.ts index 4a055f12e..f545c452c 100644 --- a/src/contracts/signals/signalToolContracts.ts +++ b/src/contracts/signals/signalToolContracts.ts @@ -1,3 +1,7 @@ +import { + inspectSignalDecodersInputSchema, + signalDecodersSchema, +} from "../../domain/signals/signalDecoders.js"; import { decodeSignalRecordingInputSchema, signalRecordingSchema, @@ -6,6 +10,22 @@ import type { ToolContract } from "../toolContractTypes.js"; import { toolContractMetadata } from "../toolEffects.js"; import { evidenceResultOf } from "../toolOutputSchemaPrimitives.js"; export const SIGNAL_TOOL_CONTRACTS = [ + { + name: "inspect_signal_decoders", + ...toolContractMetadata("inspect_signal_decoders"), + kind: "artifact-provider", + description: + "Discover installed sigrok protocol decoders or inspect one decoder's channel/option help on Linux. Omit decoder to list ID/name pairs; supply its exact ID for literal help lines with source line numbers. query is a case-insensitive literal substring. Page matches with offset/limit (default 50; null returns all remaining). Reports tool identity, version, diagnostics and examined/matching coverage. Uses configured REA_SIGROK_COMMAND and prlimit; no recording, binary target, hardware scan or acquisition. Each call probes again; decoder modules are trusted, not fully fingerprinted. Use returned IDs/options with decode_signal_recording.", + inputSchema: inspectSignalDecodersInputSchema, + outputSchema: evidenceResultOf(signalDecodersSchema), + examples: [ + { title: "Find installed UART decoders", input: { query: "uart" } }, + { + title: "Read the installed UART baud-rate option", + input: { decoder: "uart", query: "baud" }, + }, + ], + }, { name: "decode_signal_recording", ...toolContractMetadata("decode_signal_recording"), diff --git a/src/contracts/toolContractTypes.ts b/src/contracts/toolContractTypes.ts index d7e19d37e..de5f53411 100644 --- a/src/contracts/toolContractTypes.ts +++ b/src/contracts/toolContractTypes.ts @@ -33,6 +33,7 @@ export const TOOL_KINDS = [ "browser-provider", "electron-provider", "runtime-provider", + "pcap-provider", "application", "session", ] as const; diff --git a/src/contracts/toolContracts.ts b/src/contracts/toolContracts.ts index 101cca17e..2fdac1fbe 100644 --- a/src/contracts/toolContracts.ts +++ b/src/contracts/toolContracts.ts @@ -12,6 +12,7 @@ import { MANAGED_TOOL_CONTRACTS } from "./managed/managedToolContracts.js"; import { FIRMWARE_TOOL_CONTRACTS } from "./firmware/firmwareToolContracts.js"; import { ANDROID_TOOL_CONTRACTS } from "./android/androidToolContracts.js"; import { APKTOOL_TOOL_CONTRACTS } from "./apktool/apktoolToolContracts.js"; +import { PCAP_TOOL_CONTRACTS } from "./pcap/pcapToolContracts.js"; import { FLUTTER_TOOL_CONTRACTS } from "./flutter/flutterToolContracts.js"; import { JEB_TOOL_CONTRACTS } from "./jeb/jebToolContracts.js"; import { ADB_TOOL_CONTRACTS } from "./adb/adbToolContracts.js"; @@ -46,6 +47,7 @@ export const TOOL_CONTRACTS = [ ...MANAGED_TOOL_CONTRACTS, ...ANDROID_TOOL_CONTRACTS, ...APKTOOL_TOOL_CONTRACTS, + ...PCAP_TOOL_CONTRACTS, ...FLUTTER_TOOL_CONTRACTS, ...JEB_TOOL_CONTRACTS, ...ADB_TOOL_CONTRACTS, diff --git a/src/contracts/toolEffects.test.ts b/src/contracts/toolEffects.test.ts index 836d190b1..fe18c96b3 100644 --- a/src/contracts/toolEffects.test.ts +++ b/src/contracts/toolEffects.test.ts @@ -44,6 +44,8 @@ const EXECUTION_WORDING_EXCEPTIONS: Readonly> = { search_android_classes: "states it does not execute the APK", inspect_apktool_client: "inspects the operator-selected apktool launcher; never installs it", + inspect_pcap_client: + "probes the operator-selected tshark and capinfos; never installs them", inspect_jeb_client: "talks to a caller-started JEB; never launches it", open_jeb_project: "the engine analyzes the file; REA does not execute it", inspect_adb_client: diff --git a/src/contracts/toolEffects.ts b/src/contracts/toolEffects.ts index 4b0d0b745..70a40c6a1 100644 --- a/src/contracts/toolEffects.ts +++ b/src/contracts/toolEffects.ts @@ -59,11 +59,21 @@ export const TOOL_EFFECTS: Readonly> = { writesFilesystem: true, launchesProcess: true, }), + inspect_packet_fields: effects({ + mutatesSession: true, + launchesProcess: true, + writesFilesystem: true, + }), inspect_packet_capture: effects({ mutatesSession: true, launchesProcess: true, writesFilesystem: true, }), + inspect_signal_decoders: effects({ + mutatesSession: true, + launchesProcess: true, + writesFilesystem: true, + }), decode_signal_recording: effects({ mutatesSession: true, launchesProcess: true, @@ -182,6 +192,16 @@ export const TOOL_EFFECTS: Readonly> = { launchesProcess: true, writesFilesystem: true, }), + inspect_pcap_client: effects({ + mutatesSession: true, + launchesProcess: true, + writesFilesystem: true, + }), + inspect_pcap_capture: effects({ + mutatesSession: true, + launchesProcess: true, + writesFilesystem: true, + }), identify_flutter_build: effects({ mutatesSession: true, writesFilesystem: true, diff --git a/src/contracts/toolOutputReferences.test.ts b/src/contracts/toolOutputReferences.test.ts index cda49f1d5..1cd890bf3 100644 --- a/src/contracts/toolOutputReferences.test.ts +++ b/src/contracts/toolOutputReferences.test.ts @@ -6,6 +6,34 @@ import { z } from "zod"; import { toolContract } from "./toolContracts.js"; import { toolOutputSchemaWithMetadata } from "./toolSchemaMetadata.js"; +import { createEvidence } from "../domain/evidence.js"; + +it.each(["draft-2020-12", "draft-07"] as const)( + "advertises the raw-only 6502 Evidence subject constraint in %s", + (target) => { + const valid = createEvidence( + { + path: "/fixture/image.bin", + sha256: "a".repeat(64), + format: "raw-binary", + architecture: "6502", + }, + { id: "fixture", name: "Fixture", version: "1" }, + { operation: "address_name", parameters: {}, result: "entry" }, + ); + const advertised = toolOutputSchemaWithMetadata( + toolContract("address_name"), + )["~standard"].jsonSchema.output({ target }); + const validate = new AjvJsonSchemaValidator().getValidator(advertised); + expect(validate(valid).valid).toBe(true); + expect( + validate({ + ...valid, + subject: { ...valid.subject, format: "pe" }, + }).valid, + ).toBe(false); + }, +); it("keeps dialect-specific validators distinct after JSON schema serialization", () => { const contract = { diff --git a/src/contracts/toolOutputSchemaGroups.ts b/src/contracts/toolOutputSchemaGroups.ts index e828486ba..76294e2db 100644 --- a/src/contracts/toolOutputSchemaGroups.ts +++ b/src/contracts/toolOutputSchemaGroups.ts @@ -1,6 +1,7 @@ import { z } from "zod"; import { artifactInventorySummaryViewResultSchema, + machoSummaryViewResultSchema, managedArtifactSummaryViewResultSchema, } from "../domain/analysisView/analysisView.js"; import { evidenceBundleSummarySchema } from "../domain/evidenceRecordSummary.js"; @@ -356,18 +357,20 @@ export const enhancedOutputSchemas: Readonly> = { binary_overview: evidenceResultOf( z.object({ document: z.string(), + image_base: z.string().min(1).optional(), segments: z.array( z.object({ name: z.string(), start: z.string(), end: z.string(), length: z.number().min(0), + image_base: z.string().exactOptional(), + address_space: z.string().exactOptional(), }), ), segment_count: z.number().int().min(0), procedure_count: z.number().int().min(0), string_count: z.number().int().min(0), - image_base: z.string().min(1).optional(), }), ), analyze_function: functionDossierOutput, @@ -381,7 +384,9 @@ export const nativeOutputSchemas: Readonly> = { observe_native_ui: evidenceResultOf(nativeUiResultSchema), capture_native_ui_scenario: evidenceResultOf(nativeUiResultSchema), observe_native_calls: evidenceResultOf(nativeCallObservationResultSchema), - inspect_macho: evidenceResultOf(inspectMachoSchema), + inspect_macho: evidenceResultOf( + z.union([inspectMachoSchema, machoSummaryViewResultSchema]), + ), inspect_signature: evidenceResultOf(inspectSignatureSchema), inspect_plist: evidenceResultOf(inspectPlistSchema), list_architectures: evidenceResultOf(listArchitecturesSchema), @@ -439,11 +444,11 @@ const lifecycleTargetFields = { path: z.string(), kind: targetKindSchema, sha256: z.string().regex(/^[a-f0-9]{64}$/u), - architecture: z.enum(BINARY_ARCHITECTURES).nullable(), }; const detectedLifecycleTargetSchema = z.object({ ...lifecycleTargetFields, format: targetFormatSchema.exclude(["raw-binary"]), + architecture: z.enum(BINARY_ARCHITECTURES).exclude(["6502"]).nullable(), }); const rawLifecycleTargetSchema = z.object({ ...lifecycleTargetFields, diff --git a/src/contracts/toolOutputSchemaPrimitives.ts b/src/contracts/toolOutputSchemaPrimitives.ts index 20c5517ee..e63a79ff5 100644 --- a/src/contracts/toolOutputSchemaPrimitives.ts +++ b/src/contracts/toolOutputSchemaPrimitives.ts @@ -24,6 +24,7 @@ import { } from "../domain/providerOperationHealth.js"; import { analysisErrorProjectionSchema } from "./errorSchemas.js"; import { relationshipCoverageSchema } from "../domain/relationshipCoverage.js"; +import { ZIP_PACKAGE_FORMATS } from "../domain/zipPackageFormat.js"; /** Specialize the single normalized result inside a complete Evidence record. */ export const evidenceResultOf = (schema: Schema) => @@ -59,11 +60,7 @@ export const targetFormatSchema = z.enum([ "dos-mz", "dos-com", "raw-binary", - "zip", - "ipa", - "apk", - "msix", - "appx", + ...ZIP_PACKAGE_FORMATS, "asar", "dmg", "pkg", diff --git a/src/contracts/webNetworkCaptureToolContracts.ts b/src/contracts/webNetworkCaptureToolContracts.ts index cb919172f..592dd9b54 100644 --- a/src/contracts/webNetworkCaptureToolContracts.ts +++ b/src/contracts/webNetworkCaptureToolContracts.ts @@ -4,7 +4,11 @@ import { } from "../domain/webNetworkCapture.js"; import type { ToolContract } from "./toolContractTypes.js"; import { toolContractMetadata } from "./toolEffects.js"; -import { evidenceResultOf } from "./toolOutputSchemaPrimitives.js"; +import { + analysisDetailSchema, + analysisDetailOutputSchemaOf, +} from "./analysisDetail.js"; +import { webNetworkCaptureSummaryViewResultSchema } from "../domain/analysisView/analysisView.js"; /** Historical inspection is distinct from live browser capture and runtime authority. */ export const WEB_NETWORK_CAPTURE_TOOL_CONTRACTS = [ @@ -12,11 +16,24 @@ export const WEB_NETWORK_CAPTURE_TOOL_CONTRACTS = [ name: "inspect_web_network_capture", ...toolContractMetadata("inspect_web_network_capture"), description: - "Inspect retained local HAR 1.2 or native mitmproxy capture records without contacting recorded URLs. Returns original ordinals, pointers/byte ranges, preserved producer fields, precise numeric representations and exact exposed binary bytes inline. Excludes known transport credentials and explicitly marked values. Decodes every record before optional ordinal selection; malformed or oversized captures return no partial success. Duplicate HAR members are invalid; prototype-named members such as __proto__ are preserved as ordinary own JSON members without prototype mutation. Uses pinned upstream parsers, owned temporary files and bounded offline processes. Native mitmproxy requires caller-supplied mitmdump 12.2.3 on Linux. Browser transaction IDs, runtime attribution and original bytes of HAR Unicode text remain unknown.", + "Request detail summary for capture identity and counts; then page retained records with inspect_analysis_view kind capture (preview or complete), avoiding repeated decoding. Default detail complete preserves existing inline delivery. Inspect retained local HAR 1.2, native mitmproxy, Fiddler SAZ or aioquic 1.5.0 JSON qlog 0.3 events without contacting recorded URLs. Returns original ordinals, pointers/byte ranges/archive member names, preserved producer fields, precise numeric representations and exact exposed binary bytes inline. Excludes known transport credentials and explicitly marked values. Decodes every record before optional ordinal selection; malformed or oversized captures return no partial success. Duplicate JSON members are invalid; prototype-named members such as __proto__ are preserved as ordinary own JSON members without prototype mutation. Qlog reports trace-scoped STREAM interval unions and endpoint loss declarations; it does not recover bodies, prove network loss or correlate PCAP. Other qlog layouts are unsupported. Uses pinned upstream parsers, owned temporary files and bounded offline processes. SAZ preserves HTTP fields, XML metadata and encoded body/message bytes without replay or HAR conversion; encrypted archives are unsupported. Native mitmproxy requires caller-supplied mitmdump 12.2.3 on Linux. Browser transaction IDs, runtime attribution and original bytes of HAR Unicode text remain unknown.", kind: "browser-provider", - inputSchema: inspectWebNetworkCaptureInputSchema, - outputSchema: evidenceResultOf(webNetworkCaptureSchema), + inputSchema: inspectWebNetworkCaptureInputSchema.extend({ + detail: analysisDetailSchema, + }), + outputSchema: analysisDetailOutputSchemaOf( + webNetworkCaptureSchema, + webNetworkCaptureSummaryViewResultSchema, + ), examples: [ + { + title: "Inspect retained aioquic qlog events", + input: { + capture_path: "/captures/origin.qlog", + format: "qlog", + record_ordinals: [0, 2], + }, + }, { title: "Inspect historical HAR entries", input: { diff --git a/src/domain/adb/adbDeviceAnalysis.ts b/src/domain/adb/adbDeviceAnalysis.ts index 17cdc223e..c524ebd02 100644 --- a/src/domain/adb/adbDeviceAnalysis.ts +++ b/src/domain/adb/adbDeviceAnalysis.ts @@ -30,6 +30,25 @@ const packageNameSchema = z "Package name must be dot-separated Java identifiers", ); +/** Text states emitted by adb's ConnectionState formatter. */ +export const ADB_DEVICE_STATES = [ + "no permissions", + "unauthorized", + "authorizing", + "offline", + "recovery", + "sideload", + "connecting", + "bootloader", + "device", + "unknown", + "host", + "rescue", + "detached", +] as const; +export const adbDeviceStateSchema = z.enum(ADB_DEVICE_STATES); +export type AdbDeviceState = (typeof ADB_DEVICE_STATES)[number]; + const keyStringSchema = z.string().min(1).max(256); const keyPattern = /^[A-Za-z0-9._\x2d]+$/u; @@ -412,7 +431,7 @@ const client = z.strictObject({ const device = z.strictObject({ serial: z.string().min(1), - state: z.string().min(1), + state: adbDeviceStateSchema, transport: z.enum(["usb", "tcp", "unknown"]), product: z.string().nullable(), model: z.string().nullable(), diff --git a/src/domain/analysisErrorCore.ts b/src/domain/analysisErrorCore.ts index 9e08f29a3..50db91ccd 100644 --- a/src/domain/analysisErrorCore.ts +++ b/src/domain/analysisErrorCore.ts @@ -100,18 +100,24 @@ export class AnalysisOutputError extends AnalysisError { export class AnalysisCapabilityUnavailableError extends AnalysisError { readonly _tag = "AnalysisCapabilityUnavailableError"; override readonly userMessage: string | undefined; + /** Incomplete cleanup must settle before the operation can be retried. */ + readonly retryable: boolean; constructor( readonly providerId: string, readonly operation: string, readonly reason: string, - options?: AnalysisErrorOptions & { readonly userMessage?: string }, + options?: AnalysisErrorOptions & { + readonly userMessage?: string; + readonly retryable?: boolean; + }, ) { super( `Provider ${providerId} cannot execute ${operation}: ${reason}`, options, ); this.userMessage = options?.userMessage; + this.retryable = options?.retryable === true && !this.cleanupIncomplete; } } diff --git a/src/domain/analysisErrorPresentation.ts b/src/domain/analysisErrorPresentation.ts index 215ed40fc..19c3a9002 100644 --- a/src/domain/analysisErrorPresentation.ts +++ b/src/domain/analysisErrorPresentation.ts @@ -1,6 +1,7 @@ import { AnalysisAccessDeniedError, AnalysisArtifactChangedError, + AnalysisCapabilityUnavailableError, AnalysisInputError, AnalysisUnsupportedTargetError, AnalysisResourceConstraintError, @@ -33,6 +34,8 @@ import type { AnalysisErrorProjection } from "./analysisErrorProjection.js"; export const analysisErrorRemediationAction = ( error: AnalysisError, ): string => { + if (error instanceof AnalysisCapabilityUnavailableError && error.retryable) + return "Review details.reason, then retry the same analysis after the condition clears."; if (error instanceof EvidenceFileError && error.published !== undefined) return "Use the published file at details.published_path and resolve the reported staging cleanup failure."; if (error instanceof AnalysisSnapshotMismatchError) @@ -112,6 +115,8 @@ export const analysisErrorRemediationAction = ( export const analysisErrorCategory = ( error: AnalysisError, ): AnalysisErrorProjection["category"] => { + if (error instanceof AnalysisCapabilityUnavailableError && error.retryable) + return "unavailable"; if ( error instanceof BinaryTargetError && error.constraint === "load_selection_rejected" @@ -220,6 +225,8 @@ export const analysisErrorUserMessage = (error: AnalysisError): string => { const hopperMessage = hopperErrorUserMessage(error); if (hopperMessage !== undefined) return hopperMessage; if (error.userMessage !== undefined) return error.userMessage; + if (error instanceof AnalysisCapabilityUnavailableError && error.retryable) + return "Required runtime controls are unavailable for this attempt."; const standardMessage = standardErrorMessage(error._tag); if (standardMessage !== undefined) return standardMessage; if (error instanceof ArtifactOperationError) return artifactMessage(error); diff --git a/src/domain/analysisErrorProjection.test.ts b/src/domain/analysisErrorProjection.test.ts index 467704b98..0766ca6e0 100644 --- a/src/domain/analysisErrorProjection.test.ts +++ b/src/domain/analysisErrorProjection.test.ts @@ -25,6 +25,91 @@ import { ProviderSelectionError } from "./providerSelectionError.js"; import { analysisErrorProjectionSchema } from "../contracts/errorSchemas.js"; import { nativeCallPartialObservationSchema } from "./native/nativeCallPartialObservation.js"; +describe("capability failure retry disposition", () => { + it.each([ + { options: {}, retryable: false, category: "unsupported_provider" }, + { + options: { retryable: false }, + retryable: false, + category: "unsupported_provider", + }, + { options: { retryable: true }, retryable: true, category: "unavailable" }, + ] as const)( + "preserves the producer's $retryable disposition", + ({ options, retryable, category }) => { + const reason = "Native self-test failed with a sharing violation"; + const projected = projectAnalysisError( + new AnalysisCapabilityUnavailableError( + "fixture-provider", + "inspect", + reason, + options, + ), + ); + expect(projected).toMatchObject({ + code: "capability_unavailable", + category, + retryable, + details: { reason }, + }); + if (retryable) { + expect(projected.message).toBe( + "Required runtime controls are unavailable for this attempt.", + ); + expect(projected.remediation.action).toContain( + "retry the same analysis", + ); + } + expect(analysisErrorProjectionSchema.safeParse(projected).success).toBe( + true, + ); + }, + ); + + it("preserves a caller-selected message for a retryable failure", () => { + expect( + projectAnalysisError( + new AnalysisCapabilityUnavailableError( + "fixture-provider", + "inspect", + "temporary failure", + { + retryable: true, + userMessage: "Selected runtime diagnostic", + }, + ), + ).message, + ).toBe("Selected runtime diagnostic"); + }); + + it("retains cleanup ownership before allowing a transient operation to retry", () => { + expect( + projectAnalysisError( + new AnalysisCapabilityUnavailableError( + "fixture-provider", + "inspect", + "temporary failure", + { + retryable: true, + cleanup: { + reason: "owned handle is still live", + resources: ["native-owner"], + }, + }, + ), + ), + ).toMatchObject({ + code: "cleanup_incomplete", + retryable: false, + details: { + cleanup: "incomplete", + resources: ["native-owner"], + execution_failure: "capability_unavailable", + }, + }); + }); +}); + it("projects a published file receipt alongside retained cleanup failure", () => { const projected = projectAnalysisError( new EvidenceFileError("write", "io", { diff --git a/src/domain/analysisErrorProjection.ts b/src/domain/analysisErrorProjection.ts index 723200e18..bfdf7eb42 100644 --- a/src/domain/analysisErrorProjection.ts +++ b/src/domain/analysisErrorProjection.ts @@ -83,6 +83,8 @@ export const projectAnalysisError = ( message: analysisErrorUserMessage(error), retryable: RETRYABLE_CODES.has(code) || + (error instanceof AnalysisCapabilityUnavailableError && + error.retryable) || (code === "resource_constraint" && error instanceof AnalysisResourceConstraintError && error.resource === "capacity"), diff --git a/src/domain/analysisProfile.test.ts b/src/domain/analysisProfile.test.ts index 1fb18f0ae..a39fb389d 100644 --- a/src/domain/analysisProfile.test.ts +++ b/src/domain/analysisProfile.test.ts @@ -65,4 +65,11 @@ describe("analysis profile commitments", () => { analysisProfileSchema.parse({ ...profile, digest: "0".repeat(64) }), ).toThrow(/digest/u); }); + + it("projects known provider identity fields at construction", () => { + const liveProvider = { ...PROVIDER, transient_status: "starting" }; + const profile = createAnalysisProfile(liveProvider, { loader: "default" }); + expect(profile.provider).toEqual(PROVIDER); + expect(analysisProfileSchema.parse(profile)).toEqual(profile); + }); }); diff --git a/src/domain/analysisProfile.ts b/src/domain/analysisProfile.ts index 6719e2ccd..f6e6e0004 100644 --- a/src/domain/analysisProfile.ts +++ b/src/domain/analysisProfile.ts @@ -7,13 +7,13 @@ import { jsonObjectSchema, type JsonValue } from "./jsonValue.js"; import { digestSchema } from "./../domain/digests.js"; /** Concrete provider identity committed by an analysis profile. */ -export const committedProviderSchema = z.object({ +export const committedProviderSchema = z.strictObject({ id: z.string().min(1), name: z.string().min(1), version: z.string().min(1), }); -const unsignedAnalysisProfileSchema = z.object({ +const unsignedAnalysisProfileSchema = z.strictObject({ provider: committedProviderSchema, parameters: jsonObjectSchema, }); @@ -40,7 +40,11 @@ export const createAnalysisProfile = ( parameters: Readonly>, ): AnalysisProfileCommitment => { const unsigned = unsignedAnalysisProfileSchema.parse({ - provider, + provider: { + id: provider.id, + name: provider.name, + version: provider.version, + }, parameters, }); return analysisProfileSchema.parse({ diff --git a/src/domain/analysisSnapshot.test.ts b/src/domain/analysisSnapshot.test.ts index ee3920be5..d83ddac30 100644 --- a/src/domain/analysisSnapshot.test.ts +++ b/src/domain/analysisSnapshot.test.ts @@ -52,7 +52,7 @@ const snapshotWithHistoricalEvidence = () => { }; return { current, - snapshot: { + snapshot: analysisSnapshotSchema.parse({ target, binding, entries: [entry], @@ -66,11 +66,32 @@ const snapshotWithHistoricalEvidence = () => { result: { summary: "unprofiled" }, }), ]), - } satisfies AnalysisSnapshot, + }), }; }; describe("analysis snapshot target identity", () => { + it("keeps 6502 confined to executable raw snapshots", () => { + const target = { + sha256: ANALYSIS_SNAPSHOT_TARGET.sha256, + kind: "executable", + architecture: "6502", + }; + expect( + analysisSnapshotSchema.shape.target.safeParse({ + ...target, + format: "pe", + }).success, + ).toBe(false); + expect( + analysisSnapshotSchema.shape.target.safeParse({ + ...target, + format: "raw-binary", + entry_point: "0x8001", + }).success, + ).toBe(true); + }); + it("partitions raw snapshots and query identities by the canonical entry", () => { const target: BinaryTarget = { path: "/fixture/image.bin", @@ -120,6 +141,48 @@ describe("analysis snapshot target identity", () => { }); }); +describe("analysis snapshot nested subject contract", () => { + it("rejects raw-only 6502 in nested detected execution subjects", () => { + const { snapshot } = snapshotWithHistoricalEvidence(); + const entry = snapshot.entries[0]; + if (entry === undefined) throw new Error("Fixture has no snapshot entry"); + const malformedSubject = { + ...entry.execution.subject, + format: "pe", + architecture: "6502", + }; + expect( + analysisSnapshotSchema.safeParse({ + ...snapshot, + entries: [ + { + ...entry, + execution: { + ...entry.execution, + subject: malformedSubject, + }, + }, + ], + }).success, + ).toBe(false); + expect( + analysisSnapshotSchema.safeParse({ + ...snapshot, + workflow_entries: [ + { + ...entry, + execution: { + ...entry.execution, + analysis_profile: snapshot.binding.analysis_profile, + subject: malformedSubject, + }, + }, + ], + }).success, + ).toBe(false); + }); +}); + describe("analysis snapshot contract", () => { it.each([ ["dos-mz", "x86"], diff --git a/src/domain/analysisSnapshot.ts b/src/domain/analysisSnapshot.ts index 57e876653..4e1af5fa8 100644 --- a/src/domain/analysisSnapshot.ts +++ b/src/domain/analysisSnapshot.ts @@ -41,7 +41,10 @@ import { digestSchema } from "./../domain/digests.js"; import { prefixedDigestSchema } from "./../domain/digests.js"; import { canonicalJsonDigestSteps } from "./canonicalJsonDigestSteps.js"; import { freezeJsonSnapshot } from "./immutableJson.js"; -import { rawBinaryEntryPointSchema } from "./rawBinary.js"; +import { + binaryArchitectureSchema, + rawBinaryEntryPointSchema, +} from "./rawBinary.js"; import { relationshipCoverageSchema, type RelationshipCoverage, @@ -73,20 +76,30 @@ const formatSchema = z.enum([ "source-map", ]); const kindSchema = z.enum(["executable", "database", "archive", "artifact"]); -const subjectSchema = z.object({ +const subjectBaseSchema = z.object({ path: localPathStringSchema, sha256: digestSchema, - format: z.enum([ - ...formatSchema.options, - "hopper", - "directory", - "file", - "unknown", - "javascript-bundle", - "entitlements", - ]), - architecture: architectureSchema.nullable(), }); +const subjectFormatSchema = z.enum([ + ...formatSchema.options, + "hopper", + "directory", + "file", + "unknown", + "javascript-bundle", + "entitlements", +]); +// Snapshot entries retain the same raw-only architecture boundary as targets. +const subjectSchema = z.discriminatedUnion("format", [ + subjectBaseSchema.extend({ + format: subjectFormatSchema.exclude(["raw-binary"]), + architecture: architectureSchema.exclude(["6502"]).nullable(), + }), + subjectBaseSchema.extend({ + format: z.literal("raw-binary"), + architecture: architectureSchema.nullable(), + }), +]); const targetIdentitySchema = z.object({ sha256: digestSchema, kind: kindSchema, @@ -97,9 +110,12 @@ const targetSchema = z.discriminatedUnion("format", [ format: z.enum( formatSchema.options.filter((format) => format !== "raw-binary"), ), + architecture: architectureSchema.exclude(["6502"]).nullable(), }), targetIdentitySchema.extend({ format: z.literal("raw-binary"), + kind: z.literal("executable"), + architecture: binaryArchitectureSchema, entry_point: rawBinaryEntryPointSchema, }), ]); @@ -332,18 +348,21 @@ const canonicalJson = (value: unknown): string => { }; /** Project a binary into the path-free identity used for cache invalidation. */ -export const snapshotTarget = ( - target: BinaryTarget, -): AnalysisSnapshotTarget => { - const identity = { - sha256: target.sha256, - kind: target.kind, - architecture: target.architecture ?? null, - }; - return target.format === "raw-binary" - ? { ...identity, format: target.format, entry_point: target.entryPoint } - : { ...identity, format: target.format }; -}; +export const snapshotTarget = (target: BinaryTarget): AnalysisSnapshotTarget => + target.format === "raw-binary" + ? { + sha256: target.sha256, + kind: target.kind, + architecture: target.architecture, + format: target.format, + entry_point: target.entryPoint, + } + : { + sha256: target.sha256, + kind: target.kind, + architecture: target.architecture ?? null, + format: target.format, + }; /** Build the immutable provider/profile binding used by a cached snapshot. */ export const snapshotBinding = ( @@ -475,10 +494,10 @@ export const createAnalysisSnapshotWorkflowEntry = (input: { subject: execution.subject === null ? null - : { + : subjectSchema.parse({ ...execution.subject, architecture: execution.subject.architecture ?? null, - }, + }), }, }; }; @@ -538,10 +557,10 @@ export const createAnalysisSnapshotEntry = (input: { subject: execution.subject === null ? null - : { + : subjectSchema.parse({ ...execution.subject, architecture: execution.subject.architecture ?? null, - }, + }), }, }; }; diff --git a/src/domain/analysisView/analysisView.test.ts b/src/domain/analysisView/analysisView.test.ts index 631f9d506..07e6909c3 100644 --- a/src/domain/analysisView/analysisView.test.ts +++ b/src/domain/analysisView/analysisView.test.ts @@ -1,6 +1,5 @@ import { expect, it } from "vitest"; -import { resolveJavaScriptSourceMapReference } from "../javascript/javascriptSourceMapPaths.js"; import { functionDossierSchema } from "../hopperValues.js"; import { ghidraFunctionDossier } from "../ghidraValues.fixture.js"; import { JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE } from "../../contracts/javascript/javascriptRuntimeReconciliationExample.js"; @@ -24,13 +23,14 @@ import { completeWithinViewCoverage, pageViewCoverage, } from "./analysisViewCoverage.js"; +import { sourceMapReference } from "../javascript/javascriptApplicationGraph.fixture.js"; import { javascriptModulePath } from "./javascriptApplicationView.js"; - import { inspectAnalysisViewInputSchema, projectAnalysisView, type AnalysisViewParent, } from "./analysisView.js"; + const layoutParent = (): AnalysisViewParent => { const evidence = analysisViewLayoutEvidence(); return { @@ -169,7 +169,8 @@ it("pages every retained dossier collection and the native API record", () => { | "comments" | "unresolved_calls" | "referenced_strings" - | "referenced_names", + | "referenced_names" + | "swift_string_literals", result: unknown = original, ) => { const view = projectAnalysisView(nativeParent(result), { @@ -195,6 +196,23 @@ it("pages every retained dossier collection and the native API record", () => { }), ).toMatchObject({ item: [{ text: "entry" }] }); expect(page("referenced_names")).toMatchObject({ coverage: { total: 0 } }); + const literal = { + source_address: "0x401004", + value: "SleepDisabled", + representation: "small", + address: null, + registers: ["x0", "x1"], + words: ["0x7369447065656c53", "0xed000064656c6261"], + }; + expect( + page("swift_string_literals", { + ...original, + swift_string_literals: [literal], + }), + ).toMatchObject({ item: [literal], coverage: { total: 1 } }); + expect(page("swift_string_literals")).toMatchObject({ + item: { available: false, reason: "not recorded" }, + }); const { unresolved_calls: _omitted, ...withoutUnresolved } = original; expect( page("unresolved_calls", { @@ -532,7 +550,7 @@ it("pages actual modules, including unknown and empty paths, without unrelated r observations: original.observations.map((observation) => ({ ...(kind === "source-module" ? { - source_map_reference: resolveJavaScriptSourceMapReference( + source_map_reference: sourceMapReference( "", null, "renderer.js", diff --git a/src/domain/analysisView/analysisView.ts b/src/domain/analysisView/analysisView.ts index 4ab9c77da..b08001e35 100644 --- a/src/domain/analysisView/analysisView.ts +++ b/src/domain/analysisView/analysisView.ts @@ -55,7 +55,43 @@ import { wasmViewItemSchema, wasmSummarySchema, } from "./wasmArtifactView.js"; +import { + machoSummarySchema, + machoViewContextSchema, + machoViewItemSchema, + machoViewRequestSchema, + projectMachoView, +} from "./machoView.js"; +import { + projectWebNetworkCaptureView, + captureViewRequestSchema, + captureViewItemsSchema, + captureSummarySchema, +} from "./webNetworkCaptureView.js"; import type { WebNetworkApiSurfaceAccumulator } from "./WebNetworkApiSurfaceAccumulator.js"; +import { + pcapViewRequestSchema, + pcapViewPageSchema, + pcapSummarySchema, + projectPcapCaptureView, +} from "./pcapCaptureView.js"; +import { + qlogViewRequestSchema, + qlogViewPageSchema, + qlogViewContextSchema, +} from "./qlogCaptureView.js"; +import { + projectAndroidPackageView, + androidPackageSummarySchema, + androidViewRequestSchema, + androidManifestViewItemSchema, +} from "./androidPackageView.js"; +import { + projectDartAotView, + dartAotSummarySchema, + dartAotViewRequestSchema, + dartAotViewContextSchema, +} from "./dartAotView.js"; /** Public name of the selected-view workflow. */ export const INSPECT_ANALYSIS_VIEW_OPERATION = "inspect_analysis_view" as const; @@ -124,6 +160,8 @@ const nativeViewSchema = z.strictObject({ "unresolved_calls", "referenced_strings", "referenced_names", + "swift_string_literals", + "swift_string_literal_omissions", "native_api", "value_flow_summary", "value_flow_operations", @@ -167,9 +205,15 @@ export const analysisViewRequestSchema = z.discriminatedUnion("kind", [ nativeViewSchema, inventoryViewSchema, wasmViewRequestSchema, + machoViewRequestSchema, evmViewRequestSchema, managedViewRequestSchema, + androidViewRequestSchema, apiSurfaceViewSchema, + captureViewRequestSchema, + qlogViewRequestSchema, + pcapViewRequestSchema, + dartAotViewRequestSchema, ]); // The application owns admission and authenticated Evidence parsing. SDK and @@ -209,7 +253,7 @@ const artifactSchema = z.strictObject({ path: z .string() .describe("Exact recorded artifact path, including an empty string."), - sha256: digestSchema, + sha256: digestSchema.nullable(), }); const layoutSummarySchema = z.strictObject({ @@ -281,8 +325,13 @@ const viewResultBase = { "inspect_artifact", "inventory_artifact", "inspect_wasm_artifact", + "inspect_macho", "inspect_evm_interface", "inspect_managed_artifact", + "inspect_web_network_capture", + "inspect_android_package", + "inspect_pcap_capture", + "inspect_dart_aot", ]), parent_digest: digestSchema, view_digest: digestSchema, @@ -326,6 +375,13 @@ export const wasmArtifactSummaryViewResultSchema = z.strictObject({ summary: wasmSummarySchema, }); +/** Exact slice identity and collection counts for Mach-O's selected delivery. */ +export const machoSummaryViewResultSchema = z.strictObject({ + ...summaryViewResultBase, + parent_operation: z.literal("inspect_macho"), + summary: machoSummarySchema, +}); + /** Exact inferred-interface summary for EVM's caller-selected delivery. */ export const evmInterfaceSummaryViewResultSchema = z.strictObject({ ...summaryViewResultBase, @@ -340,6 +396,33 @@ export const managedArtifactSummaryViewResultSchema = z.strictObject({ summary: managedSummarySchema, }); +/** Exact historical capture summary, with no body or member-inventory payload. */ +export const webNetworkCaptureSummaryViewResultSchema = z.strictObject({ + ...summaryViewResultBase, + parent_operation: z.literal("inspect_web_network_capture"), + summary: captureSummarySchema, +}); + +/** Exact package declarations with retained decoded manifest text. */ +export const androidPackageSummaryViewResultSchema = z.strictObject({ + ...summaryViewResultBase, + parent_operation: z.literal("inspect_android_package"), + summary: androidPackageSummarySchema, +}); + +export const pcapCaptureSummaryViewResultSchema = z.strictObject({ + ...summaryViewResultBase, + parent_operation: z.literal("inspect_pcap_capture"), + summary: pcapSummarySchema, +}); + +/** Exact snapshot and pool-coverage carrier for selected AOT delivery. */ +export const dartAotSummaryViewResultSchema = z.strictObject({ + ...summaryViewResultBase, + parent_operation: z.literal("inspect_dart_aot"), + summary: dartAotSummarySchema, +}); + /** Projected facts for one selected view of retained analysis Evidence. */ export const analysisViewResultSchema = z.discriminatedUnion("kind", [ z.strictObject({ @@ -360,10 +443,38 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [ javascriptSummarySchema, inventorySummarySchema, wasmSummarySchema, + machoSummarySchema, evmSummarySchema, managedSummarySchema, + captureSummarySchema, + androidPackageSummarySchema, + pcapSummarySchema, + dartAotSummarySchema, ]), }), + z.strictObject({ + kind: z.literal("capture"), + view: captureViewRequestSchema, + items: captureViewItemsSchema, + ...viewResultBase, + parent_operation: z.literal("inspect_web_network_capture"), + }), + z.strictObject({ + kind: z.literal("qlog"), + view: qlogViewRequestSchema, + page: qlogViewPageSchema, + context: qlogViewContextSchema, + ...viewResultBase, + parent_operation: z.literal("inspect_web_network_capture"), + }), + z.strictObject({ + kind: z.literal("pcap"), + view: pcapViewRequestSchema, + page: pcapViewPageSchema, + source_coverage: z.enum(["complete", "partial"]), + ...viewResultBase, + parent_operation: z.literal("inspect_pcap_capture"), + }), z.strictObject({ kind: z.literal("facet"), view: facetViewSchema, @@ -395,6 +506,14 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [ item: wasmViewItemSchema, ...viewResultBase, }), + z.strictObject({ + kind: z.literal("macho"), + view: machoViewRequestSchema, + item: machoViewItemSchema, + ...viewResultBase, + ...machoViewContextSchema.shape, + parent_operation: z.literal("inspect_macho"), + }), z.strictObject({ kind: z.literal("evm"), view: evmViewRequestSchema, @@ -410,6 +529,21 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [ ...managedViewContextSchema.shape, parent_operation: z.literal("inspect_managed_artifact"), }), + z.strictObject({ + kind: z.literal("android"), + view: androidViewRequestSchema, + item: androidManifestViewItemSchema, + ...viewResultBase, + parent_operation: z.literal("inspect_android_package"), + }), + z.strictObject({ + kind: z.literal("dart-aot"), + view: dartAotViewRequestSchema, + items: z.array(z.string()), + ...viewResultBase, + ...dartAotViewContextSchema.shape, + parent_operation: z.literal("inspect_dart_aot"), + }), z.strictObject({ kind: z.literal("inventory"), view: inventoryViewSchema, @@ -476,10 +610,25 @@ export type UnsignedAnalysisView = "view_digest" > | Omit, "view_digest"> + | Omit, "view_digest"> | Omit, "view_digest"> + | Omit, "view_digest"> + | Omit, "view_digest"> + | Omit< + Extract, + "view_digest" + > | Omit< Extract, "view_digest" + > + | Omit< + Extract, + "view_digest" + > + | Omit< + Extract, + "view_digest" >; /** Attach the content digest that must change when projected bytes change. */ @@ -525,7 +674,7 @@ const unsupportedParent = ( new AnalysisUnsupportedTargetError( INSPECT_ANALYSIS_VIEW_OPERATION, parentArtifactPath(parent), - `Parent Evidence operation ${JSON.stringify(parent.operation)} does not support analysis views. Use completed binary layout, JavaScript application, native function, artifact inventory/inspection, WASM, EVM, managed artifact, or web network capture Evidence.`, + `Parent Evidence operation ${JSON.stringify(parent.operation)} does not support analysis views. Use completed binary layout, JavaScript application, native function, artifact inventory/inspection, WASM, Mach-O, EVM, managed artifact, Android package, Dart AOT, or web network capture Evidence.`, ); const parentArtifactPath = (parent: AnalysisViewParent): string => { @@ -552,8 +701,11 @@ const parentArtifactPath = (parent: AnalysisViewParent): string => { /** Named source owners for views that need no extra decoder or accumulator. */ const retainedViewProjectors = new Map([ ["inspect_wasm_artifact", projectWasmArtifactView], + ["inspect_macho", projectMachoView], ["inspect_evm_interface", projectEvmInterfaceView], ["inspect_managed_artifact", projectManagedArtifactView], + ["inspect_android_package", projectAndroidPackageView], + ["inspect_dart_aot", projectDartAotView], ["inspect_artifact", projectArtifactInventoryView], ["inventory_artifact", projectArtifactInventoryView], ]); @@ -575,10 +727,7 @@ export const projectAnalysisView = ( }, ]), ); - if ( - view.kind === "api-surface" || - parent.operation === "inspect_web_network_capture" - ) { + if (view.kind === "api-surface") { const projected = projectWebNetworkApiSurfaceView( parent, view, @@ -586,6 +735,14 @@ export const projectAnalysisView = ( ); return projected.ok ? ok(sealAnalysisView(projected.value)) : projected; } + if (parent.operation === "inspect_web_network_capture") { + const projected = projectWebNetworkCaptureView(parent, view); + return projected.ok ? ok(sealAnalysisView(projected.value)) : projected; + } + if (parent.operation === "inspect_pcap_capture") { + const projected = projectPcapCaptureView(parent, view); + return projected.ok ? ok(sealAnalysisView(projected.value)) : projected; + } if (parent.operation === "inspect_binary_layout") { const layout = binaryLayoutSchema.safeParse(parent.normalizedResult); if (!layout.success) diff --git a/src/domain/analysisView/androidPackageView.ts b/src/domain/analysisView/androidPackageView.ts new file mode 100644 index 000000000..caf948c10 --- /dev/null +++ b/src/domain/analysisView/androidPackageView.ts @@ -0,0 +1,136 @@ +import { z } from "zod"; +import type { AnalysisError } from "../analysisErrorBase.js"; +import { androidResultSchemas } from "../android/androidAnalysis.js"; +import { analysisInputErrorFromIssues } from "../inputIssueProjection.js"; +import { err, ok, type Result } from "../result.js"; +import { + analysisViewInputError, + type AnalysisViewParent, + type AnalysisViewRequest, + type UnsignedAnalysisView, +} from "./analysisView.js"; +import { completeWithinViewCoverage } from "./analysisViewCoverage.js"; +import { analysisTextPage } from "./analysisViewText.js"; + +const packageSchema = androidResultSchemas.inspect_android_package; + +/** All package declarations remain inline; decoded manifest text stays retained. */ +export const androidPackageSummarySchema = packageSchema.extend({ + manifest: packageSchema.shape.manifest.omit({ text: true }).extend({ + text_code_units: z.number().int().nonnegative(), + text_included: z.literal(false), + }), +}); + +/** Exact decoded text; offsets and limits count UTF-16 code units. */ +export const androidViewRequestSchema = z.strictObject({ + kind: z.literal("android"), + facet: z.literal("manifest"), + offset: z.number().int().nonnegative().default(0), + limit: z.number().int().positive(), +}); + +export const androidManifestViewItemSchema = z.strictObject({ + text: z.string(), + unit: z.literal("utf16-code-units"), + source_status: packageSchema.shape.manifest.shape.status, + reported_total_bytes: packageSchema.shape.manifest.shape.reported_total_bytes, +}); + +/** Project an authenticated package record without reopening the APK or engine. */ +export const projectAndroidPackageView = ( + parent: AnalysisViewParent, + view: AnalysisViewRequest, +): Result => { + if (view.kind !== "summary" && view.kind !== "android") + return err( + analysisViewInputError([ + { + path: ["view"], + reason: "invalid_value", + message: + "Android package Evidence supports summary and manifest views.", + }, + ]), + ); + const parsed = packageSchema.safeParse(parent.normalizedResult); + if (!parsed.success) + return err( + analysisInputErrorFromIssues( + "inspect_analysis_view", + parsed.error.issues.map((issue) => ({ + ...issue, + path: ["source", "normalized_result", ...issue.path], + })), + { source: { normalized_result: parent.normalizedResult } }, + { cause: parsed.error }, + ), + ); + // Engine artifact_sha256 identifies the JAR, never the APK. A missing + // authenticated APK subject cannot be repaired using the engine identity. + if (parent.subject == null) + return err( + analysisViewInputError([ + { + path: ["source", "subject"], + reason: "invalid_value", + message: + "Android package views require the recorded APK subject identity.", + }, + ]), + ); + const report = parsed.data; + const shared = { + parent_evidence_id: parent.evidenceId, + parent_operation: "inspect_android_package" as const, + parent_digest: parent.evidenceId.slice(3), + artifact: { + path: parent.subject.local_path, + sha256: parent.subject.digest.sha256, + }, + limitations: [...parent.limitations], + unknowns: [ + "APK signatures were not verified; package declarations do not establish authenticity or runtime behavior.", + ...(report.manifest.status === "partial" + ? [ + "The decoded manifest is partial; exhausting retained text does not establish complete manifest coverage.", + ] + : []), + ], + }; + if (view.kind === "summary") { + const { text, ...manifest } = report.manifest; + return ok({ + kind: "summary", + view, + ...shared, + summary: { + ...report, + manifest: { + ...manifest, + text_code_units: text.length, + text_included: false, + }, + }, + coverage: completeWithinViewCoverage(1, 1), + }); + } + const page = analysisTextPage( + report.manifest.text, + view.offset, + view.limit, + "Manifest", + ); + if (!page.ok) return page; + return ok({ + kind: "android", + view, + ...shared, + item: { + ...page.value.item, + source_status: report.manifest.status, + reported_total_bytes: report.manifest.reported_total_bytes, + }, + coverage: page.value.coverage, + }); +}; diff --git a/src/domain/analysisView/dartAotView.ts b/src/domain/analysisView/dartAotView.ts new file mode 100644 index 000000000..4d66aa190 --- /dev/null +++ b/src/domain/analysisView/dartAotView.ts @@ -0,0 +1,123 @@ +import { z } from "zod"; +import type { AnalysisError } from "../analysisErrorBase.js"; +import { flutterResultSchemas } from "../flutter/flutterBuildAnalysis.js"; +import { analysisInputErrorFromIssues } from "../inputIssueProjection.js"; +import { err, ok, type Result } from "../result.js"; +import { + analysisViewInputError, + type AnalysisViewParent, + type AnalysisViewRequest, + type UnsignedAnalysisView, +} from "./analysisView.js"; +import { + completeWithinViewCoverage, + pageViewCoverage, +} from "./analysisViewCoverage.js"; + +const aotSchema = flutterResultSchemas.inspect_dart_aot; + +/** Snapshot identity and original coverage, without unrelated string inventories. */ +export const dartAotSummarySchema = aotSchema.extend({ + string_pool: aotSchema.shape.string_pool + .omit({ package_uris: true, dart_uris: true, class_like_tokens: true }) + .extend({ + returned_counts: z.strictObject({ + package_uris: z.number().int().nonnegative(), + dart_uris: z.number().int().nonnegative(), + class_like_tokens: z.number().int().nonnegative(), + }), + }), +}); + +/** Pages of the producer-returned strings, preserving original order and scope. */ +export const dartAotViewRequestSchema = z.strictObject({ + kind: z.literal("dart-aot"), + collection: z.enum(["package_uris", "dart_uris", "class_like_tokens"]), + offset: z.number().int().nonnegative().default(0), + limit: z.number().int().positive().default(64), +}); + +export const dartAotViewContextSchema = z.strictObject({ + abi: aotSchema.shape.abi, + libapp: aotSchema.shape.libapp, + source_coverage: aotSchema.shape.coverage, + string_pool: dartAotSummarySchema.shape.string_pool, +}); + +/** Reuse authenticated AOT Evidence without opening the APK or rescanning its ELF. */ +export const projectDartAotView = ( + parent: AnalysisViewParent, + view: AnalysisViewRequest, +): Result => { + if (view.kind !== "summary" && view.kind !== "dart-aot") + return err( + analysisViewInputError([ + { + path: ["view"], + reason: "invalid_value", + message: "Dart AOT Evidence supports summary and string-pool pages.", + }, + ]), + ); + const parsed = aotSchema.safeParse(parent.normalizedResult); + if (!parsed.success) + return err( + analysisInputErrorFromIssues( + "inspect_analysis_view", + parsed.error.issues.map((issue) => ({ + ...issue, + path: ["source", "normalized_result", ...issue.path], + })), + { source: { normalized_result: parent.normalizedResult } }, + { cause: parsed.error }, + ), + ); + const report = parsed.data; + const { package_uris, dart_uris, class_like_tokens, ...pool } = + report.string_pool; + const string_pool = { + ...pool, + returned_counts: { + package_uris: package_uris.length, + dart_uris: dart_uris.length, + class_like_tokens: class_like_tokens.length, + }, + }; + const shared = { + parent_evidence_id: parent.evidenceId, + parent_operation: "inspect_dart_aot" as const, + parent_digest: parent.evidenceId.slice(3), + artifact: { path: report.target.path, sha256: report.target.sha256 }, + limitations: [...parent.limitations], + unknowns: [ + "String-pool entries are byte-pattern observations, not recovered typed classes or functions.", + "Counts ending in _at_least remain lower bounds; exhausting returned strings does not establish a complete snapshot inventory.", + ...(parent.subject == null + ? [ + "The parent has no recorded subject; its normalized target identity cannot be cross-checked against a subject digest.", + ] + : []), + ], + }; + if (view.kind === "summary") + return ok({ + kind: "summary", + view, + ...shared, + summary: { ...report, string_pool }, + coverage: completeWithinViewCoverage(1, 1), + }); + const collection = report.string_pool[view.collection]; + const items = collection.slice(view.offset, view.offset + view.limit); + return ok({ + kind: "dart-aot", + view, + ...shared, + items, + abi: report.abi, + libapp: report.libapp, + source_coverage: report.coverage, + string_pool, + coverage: pageViewCoverage(view.offset, items.length, collection.length), + }); +}; diff --git a/src/domain/analysisView/evmInterfaceView.ts b/src/domain/analysisView/evmInterfaceView.ts index a7b343291..d9f3994d4 100644 --- a/src/domain/analysisView/evmInterfaceView.ts +++ b/src/domain/analysisView/evmInterfaceView.ts @@ -76,6 +76,7 @@ export const projectEvmInterfaceView = ( const report = parsed.data; if ( parent.subject == null || + parent.subject.format !== "file" || parent.subject.local_path !== report.artifact.path || parent.subject.digest.sha256 !== report.artifact.sha256 ) diff --git a/src/domain/analysisView/javascriptApplicationView.ts b/src/domain/analysisView/javascriptApplicationView.ts index 5e5f9bb88..41a22f8d9 100644 --- a/src/domain/analysisView/javascriptApplicationView.ts +++ b/src/domain/analysisView/javascriptApplicationView.ts @@ -241,22 +241,14 @@ export const projectJavaScriptApplicationView = ( analysis: JavaScriptApplicationAnalysisResult, view: Exclude, ): Result => { + if (view.kind !== "summary" && view.kind !== "item" && view.kind !== "page") + return err( + jsIncompatible( + "JavaScript application Evidence supports summary, module item or module page views.", + ), + ); const shared = parentFields(parent, analysis); const modules = moduleEntries(analysis.graph.nodes); - if (view.kind === "native") - return err( - jsIncompatible("Native views require analyze_function Evidence."), - ); - if (view.kind === "inventory") - return err(jsIncompatible("Inventory views require artifact Evidence.")); - if (view.kind === "wasm") - return err( - jsIncompatible("WASM views require inspect_wasm_artifact Evidence."), - ); - if (view.kind === "evm") - return err( - jsIncompatible("EVM views require inspect_evm_interface Evidence."), - ); if (view.kind === "summary") return ok({ kind: "summary", @@ -291,12 +283,6 @@ export const projectJavaScriptApplicationView = ( coverage: completeCoverage(modules.length, analysis.graph.nodes.length), ...shared, }); - if (view.kind === "facet") - return err( - jsIncompatible( - "facet views apply to inspect_binary_layout Evidence (mitigations or linkage).", - ), - ); if (view.kind === "item") { if (view.collection !== "modules") return err( @@ -314,7 +300,7 @@ export const projectJavaScriptApplicationView = ( ...shared, }); } - if (view.collection !== "modules") + if (view.kind !== "page" || view.collection !== "modules") return err( jsIncompatible( "Section and symbol collections apply to inspect_binary_layout Evidence.", diff --git a/src/domain/analysisView/machoView.test.ts b/src/domain/analysisView/machoView.test.ts new file mode 100644 index 000000000..2018438be --- /dev/null +++ b/src/domain/analysisView/machoView.test.ts @@ -0,0 +1,246 @@ +import { describe, expect, it } from "vitest"; + +import { + projectAnalysisView, + type AnalysisViewParent, +} from "./analysisView.js"; + +const symbol = (name: string) => ({ + name, + address: null, + weak: null, + reexport: null, + source: "libSystem", +}); +const coverage = (items: T[], exhaustive = true) => ({ + items, + total: exhaustive ? items.length : null, + exhaustive, + limitations: exhaustive ? [] : ["nm output may omit stripped symbols."], +}); +const report = { + format: "mach-o", + selected_architecture: { name: "arm64", container_file_offset: 0 }, + endian: null, + word_size: 64, + file_type: "2", + flags: ["0x00200085"], + uuid: "4C4C44E9-5555-3144-A1E9-D858E24A1CCE", + entrypoints: coverage([{ file_offset: 16384 }]), + architectures: coverage([ + { + name: "arm64", + cpu_type: "16777228", + cpu_subtype: "0", + file_offset: 0, + size: 4096, + alignment: 14, + }, + ]), + build_metadata: coverage([ + { platform: "1", minimum_os: "14.0", sdk: "15.0", tools: [] }, + ]), + load_commands: coverage([{ index: 0, kind: "LC_MAIN", fields: {} }]), + dependencies: coverage([ + { + path: "/usr/lib/libSystem.B.dylib", + kind: "LC_LOAD_DYLIB", + current_version: "1351.0.0", + compatibility_version: "1.0.0", + }, + ]), + imports: coverage( + ["_malloc", "_free", "_open"].map((name) => symbol(name)), + false, + ), + exports: coverage([symbol("__mh_execute_header")]), + segments: coverage([ + { + name: "__TEXT", + vm_address: "0x100000000", + vm_size: 16384, + file_offset: 0, + file_size: 16384, + maximum_permissions: { + read: true, + write: false, + execute: true, + raw: "5", + }, + initial_permissions: { + read: true, + write: false, + execute: true, + raw: "5", + }, + sections: coverage([]), + }, + ]), + provenance: [ + { + tool: "nm", + command: ["/usr/bin/nm", "$ARTIFACT"], + tool_version: null, + version_reason: "fixture", + executable_sha256: "a".repeat(64), + exit: { code: 0, signal: null }, + stdout_bytes: 10, + stderr_bytes: 0, + }, + ], + limitations: ["Imports and exports combine dyld_info and nm."], +}; +const parent = ( + normalizedResult: unknown = report, + subject: AnalysisViewParent["subject"] = { + name: "tool", + digest: { sha256: "b".repeat(64) }, + format: "mach-o", + architecture: "arm64", + local_path: "/bin/tool", + }, +): AnalysisViewParent => ({ + evidenceId: `ev_${"c".repeat(64)}`, + operation: "inspect_macho", + normalizedResult, + subject, + limitations: ["Parent limitation."], +}); + +describe("Mach-O analysis views", () => { + it("summarizes slice identity and collection counts without rows", () => { + const projected = projectAnalysisView(parent(), { kind: "summary" }); + if (!projected.ok) throw projected.error; + expect(projected.value).toMatchObject({ + kind: "summary", + parent_operation: "inspect_macho", + artifact: { path: "/bin/tool", sha256: "b".repeat(64) }, + summary: { + selected_architecture: { name: "arm64", container_file_offset: 0 }, + uuid: report.uuid, + collections: { + imports: { returned: 3, total: null, exhaustive: false }, + exports: { returned: 1, total: 1, exhaustive: true }, + segments: { returned: 1, total: 1, exhaustive: true }, + }, + provenance_count: 1, + limitation_count: 3, + }, + coverage: { status: "complete-within-view", examined: 10, total: 10 }, + limitations: [ + "Imports and exports combine dyld_info and nm.", + "Parent limitation.", + "nm output may omit stripped symbols.", + ], + unknowns: [expect.stringContaining("exhaustive imports;")], + }); + expect(JSON.stringify(projected.value)).not.toContain("_malloc"); + }); + + it("pages original rows with the producer's own coverage", () => { + const first = projectAnalysisView(parent(), { + kind: "macho", + facet: "imports", + offset: 1, + limit: 1, + }); + if (!first.ok) throw first.error; + expect(first.value).toMatchObject({ + kind: "macho", + item: [symbol("_free")], + source_total: null, + source_exhaustive: false, + coverage: { status: "page", examined: 1, total: 3, next_offset: 2 }, + }); + expect(first.value.limitations).toContain( + "nm output may omit stripped symbols.", + ); + // Rows whose keys include an earlier row shape's keys stay whole. + for (const facet of ["architectures", "segments"] as const) { + const page = projectAnalysisView(parent(), { + kind: "macho", + facet, + offset: 0, + limit: 64, + }); + if (!page.ok) throw page.error; + expect(page.value.kind === "macho" && page.value.item).toEqual( + report[facet].items, + ); + } + const provenance = projectAnalysisView(parent(), { + kind: "macho", + facet: "provenance", + offset: 0, + limit: 64, + }); + if (!provenance.ok) throw provenance.error; + expect(provenance.value).toMatchObject({ + item: report.provenance, + source_total: 1, + source_exhaustive: true, + coverage: { exhausted: true, next_offset: null }, + }); + const past = projectAnalysisView(parent(), { + kind: "macho", + facet: "exports", + offset: 5, + limit: 64, + }); + if (!past.ok) throw past.error; + expect(past.value.coverage).toMatchObject({ + status: "empty", + exhausted: true, + }); + }); + + it("keeps a missing subject unknown instead of inventing identity", () => { + const projected = projectAnalysisView(parent(report, null), { + kind: "summary", + }); + if (!projected.ok) throw projected.error; + expect(projected.value.artifact).toEqual({ path: "", sha256: null }); + expect(projected.value.unknowns).toContain( + "The parent has no recorded subject, so the inspected artifact path and SHA-256 are unknown.", + ); + }); + + it("rejects other view kinds and malformed retained results", () => { + const wrongKind = projectAnalysisView(parent(), { + kind: "wasm", + facet: "exports", + offset: 0, + limit: 64, + }); + if (wrongKind.ok) throw new Error("expected a view-kind refusal"); + expect(wrongKind.error).toMatchObject({ + _tag: "AnalysisInputError", + issues: [ + { + path: ["view"], + message: "Mach-O inspection supports summary and macho views.", + }, + ], + }); + const contradictory = projectAnalysisView( + parent(report, { + name: "tool", + digest: { sha256: "b".repeat(64) }, + format: "elf", + architecture: null, + local_path: "/bin/tool", + }), + { kind: "summary" }, + ); + if (contradictory.ok) throw new Error("expected a subject refusal"); + expect(contradictory.error).toMatchObject({ + issues: [{ path: ["source", "subject"] }], + }); + const malformed = projectAnalysisView( + parent({ ...report, imports: { items: "x" } }), + { kind: "summary" }, + ); + expect(malformed.ok).toBe(false); + if (!malformed.ok) expect(malformed.error._tag).toBe("AnalysisInputError"); + }); +}); diff --git a/src/domain/analysisView/machoView.ts b/src/domain/analysisView/machoView.ts new file mode 100644 index 000000000..1a64d23aa --- /dev/null +++ b/src/domain/analysisView/machoView.ts @@ -0,0 +1,247 @@ +import { z } from "zod"; +import type { AnalysisError } from "../analysisErrorBase.js"; +import { uniqueSorted } from "../canonicalOrdering.js"; +import { isMachOFormat } from "../binaryTargetTypes.js"; +import { analysisInputErrorFromIssues } from "../inputIssueProjection.js"; +import { inspectMachoSchema } from "../native/nativeInspection.js"; +import { err, ok, type Result } from "../result.js"; +import { + analysisViewInputError, + type AnalysisViewParent, + type AnalysisViewRequest, + type UnsignedAnalysisView, +} from "./analysisView.js"; +import { + completeWithinViewCoverage, + pageViewCoverage, +} from "./analysisViewCoverage.js"; + +const MACHO_COLLECTIONS = [ + "entrypoints", + "architectures", + "build_metadata", + "load_commands", + "dependencies", + "imports", + "exports", + "segments", +] as const; +type MachoCollection = (typeof MACHO_COLLECTIONS)[number]; + +/** Original inspect_macho rows; offsets count rows in the retained collection. */ +export const machoViewRequestSchema = z.strictObject({ + kind: z.literal("macho"), + facet: z.enum([...MACHO_COLLECTIONS, "provenance"]), + offset: z.number().int().nonnegative().default(0), + limit: z.number().int().positive().default(64), +}); + +const { shape } = inspectMachoSchema; +/** + * Union members are tried in order and plain objects drop unknown keys, so a + * segment or architecture row would otherwise match the entrypoint row and + * lose every field but file_offset. Exact key sets keep each page's rows whole. + */ +const exactRows = ( + rows: z.ZodArray>, +) => z.array(z.strictObject(rows.element.shape)); +export const machoViewItemSchema = z.union([ + exactRows(shape.entrypoints.shape.items), + exactRows(shape.architectures.shape.items), + exactRows(shape.build_metadata.shape.items), + exactRows(shape.load_commands.shape.items), + exactRows(shape.dependencies.shape.items), + // Imports and exports share one row shape. + exactRows(shape.imports.shape.items), + exactRows(shape.segments.shape.items), + exactRows(shape.provenance), +]); + +/** The producer's own coverage of the paged collection, not of this page. */ +export const machoViewContextSchema = z.strictObject({ + source_total: z.number().int().nonnegative().nullable(), + source_exhaustive: z.boolean(), +}); + +const collectionCountSchema = z.strictObject({ + returned: z.number().int().nonnegative(), + total: z.number().int().nonnegative().nullable(), + exhaustive: z.boolean(), +}); + +/** Slice identity and per-collection counts without the collections themselves. */ +export const machoSummarySchema = z.strictObject({ + format: shape.format, + selected_architecture: shape.selected_architecture, + endian: shape.endian, + word_size: shape.word_size, + file_type: shape.file_type, + flags: shape.flags, + uuid: shape.uuid, + collections: z.strictObject( + Object.fromEntries( + MACHO_COLLECTIONS.map((name) => [name, collectionCountSchema]), + ) as Record, + ), + provenance_count: z.number().int().nonnegative(), + limitation_count: z.number().int().nonnegative(), +}); + +type MachoReport = z.output; + +/** Admit a well-formed report whose recorded subject, if any, is a Mach-O. */ +const parseMachoReport = ( + parent: AnalysisViewParent, +): Result => { + const parsed = inspectMachoSchema.safeParse(parent.normalizedResult); + if (!parsed.success) + return err( + analysisInputErrorFromIssues( + "inspect_analysis_view", + parsed.error.issues.map((issue) => ({ + ...issue, + path: ["source", "normalized_result", ...issue.path], + })), + { source: { normalized_result: parent.normalizedResult } }, + { cause: parsed.error }, + ), + ); + if (parent.subject != null && !isMachOFormat(parent.subject.format)) + return err( + analysisViewInputError([ + { + path: ["source", "subject"], + reason: "invalid_value", + message: + "Mach-O parent Evidence subject must identify a thin or universal Mach-O.", + }, + ]), + ); + return ok(parsed.data); +}; + +/** Counts replace rows; every collection caveat stays in the limitations. */ +const summarizeMacho = ( + report: MachoReport, + parentLimitations: readonly string[], +) => { + const limitations = uniqueSorted([ + ...parentLimitations, + ...report.limitations, + ...MACHO_COLLECTIONS.flatMap((name) => report[name].limitations), + ]); + const collections = Object.fromEntries( + MACHO_COLLECTIONS.map((name) => [ + name, + { + returned: report[name].items.length, + total: report[name].total, + exhaustive: report[name].exhaustive, + }, + ]), + ) as Record>; + return { + limitations, + returned: MACHO_COLLECTIONS.reduce( + (sum, name) => sum + report[name].items.length, + 0, + ), + summary: { + format: report.format, + selected_architecture: report.selected_architecture, + endian: report.endian, + word_size: report.word_size, + file_type: report.file_type, + flags: report.flags, + uuid: report.uuid, + collections, + provenance_count: report.provenance.length, + limitation_count: limitations.length, + }, + }; +}; + +/** Project retained inspect_macho Evidence without rerunning native tools. */ +export const projectMachoView = ( + parent: AnalysisViewParent, + view: AnalysisViewRequest, +): Result => { + if (view.kind !== "summary" && view.kind !== "macho") + return err( + analysisViewInputError([ + { + path: ["view"], + reason: "invalid_value", + message: "Mach-O inspection supports summary and macho views.", + }, + ]), + ); + const parsed = parseMachoReport(parent); + if (!parsed.ok) return parsed; + const report = parsed.value; + const incomplete = MACHO_COLLECTIONS.filter( + (name) => !report[name].exhaustive, + ); + const shared = { + parent_evidence_id: parent.evidenceId, + parent_operation: "inspect_macho" as const, + parent_digest: parent.evidenceId.slice(3), + artifact: + parent.subject == null + ? { path: "", sha256: null } + : { + path: parent.subject.local_path, + sha256: parent.subject.digest.sha256, + }, + unknowns: [ + ...(incomplete.length === 0 + ? [] + : [ + `Native tools did not report exhaustive ${incomplete.join(", ")}; page exhaustion does not establish absence of unreturned rows.`, + ]), + ...(parent.subject == null + ? [ + "The parent has no recorded subject, so the inspected artifact path and SHA-256 are unknown.", + ] + : []), + ], + }; + if (view.kind === "summary") { + const { summary, limitations, returned } = summarizeMacho( + report, + parent.limitations, + ); + return ok({ + kind: "summary", + view, + ...shared, + limitations, + summary, + coverage: completeWithinViewCoverage(returned, returned), + }); + } + const source = + view.facet === "provenance" + ? { + items: report.provenance, + total: report.provenance.length, + exhaustive: true, + limitations: [], + } + : report[view.facet]; + const item = source.items.slice(view.offset, view.offset + view.limit); + return ok({ + kind: "macho", + view, + item, + ...shared, + limitations: uniqueSorted([ + ...parent.limitations, + ...report.limitations, + ...source.limitations, + ]), + source_total: source.total, + source_exhaustive: source.exhaustive, + coverage: pageViewCoverage(view.offset, item.length, source.items.length), + }); +}; diff --git a/src/domain/analysisView/nativeFunctionView.ts b/src/domain/analysisView/nativeFunctionView.ts index f5a29e0e1..b56de24a1 100644 --- a/src/domain/analysisView/nativeFunctionView.ts +++ b/src/domain/analysisView/nativeFunctionView.ts @@ -79,6 +79,8 @@ const nativeRows = ( unresolved_calls: dossier.unresolved_calls, referenced_strings: dossier.referenced_strings, referenced_names: dossier.referenced_names, + swift_string_literals: dossier.swift_string_literals, + swift_string_literal_omissions: dossier.swift_string_literal_omissions, value_flow_operations: flow?.operations, value_flow_def_use: flow?.def_use, value_flow_effects: flow?.effects, @@ -163,7 +165,8 @@ export const projectNativeFunctionView = ( singleton({ available: false, // Only value-flow facets inherit the value-flow failure reason; an - // older dossier simply has no unresolved_calls record. + // older dossier simply has no unresolved_calls record, and a target + // whose provider does not decode Swift literals has no such record. reason: view.facet.startsWith("value_flow_") && flow?.available === false ? flow.reason diff --git a/src/domain/analysisView/pcapCaptureView.ts b/src/domain/analysisView/pcapCaptureView.ts new file mode 100644 index 000000000..a28398d44 --- /dev/null +++ b/src/domain/analysisView/pcapCaptureView.ts @@ -0,0 +1,327 @@ +import { z } from "zod"; +import { pcapResultSchemas } from "../pcap/pcapAnalysis.js"; +import { analysisInputErrorFromIssues } from "../inputIssueProjection.js"; +import type { AnalysisError } from "../analysisErrorBase.js"; +import { err, ok, type Result } from "../result.js"; +import { + analysisViewInputError, + type AnalysisViewParent, + type AnalysisViewRequest, + type UnsignedAnalysisView, +} from "./analysisView.js"; +import { + completeWithinViewCoverage, + pageViewCoverage, +} from "./analysisViewCoverage.js"; + +const captureSchema = pcapResultSchemas.inspect_pcap_capture; +const fields = captureSchema.shape; +const pageShape = { + kind: z.literal("pcap"), + offset: z.number().int().nonnegative().default(0), + limit: z.number().int().positive().default(64), +}; +export const pcapViewRequestSchema = z.discriminatedUnion("facet", [ + z.strictObject({ + ...pageShape, + facet: z.enum([ + "endpoints", + "dns-queries", + "tls-sni", + "protocols", + "http-uris", + "unattributed-addresses", + ]), + }), + z.strictObject({ + ...pageShape, + facet: z.literal("dns-addresses"), + query: z.string().min(1), + }), + z.strictObject({ + ...pageShape, + facet: z.literal("diagnostics"), + stream: z.enum(["tshark", "capinfos"]), + }), +]); +export const pcapViewPageSchema = z.discriminatedUnion("facet", [ + z.strictObject({ facet: z.literal("endpoints"), items: fields.endpoints }), + z.strictObject({ + facet: z.literal("dns-queries"), + items: z.array( + fields.dns_queries.element.omit({ response_addresses: true }).extend({ + retained_response_address_count: z.number().int().nonnegative(), + }), + ), + }), + z.strictObject({ facet: z.literal("tls-sni"), items: fields.tls_sni_values }), + z.strictObject({ + facet: z.literal("protocols"), + items: fields.protocol_distribution, + }), + z.strictObject({ + facet: z.literal("http-uris"), + items: fields.http_request_uris, + }), + z.strictObject({ + facet: z.literal("unattributed-addresses"), + items: fields.unattributed_dns_addresses, + }), + z.strictObject({ + facet: z.literal("dns-addresses"), + items: fields.unattributed_dns_addresses, + }), + z.strictObject({ + facet: z.literal("diagnostics"), + text: z.string(), + status: z.enum(["retained", "unknown"]), + }), +]); +export const pcapSummarySchema = captureSchema + .omit({ + endpoints: true, + dns_queries: true, + tls_sni_values: true, + protocol_distribution: true, + http_request_uris: true, + unattributed_dns_addresses: true, + diagnostics: true, + }) + .extend({ + retained_counts: z.strictObject({ + endpoints: z.number().int().nonnegative(), + dns_queries: z.number().int().nonnegative(), + tls_sni_values: z.number().int().nonnegative(), + protocol_distribution: z.number().int().nonnegative(), + http_request_uris: z.number().int().nonnegative(), + unattributed_dns_addresses: z.number().int().nonnegative(), + }), + diagnostics_code_units: z + .strictObject({ + tshark: z.number().int().nonnegative(), + capinfos: z.number().int().nonnegative(), + }) + .nullable(), + }); +type Capture = z.output; +type PcapView = z.output; +type Page = z.output; + +const parseParent = ( + parent: AnalysisViewParent, +): Result => { + const parsed = captureSchema.safeParse(parent.normalizedResult); + if (!parsed.success) + return err( + analysisInputErrorFromIssues( + "inspect_analysis_view", + parsed.error.issues, + parent.normalizedResult, + { cause: parsed.error }, + ), + ); + if ( + parent.subject?.local_path !== parsed.data.target.path || + parent.subject.digest.sha256 !== parsed.data.target.sha256 + ) + return err( + analysisViewInputError([ + { + path: ["source", "subject"], + reason: "invalid_value", + message: + "Packet parent subject must identify the report's exact path and SHA-256.", + }, + ]), + ); + return ok(parsed.data); +}; +const summarize = ({ + endpoints, + dns_queries, + tls_sni_values, + protocol_distribution, + http_request_uris, + unattributed_dns_addresses, + diagnostics, + ...metadata +}: Capture) => ({ + ...metadata, + retained_counts: { + endpoints: endpoints.length, + dns_queries: dns_queries.length, + tls_sni_values: tls_sni_values.length, + protocol_distribution: protocol_distribution.length, + http_request_uris: http_request_uris.length, + unattributed_dns_addresses: unattributed_dns_addresses.length, + }, + diagnostics_code_units: + diagnostics === undefined + ? null + : { + tshark: diagnostics.tshark_stderr.length, + capinfos: diagnostics.capinfos_stderr.length, + }, +}); + +const selectRows = ( + capture: Capture, + view: Exclude, +): Result< + { page: Exclude; total: number }, + AnalysisError +> => { + const slice = (items: readonly T[]) => + items.slice(view.offset, view.offset + view.limit); + switch (view.facet) { + case "endpoints": + return ok({ + page: { facet: view.facet, items: slice(capture.endpoints) }, + total: capture.endpoints.length, + }); + case "dns-queries": + return ok({ + page: { + facet: view.facet, + items: slice(capture.dns_queries).map( + ({ response_addresses, ...query }) => ({ + ...query, + retained_response_address_count: response_addresses.length, + }), + ), + }, + total: capture.dns_queries.length, + }); + case "tls-sni": + return ok({ + page: { facet: view.facet, items: slice(capture.tls_sni_values) }, + total: capture.tls_sni_values.length, + }); + case "protocols": + return ok({ + page: { + facet: view.facet, + items: slice(capture.protocol_distribution), + }, + total: capture.protocol_distribution.length, + }); + case "http-uris": + return ok({ + page: { facet: view.facet, items: slice(capture.http_request_uris) }, + total: capture.http_request_uris.length, + }); + case "unattributed-addresses": + return ok({ + page: { + facet: view.facet, + items: slice(capture.unattributed_dns_addresses), + }, + total: capture.unattributed_dns_addresses.length, + }); + case "dns-addresses": { + const matches = capture.dns_queries.filter( + (query) => query.query === view.query, + ); + if (matches.length !== 1) + return err( + analysisViewInputError([ + { + path: ["view", "query"], + reason: "invalid_value", + message: "Select one unambiguous exact retained DNS question.", + }, + ]), + ); + return ok({ + page: { + facet: view.facet, + items: slice(matches[0]!.response_addresses), + }, + total: matches[0]!.response_addresses.length, + }); + } + } +}; + +/** Historical packet facts are paged without reopening the source or rerunning Wireshark. */ +export const projectPcapCaptureView = ( + parent: AnalysisViewParent, + view: AnalysisViewRequest, +): Result => { + if (view.kind !== "summary" && view.kind !== "pcap") + return err( + analysisViewInputError([ + { + path: ["view"], + reason: "invalid_value", + message: + "Packet summary parents support summary and pcap fact pages.", + }, + ]), + ); + const parsed = parseParent(parent); + if (!parsed.ok) return parsed; + const capture = parsed.value; + const shared = { + parent_evidence_id: parent.evidenceId, + parent_operation: "inspect_pcap_capture" as const, + parent_digest: parent.evidenceId.slice(3), + artifact: { path: capture.target.path, sha256: capture.target.sha256 }, + limitations: [ + ...parent.limitations, + "Page totals count retained rows or UTF-16 diagnostic code units. Exhaustion does not prove whole-network completeness; source coverage and examined counts remain separate.", + ], + unknowns: [ + "Saved packet observations do not establish process identity, causality, encrypted payload contents or traffic missing from the recording.", + ], + }; + if (view.kind === "summary") + return ok({ + kind: "summary", + view, + ...shared, + summary: summarize(capture), + coverage: completeWithinViewCoverage(1, 1), + }); + if (view.facet === "diagnostics") { + const value = + view.stream === "tshark" + ? capture.diagnostics?.tshark_stderr + : capture.diagnostics?.capinfos_stderr; + const text = value?.slice(view.offset, view.offset + view.limit) ?? ""; + return ok({ + kind: "pcap", + view, + ...shared, + source_coverage: capture.coverage, + page: { + facet: "diagnostics", + text, + status: value === undefined ? "unknown" : "retained", + }, + coverage: pageViewCoverage(view.offset, text.length, value?.length ?? 0), + unknowns: + value === undefined + ? [ + ...shared.unknowns, + "The parent did not retain diagnostics; an empty text page does not establish their absence.", + ] + : shared.unknowns, + }); + } + const selected = selectRows(capture, view); + return selected.ok + ? ok({ + kind: "pcap", + view, + ...shared, + source_coverage: capture.coverage, + page: selected.value.page, + coverage: pageViewCoverage( + view.offset, + selected.value.page.items.length, + selected.value.total, + ), + }) + : selected; +}; diff --git a/src/domain/analysisView/qlogCaptureView.ts b/src/domain/analysisView/qlogCaptureView.ts new file mode 100644 index 000000000..a3bdf88c3 --- /dev/null +++ b/src/domain/analysisView/qlogCaptureView.ts @@ -0,0 +1,248 @@ +import { z } from "zod"; +import { + qlogCaptureFactsSchema, + type QlogCaptureFacts, +} from "../qlogCapture.js"; +import { err, ok, type Result } from "../result.js"; +import type { AnalysisError } from "../analysisErrorBase.js"; +import { analysisViewInputError } from "./analysisView.js"; +import { pageViewCoverage } from "./analysisViewCoverage.js"; + +const traceSchema = qlogCaptureFactsSchema.shape.traces.element; +const streamSchema = traceSchema.shape.streams.element; +export const qlogViewContextSchema = qlogCaptureFactsSchema.omit({ + traces: true, +}); +export const qlogSummarySchema = qlogViewContextSchema.extend({ + trace_count: z.number().int().nonnegative(), + retained_stream_count: z.number().int().nonnegative(), + loss_declaration_count: z.number().int().nonnegative(), + incomplete_fact_record_count: z.number().int().nonnegative(), +}); + +/** Select retained trace facts without copying event inventories or every contribution. */ +const pageRequestShape = { + kind: z.literal("qlog"), + offset: z.number().int().nonnegative().default(0), + limit: z.number().int().positive().default(64), +}; +const traceSelectorShape = { trace_index: z.number().int().nonnegative() }; +const streamSelectorShape = { + ...traceSelectorShape, + stream_id: streamSchema.shape.stream_id, + direction: streamSchema.shape.direction, +}; +export const qlogViewRequestSchema = z.discriminatedUnion("facet", [ + z.strictObject({ ...pageRequestShape, facet: z.literal("traces") }), + z.strictObject({ + ...pageRequestShape, + ...traceSelectorShape, + facet: z.literal("streams"), + }), + z.strictObject({ + ...pageRequestShape, + ...traceSelectorShape, + facet: z.literal("loss-declarations"), + }), + z.strictObject({ + ...pageRequestShape, + ...traceSelectorShape, + facet: z.literal("incomplete-records"), + }), + z.strictObject({ + ...pageRequestShape, + ...streamSelectorShape, + facet: z.literal("contributions"), + }), + z.strictObject({ + ...pageRequestShape, + ...streamSelectorShape, + facet: z.literal("ranges"), + }), + z.strictObject({ + ...pageRequestShape, + ...streamSelectorShape, + facet: z.literal("final-sizes"), + }), +]); +export type QlogViewRequest = z.output; + +const traceSummarySchema = traceSchema + .omit({ + streams: true, + loss_declarations: true, + incomplete_fact_records: true, + }) + .extend({ + stream_count: z.number().int().nonnegative(), + loss_declaration_count: z.number().int().nonnegative(), + incomplete_fact_record_count: z.number().int().nonnegative(), + }); +const streamSummarySchema = streamSchema + .omit({ + contributions: true, + covered_ranges: true, + reported_final_sizes: true, + }) + .extend({ + contribution_count: z.number().int().nonnegative(), + covered_range_count: z.number().int().nonnegative(), + reported_final_size_count: z.number().int().nonnegative(), + }); +/** Each page advertises exactly the retained item contract for its facet. */ +export const qlogViewPageSchema = z.discriminatedUnion("facet", [ + z.strictObject({ + facet: z.literal("traces"), + items: z.array(traceSummarySchema), + }), + z.strictObject({ + facet: z.literal("streams"), + items: z.array(streamSummarySchema), + }), + z.strictObject({ + facet: z.literal("loss-declarations"), + items: traceSchema.shape.loss_declarations, + }), + z.strictObject({ + facet: z.literal("incomplete-records"), + items: traceSchema.shape.incomplete_fact_records, + }), + z.strictObject({ + facet: z.literal("contributions"), + items: streamSchema.shape.contributions, + }), + z.strictObject({ + facet: z.literal("ranges"), + items: streamSchema.shape.covered_ranges, + }), + z.strictObject({ + facet: z.literal("final-sizes"), + items: streamSchema.shape.reported_final_sizes, + }), +]); +type QlogViewPage = z.output; + +export const summarizeQlogCapture = (qlog: QlogCaptureFacts) => { + const { traces, ...context } = qlog; + return { + ...context, + trace_count: traces.length, + retained_stream_count: traces.reduce( + (sum, trace) => sum + trace.streams.length, + 0, + ), + loss_declaration_count: traces.reduce( + (sum, trace) => sum + trace.loss_declarations.length, + 0, + ), + incomplete_fact_record_count: traces.reduce( + (sum, trace) => sum + trace.incomplete_fact_records.length, + 0, + ), + }; +}; + +const summarizeTrace = ({ + streams, + loss_declarations, + incomplete_fact_records, + ...trace +}: QlogCaptureFacts["traces"][number]) => ({ + ...trace, + stream_count: streams.length, + loss_declaration_count: loss_declarations.length, + incomplete_fact_record_count: incomplete_fact_records.length, +}); +const summarizeStream = ({ + contributions, + covered_ranges, + reported_final_sizes, + ...stream +}: QlogCaptureFacts["traces"][number]["streams"][number]) => ({ + ...stream, + contribution_count: contributions.length, + covered_range_count: covered_ranges.length, + reported_final_size_count: reported_final_sizes.length, +}); + +/** Array offsets refer to retained facts, whose scope remains all recorded events. */ +export const projectQlogCapturePage = ( + qlog: QlogCaptureFacts, + view: QlogViewRequest, +): Result< + { + context: z.output; + page: QlogViewPage; + coverage: ReturnType; + }, + AnalysisError +> => { + const invalid = (field: string, message: string) => + err( + analysisViewInputError([ + { path: ["view", field], reason: "invalid_value", message }, + ]), + ); + const slice = (items: readonly T[]) => + items.slice(view.offset, view.offset + view.limit); + let page: QlogViewPage; + let total: number; + if (view.facet === "traces") { + total = qlog.traces.length; + page = { + facet: "traces", + items: slice(qlog.traces).map(summarizeTrace), + }; + } else { + const matches = qlog.traces.filter( + (trace) => trace.trace_index === view.trace_index, + ); + if (matches.length !== 1) + return invalid( + "trace_index", + "Select one unambiguous retained trace index.", + ); + const trace = matches[0]!; + if (view.facet === "streams") { + total = trace.streams.length; + page = { + facet: "streams", + items: slice(trace.streams).map(summarizeStream), + }; + } else if (view.facet === "loss-declarations") { + total = trace.loss_declarations.length; + page = { facet: view.facet, items: slice(trace.loss_declarations) }; + } else if (view.facet === "incomplete-records") { + total = trace.incomplete_fact_records.length; + page = { facet: view.facet, items: slice(trace.incomplete_fact_records) }; + } else { + const matches = trace.streams.filter( + (stream) => + stream.stream_id === view.stream_id && + stream.direction === view.direction, + ); + if (matches.length !== 1) + return invalid( + "stream_id", + "Select one unambiguous retained stream ID and direction within the trace.", + ); + const stream = matches[0]!; + if (view.facet === "contributions") { + total = stream.contributions.length; + page = { facet: view.facet, items: slice(stream.contributions) }; + } else if (view.facet === "ranges") { + total = stream.covered_ranges.length; + page = { facet: view.facet, items: slice(stream.covered_ranges) }; + } else { + total = stream.reported_final_sizes.length; + page = { facet: view.facet, items: slice(stream.reported_final_sizes) }; + } + } + } + const { traces: _traces, ...context } = qlog; + return ok({ + context, + page, + coverage: pageViewCoverage(view.offset, page.items.length, total), + }); +}; diff --git a/src/domain/analysisView/wasmArtifactView.ts b/src/domain/analysisView/wasmArtifactView.ts index 111078600..dc3036b26 100644 --- a/src/domain/analysisView/wasmArtifactView.ts +++ b/src/domain/analysisView/wasmArtifactView.ts @@ -103,6 +103,7 @@ export const projectWasmArtifactView = ( : { path: parent.subject.local_path, sha256: parent.subject.digest.sha256, + format: parent.subject.format, }, ) ) diff --git a/src/domain/analysisView/webNetworkApiSurfaceView.test.ts b/src/domain/analysisView/webNetworkApiSurfaceView.test.ts index f13e392ff..b57d54b59 100644 --- a/src/domain/analysisView/webNetworkApiSurfaceView.test.ts +++ b/src/domain/analysisView/webNetworkApiSurfaceView.test.ts @@ -85,11 +85,13 @@ it("pages groups independently of record completeness and preserves null digests }); it("rejects native captures and malformed parents with typed errors", () => { const capture = analysisViewCaptureFixture(); - capture.format = "mitmproxy"; - expect(project(capture)).toMatchObject({ - ok: false, - error: { _tag: "AnalysisUnsupportedTargetError" }, - }); + for (const format of ["mitmproxy", "saz"] as const) { + capture.format = format; + expect(project(capture)).toMatchObject({ + ok: false, + error: { _tag: "AnalysisUnsupportedTargetError" }, + }); + } expect( projectAnalysisView( { diff --git a/src/domain/analysisView/webNetworkCaptureView.test.ts b/src/domain/analysisView/webNetworkCaptureView.test.ts new file mode 100644 index 000000000..ff73b29a6 --- /dev/null +++ b/src/domain/analysisView/webNetworkCaptureView.test.ts @@ -0,0 +1,176 @@ +import { expect, it } from "vitest"; +import { createHash } from "node:crypto"; +import { createEvidence } from "../evidence.js"; +import { webNetworkCaptureSchema } from "../webNetworkCapture.js"; +import { projectAnalysisView } from "./analysisView.js"; + +const capture = () => { + const body = Buffer.from("original body"); + const artifact = { + path: "/recordings/input.saz", + sha256: "b".repeat(64), + bytes: 100, + }; + const decoder = { id: "fixture", name: "Fixture", version: "1" }; + const report = webNetworkCaptureSchema.parse({ + artifact, + decoder, + format: "saz", + runtime_attribution: "unknown", + limitations: [], + container: { + reported: { members: [] }, + numeric_literals: [], + redactions: [], + records_pointer: null, + }, + total_records: 9, + records: [7, 2].map((ordinal) => ({ + ordinal, + location: { kind: "archive-members", members: [`raw/${ordinal}_c.txt`] }, + reported: { + session_id: String(ordinal), + time: null, + request: { + method: "POST", + url: "https://example.invalid/a", + body: body.toString(), + }, + }, + binary_fields: [ + { + pointer: "/request/body", + representation: "producer-bytes", + state: "retained", + content_base64: body.toString("base64"), + bytes: body.length, + sha256: createHash("sha256").update(body).digest("hex"), + }, + ], + numeric_literals: [ + { + pointer: "/time", + producer_type: "json-number", + literal: "9007199254740993", + }, + ], + redactions: [], + limitations: [], + })), + }); + const evidence = createEvidence({ ...artifact, format: "file" }, decoder, { + operation: "inspect_web_network_capture", + parameters: { record_ordinals: [7, 2] }, + result: report, + confidence: "observed", + authority: "historical-reference", + }); + return { + report, + parent: { + evidenceId: evidence.evidence_id, + operation: evidence.operation, + normalizedResult: report, + subject: evidence.subject, + limitations: evidence.limitations, + }, + }; +}; + +it("keeps selected-record page coverage distinct from capture coverage and unsafe number absence", () => { + const { parent, report } = capture(); + const summary = projectAnalysisView(parent, { kind: "summary" }); + expect(summary.ok && summary.value).toMatchObject({ + summary: { total_records: 9, retained_records: 2 }, + coverage: { examined: 2, total: 2 }, + }); + const preview = projectAnalysisView(parent, { + kind: "capture", + detail: "preview", + offset: 0, + limit: 1, + }); + expect(preview.ok && preview.value).toMatchObject({ + items: [ + { + ordinal: 7, + reported: { time: null }, + numeric_literals: [{ pointer: "/time", literal: "9007199254740993" }], + }, + ], + coverage: { total: 2, examined: 1, next_offset: 1, exhausted: false }, + }); + expect(JSON.stringify(preview)).not.toContain("original body"); + expect(JSON.stringify(preview)).not.toContain("content_base64"); + const complete = projectAnalysisView(parent, { + kind: "capture", + detail: "complete", + offset: 1, + limit: 1, + }); + expect(complete.ok && complete.value).toMatchObject({ + items: [report.records[1]], + coverage: { total: 2, examined: 1, next_offset: null, exhausted: true }, + }); + const empty = projectAnalysisView(parent, { + kind: "capture", + detail: "preview", + offset: 50, + limit: 1, + }); + expect(empty.ok && empty.value).toMatchObject({ + items: [], + coverage: { + status: "empty", + total: 2, + examined: 0, + next_offset: null, + exhausted: true, + }, + }); +}); + +it("retains explicitly unknown digests and rejects a different subject path", () => { + const { parent, report } = capture(); + const unknown = projectAnalysisView( + { + ...parent, + subject: null, + normalizedResult: { + ...report, + artifact: { ...report.artifact, sha256: null }, + }, + }, + { kind: "summary" }, + ); + expect(unknown.ok && unknown.value).toMatchObject({ + artifact: { sha256: null }, + }); + expect(unknown.ok && unknown.value.unknowns.join(" ")).toContain( + "excluded artifact digest", + ); + if (parent.subject === null) throw new Error("Expected fixture subject"); + const mismatched = projectAnalysisView( + { + ...parent, + subject: { ...parent.subject, local_path: "/different/source.saz" }, + }, + { kind: "summary" }, + ); + expect(mismatched.ok).toBe(false); +}); + +it("rejects invalid ordinal coverage in a borrowed parent", () => { + const { parent, report } = capture(); + const result = projectAnalysisView( + { + ...parent, + normalizedResult: { + ...report, + records: [report.records[0], report.records[0]], + }, + }, + { kind: "summary" }, + ); + expect(result.ok).toBe(false); +}); diff --git a/src/domain/analysisView/webNetworkCaptureView.ts b/src/domain/analysisView/webNetworkCaptureView.ts new file mode 100644 index 000000000..b335e9241 --- /dev/null +++ b/src/domain/analysisView/webNetworkCaptureView.ts @@ -0,0 +1,264 @@ +import { z } from "zod"; +import type { AnalysisError } from "../analysisErrorBase.js"; +import { analysisInputErrorFromIssues } from "../inputIssueProjection.js"; +import type { JsonValue } from "../jsonValue.js"; +import { err, ok, type Result } from "../result.js"; +import { + webNetworkCaptureRecordSchema, + webNetworkCaptureBinaryDescriptorSchema, + webNetworkCaptureSchema, + webNetworkCaptureArtifactSchema, + webNetworkCaptureDecoderSchema, + WEB_NETWORK_CAPTURE_FORMATS, + type WebNetworkCaptureRecord, + type WebNetworkCapture, +} from "../webNetworkCapture.js"; +import { + analysisViewInputError, + type AnalysisViewParent, + type AnalysisViewRequest, + type UnsignedAnalysisView, +} from "./analysisView.js"; +import { + completeWithinViewCoverage, + pageViewCoverage, +} from "./analysisViewCoverage.js"; +import { + qlogSummarySchema, + summarizeQlogCapture, + projectQlogCapturePage, +} from "./qlogCaptureView.js"; + +/** Offsets page the parent's retained record order, preserving producer ordinals. */ +export const captureViewRequestSchema = z.strictObject({ + kind: z.literal("capture"), + detail: z.enum(["preview", "complete"]).default("preview"), + offset: z.number().int().nonnegative().default(0), + limit: z.number().int().positive().default(64), +}); + +const previewSchema = webNetworkCaptureRecordSchema.extend({ + binary_fields: z.array(webNetworkCaptureBinaryDescriptorSchema), +}); +export const captureViewItemsSchema = z.array( + z.union([webNetworkCaptureRecordSchema, previewSchema]), +); + +/** Count/identity overview; bodies and the archive member inventory stay in the parent. */ +export const captureSummarySchema = z.strictObject({ + artifact: webNetworkCaptureArtifactSchema, + decoder: webNetworkCaptureDecoderSchema, + format: z.enum(WEB_NETWORK_CAPTURE_FORMATS), + total_records: z.number().int().nonnegative(), + retained_records: z.number().int().nonnegative(), + runtime_attribution: z.literal("unknown"), + qlog: qlogSummarySchema.optional(), +}); + +const object = ( + value: JsonValue, +): Readonly> | null => + value !== null && typeof value === "object" && !Array.isArray(value) + ? value + : null; + +/** Select already parsed producer facts without interpreting HTTP or payload bytes again. */ +const preview = (record: WebNetworkCaptureRecord) => { + const source = object(record.reported); + const reported: Record = {}; + const selectedPointers = new Set(); + if (source !== null) { + for (const key of ["session_id", "name", "time", "startedDateTime", "type"]) + if (Object.hasOwn(source, key) && source[key] !== undefined) { + reported[key] = source[key]; + selectedPointers.add(`/${key}`); + } + for (const key of ["request", "response"]) { + const value = source[key]; + if (!Object.hasOwn(source, key) || value === undefined) continue; + const fields = object(value); + if (fields === null) { + // Preserve a reported null/scalar instead of inventing HTTP fields. + reported[key] = value; + continue; + } + const selected: Record = {}; + for (const field of [ + "method", + "url", + "httpVersion", + "http_version", + "status", + "statusText", + "reason", + ]) + if (Object.hasOwn(fields, field) && fields[field] !== undefined) { + selected[field] = fields[field]; + selectedPointers.add(`/${key}/${field}`); + } + reported[key] = selected; + } + } + return { + ...record, + reported: source === null ? record.reported : reported, + binary_fields: record.binary_fields.map( + ({ content_base64: _content, ...field }) => field, + ), + numeric_literals: + source === null + ? record.numeric_literals + : record.numeric_literals.filter(({ pointer }) => + selectedPointers.has(pointer), + ), + }; +}; + +/** Authenticate the retained report and its subject at one owning boundary. */ +const parseCaptureParent = ( + parent: AnalysisViewParent, +): Result => { + const parsed = webNetworkCaptureSchema.safeParse(parent.normalizedResult); + if (!parsed.success) + return err( + analysisInputErrorFromIssues( + "inspect_analysis_view", + parsed.error.issues.map((issue) => ({ + ...issue, + path: ["source", "normalized_result", ...issue.path], + })), + { source: { normalized_result: parent.normalizedResult } }, + { cause: parsed.error }, + ), + ); + const report = parsed.data; + if ( + report.artifact.sha256 !== null && + (parent.subject == null || + parent.subject.local_path !== report.artifact.path || + parent.subject.digest.sha256 !== report.artifact.sha256) + ) + return err( + analysisViewInputError([ + { + path: ["source", "subject"], + reason: "invalid_value", + message: + "Capture parent subject must identify the report's exact path and SHA-256.", + }, + ]), + ); + return ok(report); +}; + +/** Page immutable retained facts; no capture file, provider or network is consulted. */ +export const projectWebNetworkCaptureView = ( + parent: AnalysisViewParent, + view: AnalysisViewRequest, +): Result => { + if ( + view.kind !== "summary" && + view.kind !== "capture" && + view.kind !== "qlog" + ) + return err( + analysisViewInputError([ + { + path: ["view"], + reason: "invalid_value", + message: + "Historical captures support summary, capture record pages, qlog fact pages and admitted api-surface views.", + }, + ]), + ); + const parsed = parseCaptureParent(parent); + if (!parsed.ok) return parsed; + const report = parsed.value; + const shared = { + parent_evidence_id: parent.evidenceId, + parent_operation: "inspect_web_network_capture" as const, + parent_digest: parent.evidenceId.slice(3), + artifact: { path: report.artifact.path, sha256: report.artifact.sha256 }, + limitations: [ + ...new Set([ + ...parent.limitations, + ...report.limitations, + "Capture record pages cover the records selected into the parent Evidence, in that selected order; total_records counts the original decoded capture.", + ]), + ], + unknowns: [ + "Historical records do not establish live runtime attribution or deployment authenticity.", + ...(report.artifact.sha256 === null + ? ["The explicitly excluded artifact digest remains unknown."] + : []), + ], + }; + if (view.kind === "summary") + return ok({ + kind: "summary", + view, + ...shared, + summary: { + artifact: report.artifact, + decoder: report.decoder, + format: report.format, + total_records: report.total_records, + retained_records: report.records.length, + runtime_attribution: report.runtime_attribution, + ...(report.format === "qlog" + ? { qlog: summarizeQlogCapture(report.qlog) } + : {}), + }, + coverage: completeWithinViewCoverage( + report.records.length, + report.records.length, + ), + }); + if (view.kind === "qlog") { + if (report.format !== "qlog") + return err( + analysisViewInputError([ + { + path: ["view", "kind"], + reason: "invalid_value", + message: "Qlog fact views require retained qlog Evidence.", + }, + ]), + ); + const projected = projectQlogCapturePage(report.qlog, view); + return projected.ok + ? ok({ + kind: "qlog", + view, + ...shared, + ...projected.value, + limitations: [ + ...shared.limitations, + "Qlog fact pages retain all-recorded-events-before-selection scope. Exhausting a page does not establish network completeness, body recovery, retransmission or actual loss.", + ], + }) + : projected; + } + const records = report.records.slice(view.offset, view.offset + view.limit); + return ok({ + kind: "capture", + view, + ...shared, + items: captureViewItemsSchema.parse( + view.detail === "complete" ? records : records.map(preview), + ), + coverage: pageViewCoverage( + view.offset, + records.length, + report.records.length, + ), + limitations: [ + ...shared.limitations, + ...(view.detail === "preview" + ? [ + "Preview selects recorded headline fields and byte descriptors; headers, XML, bodies, message data and other fields remain in complete record pages. Missing preview fields do not establish their absence from the producer record.", + ] + : []), + ], + }); +}; diff --git a/src/domain/android/androidAnalysis.ts b/src/domain/android/androidAnalysis.ts index e27be2b20..ca4d6db59 100644 --- a/src/domain/android/androidAnalysis.ts +++ b/src/domain/android/androidAnalysis.ts @@ -45,6 +45,12 @@ export const androidInputSchemas = { }), } as const; +/** Optional retained delivery is separate from the engine's APK request. */ +export const androidPackageInspectionInputSchema = + androidInputSchemas.inspect_android_package.extend({ + detail: z.enum(["complete", "summary"]).optional(), + }); + /** Supported static Android operations. */ export type AndroidOperation = keyof typeof androidInputSchemas; /** Validated Android requests carry their selected operation. */ diff --git a/src/domain/apple/dylibResolution.test.ts b/src/domain/apple/dylibResolution.test.ts index 2d63ce393..13bb7c4f6 100644 --- a/src/domain/apple/dylibResolution.test.ts +++ b/src/domain/apple/dylibResolution.test.ts @@ -597,6 +597,136 @@ describe("dyld load order", () => { }); }); +describe("loader-relative dylib resolution", () => { + it("does not reuse already loaded images by install name for @loader_path dependencies", async () => { + const pluginA = "Contents/PlugIns/A/pluginA.dylib"; + const helperA = "Contents/PlugIns/A/libhelper.dylib"; + const pluginB = "Contents/PlugIns/B/pluginB.dylib"; + const trace = await traceDylibLoading( + memoryView({ + [MAIN]: executable({ + dependencies: [ + dependency("@executable_path/../PlugIns/A/pluginA.dylib"), + dependency("@executable_path/../PlugIns/B/pluginB.dylib"), + ], + }), + [pluginA]: parsed( + slice({ + dependencies: [dependency("@loader_path/libhelper.dylib")], + }), + ), + [helperA]: parsed( + slice({ + install_name: "@loader_path/libhelper.dylib", + }), + ), + [pluginB]: parsed( + slice({ + dependencies: [dependency("@loader_path/libhelper.dylib")], + }), + ), + }), + { roots: [MAIN] }, + ); + + const edgeA = edgeFor(trace, pluginA, "@loader_path/libhelper.dylib"); + expect(edgeA.resolution).toEqual({ + status: "resolved", + image: helperA, + }); + expect(edgeA.candidates).toEqual([ + expect.objectContaining({ + source: "loader_path", + outcome: "resolved", + resolved_path: helperA, + }), + ]); + + const edgeB = edgeFor(trace, pluginB, "@loader_path/libhelper.dylib"); + expect(edgeB.resolution).toEqual({ + status: "unresolved", + image: null, + }); + expect(edgeB.candidates).toEqual([ + expect.objectContaining({ + path: "Contents/PlugIns/B/libhelper.dylib", + source: "loader_path", + outcome: "absent", + resolved_path: null, + }), + ]); + expect(trace.findings).toContainEqual( + expect.objectContaining({ + kind: "required-load-unresolved", + image: pluginB, + }), + ); + }); + + it("resolves @loader_path dependencies to the loader's own directory when multiple helpers exist", async () => { + const pluginA = "Contents/PlugIns/A/pluginA.dylib"; + const helperA = "Contents/PlugIns/A/libhelper.dylib"; + const pluginB = "Contents/PlugIns/B/pluginB.dylib"; + const helperB = "Contents/PlugIns/B/libhelper.dylib"; + const trace = await traceDylibLoading( + memoryView({ + [MAIN]: executable({ + dependencies: [ + dependency("@executable_path/../PlugIns/A/pluginA.dylib"), + dependency("@executable_path/../PlugIns/B/pluginB.dylib"), + ], + }), + [pluginA]: parsed( + slice({ + dependencies: [dependency("@loader_path/libhelper.dylib")], + }), + ), + [helperA]: parsed( + slice({ + install_name: "@loader_path/libhelper.dylib", + }), + ), + [pluginB]: parsed( + slice({ + dependencies: [dependency("@loader_path/libhelper.dylib")], + }), + ), + [helperB]: parsed( + slice({ + install_name: "@loader_path/libhelper.dylib", + }), + ), + }), + { roots: [MAIN] }, + ); + + expect( + edgeFor(trace, pluginA, "@loader_path/libhelper.dylib"), + ).toMatchObject({ + resolution: { status: "resolved", image: helperA }, + candidates: [ + expect.objectContaining({ + source: "loader_path", + outcome: "resolved", + resolved_path: helperA, + }), + ], + }); + expect( + edgeFor(trace, pluginB, "@loader_path/libhelper.dylib"), + ).toMatchObject({ + resolution: { status: "resolved", image: helperB }, + candidates: [ + expect.objectContaining({ + source: "loader_path", + outcome: "resolved", + resolved_path: helperB, + }), + ], + }); + }); +}); + describe("dyld slice compatibility and coverage", () => { it("lets an x86_64h process load generic x86_64 but not arm64e load arm64", async () => { const files = { diff --git a/src/domain/apple/dylibResolution.ts b/src/domain/apple/dylibResolution.ts index 00077efa1..cf10c4b78 100644 --- a/src/domain/apple/dylibResolution.ts +++ b/src/domain/apple/dylibResolution.ts @@ -16,6 +16,7 @@ import { import { DYLIB_RESOLUTION_LIMITATIONS, deriveFindings, + type TracedRootSlice, } from "./dylibResolutionFindings.js"; /** @@ -427,7 +428,9 @@ const resolveDependency = async ( ): Promise => { const current = context.loaded.get(loader); const chain = current?.chain ?? [loader]; - const loaded = context.byInstallName.get(dependency.install_name); + const loaded = dependency.install_name.startsWith("@rpath/") + ? context.byInstallName.get(dependency.install_name) + : undefined; const candidates: Candidate[] = []; if (loaded !== undefined) candidates.push({ @@ -506,7 +509,10 @@ const traceProcess = async ( chain: [context.root], conditional: false, }); - if (rootSlice.install_name !== null) + if ( + rootSlice.install_name !== null && + rootSlice.install_name.startsWith("@rpath/") + ) context.byInstallName.set(rootSlice.install_name, context.root); const pending = [context.root]; while (pending.length > 0) { @@ -537,6 +543,7 @@ const traceProcess = async ( }); if ( slice.install_name !== null && + slice.install_name.startsWith("@rpath/") && !context.byInstallName.has(slice.install_name) ) context.byInstallName.set(slice.install_name, image); @@ -562,6 +569,7 @@ export const traceDylibLoading = async ( for (const path of request.unclassified ?? []) images.set(path, await view.image(path)); const roots: DylibTrace["roots"][number][] = []; + const rootSlices: TracedRootSlice[] = []; const edges: Edge[] = []; const withoutArchitecture: string[] = []; const environmentRoots = new Set(); @@ -591,6 +599,7 @@ export const traceDylibLoading = async ( ) environmentRoots.add(root); roots.push({ image: root, architecture: slice.architecture }); + rootSlices.push({ image: root, slice }); edges.push( ...(await traceProcess( { @@ -635,7 +644,7 @@ export const traceDylibLoading = async ( facts.status === "parsed" ? facts.slices.map(withoutDependencies) : [], })), edges, - findings: deriveFindings(edges, roots, images), + findings: deriveFindings(edges, rootSlices), coverage: { status: unparsed.length === 0 && diff --git a/src/domain/apple/dylibResolutionFindings.ts b/src/domain/apple/dylibResolutionFindings.ts index e4cc1ddf2..eac488143 100644 --- a/src/domain/apple/dylibResolutionFindings.ts +++ b/src/domain/apple/dylibResolutionFindings.ts @@ -1,14 +1,18 @@ import type { DylibEdge as Edge, DylibFinding as Finding, - DylibTrace, - MachoImageFacts, + MachoSlice, } from "./dylibResolution.js"; +/** One traced process root with the exact slice it was traced from. */ +export interface TracedRootSlice { + readonly image: string; + readonly slice: MachoSlice; +} + export const deriveFindings = ( edges: readonly Edge[], - roots: DylibTrace["roots"], - images: ReadonlyMap, + roots: readonly TracedRootSlice[], ): Finding[] => { const findings: Finding[] = []; edges.forEach((edge, index) => { @@ -32,15 +36,11 @@ export const deriveFindings = ( explanation: `The modeled dyld search checks ${earlier.join(", ")} before ${edge.resolution.image ?? edge.install_name}. A compatible Mach-O placed at an earlier modeled path could take precedence over that fallback, subject to unmodeled search inputs and code-signing library validation; library validation is not evaluated here (see inspect_signature).${edge.loader_conditional ? " This loader loads only conditionally; if it never loads, this search never occurs." : ""}${edge.resolution.status === "conditional" ? " Unmodeled environment overrides or earlier unknown candidates may take precedence; placing an image at this rpath does not establish which image dyld will load." : ""}`, }); }); - for (const { image, architecture } of roots) { - const facts = images.get(image); - const slice = - facts?.status === "parsed" - ? facts.slices.find( - (candidate) => candidate.architecture === architecture, - ) - : undefined; - const environment = slice?.dyld_environment ?? []; + // Slices can share an architecture name (arm64e and arm64e.x1), so each + // finding uses the slice its root was traced from, not a name lookup. + for (const { image, slice } of roots) { + const { architecture } = slice; + const environment = slice.dyld_environment; if (environment.length > 0) findings.push({ kind: "dyld-environment-present", @@ -48,7 +48,7 @@ export const deriveFindings = ( image, basis: "derived", explanation: - slice?.file_type === "execute" + slice.file_type === "execute" ? `${image} (${architecture}) sets dyld environment variables through LC_DYLD_ENVIRONMENT (${environment.join(", ")}); any image-selection overrides they request are not modeled. Diagnostic settings alone do not alter resolution.` : `${image} (${architecture}) contains LC_DYLD_ENVIRONMENT (${environment.join(", ")}). Dyld reads these commands only from the main executable; entries in this non-executable root are observations and do not alter resolution.`, }); @@ -91,7 +91,7 @@ export const DYLIB_RESOLUTION_LIMITATIONS = [ "Slices are matched by dyld's graded architectures (an x86_64h process also loads x86_64). arm64e processes that disable pointer authentication can also load arm64 slices; that fallback is not modeled.", "Absolute install names and rpaths are outside the analyzed root and are not evaluated, including /System and /usr/lib libraries that the dyld shared cache usually provides.", "Leaf and relative install names depend on dyld fallback paths, DYLD_* variables, and the working directory; they are undetermined.", - "Load order follows dyld's dependents-first traversal in load-command order. An image reached through several chains is resolved once per process, with the rpath stack and conditionality of the first chain; a later request whose install name matches an already loaded image reuses it.", + "Load order follows dyld's dependents-first traversal in load-command order. An image reached through several chains is resolved once per process, with the rpath stack and conditionality of the first chain; a later @rpath request whose install name matches an already loaded image reuses it.", "@loader_path uses each image's symlink-resolved path within the analyzed root.", "Code-signing checks that can reject a found image, such as library validation and the hardened runtime, are not evaluated.", "Dylib compatibility versions are not compared. Apple's linker documentation describes a load-time rejection when an image's compatibility version is older than the loader requires, but dyld on macOS 26 was observed to load such an image.", diff --git a/src/domain/apple/interfaceBuilderGraph.test.ts b/src/domain/apple/interfaceBuilderGraph.test.ts index bfa2c71de..96c6b4915 100644 --- a/src/domain/apple/interfaceBuilderGraph.test.ts +++ b/src/domain/apple/interfaceBuilderGraph.test.ts @@ -510,6 +510,106 @@ describe("AppKit keyed archive connectors", () => { }); }); +describe("keyed archive unresolved connectors", () => { + // A rooted UIKit archive whose only connector is UID 3, with UID 1 as the + // owner, UID 2 as the destination and UID 7 as the "view" label. + const rootedOutletArchive = (connection: Record) => ({ + $archiver: "NSKeyedArchiver", + $version: 100000, + $objects: [ + "$null", + { $class: { CF$UID: 4 } }, + { $class: { CF$UID: 5 } }, + { $class: { CF$UID: 6 }, ...connection }, + { + $classname: "UIViewController", + $classes: ["UIViewController", "UIResponder", "NSObject"], + }, + { $classname: "UIView", $classes: ["UIView", "UIResponder", "NSObject"] }, + { + $classname: "UIRuntimeOutletConnection", + $classes: [ + "UIRuntimeOutletConnection", + "UIRuntimeConnection", + "NSObject", + ], + }, + "view", + { + $class: { CF$UID: 11 }, + "NS.objects": [{ CF$UID: 1 }, { CF$UID: 2 }, { CF$UID: 3 }], + }, + { $class: { CF$UID: 11 }, "NS.objects": [{ CF$UID: 3 }] }, + { + $class: { CF$UID: 11 }, + "NS.objects": [{ CF$UID: 1 }, { CF$UID: 2 }], + }, + { $classname: "NSArray", $classes: ["NSArray", "NSObject"] }, + ], + $top: { + root: { CF$UID: 8 }, + UINibConnectionsKey: { CF$UID: 9 }, + UINibObjectsKey: { CF$UID: 10 }, + UINibTopLevelObjectsKey: { CF$UID: 10 }, + }, + }); + + const project = (raw: unknown) => + buildInterfaceBuilderAnalysis({ + targetSha256: hash, + toolVersion: "test", + documents: [ + { + relativePath: "Main.nib", + archiveSha256: hash, + documentKind: "nib", + raw, + }, + ], + limits: interfaceBuilderLimitsSchema.parse({}), + }); + + it("covers a resolvable connector as a complete examined connection", () => { + const raw = rootedOutletArchive({ + UISource: { CF$UID: 1 }, + UIDestination: { CF$UID: 2 }, + UILabel: { CF$UID: 7 }, + }); + expect(parseInterfaceBuilderRecords(raw).unresolvedConnections).toBe(0); + expect(project(raw).graph.coverage).toContainEqual( + expect.objectContaining({ + facet: "connections:Main.nib", + status: "complete", + examined: 1, + omitted: 0, + }), + ); + }); + + it("reports a recognized connector with no resolvable owner as unresolved", () => { + const raw = rootedOutletArchive({ + UIDestination: { CF$UID: 2 }, + UILabel: { CF$UID: 7 }, + }); + expect(parseInterfaceBuilderRecords(raw).unresolvedConnections).toBe(1); + + const analysis = project(raw); + expect(analysis.graph.nodes.some(({ kind }) => kind === "outlet")).toBe( + false, + ); + expect(analysis.graph.coverage).toContainEqual( + expect.objectContaining({ + facet: "connections:Main.nib", + status: "partial", + reason: "connection_unresolved", + examined: 0, + omitted: 1, + }), + ); + expect(analysis.graph.truncated).toBe(true); + }); +}); + it("names the archive once per evidence record", () => { const relativePath = "Views/Main.storyboardc/scene.nib/objects.nib"; const result = buildInterfaceBuilderAnalysis({ diff --git a/src/domain/apple/interfaceBuilderGraph.ts b/src/domain/apple/interfaceBuilderGraph.ts index f25c8b9b1..b3e5216dc 100644 --- a/src/domain/apple/interfaceBuilderGraph.ts +++ b/src/domain/apple/interfaceBuilderGraph.ts @@ -410,18 +410,26 @@ export const buildInterfaceBuilderAnalysis = (input: { const connectionsTruncated = parserOmittedConnections > 0 || parsed.connections.length > input.limits.max_connections; + // A recognized connector the parser could not attach to an owner produces + // no connection record, so its coverage is partial with its own reason + // rather than absent or truncated. + const connectionsUnresolved = parsed.unresolvedConnections > 0; omittedObjects += parsed.omittedObjects + Math.max(0, parsed.objects.length - documentObjectCount); omittedConnections += Math.max(0, parsed.connections.length - input.limits.max_connections) + - parserOmittedConnections; + parserOmittedConnections + + parsed.unresolvedConnections; summaries.push({ relative_path: document.relativePath, archive_sha256: document.archiveSha256, document_kind: document.documentKind, object_count: parsed.objectCount, - connection_count: parsed.connections.length + parserOmittedConnections, + connection_count: + parsed.connections.length + + parserOmittedConnections + + parsed.unresolvedConnections, hierarchy_complete: parsed.hierarchy.length > 0 && !truncated && @@ -440,13 +448,17 @@ export const buildInterfaceBuilderAnalysis = (input: { { facet: `connections:${document.relativePath}`, status: - connectionsTruncated || documentOmittedConnections > 0 + connectionsTruncated || + documentOmittedConnections > 0 || + connectionsUnresolved ? "partial" : "complete", reason: connectionsTruncated || documentOmittedConnections > 0 ? "connection_limit_reached" - : null, + : connectionsUnresolved + ? "connection_unresolved" + : null, examined: Math.min( parsed.connections.length, input.limits.max_connections, @@ -457,7 +469,8 @@ export const buildInterfaceBuilderAnalysis = (input: { parsed.connections.length - input.limits.max_connections, ) + documentOmittedConnections + - parserOmittedConnections, + parserOmittedConnections + + parsed.unresolvedConnections, }, { facet: `hierarchy:${document.relativePath}`, diff --git a/src/domain/apple/interfaceBuilderKeyedArchive.ts b/src/domain/apple/interfaceBuilderKeyedArchive.ts index 207950bf8..57ebda803 100644 --- a/src/domain/apple/interfaceBuilderKeyedArchive.ts +++ b/src/domain/apple/interfaceBuilderKeyedArchive.ts @@ -55,6 +55,7 @@ export const parseInterfaceBuilderRecords = ( readonly objectCount: number; readonly omittedObjects: number; readonly omittedConnections: number; + readonly unresolvedConnections: number; readonly omittedHierarchyReferences: number; } => { const root = record(value); @@ -135,6 +136,7 @@ export const parseInterfaceBuilderRecords = ( objectCount: objectEntries.length, omittedObjects: Math.max(0, objectEntries.length - 20_000), omittedConnections, + unresolvedConnections: 0, omittedHierarchyReferences: 0, }; }; @@ -320,6 +322,7 @@ const parseKeyedArchive = ( const connections: InterfaceBuilderConnection[] = []; let objectCount = 0; let omittedConnections = 0; + let unresolvedConnections = 0; for (const [index, raw] of objectTable.entries()) { if (index === 0) continue; const resolved = record(resolve(raw)); @@ -371,7 +374,13 @@ const parseKeyedArchive = ( controlAction ? fields.to : fields.from, controlAction ? fields.target : fields.owner, ); - if (source === null) continue; + if (source === null) { + // A recognized connector that cannot name its owner is an unresolved + // connection, not an absent one, and cannot be represented as a + // connection because connection identity requires a source. + unresolvedConnections += 1; + continue; + } const kind = controlAction ? "action" : classifyConnection(className); const parsedConnection = connection.parse({ id: `archive:${objectId}`, @@ -426,6 +435,7 @@ const parseKeyedArchive = ( objectCount, omittedObjects: Math.max(0, objectCount - 20_000), omittedConnections, + unresolvedConnections, omittedHierarchyReferences: hierarchy.omittedReferences, }; }; diff --git a/src/domain/apple/machoContainer.ts b/src/domain/apple/machoContainer.ts index 7b552be64..c23cf7f64 100644 --- a/src/domain/apple/machoContainer.ts +++ b/src/domain/apple/machoContainer.ts @@ -63,6 +63,11 @@ export const readFatContainerHeader = ( const magic = view.getUint32(0, false); if (![FAT_MAGIC, FAT_MAGIC_64, FAT_CIGAM, FAT_CIGAM_64].includes(magic)) return { state: "not-fat" }; + if (!Number.isSafeInteger(fileSize) || fileSize < bytes.byteLength) + return { + state: "malformed", + reason: "FAT file size is invalid or contradicts retained bytes", + }; if (bytes.byteLength < 8) { return fileSize < 8 ? { state: "malformed", reason: "FAT header is truncated" } diff --git a/src/domain/binaryTarget.ts b/src/domain/binaryTarget.ts index 48da65960..94828ee6e 100644 --- a/src/domain/binaryTarget.ts +++ b/src/domain/binaryTarget.ts @@ -278,8 +278,12 @@ export const parsePeRecord = ( record.toString("binary", 0, 4) !== "PE\u0000\u0000" ) return err("invalid or truncated PE header"); - const machine = decodePeMachine(record.readUInt16LE(4)); - if (machine === undefined) return err("unsupported PE architecture"); + const machineType = record.readUInt16LE(4); + const machine = decodePeMachine(machineType); + if (machine === undefined) + return err( + `unsupported PE architecture (COFF machine 0x${machineType.toString(16).padStart(4, "0")})`, + ); const { architecture } = machine; const optionalHeaderSize = record.readUInt16LE(20); if (optionalHeaderSize > 4096 || record.length < 24 + optionalHeaderSize) @@ -347,7 +351,9 @@ const machArchitecture = (cpu: number): HeaderArchitecture | undefined => { return undefined; }; -const elfArchitecture = (machine: number): BinaryArchitecture | undefined => { +const elfArchitecture = ( + machine: number, +): Exclude | undefined => { switch (machine) { case 3: return "x86"; diff --git a/src/domain/binaryTargetTypes.ts b/src/domain/binaryTargetTypes.ts index b4d44c110..99ad8487f 100644 --- a/src/domain/binaryTargetTypes.ts +++ b/src/domain/binaryTargetTypes.ts @@ -1,10 +1,13 @@ -/** Provider-neutral CPU families detected from supported executable headers. */ +import type { ZipPackageFormat } from "./zipPackageFormat.js"; + +/** Provider-neutral CPU families detected from headers or explicitly selected raw loads. */ export const BINARY_ARCHITECTURES = [ "x86", "x86_64", "arm", "arm64", "mips", + "6502", "riscv", ] as const; export type BinaryArchitecture = (typeof BINARY_ARCHITECTURES)[number]; @@ -38,7 +41,7 @@ export const isElfMetadataArchitecture = ( /** Recognized family with no ELF metadata member; the header alone is the fact. */ export type HeaderArchitecture = Exclude< BinaryArchitecture, - ElfMetadataArchitecture + ElfMetadataArchitecture | "6502" >; /** Raw ELF facts; family recognition does not establish ISA or ABI support. */ @@ -142,8 +145,17 @@ export type BinaryTarget = readonly executableRole?: never; readonly managed?: never; }) - | (ExecutableTarget & { + | (BinaryTargetIdentity & { + readonly kind: "executable"; readonly format: "raw-binary"; + readonly architecture: Exclude< + BinaryArchitecture, + ElfMetadataArchitecture + >; + readonly availableArchitectures: readonly Exclude< + BinaryArchitecture, + ElfMetadataArchitecture + >[]; readonly entryPoint: string; readonly entryPointSource: string; readonly executableRole?: never; @@ -167,18 +179,7 @@ export type BinaryTarget = | (BinaryTargetIdentity & NonExecutableMetadata & { readonly kind: "archive"; - readonly format: - | "zip" - | "ipa" - | "apk" - | "msix" - | "appx" - | "hap" - | "hsp" - | "app-pack" - | "asar" - | "dmg" - | "pkg"; + readonly format: ZipPackageFormat | "asar" | "dmg" | "pkg"; }) | (BinaryTargetIdentity & NonExecutableMetadata & { diff --git a/src/domain/callPath.test.ts b/src/domain/callPath.test.ts index e0f343fca..e78a5f936 100644 --- a/src/domain/callPath.test.ts +++ b/src/domain/callPath.test.ts @@ -1,6 +1,7 @@ import { describe, expect, it } from "vitest"; import { buildCallPath } from "./callPath.js"; +import { combineFunctionRelationshipCoverage } from "./functionCallCoverage.js"; import { callPathResultSchema, type CallPathInput } from "./callPathSchemas.js"; import { createEvidence, type Evidence } from "./evidence.js"; import type { JsonValue } from "./jsonValue.js"; @@ -396,6 +397,60 @@ describe("explicit call relationship coverage", () => { expect(result.limitations).toEqual([reason]); }); + it("establishes not_found when only caller coverage is unknown", () => { + const reason = + "Callers: Upstream xrefs do not establish direct caller relationships."; + const result = run([ + observe("0x1000", ["0x2000"], "rea-workflow", { + relationshipCoverage: { status: "unknown", reason }, + }), + observe("0x2000", [], "rea-workflow", { + relationshipCoverage: { status: "unknown", reason }, + }), + ]); + expect(result).toMatchObject({ + status: "not_found", + search_scope: { exhaustive: true }, + }); + expect(result.limitations).toContain(reason); + }); + + it("stays unknown when callee coverage is unknown", () => { + const reason = + "Callees: Upstream callees capped results at 500; additional callees remain unknown."; + const result = run([ + observe("0x1000", ["0x2000"], "rea-workflow", { + relationshipCoverage: { status: "unknown", reason }, + }), + observe("0x2000", [], "rea-workflow", { + relationshipCoverage: { status: "unknown", reason }, + }), + ]); + expect(result).toMatchObject({ + status: "unknown", + search_scope: { exhaustive: false }, + }); + expect(result.limitations).toContain(reason); + }); + + it("stays unknown when both caller and callee coverage are unknown", () => { + const reason = + "Callers: Upstream xrefs do not establish direct caller relationships. Callees: Upstream callees are not observed."; + const result = run([ + observe("0x1000", ["0x2000"], "rea-workflow", { + relationshipCoverage: { status: "unknown", reason }, + }), + observe("0x2000", [], "rea-workflow", { + relationshipCoverage: { status: "unknown", reason }, + }), + ]); + expect(result).toMatchObject({ + status: "unknown", + search_scope: { exhaustive: false }, + }); + expect(result.limitations).toContain(reason); + }); + it("ignores incomplete relationship coverage outside the reachable search closure", () => { const reason = "An unrelated function's callees were not observed."; const result = run([ @@ -415,6 +470,34 @@ describe("explicit call relationship coverage", () => { }); }); +describe("IDA-capped callee coverage beside unknown callers", () => { + it("keeps the search non-exhaustive for the combined coverage value", () => { + const combined = combineFunctionRelationshipCoverage( + { + status: "unknown", + reason: "Upstream xrefs do not establish direct caller relationships.", + }, + { + status: "partial", + limitations: [ + "Upstream callees capped results at 500; additional callees remain unknown.", + ], + }, + ); + expect(combined.status).toBe("unknown"); + const result = run([ + observe("0x1000", [], "rea-workflow", { relationshipCoverage: combined }), + observe("0x4000", []), + ]); + expect(result).toMatchObject({ + status: "unknown", + search_scope: { exhaustive: false }, + shortest_hops: null, + paths: [], + }); + }); +}); + describe("call path evidence admission", () => { it("normalizes hex addresses and rejects duplicates and mixed providers", () => { expect( diff --git a/src/domain/callPath.ts b/src/domain/callPath.ts index 2186c8010..0bd5e88e1 100644 --- a/src/domain/callPath.ts +++ b/src/domain/callPath.ts @@ -10,6 +10,7 @@ import { parseFunctionEvidence, type FunctionSnapshot, } from "./functionDossierEvidence.js"; +import { isCallerOnlyUnknownCoverage } from "./functionCallCoverage.js"; import { compareUnicodeCodePoints } from "./unicodeCodePointOrder.js"; /** Minimal directed caller-to-callee adjacency for call-path search. */ @@ -255,10 +256,14 @@ const inspectSearch = ({ // Explicit, Evidence-bound relationship coverage owns completeness; // generic provider/dossier limitations remain contextual. Legacy Evidence // without that metadata retains its conservative limitation-based rule. + // The search only expands callee edges, so coverage that is unknown + // solely because callers are unobservable does not make the searched + // callee closure incomplete; the caller caveat stays in limitations. if ( coverage === null ? snapshot.limitations.length > 0 - : coverage.status !== "complete" + : coverage.status !== "complete" && + !isCallerOnlyUnknownCoverage(coverage) ) incompleteExpandedCoverage = true; } else if (node === goal) { diff --git a/src/domain/captures/packetFields.ts b/src/domain/captures/packetFields.ts new file mode 100644 index 000000000..23b25cfcd --- /dev/null +++ b/src/domain/captures/packetFields.ts @@ -0,0 +1,100 @@ +import { z } from "zod"; +import { packetCaptureSchema } from "./packetCapture.js"; + +export const inspectPacketFieldsInputSchema = z.strictObject({ + kind: z + .enum(["protocols", "fields"]) + .default("protocols") + .describe( + "Return protocol registrations or field definitions; defaults to protocols", + ), + protocol: z + .string() + .min(1) + .max(8192) + .nullable() + .default(null) + .describe( + "Exact protocol abbreviation; selects that protocol or its registered fields", + ), + query: z + .string() + .max(8192) + .default("") + .describe( + "Case-insensitive literal substring in name, abbreviation or field description", + ), + offset: z + .number() + .int() + .nonnegative() + .max(Number.MAX_SAFE_INTEGER) + .default(0) + .describe("Number of matching records to skip; defaults to 0"), + limit: z + .number() + .int() + .positive() + .max(Number.MAX_SAFE_INTEGER) + .nullable() + .default(50) + .describe( + "Maximum matches returned; null requests all remaining within the record budget", + ), +}); +export type InspectPacketFieldsInput = z.infer< + typeof inspectPacketFieldsInputSchema +>; +export const packetFieldRecordSchema = z.strictObject({ + source_line: z.number().int().positive(), + kind: z.enum(["protocol", "field"]), + name: z.string(), + abbreviation: z.string(), + field_type: z.string().nullable(), + parent_protocol: z.string().nullable(), + display_base: z.string().nullable(), + bitmask: z.string().nullable(), + description: z.string().nullable(), +}); +export type PacketFieldRecord = z.infer; +export const packetFieldsCatalogSchema = z + .strictObject({ + selection: inspectPacketFieldsInputSchema, + records: z.array(packetFieldRecordSchema), + output_window: z.strictObject({ + examined: z.number().int().nonnegative(), + matching: z.number().int().nonnegative(), + returned: z.number().int().nonnegative(), + has_more: z.boolean(), + }), + diagnostics: z.string(), + tool: packetCaptureSchema.shape.tool, + resource_limiter: packetCaptureSchema.shape.resource_limiter, + limits: packetCaptureSchema.shape.limits.extend({ + record_budget_bytes: z.literal(8388608), + }), + configuration: z.literal("private-empty-personal-profile"), + hardware_acquisition: z.literal("not-performed"), + }) + .superRefine((report, context) => { + const w = report.output_window; + const remaining = Math.max(0, w.matching - report.selection.offset); + const expected = Math.min(remaining, report.selection.limit ?? remaining); + if ( + w.matching > w.examined || + w.returned !== expected || + report.records.length !== expected || + w.has_more !== remaining > expected || + report.records.some( + (row) => + row.kind !== + (report.selection.kind === "protocols" ? "protocol" : "field"), + ) + ) + context.addIssue({ + code: "custom", + path: ["output_window"], + message: + "Field discovery coverage must match the selected page and record kind", + }); + }); diff --git a/src/domain/changedBehavior.test.ts b/src/domain/changedBehavior.test.ts index cc19c4c50..6d99e828f 100644 --- a/src/domain/changedBehavior.test.ts +++ b/src/domain/changedBehavior.test.ts @@ -22,6 +22,9 @@ const source = (digit: string): Evidence => const left = source("1"); const right = source("2"); +const aggregate = (comparisons: readonly unknown[]) => + findChangedBehavior({ comparisons }); + const comparison = ( operation: "compare_process_captures" | "compare_artifacts", result: JsonValue, @@ -136,9 +139,9 @@ const expectInvalidSummary = (result: ChangedBehaviorResult): void => { describe("changed behavior", () => { it("accepts process comparison Evidence", () => { - const result = findChangedBehavior([ + const result = aggregate([ comparison("compare_process_captures", processResult()), - ]); + ]).result; expect(result.behavior_status).toBe("observed_unchanged"); }); @@ -152,7 +155,7 @@ describe("changed behavior", () => { }), ); - expect(findChangedBehavior([evidence])).toMatchObject({ + expect(aggregate([evidence]).result).toMatchObject({ behavior_status: "observed_changed", summary: { observed_changes: 2 }, findings: { @@ -172,7 +175,7 @@ describe("changed behavior", () => { "compare_process_captures", processResult({ status: "changed", terminal: "changed" }), ); - const result = findChangedBehavior([evidence]); + const result = aggregate([evidence]).result; expect(changedBehaviorResultSchema.parse(result)).toMatchObject({ behavior_status: "observed_changed", summary: { observed_changes: 1, static_candidates: 0 }, @@ -193,7 +196,7 @@ describe("changed behavior", () => { it("keeps static differences as candidates, never runtime observations", () => { const evidence = comparison("compare_artifacts", artifactResult()); - const result = findChangedBehavior([evidence]); + const result = aggregate([evidence]).result; expect(result).toMatchObject({ behavior_status: "unknown", summary: { observed_changes: 0, static_candidates: 1 }, @@ -217,17 +220,17 @@ describe("changed behavior", () => { "compare_process_captures", processResult({ status: "unknown", exit: "unknown" }), ); - const result = findChangedBehavior([changed, unknown]); + const result = aggregate([changed, unknown]).result; expect(result).toMatchObject({ behavior_status: "unknown", findings: { items: expect.any(Array) }, }); - expect(findChangedBehavior([changed, unknown])).toEqual(result); + expect(aggregate([changed, unknown]).result).toEqual(result); }); it("uses complete inline artifact deltas as static candidates", () => { const complete = comparison("compare_artifacts", artifactResult()); - expect(findChangedBehavior([complete]).findings.items).toEqual( + expect(aggregate([complete]).result.findings.items).toEqual( expect.arrayContaining([ expect.objectContaining({ dimension: "artifact:main.js" }), ]), @@ -236,7 +239,7 @@ describe("changed behavior", () => { it("rejects bundles, duplicate comparisons, and malformed exact results", () => { const valid = comparison("compare_process_captures", processResult()); - expect(() => findChangedBehavior([valid, valid])).toThrow( + expect(() => aggregate([valid, valid])).toThrow( /duplicate comparison Evidence/u, ); const bundle = createEvidence( @@ -252,25 +255,25 @@ describe("changed behavior", () => { evidenceLinks: [left.evidence_id, right.evidence_id], }, ); - expect(() => findChangedBehavior([bundle])).toThrow( + expect(() => aggregate([bundle])).toThrow( /requires process, artifact, or function/u, ); const malformed = comparison("compare_process_captures", { status: "unchanged", }); - expect(() => findChangedBehavior([malformed])).toThrow(); + expect(() => aggregate([malformed])).toThrow(); const contradictory = comparison( "compare_process_captures", processResult({ status: "unchanged", terminal: "changed" }), ); - expect(() => findChangedBehavior([contradictory])).toThrow( + expect(() => aggregate([contradictory])).toThrow( /contradicts its dimensions/u, ); const danglingNested = comparison( "compare_artifacts", artifactResult([`ev_${"f".repeat(64)}`, right.evidence_id]), ); - expect(() => findChangedBehavior([danglingNested])).toThrow( + expect(() => aggregate([danglingNested])).toThrow( /outside its top-level closure/u, ); }); @@ -299,10 +302,15 @@ describe("changed behavior aggregation", () => { ), ); - const result = findChangedBehavior(comparisons); + const result = aggregate(comparisons); - expect(result.summary.observed_changes).toBe(comparisons.length); - expect(result.evidence_links).toHaveLength(comparisons.length * 201 + 2); - expect(result.findings.items[0]?.evidence_links).toHaveLength(203); + expect(result.result.summary.observed_changes).toBe(comparisons.length); + expect(result.result.evidence_links).toHaveLength( + comparisons.length * 201 + 2, + ); + expect(result.result.findings.items[0]?.evidence_links).toHaveLength(203); + expect(result.comparisonEvidenceIds).toEqual( + comparisons.map(({ evidence_id }) => evidence_id), + ); }); }); diff --git a/src/domain/changedBehavior.ts b/src/domain/changedBehavior.ts index b785cdd23..56e9b6ad9 100644 --- a/src/domain/changedBehavior.ts +++ b/src/domain/changedBehavior.ts @@ -91,6 +91,22 @@ export const changedBehaviorResultSchema = z export type ChangedBehaviorResult = z.infer; type Finding = z.infer; type RuntimeStatus = z.infer["status"]; +type ParsedComparison = + | { + readonly operation: "compare_process_captures"; + readonly evidence: Evidence; + readonly result: z.infer; + } + | { + readonly operation: "compare_functions"; + readonly evidence: Evidence; + readonly result: z.infer; + } + | { + readonly operation: "compare_artifacts"; + readonly evidence: Evidence; + readonly result: z.infer; + }; const EXPECTED_PROVIDERS = { compare_process_captures: { @@ -117,24 +133,37 @@ const acceptsPredicate = ( ): boolean => expected.some((predicate) => predicate === actual); /** Find observed runtime changes and separately report static change candidates. */ -export const findChangedBehavior = ( - comparisonsInput: unknown, -): ChangedBehaviorResult => { - const parsedInput = changedBehaviorInputSchema.parse({ - comparisons: comparisonsInput, - }); - const comparisons = parsedInput.comparisons; +interface FoundChangedBehavior { + readonly result: ChangedBehaviorResult; + /** Authenticated IDs retained from the comparisons used to compute result. */ + readonly comparisonEvidenceIds: readonly string[]; +} + +export const findChangedBehavior = (input: unknown): FoundChangedBehavior => { + // The public input schema remains available to clients; this internal + // admission only checks the list shape because parseEvidence below owns + // validation and authentication of each retained Evidence item. + const { comparisons } = z + .strictObject({ comparisons: z.array(z.unknown()).min(1) }) + .parse(input); const evidence = comparisons.map(parseComparisonEvidence); assertUnique( - evidence.map(({ evidence_id: id }) => id), + evidence.map(({ evidence: { evidence_id: id } }) => id), "comparison Evidence", ); const findings = evidence.flatMap(findingsFor).sort(compareFindings); - const runtimeStatuses = evidence.flatMap(runtimeStatus); - const links = uniqueSorted( - evidence.flatMap((item) => [item.evidence_id, ...item.evidence_links]), + const runtimeStatuses = evidence.flatMap((comparison) => + comparison.operation === "compare_process_captures" + ? [comparison.result.status] + : [], ); - return changedBehaviorResultSchema.parse({ + const links = uniqueSorted( + evidence.flatMap(({ evidence: item }) => [ + item.evidence_id, + ...item.evidence_links, + ]), + ); + const result = changedBehaviorResultSchema.parse({ behavior_status: behaviorStatus(runtimeStatuses), summary: { observed_changes: count(findings, "observed_change"), @@ -147,7 +176,7 @@ export const findChangedBehavior = ( }, evidence_links: links, limitations: uniqueSorted([ - ...evidence.flatMap(({ limitations }) => limitations), + ...evidence.flatMap(({ evidence: item }) => item.limitations), ...(evidence.some( ({ operation }) => operation !== "compare_process_captures", ) @@ -160,9 +189,15 @@ export const findChangedBehavior = ( : []), ]), }); + return { + result, + comparisonEvidenceIds: evidence.map( + ({ evidence: { evidence_id } }) => evidence_id, + ), + }; }; -const parseComparisonEvidence = (input: unknown): Evidence => { +const parseComparisonEvidence = (input: unknown): ParsedComparison => { const evidence = parseEvidence(input); if (!(evidence.operation in EXPECTED_PROVIDERS)) throw new TypeError( @@ -198,12 +233,17 @@ const parseComparisonEvidence = (input: unknown): Evidence => { throw new TypeError( "Comparison Evidence must cite both selected source observations", ); - parseResult(evidence); - return evidence; + return parseResult(evidence, operation); }; -const parseResult = (evidence: Evidence): void => { - if (evidence.operation === "compare_process_captures") { +const parseResult = ( + evidence: Evidence, + operation: + | "compare_process_captures" + | "compare_artifacts" + | "compare_functions", +): ParsedComparison => { + if (operation === "compare_process_captures") { const result = processCaptureComparisonSchema.parse( evidence.normalized_result, ); @@ -214,9 +254,9 @@ const parseResult = (evidence: Evidence): void => { throw new TypeError( "Process comparison status contradicts its dimensions", ); - return; + return { operation, evidence, result }; } - if (evidence.operation === "compare_functions") { + if (operation === "compare_functions") { const result = functionComparisonResultSchema.parse( evidence.normalized_result, ); @@ -224,7 +264,7 @@ const parseResult = (evidence: Evidence): void => { evidence, result.dimensions.flatMap(({ evidence_links: links }) => links), ); - return; + return { operation, evidence, result }; } const result = artifactComparisonResultSchema.parse( evidence.normalized_result, @@ -233,6 +273,7 @@ const parseResult = (evidence: Evidence): void => { evidence, result.changes.flatMap(({ evidence_links: links }) => links), ); + return { operation, evidence, result }; }; const assertNestedLinks = ( @@ -246,18 +287,21 @@ const assertNestedLinks = ( ); }; -const findingsFor = (evidence: Evidence): Finding[] => { - if (evidence.operation === "compare_process_captures") - return processFindings(evidence); - if (evidence.operation === "compare_functions") - return functionFindings(evidence); - return artifactFindings(evidence); +const findingsFor = (comparison: ParsedComparison): Finding[] => { + if (comparison.operation === "compare_process_captures") + return processFindings(comparison); + if (comparison.operation === "compare_functions") + return functionFindings(comparison); + return artifactFindings(comparison); }; -const processFindings = (evidence: Evidence): Finding[] => { - const result = processCaptureComparisonSchema.parse( - evidence.normalized_result, - ); +const processFindings = ( + comparison: Extract< + ParsedComparison, + { readonly operation: "compare_process_captures" } + >, +): Finding[] => { + const { evidence, result } = comparison; const dimensions = [ ["terminal", result.terminal, "runtime"], ["interaction", result.interaction, "runtime"], @@ -283,10 +327,13 @@ const processFindings = (evidence: Evidence): Finding[] => { ); }; -const functionFindings = (evidence: Evidence): Finding[] => { - const result = functionComparisonResultSchema.parse( - evidence.normalized_result, - ); +const functionFindings = ( + comparison: Extract< + ParsedComparison, + { readonly operation: "compare_functions" } + >, +): Finding[] => { + const { evidence, result } = comparison; return result.dimensions.flatMap((dimension) => dimension.status === "unchanged" ? [] @@ -307,10 +354,13 @@ const functionFindings = (evidence: Evidence): Finding[] => { ); }; -const artifactFindings = (evidence: Evidence): Finding[] => { - const result = artifactComparisonResultSchema.parse( - evidence.normalized_result, - ); +const artifactFindings = ( + comparison: Extract< + ParsedComparison, + { readonly operation: "compare_artifacts" } + >, +): Finding[] => { + const { evidence, result } = comparison; const changes = result.changes.map((change) => makeFinding({ evidence, @@ -367,11 +417,6 @@ const makeFinding = (input: { limitations: uniqueSorted(input.limitations), }); -const runtimeStatus = (evidence: Evidence): RuntimeStatus[] => - evidence.operation === "compare_process_captures" - ? [processCaptureComparisonSchema.parse(evidence.normalized_result).status] - : []; - const behaviorStatus = ( statuses: readonly RuntimeStatus[], ): ChangedBehaviorResult["behavior_status"] => { diff --git a/src/domain/evidence.test.ts b/src/domain/evidence.test.ts index c5898aecc..be923e3a6 100644 --- a/src/domain/evidence.test.ts +++ b/src/domain/evidence.test.ts @@ -19,6 +19,33 @@ const TARGET: BinaryTarget = { const PROVIDER = { id: "fixture", name: "Fixture provider", version: "1" }; const PROFILE = createAnalysisProfile(PROVIDER, { loader: "default" }); +it("rejects a detected Evidence subject with the raw-only 6502 architecture", () => { + const selected = { + path: TARGET.path, + sha256: TARGET.sha256, + format: "raw-binary" as const, + architecture: "6502" as const, + }; + expect( + createEvidence(selected, PROVIDER, { + operation: "inspect", + parameters: {}, + result: null, + }).subject, + ).toMatchObject({ format: "raw-binary", architecture: "6502" }); + const malformed = { + ...selected, + format: "pe" as const, + }; + expect(() => + createEvidence(malformed, PROVIDER, { + operation: "inspect", + parameters: {}, + result: null, + }), + ).toThrow(); +}); + it("preserves omitted relationship diagnostics and authenticates reported diagnostics", () => { const observation = { operation: "procedure_callees", @@ -52,6 +79,30 @@ it("preserves omitted relationship diagnostics and authenticates reported diagno ).toThrow("semantic identifier"); }); +it("projects known provider fields and preserves open JSON payloads", () => { + const liveProvider = { ...PROVIDER, transient_status: "starting" }; + const evidence = createEvidence(TARGET, liveProvider, { + operation: "inspect", + parameters: { caller_metadata: { note: "kept" } }, + result: { producer_extension: { value: "kept" } }, + analysisProfile: PROFILE, + environment: { + id: "fixture", + platform: "linux", + architecture: "x86_64", + isolation: "process", + }, + locations: [{ kind: "address", address: "0x1000" }], + }); + + expect(evidence.provider).toEqual(PROVIDER); + expect(evidence.parameters).toEqual({ caller_metadata: { note: "kept" } }); + expect(evidence.normalized_result).toEqual({ + producer_extension: { value: "kept" }, + }); + expect(parseEvidence(evidence)).toEqual(evidence); +}); + it("preserves identity and bytes when reusing an owned immutable result", () => { const value = { nested: { values: ["observed", 1] } }; const mutable = createEvidence(TARGET, PROVIDER, { diff --git a/src/domain/evidence.ts b/src/domain/evidence.ts index 8e21686e5..49cb2a229 100644 --- a/src/domain/evidence.ts +++ b/src/domain/evidence.ts @@ -33,59 +33,72 @@ import { } from "./relationshipCoverage.js"; /** Provider identity schema shared by evidence-bearing persistence formats. */ -export const providerSchema = z.object({ +export const providerSchema = z.strictObject({ id: z.string().min(1), name: z.string().min(1), version: z.string().nullable(), }); -const subjectSchema = z.object({ +const subjectFormatSchema = z.enum([ + "hopper", + "analysis-database", + "mach-o", + "elf", + "pe", + "dos-mz", + "dos-com", + "raw-binary", + "zip", + "ipa", + "apk", + "msix", + "appx", + "hap", + "hsp", + "app-pack", + "asar", + "dmg", + "pkg", + "plist", + "javascript", + "source-map", + "directory", + "file", + "unknown", + "mach-o-universal", + "javascript-bundle", + "entitlements", +]); +const subjectBaseSchema = z.strictObject({ name: z.string().min(1), - digest: z.object({ sha256: digestSchema }), - format: z.enum([ - "hopper", - "analysis-database", - "mach-o", - "elf", - "pe", - "dos-mz", - "dos-com", - "raw-binary", - "zip", - "ipa", - "apk", - "msix", - "appx", - "hap", - "hsp", - "app-pack", - "asar", - "dmg", - "pkg", - "plist", - "javascript", - "source-map", - "directory", - "file", - "unknown", - "mach-o-universal", - "javascript-bundle", - "entitlements", - ]), - architecture: z.enum(BINARY_ARCHITECTURES).nullable(), + digest: z.strictObject({ sha256: digestSchema }), local_path: nulFreeStringSchema, }); +// The format owns the architecture claim, including nested Evidence subjects. +const subjectSchema = z.discriminatedUnion("format", [ + subjectBaseSchema.extend({ + format: subjectFormatSchema.exclude(["raw-binary"]), + architecture: z.enum(BINARY_ARCHITECTURES).exclude(["6502"]).nullable(), + }), + subjectBaseSchema.extend({ + format: z.literal("raw-binary"), + architecture: z.enum(BINARY_ARCHITECTURES).nullable(), + }), +]); /** Source location attached to an evidence observation. */ /** Source-location schema shared by evidence-bearing persistence formats. */ export const evidenceLocationSchema = z.discriminatedUnion("kind", [ - z.object({ kind: z.literal("address"), address: z.string().min(1) }), - z.object({ + z.strictObject({ kind: z.literal("address"), address: z.string().min(1) }), + z.strictObject({ kind: z.literal("address-range"), start: z.string().min(1), end: z.string().min(1), }), - z.object({ kind: z.literal("artifact-path"), path: z.string().min(1) }), - z.object({ kind: z.literal("file-offset"), offset: z.number().int().min(0) }), - z.object({ + z.strictObject({ kind: z.literal("artifact-path"), path: z.string().min(1) }), + z.strictObject({ + kind: z.literal("file-offset"), + offset: z.number().int().min(0), + }), + z.strictObject({ kind: z.literal("file-offset-range"), start: z.number().int().min(0), end: z.number().int().min(0), @@ -100,7 +113,7 @@ const evidenceAuthoritySchema = z.enum([ "analyst-inference", ]); -const executionEnvironmentSchema = z.object({ +const executionEnvironmentSchema = z.strictObject({ id: z.string().min(1), platform: z.string().min(1), architecture: z.string().min(1), @@ -183,7 +196,7 @@ const immutableResultSchema = z }); export type EvidenceLocation = z.infer; -/** Minimal immutable local artifact identity accepted by Evidence. */ +/** Candidate local artifact identity validated by the Evidence subject schema. */ export interface EvidenceSubjectTarget { readonly path: string; readonly sha256: string; diff --git a/src/domain/evidenceBundle.ts b/src/domain/evidenceBundle.ts index c1fa3ce1c..a70969b8e 100644 --- a/src/domain/evidenceBundle.ts +++ b/src/domain/evidenceBundle.ts @@ -16,24 +16,30 @@ import { } from "./residualUnknown.js"; import { freezeJsonSnapshot } from "./immutableJson.js"; -const artifactManifestSchema = evidenceSchema.shape.subject.unwrap().pick({ +const [detectedSubjectSchema, rawSubjectSchema] = + evidenceSchema.shape.subject.unwrap().options; +const manifestFields = { digest: true, format: true, architecture: true, -}); +} as const; +const artifactManifestSchema = z.discriminatedUnion("format", [ + detectedSubjectSchema.pick(manifestFields), + rawSubjectSchema.pick(manifestFields), +]); const providerManifestSchema = providerSchema; const environmentManifestSchema = evidenceSchema.shape.environment.unwrap(); -const scenarioManifestSchema = z.object({ +const scenarioManifestSchema = z.strictObject({ evidence_id: evidenceSchema.shape.evidence_id, operation: z.string().min(1), authority: z.literal("controlled-replay"), }); -const captureManifestSchema = z.object({ +const captureManifestSchema = z.strictObject({ evidence_id: evidenceSchema.shape.evidence_id, predicate_type: z.string().min(1), }); -export const evidenceBundleSchema = z.object({ +export const evidenceBundleSchema = z.strictObject({ artifacts: z.array(artifactManifestSchema), providers: z.array(providerManifestSchema), environments: z.array(environmentManifestSchema), @@ -82,11 +88,11 @@ export const createEvidenceBundle = ( subject === null ? [] : [ - { + artifactManifestSchema.parse({ digest: subject.digest, format: subject.format, architecture: subject.architecture, - }, + }), ], ), ), diff --git a/src/domain/explicitSensitiveFailure.test.ts b/src/domain/explicitSensitiveFailure.test.ts index 68aaf9f8f..757e9887a 100644 --- a/src/domain/explicitSensitiveFailure.test.ts +++ b/src/domain/explicitSensitiveFailure.test.ts @@ -116,6 +116,24 @@ it("keeps unsupported profile reasons distinct from malformed input", () => { expect(JSON.stringify(projectAnalysisError(result))).not.toContain("private"); }); +it("preserves a retryable capability failure through explicit-value redaction", () => { + // Retryability is a semantic field of the error, not presentation: dropping + // it turns a transient admission failure into a permanent one and changes + // the remediation the caller is shown. + const error = new AnalysisCapabilityUnavailableError( + "mitmproxy", + "inspect_web_network_capture", + "Runtime admission for /targets/private/capture failed transiently", + { retryable: true }, + ); + expect(projectAnalysisError(error).retryable).toBe(true); + const result = redactExplicitFailure(error, ["private"]); + expect(result).toBeInstanceOf(AnalysisCapabilityUnavailableError); + expect((result as AnalysisCapabilityUnavailableError).retryable).toBe(true); + expect(projectAnalysisError(result).retryable).toBe(true); + expect(JSON.stringify(projectAnalysisError(result))).not.toContain("private"); +}); + it("coarsens format issue pointers without creating replacement coordinates", () => { const error = new AnalysisInputError( "inspect_web_network_capture", diff --git a/src/domain/explicitSensitiveFailure.ts b/src/domain/explicitSensitiveFailure.ts index 05a4f2f88..8a0fac095 100644 --- a/src/domain/explicitSensitiveFailure.ts +++ b/src/domain/explicitSensitiveFailure.ts @@ -154,6 +154,7 @@ export const redactExplicitFailure = ( ...(error.userMessage === undefined ? {} : { userMessage: text(error.userMessage) }), + ...(error.retryable ? { retryable: true } : {}), }, ); if (error instanceof AnalysisResourceConstraintError) diff --git a/src/domain/flutter/flutterBuildAnalysis.ts b/src/domain/flutter/flutterBuildAnalysis.ts index 068149f68..10c5b7b00 100644 --- a/src/domain/flutter/flutterBuildAnalysis.ts +++ b/src/domain/flutter/flutterBuildAnalysis.ts @@ -30,6 +30,12 @@ export const flutterInputSchemas = { }), } as const; +/** Retained delivery is selected independently of the provider's ABI request. */ +export const dartAotInspectionInputSchema = + flutterInputSchemas.inspect_dart_aot.extend({ + detail: z.enum(["complete", "summary"]).optional(), + }); + /** Supported Flutter-backed operations. */ export type FlutterOperation = keyof typeof flutterInputSchemas; /** Validated Flutter requests carry their selected operation. */ diff --git a/src/domain/functionCallCoverage.test.ts b/src/domain/functionCallCoverage.test.ts new file mode 100644 index 000000000..6fd3fb045 --- /dev/null +++ b/src/domain/functionCallCoverage.test.ts @@ -0,0 +1,64 @@ +import { describe, expect, it } from "vitest"; + +import { + combineFunctionRelationshipCoverage, + isCallerOnlyUnknownCoverage, +} from "./functionCallCoverage.js"; +import type { RelationshipCoverage } from "./relationshipCoverage.js"; + +const unknownCallers: RelationshipCoverage = { + status: "unknown", + reason: "Upstream xrefs do not establish direct caller relationships.", +}; +const unknownCallees: RelationshipCoverage = { + status: "unknown", + reason: "Upstream callees are not observed in this profile.", +}; +const cappedCallees: RelationshipCoverage = { + status: "partial", + limitations: [ + "Upstream callees capped results at 500; additional callees remain unknown.", + ], +}; +const complete: RelationshipCoverage = { status: "complete" }; + +describe("isCallerOnlyUnknownCoverage", () => { + it("detects unknown callers beside complete callees", () => { + const combined = combineFunctionRelationshipCoverage( + unknownCallers, + complete, + ); + expect(combined.status).toBe("unknown"); + expect(isCallerOnlyUnknownCoverage(combined)).toBe(true); + }); + + it("rejects unknown callees beside complete callers", () => { + const combined = combineFunctionRelationshipCoverage( + complete, + unknownCallees, + ); + expect(isCallerOnlyUnknownCoverage(combined)).toBe(false); + }); + + it("rejects unknown coverage on both sides", () => { + const combined = combineFunctionRelationshipCoverage( + unknownCallers, + unknownCallees, + ); + expect(isCallerOnlyUnknownCoverage(combined)).toBe(false); + }); + + it("rejects unknown callers beside IDA-capped callees", () => { + const combined = combineFunctionRelationshipCoverage( + unknownCallers, + cappedCallees, + ); + expect(combined.status).toBe("unknown"); + expect(isCallerOnlyUnknownCoverage(combined)).toBe(false); + }); + + it("rejects complete coverage", () => { + const combined = combineFunctionRelationshipCoverage(complete, complete); + expect(isCallerOnlyUnknownCoverage(combined)).toBe(false); + }); +}); diff --git a/src/domain/functionCallCoverage.ts b/src/domain/functionCallCoverage.ts index eb2c05a13..b64e78cd6 100644 --- a/src/domain/functionCallCoverage.ts +++ b/src/domain/functionCallCoverage.ts @@ -2,6 +2,9 @@ import type { RelationshipCoverage } from "./relationshipCoverage.js"; import type { FunctionDossier } from "./hopperValues.js"; import { uniqueSorted } from "./canonicalOrdering.js"; +const callerCoveragePrefix = "Callers: "; +const calleeCoveragePrefix = "Callees: "; + /** Combine caller and callee read outcomes without losing either explanation. */ export const combineFunctionRelationshipCoverage = ( callers: RelationshipCoverage | undefined, @@ -20,7 +23,8 @@ export const combineFunctionRelationshipCoverage = ( ); const details = diagnostics.length > 0 ? { diagnostics } : {}; const reasons = reads.flatMap(({ direction, coverage }) => { - const label = direction === "callers" ? "Callers" : "Callees"; + const label = + direction === "callers" ? callerCoveragePrefix : calleeCoveragePrefix; const limitations = coverage === undefined ? ["Provider did not report relationship coverage."] @@ -29,7 +33,7 @@ export const combineFunctionRelationshipCoverage = ( : coverage.status === "partial" ? coverage.limitations : []; - return limitations.map((reason) => `${label}: ${reason}`); + return limitations.map((reason) => `${label}${reason}`); }); if ( reads.some( @@ -50,6 +54,20 @@ export const combineFunctionRelationshipCoverage = ( return { status: "complete", ...details }; }; +/** + * Whether combined relationship coverage is unknown only on the caller side. + * The combine helper above prefixes each direction's reasons, so a reason + * that carries the caller prefix and no callee prefix records observed + * callees with unobservable callers. Forward (callee-edge) searches stay + * exhaustive under that caveat; the caller reason remains a limitation. + */ +export const isCallerOnlyUnknownCoverage = ( + coverage: RelationshipCoverage, +): boolean => + coverage.status === "unknown" && + coverage.reason.startsWith(callerCoveragePrefix) && + !coverage.reason.includes(calleeCoveragePrefix); + /** Explain why two resolved call sets cannot establish call-set equality. */ export const functionCallCoverageLimitations = ( coverage: RelationshipCoverage | null, diff --git a/src/domain/functionComparison.test.ts b/src/domain/functionComparison.test.ts index acb505dac..677fdc0cc 100644 --- a/src/domain/functionComparison.test.ts +++ b/src/domain/functionComparison.test.ts @@ -119,6 +119,34 @@ const observe = ( ); describe("function comparison normalized identity", () => { + it("keeps differing generated names ambiguous and stable names mismatched", () => { + const named = (base: "0x1000" | "0x2000", name: string) => + functionDossierSchema.parse({ + ...dossier("return 0;", base), + procedure: { + ...procedure(base, name), + signature: "int helper(void)", + locals: [], + }, + }); + const generated = compareFunctions( + observe("1", named("0x1000", "sub_1000")), + observe("2", named("0x2000", "sub_2000")), + ); + expect(generated.function_match.status).toBe("ambiguous"); + expect(generated.status).toBe("unknown"); + expect( + generated.dimensions.find(({ dimension }) => dimension === "identity"), + ).toMatchObject({ status: "unknown" }); + + const stable = compareFunctions( + observe("3", named("0x1000", "firstFunction")), + observe("4", named("0x2000", "secondFunction")), + ); + expect(stable.function_match.status).toBe("mismatched"); + expect(stable.status).toBe("changed"); + }); + it.each(["sub_deallocate", "sub_deadbeef_handler", "fcn.dispatch"])( "recognizes the explicit symbol %s without a generated-name prefix false positive", (name) => { @@ -688,6 +716,236 @@ describe("function comparison identity and malformed CFG", () => { }); }); +const withLiterals = (base: "0x1000" | "0x2000", values: readonly string[]) => + functionDossierSchema.parse({ + ...dossier("return 0;", base), + swift_string_literals: values.map((value) => ({ + source_address: base, + value, + representation: "small" as const, + address: null, + registers: ["x0", "x1"] as const, + words: ["0x0", "0xe"] as const, + })), + }); +const stringsDimension = ( + leftDigit: string, + left: ReturnType | ReturnType, + rightDigit: string, + right: ReturnType | ReturnType, +) => + compareFunctions( + observe(leftDigit, left), + observe(rightDigit, right), + ).dimensions.find(({ dimension }) => dimension === "strings_names"); + +describe("function comparison Swift string literals", () => { + it("treats differing decoded literals as a strings and names change", () => { + expect( + stringsDimension( + "1", + withLiterals("0x1000", ["SleepDisabled"]), + "2", + withLiterals("0x2000", ["Holding awake"]), + ), + ).toMatchObject({ status: "changed" }); + }); + + it("matches the same decoded literal after the function is relocated", () => { + expect( + stringsDimension( + "3", + withLiterals("0x1000", ["SleepDisabled"]), + "4", + withLiterals("0x2000", ["SleepDisabled"]), + ), + ).toMatchObject({ status: "unchanged" }); + }); + + it("keeps strings and names unknown when only one dossier recorded Swift literals", () => { + expect( + stringsDimension( + "5", + withLiterals("0x1000", []), + "6", + dossier("return 0;", "0x2000"), + ), + ).toMatchObject({ + status: "unknown", + limitations: [ + "Swift String literal coverage differs or is partial; referenced strings and names match, and complete coverage does not establish a differing literal.", + ], + }); + }); + + it("reports a referenced name change even when Swift literal coverage differs", () => { + const named = functionDossierSchema.parse({ + ...dossier("return 0;", "0x2000"), + referenced_names: [ + { address: "0x3000", value: "helper", source_address: "0x2000" }, + ], + }); + expect( + stringsDimension("7", withLiterals("0x1000", []), "8", named), + ).toMatchObject({ status: "changed" }); + }); +}); + +describe("function comparison partial Swift literal coverage", () => { + it("does not equate a partial literal record with a complete one", () => { + const partial = functionDossierSchema.parse({ + ...withLiterals("0x2000", ["SleepDisabled"]), + swift_string_literal_omissions: [ + { + source_address: "0x2000", + representation: "large", + address: "0x3000", + byte_length: 64, + reason: "capacity", + registers: ["x0", "x1"], + words: ["0xd000000000000040", "0x8000000000002fe0"], + }, + ], + }); + expect( + stringsDimension( + "9", + withLiterals("0x1000", ["SleepDisabled"]), + "a", + partial, + ), + ).toMatchObject({ status: "unknown" }); + }); + + it("reports a literal that a complete dossier shows absent at a site the partial one decoded", () => { + const partial = functionDossierSchema.parse({ + ...withLiterals("0x2000", ["SleepDisabled"]), + swift_string_literal_omissions: [ + { + source_address: "0x2004", + representation: "small", + address: null, + byte_length: 4, + reason: "capacity", + registers: ["x0", "x1"], + words: ["0x732f4220", "0xe400000000000000"], + }, + ], + }); + expect( + stringsDimension("d", withLiterals("0x1000", []), "e", partial), + ).toMatchObject({ status: "changed" }); + }); + + it("does not interpret an unexamined path as a missing literal at a retained site", () => { + const partial = functionDossierSchema.parse({ + ...withLiterals("0x2000", ["A"]), + swift_string_literal_omissions: [ + { + source_address: "0x2004", + representation: "small", + address: null, + byte_length: 0, + reason: "capacity", + registers: ["x0", "x1"], + words: ["0x0", "0xe000000000000000"], + }, + ], + }); + const complete = withLiterals("0x1000", ["A", "B"]); + expect(stringsDimension("1", complete, "2", partial)).toMatchObject({ + status: "unknown", + }); + expect(stringsDimension("3", partial, "4", complete)).toMatchObject({ + status: "unknown", + }); + expect( + stringsDimension("5", withLiterals("0x1000", []), "6", partial), + ).toMatchObject({ status: "changed" }); + }); + + it("treats an omission record without decoded literals as partial coverage", () => { + const omissionOnly = functionDossierSchema.parse({ + ...dossier("return 0;", "0x2000"), + swift_string_literal_omissions: [ + { + source_address: "0x2004", + representation: "small", + address: null, + byte_length: 4, + reason: "capacity", + registers: ["x0", "x1"], + words: ["0x732f4220", "0xe400000000000000"], + }, + ], + }); + expect( + stringsDimension("f", dossier("return 0;", "0x1000"), "0", omissionOnly), + ).toMatchObject({ status: "unknown" }); + }); + + it("keeps different retained representations unknown when both dossiers are partial", () => { + const omission = { + source_address: "0x1004", + representation: "small", + address: null, + byte_length: 4, + reason: "capacity", + registers: ["x0", "x1"], + words: ["0x732f4220", "0xe400000000000000"], + } as const; + const small = functionDossierSchema.parse({ + ...withLiterals("0x1000", ["SleepDisabled"]), + swift_string_literal_omissions: [omission], + }); + const large = functionDossierSchema.parse({ + ...dossier("return 0;", "0x2000"), + swift_string_literals: [ + { + source_address: "0x2000", + value: "SleepDisabled", + representation: "large", + address: "0x3000", + registers: ["x0", "x1"], + words: ["0xd00000000000000d", "0x8000000000002fe0"], + }, + ], + swift_string_literal_omissions: [ + { ...omission, source_address: "0x2004" }, + ], + }); + expect(stringsDimension("1", small, "2", large)).toMatchObject({ + status: "unknown", + }); + }); + + it("keeps different retained literals unknown when both dossiers are partial", () => { + const partial = (base: "0x1000" | "0x2000", value: string) => + functionDossierSchema.parse({ + ...withLiterals(base, [value]), + swift_string_literal_omissions: [ + { + source_address: base === "0x1000" ? "0x1004" : "0x2004", + representation: "large", + address: "0x3000", + byte_length: 64, + reason: "capacity", + registers: ["x0", "x1"], + words: ["0xd000000000000040", "0x8000000000002fe0"], + }, + ], + }); + expect( + stringsDimension( + "b", + partial("0x1000", "SleepDisabled"), + "c", + partial("0x2000", "Holding awake"), + ), + ).toMatchObject({ status: "unknown" }); + }); +}); + const expectInlineComparisonEvidence = ( result: ReturnType, ): void => { diff --git a/src/domain/functionComparisonDimensions.ts b/src/domain/functionComparisonDimensions.ts index 27bb7b176..c5a0f373f 100644 --- a/src/domain/functionComparisonDimensions.ts +++ b/src/domain/functionComparisonDimensions.ts @@ -11,6 +11,7 @@ import { referenceProjection, sorted, stringAndNameProjection, + swiftLiteralSitesConflict, } from "./functionComparisonNormalization.js"; import type { DimensionName, @@ -139,8 +140,12 @@ const compareStringsAndNames = ( right: FunctionSnapshot, context: ComparisonContext, ): FunctionDimension => { - const leftValues = stringAndNameProjection(left); - const rightValues = stringAndNameProjection(right); + const complete = + swiftLiteralCoverage(left) === swiftLiteralCoverage(right) && + swiftLiteralCoverage(left) !== "partial"; + // Facts both dossiers record decide a change before Swift literal coverage can. + const leftValues = stringAndNameProjection(left, complete); + const rightValues = stringAndNameProjection(right, complete); if (leftValues === null || rightValues === null) return unresolvedDimension({ dimension: "strings_names", @@ -151,9 +156,40 @@ const compareStringsAndNames = ( "Reference source locations could not be normalized to function offsets.", ], }); - return compareValues("strings_names", leftValues, rightValues, context); + const compared = compareValues( + "strings_names", + leftValues, + rightValues, + context, + ); + if (complete || compared.status === "changed") return compared; + // A retained literal differs only when the opposite dossier establishes its absence. + if (swiftLiteralSitesConflict(left, right)) { + const leftAll = stringAndNameProjection(left); + const rightAll = stringAndNameProjection(right); + if (leftAll !== null && rightAll !== null) + return compareValues("strings_names", leftAll, rightAll, context); + } + return unresolvedDimension({ + dimension: "strings_names", + links: context.links, + leftCount: null, + rightCount: null, + limitations: [ + "Swift String literal coverage differs or is partial; referenced strings and names match, and complete coverage does not establish a differing literal.", + ], + }); }; +const swiftLiteralCoverage = ( + snapshot: FunctionSnapshot, +): "absent" | "complete" | "partial" => + (snapshot.dossier.swift_string_literal_omissions ?? []).length > 0 + ? "partial" + : snapshot.dossier.swift_string_literals === undefined + ? "absent" + : "complete"; + const compareReferences = ( left: FunctionSnapshot, right: FunctionSnapshot, diff --git a/src/domain/functionComparisonNormalization.ts b/src/domain/functionComparisonNormalization.ts index a676e1e49..f32220b0b 100644 --- a/src/domain/functionComparisonNormalization.ts +++ b/src/domain/functionComparisonNormalization.ts @@ -16,24 +16,21 @@ export const functionMatch = ( left: FunctionSnapshot, right: FunctionSnapshot, ): FunctionComparisonResult["function_match"] => { - if ( - !isAutoName(left.dossier.procedure.name) && - left.dossier.procedure.name === right.dossier.procedure.name - ) + const leftName = left.dossier.procedure.name; + const rightName = right.dossier.procedure.name; + const stableNames = !isAutoName(leftName) && !isAutoName(rightName); + if (stableNames && leftName === rightName) return { status: "matched", method: "symbol", - left_name: left.dossier.procedure.name, - right_name: right.dossier.procedure.name, + left_name: leftName, + right_name: rightName, }; return { - status: - left.dossier.procedure.name === right.dossier.procedure.name - ? "ambiguous" - : "mismatched", + status: stableNames ? "mismatched" : "ambiguous", method: "explicit", - left_name: left.dossier.procedure.name, - right_name: right.dossier.procedure.name, + left_name: leftName, + right_name: rightName, }; }; @@ -145,6 +142,7 @@ export const commentProjection = ( export const stringAndNameProjection = ( snapshot: FunctionSnapshot, + includeSwiftLiterals = true, ): readonly unknown[] | null => { const values = [ ...snapshot.dossier.referenced_strings.map( @@ -167,12 +165,90 @@ export const stringAndNameProjection = ( value, }), ), + ...(includeSwiftLiterals + ? (snapshot.dossier.swift_string_literals ?? []) + : [] + ).map( + ({ + source_address: sourceAddress, + value, + representation, + registers, + }) => ({ + kind: "swift_string", + source_offset: relativeAddress( + sourceAddress, + snapshot.dossier.procedure.address, + ), + value, + representation, + registers, + }), + ), ]; return values.some(({ source_offset: offset }) => offset === null) ? null : sorted(values); }; +/** + * A retained literal proves presence, not that every path through its site was examined. + * A differing retained fact proves a change only when the opposite dossier is complete + * and therefore establishes its absence. + */ +export const swiftLiteralSitesConflict = ( + left: FunctionSnapshot, + right: FunctionSnapshot, +): boolean => { + const leftSites = swiftLiteralSites(left); + const rightSites = swiftLiteralSites(right); + for (const site of new Set([ + ...leftSites.values.keys(), + ...rightSites.values.keys(), + ])) { + const leftValues = + leftSites.values.get(site) ?? (leftSites.complete ? [] : undefined); + const rightValues = + rightSites.values.get(site) ?? (rightSites.complete ? [] : undefined); + if (leftValues === undefined || rightValues === undefined) continue; + const leftFacts = new Set(leftValues); + const rightFacts = new Set(rightValues); + if ( + (rightSites.complete && + leftValues.some((value) => !rightFacts.has(value))) || + (leftSites.complete && rightValues.some((value) => !leftFacts.has(value))) + ) + return true; + } + return false; +}; + +const swiftLiteralSites = (snapshot: FunctionSnapshot) => { + const base = snapshot.dossier.procedure.address; + const omissions = snapshot.dossier.swift_string_literal_omissions ?? []; + const site = (sourceAddress: string, registers: readonly string[]) => { + const offset = relativeAddress(sourceAddress, base); + return offset === null ? null : canonicalJson([offset, registers]); + }; + // Keep the same literal facts the full strings_names projection compares. + const values = new Map(); + for (const literal of snapshot.dossier.swift_string_literals ?? []) { + const key = site(literal.source_address, literal.registers); + if (key === null) continue; + values.set( + key, + [ + ...(values.get(key) ?? []), + canonicalJson([literal.value, literal.representation]), + ].sort(compareUnicodeCodePoints), + ); + } + const complete = + snapshot.dossier.swift_string_literals !== undefined && + omissions.length === 0; + return { values, complete }; +}; + export const sorted = (values: readonly unknown[]): readonly unknown[] => values .map((value) => ({ diff --git a/src/domain/hopperValues.test.ts b/src/domain/hopperValues.test.ts index 7af5df704..c5931cd35 100644 --- a/src/domain/hopperValues.test.ts +++ b/src/domain/hopperValues.test.ts @@ -8,7 +8,6 @@ import { parseNames, parseProcedures, parseRelatedAddresses, - parseSegmentImageBase, parseSegments, } from "./hopperValues.js"; @@ -66,19 +65,6 @@ describe("Hopper boundary values", () => { ok: true, value: ["fixture"], }); - expect( - parseSegmentImageBase([ - { image_base: "0x100000" }, - { image_base: "0x100000" }, - ]), - ).toBe("0x100000"); - expect( - parseSegmentImageBase([ - { image_base: "0x100000" }, - { image_base: "0x200000" }, - ]), - ).toBeUndefined(); - expect(parseSegmentImageBase([{ name: "__TEXT" }])).toBeUndefined(); }); it.each([ diff --git a/src/domain/hopperValues.ts b/src/domain/hopperValues.ts index 5e5a3987b..d6344ff3e 100644 --- a/src/domain/hopperValues.ts +++ b/src/domain/hopperValues.ts @@ -20,6 +20,8 @@ export interface SegmentSummary { readonly readable: boolean | null; readonly writable: boolean | null; readonly executable: boolean | null; + readonly image_base?: string; + readonly address_space?: string; } /** Return a bounded byte distance between canonical addresses in one space. */ @@ -58,10 +60,9 @@ const segmentSchema = z.object({ readable: z.boolean().nullable(), writable: z.boolean().nullable(), executable: z.boolean().nullable(), + image_base: z.string().min(1).exactOptional(), + address_space: z.string().min(1).exactOptional(), }); -const segmentImageBaseSchema = z - .object({ image_base: z.string().min(1) }) - .passthrough(); const unavailableAnalysisFactSchema = z .object({ available: z.literal(false), reason: z.string() }) .strict(); @@ -289,6 +290,70 @@ export const unresolvedCallSchema = z.strictObject({ reason: z.string(), }); +const swiftStringLiteralFields = { + source_address: z + .string() + .describe( + "Instruction that consumes both words: a call, tail call, return, or paired store.", + ), + value: z + .string() + .describe("Decoded UTF-8 text, including the canonical empty string."), + registers: z.tuple([z.string(), z.string()]), + words: z + .tuple([z.string(), z.string()]) + .describe("Observed constant register values, in register order."), +}; + +/** A Swift String literal decoded from the two words of its ABI value, not a data reference. */ +export const swiftStringLiteralSchema = z.discriminatedUnion("representation", [ + z.strictObject({ + ...swiftStringLiteralFields, + representation: z + .literal("small") + .describe("The UTF-8 bytes are inside the two words."), + address: z.null().describe("A small literal has no storage of its own."), + }), + z.strictObject({ + ...swiftStringLiteralFields, + representation: z + .literal("large") + .describe( + "An immortal literal whose words hold its count and a pointer to its bytes.", + ), + address: z.string().describe("Address of a large literal's bytes."), + }), +]); + +const swiftStringLiteralOmissionFields = { + source_address: swiftStringLiteralFields.source_address, + byte_length: z.number().int().min(0).max(Number.MAX_SAFE_INTEGER), + reason: z + .literal("capacity") + .describe( + "The record did not fit the provider's memory budget for this dossier; decoding stopped here, so later use sites were not examined.", + ), + registers: swiftStringLiteralFields.registers, + words: swiftStringLiteralFields.words, +}; + +/** A Swift literal whose layout matched but whose record did not fit; decoding stopped there. */ +export const swiftStringLiteralOmissionSchema = z.discriminatedUnion( + "representation", + [ + z.strictObject({ + ...swiftStringLiteralOmissionFields, + representation: z.literal("small"), + address: z.null(), + }), + z.strictObject({ + ...swiftStringLiteralOmissionFields, + representation: z.literal("large"), + address: z.string().describe("Address of the literal's bytes."), + }), + ], +); + /** Observed endpoints with complete or partial reference classification evidence. */ export const referenceEdgeSchema = z .object({ @@ -340,6 +405,18 @@ export const functionDossierSchema = z }) .strict(), ), + swift_string_literals: z + .array(swiftStringLiteralSchema) + .exactOptional() + .describe( + "Swift String literals decoded from constant register values. Absent when the provider does not decode them for this target.", + ), + swift_string_literal_omissions: z + .array(swiftStringLiteralOmissionSchema) + .exactOptional() + .describe( + "The literal at which decoding stopped for capacity. Present exactly when swift_string_literals is present; empty when decoding examined every use site.", + ), basic_blocks: z.array( z .object({ @@ -415,7 +492,7 @@ export const parseRelatedAddresses = ( : invalid(`${relation} list`, parsed.error); }; -/** Parse direct or wrapped Hopper segment records. */ +/** Preserve reported address context in direct or wrapped provider segments. */ export const parseSegments = ( value: JsonValue, ): Result => { @@ -427,14 +504,15 @@ export const parseSegments = ( : invalid("segment list", parsed.error); }; -/** Return a provider-reported image base only when every segment agrees on it. */ -export const parseSegmentImageBase = (value: JsonValue): string | undefined => { - const parsed = z - .array(segmentImageBaseSchema) - .safeParse(unwrapProperty(value, "segments")); - if (!parsed.success) return undefined; - const bases = new Set(parsed.data.map(({ image_base }) => image_base)); - return bases.size === 1 ? parsed.data[0]?.image_base : undefined; +/** Return a reported overview base only when every parsed segment agrees. */ +export const parseSegmentImageBase = ( + segments: readonly SegmentSummary[], +): string | undefined => { + const base = segments[0]?.image_base; + return base !== undefined && + segments.every((segment) => segment.image_base === base) + ? base + : undefined; }; /** Parse direct or wrapped Hopper document names. */ diff --git a/src/domain/javascript/javascriptApplicationGraph.fixture.ts b/src/domain/javascript/javascriptApplicationGraph.fixture.ts index 3002b822f..0e78e5115 100644 --- a/src/domain/javascript/javascriptApplicationGraph.fixture.ts +++ b/src/domain/javascript/javascriptApplicationGraph.fixture.ts @@ -6,6 +6,7 @@ import { } from "./javascriptApplicationGraph.js"; import type { ApplicationNode } from "./javascriptApplicationGraphSchemas.js"; import type { ApplicationGraphEvidence } from "./javascriptApplicationEvidenceSchemas.js"; +import { resolveJavaScriptSourceMapReferences } from "./javascriptSourceMapPaths.js"; export const APPLICATION_GRAPH_DIGESTS = { package: "1".repeat(64), @@ -22,6 +23,20 @@ export const completeCoverage = { limits: [], } satisfies ApplicationGraphEvidence["coverage"]; +export const sourceMapReference = ( + sourceName: string, + sourceRoot: string | null, + mapPath: string, +) => { + const reference = resolveJavaScriptSourceMapReferences( + [sourceName], + sourceRoot, + mapPath, + )[0]; + if (reference == null) throw new TypeError("Expected a source-map reference"); + return reference; +}; + const extractor = (operation: string) => ({ name: "rea-test-fixture", version: "1.0.0", diff --git a/src/domain/javascript/javascriptApplicationGraph.test.ts b/src/domain/javascript/javascriptApplicationGraph.test.ts index 5e396a545..5bf917480 100644 --- a/src/domain/javascript/javascriptApplicationGraph.test.ts +++ b/src/domain/javascript/javascriptApplicationGraph.test.ts @@ -20,10 +20,9 @@ import { inferredArtifactEvidence, nodeByLabel, runtimeEvidence, + sourceMapReference, unknownEvidence, } from "./javascriptApplicationGraph.fixture.js"; - -import { resolveJavaScriptSourceMapReference } from "./javascriptSourceMapPaths.js"; const createLargeNativeExportGraph = (): JavaScriptApplicationGraph => { const properties = Object.fromEntries( Array.from({ length: 1_001 }, (_, index) => [ @@ -438,7 +437,7 @@ describe("JavaScript Application Graph: source maps and observations", () => { observations: [ { label: "src/editor.ts", - source_map_reference: resolveJavaScriptSourceMapReference( + source_map_reference: sourceMapReference( "webpack:///src/editor.ts", null, "dist/app.js.map", @@ -535,7 +534,7 @@ describe("JavaScript Application Graph: source-map digest consistency", () => { observations: [ { label: null, - source_map_reference: resolveJavaScriptSourceMapReference( + source_map_reference: sourceMapReference( "src/editor.ts", null, "dist/app.js.map", diff --git a/src/domain/javascript/javascriptBundlerRegistrationIdentity.test.ts b/src/domain/javascript/javascriptBundlerRegistrationIdentity.test.ts new file mode 100644 index 000000000..1b251e313 --- /dev/null +++ b/src/domain/javascript/javascriptBundlerRegistrationIdentity.test.ts @@ -0,0 +1,78 @@ +import { expect, it } from "vitest"; +import { analyzeJavaScriptStaticSource } from "./javascriptStaticAnalysis.js"; + +const registration = (runtime: string, callback: string, chunk = 1) => + "(globalThis." + + runtime + + " = globalThis." + + runtime + + " || []).push([[" + + chunk + + "], {a: function() {}}, " + + callback + + "]);"; + +it.each(["webpackChunkFixture", "rspackChunkFixture"])( + "keeps differing callback facts for %s independent of source order", + (runtime) => { + const first = registration( + runtime, + 'function(r) { r(7); r.e("chunk-a"); }', + ); + const second = registration( + runtime, + 'function(r) { r(8); r.e("chunk-b"); }', + ); + for (const source of [first + "\n" + second, second + "\n" + first]) { + const result = analyzeJavaScriptStaticSource(source); + expect(result.parse_status).toBe("complete"); + expect(result.bundler_registrations).toHaveLength(2); + expect( + result.bundler_registrations.flatMap((r) => r.entry_module_keys).sort(), + ).toEqual(["7", "8"]); + expect( + result.bundler_registrations.flatMap((r) => r.async_chunk_keys).sort(), + ).toEqual(["chunk-a", "chunk-b"]); + } + expect( + analyzeJavaScriptStaticSource(first + "\n" + first).bundler_registrations, + ).toHaveLength(1); + expect( + analyzeJavaScriptStaticSource(first).bundler_registrations[0], + ).toMatchObject({ + entry_module_keys: ["7"], + async_chunk_keys: ["chunk-a"], + }); + expect( + analyzeJavaScriptStaticSource(second).bundler_registrations[0], + ).toMatchObject({ + entry_module_keys: ["8"], + async_chunk_keys: ["chunk-b"], + }); + }, +); + +it.each([ + ["function(r) { r(7); }", "function(r) { r(8); }"], + ['function(r) { r.e("a"); }', 'function(r) { r.e("b"); }'], + ["function(r) {}", "function(s) {}"], + ["function(r) {}", "function(r) { r(dynamic); }"], + ["function(r) {}", "function(r) { r.e(dynamic); }"], +])("retains callback metadata differences: %s / %s", (left, right) => { + const result = analyzeJavaScriptStaticSource( + registration("webpackChunkFixture", left) + + "\n" + + registration("webpackChunkFixture", right), + ); + expect(result.bundler_registrations).toHaveLength(2); +}); + +it("retains distinct chunk identities with identical callbacks", () => { + const callback = "function(r) { r(7); }"; + const result = analyzeJavaScriptStaticSource( + registration("webpackChunkFixture", callback, 1) + + "\n" + + registration("webpackChunkFixture", callback, 2), + ); + expect(result.bundler_registrations).toHaveLength(2); +}); diff --git a/src/domain/javascript/javascriptModuleDeclaration.ts b/src/domain/javascript/javascriptModuleDeclaration.ts new file mode 100644 index 000000000..d7a8d6754 --- /dev/null +++ b/src/domain/javascript/javascriptModuleDeclaration.ts @@ -0,0 +1,60 @@ +import * as t from "@babel/types"; + +/** Whether TypeScript explicitly erases this entire import declaration. */ +export const isTypeOnlyImportDeclaration = ( + node: t.ImportDeclaration, +): boolean => node.importKind === "type" || node.importKind === "typeof"; + +/** Whether TypeScript explicitly erases this entire export declaration. */ +export const isTypeOnlyExportDeclaration = ( + node: t.ExportNamedDeclaration | t.ExportAllDeclaration, +): boolean => + node.exportKind === "type" || + (t.isExportNamedDeclaration(node) && + node.declaration !== null && + (t.isTSInterfaceDeclaration(node.declaration) || + t.isTSTypeAliasDeclaration(node.declaration) || + t.isTSDeclareFunction(node.declaration))); + +/** Whether this import binding is explicitly type-only. */ +export const isTypeOnlyImportSpecifier = (node: t.ImportSpecifier): boolean => + node.importKind === "type" || node.importKind === "typeof"; + +/** Whether this exported binding is explicitly type-only. */ +export const isTypeOnlyExportSpecifier = (node: t.ExportSpecifier): boolean => + node.exportKind === "type"; + +/** A regular import whose bindings are all erased but may retain side effects. */ +export const hasOnlyInlineTypeImportSpecifiers = ( + node: t.ImportDeclaration, +): boolean => + !isTypeOnlyImportDeclaration(node) && + node.specifiers.length > 0 && + node.specifiers.every( + (specifier) => + t.isImportSpecifier(specifier) && isTypeOnlyImportSpecifier(specifier), + ); + +/** A regular re-export whose bindings are all erased but may retain side effects. */ +export const hasOnlyInlineTypeExportSpecifiers = ( + node: t.ExportNamedDeclaration, +): boolean => + !isTypeOnlyExportDeclaration(node) && + node.source != null && + node.specifiers.length > 0 && + node.specifiers.every( + (specifier) => + t.isExportSpecifier(specifier) && isTypeOnlyExportSpecifier(specifier), + ); + +/** Whether compiler options leave an inline type-only declaration's side effect unknown. */ +export const hasUncertainTypeOnlyModuleSideEffect = ( + program: t.Program, +): boolean => + program.body.some( + (node) => + (t.isImportDeclaration(node) && + hasOnlyInlineTypeImportSpecifiers(node)) || + (t.isExportNamedDeclaration(node) && + hasOnlyInlineTypeExportSpecifiers(node)), + ); diff --git a/src/domain/javascript/javascriptModuleResolutionMode.ts b/src/domain/javascript/javascriptModuleResolutionMode.ts new file mode 100644 index 000000000..d14b91bd9 --- /dev/null +++ b/src/domain/javascript/javascriptModuleResolutionMode.ts @@ -0,0 +1,23 @@ +import { posix } from "node:path"; + +/** Loader evidence retained for a source-level module reference. */ +export type JavaScriptModuleResolutionMode = + | "import" + | "require" + | "compiler-dependent"; + +/** Derive source mode only when syntax or an explicit source format owns it. */ +export const moduleResolutionModeForJavaScriptReference = ( + sourcePath: string, + referenceKind: "static-import" | "dynamic-import" | "require", + explicitMode?: JavaScriptModuleResolutionMode, +): JavaScriptModuleResolutionMode => { + if (explicitMode !== undefined) return explicitMode; + if (referenceKind === "require") return "require"; + if (referenceKind === "dynamic-import") return "import"; + const extension = posix.extname(sourcePath).toLowerCase(); + if (extension === ".cts") return "require"; + if (extension === ".mts") return "import"; + if (extension === ".ts" || extension === ".tsx") return "compiler-dependent"; + return "import"; +}; diff --git a/src/domain/javascript/javascriptModuleSpecifier.ts b/src/domain/javascript/javascriptModuleSpecifier.ts new file mode 100644 index 000000000..f52f081dc --- /dev/null +++ b/src/domain/javascript/javascriptModuleSpecifier.ts @@ -0,0 +1,6 @@ +/** Relative module syntax differs from an arbitrary dot-prefixed filename. */ +export const isRelativeJavaScriptModuleSpecifier = (value: string): boolean => + value === "." || + value === ".." || + value.startsWith("./") || + value.startsWith("../"); diff --git a/src/domain/javascript/javascriptRuntimeStaticCandidates.ts b/src/domain/javascript/javascriptRuntimeStaticCandidates.ts index 1f02c820d..810ab7f5e 100644 --- a/src/domain/javascript/javascriptRuntimeStaticCandidates.ts +++ b/src/domain/javascript/javascriptRuntimeStaticCandidates.ts @@ -1,4 +1,4 @@ -import { posix, relative, win32 } from "node:path"; +import { posix, win32 } from "node:path"; import type { ApplicationNode } from "./javascriptApplicationGraphSchemas.js"; import type { JavaScriptApplicationGraph } from "./javascriptApplicationGraph.js"; @@ -216,7 +216,7 @@ const pathBelowRoot = (root: string, value: string): string | null => { const remainder = rootStyle === "windows" ? win32.relative(root, value) - : relative(root, value); + : posix.relative(root, value); const portableRemainder = remainder.replaceAll("\\", "/"); if ( remainder === "" || diff --git a/src/domain/javascript/javascriptSemanticAnalysis.getter_aliases.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.getter_aliases.test.ts new file mode 100644 index 000000000..1eeaec350 --- /dev/null +++ b/src/domain/javascript/javascriptSemanticAnalysis.getter_aliases.test.ts @@ -0,0 +1,351 @@ +import { describe, expect, it } from "vitest"; +import { analyzeJavaScriptSemantics } from "./javascriptSemanticAnalysis.js"; +import { onlyCallable } from "./javascriptSemanticAnalysis.fixture.js"; + +const returnedValue = (getter: string, write = "box.v.x = 2;") => + onlyCallable( + analyzeJavaScriptSemantics(`export function result() { + const shared = { x: 1, y: 1 }; + const box = { s: shared, get v() { ${getter} } }; + ${write} + return shared.x; + }`), + "result", + ).returnSites[0]?.value; + +describe("projected private getter receivers", () => { + it.each([ + [ + "destructured self field", + "shared", + "peer = this;", + "const { peer: self } = this; return self.#s;", + true, + "unknown", + ], + [ + "unrelated private child", + "{ peer: shared }", + "peer = this;", + "const { peer: self } = this; return self.#s;", + true, + "literal", + ], + [ + "read without write", + "shared", + "peer = this;", + "const { peer: self } = this; return self.#s;", + false, + "literal", + ], + [ + "two self fields", + "shared", + "peer = this; next = this.peer;", + "const { next: self } = this; return self.#s;", + true, + "unknown", + ], + [ + "nested receiver projection", + "shared", + "peer = this;", + "const { peer: { peer: self } } = this; return self.#s;", + true, + "unknown", + ], + [ + "member receiver alias", + "shared", + "peer = this;", + "const self = this.peer.peer; return self.#s;", + true, + "unknown", + ], + [ + "projected result suffix", + "{ child: shared }", + "peer = this;", + "const { peer: self } = this; return self.#s.child;", + true, + "unknown", + ], + [ + "external receiver", + "shared", + "peer = external;", + "const { peer: self } = this; return self.#s;", + true, + "literal", + ], + [ + "field receiver cycle", + "shared", + "peer = this.other; other = this.peer;", + "const { peer: self } = this; return self.#s;", + true, + "literal", + ], + ] as const)( + "keeps receiver and result paths separate for %s", + (_name, value, fields, getter, write, status) => { + const ir = analyzeJavaScriptSemantics(`export function result() { + const shared = { x: 1 }; + class Box { + #s = ${value}; ${fields} + get v() { ${getter} } + } + const box = new Box(); + ${write ? "box.v.x = 2;" : "const saved = box.v;"} + return shared.x; + }`); + expect(onlyCallable(ir, "result").returnSites[0]?.value).toMatchObject( + status === "literal" ? { status, value: 1 } : { status }, + ); + }, + ); +}); + +describe("private receiver field ownership", () => { + it("bounds growing receiver projections under cyclic class inheritance", () => { + const ir = analyzeJavaScriptSemantics(`export function result() { + const shared = { x: 1 }; + class A extends B { + #s = shared; + peer = this.other.other; + get v() { const { peer: self } = this; return self.#s; } + } + class B extends A { other = this.peer.peer; } + new A().v.x = 2; + return shared.x; + }`); + expect(onlyCallable(ir, "result").returnSites[0]?.value).toMatchObject({ + status: "literal", + value: 1, + }); + }, 1000); + + it.each([ + [ + "own static receiver", + "class Box { static #s=shared; static peer=this; static get v(){const {peer:self}=this;return self.#s;} } Box.v.x=2;", + "unknown", + ], + [ + "inherited instance receiver", + "class Base {peer=this;} class Box extends Base {#s=shared;get v(){const {peer:self}=this;return self.#s;}} new Box().v.x=2;", + "unknown", + ], + [ + "inherited static field retains its declaring class", + "class Base {static peer=this;} class Box extends Base {static #s=shared;static get v(){const {peer:self}=this;return self.#s;}} Box.v.x=2;", + "literal", + ], + [ + "inherited static getter's own private field", + "class Base {static #s=shared;static peer=this;static get v(){const {peer:self}=this;return self.#s;}} class Box extends Base {} Box.v.x=2;", + "unknown", + ], + [ + "a field that is not initialized yet", + "class Box {#s=shared;peer=this.later;later=this;get v(){const {peer:self}=this;return self.#s;}} new Box().v.x=2;", + "literal", + ], + [ + "remaining getter projection after a captured field", + "class Box {#s=shared;peer=this;next=this.peer;later=this;get v(){const {next:{later:self}}=this;return self.#s;}} new Box().v.x=2;", + "unknown", + ], + [ + "a replaced self field", + "class Box {#s=shared;peer=this;peer=external;get v(){const {peer:self}=this;return self.#s;}} new Box().v.x=2;", + "literal", + ], + ] as const)("respects %s", (_name, body, status) => { + const ir = analyzeJavaScriptSemantics(`export function result() { + const shared = { x: 1 }; + ${body} + return shared.x; + }`); + expect(onlyCallable(ir, "result").returnSites[0]?.value).toMatchObject( + status === "literal" ? { status, value: 1 } : { status }, + ); + }); +}); + +describe("getter-local receiver aliases (#2653)", () => { + it.each([ + ["initialized alias", "const self = this; return self.s;"], + ["assigned alias", "let self; self = this; return self.s;"], + [ + "chained assignment", + "let self; let alias; self = alias = this; return self.s;", + ], + ["destructured receiver", "const { s: value } = this; return value;"], + ["alias chain", "const self = this; const saved = self; return saved.s;"], + [ + "capture before reassignment", + "let self = this; const saved = self; self = { s: { x: 1 } }; return saved.s;", + ], + [ + "conditional receiver alias", + "const self = flag ? this : { s: { x: 1 } }; return self.s;", + ], + ])("invalidates shared values through %s", (_name, getter) => { + expect(returnedValue(getter)?.status).toBe("unknown"); + }); + + it.each([ + ["no write", "const self = this; return self.s;", "const saved = box.v;"], + [ + "an unrelated property write", + "const self = this; return self.s;", + "box.v.y = 2;", + ], + [ + "a cycle that does not grow the selected path", + "let self = this; while (flag) self = self; return self.s;", + "box.v.y = 2;", + ], + [ + "an alias replaced before return", + "let self = this; self = { s: { x: 1 } }; return self.s;", + "box.v.x = 2;", + ], + [ + "a shadowing block binding", + "const self = this; { const self = { s: { x: 1 } }; return self.s; }", + "box.v.x = 2;", + ], + [ + "a nested function's receiver", + "function other() { const self = this; return self.s; } return other();", + "box.v.x = 2;", + ], + [ + "an alias assigned in a different receiver scope", + "let self; function assign() { self = this; } assign(); return self.s;", + "box.v.x = 2;", + ], + [ + "an excluded destructuring default", + "const { value = shared } = { value: { x: 1 } }; return value;", + "box.v.x = 2;", + ], + ])("preserves known values with %s", (_name, getter, write) => { + expect(returnedValue(getter, write)).toMatchObject({ + status: "literal", + value: 1, + }); + }); +}); + +describe("getter receiver alias boundaries", () => { + it("does not reinterpret an outer this binding as the getter receiver", () => { + const ir = analyzeJavaScriptSemantics(`export function result() { + const shared = { x: 1 }; + const outside = this; + const box = { s: shared, get v() { return outside.s; } }; + box.v.x = 2; + return shared.x; + }`); + expect(onlyCallable(ir, "result").returnSites[0]?.value).toMatchObject({ + status: "literal", + value: 1, + }); + }); + + it("retains a private field's lexical owner through a receiver alias", () => { + const ir = analyzeJavaScriptSemantics(`export function result() { + const shared = { x: 1 }; + class Box { + #s = shared; + get v() { const self = this; return self.#s; } + } + new Box().v.x = 2; + return shared.x; + }`); + expect(onlyCallable(ir, "result").returnSites[0]?.value.status).toBe( + "unknown", + ); + }); + + it("does not append a receiver projection to a private field's result path", () => { + const ir = analyzeJavaScriptSemantics(`export function result() { + const shared = { x: 1 }; + class Box { + #s = { peer: shared }; + peer = this; + get v() { const { peer: self } = this; return self.#s; } + } + new Box().v.x = 2; + return shared.x; + }`); + expect(onlyCallable(ir, "result").returnSites[0]?.value).toMatchObject({ + status: "literal", + value: 1, + }); + }); + + it("resolves a long local alias chain without recursive expansion", () => { + const aliases = ["const a0 = this;"]; + for (let index = 1; index <= 2000; index++) + aliases.push(`const a${String(index)} = a${String(index - 1)};`); + expect(returnedValue(`${aliases.join("\n")} return a2000.s;`)?.status).toBe( + "unknown", + ); + }); + + it("bounds a growing alias cycle without losing the original receiver", () => { + expect( + returnedValue( + "let self = this; while (flag) self = self.s; return self.s;", + )?.status, + ).toBe("unknown"); + }); + + it.each([true, false])( + "retains deeper shared references through an alias cycle (write=%s)", + (write) => { + const ir = analyzeJavaScriptSemantics(`export function result() { + const shared = { x: 1 }; + const box = { + start: { child: { child: shared } }, + get v() { + let self = this.start; + while (flag) self = self.child; + return self; + } + }; + ${write ? "box.v.x = 2;" : "const saved = box.v;"} + return shared.x; + }`); + expect(onlyCallable(ir, "result").returnSites[0]?.value).toMatchObject( + write ? { status: "unknown" } : { status: "literal", value: 1 }, + ); + }, + ); + + it("does not widen a growing cycle to unrelated receiver properties", () => { + const ir = analyzeJavaScriptSemantics(`export function result() { + const shared = { x: 1 }; + const unrelated = { x: 1 }; + const box = { + start: { child: { child: shared } }, + unrelated, + get v() { + let self = this.start; + while (flag) self = self.child; + return self; + } + }; + box.v.x = 2; + return unrelated.x; + }`); + expect(onlyCallable(ir, "result").returnSites[0]?.value).toMatchObject({ + status: "literal", + value: 1, + }); + }); +}); diff --git a/src/domain/javascript/javascriptSemanticAnalysis.getter_receivers.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.getter_receivers.test.ts index 3b83bf482..c3b38842a 100644 --- a/src/domain/javascript/javascriptSemanticAnalysis.getter_receivers.test.ts +++ b/src/domain/javascript/javascriptSemanticAnalysis.getter_receivers.test.ts @@ -33,6 +33,38 @@ const getterReceivers = [ "nested owner", "const outer = { box: { x: 1, get self() { return this; } } }; outer.box.self.x = 2; return outer.box.x;", ], + [ + "array element", + "const arr = [{ x: 1, get self() { return this; } }]; arr[0].self.x = 2; return arr[0].x;", + ], + [ + "array element after a hole", + "const arr = [, { x: 1, get self() { return this; } }]; arr[1].self.x = 2; return arr[1].x;", + ], + [ + "mixed object and array owners", + "const outer = { rows: [{ box: { x: 1, get self() { return this; } } }] }; outer.rows[0].box.self.x = 2; return outer.rows[0].box.x;", + ], + [ + "assigned owner", + "let box; box = { x: 1, get self() { return this; } }; box.self.x = 2; return box.x;", + ], + [ + "saved array receiver alias", + "const arr = [{ x: 1, get self() { return this; } }]; const saved = arr[0].self; saved.x = 2; return arr[0].x;", + ], + [ + "old receiver captured inside its replacement initializer", + "const old = { x: 1 }; let box = old; box = { old: box, get self() { return this; } }; box.self.old.x = 2; return old.x;", + ], + [ + "array receiver after an unknown spread", + "const arr = [...items, { x: 1, get self() { return this; } }]; arr[index].self.x = 2; return arr[index].x;", + ], + [ + "saved receiver before its array alias is reassigned", + "const original = [{ x: 1, get self() { return this; } }]; let arr = original; const saved = arr[0].self; arr = [{ x: 1 }]; saved.x = 2; return original[0].x;", + ], [ "inherited receiver field", "const shared = { x: 1 }; class A { get v() { return this.s; } } class B extends A { s = shared; } new B().v.x = 2; return shared.x;", @@ -75,7 +107,7 @@ const getterReceivers = [ ], ] as const; -describe("getter receiver mutation aliases (#1665)", () => { +describe("getter receiver mutation aliases (#1665, #2654)", () => { it.each(getterReceivers)( "does not report a stale literal through %s", (_shape, body) => { @@ -103,6 +135,42 @@ describe("getter receiver mutation aliases (#1665)", () => { "unrelated sibling", "const box = { x: 1, y: 1, get self() { return this; } }; box.self.y = 2; return box.x;", ], + [ + "array receiver read without write", + "const arr = [{ x: 1, get self() { return this; } }]; const saved = arr[0].self; return arr[0].x;", + ], + [ + "assigned receiver read without write", + "let box; box = { x: 1, get self() { return this; } }; const saved = box.self; return box.x;", + ], + [ + "unrelated array receiver property", + "const arr = [{ x: 1, y: 1, get self() { return this; } }]; arr[0].self.y = 2; return arr[0].x;", + ], + [ + "unrelated array element", + "const arr = [{ x: 1, get self() { return this; } }, { x: 1 }]; arr[0].self.x = 2; return arr[1].x;", + ], + [ + "array replacement after a receiver was captured", + "const original = [{ x: 1, get self() { return this; } }]; const replacement = [{ x: 1 }]; let arr = original; const saved = arr[0].self; arr = replacement; saved.x = 2; return replacement[0].x;", + ], + [ + "known array prefix before a spread receiver", + "const arr = [{ x: 1 }, ...items, { x: 1, get self() { return this; } }]; arr[1].self.x = 2; return arr[0].x;", + ], + [ + "outer binding shadowed by an assigned receiver", + "let box; box = { x: 1 }; { let box; box = { x: 1, get self() { return this; } }; box.self.x = 2; } return box.x;", + ], + [ + "old object before an unconditional receiver assignment", + "const old = { x: 1 }; let box = old; box = { x: 1, get self() { return this; } }; box.self.x = 2; return old.x;", + ], + [ + "unmodified old receiver captured inside its replacement initializer", + "const old = { x: 1 }; let box = old; box = { x: 1, old: box, get self() { return this; } }; box.self.x = 2; return old.x;", + ], [ "replace getter", "const shared = { x: 1 }; const box = { s: shared, get v() { return this.s; } }; box.v = {}; return shared.x;", diff --git a/src/domain/javascript/javascriptSemanticAnalysis.require_links.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.require_links.test.ts index 333ad4f9c..a0d93c3c3 100644 --- a/src/domain/javascript/javascriptSemanticAnalysis.require_links.test.ts +++ b/src/domain/javascript/javascriptSemanticAnalysis.require_links.test.ts @@ -22,6 +22,11 @@ describe("TypeScript import-equals links", () => { localName: "dependency", exportedName: null, }), + expect.objectContaining({ + kind: "type-import", + specifier: "./types", + localName: "types", + }), ]); }); @@ -51,6 +56,57 @@ describe("TypeScript import-equals links", () => { }); }); +describe("TypeScript type-only module links", () => { + it("retains declarations without runtime bindings or callable claims", () => { + const ir = analyzeJavaScriptSemantics(` + import type { Whole } from "./whole"; + import { type Inline } from "./inline"; + import { type Erased, value } from "./mixed"; + export type { Reexported } from "./reexported"; + export { type InlineExport } from "./inline-export"; + export type LocalType = string; + export interface LocalInterface { value: string } + function hiddenFunction() { return 1; } + export type { hiddenFunction }; + `); + + const externalLinks = ir.moduleLinks + .filter(({ specifier }) => specifier !== null) + .map( + ({ kind, specifier, importedName, exportedName }) => + [kind, specifier, importedName, exportedName] as const, + ); + expect(externalLinks).toEqual([ + ["type-import", "./whole", "Whole", null], + ["type-import", "./inline", "Inline", null], + ["type-import", "./mixed", "Erased", null], + ["import", "./mixed", "value", null], + ["type-import", "./reexported", "Reexported", "Reexported"], + ["type-import", "./inline-export", "InlineExport", "InlineExport"], + ]); + + const typeLinks = ir.moduleLinks + .filter(({ kind }) => kind === "type-import") + .map( + ({ specifier, localName, exportedName, callableId }) => + [specifier, localName, exportedName, callableId] as const, + ); + expect(typeLinks).toEqual([ + ["./whole", "Whole", null, null], + ["./inline", "Inline", null, null], + ["./mixed", "Erased", null, null], + ["./reexported", null, "Reexported", null], + ["./inline-export", null, "InlineExport", null], + [null, "LocalType", "LocalType", null], + [null, "LocalInterface", "LocalInterface", null], + [null, "hiddenFunction", "hiddenFunction", null], + ]); + expect(ir.limitations).toContain( + "Compiler options are unavailable; inline type-only imports or re-exports may preserve a module side-effect dependency.", + ); + }); +}); + describe("CommonJS dependency links", () => { it("retains an import-equals binding and its module origin", () => { const ir = analyzeJavaScriptSemantics(` @@ -76,6 +132,12 @@ describe("CommonJS dependency links", () => { importedName: null, localName: "typed", }), + expect.objectContaining({ + kind: "type-import", + specifier: "type-only", + importedName: null, + localName: "TypeOnly", + }), ]); expect(ir.references.filter(({ name }) => name === "typed")).toHaveLength( 1, diff --git a/src/domain/javascript/javascriptSemanticAnalysis.structure.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.structure.test.ts index 2cd9fba80..2312eff5d 100644 --- a/src/domain/javascript/javascriptSemanticAnalysis.structure.test.ts +++ b/src/domain/javascript/javascriptSemanticAnalysis.structure.test.ts @@ -72,9 +72,9 @@ describe("JavaScript semantic analysis: structure 1", () => { }); const shadow = onlyBinding(ir, "ipcRenderer"); expect(shadow.provenance).toEqual({ - status: "local", + status: "unknown", origins: [], - reason: null, + reason: "Parameter provenance depends on caller arguments.", }); const shadowReference = ir.references.find( ({ name, bindingId, role }) => diff --git a/src/domain/javascript/javascriptSemanticAnalysis.ts b/src/domain/javascript/javascriptSemanticAnalysis.ts index b56e1a112..f6c6db25d 100644 --- a/src/domain/javascript/javascriptSemanticAnalysis.ts +++ b/src/domain/javascript/javascriptSemanticAnalysis.ts @@ -43,6 +43,7 @@ import { import { semanticRequireOrigin } from "./javascriptSemanticRequireOrigin.js"; import { collectSemanticMemberMutationsSteps } from "./javascriptSemanticMemberMutations.js"; import { indexJavaScriptAmbientWritesSteps } from "./javascriptSemanticAmbientWrites.js"; +import { hasUncertainTypeOnlyModuleSideEffect } from "./javascriptModuleDeclaration.js"; import { collectSemanticReturns, resolveSemanticModuleCallables, @@ -67,6 +68,7 @@ import { import { isJavaScriptRpcClientFactory } from "./javascriptRpcEndpoints.js"; import { createSemanticModuleInterop } from "./javascriptModuleInterop.js"; import { SemanticPropertyPathCoverage } from "./javascriptSemanticPropertyPaths.js"; +import type { JavaScriptModuleResolutionMode } from "./javascriptModuleResolutionMode.js"; interface BindPatternInput { readonly pattern: t.Node; @@ -137,7 +139,7 @@ export function* indexJavaScriptBindingsSteps( /** Recover semantics from an already parsed JavaScript artifact. */ export const analyzeParsedJavaScriptSemantics = ( file: ParsedJavaScriptSource, - staticImportResolutionMode: "import" | "require" = "import", + staticImportResolutionMode: JavaScriptModuleResolutionMode = "import", ): JavaScriptSemanticIr => completeSemanticSteps( analyzeParsedJavaScriptSemanticsSteps(file, staticImportResolutionMode), @@ -151,7 +153,7 @@ export const analyzeParsedJavaScriptSemantics = ( */ export function* analyzeParsedJavaScriptSemanticsSteps( file: ParsedJavaScriptSource, - staticImportResolutionMode: "import" | "require" = "import", + staticImportResolutionMode: JavaScriptModuleResolutionMode = "import", ): Generator { const state = createState(file.program, staticImportResolutionMode); yield* collectDefinitionsSteps(file.program, state); @@ -200,6 +202,11 @@ export function* analyzeParsedJavaScriptSemanticsSteps( "The parser recovered from syntax errors; affected bindings are partial.", ] : []), + ...(hasUncertainTypeOnlyModuleSideEffect(file.program) + ? [ + "Compiler options are unavailable; inline type-only imports or re-exports may preserve a module side-effect dependency.", + ] + : []), "Values and aliases were recovered from inert syntax only; no JavaScript was executed.", "Return sites include only direct callable returns; nested callable returns remain separate.", "Local call, argument, return, and closure relations are static candidates and do not prove runtime invocation.", @@ -533,7 +540,7 @@ const bindingOrigin = ( const createState = ( program: t.Program, - staticImportResolutionMode: "import" | "require" = "import", + staticImportResolutionMode: JavaScriptModuleResolutionMode = "import", ): JavaScriptSemanticAnalysisState => { const root: JavaScriptSemanticScopeState = { scopeId: semanticScopeId("program", program), @@ -1210,8 +1217,11 @@ const addBinding = (input: AddBindingInput): void => { if (directOrigin !== undefined) binding.directOrigins.push( directOrigin.resolutionMode === undefined && - state.staticImportResolutionMode === "require" - ? { ...directOrigin, resolutionMode: "require" } + state.staticImportResolutionMode !== "import" + ? { + ...directOrigin, + resolutionMode: state.staticImportResolutionMode, + } : directOrigin, ); binding.runtimeImport ||= runtimeImport === true; diff --git a/src/domain/javascript/javascriptSemanticIr.ts b/src/domain/javascript/javascriptSemanticIr.ts index 18f1eb485..d8349e895 100644 --- a/src/domain/javascript/javascriptSemanticIr.ts +++ b/src/domain/javascript/javascriptSemanticIr.ts @@ -4,15 +4,19 @@ import type { JavaScriptSemanticCoverage, JavaScriptSemanticReturnCoverage, } from "./javascriptSemanticCoverage.js"; +import type { JavaScriptModuleResolutionMode } from "./javascriptModuleResolutionMode.js"; /** One exact module origin followed through imports, requires, or aliases. */ export interface JavaScriptModuleOrigin { - readonly resolutionMode?: "import" | "require"; + readonly resolutionMode?: JavaScriptModuleResolutionMode; readonly specifier: string; readonly importedPath: readonly string[]; } -/** Fail-closed provenance classification for one binding. */ +/** + * Fail-closed module-alias provenance for one binding, not transitive data flow. + * Locally constructed values may contain or derive from imported values. + */ export type JavaScriptBindingProvenance = | { readonly status: "module"; @@ -393,7 +397,7 @@ export interface JavaScriptSemanticFrontier { /** Static import/export relationship retained for cross-file composition. */ export interface JavaScriptSemanticModuleLink { - readonly resolutionMode?: "import" | "require"; + readonly resolutionMode?: JavaScriptModuleResolutionMode; readonly kind: | "import" | "type-import" diff --git a/src/domain/javascript/javascriptSemanticMemberMutations.ts b/src/domain/javascript/javascriptSemanticMemberMutations.ts index c2f7efba3..7b988af88 100644 --- a/src/domain/javascript/javascriptSemanticMemberMutations.ts +++ b/src/domain/javascript/javascriptSemanticMemberMutations.ts @@ -10,19 +10,21 @@ import { evaluateSemanticBinding, evaluateSemanticExpression, } from "./javascriptSemanticValues.js"; -import { - semanticContainer, - semanticSlotAtPath, -} from "./javascriptSemanticSlots.js"; +import { semanticContainer } from "./javascriptSemanticSlots.js"; import { semanticClassCandidates, semanticClassPropertyReferences, semanticObjectPropertyReferences, + type JavaScriptSemanticObjectReference, } from "./javascriptSemanticObjectReferences.js"; import { semanticIterationSources } from "./javascriptSemanticIterationSources.js"; import { semanticCallableResultExpressions } from "./javascriptSemanticCallableResults.js"; import { createSemanticReturnedReferences } from "./javascriptSemanticReturnedReferences.js"; import { semanticMutationInitializers } from "./javascriptSemanticMutationInitializers.js"; +import { + semanticReferenceSourcesInactive, + semanticSelectedReferenceSlot, +} from "./javascriptSemanticReferenceSources.js"; import { semanticArrayIndex, SemanticPropertyPathCoverage, @@ -98,8 +100,20 @@ export function* collectSemanticMemberMutationsSteps( effect: ValueEffect = "write", mutation?: t.Node, originAt: t.Node | undefined = mutation, + ownerInitializer?: t.Node, ): void => { - const pending = [{ node, path, bindings, originAt }]; + const pending: (ReferencedValue & { + readonly bindings: ReadonlySet; + readonly originAt: t.Node | undefined; + })[] = [ + { + node, + path, + bindings, + originAt, + ...(ownerInitializer === undefined ? {} : { ownerInitializer }), + }, + ]; // Visit shallower alias traversals first so their exact prefixes suppress // longer loop paths before those paths enumerate every assignment order. for (let cursor = 0; cursor < pending.length; cursor++) { @@ -132,6 +146,8 @@ export function* collectSemanticMemberMutationsSteps( binding.bindingId, current.originAt?.start, current.originAt?.end, + current.ownerInitializer?.start, + current.ownerInitializer?.end, ]; const lifetime = JSON.stringify([...capture, current.path]); if (current.bindings.has(lifetime)) continue; @@ -165,6 +181,7 @@ export function* collectSemanticMemberMutationsSteps( "escape", mutation, current.originAt, + current.ownerInitializer, ); continue; } @@ -195,6 +212,8 @@ export function* collectSemanticMemberMutationsSteps( mutation?.end, current.originAt?.start, current.originAt?.end, + current.ownerInitializer?.start, + current.ownerInitializer?.end, ]); const previous = selections.get(selectionIdentity); const excludedKeys = @@ -220,6 +239,8 @@ export function* collectSemanticMemberMutationsSteps( mutation?.end, current.originAt?.start, current.originAt?.end, + current.ownerInitializer?.start, + current.ownerInitializer?.end, ]); let covering = coveringPaths.get(pathState); if (covering === undefined) { @@ -235,6 +256,8 @@ export function* collectSemanticMemberMutationsSteps( mutation?.end, current.originAt?.start, current.originAt?.end, + current.ownerInitializer?.start, + current.ownerInitializer?.end, ]); if (recordedEffects.has(identity)) continue; // A recorded write invalidates the binding to a non-primitive value, so @@ -248,10 +271,23 @@ export function* collectSemanticMemberMutationsSteps( value?.status === "literal" || value?.status === "union"; recordedEffects.add(identity); const nested = new Set([...current.bindings, lifetime, captureKey]); + // Returning a literal's receiver reaches that allocation, not the + // binding's older values read while evaluating its initializer. const origins = - current.originAt === undefined - ? binding - : semanticMutationInitializers(binding, current.originAt, parents); + current.ownerInitializer !== undefined + ? { + initializers: binding.initializers.filter( + ({ node }) => node === current.ownerInitializer, + ), + referenceInitializers: [], + } + : current.originAt === undefined + ? binding + : semanticMutationInitializers( + binding, + current.originAt, + parents, + ); if (!primitiveWrite) { if (effect === "escape") binding.escapedPaths.push({ path, node: mutation }); @@ -294,7 +330,17 @@ export function* collectSemanticMemberMutationsSteps( for (const value of effect === "escape" ? escapedReferenceLeaves(expression, current.path) : expandedValues(expression, current.path, effect)) { - if (value.iterableFallbackPath !== undefined) + if (value.escapes && effect !== "escape") + markValue( + value.node, + value.path, + new Set(), + "escape", + mutation, + current.originAt, + value.ownerInitializer, + ); + else if (value.iterableFallbackPath !== undefined) deferIterable({ node: value.node, projection: [], @@ -367,6 +413,8 @@ export function* collectSemanticMemberMutationsSteps( binding === undefined ? null : binding.bindingId, binding === undefined ? node.name : null, pathKey, + value.ownerInitializer?.start, + value.ownerInitializer?.end, ]); if (leafIdentities.has(identity)) continue; leafIdentities.add(identity); @@ -467,7 +515,7 @@ export function* collectSemanticMemberMutationsSteps( for (const [index, reference] of pendingReferences.splice(0).entries()) { if (index % 64 === 0) yield; const { initializer } = reference; - if (referenceSourcesInactive(initializer, state)) + if (semanticReferenceSourcesInactive(initializer, state)) pendingReferences.push(reference); else if (t.isForOfStatement(initializer.node)) deferIterable({ @@ -512,7 +560,7 @@ export function* collectSemanticMemberMutationsSteps( } for (const [index, reference] of [...iterableReferences].entries()) { if (index % 64 === 0) yield; - if (referenceSourcesInactive(reference, state)) continue; + if (semanticReferenceSourcesInactive(reference, state)) continue; if (reference.iterationOnly) { const expanded = expandedIterationReferences(reference, state); if (expanded !== null) { @@ -520,7 +568,7 @@ export function* collectSemanticMemberMutationsSteps( continue; } } - const slot = selectedReferenceSlot(reference, state); + const slot = semanticSelectedReferenceSlot(reference, state); // Custom iteration can yield the iterable itself as well as any child. // Treat its root as escaped, including writes through yielded references. const knownArray = @@ -551,9 +599,7 @@ export function* collectSemanticMemberMutationsSteps( clearSemanticValueCaches(state); } -interface ReferencedValue { - readonly node: t.Node; - readonly path: PropertyPath; +interface ReferencedValue extends JavaScriptSemanticObjectReference { readonly iterableFallbackPath?: PropertyPath; } @@ -575,27 +621,6 @@ interface IterableReference extends ReferencedValue { }[]; } -const referenceSourcesInactive = ( - reference: Pick, - state: JavaScriptSemanticAnalysisState, -): boolean => - reference.requiredSources?.some( - (source) => - selectedReferenceSlot(source, state, source.node)?.presence === "absent", - ) === true || - reference.fallbackSources?.some((source) => { - // Selection precedes effects through the chosen reference. A later escape - // cannot make a fallback that was excluded at destructuring reachable. - const slot = selectedReferenceSlot(source, state, source.node); - return ( - slot?.presence === "present" && - (slot.value.status === "literal" || - slot.value.status === "union" || - slot.value.status === "object" || - slot.value.status === "array") - ); - }) === true; - const expandedIterationReferences = ( reference: IterableReference, state: JavaScriptSemanticAnalysisState, @@ -610,7 +635,10 @@ const expandedIterationReferences = ( ...(reference.fallbackSources ?? []), ].find((source) => source.node === iterable && source.projection[0] === null); if (source === undefined) return null; - const slot = selectedReferenceSlot({ node: iterable, projection: [] }, state); + const slot = semanticSelectedReferenceSlot( + { node: iterable, projection: [] }, + state, + ); const container = slot === undefined ? null : semanticContainer(slot.value); if (container?.coverage.status !== "complete") return null; const select = ( @@ -1106,19 +1134,3 @@ const copiedReferencePath = ( } return [...initializer.projection.slice(0, offset), key, ...remaining]; }; - -const selectedReferenceSlot = ( - source: { readonly node: t.Node; readonly projection: PropertyPath }, - state: JavaScriptSemanticAnalysisState, - capturePoint?: t.Node, -) => { - const path: string[] = []; - for (const key of source.projection) { - if (key === null || typeof key === "object") return undefined; - path.push(String(key)); - } - return semanticSlotAtPath( - evaluateSemanticExpression(source.node, state, capturePoint), - path, - ); -}; diff --git a/src/domain/javascript/javascriptSemanticMutationInitializers.ts b/src/domain/javascript/javascriptSemanticMutationInitializers.ts index cfaaf0ef5..6de7217c1 100644 --- a/src/domain/javascript/javascriptSemanticMutationInitializers.ts +++ b/src/domain/javascript/javascriptSemanticMutationInitializers.ts @@ -149,7 +149,50 @@ export const semanticMutationInitializers = ( const mutationPosition = directMutationPosition(mutation, parents) ?? referenceInitializerPosition(mutation, parents); - if (mutationPosition === null) return binding; + return selectInitializers(binding, mutationPosition, parents); +}; + +/** Select a local alias at its read, before any later straight-line reassignment. */ +export const semanticReadInitializers = ( + binding: JavaScriptSemanticBindingState, + read: t.Node, + parents: WeakMap, +): Pick< + JavaScriptSemanticBindingState, + "initializers" | "referenceInitializers" +> => { + let position: StatementPosition | null = null; + let current = read; + for ( + let parent = parents.get(current); + parent; + parent = parents.get(current) + ) { + if (t.isLoop(parent)) return binding; + if (t.isFunction(parent)) break; + if (position === null) { + if (t.isVariableDeclarator(parent)) { + const declaration = parents.get(parent); + if (t.isVariableDeclaration(declaration)) + position = statementPosition(declaration, parents, parent); + } else if (t.isStatement(parent)) + position = statementPosition(parent, parents); + } + current = parent; + } + return selectInitializers(binding, position, parents, true); +}; + +const selectInitializers = ( + binding: JavaScriptSemanticBindingState, + position: StatementPosition | null, + parents: WeakMap, + atRead = false, +): Pick< + JavaScriptSemanticBindingState, + "initializers" | "referenceInitializers" +> => { + if (position === null) return binding; const origins = [ ...binding.initializers.map((initializer) => ({ initializer, @@ -169,12 +212,20 @@ export const semanticMutationInitializers = ( ]; if ( origins.some( - ({ position }) => - position === null || !positionPrecedes(position, mutationPosition), + ({ position: origin }) => + origin === null || + origin.body !== position.body || + (!atRead && !positionPrecedes(origin, position)), ) ) return binding; - const latest = origins.reduce( + const preceding = atRead + ? origins.filter( + ({ position: origin }) => + origin !== null && positionPrecedes(origin, position), + ) + : origins; + const latest = preceding.reduce( (latest, { position }) => position !== null && (latest === null || positionPrecedes(latest, position)) @@ -183,7 +234,7 @@ export const semanticMutationInitializers = ( null, ); const retained = new Set( - origins + preceding .filter( ({ position }) => position?.index === latest?.index && diff --git a/src/domain/javascript/javascriptSemanticObjectReferences.ts b/src/domain/javascript/javascriptSemanticObjectReferences.ts index 387ba09d2..425e5fea5 100644 --- a/src/domain/javascript/javascriptSemanticObjectReferences.ts +++ b/src/domain/javascript/javascriptSemanticObjectReferences.ts @@ -4,6 +4,8 @@ import { type JavaScriptSemanticAnalysisState, } from "./javascriptSemanticState.js"; import { semanticCallableResultExpressions } from "./javascriptSemanticCallableResults.js"; +import { semanticReadInitializers } from "./javascriptSemanticMutationInitializers.js"; +import { semanticReferenceSourcesInactive } from "./javascriptSemanticReferenceSources.js"; import { semanticStaticPropertyKey, unwrapJavaScriptExpression, @@ -14,9 +16,12 @@ import { } from "./javascriptSemanticPropertyPaths.js"; import { compareUnicodeCodePoints } from "../unicodeCodePointOrder.js"; -interface ReferencedValue { +export interface JavaScriptSemanticObjectReference { readonly node: t.Node; readonly path: PropertyPath; + readonly escapes?: true; + /** The allocation owned by this binding, rather than a read of its prior value. */ + readonly ownerInitializer?: t.Node; } export const semanticClassCandidates = ( @@ -71,11 +76,12 @@ const instanceFieldReferences = ( propertyName: string | null, path: PropertyPath, state: JavaScriptSemanticAnalysisState, + initializedBefore?: t.ClassProperty, ): { - readonly references: readonly ReferencedValue[]; + readonly references: readonly JavaScriptSemanticObjectReference[]; readonly definiteNames: ReadonlySet; } => { - const references: ReferencedValue[] = []; + const references: JavaScriptSemanticObjectReference[] = []; const pending = [{ klass: root, excluded: new Set() }]; const selections = new Map>(); let definiteNames: ReadonlySet | undefined; @@ -92,7 +98,15 @@ const instanceFieldReferences = ( if (previous !== undefined && previous.size === excluded.size) continue; selections.set(current.klass, excluded); const shadowed = new Set(excluded); - for (const member of [...current.klass.body.body].reverse()) { + const before = + initializedBefore === undefined + ? -1 + : current.klass.body.body.indexOf(initializedBefore); + const members = + before < 0 + ? current.klass.body.body + : current.klass.body.body.slice(0, before); + for (const member of [...members].reverse()) { if (!t.isClassProperty(member) || member.static) continue; const key = semanticStaticPropertyKey(member.key, member.computed); if (key !== null && shadowed.has(key)) continue; @@ -121,12 +135,15 @@ interface ClassPropertyLookup { readonly owner: t.ClassDeclaration | t.ClassExpression; readonly receiver?: t.ClassDeclaration | t.ClassExpression; readonly kind: "instance" | "static" | "prototype"; + readonly fieldsOnly?: true; + readonly initializedBefore?: t.ClassProperty; } const classPropertyDescriptors = ( klass: t.ClassDeclaration | t.ClassExpression, propertyName: string | null, kind: ClassPropertyLookup["kind"], + initializedBefore?: t.ClassProperty, ) => { const isStatic = kind === "static"; const selectedGetters: t.ClassMethod[] = []; @@ -136,7 +153,12 @@ const classPropertyDescriptors = ( readonly index: number; readonly key: string | null; }[] = []; + const before = + initializedBefore === undefined + ? -1 + : klass.body.body.indexOf(initializedBefore); klass.body.body.forEach((member, index) => { + if (before >= 0 && index >= before && t.isClassProperty(member)) return; if ( (t.isClassMethod(member) || (kind === "static" && t.isClassProperty(member))) && @@ -208,7 +230,7 @@ export const semanticClassPropertyReferences = ( selection: PropertyPath, state: JavaScriptSemanticAnalysisState, activeGetters: ReadonlyMap = new Map(), -): readonly ReferencedValue[] => { +): readonly JavaScriptSemanticObjectReference[] => { const root = lookup.owner; const receiver = lookup.receiver ?? root; const [key, ...path] = selection; @@ -216,12 +238,18 @@ export const semanticClassPropertyReferences = ( key == null || typeof key === "object" ? null : String(key); const pending: (t.ClassDeclaration | t.ClassExpression)[] = [root]; const seen = new Set(); - const possibleResults: ReferencedValue[] = []; + const possibleResults: JavaScriptSemanticObjectReference[] = []; // Instance fields are own properties, including fields initialized by base // constructors. They take precedence over every prototype descriptor. let instanceFields: ReadonlySet = new Set(); if (lookup.kind === "instance") { - const fields = instanceFieldReferences(root, propertyName, path, state); + const fields = instanceFieldReferences( + root, + propertyName, + path, + state, + lookup.initializedBefore, + ); possibleResults.push(...fields.references); instanceFields = fields.definiteNames; if (propertyName !== null && instanceFields.has(propertyName)) @@ -235,11 +263,12 @@ export const semanticClassPropertyReferences = ( klass, propertyName, lookup.kind, + lookup.initializedBefore, ); for (const field of descriptors.fields) if (field.value != null) possibleResults.push({ node: field.value, path }); - for (const getter of descriptors.getters) { + for (const getter of lookup.fieldsOnly ? [] : descriptors.getters) { const key = semanticStaticPropertyKey(getter.key, getter.computed); if (key !== null && instanceFields.has(key)) continue; possibleResults.push( @@ -267,11 +296,11 @@ const objectReceiverReference = ( object: t.ObjectExpression, path: PropertyPath, state: JavaScriptSemanticAnalysisState, -): readonly ReferencedValue[] => { +): readonly JavaScriptSemanticObjectReference[] => { // A literal's enclosing binding owns its slots. Returning the literal // itself would bypass invalidation of that binding's observed properties. let current: t.Node = object; - const projection: (string | number | null)[] = []; + const projection: PropertyPath[number][] = []; while (true) { const parent = state.parentsByNode.get(current); if ( @@ -279,7 +308,43 @@ const objectReceiverReference = ( parent.init === current && t.isIdentifier(parent.id) ) - return [{ node: parent.id, path: [...projection, ...path] }]; + return [ + { + node: parent.id, + path: [...projection, ...path], + ownerInitializer: current, + }, + ]; + if ( + t.isAssignmentExpression(parent, { operator: "=" }) && + parent.right === current && + t.isIdentifier(parent.left) + ) + return [ + { + node: parent.left, + path: [...projection, ...path], + ownerInitializer: current, + }, + ]; + if (t.isArrayExpression(parent)) { + let minimumIndex = 0; + let uncertainIndex = false; + for (const element of parent.elements) { + if (element === current) break; + if (t.isSpreadElement(element)) uncertainIndex = true; + else minimumIndex++; + } + // A spread contributes an unknown number of slots. Retain the known + // prefix without treating the element's syntax position as its index. + projection.unshift( + uncertainIndex + ? { excludedKeys: [], startIndex: minimumIndex } + : minimumIndex, + ); + current = parent; + continue; + } if (t.isObjectProperty(parent) && parent.value === current) { projection.unshift( semanticStaticPropertyKey(parent.key, parent.computed), @@ -307,12 +372,108 @@ interface GetterInvocation { | t.ClassExpression; } +/** Follow only data fields whose initializers retain a class receiver. */ +const privateReceiverProjection = ( + receiver: t.ClassDeclaration | t.ClassExpression, + owner: t.ClassDeclaration | t.ClassExpression, + path: PropertyPath, + isStatic: boolean, + state: JavaScriptSemanticAnalysisState, +): boolean => { + if (path.some((key) => key === null || typeof key === "object")) return false; + type Selection = { readonly key: string; readonly before?: t.ClassProperty }; + const pending: { + receiver: typeof receiver; + path: readonly Selection[]; + visits: ReadonlyMap; + }[] = [ + { + receiver, + path: path.map((key) => ({ key: String(key) })), + visits: new Map(), + }, + ]; + const seen = new Set(); + const declaringClass = (field: t.ClassProperty) => { + const body = state.parentsByNode.get(field); + return body === undefined ? undefined : state.parentsByNode.get(body); + }; + for (let cursor = 0; cursor < pending.length; cursor++) { + const current = pending[cursor]; + if (current === undefined) break; + const [selection, ...remaining] = current.path; + if (selection === undefined) { + if (!isStatic || current.receiver === owner) return true; + continue; + } + const origin = JSON.stringify([ + current.receiver.start, + selection.key, + selection.before?.start, + ]); + const previous = current.visits.get(origin); + // Repeated fields may consume a finite getter path. An initializer cycle + // that preserves or grows it cannot establish another receiver identity. + if (previous !== undefined && previous <= current.path.length) continue; + const visits = new Map(current.visits).set(origin, current.path.length); + const identity = JSON.stringify([ + current.receiver.start, + current.path.map(({ key, before }) => [key, before?.start]), + ]); + if (seen.has(identity)) continue; + seen.add(identity); + const lookupOwner = + selection.before === undefined + ? current.receiver + : declaringClass(selection.before); + if (!t.isClass(lookupOwner)) continue; + for (const reference of semanticClassPropertyReferences( + { + owner: lookupOwner, + receiver: current.receiver, + kind: isStatic ? "static" : "instance", + fieldsOnly: true, + ...(selection.before === undefined + ? {} + : { initializedBefore: selection.before }), + }, + [selection.key], + state, + )) { + const field = state.parentsByNode.get(reference.node); + if (!t.isClassProperty(field)) continue; + const fieldOwner = declaringClass(field); + if (!t.isClass(fieldOwner)) continue; + const projection: Selection[] = []; + let value = unwrapJavaScriptExpression(reference.node).node; + while ( + t.isMemberExpression(value) || + t.isOptionalMemberExpression(value) + ) { + const key = semanticStaticPropertyKey(value.property, value.computed); + if (key === null || t.isPrivateName(value.property)) break; + projection.unshift({ key, before: field }); + value = unwrapJavaScriptExpression(value.object).node; + } + if (!t.isThisExpression(value)) continue; + // Instance fields retain the actual instance. Static field `this` belongs + // to its declaring class, even when a subclass inherits the field. + pending.push({ + receiver: isStatic ? fieldOwner : current.receiver, + path: [...projection, ...remaining], + visits, + }); + } + } + return false; +}; + const getterResultReferences = ( invocation: GetterInvocation, path: PropertyPath, state: JavaScriptSemanticAnalysisState, activeGetters: ReadonlyMap = new Map(), -): readonly ReferencedValue[] => { +): readonly JavaScriptSemanticObjectReference[] => { const { getter, owner, receiver = owner } = invocation; const previousLength = activeGetters.get(getter); if (previousLength !== undefined && previousLength <= path.length) return []; @@ -320,40 +481,206 @@ const getterResultReferences = ( // path; recursive getters that preserve or grow that path cannot progress. const active = new Map(activeGetters).set(getter, path.length); const isStatic = t.isClassMethod(getter) && getter.static; - const resolve = ( - node: t.Node, - remaining: PropertyPath, - ): readonly ReferencedValue[] => { - const expression = unwrapJavaScriptExpression(node).node; - if (t.isThisExpression(expression)) { - if (t.isClass(receiver)) { - return semanticClassPropertyReferences( - { owner: receiver, kind: isStatic ? "static" : "instance" }, - remaining, - state, - active, - ); + interface AliasVisit { + readonly identity: string; + readonly pathLength: number; + readonly parent: AliasVisit | undefined; + } + const pending: { + readonly node: t.Node; + readonly path: PropertyPath; + readonly aliases: AliasVisit | undefined; + readonly escapes?: true; + readonly privateRead?: { + readonly node: t.MemberExpression | t.OptionalMemberExpression; + readonly name: string; + readonly path: PropertyPath; + }; + }[] = semanticCallableResultExpressions(getter).flatMap(({ node }) => + node === null ? [] : [{ node, path, aliases: undefined }], + ); + const references: JavaScriptSemanticObjectReference[] = []; + const expandedAliases = new Map>(); + const getterScope = state.scopeByNode.get(getter)?.scopeId; + for (let cursor = 0; cursor < pending.length; cursor++) { + const current = pending[cursor]; + if (current === undefined) break; + const { path: remaining } = current; + const expression = unwrapJavaScriptExpression(current.node).node; + const retain = (reference: JavaScriptSemanticObjectReference): void => { + references.push( + current.escapes ? { ...reference, escapes: true } : reference, + ); + }; + const retainPrivateRead = (): void => { + if (current.privateRead !== undefined) + retain({ + node: current.privateRead.node, + path: current.escapes ? [] : current.privateRead.path, + }); + }; + if (t.isIdentifier(expression)) { + const binding = resolveSemanticBindingState( + state, + expression, + expression.name, + ); + let scope = + binding === undefined + ? undefined + : state.scopesById.get(binding.scopeId); + while (scope !== undefined && scope.scopeId !== getterScope) { + if (scope.kind === "function" || scope.parentScopeId === null) { + scope = undefined; + break; + } + scope = state.scopesById.get(scope.parentScopeId); } - return objectReceiverReference(receiver, remaining, state); + if ( + binding !== undefined && + scope !== undefined && + binding.referenceInitializers.every( + (initializer) => initializer.copyKind === undefined, + ) + ) { + const identity = JSON.stringify([ + binding.bindingId, + expression.start, + expression.end, + ]); + const known = expandedAliases.get(identity); + const key = JSON.stringify([ + remaining, + current.privateRead?.name, + current.privateRead?.path, + current.escapes, + ]); + let cyclic = false; + let growing = false; + if (known !== undefined) + for ( + let visit = current.aliases; + visit !== undefined; + visit = visit.parent + ) + if (visit.identity === identity) { + cyclic = true; + growing = remaining.length > visit.pathLength; + break; + } + if (cyclic) { + // A repeated alias can select arbitrarily deep children. Revisit its + // roots as escapes rather than enumerating iterations or dropping them. + if (growing && !current.escapes) + pending.push({ + ...current, + path: [], + aliases: undefined, + escapes: true, + }); + continue; + } + if (known?.has(key)) continue; + const selections = known ?? new Set(); + selections.add(key); + expandedAliases.set(identity, selections); + const aliases = { + identity, + pathLength: remaining.length, + parent: current.aliases, + }; + const origins = semanticReadInitializers( + binding, + expression, + state.parentsByNode, + ); + for (const initializer of [ + ...origins.initializers, + ...origins.referenceInitializers.filter( + (initializer) => + !semanticReferenceSourcesInactive(initializer, state), + ), + ]) + pending.push({ + ...current, + node: initializer.node, + path: [...initializer.projection, ...remaining], + aliases, + }); + continue; + } + } + if (t.isThisExpression(expression)) { + // An outer binding or a nested ordinary function has a different receiver. + let ownerNode = state.parentsByNode.get(expression); + while ( + ownerNode !== undefined && + !t.isClass(ownerNode) && + !(t.isFunction(ownerNode) && !t.isArrowFunctionExpression(ownerNode)) + ) + ownerNode = state.parentsByNode.get(ownerNode); + if (ownerNode !== getter) continue; + if (current.privateRead !== undefined && t.isClass(owner)) { + if ( + !t.isClass(receiver) || + !privateReceiverProjection( + receiver, + owner, + remaining, + isStatic, + state, + ) + ) { + retainPrivateRead(); + continue; + } + for (const member of owner.body.body) + if ( + t.isClassPrivateProperty(member) && + member.static === isStatic && + member.key.id.name === current.privateRead.name && + member.value != null + ) + retain({ + node: member.value, + path: current.escapes ? [] : current.privateRead.path, + }); + continue; + } + for (const reference of t.isClass(receiver) + ? semanticClassPropertyReferences( + { owner: receiver, kind: isStatic ? "static" : "instance" }, + remaining, + state, + active, + ) + : objectReceiverReference(receiver, remaining, state)) + retain(reference); + continue; } if ( t.isMemberExpression(expression) || t.isOptionalMemberExpression(expression) ) { if ( - t.isPrivateName(expression.property) && - t.isThisExpression(expression.object) && - t.isClass(owner) + current.privateRead !== undefined && + t.isPrivateName(expression.property) ) { - const privateName = expression.property.id.name; - return owner.body.body.flatMap((member) => - t.isClassPrivateProperty(member) && - member.static === isStatic && - member.key.id.name === privateName && - member.value != null - ? [{ node: member.value, path: remaining }] - : [], - ); + retainPrivateRead(); + continue; + } + if (t.isPrivateName(expression.property) && t.isClass(owner)) { + pending.push({ + ...current, + node: expression.object, + path: [], + privateRead: { + node: expression, + name: expression.property.id.name, + path: remaining, + }, + }); + continue; } const key = semanticStaticPropertyKey( expression.property, @@ -364,50 +691,61 @@ const getterResultReferences = ( t.isClass(owner) && owner.superClass != null ) { - return semanticClassCandidates(owner.superClass, state).flatMap( - (base) => - semanticClassPropertyReferences( - { - owner: base, - receiver: t.isClass(receiver) ? receiver : owner, - kind: isStatic ? "static" : "prototype", - }, - [key, ...remaining], - state, - active, - ), - ); + for (const base of semanticClassCandidates(owner.superClass, state)) + for (const reference of semanticClassPropertyReferences( + { + owner: base, + receiver: t.isClass(receiver) ? receiver : owner, + kind: isStatic ? "static" : "prototype", + }, + [key, ...remaining], + state, + active, + )) + retain(reference); + continue; } - return resolve(expression.object, [key, ...remaining]); - } - if (t.isConditionalExpression(expression)) - return [ - ...resolve(expression.consequent, remaining), - ...resolve(expression.alternate, remaining), - ]; - if (t.isLogicalExpression(expression)) - return [ - ...resolve(expression.left, remaining), - ...resolve(expression.right, remaining), - ]; - if (t.isSequenceExpression(expression)) { + pending.push({ + ...current, + node: expression.object, + path: [key, ...remaining], + }); + } else if (t.isConditionalExpression(expression)) + pending.push( + { ...current, node: expression.consequent }, + { ...current, node: expression.alternate }, + ); + else if (t.isLogicalExpression(expression)) + pending.push( + { ...current, node: expression.left }, + { ...current, node: expression.right }, + ); + else if (t.isSequenceExpression(expression)) { const last = expression.expressions.at(-1); - return last === undefined ? [] : resolve(last, remaining); - } - return [{ node: expression, path: remaining }]; - }; - return semanticCallableResultExpressions(getter).flatMap(({ node }) => - node === null ? [] : resolve(node, path), - ); + if (last !== undefined) pending.push({ ...current, node: last }); + } else if (t.isAssignmentExpression(expression, { operator: "=" })) + pending.push({ ...current, node: expression.right }); + else if ( + t.isAssignmentExpression(expression) && + ["&&=", "||=", "??="].includes(expression.operator) + ) + pending.push( + { ...current, node: expression.left }, + { ...current, node: expression.right }, + ); + else if (current.privateRead !== undefined) retainPrivateRead(); + else retain({ node: expression, path: remaining }); + } + return references; }; export const semanticObjectPropertyReferences = ( node: t.ObjectExpression, path: PropertyPath, state: JavaScriptSemanticAnalysisState, -): readonly ReferencedValue[] => { +): readonly JavaScriptSemanticObjectReference[] => { const [key, ...remaining] = path; - const references: ReferencedValue[] = []; + const references: JavaScriptSemanticObjectReference[] = []; const overwritten = new Set(); const dataOverwritten = new Set(); const accessorNames = new Set(); diff --git a/src/domain/javascript/javascriptSemanticProjection.ts b/src/domain/javascript/javascriptSemanticProjection.ts index 7e4663efe..b477bb5af 100644 --- a/src/domain/javascript/javascriptSemanticProjection.ts +++ b/src/domain/javascript/javascriptSemanticProjection.ts @@ -28,6 +28,12 @@ import { propertyName, semanticStaticPropertyKey, } from "./javascriptAstValues.js"; +import { + isTypeOnlyExportDeclaration, + isTypeOnlyExportSpecifier, + isTypeOnlyImportDeclaration, + isTypeOnlyImportSpecifier, +} from "./javascriptModuleDeclaration.js"; interface CollectCallableInput { readonly node: t.Node; @@ -75,7 +81,7 @@ export const collectSemanticModuleLink = ( addModuleLink( state, { - kind: "re-export", + kind: isTypeOnlyExportDeclaration(node) ? "type-import" : "re-export", specifier: node.source.value, importedName: "*", localName: null, @@ -108,12 +114,19 @@ const collectImportEquals = ( node: t.TSImportEqualsDeclaration, state: JavaScriptSemanticAnalysisState, ): void => { - const specifier = semanticImportEqualsSpecifier(node); + const moduleReference = node.moduleReference; + const specifier = + semanticImportEqualsSpecifier(node) ?? + (node.importKind === "type" && + t.isTSExternalModuleReference(moduleReference) && + t.isStringLiteral(moduleReference.expression) + ? moduleReference.expression.value + : null); if (specifier === null) return; addModuleLink( state, { - kind: "require", + kind: node.importKind === "type" ? "type-import" : "require", specifier, importedName: null, localName: node.id.name, @@ -287,7 +300,7 @@ const collectImports = ( addModuleLink( state, { - kind: "import", + kind: isTypeOnlyImportDeclaration(node) ? "type-import" : "import", specifier: node.source.value, importedName: null, localName: null, @@ -300,7 +313,12 @@ const collectImports = ( addModuleLink( state, { - kind: "import", + kind: + isTypeOnlyImportDeclaration(node) || + (t.isImportSpecifier(specifier) && + isTypeOnlyImportSpecifier(specifier)) + ? "type-import" + : "import", specifier: node.source.value, importedName: t.isImportDefaultSpecifier(specifier) ? "default" @@ -319,12 +337,13 @@ const collectNamedExports = ( node: t.ExportNamedDeclaration, state: JavaScriptSemanticAnalysisState, ): void => { + const declarationTypeOnly = isTypeOnlyExportDeclaration(node); if (node.declaration !== null && node.declaration !== undefined) for (const localName of declarationNames(node.declaration)) addModuleLink( state, { - kind: "export", + kind: declarationTypeOnly ? "type-import" : "export", specifier: null, importedName: null, localName, @@ -342,7 +361,14 @@ const collectNamedExports = ( addModuleLink( state, { - kind: node.source == null ? "export" : "re-export", + kind: + declarationTypeOnly || + (t.isExportSpecifier(specifier) && + isTypeOnlyExportSpecifier(specifier)) + ? "type-import" + : node.source == null + ? "export" + : "re-export", specifier: node.source?.value ?? null, importedName: t.isExportNamespaceSpecifier(specifier) ? "*" @@ -507,7 +533,20 @@ const addModuleLink = ( }, referenceNode: t.Node, ): void => { - const collected = { ...link, callableId: link.callableId ?? null }; + const resolutionMode = + link.resolutionMode ?? + (link.kind === "import" || + link.kind === "re-export" || + link.kind === "type-import" + ? state.staticImportResolutionMode + : link.kind === "require" + ? "require" + : undefined); + const collected = { + ...link, + ...(resolutionMode === undefined ? {} : { resolutionMode }), + callableId: link.callableId ?? null, + }; state.moduleLinks.push(collected); if (link.localName !== null) { const binding = resolveSemanticBindingState( @@ -745,6 +784,13 @@ const declarationNames = (declaration: t.Declaration): string[] => { t.isIdentifier(declaration.id) ) return [declaration.id.name]; + if ( + (t.isTSInterfaceDeclaration(declaration) || + t.isTSTypeAliasDeclaration(declaration) || + t.isTSDeclareFunction(declaration)) && + t.isIdentifier(declaration.id) + ) + return [declaration.id.name]; if ( t.isTSImportEqualsDeclaration(declaration) && semanticImportEqualsSpecifier(declaration) !== null diff --git a/src/domain/javascript/javascriptSemanticProvenance.test.ts b/src/domain/javascript/javascriptSemanticProvenance.test.ts new file mode 100644 index 000000000..82a77a449 --- /dev/null +++ b/src/domain/javascript/javascriptSemanticProvenance.test.ts @@ -0,0 +1,94 @@ +import { describe, expect, it } from "vitest"; + +import { analyzeJavaScriptSemantics } from "./javascriptSemanticAnalysis.js"; +import { + onlyBinding, + topLevelBinding, +} from "./javascriptSemanticAnalysis.fixture.js"; + +describe("JavaScript semantic binding provenance", () => { + it("keeps opaque expression results unknown instead of declaring them local", () => { + const ir = analyzeJavaScriptSemantics(` + const called = loadFeature(); + const constructed = new Feature(); + const tagged = tag\`feature\`; + const dynamicRequired = require(specifier); + const sequence = (0, loadFeature()); + const assignment = (slot = loadFeature()); + async function readLater(parameter = require("fallback")) { + const awaited = await loadFeature(); + return awaited; + } + try { loadFeature(); } catch (caught) { void caught; } + function* yieldLater() { + const yielded = yield loadFeature(); + return yielded; + } + `); + + for (const name of [ + "called", + "constructed", + "tagged", + "dynamicRequired", + "sequence", + "assignment", + "awaited", + "yielded", + "parameter", + "caught", + ]) { + expect(onlyBinding(ir, name).provenance).toMatchObject({ + status: "unknown", + origins: [], + }); + } + }); + + it("recognizes explicit local values and preserves exact module aliases", () => { + const ir = analyzeJavaScriptSemantics(` + const object = { value: 1 }; + const array = [1, 2]; + const template = \`feature\`; + const primitive = -1 + 2; + const derived = loadFeature() + 1; + const derivedText = \`value: \${loadFeature()}\`; + let uninitialized; + const callable = () => 1; + const Constructed = class {}; + const imported = require("feature"); + const alias = imported; + const sequence = (0, require("sequence-feature")); + const assignment = (slot = require("assignment-feature")); + `); + + for (const name of [ + "object", + "array", + "template", + "primitive", + "derived", + "derivedText", + "uninitialized", + "callable", + "Constructed", + ]) + expect(topLevelBinding(ir, name).provenance).toMatchObject({ + status: "local", + origins: [], + }); + + expect(topLevelBinding(ir, "alias").provenance).toMatchObject({ + status: "module", + origins: [{ specifier: "feature" }], + }); + expect(topLevelBinding(ir, "sequence").provenance).toMatchObject({ + status: "module", + origins: [{ specifier: "sequence-feature" }], + }); + expect(topLevelBinding(ir, "assignment").provenance).toMatchObject({ + status: "module", + origins: [{ specifier: "assignment-feature" }], + }); + }); +}); diff --git a/src/domain/javascript/javascriptSemanticReferenceSources.ts b/src/domain/javascript/javascriptSemanticReferenceSources.ts new file mode 100644 index 000000000..54e1d3ba9 --- /dev/null +++ b/src/domain/javascript/javascriptSemanticReferenceSources.ts @@ -0,0 +1,53 @@ +import type * as t from "@babel/types"; +import type { JavaScriptSemanticAnalysisState } from "./javascriptSemanticState.js"; +import type { JavaScriptSemanticPropertyPath as PropertyPath } from "./javascriptSemanticPropertyPaths.js"; +import { evaluateSemanticExpression } from "./javascriptSemanticValues.js"; +import { semanticSlotAtPath } from "./javascriptSemanticSlots.js"; + +interface ReferenceSource { + readonly node: t.Node; + readonly projection: PropertyPath; +} + +/** Exclude references whose source slot or destructuring fallback is absent. */ +export const semanticReferenceSourcesInactive = ( + reference: { + readonly requiredSources?: readonly ReferenceSource[]; + readonly fallbackSources?: readonly ReferenceSource[]; + }, + state: JavaScriptSemanticAnalysisState, +): boolean => + reference.requiredSources?.some( + (source) => + semanticSelectedReferenceSlot(source, state, source.node)?.presence === + "absent", + ) === true || + reference.fallbackSources?.some((source) => { + // Selection precedes effects through the chosen reference. A later escape + // cannot make a fallback that was excluded at destructuring reachable. + const slot = semanticSelectedReferenceSlot(source, state, source.node); + return ( + slot?.presence === "present" && + (slot.value.status === "literal" || + slot.value.status === "union" || + slot.value.status === "object" || + slot.value.status === "array") + ); + }) === true; + +/** Select a retained source slot at its capture, before later effects. */ +export const semanticSelectedReferenceSlot = ( + source: ReferenceSource, + state: JavaScriptSemanticAnalysisState, + capturePoint?: t.Node, +) => { + const path: string[] = []; + for (const key of source.projection) { + if (key === null || typeof key === "object") return undefined; + path.push(String(key)); + } + return semanticSlotAtPath( + evaluateSemanticExpression(source.node, state, capturePoint), + path, + ); +}; diff --git a/src/domain/javascript/javascriptSemanticReturns.ts b/src/domain/javascript/javascriptSemanticReturns.ts index 189a1385e..27bb030d0 100644 --- a/src/domain/javascript/javascriptSemanticReturns.ts +++ b/src/domain/javascript/javascriptSemanticReturns.ts @@ -101,6 +101,7 @@ export const resolveSemanticModuleCallables = ( bindings.map(({ bindingId, provenance }) => [bindingId, provenance]), ); return state.moduleLinks.map((link) => { + if (link.kind === "type-import") return { ...link, callableId: null }; // Named exports follow their lexical binding, including every assignment. // A declaration's ID or a callable's display name cannot prove that link. if (link.localName === null) @@ -122,9 +123,7 @@ export const resolveSemanticModuleCallables = ( : undefined; return { ...link, - ...(resolutionMode === "import" || resolutionMode === "require" - ? { resolutionMode } - : {}), + ...(resolutionMode !== undefined ? { resolutionMode } : {}), callableId: resolved.length === 1 ? (resolved[0] ?? null) : null, }; }); diff --git a/src/domain/javascript/javascriptSemanticState.ts b/src/domain/javascript/javascriptSemanticState.ts index b82abc62d..d01a4be52 100644 --- a/src/domain/javascript/javascriptSemanticState.ts +++ b/src/domain/javascript/javascriptSemanticState.ts @@ -12,6 +12,7 @@ import type { JavaScriptSemanticCallable, JavaScriptSemanticScope, } from "./javascriptSemanticIr.js"; +import type { JavaScriptModuleResolutionMode } from "./javascriptModuleResolutionMode.js"; /** Internal initializer plus a destructuring/member projection. */ interface JavaScriptSemanticInitializer { @@ -63,7 +64,7 @@ export interface JavaScriptSemanticScopeState { /** Shared state for semantic collection and evaluation. */ export interface JavaScriptSemanticAnalysisState { /** Effective loader for static imports in this source artifact. */ - readonly staticImportResolutionMode: "import" | "require"; + readonly staticImportResolutionMode: JavaScriptModuleResolutionMode; readonly parentsByNode: WeakMap; readonly scopes: JavaScriptSemanticScopeState[]; readonly scopesById: Map; diff --git a/src/domain/javascript/javascriptSemanticValues.ts b/src/domain/javascript/javascriptSemanticValues.ts index cf31e0575..d15c5edfc 100644 --- a/src/domain/javascript/javascriptSemanticValues.ts +++ b/src/domain/javascript/javascriptSemanticValues.ts @@ -817,6 +817,13 @@ const provenanceForBindingValue = ( binding: JavaScriptSemanticBindingState, context: EvaluationContext, ): JavaScriptBindingProvenance => { + if (binding.kind === "parameter" || binding.kind === "catch") + return semanticUnresolvedProvenance( + "unknown", + binding.kind === "parameter" + ? "Parameter provenance depends on caller arguments." + : "Catch binding provenance depends on the thrown value.", + ); if (binding.directOrigins.length > 0) return semanticOriginsProvenance(binding.directOrigins); if (binding.initializers.length === 0) return semanticLocalProvenance(); @@ -890,6 +897,14 @@ const provenanceForExpression = ( const required = semanticRequireOrigin(node, context.state); if (required !== undefined) return semanticOriginsProvenance([required.origin]); + if (t.isSequenceExpression(node)) { + const finalExpression = node.expressions.at(-1); + return finalExpression === undefined + ? semanticUnresolvedProvenance("unknown", "Empty sequence expression.") + : provenanceForExpression(finalExpression, nestedContext(context)); + } + if (t.isAssignmentExpression(node) && node.operator === "=") + return provenanceForExpression(node.right, nestedContext(context)); if (t.isIdentifier(node)) { const binding = resolveSemanticBindingState(context.state, node, node.name); return binding === undefined @@ -961,9 +976,25 @@ const provenanceForExpression = ( "Conditional provenance is unresolved.", ); } - return semanticLocalProvenance(); + if (isLocallyCreatedValue(node)) return semanticLocalProvenance(); + return semanticUnresolvedProvenance( + "unknown", + `Unsupported ${node.type} provenance.`, + ); }; +const isLocallyCreatedValue = (node: t.Node): boolean => + readExactJavaScriptLiteral(node).found || + t.isObjectExpression(node) || + t.isArrayExpression(node) || + t.isFunctionExpression(node) || + t.isArrowFunctionExpression(node) || + t.isClassExpression(node) || + t.isTemplateLiteral(node) || + t.isUnaryExpression(node) || + t.isBinaryExpression(node) || + t.isUpdateExpression(node); + const mergeValues = ( values: readonly JavaScriptSemanticValue[], ): JavaScriptSemanticValue => { diff --git a/src/domain/javascript/javascriptSourceMapPaths.ts b/src/domain/javascript/javascriptSourceMapPaths.ts index d897bdd26..06b5f8265 100644 --- a/src/domain/javascript/javascriptSourceMapPaths.ts +++ b/src/domain/javascript/javascriptSourceMapPaths.ts @@ -1,5 +1,6 @@ import { posix } from "node:path"; import { domainToASCII, fileURLToPath, pathToFileURL } from "node:url"; +import { TraceMap } from "@jridgewell/trace-mapping"; import { z } from "zod"; @@ -25,22 +26,64 @@ export type JavaScriptSourceMapReference = z.infer< typeof javascriptSourceMapReferenceSchema >; -/** Interpret a declaration once at the source-map ingestion boundary. */ -export const resolveJavaScriptSourceMapReference = ( - sourceName: string, +/** Interpret declarations together once at their source-map ingestion boundary. */ +export const resolveJavaScriptSourceMapReferences = ( + sourceNames: readonly (string | null)[], sourceRoot: string | null, mapPath: string, -): JavaScriptSourceMapReference => { - // TraceMap's producer prefixes sourceRoot before resolving the URL reference. - const reference = `${sourceRoot ? `${sourceRoot}/` : ""}${sourceName}`; - return { - source_name: sourceName, - source_root: sourceRoot, - map_path: mapPath, - resolution: resolveJavaScriptSourceMapPath(reference, mapPath), - }; +): (JavaScriptSourceMapReference | null)[] => { + if (sourceNames.length === 0) return []; + const resolvedSources = resolveSourceMapUrls( + sourceNames.map((sourceName) => sourceName ?? ""), + sourceRoot, + "", + ); + return sourceNames.map((sourceName, index) => { + if (sourceName === null) return null; + const reference = resolvedSources[index]; + if (reference === undefined) + throw new TypeError( + `TraceMap did not resolve source declaration ${String(index)}`, + ); + return { + source_name: sourceName, + source_root: sourceRoot, + map_path: mapPath, + resolution: resolveJavaScriptSourceMapPath(reference, mapPath), + }; + }); }; +/** Resolve a source declaration using the same pinned format implementation as map consumers. */ +export const resolveJavaScriptSourceMapUrl = ( + source: string | null, + sourceRoot: string | null | undefined, + mapUrl: string, +): string => { + const resolved = resolveSourceMapUrls([source ?? ""], sourceRoot, mapUrl)[0]; + if (resolved === undefined) + throw new TypeError( + "TraceMap did not resolve its single source declaration", + ); + return resolved; +}; + +const resolveSourceMapUrls = ( + sources: string[], + sourceRoot: string | null | undefined, + mapUrl: string, +): readonly string[] => + new TraceMap( + { + version: 3, + names: [], + sources, + ...(sourceRoot == null ? {} : { sourceRoot }), + mappings: "", + }, + mapUrl, + ).resolvedSources; + /** Resolve a source-map URL reference without changing its recorded identity. */ const resolveJavaScriptSourceMapPath = ( source: string, diff --git a/src/domain/javascript/javascriptSourceParser.ts b/src/domain/javascript/javascriptSourceParser.ts index ce5619ead..630d5e9bf 100644 --- a/src/domain/javascript/javascriptSourceParser.ts +++ b/src/domain/javascript/javascriptSourceParser.ts @@ -37,10 +37,19 @@ export const parserPluginsForPath = (sourcePath?: string): ParserPlugin[] => { : ["decorators-legacy", typescript]; }; -/** `.mjs` and `.mts` are modules by extension; other paths leave the mode to syntax. */ +/** `.mjs` and `.mts` explicitly select module mode. */ export const isExplicitModuleSourcePath = (path?: string): boolean => /\.m[jt]s$/iu.test(path ?? ""); +const runtimeSourceTypeForPath = ( + path?: string, +): "commonjs" | "module" | "unambiguous" => + /\.cjs$/iu.test(path ?? "") + ? "commonjs" + : isExplicitModuleSourcePath(path) + ? "module" + : "unambiguous"; + export interface RecoveredJavaScriptParse { readonly file: ParsedJavaScriptSource; /** Errors that apply only if a source of unknown mode is an ES module. */ @@ -58,25 +67,25 @@ export const parseRecoveredJavaScript = ( sourcePath: string | undefined, options: Omit, ): RecoveredJavaScriptParse => { - // `.cjs` is CommonJS by extension, so module-only errors cannot apply. - if (/\.cjs$/iu.test(sourcePath ?? "")) + const sourceType = runtimeSourceTypeForPath(sourcePath); + if (sourceType !== "unambiguous") { + const normalizedOptions = { ...options }; + if (sourceType === "commonjs") { + // Babel owns these wrapper semantics in CommonJS mode and rejects + // explicit script overrides, including the reference-import options. + delete normalizedOptions.allowAwaitOutsideFunction; + delete normalizedOptions.allowReturnOutsideFunction; + delete normalizedOptions.allowNewTargetOutsideFunction; + } return { file: parse(source, { - ...options, - sourceType: "script", - errorRecovery: true, - }), - moduleOnlyErrors: [], - }; - if (isExplicitModuleSourcePath(sourcePath)) - return { - file: parse(source, { - ...options, - sourceType: "module", + ...normalizedOptions, + sourceType, errorRecovery: true, }), moduleOnlyErrors: [], }; + } const file = parse(source, { ...options, sourceType: "unambiguous", @@ -98,18 +107,34 @@ export const parseRecoveredJavaScript = ( }; }; -/** Parse JavaScript or TypeScript once without attaching comments to AST nodes. */ +/** Parse inert JavaScript or TypeScript facts without attaching comments to AST nodes. */ export const parseJavaScriptSource = ( source: string, sourcePath?: string, ): ParsedJavaScriptSource | null => { try { - return parse(source, { - sourceType: "unambiguous", + const runtimeSourceType = runtimeSourceTypeForPath(sourcePath); + const commonjs = runtimeSourceType === "commonjs"; + const options = { errorRecovery: true, attachComment: false, plugins: parserPluginsForPath(sourcePath), + }; + const file = parse(source, { + ...options, + // Inert artifact analysis retains module declarations even in .cjs assets. + // Infer their syntax here; reference imports keep filename-mode diagnostics. + sourceType: commonjs ? "unambiguous" : runtimeSourceType, + allowReturnOutsideFunction: commonjs, + allowNewTargetOutsideFunction: commonjs, }); + // Unambiguous mode retains module-only diagnostics after choosing a script. + // The .cjs extension establishes wrapper mode, so obtain its own diagnostics. + return commonjs && + file.program.sourceType === "script" && + file.errors.length > 0 + ? parse(source, { ...options, sourceType: "commonjs" }) + : file; } catch (cause: unknown) { // Unparseable source is represented by the null return. void cause; diff --git a/src/domain/javascript/javascriptStaticAnalysis.ts b/src/domain/javascript/javascriptStaticAnalysis.ts index 4d10638bc..c34462498 100644 --- a/src/domain/javascript/javascriptStaticAnalysis.ts +++ b/src/domain/javascript/javascriptStaticAnalysis.ts @@ -53,6 +53,11 @@ import { import { createJavaScriptRpcScan } from "./javascriptRpcEndpoints.js"; import { semanticImportEqualsSpecifier } from "./javascriptSemanticProjection.js"; +import { + hasUncertainTypeOnlyModuleSideEffect, + isTypeOnlyExportDeclaration, + isTypeOnlyImportDeclaration, +} from "./javascriptModuleDeclaration.js"; /** Parse one JavaScript artifact and recover static structure only. */ export const analyzeJavaScriptStaticSource = ( @@ -154,6 +159,11 @@ const finalizeStaticAnalysis = ( : [ "One or more static keys, expressions, or Electron boundary values were dynamic and remain unknown.", ]), + ...(hasUncertainTypeOnlyModuleSideEffect(file.program) + ? [ + "Compiler options are unavailable; inline type-only imports or re-exports may preserve a module side-effect dependency.", + ] + : []), ...(accumulator.references.some( ({ value }) => value.specifier !== null && ELECTRON_MODULE.test(value.specifier), @@ -246,14 +256,20 @@ const inspectNode = ( addReference(findings, { node, kind: "require", specifier }); } else if ( (t.isImportDeclaration(node) || t.isExportAllDeclaration(node)) && - node.source !== undefined + node.source !== undefined && + !(t.isImportDeclaration(node) && isTypeOnlyImportDeclaration(node)) && + !(t.isExportAllDeclaration(node) && isTypeOnlyExportDeclaration(node)) ) addReference(findings, { node, kind: "static-import", specifier: node.source.value, }); - else if (t.isExportNamedDeclaration(node) && node.source != null) + else if ( + t.isExportNamedDeclaration(node) && + node.source != null && + !isTypeOnlyExportDeclaration(node) + ) addReference(findings, { node, kind: "static-import", diff --git a/src/domain/javascript/javascriptStaticAnalysisHelpers.ts b/src/domain/javascript/javascriptStaticAnalysisHelpers.ts index a6a6ebc44..f2045a1eb 100644 --- a/src/domain/javascript/javascriptStaticAnalysisHelpers.ts +++ b/src/domain/javascript/javascriptStaticAnalysisHelpers.ts @@ -181,18 +181,37 @@ export const staticPath = (node: t.Node): string | undefined => { if (t.isStringLiteral(node)) return node.value; if (t.isTemplateLiteral(node) && node.expressions.length === 0) return stringValue(node); - if (t.isBinaryExpression(node, { operator: "+" })) { - const left = staticPath(node.left); - const right = staticPath(node.right); - return left === undefined || right === undefined - ? undefined - : `${left}${right}`; - } + if (t.isBinaryExpression(node, { operator: "+" })) + return staticConcatenation(node); if (t.isCallExpression(node) || t.isNewExpression(node)) return staticCallPath(node); return undefined; }; +/** + * Concatenate inert text. A leading source directory contributes no text: + * `__dirname + "/preload.js"` names a file inside it, while text without a + * leading separator extends the directory's own name and stays unknown. + */ +const staticConcatenation = (node: t.BinaryExpression): string | undefined => { + const operands: t.Node[] = []; + let current: t.Node = node; + while (t.isBinaryExpression(current, { operator: "+" })) { + operands.push(current.right); + current = current.left; + } + const anchored = isDirectoryPath(current); + if (!anchored) operands.push(current); + let text = ""; + for (const operand of operands.reverse()) { + const value = staticPath(operand); + if (value === undefined) return undefined; + text += value; + } + if (!anchored) return text; + return text.startsWith("/") ? posix.join(".", text) : undefined; +}; + const staticCallPath = ( node: t.CallExpression | t.NewExpression, ): string | undefined => { @@ -204,11 +223,6 @@ const staticCallPath = ( if (name === "fileURLToPath" || name.endsWith(".fileURLToPath")) { return staticFileUrlPath(argumentNode(node.arguments[0])); } - if ( - (name === "dirname" || name.endsWith(".dirname")) && - isFileIdentity(argumentNode(node.arguments[0])) - ) - return ""; if (name.endsWith(".resolve")) return name === "win32.resolve" || name.endsWith(".win32.resolve") ? undefined @@ -216,7 +230,12 @@ const staticCallPath = ( if (!name.endsWith(".join") && name !== "join") return undefined; const parts: string[] = []; for (const argument of node.arguments) { - if (isDirectoryIdentity(argumentNode(argument))) continue; + if (isDirectoryIdentity(argumentNode(argument))) { + // join never resets at an absolute segment, so `/preload.js` after the + // source directory stays inside it. + parts.push("."); + continue; + } const value = t.isNode(argument) ? staticPath(argument) : undefined; if (value === undefined) return undefined; parts.push(value); @@ -331,11 +350,20 @@ const isFilesystemPathExpression = (node: t.Node): boolean => { }; const isDirectoryIdentity = (node: t.Node | undefined): boolean => { - if (t.isIdentifier(node, { name: "__dirname" })) return true; + if (isDirectoryPath(node)) return true; if (!t.isCallExpression(node) && !t.isNewExpression(node)) return false; const name = calleeName(node.callee); - if (name === "URL" || name.endsWith(".URL")) - return isFileIdentity(argumentNode(node.arguments[1])); + return ( + (name === "URL" || name.endsWith(".URL")) && + isFileIdentity(argumentNode(node.arguments[1])) + ); +}; + +/** `__dirname` or the dirname of the source file, as a string. */ +const isDirectoryPath = (node: t.Node | undefined): boolean => { + if (t.isIdentifier(node, { name: "__dirname" })) return true; + if (!t.isCallExpression(node)) return false; + const name = calleeName(node.callee); return ( (name === "dirname" || name.endsWith(".dirname")) && isFileIdentity(argumentNode(node.arguments[0])) @@ -691,6 +719,14 @@ export const registrationKey = ( registration.runtime, registration.chunk_keys, String(registration.unknown_chunk_keys), + // Equal factory tables can be registered with different runtime callbacks. + // Both early collection and finalization must retain those recovered facts. + registration.runtime_require_name, + registration.runtime_module_cache_status, + registration.entry_module_keys, + String(registration.unknown_entry_module_keys), + registration.async_chunk_keys, + String(registration.unknown_async_chunk_keys), registration.modules.map(({ module_key: key, source_sha256: digest }) => [ key, digest, diff --git a/src/domain/javascript/javascriptStaticPathResolution.test.ts b/src/domain/javascript/javascriptStaticPathResolution.test.ts index e54122440..30bbc317f 100644 --- a/src/domain/javascript/javascriptStaticPathResolution.test.ts +++ b/src/domain/javascript/javascriptStaticPathResolution.test.ts @@ -38,6 +38,9 @@ const anchoredExpressions = [ ["path.resolve(__dirname)", "."], ['path.posix.resolve(`/app`, `/shared`, "preload.js")', "/shared/preload.js"], ['path.posix.resolve("" + "/app", "preload.js")', "/app/preload.js"], + ['__dirname + "/preload.js"', "preload.js"], + ['__dirname + "/" + "preload.js"', "preload.js"], + ['path.dirname(__filename) + "/lib" + "/preload.js"', "lib/preload.js"], ] as const; const unknownExpressions = [ @@ -58,6 +61,8 @@ const unknownExpressions = [ 'path.win32.resolve(__dirname, "preload.js")', 'win32.resolve("/app", "preload.js")', 'resolve("/app", "preload.js")', + '__dirname + "preload.js"', + 'path.dirname(__filename) + "preload.js"', ]; describe("inert JavaScript path resolution", () => { @@ -100,6 +105,16 @@ describe("inert JavaScript path resolution", () => { "preload.js", "filesystem-expression", ], + [ + 'path.join(__dirname, "/preload.js")', + "preload.js", + "filesystem-expression", + ], + [ + 'path.join(dirname(fileURLToPath(import.meta.url)), "/lib", "/preload.js")', + "lib/preload.js", + "filesystem-expression", + ], [ 'path.join("ignored", __dirname, "preload.js")', "ignored/preload.js", @@ -111,6 +126,7 @@ describe("inert JavaScript path resolution", () => { "filesystem-expression", ], ['join("sub", "preload.js")', "sub/preload.js", "module-specifier"], + ["path.join(__dirname)", ".", "filesystem-expression"], ] as const)("preserves join behavior in %s", (expression, path, context) => { assertPathProjection(expression, path, context); }); @@ -182,7 +198,6 @@ describe("inert JavaScript path resolution", () => { it.each([ "path.join()", - "path.join(__dirname)", 'path.join("/app", runtimeFile)', "path.join(...parts)", ])("preserves unresolved join syntax in %s", (expression) => diff --git a/src/domain/javascript/sourceToBundleComparison.test.ts b/src/domain/javascript/sourceToBundleComparison.test.ts index f488ee598..d231cb5ec 100644 --- a/src/domain/javascript/sourceToBundleComparison.test.ts +++ b/src/domain/javascript/sourceToBundleComparison.test.ts @@ -1,10 +1,11 @@ import { describe, expect, it } from "vitest"; +import { TraceMap } from "@jridgewell/trace-mapping"; -import { resolveJavaScriptSourceMapReference } from "./javascriptSourceMapPaths.js"; import { createJavaScriptApplicationGraph, createJavaScriptApplicationNode, } from "./javascriptApplicationGraph.js"; +import type { ApplicationNode } from "./javascriptApplicationGraphSchemas.js"; import { createHistoricalSourceGraph } from "../referenceSourceGraph.js"; import { compareUnicodeCodePoints } from "../unicodeCodePointOrder.js"; import { compareSourceToBundle } from "./sourceToBundleComparison.js"; @@ -12,7 +13,10 @@ import { SOURCE_TO_BUNDLE_SIGNAL_WEIGHTS, sourceToBundleComparisonResultSchema, } from "./sourceToBundleComparisonSchemas.js"; -import { artifactEvidence } from "./javascriptApplicationGraph.fixture.js"; +import { + artifactEvidence, + sourceMapReference, +} from "./javascriptApplicationGraph.fixture.js"; const HASH = { artifact: "1".repeat(64), @@ -42,9 +46,6 @@ describe("historical source to bundle comparison", () => { const second = compareSourceToBundle(input); expect(first).toEqual(second); - expect(() => - sourceToBundleComparisonResultSchema.parse(first), - ).not.toThrow(); expect(first.summary).toEqual({ unchanged: 1, modified: 0, @@ -153,7 +154,7 @@ describe("source to bundle digest changes", () => { observations: [ { label: "display label", - source_map_reference: resolveJavaScriptSourceMapReference( + source_map_reference: sourceMapReference( "webpack://fixture/./src/modified.ts", null, "dist/main.js.map", @@ -202,6 +203,90 @@ describe("source to bundle digest changes", () => { }); }); +describe("source to bundle Unicode path collisions", () => { + it("keeps colliding canonical-equivalent current paths ambiguous", () => { + const historicalPath = "src/a\u0301\u0323.ts"; + const currentPaths = ["src/a\u0323\u0301.ts", "src/\u1ea1\u0301.ts"]; + const nodes = currentPaths.map((path, index) => + moduleNode(`unicode-${index}`, { path }), + ); + const result = comparePathToNodes(historicalPath, nodes); + const mapping = item(result, historicalPath); + + expect(mapping).toMatchObject({ + status: "unknown", + confidence: "unknown", + current_node_ids: [], + candidates: [ + expect.objectContaining({ + signals: expect.arrayContaining([ + expect.objectContaining({ kind: "unicode-equivalent-path" }), + ]), + }), + expect.objectContaining({ + signals: expect.arrayContaining([ + expect.objectContaining({ kind: "unicode-equivalent-path" }), + ]), + }), + ], + limitations: [ + expect.stringContaining( + "filesystem paths with distinct Unicode spellings", + ), + ], + }); + }); + + it("prefers a raw source-map path over canonically equivalent alternatives", () => { + const historicalPath = "src/cafe\u0301.ts"; + const exact = sourceMapNode( + "exact", + "cafe\u0301.ts", + "../src/", + HASH.modified, + ); + const equivalent = sourceMapNode( + "equivalent", + "caf\u00e9.ts", + "../src/", + HASH.removed, + ); + const result = comparePathToNodes(historicalPath, [exact, equivalent]); + + expect(item(result, historicalPath)).toMatchObject({ + status: "modified", + current_node_ids: [exact.node_id], + }); + }); + + it("finds Unicode-equivalent suffixes of absolute source-map URLs", () => { + const historicalPath = "src/caf\u00e9.ts"; + const node = sourceMapNode( + "remote", + "cafe\u0301.ts", + "https://cdn.example/src/", + HASH.modified, + ); + const result = comparePathToNodes(historicalPath, [node]); + + expect(item(result, historicalPath)).toMatchObject({ + status: "unknown", + current_node_ids: [], + candidates: [ + expect.objectContaining({ + current_node_id: node.node_id, + signals: expect.arrayContaining([ + expect.objectContaining({ + kind: "unicode-equivalent-path", + current_values: ["cdn.example/src/cafe\u0301.ts"], + }), + ]), + }), + ], + }); + }); +}); + describe("source to bundle path syntax", () => { it.each([ "src/modified#part.ts", @@ -282,9 +367,60 @@ describe("source to bundle path syntax", () => { }), ], }); + expect( + item(result, "src/modified.ts").candidates.flatMap( + ({ signals }) => signals, + ), + ).not.toContainEqual( + expect.objectContaining({ kind: "unicode-equivalent-path" }), + ); }); }); +describe("source-map resolver equivalence", () => { + it.each([ + { + sourceRoot: "https://cdn.example/src/", + expectedSource: "https://cdn.example/src/a.js", + expectedResolution: { kind: "suffix", path: "cdn.example/src/a.js" }, + }, + { + sourceRoot: "https://cdn.example/src//", + expectedSource: "https://cdn.example/src/a.js", + expectedResolution: { kind: "suffix", path: "cdn.example/src/a.js" }, + }, + { + sourceRoot: "webpack://", + expectedSource: "webpack:///a.js", + expectedResolution: { kind: "suffix", path: "a.js" }, + }, + ])( + "matches TraceMap for sourceRoot $sourceRoot", + ({ sourceRoot, expectedSource, expectedResolution }) => { + const producerSource = new TraceMap( + { + version: 3, + file: "main.js", + sourceRoot, + sources: ["a.js"], + names: [], + mappings: "", + }, + "file:///artifact/dist/main.js.map", + ).resolvedSources[0]; + const reference = sourceMapReference( + "a.js", + sourceRoot, + "dist/main.js.map", + ); + + expect(producerSource).toBe(expectedSource); + expect(reference.source_root).toBe(sourceRoot); + expect(reference.resolution).toEqual(expectedResolution); + }, + ); +}); + describe("source to bundle complete metadata", () => { it("accepts long source metadata and all reported limitations", () => { const result = compareSourceToBundle({ @@ -328,6 +464,61 @@ describe("source to bundle complete metadata", () => { }); }); +const sourceMapNode = ( + key: string, + sourceName: string, + sourceRoot: string, + digest: string, +) => + createJavaScriptApplicationNode({ + kind: "source-module", + identity: { + strategy: "source-map-original", + source_root: sourceRoot, + stability: "source-map-exact", + source_map_sha256: HASH.artifact, + original_source: sourceName, + source_sha256: digest, + }, + observations: [ + { + label: `${key} source-map spelling`, + source_map_reference: sourceMapReference( + sourceName, + sourceRoot, + `dist/${key}.map`, + ), + properties: { content_available: true, content_sha256: digest }, + evidence: artifactEvidence(HASH.artifact, `dist/${key}.map`), + }, + ], + }); + +const comparePathToNodes = ( + sourcePath: string, + nodes: readonly ApplicationNode[], +) => + compareSourceToBundle({ + reference: historicalGraph("complete", [sourcePath]), + application: { + evidenceId: EVIDENCE_ID, + rootArtifactSha256: HASH.artifact, + graph: createJavaScriptApplicationGraph({ + schema: "JavaScriptApplicationGraph", + root_node_ids: nodes.map(({ node_id: nodeId }) => nodeId), + nodes, + edges: [], + coverage: { + status: "complete", + truncated: false, + omitted_count: 0, + limits: [], + }, + limitations: [], + }), + }, + }); + const historicalGraph = ( inventoryState: "complete" | "partial", paths = [ diff --git a/src/domain/javascript/sourceToBundleComparison.ts b/src/domain/javascript/sourceToBundleComparison.ts index ac0007cb4..5a70c57f3 100644 --- a/src/domain/javascript/sourceToBundleComparison.ts +++ b/src/domain/javascript/sourceToBundleComparison.ts @@ -258,7 +258,11 @@ const classifyMapping = ( confidence: "unknown", currentNodeIds: [], limitations: [ - "Only weak basename or extension signals were available; no mapping was forced.", + candidates.some((candidate) => + hasSignal(candidate, "unicode-equivalent-path"), + ) + ? "Unicode-equivalent paths are inferred candidates only; filesystem paths with distinct Unicode spellings may identify different files, so no mapping was forced." + : "Only weak basename or extension signals were available; no mapping was forced.", ], }; return absenceComplete diff --git a/src/domain/javascript/sourceToBundleComparisonSchemas.ts b/src/domain/javascript/sourceToBundleComparisonSchemas.ts index 2e27cadb3..9d6bd6b2f 100644 --- a/src/domain/javascript/sourceToBundleComparisonSchemas.ts +++ b/src/domain/javascript/sourceToBundleComparisonSchemas.ts @@ -15,6 +15,7 @@ export const SOURCE_TO_BUNDLE_SIGNAL_WEIGHTS = [ ["source-map-original-path", 70], ["current-path-exact", 60], ["current-path-suffix", 40], + ["unicode-equivalent-path", 20], ["basename-match", 20], ["language-extension", 5], ] as const; @@ -24,6 +25,7 @@ const sourceToBundleSignalKindSchema = z.enum([ "source-map-original-path", "current-path-exact", "current-path-suffix", + "unicode-equivalent-path", "basename-match", "language-extension", ]); diff --git a/src/domain/javascript/sourceToBundleSignals.ts b/src/domain/javascript/sourceToBundleSignals.ts index 2190988fe..b3546aecc 100644 --- a/src/domain/javascript/sourceToBundleSignals.ts +++ b/src/domain/javascript/sourceToBundleSignals.ts @@ -102,8 +102,12 @@ export const buildSourceToBundleCandidateIndex = ( for (const path of projection.paths) { for (const suffix of path.allowSuffix ? pathSuffixes(path.value) - : [path.value]) + : [path.value]) { addIndexValue(byPathSuffix, suffix, projection.node.node_id); + const canonical = suffix.normalize("NFC"); + if (canonical !== suffix) + addIndexValue(byPathSuffix, canonical, projection.node.node_id); + } addIndexValue( byBasename, posix.basename(path.value), @@ -126,9 +130,13 @@ export const candidateIdsForSource = ( source: SourceFile, index: SourceToBundleCandidateIndex, ): string[] => { + const canonicalPath = source.path.normalize("NFC"); const groups = [ source.sha256 === null ? undefined : index.byDigest.get(source.sha256), index.byPathSuffix.get(source.path), + canonicalPath === source.path + ? undefined + : index.byPathSuffix.get(canonicalPath), index.byBasename.get(posix.basename(source.path)), ]; const seen = new Set(); @@ -220,6 +228,14 @@ const candidateSignals = ( signals.push( signal("source-map-original-path", source.path, sourceMapPaths), ); + const unicodeEquivalentPaths = matchingUnicodePaths( + source.path, + current.paths, + ); + if (unicodeEquivalentPaths.length > 0) + signals.push( + signal("unicode-equivalent-path", source.path, unicodeEquivalentPaths), + ); const otherPaths = current.paths.filter( ({ kind }) => kind !== "source-map-original", ); @@ -286,6 +302,27 @@ const matchingPaths = ( ) .map(({ value }) => value); +const matchingUnicodePaths = ( + sourcePath: string, + currentPaths: readonly CurrentPath[], +): string[] => { + const canonicalSource = sourcePath.normalize("NFC"); + return currentPaths + .filter(({ value, allowSuffix }) => { + if ( + value === sourcePath || + (allowSuffix && value.endsWith(`/${sourcePath}`)) + ) + return false; + const canonicalValue = value.normalize("NFC"); + return ( + canonicalValue === canonicalSource || + (allowSuffix && canonicalValue.endsWith(`/${canonicalSource}`)) + ); + }) + .map(({ value }) => value); +}; + const addJsonPath = (paths: CurrentPath[], value: unknown): void => { if (typeof value === "string") addPath(paths, "observation-path", value); }; diff --git a/src/domain/managed/managedArtifact.ts b/src/domain/managed/managedArtifact.ts index 297b60a4e..94eff2398 100644 --- a/src/domain/managed/managedArtifact.ts +++ b/src/domain/managed/managedArtifact.ts @@ -203,10 +203,18 @@ const managedMethodSchema = z.object({ body: managedMethodBodySchema, }); +// Declaring types are namespace-qualified like TypeDef `full_name`, nested +// types keep bare names, and a generic instantiation names its generic type. +// Evidence recorded before declaring types were named reads them as null. +const declaringTypeNameSchema = z.string().nullable().default(null); + const managedMemberReferenceSchema = z.object({ token: tokenSchema, row_offset: offsetSchema, parent_token: tokenSchema.nullable(), + parent_name: declaringTypeNameSchema.describe( + "Name of a TypeDef, TypeRef, or generic-instantiation TypeSpec parent; null for other parents or when unknown", + ), name: z.string(), signature: managedSignatureSchema, }); @@ -281,7 +289,13 @@ const managedCallEdgeSchema = z.object({ opcode: z.string(), target_token: tokenSchema, target_kind: z.enum(["method-def", "member-ref", "method-spec", "unknown"]), - target_name: z.string().nullable(), + target_name: z + .string() + .nullable() + .describe("Method name; a method-spec target reports its generic method"), + target_declaring_type: declaringTypeNameSchema.describe( + "Declaring type of the called method; null when unknown", + ), }); const managedFieldAccessSchema = z.object({ @@ -290,6 +304,9 @@ const managedFieldAccessSchema = z.object({ opcode: z.string(), field_token: tokenSchema, field_name: z.string().nullable(), + field_declaring_type: declaringTypeNameSchema.describe( + "Declaring type of the accessed field; null when unknown", + ), }); const managedParseIssueSchema = z.object({ diff --git a/src/domain/managed/managedReconstruction.ts b/src/domain/managed/managedReconstruction.ts index e10be0ffd..9600166ae 100644 --- a/src/domain/managed/managedReconstruction.ts +++ b/src/domain/managed/managedReconstruction.ts @@ -204,7 +204,8 @@ const buildReconstructionSummary = ( ({ reconstruction }) => reconstruction.kind === "semantic-pseudocode", ).length, total_text_bytes: methods.reduce( - (total, { reconstruction }) => total + reconstruction.text_length, + (total, { reconstruction }) => + total + Buffer.byteLength(reconstruction.text, "utf8"), 0, ), }); diff --git a/src/domain/native/nativeAotPe.test.ts b/src/domain/native/nativeAotPe.test.ts index 8ae624d44..ab031cc85 100644 --- a/src/domain/native/nativeAotPe.test.ts +++ b/src/domain/native/nativeAotPe.test.ts @@ -159,3 +159,22 @@ describe("read-only NativeAOT PE parser", () => { }); }); }); + +describe("NativeAOT PE signature admission", () => { + it.each([ + ["MZ", "Not a PE image"], + ["PE", "Invalid PE signature"], + ])( + "rejects a high-bit lookalike of the %s signature", + (signature, reason) => { + const bytes = nativeAotPeFixture(); + const offset = signature === "MZ" ? 0 : bytes.readUInt32LE(0x3c); + for (const position of [offset, offset + 1]) + bytes.writeUInt8(bytes.readUInt8(position) | 0x80, position); + expect( + parseNativeAotPe(bytes, nativeAotPeDigest(bytes), SUMMARY_BUDGET) + .summary, + ).toMatchObject({ status: "not_applicable", reason }); + }, + ); +}); diff --git a/src/domain/native/nativeAotPe.ts b/src/domain/native/nativeAotPe.ts index b67a58398..7f9c5eb55 100644 --- a/src/domain/native/nativeAotPe.ts +++ b/src/domain/native/nativeAotPe.ts @@ -384,11 +384,11 @@ class PeImage { constructor(bytes: Buffer) { this.#bytes = bytes; - if (bytes.length < 0x40 || bytes.toString("ascii", 0, 2) !== "MZ") + if (bytes.length < 0x40 || bytes.toString("latin1", 0, 2) !== "MZ") throw new Error("Not a PE image"); const peOffset = this.#u32(0x3c); this.#range(peOffset, 24); - if (bytes.toString("ascii", peOffset, peOffset + 4) !== "PE\0\0") + if (bytes.toString("latin1", peOffset, peOffset + 4) !== "PE\0\0") throw new Error("Invalid PE signature"); this.#machine = this.#u16(peOffset + 4); const count = this.#u16(peOffset + 6); diff --git a/src/domain/native/nativeInvestigationGraph.test.ts b/src/domain/native/nativeInvestigationGraph.test.ts index 015458319..684b2a02e 100644 --- a/src/domain/native/nativeInvestigationGraph.test.ts +++ b/src/domain/native/nativeInvestigationGraph.test.ts @@ -83,6 +83,61 @@ const graph = nativeInvestigationGraphSchema.parse({ }); describe("native investigation graph traces", () => { + it.each(["forward", "backward"] as const)( + "keeps both endpoints of retained edges within the %s node budget", + (direction) => { + const trace = traceNativeInvestigationGraph(graph, { + start: direction === "forward" ? "a" : "c", + direction, + limits: { max_nodes: 1 }, + }); + expect(trace.nodes).toHaveLength(1); + expect(trace.edges).toEqual([]); + expect(trace).toMatchObject({ + truncated: true, + reason: "max_nodes_reached", + reached_depth: 0, + }); + }, + ); + + it.each([{ max_nodes: 2 }, { max_edges: 1 }])( + "reports retained node depth when a sibling hits the budget %j", + (limits) => { + const branching = { + ...graph, + edges: graph.edges + .filter((edge) => edge.resolution !== "unresolved") + .map((edge) => ({ ...edge, from: "a" })), + }; + const trace = traceNativeInvestigationGraph(branching, { + start: "a", + limits, + }); + expect(trace.nodes.map(({ id }) => id)).toEqual(["a", "b"]); + expect(trace.edges.map(({ id }) => id)).toEqual(["1"]); + expect(trace.reached_depth).toBe(1); + expect(trace.truncated).toBe(true); + }, + ); + + it("retains cycle edges to admitted nodes when the node budget is full", () => { + const cycle = { + ...graph, + nodes: graph.nodes.slice(0, 2), + edges: [ + graph.edges[0], + { ...graph.edges[0], id: "back", from: "b", to: "a" }, + ], + }; + const trace = traceNativeInvestigationGraph( + nativeInvestigationGraphSchema.parse(cycle), + { start: "a", limits: { max_nodes: 2 } }, + ); + expect(trace.edges.map(({ id }) => id)).toEqual(["1", "back"]); + expect(trace).toMatchObject({ reached_depth: 1, truncated: false }); + }); + it("retains observed and inferred route edges plus unresolved dispatch", () => { const trace = traceNativeInvestigationGraph(graph, { start: "a" }); expect(trace.nodes.map(({ id }) => id)).toEqual(["a", "b", "c"]); diff --git a/src/domain/native/nativeInvestigationGraph.ts b/src/domain/native/nativeInvestigationGraph.ts index 0dd0dbe8b..9b9efc412 100644 --- a/src/domain/native/nativeInvestigationGraph.ts +++ b/src/domain/native/nativeInvestigationGraph.ts @@ -207,7 +207,6 @@ export const traceNativeInvestigationGraph = ( while (queue.length > 0) { const current = queue.shift(); if (current === undefined) break; - reachedDepth = Math.max(reachedDepth, current.depth); const adjacent = adjacency.get(current.id) ?? []; for (const edge of adjacent) { @@ -227,19 +226,24 @@ export const traceNativeInvestigationGraph = ( reason ??= "max_depth_reached"; continue; } - selectedEdges.set(edge.id, edge); - if (alreadyVisited) continue; - if (visited.size >= limits.max_nodes) { + // Check destination capacity before retaining its edge. Otherwise the node + // bound leaves a resolved edge pointing outside the returned trace. + if (!alreadyVisited && visited.size >= limits.max_nodes) { truncated = true; reason = "max_nodes_reached"; break; } + selectedEdges.set(edge.id, edge); + if (alreadyVisited) continue; if (!nodeById.has(nextId)) { truncated = true; reason ??= "edge_target_missing_from_graph"; continue; } visited.add(nextId); + // A retained node has already been reached even if a later sibling + // exhausts a budget before this node is dequeued for expansion. + reachedDepth = Math.max(reachedDepth, current.depth + 1); queue.push({ id: nextId, depth: current.depth + 1 }); } if (reason === "max_edges_reached" || reason === "max_nodes_reached") break; diff --git a/src/domain/native/nativeLoadImage.ts b/src/domain/native/nativeLoadImage.ts index 6a3be4421..a4efc2847 100644 --- a/src/domain/native/nativeLoadImage.ts +++ b/src/domain/native/nativeLoadImage.ts @@ -1,6 +1,7 @@ import { z } from "zod"; import { nativeMetadataRecoverySummarySchema } from "./nativeMetadataRecovery.js"; import { jsonValueSchema } from "../jsonValue.js"; +import { binaryArchitectureSchema } from "../rawBinary.js"; const integer = z.number().int().min(0).max(Number.MAX_SAFE_INTEGER); const digest = z.string().regex(/^[a-f0-9]{64}$/u); @@ -112,7 +113,7 @@ export const nativeLoadImageSchema = z.union([ status: z.enum(["verified", "mismatch"]), format: z.literal("raw-binary"), target_sha256: digest, - architecture: z.enum(["x86", "x86_64", "arm", "arm64"]), + architecture: binaryArchitectureSchema, language_id: z.string().min(1), compiler_spec_id: z.string().min(1), base_address: address, diff --git a/src/domain/offsetDateTime.ts b/src/domain/offsetDateTime.ts index bccda7494..b7d38550f 100644 --- a/src/domain/offsetDateTime.ts +++ b/src/domain/offsetDateTime.ts @@ -1,5 +1,11 @@ import { z } from "zod"; +export interface OffsetDateTimeInstant { + readonly epochSecond: number; + /** Exact fractional second with insignificant trailing zeroes removed. */ + readonly fraction: string; +} + /** * ISO-8601 date-time that requires an explicit UTC offset. * @@ -19,3 +25,80 @@ export const offsetDateTimeSchema = z /^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z|(?:[+\x2d](?:[01]\d|2[0-3]):[0-5]\d)))$/u, "Use an ISO-8601 date-time with an explicit UTC offset", ); + +/** Parse an already validated offset timestamp without losing sub-millisecond precision. */ +export const parseOffsetDateTimeInstant = ( + value: string, +): OffsetDateTimeInstant => { + const match = /^(.*?)(?:\.(\d+))?(Z|[+\x2d]\d{2}:\d{2})$/u.exec(value); + if (match === null) + throw new TypeError("Expected a validated offset date-time."); + const [, dateTime, fractionalDigits, offset] = match; + if (dateTime === undefined || offset === undefined) + throw new TypeError("Expected a validated offset date-time."); + const wholeSecondMilliseconds = Date.parse(`${dateTime}${offset}`); + if ( + !Number.isSafeInteger(wholeSecondMilliseconds) || + wholeSecondMilliseconds % 1000 !== 0 + ) + throw new TypeError( + "Offset date-time is outside the supported time range.", + ); + return { + epochSecond: wholeSecondMilliseconds / 1000, + fraction: fractionalDigits?.replace(/0+$/u, "") ?? "", + }; +}; + +/** Convert a Date.now-style integer into the same exact comparison shape. */ +export const offsetDateTimeInstantFromEpochMilliseconds = ( + epochMilliseconds: number, +): OffsetDateTimeInstant => { + if (!Number.isFinite(epochMilliseconds)) + throw new TypeError("Expected finite epoch milliseconds."); + if (Math.abs(epochMilliseconds) > Number.MAX_SAFE_INTEGER) + return { epochSecond: epochMilliseconds / 1000, fraction: "" }; + const epochSecond = Math.floor(epochMilliseconds / 1000); + const millisecondsWithinSecond = epochMilliseconds - epochSecond * 1000; + return { + epochSecond, + fraction: fractionalSecondFromMilliseconds(millisecondsWithinSecond), + }; +}; + +const fractionalSecondFromMilliseconds = (milliseconds: number): string => { + if (milliseconds === 0) return ""; + const [coefficient = "", exponentText] = String(milliseconds) + .toLowerCase() + .split("e"); + const exponent = exponentText === undefined ? 0 : Number(exponentText); + const point = coefficient.indexOf("."); + const decimalPosition = + (point < 0 ? coefficient.length : point) + exponent - 3; + const digits = coefficient.replace(".", ""); + const fraction = + decimalPosition <= 0 + ? `0.${"0".repeat(-decimalPosition)}${digits}` + : decimalPosition < digits.length + ? `${digits.slice(0, decimalPosition)}.${digits.slice(decimalPosition)}` + : `${digits}${"0".repeat(decimalPosition - digits.length)}`; + return ( + fraction.includes(".") ? fraction.slice(fraction.indexOf(".") + 1) : "" + ).replace(/0+$/u, ""); +}; + +/** Compare instants exactly while treating absent fraction digits as zeroes. */ +export const compareOffsetDateTimeInstants = ( + left: OffsetDateTimeInstant, + right: OffsetDateTimeInstant, +): number => { + if (left.epochSecond !== right.epochSecond) + return left.epochSecond < right.epochSecond ? -1 : 1; + const length = Math.max(left.fraction.length, right.fraction.length); + for (let index = 0; index < length; index += 1) { + const leftDigit = left.fraction.charCodeAt(index) || 48; + const rightDigit = right.fraction.charCodeAt(index) || 48; + if (leftDigit !== rightDigit) return leftDigit < rightDigit ? -1 : 1; + } + return 0; +}; diff --git a/src/domain/pcap/pcapAnalysis.ts b/src/domain/pcap/pcapAnalysis.ts new file mode 100644 index 000000000..21c64f365 --- /dev/null +++ b/src/domain/pcap/pcapAnalysis.ts @@ -0,0 +1,110 @@ +import { z } from "zod"; +import { localPathStringSchema } from "../localPath.js"; +import { digestSchema } from "../digests.js"; + +/** + * Caller intent for packet-capture analysis through caller-supplied tshark + * and capinfos. REA never installs Wireshark or its tools. + */ +export const pcapInputSchemas = { + inspect_pcap_client: z.strictObject({}), + inspect_pcap_capture: z.strictObject({ + path: localPathStringSchema.describe( + "Local pcap or pcapng capture file to inspect", + ), + }), +} as const; + +/** Supported pcap-backed operations. */ +export type PcapOperation = keyof typeof pcapInputSchemas; +/** Validated pcap requests carry their selected operation. */ +export type PcapRequest = { + [Name in PcapOperation]: { + operation: Name; + input: z.infer<(typeof pcapInputSchemas)[Name]>; + }; +}[PcapOperation]; + +/** Validate operation and input together so their types remain correlated. */ +export const pcapRequestSchema = z.discriminatedUnion("operation", [ + z.strictObject({ + operation: z.literal("inspect_pcap_client"), + input: pcapInputSchemas.inspect_pcap_client, + }), + z.strictObject({ + operation: z.literal("inspect_pcap_capture"), + input: pcapInputSchemas.inspect_pcap_capture, + }), +]); + +const client = z.strictObject({ + tshark_command: z.string().min(1), + tshark_source: z.enum(["environment", "path"]), + tshark_version: z.string().min(1).nullable(), + capinfos_command: z.string().min(1), + capinfos_source: z.enum(["environment", "path"]), + capinfos_version: z.string().min(1).nullable(), +}); + +const endpoint = z.strictObject({ + address: z.string().min(1), + port: z.number().int().positive().nullable(), + packet_count: z.number().int().positive(), +}); + +const dnsQuery = z.strictObject({ + query: z.string().min(1), + response_addresses: z.array(z.string().min(1)), + response_address_count: z.number().int().nonnegative(), +}); + +const protocolCount = z.strictObject({ + protocol: z.string().min(1), + packet_count: z.number().int().positive(), +}); + +/** Portable pcap observations with explicit coverage limitations. */ +export const pcapResultSchemas = { + inspect_pcap_client: z.strictObject({ + client, + }), + inspect_pcap_capture: z.strictObject({ + client, + target: z.strictObject({ + path: localPathStringSchema, + bytes: z.number().int().nonnegative(), + sha256: digestSchema, + }), + /** File format as capinfos reports it (pcap vs pcapng). */ + file_type: z.string().min(1).nullable(), + encapsulation: z.string().min(1).nullable(), + packet_count: z.number().int().nonnegative().nullable(), + capture_start: z.string().nullable(), + capture_end: z.string().nullable(), + capture_duration_seconds: z.number().nonnegative().nullable(), + snapshot_length: z.number().int().positive().nullable(), + /** Distinct endpoints with packet counts, bounded. */ + endpoints: z.array(endpoint), + endpoint_count: z.number().int().nonnegative(), + /** DNS queries with resolved addresses, bounded. */ + dns_queries: z.array(dnsQuery), + dns_query_count: z.number().int().nonnegative(), + /** TLS Server Name Indication values observed in handshakes. */ + tls_sni_values: z.array(z.string().min(1)), + tls_sni_count: z.number().int().nonnegative(), + /** Protocol distribution (frame.protocols column counts). */ + protocol_distribution: z.array(protocolCount), + protocol_count: z.number().int().nonnegative(), + /** HTTP request URIs seen on plain-text ports, bounded. */ + http_request_uris: z.array(z.string().min(1)), + http_request_uri_count: z.number().int().nonnegative(), + http_uri_redaction_count: z.number().int().nonnegative(), + unattributed_dns_addresses: z.array(z.string().min(1)), + unattributed_dns_address_count: z.number().int().nonnegative(), + unattributed_dns_packet_count: z.number().int().nonnegative(), + coverage: z.enum(["complete", "partial"]), + diagnostics: z + .strictObject({ capinfos_stderr: z.string(), tshark_stderr: z.string() }) + .optional(), + }), +} as const; diff --git a/src/domain/pcap/pcapEvidence.ts b/src/domain/pcap/pcapEvidence.ts new file mode 100644 index 000000000..848350fe3 --- /dev/null +++ b/src/domain/pcap/pcapEvidence.ts @@ -0,0 +1,34 @@ +import { createEvidence } from "../evidence.js"; +import { jsonObjectSchema, type JsonValue } from "../jsonValue.js"; +import type { PcapRequest } from "./pcapAnalysis.js"; + +/** Domain facts needed by both successful and retained partial observations. */ +interface PcapObservation { + readonly subject: Parameters[0] | null; + readonly provider: Parameters[1]; + readonly result: JsonValue; + readonly rawResult: JsonValue | null; + readonly limitations: readonly string[]; + readonly locations: NonNullable< + Parameters[2]["locations"] + >; +} + +/** One owner commits packet aggregation provenance and caller intent into Evidence. */ +export const createPcapEvidence = ( + request: PcapRequest, + observation: PcapObservation, +) => + createEvidence(observation.subject ?? undefined, observation.provider, { + operation: request.operation, + parameters: jsonObjectSchema.parse(request.input), + result: observation.result, + rawResult: observation.rawResult, + limitations: observation.limitations, + locations: observation.locations, + confidence: + request.operation === "inspect_pcap_capture" ? "derived" : "observed", + ...(request.operation === "inspect_pcap_capture" + ? { authority: "historical-reference" as const } + : {}), + }); diff --git a/src/domain/pngHeader.test.ts b/src/domain/pngHeader.test.ts index e8df531bc..676a832e0 100644 --- a/src/domain/pngHeader.test.ts +++ b/src/domain/pngHeader.test.ts @@ -33,3 +33,24 @@ it("rejects incomplete headers, invalid signatures and zero dimensions", () => { expect(readPngHeaderDimensions(zero)).toBeUndefined(); } }); + +it.each([0, 12, 14, 0xffff_ffff])("rejects IHDR data length %s", (length) => { + const bytes = Buffer.from(png); + bytes.writeUInt32BE(length, 8); + expect(readPngHeaderDimensions(bytes)).toBeUndefined(); +}); + +it.each([16, 20])("rejects out-of-range PNG dimension at %s", (offset) => { + const bytes = Buffer.from(png); + bytes.writeUInt32BE(0x8000_0000, offset); + expect(readPngHeaderDimensions(bytes)).toBeUndefined(); +}); + +it("accepts the largest dimension representable by the PNG specification", () => { + const bytes = Buffer.from(png); + bytes.writeUInt32BE(0x7fff_ffff, 16); + expect(readPngHeaderDimensions(bytes)).toEqual({ + width: 0x7fff_ffff, + height: 1, + }); +}); diff --git a/src/domain/pngHeader.ts b/src/domain/pngHeader.ts index 58cdd7d3f..5ab8c7c3d 100644 --- a/src/domain/pngHeader.ts +++ b/src/domain/pngHeader.ts @@ -8,10 +8,16 @@ export const readPngHeaderDimensions = ( if ( bytes.byteLength < 24 || !bytes.subarray(0, 8).equals(PNG_SIGNATURE) || + bytes.readUInt32BE(8) !== 13 || !bytes.subarray(12, 16).equals(IHDR_TYPE) ) return undefined; const width = bytes.readUInt32BE(16); const height = bytes.readUInt32BE(20); - return width === 0 || height === 0 ? undefined : { width, height }; + return width === 0 || + height === 0 || + width > 0x7fff_ffff || + height > 0x7fff_ffff + ? undefined + : { width, height }; }; diff --git a/src/domain/process/processCapture.fixture.ts b/src/domain/process/processCapture.fixture.ts index 701e76e2a..a8f2e669f 100644 --- a/src/domain/process/processCapture.fixture.ts +++ b/src/domain/process/processCapture.fixture.ts @@ -1,3 +1,4 @@ +import type { ProcessTraceSpecification } from "./processTraceSpecification.js"; import { compareProcessCaptures } from "./processComparison.js"; import { parseProcessCapture } from "./processCaptureParsing.js"; import { EMPTY_PROCESS_CAPTURE_EXAMPLE } from "./processCaptureExample.js"; @@ -112,3 +113,56 @@ export const compareUnverifiedProcessCaptures = ( parseProcessCapture(right), options, ); + +/** One journal and its matching declared trace, shared by domain and adapter checks. */ +export const processTraceCaptureFixture = () => { + const capture = { + ...EMPTY_PROCESS_CAPTURE_EXAMPLE, + event_journal: [ + { capture_order: 0, collection: "filesystem_checkpoints", index: 0 }, + { capture_order: 1, collection: "lifecycle", index: 0 }, + { capture_order: 2, collection: "lifecycle", index: 1 }, + { capture_order: 3, collection: "filesystem_checkpoints", index: 1 }, + ], + } as const; + const [before, after] = capture.filesystem_checkpoints; + if (before === undefined || after === undefined) + throw new Error( + "Trace fixture requires before and after filesystem checkpoints", + ); + const traceSpecification: ProcessTraceSpecification = { + events: [ + { + id: "before", + source: "filesystem", + exact: before, + cardinality: { kind: "required" }, + }, + { + id: "exit", + source: "lifecycle", + exact: { event: "exit", ...capture.exit }, + cardinality: { kind: "required" }, + }, + { + id: "settlement", + source: "lifecycle", + exact: { event: "settlement", ...capture.settlement }, + cardinality: { kind: "required" }, + }, + { + id: "after", + source: "filesystem", + exact: after, + cardinality: { kind: "required" }, + }, + ], + language: { + kind: "finite_traces", + variants: [ + { id: "normal", trace: ["before", "exit", "settlement", "after"] }, + ], + }, + }; + return { capture, traceSpecification }; +}; diff --git a/src/domain/process/processCaptureValidation.ts b/src/domain/process/processCaptureValidation.ts index 10d6b831b..33c6d5483 100644 --- a/src/domain/process/processCaptureValidation.ts +++ b/src/domain/process/processCaptureValidation.ts @@ -47,10 +47,6 @@ const validateCommitments = ( digestProcessCommitment( value, ), `manifest.${field}`, "commitment does not match its canonical value"); - require(Date.parse(manifest.started_at) <= - Date.parse( - manifest.completed_at, - ), "manifest.completed_at", "completion precedes start"); }; const validateOrdering = ( diff --git a/src/domain/process/processScenario.ts b/src/domain/process/processScenario.ts index a87ffd86e..e98d7a537 100644 --- a/src/domain/process/processScenario.ts +++ b/src/domain/process/processScenario.ts @@ -128,6 +128,9 @@ export const processComparisonContract = ( : {}), limits: scenario.limits, normalization: scenario.normalization, + ...(scenario.normalization.paths + ? { path_normalization_version: "longest-original-paths-v1" } + : {}), ...(scenario.normalization.ports ? { port_normalization_version: "contextual-endpoints-v2" } : {}), diff --git a/src/domain/process/processTraceAdapters.test.ts b/src/domain/process/processTraceAdapters.test.ts index 18cdea5c6..959ca76c8 100644 --- a/src/domain/process/processTraceAdapters.test.ts +++ b/src/domain/process/processTraceAdapters.test.ts @@ -1,72 +1,13 @@ import { describe, expect, it } from "vitest"; -import { EMPTY_PROCESS_CAPTURE_EXAMPLE } from "./processCaptureExample.js"; -import { accountFullyObservedProcessCapture } from "./processCapture.fixture.js"; +import { + processTraceCaptureFixture, + accountFullyObservedProcessCapture, +} from "./processCapture.fixture.js"; import { compareProcessCaptures } from "./processComparison.js"; import { parseProcessCapture } from "./processCaptureParsing.js"; import type { ProcessTraceSpecification } from "./processTraceSpecification.js"; -const beforeCheckpoint = { - name: "before", - at_ms: 0, - files: [], - effects: [], - truncated: false, -}; -const afterCheckpoint = { - name: "after_settlement", - at_ms: 50, - files: [], - effects: [], - truncated: false, -}; - -const capture = { - ...EMPTY_PROCESS_CAPTURE_EXAMPLE, - event_journal: [ - { capture_order: 0, collection: "filesystem_checkpoints", index: 0 }, - { capture_order: 1, collection: "lifecycle", index: 0 }, - { capture_order: 2, collection: "lifecycle", index: 1 }, - { capture_order: 3, collection: "filesystem_checkpoints", index: 1 }, - ], -} as const; - -const traceSpecification: ProcessTraceSpecification = { - events: [ - { - id: "before", - source: "filesystem", - exact: beforeCheckpoint, - cardinality: { kind: "required" }, - }, - { - id: "exit", - source: "lifecycle", - exact: { event: "exit", ...capture.exit }, - cardinality: { kind: "required" }, - }, - { - id: "settlement", - source: "lifecycle", - exact: { event: "settlement", ...capture.settlement }, - cardinality: { kind: "required" }, - }, - { - id: "after", - source: "filesystem", - exact: afterCheckpoint, - cardinality: { kind: "required" }, - }, - ], - language: { - kind: "finite_traces", - variants: [ - { - id: "normal", - trace: ["before", "exit", "settlement", "after"], - }, - ], - }, -}; +const { capture, traceSpecification } = processTraceCaptureFixture(); describe("declared trace comparison nonconformance", () => { it("does not call identical captures changed when both violate the language", () => { diff --git a/src/domain/process/processTraceComparison.test.ts b/src/domain/process/processTraceComparison.test.ts index caeaf7999..2207352e3 100644 --- a/src/domain/process/processTraceComparison.test.ts +++ b/src/domain/process/processTraceComparison.test.ts @@ -1,6 +1,7 @@ import { describe, expect, it } from "vitest"; import { parseProcessCapture } from "./processCaptureParsing.js"; import type { ProcessTraceSpecification } from "./processTraceSpecification.js"; +import { processTraceSpecificationSchema as specificationSchema } from "./processTraceSpecification.js"; import { processTraceComparisonResultSchema } from "./processTraceEvaluation.js"; import { capture, @@ -100,6 +101,33 @@ describe("generic capture trace comparison", () => { right: { matched_variant: "two" }, }); }); + + it("compares a declared identifier the comparison result can represent", () => { + // The specification owns the trace identifier vocabulary and the result + // echoes those identifiers verbatim, so both must accept the same values + // rather than handing the caller a raw schema error for its own input. + const eventId = `e${"0".repeat(64)}`; + const specification: ProcessTraceSpecification = { + events: [ + { + id: eventId, + source: "terminal_raw", + exact: terminal, + cardinality: { kind: "required" }, + }, + ], + language: { + kind: "finite_traces", + variants: [{ id: "only", trace: [eventId] }], + }, + }; + expect(specificationSchema.safeParse(specification).success).toBe(true); + const left = capture(values(["terminal", "process"])); + expect(compareProcessTraces(left, left, specification)).toMatchObject({ + verdict: "equivalent", + left: { matched_variant: "only", raw_trace: [{ event_id: eventId }] }, + }); + }); }); describe("process trace constraints and evidence sufficiency", () => { diff --git a/src/domain/process/processTraceEvaluation.ts b/src/domain/process/processTraceEvaluation.ts index 51a2a15f8..7c232a480 100644 --- a/src/domain/process/processTraceEvaluation.ts +++ b/src/domain/process/processTraceEvaluation.ts @@ -13,13 +13,12 @@ import { import { canonicalTraceJson, processTraceCardinalityBounds, + processTraceIdentifierSchema, type ProcessTraceSpecification, } from "./processTraceSpecification.js"; -const identifierSchema = z.string().regex(/^[A-Za-z][A-Za-z0-9._\x2d]{0,63}$/u); - const matchedEventSchema = z.strictObject({ - event_id: identifierSchema, + event_id: processTraceIdentifierSchema, location: processObservationLocationSchema, }); @@ -32,7 +31,7 @@ const sideResultSchema = z.union([ z.strictObject({ ...sideResultShape, status: z.literal("pass"), - matched_variant: identifierSchema.nullable(), + matched_variant: processTraceIdentifierSchema.nullable(), }), z.strictObject({ ...sideResultShape, @@ -53,7 +52,7 @@ const diagnosticSchema = z.strictObject({ ]), side: z.enum(["left", "right"]), message: z.string(), - event_ids: z.array(identifierSchema), + event_ids: z.array(processTraceIdentifierSchema), locations: z.array(processObservationLocationSchema), }); diff --git a/src/domain/process/processTraceSpecification.test.ts b/src/domain/process/processTraceSpecification.test.ts index aeedf4d31..ab649d2aa 100644 --- a/src/domain/process/processTraceSpecification.test.ts +++ b/src/domain/process/processTraceSpecification.test.ts @@ -39,30 +39,110 @@ const specification = (): ProcessTraceSpecification => ({ }); describe("process trace specification", () => { - it("summarizes an incomplete language with its exact missing pair count", () => { - const events = Array.from({ length: 200 }, (_, index) => ({ - id: `e${String(index)}`, + it("reports independent declaration errors alongside a cycle", () => { + const base = specification(); + const result = processTraceSpecificationSchema.safeParse({ + ...base, + language: { + ...base.language, + happens_before: [ + { before: "ready", after: "worker" }, + { before: "worker", after: "ready" }, + ], + not_before: [{ event: "missing", anchor: "ready" }], + unordered_groups: [{ events: ["missing", "missing"] }], + prefix: ["missing"], + suffix: ["missing"], + }, + }); + if (result.success) throw new Error("Expected invalid trace declarations"); + expect(result.error.issues).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + message: "happens-before graph must be acyclic", + }), + expect.objectContaining({ + path: ["language", "not_before", 0, "event"], + }), + expect.objectContaining({ + path: ["language", "unordered_groups", 0, "events"], + }), + expect.objectContaining({ path: ["language", "prefix", 0] }), + expect.objectContaining({ path: ["language", "suffix", 0] }), + ]), + ); + }); + + it("accepts an acyclic order with shared successors and explicit unordered peers", () => { + const result = processTraceSpecificationSchema.parse({ + events: ["a", "b", "c", "d"].map((id) => ({ + id, + source: "lifecycle", + exact: { marker: id }, + })), + language: { + kind: "partial_order", + happens_before: [ + { before: "a", after: "b" }, + { before: "a", after: "c" }, + { before: "b", after: "d" }, + { before: "c", after: "d" }, + ], + unordered_groups: [{ events: ["b", "c"] }], + prefix: ["a"], + suffix: ["d"], + }, + }); + expect(result.language.kind).toBe("partial_order"); + }); +}); + +describe("process trace ordering validation", () => { + it("uses a unique topological order for transitive declarations and conflicts", () => { + const events = ["a", "b", "c"].map((id) => ({ + id, source: "lifecycle" as const, - exact: { fixture: index }, + exact: { marker: id }, cardinality: { kind: "optional" as const }, })); - const result = processTraceSpecificationSchema.safeParse({ + const accepted = processTraceSpecificationSchema.safeParse({ events, - language: { kind: "partial_order" }, + language: { + kind: "partial_order", + happens_before: [ + { before: "a", after: "b" }, + { before: "b", after: "c" }, + ], + }, }); - expect(result.success).toBe(false); - if (result.success) - throw new Error("Expected incomplete pair declarations"); - expect(result.error.issues).toEqual([ + expect(accepted.success).toBe(true); + + const rejected = processTraceSpecificationSchema.safeParse({ + events, + language: { + kind: "partial_order", + happens_before: [ + { before: "a", after: "b" }, + { before: "b", after: "c" }, + ], + not_before: [{ event: "a", anchor: "c" }], + unordered_groups: [{ events: ["a", "b", "c"] }], + }, + }); + if (rejected.success) throw new Error("Expected ordered pair conflicts"); + expect(rejected.error.issues).toEqual([ expect.objectContaining({ - path: ["language"], - message: - "19900 event pairs must be explicitly ordered or unordered; first undeclared pair: e0/e1", + path: ["language", "not_before", 0], + message: "not-before constraint conflicts with happens-before order", + }), + expect.objectContaining({ + path: ["language", "unordered_groups", 0], + message: "ordered events cannot also be declared unordered", }), ]); }); - it("counts only undeclared pairs after transitive, not-before and unordered relations", () => { + it("counts only undeclared pairs across transitive and disconnected relations", () => { const events = ["a", "b", "c", "d"].map((id) => ({ id, source: "lifecycle" as const, @@ -91,7 +171,7 @@ describe("process trace specification", () => { ]); }); - it("retains duplicate unordered-pair diagnostics for reversed declarations", () => { + it("reports one duplicate-pair diagnostic per overlapping group", () => { const result = processTraceSpecificationSchema.safeParse({ events: ["z", "a", "b"].map((id) => ({ id, @@ -101,7 +181,10 @@ describe("process trace specification", () => { })), language: { kind: "partial_order", - unordered_groups: [{ events: ["z", "a", "b"] }, { events: ["a", "z"] }], + unordered_groups: [ + { events: ["z", "a", "b"] }, + { events: ["b", "a", "z"] }, + ], }, }); expect(result.success).toBe(false); @@ -113,27 +196,84 @@ describe("process trace specification", () => { }), ]); }); +}); - it("rejects cycles, implicit ordering gaps, invalid ignores, and unsatisfiable variants", () => { - const base = specification(); - expect( - processTraceSpecificationSchema.safeParse({ - ...base, - language: { - kind: "partial_order", - happens_before: [ - { before: "ready", after: "worker" }, - { before: "worker", after: "ready" }, - ], +describe("malformed process trace declarations", () => { + it("preserves independent diagnostics when event ids are duplicated", () => { + const result = processTraceSpecificationSchema.safeParse({ + events: [ + { + id: "a", + source: "lifecycle", + exact: { fixture: "first" }, + cardinality: { kind: "optional" }, }, - }).success, - ).toBe(false); - expect( - processTraceSpecificationSchema.safeParse({ - ...base, - language: { kind: "partial_order" }, - }).success, - ).toBe(false); + { + id: "a", + source: "lifecycle", + exact: { fixture: "second" }, + cardinality: { kind: "optional" }, + }, + ], + language: { + kind: "partial_order", + unordered_groups: [{ events: ["a", "b"] }], + }, + }); + expect(result.success).toBe(false); + if (result.success) throw new Error("Expected a duplicate event id"); + expect(result.error.issues).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + path: ["events", 1, "id"], + message: "event ids must be unique", + }), + expect.objectContaining({ + path: ["language", "unordered_groups", 0, "events", 1], + message: "unknown event id: b", + }), + ]), + ); + }); + + it("keeps reachability diagnostics for unknown referenced successors", () => { + const result = processTraceSpecificationSchema.safeParse({ + events: ["a", "b"].map((id) => ({ + id, + source: "lifecycle", + exact: { fixture: id }, + cardinality: { kind: "optional" }, + })), + language: { + kind: "partial_order", + happens_before: [{ before: "a", after: "missing" }], + not_before: [{ event: "a", anchor: "missing" }], + }, + }); + expect(result.success).toBe(false); + if (result.success) throw new Error("Expected an unknown event reference"); + expect(result.error.issues).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + path: ["language", "happens_before", 0, "after"], + message: "unknown event id: missing", + }), + expect.objectContaining({ + path: ["language", "not_before", 0, "anchor"], + message: "unknown event id: missing", + }), + expect.objectContaining({ + path: ["language", "not_before", 0], + message: "not-before constraint conflicts with happens-before order", + }), + ]), + ); + }); +}); + +describe("finite trace specification validation", () => { + it("rejects invalid ignores and unsatisfiable variants", () => { + const base = specification(); expect( processTraceSpecificationSchema.safeParse({ events: [ diff --git a/src/domain/process/processTraceSpecification.ts b/src/domain/process/processTraceSpecification.ts index 61ace7ca1..4e28774e0 100644 --- a/src/domain/process/processTraceSpecification.ts +++ b/src/domain/process/processTraceSpecification.ts @@ -9,7 +9,16 @@ import { processObservationSourceSchema, } from "./processObservation.js"; -const identifierSchema = z.string().regex(IDENTIFIER_PATTERN); +/** + * A declared trace, event or variant identifier. The specification owns this + * vocabulary and the comparison result echoes those identifiers verbatim, so + * the result module must reuse this schema rather than restate the pattern: + * two bounds on the same identifier would reject the result of an accepted + * specification. + */ +export const processTraceIdentifierSchema = z + .string() + .regex(IDENTIFIER_PATTERN); const cardinalitySchema = z.discriminatedUnion("kind", [ z.strictObject({ kind: z.literal("required") }), @@ -31,7 +40,7 @@ const cardinalitySchema = z.discriminatedUnion("kind", [ ]); const eventDeclarationSchema = z.strictObject({ - id: identifierSchema, + id: processTraceIdentifierSchema, source: processObservationSourceSchema, exact: jsonValueSchema, ignore_fields: z @@ -56,29 +65,31 @@ const partialOrderLanguageSchema = z.strictObject({ happens_before: z .array( z.strictObject({ - before: identifierSchema, - after: identifierSchema, + before: processTraceIdentifierSchema, + after: processTraceIdentifierSchema, }), ) .default([]), not_before: z .array( z.strictObject({ - event: identifierSchema, - anchor: identifierSchema, + event: processTraceIdentifierSchema, + anchor: processTraceIdentifierSchema, }), ) .default([]), unordered_groups: z - .array(z.strictObject({ events: z.array(identifierSchema).min(2) })) + .array( + z.strictObject({ events: z.array(processTraceIdentifierSchema).min(2) }), + ) .default([]), - prefix: z.array(identifierSchema).default([]), - suffix: z.array(identifierSchema).default([]), + prefix: z.array(processTraceIdentifierSchema).default([]), + suffix: z.array(processTraceIdentifierSchema).default([]), }); const finiteTraceTokenSchema = z.union([ - identifierSchema, - z.strictObject({ unordered: z.array(identifierSchema).min(2) }), + processTraceIdentifierSchema, + z.strictObject({ unordered: z.array(processTraceIdentifierSchema).min(2) }), ]); const finiteTraceLanguageSchema = z.strictObject({ @@ -86,7 +97,7 @@ const finiteTraceLanguageSchema = z.strictObject({ variants: z .array( z.strictObject({ - id: identifierSchema, + id: processTraceIdentifierSchema, trace: z.array(finiteTraceTokenSchema), }), ) @@ -106,6 +117,17 @@ export type ProcessTraceSpecification = z.infer< >; type PartialOrderLanguage = z.infer; +type AcyclicGraphFacts = + | { + readonly kind: "total-order"; + readonly rank: ReadonlyMap; + } + | { + readonly kind: "partial-order"; + readonly reverseAdjacency: ReadonlyMap>; + }; +type GraphFacts = { readonly kind: "cyclic" } | AcyclicGraphFacts; + export const canonicalTraceJson = (value: unknown): string => { const serialized = canonicalize(value); if (serialized === undefined) @@ -170,7 +192,6 @@ const validateNotBefore = ( language: PartialOrderLanguage, context: z.RefinementCtx, known: ReadonlySet, - reachable: ReadonlyMap>, ): ReadonlySet => { const pairs = new Set(); for (const [index, constraint] of language.not_before.entries()) { @@ -191,76 +212,101 @@ const validateNotBefore = ( path, }); pairs.add(key); - if (reachable.get(constraint.event)?.has(constraint.anchor) === true) - context.addIssue({ - code: "custom", - message: "not-before constraint conflicts with happens-before order", - path, - }); } return pairs; }; -const graphHasCycle = ( +const classifyGraph = ( ids: readonly string[], adjacency: ReadonlyMap>, -): boolean => { - const active = new Set(); - const complete = new Set(); - const visit = (current: string): boolean => { - if (active.has(current)) return true; - if (complete.has(current)) return false; - active.add(current); - const cyclic = [...(adjacency.get(current) ?? [])].some(visit); - active.delete(current); - complete.add(current); - return cyclic; - }; - return ids.some(visit); -}; - -const computeReachability = ( - ids: readonly string[], - adjacency: ReadonlyMap>, -): ReadonlyMap> => { - const reachable = new Map>(); - for (const origin of ids) { - const seen = new Set(); - const pending = [...(adjacency.get(origin) ?? [])]; - while (pending.length > 0) { - const current = pending.pop(); - if (current === undefined || seen.has(current)) continue; - seen.add(current); - pending.push(...(adjacency.get(current) ?? [])); +): GraphFacts => { + const incoming = new Map(ids.map((id) => [id, 0])); + let hasUnknownSuccessor = false; + for (const successors of adjacency.values()) + for (const successor of successors) { + const count = incoming.get(successor); + if (count === undefined) hasUnknownSuccessor = true; + else incoming.set(successor, count + 1); + } + const ready = [...incoming].flatMap(([id, count]) => + count === 0 ? [id] : [], + ); + const order: string[] = []; + let unique = true; + while (ready.length > 0) { + if (ready.length !== 1) unique = false; + const current = ready.pop(); + if (current === undefined) continue; + order.push(current); + for (const successor of adjacency.get(current) ?? []) { + const count = incoming.get(successor); + if (count === undefined) continue; + incoming.set(successor, count - 1); + if (count === 1) ready.push(successor); } - reachable.set(origin, seen); } - return reachable; + if (order.length !== incoming.size) return { kind: "cyclic" }; + if (unique && order.length === ids.length && !hasUnknownSuccessor) { + return { + kind: "total-order", + rank: new Map(order.map((id, index) => [id, index])), + }; + } + const reverseAdjacency = new Map( + [...incoming.keys()].map((id) => [id, new Set()]), + ); + for (const [before, successors] of adjacency) + for (const after of successors) reverseAdjacency.get(after)?.add(before); + return { kind: "partial-order", reverseAdjacency }; }; -const isOrdered = ( - reachable: ReadonlyMap>, - left: string, - right: string, -): boolean => - reachable.get(left)?.has(right) === true || - reachable.get(right)?.has(left) === true; - -type OrderingRelations = { - readonly reachable: ReadonlyMap>; - readonly notBeforePairs: ReadonlySet; +const collectReachable = ( + origin: string, + adjacency: ReadonlyMap>, +): ReadonlySet => { + const seen = new Set(); + const pending = [...(adjacency.get(origin) ?? [])]; + while (pending.length > 0) { + const current = pending.pop(); + if (current === undefined || seen.has(current)) continue; + seen.add(current); + for (const successor of adjacency.get(current) ?? []) + pending.push(successor); + } + return seen; }; const validateUnorderedGroups = ( language: PartialOrderLanguage, context: z.RefinementCtx, known: ReadonlySet, - relations: OrderingRelations, -): ReadonlySet => { - const pairs = new Set(); +): { + readonly groups: readonly ReadonlySet[]; + hasPair(left: string, right: string): boolean; + markDeclaredPairConflicts( + pairs: ReadonlySet, + conflicts: boolean[], + ): void; + markSharedGroups(left: string, right: string, conflicts: boolean[]): void; +} => { + const memberships = new Map>(); + const groups: Set[] = []; + const repeatedMemberships = new Map>(); for (const [groupIndex, group] of language.unordered_groups.entries()) { const path = ["language", "unordered_groups", groupIndex]; - if (new Set(group.events).size !== group.events.length) + const uniqueIds = new Set(); + const repeatedIds = new Set(); + for (const id of group.events) { + if (uniqueIds.has(id)) repeatedIds.add(id); + else uniqueIds.add(id); + } + groups.push(uniqueIds); + for (const id of repeatedIds) { + const groupIndexes = repeatedMemberships.get(id) ?? new Set(); + groupIndexes.add(groupIndex); + repeatedMemberships.set(id, groupIndexes); + } + if (uniqueIds.size !== group.events.length) context.addIssue({ code: "custom", message: "unordered groups must not repeat event ids", @@ -268,46 +314,136 @@ const validateUnorderedGroups = ( }); for (const [eventIndex, id] of group.events.entries()) addReferenceIssue(context, known, id, [...path, "events", eventIndex]); - for (const [leftIndex, leftId] of group.events.entries()) { - for ( - let rightIndex = leftIndex + 1; - rightIndex < group.events.length; - rightIndex++ - ) { - const rightId = group.events[rightIndex]; - if (rightId === undefined) continue; - const key = pairKey(leftId, rightId); - if (pairs.has(key)) - context.addIssue({ - code: "custom", - message: "unordered event pairs must be declared once", - path, - }); - pairs.add(key); - if ( - isOrdered(relations.reachable, leftId, rightId) || - relations.notBeforePairs.has(key) - ) - context.addIssue({ - code: "custom", - message: "ordered events cannot also be declared unordered", - path, - }); - } + const overlaps = new Map(); + for (const id of uniqueIds) { + const priorGroups = memberships.get(id) ?? new Set(); + for (const prior of priorGroups) + overlaps.set(prior, (overlaps.get(prior) ?? 0) + 1); + priorGroups.add(groupIndex); + memberships.set(id, priorGroups); } + if ([...overlaps.values()].some((count) => count > 1)) + context.addIssue({ + code: "custom", + message: "unordered event pairs must be declared once", + path, + }); } - return pairs; + const markSharedGroups = ( + left: string, + right: string, + conflicts: boolean[], + ): void => { + if (left === right) { + for (const groupIndex of repeatedMemberships.get(left) ?? []) + conflicts[groupIndex] = true; + return; + } + const rightGroups = memberships.get(right); + for (const group of memberships.get(left) ?? []) + if (rightGroups?.has(group) === true) conflicts[group] = true; + }; + return { + groups, + hasPair(left, right) { + if (left === right) return memberships.has(left); + const rightGroups = memberships.get(right); + for (const group of memberships.get(left) ?? []) + if (rightGroups?.has(group) === true) return true; + return false; + }, + markDeclaredPairConflicts(pairs, conflicts) { + for (const key of pairs) { + const separator = key.indexOf("\0"); + const left = key.slice(0, separator); + const right = key.slice(separator + 1); + markSharedGroups(left, right, conflicts); + } + }, + markSharedGroups, + }; }; -const validateCompletePairDeclarations = ( +const validateTotalOrderRelations = ( + language: PartialOrderLanguage, + graph: Extract, + unorderedGroups: ReturnType, + notBeforePairs: ReadonlySet, + context: z.RefinementCtx, +): void => { + for (const [index, constraint] of language.not_before.entries()) { + const before = graph.rank.get(constraint.event); + const after = graph.rank.get(constraint.anchor); + if (before !== undefined && after !== undefined && before < after) + context.addIssue({ + code: "custom", + message: "not-before constraint conflicts with happens-before order", + path: ["language", "not_before", index], + }); + } + + const conflicts = new Array(language.unordered_groups.length).fill( + false, + ); + for (const [groupIndex, group] of unorderedGroups.groups.entries()) { + let comparableEvents = 0; + for (const id of group) + if (graph.rank.has(id) && ++comparableEvents === 2) { + conflicts[groupIndex] = true; + break; + } + } + unorderedGroups.markDeclaredPairConflicts(notBeforePairs, conflicts); + addUnorderedConflictIssues(conflicts, context); +}; + +const addUnorderedConflictIssues = ( + conflicts: readonly boolean[], + context: z.RefinementCtx, +): void => { + for (const [groupIndex, conflict] of conflicts.entries()) + if (conflict) + context.addIssue({ + code: "custom", + message: "ordered events cannot also be declared unordered", + path: ["language", "unordered_groups", groupIndex], + }); +}; + +const validatePartialOrderRelations = ( ids: readonly string[], - relations: OrderingRelations, - unorderedPairs: ReadonlySet, + language: PartialOrderLanguage, + adjacency: ReadonlyMap>, + graph: Extract, + unorderedGroups: ReturnType, + notBeforePairs: ReadonlySet, context: z.RefinementCtx, ): void => { let missingPairCount = 0; let firstMissingPair = ""; + const groupConflicts = new Array( + language.unordered_groups.length, + ).fill(false); + unorderedGroups.markDeclaredPairConflicts(notBeforePairs, groupConflicts); + const notBeforeConflictFlags = new Uint8Array(language.not_before.length); + const constraintsBySource = new Map(); + for (const [index, constraint] of language.not_before.entries()) { + const indexes = constraintsBySource.get(constraint.event) ?? []; + indexes.push(index); + constraintsBySource.set(constraint.event, indexes); + } + + // Keep only one origin's forward and backward closure live at a time. Pair + // iteration stays in declaration order so the count and first missing pair + // retain the established diagnostic semantics. for (const [leftIndex, leftId] of ids.entries()) { + const forward = collectReachable(leftId, adjacency); + const backward = collectReachable(leftId, graph.reverseAdjacency); + for (const constraintIndex of constraintsBySource.get(leftId) ?? []) { + const constraint = language.not_before[constraintIndex]; + if (constraint !== undefined && forward.has(constraint.anchor)) + notBeforeConflictFlags[constraintIndex] = 1; + } for ( let rightIndex = leftIndex + 1; rightIndex < ids.length; @@ -315,14 +451,32 @@ const validateCompletePairDeclarations = ( ) { const rightId = ids[rightIndex]; if (rightId === undefined) continue; - if (isOrdered(relations.reachable, leftId, rightId)) continue; + const sameEvent = leftId === rightId; + const ordered = + !sameEvent && (forward.has(rightId) || backward.has(rightId)); const key = pairKey(leftId, rightId); - if (!relations.notBeforePairs.has(key) && !unorderedPairs.has(key)) { + const notBefore = notBeforePairs.has(key); + const unordered = unorderedGroups.hasPair(leftId, rightId); + if (ordered || notBefore) { + if (!sameEvent) + unorderedGroups.markSharedGroups(leftId, rightId, groupConflicts); + } + if (!ordered && !notBefore && !unordered) { if (missingPairCount === 0) firstMissingPair = `${leftId}/${rightId}`; missingPairCount++; } } } + + for (const [index, conflict] of notBeforeConflictFlags.entries()) + if (conflict === 1) + context.addIssue({ + code: "custom", + message: "not-before constraint conflicts with happens-before order", + path: ["language", "not_before", index], + }); + addUnorderedConflictIssues(groupConflicts, context); + if (missingPairCount > 0) context.addIssue({ code: "custom", @@ -342,27 +496,44 @@ const validatePartialOrder = ( const ids = specification.events.map(({ id }) => id); const known = new Set(ids); const adjacency = buildAdjacency(specification.language, context, known); - if (graphHasCycle(ids, adjacency)) + const graph = classifyGraph(ids, adjacency); + if (graph.kind === "cyclic") context.addIssue({ code: "custom", message: "happens-before graph must be acyclic", path: ["language"], }); - const reachable = computeReachability(ids, adjacency); const notBeforePairs = validateNotBefore( specification.language, context, known, - reachable, ); - const relations = { reachable, notBeforePairs }; - const unorderedPairs = validateUnorderedGroups( + const unorderedGroups = validateUnorderedGroups( specification.language, context, known, - relations, ); - validateCompletePairDeclarations(ids, relations, unorderedPairs, context); + // Preserve independent declaration diagnostics on cyclic input. Computing + // all pair relations cannot establish a valid partial order in that case. + if (graph.kind === "total-order") { + validateTotalOrderRelations( + specification.language, + graph, + unorderedGroups, + notBeforePairs, + context, + ); + } else if (graph.kind === "partial-order") { + validatePartialOrderRelations( + ids, + specification.language, + adjacency, + graph, + unorderedGroups, + notBeforePairs, + context, + ); + } for (const [kind, values] of [ ["prefix", specification.language.prefix], ["suffix", specification.language.suffix], diff --git a/src/domain/qlogCapture.ts b/src/domain/qlogCapture.ts new file mode 100644 index 000000000..575cf2a60 --- /dev/null +++ b/src/domain/qlogCapture.ts @@ -0,0 +1,76 @@ +import { z } from "zod"; + +const decimal = z.string().regex(/^(0|[1-9][0-9]*)$/u); +const unsignedFact = z.discriminatedUnion("state", [ + z.strictObject({ state: z.literal("reported"), value: decimal }), + z.strictObject({ + state: z.literal("unknown"), + reason: z.literal("omitted-null-or-redacted"), + }), +]); +const traceLocation = z.discriminatedUnion("kind", [ + z.strictObject({ kind: z.literal("json-pointer"), pointer: z.string() }), + z.strictObject({ + kind: z.literal("unknown"), + reason: z.literal("explicit-sensitive-value"), + }), +]); + +/** Exact, source-bound transport facts; coverage never establishes network completeness. */ +export const qlogCaptureFactsSchema = z.strictObject({ + profile: z.literal("aioquic-1.5.0-json-0.3"), + scope: z.literal("all-recorded-events-before-selection"), + body_bytes: z.literal("not-reported"), + clock: z.strictObject({ + unit: z.literal("milliseconds"), + domain: z.literal("unix-epoch"), + basis: z.literal("selected-aioquic-profile-not-producer-authentication"), + }), + traces: z.array( + z.strictObject({ + trace_index: z.number().int().nonnegative(), + location: traceLocation, + event_count: z.number().int().nonnegative(), + odcid: z.string().nullable(), + vantage_point: z.enum(["client", "server"]).nullable(), + incomplete_fact_records: z.array(z.number().int().nonnegative()), + loss_declarations: z.array( + z.strictObject({ + record_ordinal: z.number().int().nonnegative(), + confidence: z.literal("observed"), + packet_number: unsignedFact, + packet_type: z.string().nullable(), + packet_number_space: z + .enum(["initial", "handshake", "application"]) + .nullable(), + }), + ), + streams: z.array( + z.strictObject({ + stream_id: decimal, + direction: z.enum(["sent", "received"]), + confidence: z.literal("derived"), + scope: z.literal("observed-stream-frame-ranges"), + contributions: z.array( + z.strictObject({ + record_ordinal: z.number().int().nonnegative(), + frame_index: z.number().int().nonnegative(), + offset: decimal, + length: decimal, + fin: z.boolean().nullable(), + }), + ), + covered_ranges: z.array( + z.strictObject({ start: decimal, end: decimal }), + ), + unique_bytes: decimal, + frame_length_sum: decimal, + repeated_bytes: decimal, + reported_final_sizes: z.array(decimal), + }), + ), + }), + ), +}); + +export type QlogCaptureFacts = z.output; diff --git a/src/domain/rawBinary.test.ts b/src/domain/rawBinary.test.ts index f22e5fced..5cb6ed129 100644 --- a/src/domain/rawBinary.test.ts +++ b/src/domain/rawBinary.test.ts @@ -39,6 +39,44 @@ describe("raw-binary entry addresses", () => { }); describe("raw-binary admission", () => { + it("admits byte-aligned 6502 images up to the 16-bit address boundary", () => { + for (const [entryPoint, fileBytes] of [ + ["0x8001", 8], + ["0xffff", 1], + ["0x0", 65536], + ] as const) { + const result = admitRawBinary({ + architecture: "6502", + entryPoint, + fileBytes, + }); + if (!result.ok) throw new Error(result.error.reason); + expect(result.value).toEqual({ + architecture: "6502", + entryPoint, + entryPointSource: entryPoint, + }); + } + }); + it.each([ + ["0x10000", 1], + ["0xffff", 2], + ["0x0", 65537], + ])( + "rejects a 6502 mapping at %s with %i bytes that exceeds 16 bits", + (entryPoint, fileBytes) => { + const result = admitRawBinary({ + architecture: "6502", + entryPoint, + fileBytes, + }); + if (result.ok) throw new Error("expected 16-bit overflow rejection"); + expect(result.error).toEqual({ + reason: `raw-binary mapping from ${entryPoint} must fit in the 6502 address width`, + constraint: "load_selection_rejected", + }); + }, + ); it("admits a little-endian ARM image mapped at the caller entry", () => { expect( admitRawBinary({ diff --git a/src/domain/rawBinary.ts b/src/domain/rawBinary.ts index 94fdf6ce7..22d94f4d6 100644 --- a/src/domain/rawBinary.ts +++ b/src/domain/rawBinary.ts @@ -13,6 +13,7 @@ export const BINARY_ARCHITECTURES = [ "x86_64", "arm", "arm64", + "6502", ] as const satisfies readonly BinaryArchitecture[]; /** Provider-neutral CPU family for a caller-supplied raw-binary load. */ @@ -58,7 +59,7 @@ export const validateRawBinaryLength = ( /** Admitted headerless load identity used by target resolution and profiles. */ export interface RawBinaryAdmission { - readonly architecture: HeaderArchitecture; + readonly architecture: HeaderArchitecture | "6502"; readonly entryPoint: string; readonly entryPointSource: string; } @@ -74,9 +75,11 @@ export interface RawBinaryAdmissionRejection { } const addressWidth = (architecture: BinaryArchitecture): bigint => - architecture === "x86" || architecture === "arm" - ? 0xffffffffn - : 0xffffffffffffffffn; + architecture === "6502" + ? 0xffffn + : architecture === "x86" || architecture === "arm" + ? 0xffffffffn + : 0xffffffffffffffffn; /** Validate architecture, entry, and that the mapped image fits the CPU width. */ export const admitRawBinary = (input: { @@ -88,7 +91,7 @@ export const admitRawBinary = (input: { if (!architecture.success) return err({ reason: - "raw-binary format requires architecture (x86, x86_64, arm, or arm64)", + "raw-binary format requires architecture (x86, x86_64, arm, arm64, or 6502)", constraint: "load_selection_rejected", }); if (typeof input.entryPoint !== "string") diff --git a/src/domain/reconstructionCoverage.test.ts b/src/domain/reconstructionCoverage.test.ts index 4c0b41831..ac7552571 100644 --- a/src/domain/reconstructionCoverage.test.ts +++ b/src/domain/reconstructionCoverage.test.ts @@ -286,14 +286,18 @@ describe("reconstruction coverage verifier selection", () => { }); }); -describe("reconstruction coverage verifier result ordering", () => { - it("preserves the first input result when verifier timestamps tie", () => { +describe("reconstruction coverage conflicting latest verifier observations", () => { + it("keeps conflicting tied assessments unknown regardless of input order", () => { const workspace = completeReconstructionCoverageData(); const result = workspace.verifier_results[0]; const boundary = workspace.boundaries[0]; if (result === undefined || boundary === undefined) throw new Error("Expected complete reconstruction coverage fixture"); - const failed = { ...result, status: "fail" as const }; + const failed = { + ...result, + status: "fail" as const, + evidence_ids: [reconstructionCoverageEvidenceId("1")], + }; const firstPass = createReconstructionCoverageData({ ...workspace, verifier_results: [result, failed], @@ -303,23 +307,109 @@ describe("reconstruction coverage verifier result ordering", () => { verifier_results: [failed, result], }); + const firstPassResult = evaluateReconstructionClosure( + firstPass, + boundary.boundary_id, + RECONSTRUCTION_COVERAGE_NOW, + ); + const firstFailResult = evaluateReconstructionClosure( + firstFail, + boundary.boundary_id, + RECONSTRUCTION_COVERAGE_NOW, + ); + expect(firstPassResult).toEqual(firstFailResult); + expect(firstPassResult).toMatchObject({ + status: "unknown", + reasons: [ + expect.objectContaining({ + code: "verifier-result-ambiguous", + subject_id: "claim.cli.help", + }), + ], + }); + expect(firstPassResult.evidence_ids).toEqual( + expect.arrayContaining([ + reconstructionCoverageEvidenceId("3"), + reconstructionCoverageEvidenceId("1"), + ]), + ); + }); +}); + +describe("reconstruction coverage equivalent latest verifier observations", () => { + it("coalesces equivalent equal-instant observations and retains all evidence", () => { + const workspace = completeReconstructionCoverageData(); + const result = workspace.verifier_results[0]; + const boundary = workspace.boundaries[0]; + if (result === undefined || boundary === undefined) + throw new Error("Expected complete reconstruction coverage fixture"); + const first = { + ...result, + observed_at: "2026-07-16T11:00:00.123456789012345678900Z", + }; + const equivalent = { + ...first, + observed_at: "2026-07-16T06:00:00.1234567890123456789-05:00", + covered_claim_ids: [...result.covered_claim_ids].reverse(), + covered_dimensions: [...result.covered_dimensions].reverse(), + artifact_sha256s: [...result.artifact_sha256s].reverse(), + owner_sha256s: [...result.owner_sha256s].reverse(), + evidence_ids: [reconstructionCoverageEvidenceId("1")], + }; + const forward = createReconstructionCoverageData({ + ...workspace, + verifier_results: [first, equivalent], + }); + const reversed = createReconstructionCoverageData({ + ...workspace, + verifier_results: [equivalent, first], + }); + const forwardResult = evaluateReconstructionClosure( + forward, + boundary.boundary_id, + RECONSTRUCTION_COVERAGE_NOW, + ); + const reversedResult = evaluateReconstructionClosure( + reversed, + boundary.boundary_id, + RECONSTRUCTION_COVERAGE_NOW, + ); + + expect(forwardResult).toEqual(reversedResult); + expect(forwardResult.status).toBe("ready"); + expect(forwardResult.evidence_ids).toEqual( + expect.arrayContaining([ + reconstructionCoverageEvidenceId("3"), + reconstructionCoverageEvidenceId("1"), + ]), + ); + }); +}); + +describe("reconstruction coverage verifier timestamp selection", () => { + it("lets a later unambiguous observation supersede an older tie", () => { + const workspace = completeReconstructionCoverageData(); + const result = workspace.verifier_results[0]; + const boundary = workspace.boundaries[0]; + if (result === undefined || boundary === undefined) + throw new Error("Expected complete reconstruction coverage fixture"); + const failed = { ...result, status: "fail" as const }; + const latestPass = { + ...result, + observed_at: "2026-07-16T11:30:00.000Z", + }; + const changed = createReconstructionCoverageData({ + ...workspace, + verifier_results: [result, failed, latestPass], + }); + expect( evaluateReconstructionClosure( - firstPass, - boundary.boundary_id, - RECONSTRUCTION_COVERAGE_NOW, - ).status, - ).toBe("ready"); - expect( - evaluateReconstructionClosure( - firstFail, + changed, boundary.boundary_id, RECONSTRUCTION_COVERAGE_NOW, ), - ).toMatchObject({ - status: "failed", - reasons: [expect.objectContaining({ code: "verifier-failed" })], - }); + ).toMatchObject({ status: "ready", reasons: [] }); }); it("orders offset-bearing verifier timestamps chronologically", () => { @@ -352,6 +442,68 @@ describe("reconstruction coverage verifier result ordering", () => { }); }); + it("selects the later result when timestamps differ below millisecond precision", () => { + const workspace = completeReconstructionCoverageData(); + const result = workspace.verifier_results[0]; + if (result === undefined) + throw new Error("Expected fixture verifier result"); + const changed = createReconstructionCoverageData({ + ...workspace, + verifier_results: [ + { ...result, observed_at: "2026-07-16T11:00:00.1231Z" }, + { + ...result, + observed_at: "2026-07-16T11:00:00.1239Z", + status: "fail", + }, + ], + }); + + expect( + evaluateReconstructionClosure( + changed, + "replacement.cli", + RECONSTRUCTION_COVERAGE_NOW, + ), + ).toMatchObject({ + status: "failed", + reasons: [expect.objectContaining({ code: "verifier-failed" })], + }); + }); +}); + +describe("reconstruction coverage verifier future timestamps", () => { + it("treats an observation beyond millisecond-precision now as future", () => { + const workspace = completeReconstructionCoverageData(); + const result = workspace.verifier_results[0]; + if (result === undefined) + throw new Error("Expected fixture verifier result"); + const changed = createReconstructionCoverageData({ + ...workspace, + verifier_results: [ + { ...result, observed_at: "2026-07-16T12:00:00.1239Z" }, + ], + }); + + expect( + evaluateReconstructionClosure( + changed, + "replacement.cli", + Date.parse("2026-07-16T12:00:00.123Z") + 0.5, + ), + ).toMatchObject({ + status: "unknown", + reasons: [expect.objectContaining({ code: "verifier-result-stale" })], + }); + }); + + it("rejects a non-finite evaluation clock", () => { + const workspace = completeReconstructionCoverageData(); + expect(() => + evaluateReconstructionClosure(workspace, "replacement.cli", Number.NaN), + ).toThrow("Expected finite epoch milliseconds."); + }); + it("rejects verifier observations from the future", () => { const workspace = completeReconstructionCoverageData(); const result = workspace.verifier_results[0]; diff --git a/src/domain/reconstructionCoverage.ts b/src/domain/reconstructionCoverage.ts index 485b66a5a..04913d4ae 100644 --- a/src/domain/reconstructionCoverage.ts +++ b/src/domain/reconstructionCoverage.ts @@ -19,7 +19,10 @@ import { } from "./reconstructionCoverageEvaluation.js"; import { digestSchema } from "./../domain/digests.js"; import { prefixedDigestSchema } from "./../domain/digests.js"; -import { offsetDateTimeSchema } from "./offsetDateTime.js"; +import { + offsetDateTimeInstantFromEpochMilliseconds, + offsetDateTimeSchema, +} from "./offsetDateTime.js"; const evidenceIdSchema = prefixedDigestSchema("ev"); const stableIdSchema = z @@ -213,6 +216,7 @@ const closureReasonSchema = z.strictObject({ "verifier-result-missing", "verifier-result-stale", "verifier-result-incompatible", + "verifier-result-ambiguous", "verifier-failed", "verifier-unknown", "active-unknown", @@ -274,10 +278,12 @@ export const evaluateReconstructionClosure = ( ); if (boundary === undefined) throw new TypeError(`Unknown reconstruction boundary: ${boundaryId}`); + const nowInstant = offsetDateTimeInstantFromEpochMilliseconds(nowEpochMs); const context: ReconstructionEvaluationContext = { coverage, boundary, nowEpochMs, + nowInstant, reasons: [], evidenceIds: new Set(), }; @@ -394,6 +400,7 @@ const closureStatus = ( [ "verifier-result-stale", "verifier-result-incompatible", + "verifier-result-ambiguous", "verifier-unknown", "active-unknown", "owner-path-unknown", diff --git a/src/domain/reconstructionCoverageEvaluation.ts b/src/domain/reconstructionCoverageEvaluation.ts index ea82e5636..f8a56b261 100644 --- a/src/domain/reconstructionCoverageEvaluation.ts +++ b/src/domain/reconstructionCoverageEvaluation.ts @@ -3,6 +3,16 @@ import type { ReconstructionCoverageData, ReconstructionVerifierContract, } from "./reconstructionCoverage.js"; +import { + compareOffsetDateTimeInstants, + offsetDateTimeInstantFromEpochMilliseconds, + type OffsetDateTimeInstant, +} from "./offsetDateTime.js"; +import { + latestVerifierObservationsByVerifier, + sameVerifierAssessment, + type VerifierObservation, +} from "./reconstructionCoverageVerifierResults.js"; export const recommendedReconstructionProbes = ( reasons: readonly ClosureReason[], @@ -29,6 +39,7 @@ export interface ReconstructionEvaluationContext { readonly coverage: ReconstructionCoverageData; readonly boundary: Boundary; readonly nowEpochMs: number; + readonly nowInstant: OffsetDateTimeInstant; readonly reasons: ClosureReason[]; readonly evidenceIds: Set; } @@ -184,7 +195,7 @@ interface VerifierIndexes { >; readonly latestResultsByVerifier: ReadonlyMap< string, - ReconstructionCoverageData["verifier_results"][number] + readonly VerifierObservation[] >; readonly artifactDigests: ReadonlySet; readonly ownerDigests: ReadonlySet; @@ -213,20 +224,6 @@ const createVerifierIndexes = ( contractsByClaim.set(claimId, contracts); } - const latestResultsByVerifier = new Map< - string, - ReconstructionCoverageData["verifier_results"][number] - >(); - for (const result of coverage.verifier_results) { - const latest = latestResultsByVerifier.get(result.verifier_id); - // Stable descending sort used to choose the first input record on ties. - if ( - latest === undefined || - Date.parse(result.observed_at) > Date.parse(latest.observed_at) - ) - latestResultsByVerifier.set(result.verifier_id, result); - } - const ownersBySurface = new Map< string, ReconstructionCoverageData["owners"][number][] @@ -239,7 +236,9 @@ const createVerifierIndexes = ( return { contractsByClaim, - latestResultsByVerifier, + latestResultsByVerifier: latestVerifierObservationsByVerifier( + coverage.verifier_results, + ), artifactDigests: new Set( coverage.artifacts.map(({ artifact_sha256: value }) => value), ), @@ -260,8 +259,10 @@ const evaluateVerifierResult = ( contract: ReconstructionVerifierContract, indexes: VerifierIndexes, ): void => { - const result = indexes.latestResultsByVerifier.get(contract.verifier_id); - if (result === undefined) { + const latestResults = indexes.latestResultsByVerifier.get( + contract.verifier_id, + ); + if (latestResults === undefined) { addReason( context, "verifier-result-missing", @@ -270,7 +271,26 @@ const evaluateVerifierResult = ( ); return; } - addEvidence(context, result.evidence_ids); + addEvidence( + context, + latestResults.flatMap(({ result }) => result.evidence_ids), + ); + const latest = latestResults[0]; + if (latest === undefined) return; + const { result } = latest; + if ( + !latestResults.every((candidate) => + sameVerifierAssessment(result, candidate.result), + ) + ) { + addReason( + context, + "verifier-result-ambiguous", + claim.claim_id, + "Latest verifier observations have conflicting assessment content.", + ); + return; + } if (!verifierResultIsCompatible(claim, contract, result, indexes)) { addReason( context, @@ -280,10 +300,14 @@ const evaluateVerifierResult = ( ); return; } - const observedAt = Date.parse(result.observed_at); + const cutoffEpochMs = context.nowEpochMs - contract.max_age_ms; if ( - observedAt > context.nowEpochMs || - context.nowEpochMs - observedAt > contract.max_age_ms + compareOffsetDateTimeInstants(latest.instant, context.nowInstant) > 0 || + (cutoffEpochMs >= -Number.MAX_SAFE_INTEGER && + compareOffsetDateTimeInstants( + latest.instant, + offsetDateTimeInstantFromEpochMilliseconds(cutoffEpochMs), + ) < 0) ) { addReason( context, diff --git a/src/domain/reconstructionCoverageVerifierResults.ts b/src/domain/reconstructionCoverageVerifierResults.ts new file mode 100644 index 000000000..de7f55874 --- /dev/null +++ b/src/domain/reconstructionCoverageVerifierResults.ts @@ -0,0 +1,57 @@ +import type { ReconstructionCoverageData } from "./reconstructionCoverage.js"; +import { uniqueSorted } from "./canonicalOrdering.js"; +import { + compareOffsetDateTimeInstants, + parseOffsetDateTimeInstant, + type OffsetDateTimeInstant, +} from "./offsetDateTime.js"; + +type VerifierResult = ReconstructionCoverageData["verifier_results"][number]; + +export interface VerifierObservation { + readonly result: VerifierResult; + readonly instant: OffsetDateTimeInstant; +} + +/** Index each verifier's latest timestamp group without relying on array order. */ +export const latestVerifierObservationsByVerifier = ( + results: readonly VerifierResult[], +): ReadonlyMap => { + const latestByVerifier = new Map(); + for (const result of results) { + const observation = { + result, + instant: parseOffsetDateTimeInstant(result.observed_at), + }; + const latest = latestByVerifier.get(result.verifier_id); + const latestInstant = latest?.[0]?.instant; + const comparison = + latestInstant === undefined + ? 1 + : compareOffsetDateTimeInstants(observation.instant, latestInstant); + if (latest === undefined || comparison > 0) + latestByVerifier.set(result.verifier_id, [observation]); + else if (comparison === 0) latest.push(observation); + } + return latestByVerifier; +}; + +/** Compare assessment content while excluding timestamp spelling and provenance. */ +export const sameVerifierAssessment = ( + left: VerifierResult, + right: VerifierResult, +): boolean => + verifierAssessmentIdentity(left) === verifierAssessmentIdentity(right); + +const verifierAssessmentIdentity = (result: VerifierResult): string => + JSON.stringify({ + verifier_id: result.verifier_id, + contract_sha256: result.contract_sha256, + status: result.status, + covered_claim_ids: uniqueSorted(result.covered_claim_ids), + covered_dimensions: uniqueSorted(result.covered_dimensions), + artifact_sha256s: uniqueSorted(result.artifact_sha256s), + owner_sha256s: uniqueSorted(result.owner_sha256s), + normalization_sha256: result.normalization_sha256, + repeats: result.repeats, + }); diff --git a/src/domain/referenceSourceImportParsing.ts b/src/domain/referenceSourceImportParsing.ts index 998e368c4..18f76a358 100644 --- a/src/domain/referenceSourceImportParsing.ts +++ b/src/domain/referenceSourceImportParsing.ts @@ -11,6 +11,7 @@ import { parserPluginsForPath, } from "./javascript/javascriptSourceParser.js"; import { unshadowedReferenceSourceRequireCalls } from "./referenceSourceRequireBindings.js"; +import { isRelativeJavaScriptModuleSpecifier } from "./javascript/javascriptModuleSpecifier.js"; import { isCallExpression, isExportAllDeclaration, @@ -60,7 +61,8 @@ export interface ReferenceSourceImportParseResult { readonly limitations: readonly string[]; } -const codeSourceLanguages = new Set(["JavaScript", "TypeScript", "JSX", "TSX"]); +export const PARSEABLE_REFERENCE_SOURCE_LANGUAGES: ReadonlySet = + new Set(["JavaScript", "TypeScript", "JSX", "TSX"]); const safeModuleName = (node: Node | null | undefined): string | undefined => { if (isStringLiteral(node)) return node.value; @@ -70,9 +72,11 @@ const safeModuleName = (node: Node | null | undefined): string | undefined => { const resolveSpecifier = ( specifier: string, ): ReferenceSourceImportResolution => { - if (specifier.startsWith(".") || specifier.startsWith("/")) return "internal"; - if (specifier.startsWith("node:") || specifier.startsWith("data:")) - return "external"; + if ( + isRelativeJavaScriptModuleSpecifier(specifier) || + specifier.startsWith("/") + ) + return "internal"; return "external"; }; @@ -314,7 +318,7 @@ export const parseReferenceSourceImports = ( bytes: Uint8Array, language: string | null, ): ReferenceSourceImportParseResult => { - if (language !== null && !codeSourceLanguages.has(language)) + if (language !== null && !PARSEABLE_REFERENCE_SOURCE_LANGUAGES.has(language)) return { relationships: [], parse_failures: [], limitations: [] }; let source: string; diff --git a/src/domain/residualUnknown.ts b/src/domain/residualUnknown.ts index 0819a3964..e5db1cc54 100644 --- a/src/domain/residualUnknown.ts +++ b/src/domain/residualUnknown.ts @@ -12,7 +12,7 @@ import { functionComparisonDimensionNameSchema } from "./functionComparisonSchem const evidenceIdSchema = prefixedDigestSchema("ev"); const unknownIdSchema = prefixedDigestSchema("unk"); const text = z.string().trim().min(1); -const environmentRequirementSchema = z.object({ +const environmentRequirementSchema = z.strictObject({ id: z.string().min(1).nullable(), platform: z.string().min(1).nullable(), architecture: z.string().min(1).nullable(), @@ -39,7 +39,7 @@ const stableUnknownId = ( ) .digest("hex")}`; -const probeSchema = z.object({ +const probeSchema = z.strictObject({ operation: z.string().min(1), rationale: text, }); @@ -60,17 +60,17 @@ const comparisonConclusionScopeSchema = z message: "Comparison dimensions must be sorted and unique", }); }); -const relationshipSchema = z.object({ +const relationshipSchema = z.strictObject({ type: z.enum(["depends-on", "related-to", "contradicts"]), unknown_id: unknownIdSchema, }); const resolutionSchema = z.discriminatedUnion("disposition", [ - z.object({ + z.strictObject({ disposition: z.literal("verified"), rationale: text, evidence_ids: z.array(evidenceIdSchema).min(1), }), - z.object({ + z.strictObject({ disposition: z.enum(["withdrawn", "out-of-scope"]), rationale: text, evidence_ids: z.array(evidenceIdSchema), @@ -78,7 +78,7 @@ const resolutionSchema = z.discriminatedUnion("disposition", [ ]); /** Strict durable record for one unresolved or evidence-qualified question. */ -const residualUnknownObjectSchema = z.object({ +const residualUnknownObjectSchema = z.strictObject({ unknown_id: unknownIdSchema, revision: z.number().int().min(1), previous_revision_digest: z diff --git a/src/domain/signals/signalDecoders.ts b/src/domain/signals/signalDecoders.ts new file mode 100644 index 000000000..ae6754cca --- /dev/null +++ b/src/domain/signals/signalDecoders.ts @@ -0,0 +1,86 @@ +import { z } from "zod"; +import { signalRecordingSchema } from "./signalRecording.js"; + +export const inspectSignalDecodersInputSchema = z.strictObject({ + decoder: z + .string() + .regex(/^[a-zA-Z0-9_+\x2d]+$/u) + .max(256) + .nullable() + .default(null) + .describe( + "Exact installed decoder ID for help; null searches the decoder catalog", + ), + query: z + .string() + .max(8192) + .default("") + .describe( + "Case-insensitive literal substring in decoder ID/name or help lines; empty selects all", + ), + offset: z + .number() + .int() + .nonnegative() + .max(Number.MAX_SAFE_INTEGER) + .default(0) + .describe( + "Number of matching decoders or help lines to skip; defaults to 0", + ), + limit: z + .number() + .int() + .positive() + .max(Number.MAX_SAFE_INTEGER) + .nullable() + .default(50) + .describe( + "Maximum matching decoders or help lines returned; null requests all remaining", + ), +}); +export type InspectSignalDecodersInput = z.infer< + typeof inspectSignalDecodersInputSchema +>; +export const signalDecodersSchema = z + .strictObject({ + selection: inspectSignalDecodersInputSchema, + decoders: z.array(z.strictObject({ id: z.string(), name: z.string() })), + help_lines: z.array( + z.strictObject({ + line_number: z.number().int().positive(), + text: z.string(), + }), + ), + output_window: z.strictObject({ + unit: z.enum(["decoders", "help-lines"]), + examined: z.number().int().nonnegative(), + matching: z.number().int().nonnegative(), + returned: z.number().int().nonnegative(), + has_more: z.boolean(), + }), + diagnostics: z.string(), + tool: signalRecordingSchema.shape.tool, + resource_limiter: signalRecordingSchema.shape.resource_limiter, + limits: signalRecordingSchema.shape.limits, + hardware_acquisition: z.literal("not-performed"), + }) + .superRefine((report, context) => { + const w = report.output_window; + const catalog = report.selection.decoder === null; + const remaining = Math.max(0, w.matching - report.selection.offset); + const expected = Math.min(remaining, report.selection.limit ?? remaining); + if ( + w.unit !== (catalog ? "decoders" : "help-lines") || + w.matching > w.examined || + w.returned !== expected || + (catalog ? report.decoders : report.help_lines).length !== expected || + (catalog ? report.help_lines : report.decoders).length !== 0 || + w.has_more !== remaining > expected + ) + context.addIssue({ + code: "custom", + path: ["output_window"], + message: + "Discovery coverage must match the selected page and result kind", + }); + }); diff --git a/src/domain/wasm/wasmArtifact.ts b/src/domain/wasm/wasmArtifact.ts index 254a55a3a..d8c89ddf3 100644 --- a/src/domain/wasm/wasmArtifact.ts +++ b/src/domain/wasm/wasmArtifact.ts @@ -1,5 +1,6 @@ import { z } from "zod"; import { createHash } from "node:crypto"; +import type { EvidenceSubjectTarget } from "../evidence.js"; import { localPathStringSchema } from "../localPath.js"; @@ -137,11 +138,20 @@ export type InspectWasmArtifactInput = z.infer< >; export type WasmArtifact = z.infer; -/** A WASM observation must identify the exact selected path and byte digest. */ +/** A WASM observation must identify the selected file path and byte digest. */ +export type WasmArtifactSubject = Pick< + EvidenceSubjectTarget, + "path" | "sha256" | "format" +>; +type WasmArtifactFileSubject = WasmArtifactSubject & { + readonly format: "file"; +}; + export const wasmArtifactMatchesSubject = ( artifact: WasmArtifact["artifact"], - subject: Pick | null | undefined, -): subject is Pick => + subject: WasmArtifactSubject | null | undefined, +): subject is WasmArtifactFileSubject => subject != null && subject.path === artifact.path && - subject.sha256 === artifact.sha256; + subject.sha256 === artifact.sha256 && + subject.format === "file"; diff --git a/src/domain/webModuleImports.test.ts b/src/domain/webModuleImports.test.ts index 26db977a1..c0bcabdbf 100644 --- a/src/domain/webModuleImports.test.ts +++ b/src/domain/webModuleImports.test.ts @@ -25,11 +25,18 @@ describe("native module syntax observations", () => { item.expression, ); }); - it("excludes erased TypeScript imports/exports and bundler inference", () => { + it("excludes whole type declarations and retains inline type dependency syntax", () => { const result = collectWebModuleImports( 'import type {T} from "types"; import {type X} from "also-types"; export type {T} from "export-types"; export {type Y} from "only-types"; import {type T, value} from "mixed"; __webpack_require__.e(4); __vitePreload(() => 1, ["chunk.js"]);', ); - expect(result.imports.map((item) => item.specifier)).toEqual(["mixed"]); + expect(result.imports.map((item) => item.specifier)).toEqual([ + "also-types", + "only-types", + "mixed", + ]); + expect(result.diagnostics).toContain( + "Compiler options are unavailable; inline type-only imports or re-exports may preserve a module side-effect dependency.", + ); }); it("uses UTF-16 source locations after astral characters", () => { const source = 'const emoji = "😀";\nimport "./x.js";'; diff --git a/src/domain/webModuleImports.ts b/src/domain/webModuleImports.ts index 026868955..0f9846871 100644 --- a/src/domain/webModuleImports.ts +++ b/src/domain/webModuleImports.ts @@ -4,8 +4,13 @@ import { traverseJavaScriptAst } from "./javascript/javascriptSemanticTraversal. import { parseJavaScriptSource } from "./javascript/javascriptSourceParser.js"; import type { WebModuleImport } from "./webModuleTrace.js"; import type { ExportedWebScript } from "./webScriptExport.js"; +import { + hasUncertainTypeOnlyModuleSideEffect, + isTypeOnlyExportDeclaration, + isTypeOnlyImportDeclaration, +} from "./javascript/javascriptModuleDeclaration.js"; -/** Collect native import syntax, excluding erased type-only references. */ +/** Collect module declarations, omitting whole-statement type-only forms. */ export const collectWebModuleImports = ( source: string, ): { @@ -55,35 +60,18 @@ export const collectWebModuleImports = ( traverseJavaScriptAst(ast, { enter: (node) => { if (node.type === "ImportDeclaration") { - if (node.importKind === "type" || node.importKind === "typeof") return; - if ( - node.specifiers.length > 0 && - node.specifiers.every( - (s) => - s.type === "ImportSpecifier" && - (s.importKind === "type" || s.importKind === "typeof"), - ) - ) - return; + if (isTypeOnlyImportDeclaration(node)) return; add("static-import", node.source); } else if ( node.type === "ExportAllDeclaration" || node.type === "ExportNamedDeclaration" ) { if ( - node.exportKind === "type" || + isTypeOnlyExportDeclaration(node) || node.source === null || node.source === undefined ) return; - if ( - node.type === "ExportNamedDeclaration" && - node.specifiers.length > 0 && - node.specifiers.every( - (s) => s.type === "ExportSpecifier" && s.exportKind === "type", - ) - ) - return; add("re-export", node.source); } else if (node.type === "ImportExpression") add("dynamic-import", node.source); @@ -99,7 +87,14 @@ export const collectWebModuleImports = ( return { imports, state: "parsed", - diagnostics: ast.errors?.map((error) => error.message) ?? [], + diagnostics: [ + ...(ast.errors?.map((error) => error.message) ?? []), + ...(hasUncertainTypeOnlyModuleSideEffect(ast.program) + ? [ + "Compiler options are unavailable; inline type-only imports or re-exports may preserve a module side-effect dependency.", + ] + : []), + ], }; }; diff --git a/src/domain/webNetworkCapture.ts b/src/domain/webNetworkCapture.ts index ed31d5c70..77752f8e5 100644 --- a/src/domain/webNetworkCapture.ts +++ b/src/domain/webNetworkCapture.ts @@ -1,6 +1,7 @@ import { z } from "zod"; import { jsonValueSchema } from "./jsonValue.js"; import { localPathStringSchema } from "./localPath.js"; +import { qlogCaptureFactsSchema } from "./qlogCapture.js"; /** Complete historical evidence budgets; exceeding a budget returns no partial capture. */ export const WEB_NETWORK_CAPTURE_LIMITS = { @@ -10,12 +11,19 @@ export const WEB_NETWORK_CAPTURE_LIMITS = { depth: 64, } as const; +export const WEB_NETWORK_CAPTURE_FORMATS = [ + "har", + "mitmproxy", + "saz", + "qlog", +] as const; + /** Inspect an explicit retained capture without contacting its recorded destinations. */ export const inspectWebNetworkCaptureInputSchema = z.strictObject({ capture_path: localPathStringSchema.describe( - "Absolute local HAR or mitmproxy capture path.", + "Absolute local HAR, mitmproxy, Fiddler SAZ or aioquic qlog capture path.", ), - format: z.enum(["har", "mitmproxy"]), + format: z.enum(WEB_NETWORK_CAPTURE_FORMATS), record_ordinals: z .array(z.number().int().nonnegative()) .min(1) @@ -31,6 +39,21 @@ export const inspectWebNetworkCaptureInputSchema = z.strictObject({ ), }); +/** Read declarations through their owning schema for service and adapter failures. */ +export const captureSensitiveValues = ( + rawInput: unknown, +): readonly string[] => { + const parsed = + inspectWebNetworkCaptureInputSchema.shape.sensitive_values.safeParse( + typeof rawInput === "object" && + rawInput !== null && + "sensitive_values" in rawInput + ? rawInput.sensitive_values + : undefined, + ); + return parsed.success ? parsed.data : []; +}; + const pointerSchema = z .string() .describe("RFC 6901 pointer in the producer's reported object."); @@ -53,21 +76,26 @@ const binaryBase = { pointer: pointerSchema, representation: z.enum(["producer-bytes", "har-base64-content"]), }; +const retainedBinaryDescriptorSchema = z.strictObject({ + ...binaryBase, + state: z.literal("retained"), + bytes: z.number().int().nonnegative(), + sha256: z.string().regex(/^[0-9a-f]{64}$/), +}); +const redactedBinaryDescriptorSchema = z.strictObject({ + ...binaryBase, + state: z.literal("redacted"), + bytes: z.null(), + sha256: z.null(), +}); +/** Exact byte facts shared by complete records and previews that omit content. */ +export const webNetworkCaptureBinaryDescriptorSchema = z.discriminatedUnion( + "state", + [retainedBinaryDescriptorSchema, redactedBinaryDescriptorSchema], +); const binarySchema = z.discriminatedUnion("state", [ - z.strictObject({ - ...binaryBase, - state: z.literal("retained"), - content_base64: z.string(), - bytes: z.number().int().nonnegative(), - sha256: z.string().regex(/^[0-9a-f]{64}$/), - }), - z.strictObject({ - ...binaryBase, - state: z.literal("redacted"), - content_base64: z.null(), - bytes: z.null(), - sha256: z.null(), - }), + retainedBinaryDescriptorSchema.extend({ content_base64: z.string() }), + redactedBinaryDescriptorSchema.extend({ content_base64: z.null() }), ]); const numberSchema = z.strictObject({ pointer: pointerSchema, @@ -84,6 +112,10 @@ export const webNetworkCaptureRecordSchema = z.strictObject({ reason: z.literal("explicit-sensitive-value"), }), z.strictObject({ kind: z.literal("json-pointer"), pointer: pointerSchema }), + z.strictObject({ + kind: z.literal("archive-members"), + members: z.array(z.string()).min(1), + }), z.strictObject({ kind: z.literal("byte-range"), offset: z.number().int().nonnegative(), @@ -97,69 +129,85 @@ export const webNetworkCaptureRecordSchema = z.strictObject({ limitations: z.array(z.string()), }); -/** Decoded records retain unique producer ordinals within their declared coverage. */ -export const webNetworkCapturePayloadSchema = z - .strictObject({ - decoder: z.strictObject({ - id: z.string().min(1), - name: z.string().min(1), - version: z.string().min(1), - }), - container: z.strictObject({ - reported: jsonValueSchema, - numeric_literals: z.array(numberSchema), - redactions: z.array(redactionSchema), - records_pointer: pointerSchema.nullable(), - }), - total_records: z.number().int().nonnegative(), - records: z.array(webNetworkCaptureRecordSchema), - }) - .superRefine((capture, context) => { - if (capture.records.length > capture.total_records) { +/** The selected offline parser profile, independent of authenticated producer identity. */ +export const webNetworkCaptureDecoderSchema = z.strictObject({ + id: z.string().min(1), + name: z.string().min(1), + version: z.string().min(1), +}); + +/** Source coverage is shared by every offline decoder. */ +const payloadShape = { + decoder: webNetworkCaptureDecoderSchema, + container: z.strictObject({ + reported: jsonValueSchema, + numeric_literals: z.array(numberSchema), + redactions: z.array(redactionSchema), + records_pointer: pointerSchema.nullable(), + }), + total_records: z.number().int().nonnegative(), + records: z.array(webNetworkCaptureRecordSchema), +}; +const refineCoverage = ( + capture: { total_records: number; records: { ordinal: number }[] }, + context: z.RefinementCtx, +): void => { + const seen = new Set(); + for (const [index, record] of capture.records.entries()) { + if (record.ordinal >= capture.total_records || seen.has(record.ordinal)) { context.addIssue({ code: "custom", - path: ["records"], - message: "Retained record count exceeds the declared capture total.", + path: ["records", index, "ordinal"], + message: + "Retained ordinals must be unique and within the declared capture total.", }); return; } - const seen = new Set(); - for (const [index, record] of capture.records.entries()) { - if (record.ordinal >= capture.total_records || seen.has(record.ordinal)) { - context.addIssue({ - code: "custom", - path: ["records", index, "ordinal"], - message: - "Retained ordinals must be unique and within the declared capture total.", - }); - return; - } - seen.add(record.ordinal); - } - }); - -/** Historical capture inspection has file identity, producer provenance and explicit unknowns. */ -export const webNetworkCaptureSchema = - webNetworkCapturePayloadSchema.safeExtend({ - artifact: z.strictObject({ - path: z - .string() - .describe( - "Selected artifact path; empty when explicitly excluded from Evidence.", - ), - sha256: z - .string() - .regex(/^[0-9a-f]{64}$/) - .nullable() - .describe( - "Observed artifact SHA-256; null only when explicitly excluded by a sensitive declaration.", - ), - bytes: z.number().int().nonnegative(), + seen.add(record.ordinal); + } +}; +/** Strict alternatives prevent undefined-valued facets from entering JSON Evidence. */ +export const webNetworkCapturePayloadSchema = z + .union([ + z.strictObject({ ...payloadShape, qlog: qlogCaptureFactsSchema }), + z.strictObject(payloadShape), + ]) + .superRefine(refineCoverage); +/** Caller-selected file identity; explicit exclusions remain unknown. */ +export const webNetworkCaptureArtifactSchema = z.strictObject({ + path: z + .string() + .describe( + "Selected artifact path; empty when explicitly excluded from Evidence.", + ), + sha256: z + .string() + .regex(/^[0-9a-f]{64}$/) + .nullable() + .describe("Observed SHA-256; null only when explicitly excluded."), + bytes: z.number().int().nonnegative(), +}); +const artifactShape = { + artifact: webNetworkCaptureArtifactSchema, + runtime_attribution: z.literal("unknown"), + limitations: z.array(z.string()), +}; +/** Qlog transport facts occur exactly on the qlog alternative. */ +export const webNetworkCaptureSchema = z + .discriminatedUnion("format", [ + z.strictObject({ + ...payloadShape, + ...artifactShape, + format: z.enum(["har", "mitmproxy", "saz"]), }), - format: z.enum(["har", "mitmproxy"]), - runtime_attribution: z.literal("unknown"), - limitations: z.array(z.string()), - }); + z.strictObject({ + ...payloadShape, + ...artifactShape, + format: z.literal("qlog"), + qlog: qlogCaptureFactsSchema, + }), + ]) + .superRefine(refineCoverage); export type InspectWebNetworkCaptureInput = z.output< typeof inspectWebNetworkCaptureInputSchema diff --git a/src/domain/webNetworkCaptureCoordinates.ts b/src/domain/webNetworkCaptureCoordinates.ts index 6cba46cd9..a975c6b4a 100644 --- a/src/domain/webNetworkCaptureCoordinates.ts +++ b/src/domain/webNetworkCaptureCoordinates.ts @@ -22,6 +22,8 @@ export const excludeCaptureCoordinates = ( const location = (record.location.kind === "json-pointer" && sensitive(record.location.pointer)) || + (record.location.kind === "archive-members" && + record.location.members.some(sensitive)) || (record.location.kind === "byte-range" && (sensitive(String(record.location.offset)) || sensitive(String(record.location.bytes)))) @@ -51,8 +53,30 @@ export const excludeCaptureCoordinates = ( numeric_literals: sidecars(value.container.numeric_literals), redactions: sidecars(value.container.redactions), }; + const qlog = + value.format === "qlog" + ? { + ...value.qlog, + traces: value.qlog.traces.map((trace) => { + if ( + trace.location.kind !== "json-pointer" || + !sensitive(trace.location.pointer) + ) + return trace; + excluded = true; + return { + ...trace, + location: { + kind: "unknown" as const, + reason: "explicit-sensitive-value" as const, + }, + }; + }), + } + : undefined; return { ...value, + ...(qlog === undefined ? {} : { qlog }), container, records, limitations: excluded diff --git a/src/dotnet/ManagedArtifactInspector.test.ts b/src/dotnet/ManagedArtifactInspector.test.ts index 2d30bf580..50de3a39b 100644 --- a/src/dotnet/ManagedArtifactInspector.test.ts +++ b/src/dotnet/ManagedArtifactInspector.test.ts @@ -292,6 +292,30 @@ describe("managed artifact inventory", () => { ]); expect(result.coverage).toMatchObject({ state: "complete", issues: [] }); }); + + // dotnet/runtime corhdr.h: 32BITREQUIRED and 32BITPREFERRED form one + // two-bit field, so neither bit can be read on its own. + it.each([ + ["neither bit", 0x0000_0001, "anycpu", false], + ["32BITREQUIRED alone", 0x0000_0003, "x86", true], + ["both bits", 0x0002_0003, "anycpu-prefer-32", false], + ["the reserved 32BITPREFERRED alone", 0x0002_0001, "anycpu", false], + ] as const)( + "reads %s as one 32-bit requirement", + (_label, cliFlags, architecture, requires32Bit) => { + const bytes = buildManagedPeFixture({ cliFlags }); + const target = managedPeFixtureTarget(bytes); + + expect( + inspectManagedArtifactBytes(bytes, target).classification + .managed_architecture, + ).toBe(architecture); + expect( + inspectManagedNativeBoundariesBytes(bytes, target).cli_native + ?.requires_32bit, + ).toBe(requires32Bit); + }, + ); }); describe("managed artifact coded indexes", () => { @@ -523,6 +547,63 @@ describe("managed fixture data layout", () => { }); }); +describe("managed implementation classification", () => { + it("classifies crossgen2 ReadyToRun output without ILONLY as modern .NET", () => { + // crossgen2 writes IL_LIBRARY | STRONGNAMESIGNED and clears ILONLY. + const bytes = buildManagedPeFixture({ readyToRun: true, cliFlags: 0xc }); + const result = inspectManagedArtifactBytes( + bytes, + managedPeFixtureTarget(bytes), + ); + + expect(result.classification).toMatchObject({ + runtime_family: "modern-dotnet", + implementation: "cil-and-ready-to-run", + }); + expect(result.classification.evidence).toContainEqual( + expect.objectContaining({ code: "ready-to-run-signature-observed" }), + ); + expect(result.classification.evidence).not.toContainEqual( + expect.objectContaining({ code: "cli-il-only-clear" }), + ); + }); + + it("preserves ReadyToRun runtime identity after metadata capacity is exhausted", () => { + const bytes = buildManagedPeFixture({ + readyToRun: true, + cliFlags: 0xc, + references: ["x".repeat(12 * 1024 * 1024)], + }); + const result = inspectManagedArtifactBytes( + bytes, + managedPeFixtureTarget(bytes), + ); + expect(result.classification).toMatchObject({ + status: "managed", + runtime_family: "modern-dotnet", + }); + expect(result.coverage.issues).toContainEqual( + expect.objectContaining({ code: "resource-limit" }), + ); + }); + + it("keeps a clear ILONLY flag without ReadyToRun as mixed CLR/native", () => { + const bytes = buildManagedPeFixture({ cliFlags: 0 }); + const result = inspectManagedArtifactBytes( + bytes, + managedPeFixtureTarget(bytes), + ); + + expect(result.classification).toMatchObject({ + runtime_family: "mixed-clr-native", + implementation: "cpp-cli-mixed", + }); + expect(result.classification.evidence).toContainEqual( + expect.objectContaining({ code: "cli-il-only-clear" }), + ); + }); +}); + describe("managed artifact metadata identity", () => { it("accepts CLI metadata GUIDs without RFC UUID version or variant bits", () => { const bytes = buildManagedPeFixture({ diff --git a/src/dotnet/ManagedArtifactInspector.ts b/src/dotnet/ManagedArtifactInspector.ts index 7df8aae83..625969aba 100644 --- a/src/dotnet/ManagedArtifactInspector.ts +++ b/src/dotnet/ManagedArtifactInspector.ts @@ -9,6 +9,7 @@ import { type ManagedParseIssue, } from "../domain/managed/managedArtifact.js"; import { + managedCli32BitRequirement, readManagedPeLayout, type ManagedCliHeader, type ManagedPeLayout, @@ -157,6 +158,15 @@ interface RuntimeClassificationInput { readonly inventory: ManagedMetadataInventory; } +const runtimeFamilyFromCliHeader = ( + cli: ManagedCliHeader | null, +): ManagedArtifactInspection["classification"]["runtime_family"] => + cli?.readyToRunSignature === true + ? "modern-dotnet" + : cli !== null && (cli.flags & 1) === 0 + ? "mixed-clr-native" + : "unknown"; + const runtimeFamilyFor = ( cli: ManagedCliHeader, inventory: ManagedMetadataInventory, @@ -179,7 +189,9 @@ const runtimeFamilyFor = ( ); let runtimeFamily: ManagedArtifactInspection["classification"]["runtime_family"] = "unknown"; - if ((cli.flags & 1) === 0) { + // crossgen2 clears ILONLY on ReadyToRun output, so only a clear flag + // without the ReadyToRun signature indicates mixed CLR/native code. + if (runtimeFamilyFromCliHeader(cli) === "mixed-clr-native") { runtimeFamily = "mixed-clr-native"; evidence.push({ code: "cli-il-only-clear", @@ -238,8 +250,8 @@ const implementationFor = ( cli: ManagedCliHeader, metadata: ManagedMetadataLayout, ): ManagedArtifactInspection["classification"]["implementation"] => { - if ((cli.flags & 1) === 0) return "cpp-cli-mixed"; if (cli.readyToRunSignature) return "cil-and-ready-to-run"; + if ((cli.flags & 1) === 0) return "cpp-cli-mixed"; return (metadata.rowCounts[6] ?? 0) === 0 ? "metadata-only" : "cil"; }; @@ -249,9 +261,9 @@ const managedArchitectureFor = ( ): ManagedArtifactInspection["classification"]["managed_architecture"] => { if (layout.architecture !== "x86" || (cli.flags & 1) === 0) return layout.architecture; - if ((cli.flags & 0x0000_0002) !== 0) return "x86"; - if ((cli.flags & 0x0002_0000) !== 0) return "anycpu-prefer-32"; - return "anycpu"; + const requirement = managedCli32BitRequirement(cli.flags); + if (requirement === "required") return "x86"; + return requirement === "preferred" ? "anycpu-prefer-32" : "anycpu"; }; const runtimeClassification = ({ @@ -302,8 +314,7 @@ const partialMetadataResult = ({ classification: { status: issue.code === "resource-limit" ? "managed" : "malformed", container: "pe", - runtime_family: - (layout.cli?.flags ?? 1) & 1 ? "unknown" : "mixed-clr-native", + runtime_family: runtimeFamilyFromCliHeader(layout.cli), implementation: "unknown", managed_architecture: "unknown", evidence: [ diff --git a/src/dotnet/ManagedCilDecoding.test.ts b/src/dotnet/ManagedCilDecoding.test.ts index c1ed21130..4ba33f4c9 100644 --- a/src/dotnet/ManagedCilDecoding.test.ts +++ b/src/dotnet/ManagedCilDecoding.test.ts @@ -345,6 +345,31 @@ describe("managed exception region prefix boundaries", () => { }); }); + it("decodes the no. prefix and keeps it with its opcode", () => { + // ldarg.0; no. nullcheck; ldfld 0x04000001; pop; ret + const il = Buffer.from([2, 0xfe, 0x19, 4, 0x7b, 1, 0, 0, 4, 0x26, 0x2a]); + expect( + managedBodyWithRegion({ il, tryLength: 10, handlerOffset: 10 }), + ).toMatchObject({ + status: "present", + issue: null, + decoded_instruction_count: 5, + opcode_counts: { "ldarg.0": 1, "no.": 1, ldfld: 1, pop: 1, ret: 1 }, + }); + expect( + managedBodyWithRegion({ + il, + tryOffset: 4, + tryLength: 5, + handlerOffset: 10, + }), + ).toMatchObject({ + status: "malformed", + issue: + "Exception clause try range does not align with CIL instruction boundaries", + }); + }); + it.each([ ["start", { tryOffset: 5, tryLength: 1, handlerOffset: 6 }], [ @@ -510,3 +535,41 @@ describe("managed exception section decoding", () => { }); }); }); + +describe("managed CIL truncated operands", () => { + const tinyBody = (il: readonly number[]) => { + const bytes = buildManagedPeFixture({ + ilBody: Buffer.from([(il.length << 2) | 0x02, ...il]), + }); + const inspection = inspectManagedMembersBytes( + bytes, + managedPeFixtureTarget(bytes), + ); + return { body: inspection.methods[0]?.body, coverage: inspection.coverage }; + }; + + it.each([ + [[0x20], 0, "Instruction ldc.i4 at IL offset 0 leaves method body"], + [ + [0x00, 0x20, 0x01], + 1, + "Instruction ldc.i4 at IL offset 1 leaves method body", + ], + [[0x2b], 0, "Instruction br.s at IL offset 0 leaves method body"], + [[0x45, 0x02], 0, "Instruction switch at IL offset 0 leaves method body"], + [ + [0xfe, 0x09, 0x00], + 0, + "Instruction ldarg at IL offset 0 leaves method body", + ], + [[0x00, 0xfe], 1, "Two-byte CIL opcode at IL offset 1 leaves method body"], + ])("names the instruction and IL offset for IL %j", (il, decoded, issue) => { + const { body, coverage } = tinyBody(il); + expect(body).toMatchObject({ + status: "malformed", + decoded_instruction_count: decoded, + issue, + }); + expect(coverage.state).toBe("partial"); + }); +}); diff --git a/src/dotnet/ManagedMemberInspector.test.ts b/src/dotnet/ManagedMemberInspector.test.ts index 81d64f4a1..9ada1bf55 100644 --- a/src/dotnet/ManagedMemberInspector.test.ts +++ b/src/dotnet/ManagedMemberInspector.test.ts @@ -110,6 +110,8 @@ describe("managed member inspection", () => { expect(result.member_refs).toEqual([ expect.objectContaining({ token: "0x0a000001", + parent_token: "0x01000001", + parent_name: "System.Runtime.Versioning.TargetFrameworkAttribute", name: ".ctor", signature: expect.objectContaining({ kind: "method", @@ -126,6 +128,8 @@ describe("managed member inspection", () => { target_token: "0x0a000001", target_kind: "member-ref", target_name: ".ctor", + target_declaring_type: + "System.Runtime.Versioning.TargetFrameworkAttribute", }, ]); expect(result.field_accesses).toEqual([ @@ -135,12 +139,366 @@ describe("managed member inspection", () => { opcode: "ldfld", field_token: "0x04000001", field_name: "counter", + field_declaring_type: "Fixture.Program", }, ]); expect(result.coverage).toMatchObject({ state: "complete", issues: [] }); }); }); +/** A tiny-header body of token-operand instructions followed by `ret`. */ +const tokenBody = ( + instructions: readonly (readonly [opcode: number, token: number])[], +): Buffer => { + const il = Buffer.concat([ + ...instructions.map(([opcode, token]) => { + const instruction = Buffer.alloc(5); + instruction.writeUInt8(opcode, 0); + instruction.writeUInt32LE(token, 1); + return instruction; + }), + Buffer.from([0x2a]), + ]); + return Buffer.concat([Buffer.from([(il.length << 2) | 2]), il]); +}; + +const attribute = "System.Runtime.Versioning.TargetFrameworkAttribute"; +const inspect = (options: Parameters[0]) => { + const bytes = buildManagedPeFixture(options); + return inspectManagedMembersBytes(bytes, managedPeFixtureTarget(bytes)); +}; +const call = 0x28; +const ldsfld = 0x7e; +const memberRefOnTypeSpec = (signature: Buffer) => + inspect({ + typeSpecSignatures: [signature], + memberRefParentRaw: (1 << 3) | 4, + }); + +// HASTHIS | GENERIC, one generic parameter, no parameters, void. +const genericRef = Buffer.from([0x30, 0x01, 0x00, 0x01]); +// GENERIC, one generic parameter, no parameters, void. +const genericDef = Buffer.from([0x10, 0x01, 0x00, 0x01]); + +describe("managed edge target declaring types", () => { + it("names a generic TypeSpec parent by its generic type and leaves other TypeSpecs unknown", () => { + // GENERICINST CLASS TypeRef(1) + const generic = memberRefOnTypeSpec( + Buffer.from([0x15, 0x12, (1 << 2) | 1, 0x01, 0x0e]), + ); + expect(generic.member_refs[0]?.parent_name).toBe(attribute); + expect(generic.call_edges[0]?.target_declaring_type).toBe(attribute); + + // SZARRAY i4 + const array = memberRefOnTypeSpec(Buffer.from([0x1d, 0x08])); + expect(array.member_refs[0]?.parent_name).toBeNull(); + expect(array.coverage).toEqual({ state: "complete", issues: [] }); + }); + + it.each([ + [ + "names a TypeSpec as its generic type", + [0x15, 0x12, (1 << 2) | 2, 0x01, 0x0e], + "invalid-row", + "metadata.TypeSpec:0x1b000001", + ], + [ + "has no type arguments", + [0x15, 0x12, (1 << 2) | 1], + "invalid-blob", + "metadata.TypeSpec:0x1b000001", + ], + [ + "names a missing TypeRef", + [0x15, 0x12, (9 << 2) | 1, 0x01, 0x0e], + "invalid-row", + "metadata.TypeSpec:0x1b000001", + ], + [ + "names a missing TypeDef", + [0x15, 0x12, (9 << 2) | 0, 0x01, 0x0e], + "invalid-row", + "metadata.TypeSpec:0x1b000001", + ], + [ + "has a type argument naming a missing TypeRef", + [0x15, 0x12, (1 << 2) | 1, 0x01, 0x12, (9 << 2) | 1], + "invalid-row", + "metadata.TypeSpec:0x1b000001", + ], + ])( + "reports a generic TypeSpec parent that %s and leaves it unknown", + (_case, signature, code, scope) => { + const result = memberRefOnTypeSpec(Buffer.from(signature)); + + expect(result.member_refs[0]?.parent_name).toBeNull(); + expect(result.coverage).toMatchObject({ + state: "partial", + issues: [{ code, scope }], + }); + }, + ); + + it("reports unsupported TypeSpec and MethodSpec forms instead of treating them as unknown", () => { + const typeSpec = memberRefOnTypeSpec(Buffer.from([0xff])); + const methodSpec = inspect({ + memberRefSignature: genericRef, + methodSpecMethods: [(1 << 1) | 1], + methodSpecInstantiation: Buffer.from([0x0a, 0x01, 0xff]), + ilBody: tokenBody([[call, 0x2b00_0001]]), + }); + + expect(typeSpec.member_refs[0]?.parent_name).toBeNull(); + expect(typeSpec.coverage).toMatchObject({ + state: "partial", + issues: [ + { + code: "unsupported-signature", + scope: "metadata.TypeSpec:0x1b000001", + }, + ], + }); + expect(methodSpec.coverage).toMatchObject({ + state: "partial", + issues: [ + { + code: "unsupported-signature", + scope: "metadata.MethodSpec:0x2b000001", + }, + ], + }); + }); + + it("names field MemberRefs as fields and leaves mismatched member kinds unnamed", () => { + const result = inspect({ + memberRefName: "Empty", + // FIELD string + memberRefSignature: Buffer.from([0x06, 0x0e]), + ilBody: tokenBody([ + [ldsfld, 0x0a00_0001], + [call, 0x0400_0001], + ]), + }); + const methodRef = inspect({ ilBody: tokenBody([[ldsfld, 0x0a00_0001]]) }); + const flaggedField = inspect({ + memberRefName: "Empty", + // GENERIC | FIELD string: FieldSig allows no flags. + memberRefSignature: Buffer.from([0x16, 0x0e]), + ilBody: tokenBody([[ldsfld, 0x0a00_0001]]), + }); + + expect(result.field_accesses).toMatchObject([ + { field_name: "Empty", field_declaring_type: attribute }, + ]); + expect(methodRef.field_accesses).toMatchObject([ + { field_name: null, field_declaring_type: null }, + ]); + expect(flaggedField.field_accesses).toMatchObject([ + { field_name: null, field_declaring_type: null }, + ]); + expect(flaggedField.coverage.state).toBe("partial"); + expect(result.call_edges).toMatchObject([ + { + target_kind: "unknown", + target_name: null, + target_declaring_type: null, + }, + ]); + }); + + it("reads Evidence recorded before declaring types were named as unknown", () => { + const current = inspect({}); + const parsed = managedMemberInspectionSchema.parse({ + ...current, + member_refs: current.member_refs.map( + ({ parent_name: _omitted, ...legacy }) => legacy, + ), + call_edges: current.call_edges.map( + ({ target_declaring_type: _omitted, ...legacy }) => legacy, + ), + field_accesses: current.field_accesses.map( + ({ field_declaring_type: _omitted, ...legacy }) => legacy, + ), + }); + + expect(parsed.member_refs[0]?.parent_name).toBeNull(); + expect(parsed.call_edges[0]?.target_declaring_type).toBeNull(); + expect(parsed.field_accesses[0]?.field_declaring_type).toBeNull(); + }); +}); + +describe("managed TypeRef resolution scopes", () => { + it.each([ + // AssemblyRef 9 does not exist. + ["an invalid", (9 << 2) | 2, "is invalid"], + // The TypeRef is its own enclosing type. + ["a cyclic", (1 << 2) | 3, "contains a cycle"], + ])( + "keeps a TypeRef name beside %s resolution scope", + (_case, typeRefScopeRaw, detail) => { + const result = inspect({ typeRefScopeRaw }); + + expect(result.member_refs[0]?.parent_name).toBe(attribute); + expect(result.call_edges[0]?.target_declaring_type).toBe(attribute); + expect(result.coverage).toMatchObject({ + state: "partial", + issues: [ + { + code: "invalid-row", + scope: "metadata.TypeRef:0x01000001", + detail: expect.stringContaining(detail), + }, + ], + }); + }, + ); +}); + +describe("managed MethodSpec call targets", () => { + it("names generic method calls after the method they instantiate", () => { + const result = inspect({ + memberRefSignature: genericRef, + methodSignature: genericDef, + methodSpecMethods: [(1 << 1) | 1, (1 << 1) | 0], + ilBody: tokenBody([ + [call, 0x2b00_0001], + [call, 0x2b00_0002], + ]), + }); + + expect( + result.call_edges.map((edge) => [ + edge.target_kind, + edge.target_name, + edge.target_declaring_type, + ]), + ).toEqual([ + ["method-spec", ".ctor", attribute], + ["method-spec", "Main", "Fixture.Program"], + ]); + expect(result.coverage).toEqual({ state: "complete", issues: [] }); + }); + + it.each([ + [ + "a non-generic method", + {}, + Buffer.from([0x0a, 0x01, 0x0e]), + "but its method declares 0 generic parameter(s)", + ], + [ + "the wrong number of type arguments", + { memberRefSignature: genericRef }, + Buffer.from([0x0a, 0x02, 0x0e, 0x08]), + "supplies 2 type argument(s), but its method declares 1", + ], + [ + "a field reference", + { memberRefSignature: Buffer.from([0x06, 0x0e]) }, + Buffer.from([0x0a, 0x01, 0x0e]), + "does not have a decodable method signature", + ], + [ + "a property signature", + // PROPERTY | GENERIC, one generic parameter, no parameters, void. + { memberRefSignature: Buffer.from([0x18, 0x01, 0x00, 0x01]) }, + Buffer.from([0x0a, 0x01, 0x0e]), + "does not have a decodable method signature", + ], + [ + "a truncated signature", + // HASTHIS | GENERIC, one generic parameter, no parameters, no return type. + { memberRefSignature: Buffer.from([0x30, 0x01, 0x00]) }, + Buffer.from([0x0a, 0x01, 0x0e]), + "does not have a decodable method signature", + ], + [ + "a type argument naming a missing TypeRef", + { memberRefSignature: genericRef }, + Buffer.from([0x0a, 0x01, 0x12, (9 << 2) | 1]), + "TypeDefOrRefOrSpec operand 0x25 is invalid", + ], + [ + "an unmanaged-convention signature", + // C | GENERIC: a function pointer convention, not a MethodRefSig one. + { memberRefSignature: Buffer.from([0x11, 0x01, 0x00, 0x01]) }, + Buffer.from([0x0a, 0x01, 0x0e]), + "does not have a decodable method signature", + ], + ])( + "reports a MethodSpec that instantiates %s", + (_case, options, methodSpecInstantiation, detail) => { + const result = inspect({ + ...options, + methodSpecMethods: [(1 << 1) | 1], + methodSpecInstantiation, + ilBody: tokenBody([[call, 0x2b00_0001]]), + }); + + expect(result.coverage).toMatchObject({ + state: "partial", + issues: [ + { + code: "invalid-row", + scope: "metadata.MethodSpec:0x2b000001", + detail: expect.stringContaining(detail), + }, + ], + }); + }, + ); + + it("never names a field through a MethodSpec", () => { + const result = inspect({ + memberRefName: "Empty", + // FIELD string: a MethodSpec on it is malformed. + memberRefSignature: Buffer.from([0x06, 0x0e]), + methodSpecMethods: [(1 << 1) | 1], + ilBody: tokenBody([ + [ldsfld, 0x2b00_0001], + [call, 0x2b00_0001], + ]), + }); + + expect(result.field_accesses).toMatchObject([ + { field_name: null, field_declaring_type: null }, + ]); + expect(result.call_edges).toMatchObject([ + { target_name: null, target_declaring_type: null }, + ]); + expect(result.coverage.state).toBe("partial"); + }); + + it("keeps a readable MethodSpec method beside a malformed instantiation", () => { + const result = inspect({ + methodSpecMethods: [(1 << 1) | 1, (9 << 1) | 1], + methodSpecInstantiation: Buffer.from([0x0a, 0x00]), + ilBody: tokenBody([ + [call, 0x2b00_0001], + [call, 0x2b00_0002], + ]), + }); + + expect( + result.call_edges.map(({ target_name, target_declaring_type }) => [ + target_name, + target_declaring_type, + ]), + ).toEqual([ + [".ctor", attribute], + [null, null], + ]); + expect(result.coverage.state).toBe("partial"); + expect( + result.coverage.issues.map(({ code, scope }) => [code, scope]), + ).toEqual([ + ["invalid-blob", "metadata.MethodSpec:0x2b000001"], + ["invalid-row", "metadata.MethodSpec:0x2b000002"], + ["invalid-blob", "metadata.MethodSpec:0x2b000002"], + ]); + }); +}); + describe("managed exported types", () => { it("reports type forwarders and nested exported types with their implementation scope", () => { const bytes = buildManagedPeFixture({ @@ -373,15 +731,9 @@ describe("managed derived edges", () => { body: { anchors: Array.from({ length: 40_000 }, () => anchor) }, } as unknown as ManagedMethod; - expect(() => - edges( - [method], - new Map(), - new Map(), - new Map(), - new ManagedDecodeBudget(), - ), - ).toThrow(ManagedReaderFailure); + expect(() => edges([method], new Map(), new ManagedDecodeBudget())).toThrow( + ManagedReaderFailure, + ); }); }); @@ -413,15 +765,9 @@ describe("managed derived caller names", () => { }, } as unknown as ManagedMethod; - expect(() => - edges( - [method], - new Map(), - new Map(), - new Map(), - new ManagedDecodeBudget(), - ), - ).toThrow(ManagedReaderFailure); + expect(() => edges([method], new Map(), new ManagedDecodeBudget())).toThrow( + ManagedReaderFailure, + ); expect(built).toBe(false); }); }); @@ -671,6 +1017,18 @@ it.each([ }); }); +it.each([ + ["a property header", [0x08, 0x00, 0x08]], + ["an explicit-this header without has-this", [0x40, 0x00, 0x01]], + ["a C calling convention header", [0x01, 0x00, 0x01]], + ["a generic vararg header", [0x15, 0x01, 0x00, 0x01]], +])("reports %s as a malformed member signature", (_name, blob) => { + expect(signature(Buffer.from(blob))).toMatchObject({ + kind: "unknown", + parse_status: "malformed", + }); +}); + it("rejects array ranks beyond the CLR limit before rendering dimensions", () => { expect( signature( diff --git a/src/dotnet/ManagedMemberInspector.ts b/src/dotnet/ManagedMemberInspector.ts index 24b97f9eb..dba20ac57 100644 --- a/src/dotnet/ManagedMemberInspector.ts +++ b/src/dotnet/ManagedMemberInspector.ts @@ -27,6 +27,8 @@ import { typeRanges, type TypeRange, } from "./ManagedMemberInspectorCore.js"; +import { edgeTargets } from "./ManagedMethodSpecs.js"; +import { typeNameLookup } from "./ManagedTypeNames.js"; import { edges, parseExportedTypes, @@ -171,7 +173,11 @@ export const inspectManagedMembersBytes = ( const exportedTypes = parseExportedTypes(bytes, layout); issues.push(...exportedTypes.issues); const fields = parseFields(bytes, layout, ranges, selected); - const memberRefs = parseMemberRefs(bytes, layout); + const memberRefs = parseMemberRefs( + bytes, + layout, + typeNameLookup(bytes, layout, ranges, issues), + ); issues.push(...memberRefs.issues); const methods = parseMethods({ bytes, @@ -181,6 +187,12 @@ export const inspectManagedMembersBytes = ( selected, }); issues.push(...fields.issues, ...methods.issues); + const targets = edgeTargets(bytes, layout, methods.methods, [ + fields.core, + memberRefs.core, + methods.core, + ]); + issues.push(...targets.issues); const coverageIssues = uniqueManagedParseIssues(issues); const incompleteMembers = [...fields.fields, ...memberRefs.refs, ...methods.methods].some( @@ -193,9 +205,7 @@ export const inspectManagedMembersBytes = ( ); const related = edges( methods.methods, - methods.core, - fields.core, - memberRefs.core, + targets.members, managedDecodeBudget(layout), ); return managedMemberInspectionSchema.parse( diff --git a/src/dotnet/ManagedMemberInspectorCore.ts b/src/dotnet/ManagedMemberInspectorCore.ts index f54a11769..7776c0090 100644 --- a/src/dotnet/ManagedMemberInspectorCore.ts +++ b/src/dotnet/ManagedMemberInspectorCore.ts @@ -7,6 +7,7 @@ import type { ManagedMemberInspection, ManagedParseIssue, } from "../domain/managed/managedArtifact.js"; +import { managedDecodeBudget } from "./ManagedDecodeBudget.js"; import type { ManagedMetadataLayout, MetadataTableLayout, @@ -49,20 +50,22 @@ export interface TypeRange { readonly methodEnd: number; } -export interface MethodCore { +/** + * Name and declaring type of a field, method, member reference, or method + * specification, keyed by its token so edges resolve any target uniformly. + */ +export interface MemberCore { readonly token: string; readonly name: string; readonly declaringType: string | null; -} - -export interface FieldCore { - readonly token: string; - readonly name: string; -} - -export interface MemberRefCore { - readonly token: string; - readonly name: string; + /** Fields and methods by table; member references by decoded signature. */ + readonly kind: ManagedSignature["kind"]; + /** + * Generic arity from the owner's decoded signature (see + * `signatureGenericArity`); absent when no owner decoded it, as for a + * MethodDef outside the type selection. + */ + readonly genericArity?: number | null; } export interface ParsedInstruction { @@ -88,7 +91,7 @@ const rowRange = ( return { first: start, last: end - 1, count: end - start }; }; -const fullName = (namespace: string, name: string): string => +export const fullName = (namespace: string, name: string): string => namespace.length === 0 ? name : `${namespace}.${name}`; const readTypeRange = ( @@ -242,7 +245,7 @@ export const parseTypes = ( return { types: items, issues }; }; -const readCompressed = ( +export const readCompressed = ( bytes: Buffer, offset: number, ): { readonly value: number; readonly next: number } => { @@ -341,13 +344,38 @@ interface MethodSignatureHeader { readonly next: number; } +// MethodDefSig and MethodRefSig headers (ECMA-335 II.23.2.1-2) without +// HASTHIS and EXPLICITTHIS: DEFAULT, VARARG, or GENERIC, which implies DEFAULT. +const MEMBER_METHOD_HEADERS = new Set([0x00, 0x05, 0x10]); + +// FNPTR also carries the unmanaged C, STDCALL, THISCALL, FASTCALL, and +// UNMANAGED conventions that compilers emit for unmanaged function pointers. +const FUNCTION_POINTER_CONVENTIONS = new Set([0, 1, 2, 3, 4, 5, 9]); + +/** + * Read a method signature's calling convention and counts. A header that is + * not a method signature in its context (a MethodDef or MemberRef row, or an + * FNPTR type), sets the reserved bit, or sets EXPLICITTHIS without HASTHIS is + * malformed, so no caller consumes its counts. FIELD, LOCAL_SIG, PROPERTY, + * and GENERICINST headers are never method signatures. + */ const readMethodSignatureHeader = ( blob: Buffer, offset: number, + context: "member" | "function-pointer", ): MethodSignatureHeader => { const first = blob[offset]; if (first === undefined) return malformedSignature("truncated method signature", offset); + const allowed = + context === "member" + ? MEMBER_METHOD_HEADERS.has(first & 0x9f) + : (first & 0x80) === 0 && FUNCTION_POINTER_CONVENTIONS.has(first & 0x0f); + if (!allowed || ((first & 0x40) !== 0 && (first & 0x20) === 0)) + return malformedSignature( + `calling convention 0x${first.toString(16)} does not head a ${context} method signature`, + offset, + ); let next = offset + 1; let genericParameterCount: number | null = null; if ((first & 0x10) !== 0) { @@ -366,12 +394,6 @@ const readMethodSignatureHeader = ( const MAX_ARRAY_RANK = 32; -// FNPTR carries a method signature: managed DEFAULT/VARARG, or the unmanaged -// C, STDCALL, THISCALL, FASTCALL, and UNMANAGED conventions that compilers -// emit for unmanaged function pointers. FIELD, LOCAL_SIG, PROPERTY, and -// GENERICINST headers are not method signatures. -const FUNCTION_POINTER_CONVENTIONS = new Set([0, 1, 2, 3, 4, 5, 9]); - const arrayShape = ( blob: Buffer, offset: number, @@ -449,10 +471,40 @@ const POSITION_ELEMENTS: ReadonlyMap = [0x10, ["return", "parameter", "field"]], ]); +/** Decode a type operand once; spec callers also validate its metadata coordinate. */ +const readTypeOperand = ( + blob: Buffer, + offset: number, + layout?: ManagedMetadataLayout, + tables: readonly number[] = [2, 1, 27], +): ReturnType => { + const token = readCompressed(blob, offset); + if (layout !== undefined) { + const reason = + token.value === 0 + ? "type operand is null" + : metadataCodedTokenInvalidReason( + token.value, + 2, + tables, + layout.rowCounts, + ); + if (reason !== null) + throw managedFailure( + "invalid-row", + "signature", + `TypeDefOrRefOrSpec operand 0x${token.value.toString(16)} is invalid: ${reason}`, + offset, + ); + } + return token; +}; + const readTypeSignature = ( blob: Buffer, offset: number, base: SignaturePosition, + layout?: ManagedMetadataLayout, ): { readonly value: string; readonly next: number } => { const stack: TypeContinuation[] = []; let cursor = offset; @@ -479,7 +531,7 @@ const readTypeSignature = ( stack.push({ kind: "suffix", suffix }); cursor += 1; } else if (kind === 0x1f || kind === 0x20) { - const token = readCompressed(blob, cursor + 1); + const token = readTypeOperand(blob, cursor + 1, layout); const modifier = `${kind === 0x1f ? "modreq" : "modopt"}:${String(token.value)}`; // A modifier frame on top was pushed by the directly preceding byte; // keep an adjacent run together and in source order. @@ -488,7 +540,7 @@ const readTypeSignature = ( else stack.push({ kind: "modifiers", modifiers: [modifier] }); cursor = token.next; } else if (kind === 0x11 || kind === 0x12) { - const token = readCompressed(blob, cursor + 1); + const token = readTypeOperand(blob, cursor + 1, layout); value = `${kind === 0x11 ? "valuetype" : "class"}:${String(token.value)}`; cursor = token.next; } else if (kind === 0x1e || kind === 0x13) { @@ -508,7 +560,7 @@ const readTypeSignature = ( "generic instantiation does not name a class or value type", cursor + 1, ); - const token = readCompressed(blob, cursor + 2); + const token = readTypeOperand(blob, cursor + 2, layout, [2, 1]); const argumentCount = readCompressed(blob, token.next); if (argumentCount.value === 0) return malformedSignature( @@ -525,16 +577,11 @@ const readTypeSignature = ( }); cursor = argumentCount.next; } else if (kind === 0x1b) { - const header = readMethodSignatureHeader(blob, cursor + 1); - if ( - !FUNCTION_POINTER_CONVENTIONS.has(header.first & 0x0f) || - (header.first & 0x80) !== 0 || - ((header.first & 0x40) !== 0 && (header.first & 0x20) === 0) - ) - return malformedSignature( - `function pointer calling convention 0x${header.first.toString(16)} does not head a method signature`, - cursor + 1, - ); + const header = readMethodSignatureHeader( + blob, + cursor + 1, + "function-pointer", + ); const convention = callingConvention(header.first); const head = header.genericParameterCount === null @@ -602,22 +649,81 @@ const callingConvention = (value: number): string => { return flags.length === 0 ? name : `${name} ${flags.join(" ")}`; }; -export const signature = ( +/** + * Decode a whole TypeSpec blob, or a MethodSpec instantiation (GENERICINST, + * argument count, Types), with the member-signature grammar and return the + * rendered types. A malformed or unsupported blob throws invalid-blob or + * unsupported-signature under the owning row, so coverage records it. + */ +export const specSignature = ( + layout: ManagedMetadataLayout, blob: Buffer, - rawSha256?: string, -): ManagedSignature => { - const raw = { - raw_length: blob.length, - raw_sha256: rawSha256 ?? sha256Bytes(blob), - }; + owner: { + readonly form: "type" | "instantiation"; + readonly scope: string; + readonly offset: number; + }, +): readonly string[] => { + const budget = managedDecodeBudget(layout); + // Rendering is transient, so charge it as member signatures are charged. + budget.admitTransient(blob.length * 256, owner.offset); + try { + let cursor = 0; + let count = 1; + if (owner.form === "instantiation") { + if (blob[0] !== 0x0a) + malformedSignature("MethodSpec instantiation is not GENERICINST", 0); + const header = readCompressed(blob, 1); + if (header.value === 0) + malformedSignature("MethodSpec instantiation has no type arguments", 1); + count = header.value; + cursor = header.next; + } + const types: string[] = []; + // Each Type consumes at least one byte, so a false count fails on truncation. + for (let index = 0; index < count; index += 1) { + const type = readTypeSignature(blob, cursor, "type", layout); + types.push(type.value); + cursor = type.next; + } + if (cursor !== blob.length) + malformedSignature("Trailing signature data", cursor); + return types; + } catch (cause: unknown) { + if ( + !(cause instanceof ManagedReaderFailure) || + (cause.issue.code !== "invalid-blob" && + cause.issue.code !== "invalid-row" && + cause.issue.code !== "unsupported-signature") + ) + throw cause; + throw managedFailure( + cause.issue.code, + owner.scope, + cause.issue.detail, + owner.offset, + ); + } finally { + budget.releaseTransient(); + } +}; + +type DecodedSignature = Omit; + +/** + * Decode a field or member method signature. A malformed or unsupported blob + * decodes as kind `unknown` with its reason. + */ +const decodeSignature = (blob: Buffer): DecodedSignature => { try { if (blob.length === 0) throw managedFailure("invalid-blob", "signature", "empty signature", 0); const first = blob[0] ?? 0; - if ((first & 0x0f) === 6) { + // FieldSig (II.23.2.4) is exactly FIELD; a flagged FIELD header is not a + // method header either, so it is malformed below. + if (first === 0x06) { const fieldType = readTypeSignature(blob, 1, "field"); return { - ...raw, kind: "field", parse_status: fieldType.next === blob.length ? "decoded" : "partial", calling_convention: callingConvention(first), @@ -630,7 +736,7 @@ export const signature = ( fieldType.next === blob.length ? null : "Trailing signature data", }; } - const header = readMethodSignatureHeader(blob, 0); + const header = readMethodSignatureHeader(blob, 0, "member"); const returnType = readTypeSignature(blob, header.next, "return"); let offset = returnType.next; const parameters: string[] = []; @@ -640,7 +746,6 @@ export const signature = ( offset = parameter.next; } return { - ...raw, kind: "method", parse_status: offset === blob.length ? "decoded" : "partial", calling_convention: callingConvention(first), @@ -659,7 +764,6 @@ export const signature = ( ) throw cause; return { - ...raw, kind: "unknown", parse_status: cause.issue.code === "unsupported-signature" @@ -675,3 +779,41 @@ export const signature = ( }; } }; + +/** + * Generic parameter count of a method signature that decodes completely, 0 + * when it is not generic, or null when the blob is not a decodable method + * signature. The rendered decode is transient. + */ +export const methodGenericArity = ( + layout: ManagedMetadataLayout, + blob: Buffer, +): number | null => { + const budget = managedDecodeBudget(layout); + budget.admitTransient(blob.length * 256, layout.blob.offset); + try { + return signatureGenericArity(decodeSignature(blob)); + } finally { + budget.releaseTransient(); + } +}; + +/** + * Generic parameter count of a completely decoded method signature, 0 when it + * is not generic, or null for any other signature. + */ +export const signatureGenericArity = ( + decoded: DecodedSignature, +): number | null => + decoded.kind === "method" && decoded.parse_status === "decoded" + ? (decoded.generic_parameter_count ?? 0) + : null; + +export const signature = ( + blob: Buffer, + rawSha256?: string, +): ManagedSignature => ({ + raw_length: blob.length, + raw_sha256: rawSha256 ?? sha256Bytes(blob), + ...decodeSignature(blob), +}); diff --git a/src/dotnet/ManagedMemberInstructionDecoder.ts b/src/dotnet/ManagedMemberInstructionDecoder.ts index f638e3e83..01c2b7814 100644 --- a/src/dotnet/ManagedMemberInstructionDecoder.ts +++ b/src/dotnet/ManagedMemberInstructionDecoder.ts @@ -12,6 +12,7 @@ const PREFIX_OPCODES = new Set([ "tail.", "constrained.", "readonly.", + "no.", ]); const OPCODES = new Map< @@ -232,6 +233,7 @@ const OPCODES = new Map< [0xfe16, { name: "constrained.", operand: "type" }], [0xfe17, { name: "cpblk", operand: "none" }], [0xfe18, { name: "initblk", operand: "none" }], + [0xfe19, { name: "no.", operand: "short-i" }], [0xfe1a, { name: "rethrow", operand: "none" }], [0xfe1c, { name: "sizeof", operand: "type" }], [0xfe1d, { name: "refanytype", operand: "none" }], @@ -265,6 +267,10 @@ type OperandResult = { type OperandReader = (il: Buffer, offset: number) => OperandResult; +// String(-0) loses a CIL constant's sign in both anchors and normalized hashes. +const floatingPointOperand = (value: number): string => + Object.is(value, -0) ? "-0" : String(value); + const FIXED_OPERAND_READERS = new Map([ [ "none", @@ -315,7 +321,7 @@ const FIXED_OPERAND_READERS = new Map([ (il, offset) => ({ next: offset + 4, anchorKind: "constant", - value: String(il.readFloatLE(offset)), + value: floatingPointOperand(il.readFloatLE(offset)), }), ], [ @@ -323,7 +329,7 @@ const FIXED_OPERAND_READERS = new Map([ (il, offset) => ({ next: offset + 8, anchorKind: "constant", - value: String(il.readDoubleLE(offset)), + value: floatingPointOperand(il.readDoubleLE(offset)), }), ], [ @@ -380,6 +386,22 @@ const readOperand = ( return { next: offset, anchorKind: "unknown", value: null }; }; +/** An operand cut off by the end of the IL body has no decodable value. */ +const readOperandWithin = ( + il: Buffer, + offset: number, + kind: string, +): OperandResult | undefined => { + try { + return readOperand(il, offset, kind); + } catch (cause: unknown) { + // Buffer reads past the IL body throw RangeError; report it as the + // instruction leaving the method body rather than as a Buffer message. + if (cause instanceof RangeError) return undefined; + throw cause; + } +}; + export const decodeInstructions = ( il: Buffer, admitInstruction?: (offset: number) => void, @@ -400,14 +422,22 @@ export const decodeInstructions = ( const start = offset; const first = il.readUInt8(offset); offset += 1; + if (first === 0xfe && offset >= il.length) { + issue = `Two-byte CIL opcode at IL offset ${String(start)} leaves method body`; + break; + } const code = first === 0xfe ? 0xfe00 + il.readUInt8(offset++) : first; const descriptor = OPCODES.get(code); if (descriptor === undefined) { issue = `Unsupported CIL opcode 0x${code.toString(16)} at IL offset ${String(start)}`; break; } - const operand = readOperand(il, offset, descriptor.operand); - if (operand.next < offset || operand.next > il.length) { + const operand = readOperandWithin(il, offset, descriptor.operand); + if ( + operand === undefined || + operand.next < offset || + operand.next > il.length + ) { issue = `Instruction ${descriptor.name} at IL offset ${String(start)} leaves method body`; break; } diff --git a/src/dotnet/ManagedMemberRows.ts b/src/dotnet/ManagedMemberRows.ts index f4a63db5a..201ff1c45 100644 --- a/src/dotnet/ManagedMemberRows.ts +++ b/src/dotnet/ManagedMemberRows.ts @@ -9,15 +9,14 @@ import type { ManagedParseIssue } from "../domain/managed/managedArtifact.js"; import type { ManagedMetadataLayout } from "./ManagedMetadataLayout.js"; import { signature, - type FieldCore, + signatureGenericArity, type ManagedCallEdge, type ManagedExportedType, type ManagedField, type ManagedFieldAccess, type ManagedMemberRef, type ManagedMethod, - type MemberRefCore, - type MethodCore, + type MemberCore, type TypeRange, } from "./ManagedMemberInspectorCore.js"; import { @@ -51,11 +50,11 @@ export const parseFields = ( selected: SelectedTypeTokens = null, ): { readonly fields: readonly ManagedField[]; - readonly core: ReadonlyMap; + readonly core: ReadonlyMap; readonly issues: readonly ManagedParseIssue[]; } => { const fields: ManagedField[] = []; - const core = new Map(); + const core = new Map(); const issues: ManagedParseIssue[] = []; const table = layout.table(4); const ownerRow = readManagedValue( @@ -78,7 +77,12 @@ export const parseFields = ( : (readManagedValue(() => ownerRow(row), issues) ?? 0); const declared = ranges[owner - 1]; const token = metadataToken(4, row); - core.set(token, { token, name }); + core.set(token, { + token, + name, + declaringType: declared?.fullName ?? null, + kind: "field", + }); if (!isSelected(selected, declared)) return; const sig = readMetadataBlob( bytes, @@ -174,13 +178,14 @@ export const parseExportedTypes = ( export const parseMemberRefs = ( bytes: Buffer, layout: ManagedMetadataLayout, + typeName: (token: string | null) => string | null, ): { readonly refs: readonly ManagedMemberRef[]; - readonly core: ReadonlyMap; + readonly core: ReadonlyMap; readonly issues: readonly ManagedParseIssue[]; } => { const refs: ManagedMemberRef[] = []; - const core = new Map(); + const core = new Map(); const issues: ManagedParseIssue[] = []; const table = layout.table(10); for (let row = 1; row <= (table?.rowCount ?? 0); row += 1) { @@ -218,19 +223,29 @@ export const parseMemberRefs = ( ? "MemberRef parent coded index 0x0 is null, but the Class column must reference a row" : `MemberRef parent coded index 0x${parentRaw.toString(16)} is invalid: ${parentReason}`, }); + const parentToken = metadataCodedToken( + parentRaw, + 3, + [2, 1, 26, 6, 27], + layout.rowCounts, + ); + const parentName = typeName(parentToken); + const decoded = budgetedSignature(sig, layout); refs.push({ token, row_offset: cursor.start, - parent_token: metadataCodedToken( - parentRaw, - 3, - [2, 1, 26, 6, 27], - layout.rowCounts, - ), + parent_token: parentToken, + parent_name: parentName, name, - signature: budgetedSignature(sig, layout), + signature: decoded, + }); + core.set(token, { + token, + name, + declaringType: parentName, + kind: decoded.kind, + genericArity: signatureGenericArity(decoded), }); - core.set(token, { token, name }); }, issues); } return { refs, core, issues }; @@ -252,11 +267,11 @@ export const parseMethods = ({ selected = null, }: ParseMethodsInput): { readonly methods: readonly ManagedMethod[]; - readonly core: ReadonlyMap; + readonly core: ReadonlyMap; readonly issues: readonly ManagedParseIssue[]; } => { const methods: ManagedMethod[] = []; - const core = new Map(); + const core = new Map(); const issues: ManagedParseIssue[] = []; const table = layout.table(6); const ownerRow = readManagedValue( @@ -281,11 +296,13 @@ export const parseMethods = ({ : (readManagedValue(() => ownerRow(row), issues) ?? 0); const declared = ranges[owner - 1]; const token = metadataToken(6, row); - core.set(token, { + const member = { token, name, declaringType: declared?.fullName ?? null, - }); + kind: "method", + } as const; + core.set(token, member); if (!isSelected(selected, declared)) return; const sig = readMetadataBlob( bytes, @@ -294,6 +311,11 @@ export const parseMethods = ({ layout.blob.size, ); cursor.readIndex(layout.tableIndexSize(8)); + const decoded = budgetedSignature(sig, layout); + core.set(token, { + ...member, + genericArity: signatureGenericArity(decoded), + }); methods.push({ token, row_offset: cursor.start, @@ -303,7 +325,7 @@ export const parseMethods = ({ rva, impl_flags: implFlags, flags, - signature: budgetedSignature(sig, layout), + signature: decoded, body: methodBody( bytes, pe, @@ -317,6 +339,15 @@ export const parseMethods = ({ return { methods, core, issues }; }; +/** + * Member kinds each operand kind may name. A MemberRef whose signature does + * not decode keeps its name on calls, as before, but is never named a field. + */ +const NAMEABLE_KINDS = new Map>([ + ["method", new Set(["method", "unknown"])], + ["field", new Set(["field"])], +]); + const targetKind = (token: string): ManagedCallEdge["target_kind"] => token.startsWith("0x06") ? "method-def" @@ -326,11 +357,22 @@ const targetKind = (token: string): ManagedCallEdge["target_kind"] => ? "method-spec" : "unknown"; +/** The member an operand names, if the operand kind can name that member's kind. */ +const namedTarget = ( + members: ReadonlyMap, + operandKind: string, + operand: string, +): MemberCore | undefined => { + const member = members.get(operand); + return member !== undefined && + NAMEABLE_KINDS.get(operandKind)?.has(member.kind) === true + ? member + : undefined; +}; + export const edges = ( methods: readonly ManagedMethod[], - methodCore: ReadonlyMap, - fieldCore: ReadonlyMap, - refCore: ReadonlyMap, + members: ReadonlyMap, budget: ManagedDecodeBudget, ): { readonly callEdges: readonly ManagedCallEdge[]; @@ -350,11 +392,10 @@ export const edges = ( const caller = derives ? callerName(method) : method.name; for (const anchor of method.body.anchors) { if (anchor.opcode === "ldtoken") continue; - if (anchor.operand_kind === "method" && anchor.operand !== null) { - const named = - methodCore.get(anchor.operand)?.name ?? - refCore.get(anchor.operand)?.name ?? - null; + const { operand } = anchor; + if (operand === null) continue; + const target = namedTarget(members, anchor.operand_kind, operand); + if (anchor.operand_kind === "method") { // Each derived record is admitted before the graph grows further. callEdges.push( admitManagedProjection( @@ -362,23 +403,25 @@ export const edges = ( caller_token: method.token, caller, opcode: anchor.opcode, - target_token: anchor.operand, - target_kind: targetKind(anchor.operand), - target_name: named, + target_token: operand, + target_kind: targetKind(operand), + target_name: target?.name ?? null, + target_declaring_type: target?.declaringType ?? null, }, budget, ), ); } - if (anchor.operand_kind === "field" && anchor.operand !== null) { + if (anchor.operand_kind === "field") { fieldAccesses.push( admitManagedProjection( { method_token: method.token, method: caller, opcode: anchor.opcode, - field_token: anchor.operand, - field_name: fieldCore.get(anchor.operand)?.name ?? null, + field_token: operand, + field_name: target?.name ?? null, + field_declaring_type: target?.declaringType ?? null, }, budget, ), diff --git a/src/dotnet/ManagedMethodSpecs.ts b/src/dotnet/ManagedMethodSpecs.ts new file mode 100644 index 000000000..057090862 --- /dev/null +++ b/src/dotnet/ManagedMethodSpecs.ts @@ -0,0 +1,154 @@ +import type { ManagedParseIssue } from "../domain/managed/managedArtifact.js"; +import { + methodGenericArity, + specSignature, + type ManagedMethod, + type MemberCore, +} from "./ManagedMemberInspectorCore.js"; +import type { ManagedMetadataLayout } from "./ManagedMetadataLayout.js"; +import { + metadataCodedToken, + metadataCodedTokenInvalidReason, + metadataRowCursor, + readMetadataBlob, +} from "./ManagedMetadataHeaps.js"; +import { managedFailure, readManagedValue } from "./ManagedReaderFailure.js"; + +/** + * Index fields, member references, and methods by token, then name each + * MethodSpec an inspected body calls after the generic method it instantiates, + * so every edge target resolves through one lookup. + */ +export const edgeTargets = ( + bytes: Buffer, + layout: ManagedMetadataLayout, + methods: readonly ManagedMethod[], + cores: readonly ReadonlyMap[], +): { + readonly members: ReadonlyMap; + readonly issues: readonly ManagedParseIssue[]; +} => { + const members = new Map(cores.flatMap((core) => [...core])); + const issues: ManagedParseIssue[] = []; + const seen = new Set(); + // Generic arity by method token, so instantiations of one method decode it once. + const arities = new Map(); + for (const method of methods) + for (const { operand } of method.body.anchors) { + if (operand?.startsWith("0x2b") !== true || seen.has(operand)) continue; + seen.add(operand); + readManagedValue( + () => + readMethodSpec({ bytes, layout, members, issues, arities }, operand), + issues, + ); + } + return { members, issues }; +}; + +/** + * Map one MethodSpec to its generic method, then check its instantiation + * decodes and matches that method's generic arity. A readable Method column + * keeps its name when the rest is malformed; the failure makes coverage partial. + */ +const readMethodSpec = ( + { + bytes, + layout, + members, + issues, + arities, + }: { + readonly bytes: Buffer; + readonly layout: ManagedMetadataLayout; + readonly members: Map; + readonly issues: ManagedParseIssue[]; + readonly arities: Map; + }, + token: string, +): void => { + const scope = `metadata.MethodSpec:${token}`; + const cursor = metadataRowCursor( + bytes, + layout, + 0x2b, + Number.parseInt(token.slice(4), 16), + ); + const methodRaw = cursor.readIndex(layout.codedIndexSize("MethodDefOrRef")); + const instantiation = cursor.readIndex(layout.blobIndexSize); + const methodReason = metadataCodedTokenInvalidReason( + methodRaw, + 1, + [6, 10], + layout.rowCounts, + ); + if (methodReason !== null || methodRaw === 0) + issues.push({ + code: "invalid-row", + scope, + offset: cursor.start, + detail: + methodReason === null + ? "MethodSpec method coded index 0x0 is null, but the Method column must reference a row" + : `MethodSpec method coded index 0x${methodRaw.toString(16)} is invalid: ${methodReason}`, + }); + const generic = metadataCodedToken(methodRaw, 1, [6, 10], layout.rowCounts); + const target = generic === null ? undefined : members.get(generic); + // A MethodSpec instantiates a method, so a field target never lends it a + // name, even to a field operand. + if (target !== undefined && target.kind !== "field") + members.set(token, { ...target, token }); + const types = specSignature( + layout, + readMetadataBlob(bytes, layout, instantiation, layout.blob.size), + { form: "instantiation", scope, offset: cursor.start }, + ); + if (generic === null) return; + // Prefer the arity the target's owner decoded; read the raw signature only + // for a target nothing decoded, such as a MethodDef outside the selection. + if (target?.genericArity === undefined && !arities.has(generic)) + arities.set( + generic, + methodGenericArity(layout, methodSignatureBlob(bytes, layout, generic)), + ); + const arity = + target?.genericArity === undefined + ? (arities.get(generic) ?? null) + : target.genericArity; + if (arity !== types.length) + throw managedFailure( + "invalid-row", + scope, + arity === null + ? "MethodSpec method does not have a decodable method signature" + : `MethodSpec supplies ${String(types.length)} type argument(s), but its method declares ${String(arity)} generic parameter(s)`, + cursor.start, + ); +}; + +/** The Signature column of a MethodDef or MemberRef row. */ +const methodSignatureBlob = ( + bytes: Buffer, + layout: ManagedMetadataLayout, + token: string, +): Buffer => { + const table = Number.parseInt(token.slice(2, 4), 16); + const cursor = metadataRowCursor( + bytes, + layout, + table, + Number.parseInt(token.slice(4), 16), + ); + if (table === 6) { + cursor.readUInt32(); + cursor.readUInt16(); + cursor.readUInt16(); + } else cursor.readIndex(layout.codedIndexSize("MemberRefParent")); + cursor.readIndex(layout.stringIndexSize); + return readMetadataBlob( + bytes, + layout, + cursor.readIndex(layout.blobIndexSize), + layout.blob.size, + ); +}; diff --git a/src/dotnet/ManagedNativeBoundaryHelpers.ts b/src/dotnet/ManagedNativeBoundaryHelpers.ts index fb81e1d28..e4b1a5960 100644 --- a/src/dotnet/ManagedNativeBoundaryHelpers.ts +++ b/src/dotnet/ManagedNativeBoundaryHelpers.ts @@ -18,7 +18,10 @@ import { } from "./ManagedMetadataHeaps.js"; import { managedTableRowCounts } from "./ManagedMetadataInventory.js"; import type { ManagedMetadataInventory } from "./ManagedMetadataInventory.js"; -import type { ManagedPeLayout } from "./ManagedPeReader.js"; +import { + managedCli32BitRequirement, + type ManagedPeLayout, +} from "./ManagedPeReader.js"; import { readManagedValue } from "./ManagedReaderFailure.js"; type ModuleRef = ManagedNativeBoundaryInspection["module_refs"][number]; @@ -380,7 +383,7 @@ export const cliNative = ( if (pe.cli === null) return null; return { il_only: (pe.cli.flags & 0x0000_0001) !== 0, - requires_32bit: (pe.cli.flags & 0x0000_0002) !== 0, + requires_32bit: managedCli32BitRequirement(pe.cli.flags) === "required", strong_name_signed: (pe.cli.flags & 0x0000_0008) !== 0, native_entry_point: (pe.cli.flags & 0x0000_0010) !== 0, ready_to_run_signature: pe.cli.readyToRunSignature, diff --git a/src/dotnet/ManagedPe.fixture.ts b/src/dotnet/ManagedPe.fixture.ts index 25f819925..7ea2b06b4 100644 --- a/src/dotnet/ManagedPe.fixture.ts +++ b/src/dotnet/ManagedPe.fixture.ts @@ -29,6 +29,16 @@ interface ManagedPeFixtureOptions { readonly moduleRowCount?: number; readonly assemblyRowCount?: number; readonly memberRefParentRaw?: number; + readonly memberRefName?: string; + readonly memberRefSignature?: Buffer; + /** Raw MethodDefOrRef coded values, one MethodSpec row each. */ + readonly methodSpecMethods?: readonly number[]; + /** Instantiation blob shared by every MethodSpec row. */ + readonly methodSpecInstantiation?: Buffer; + /** TypeSpec signature blobs, one TypeSpec row each. */ + readonly typeSpecSignatures?: readonly Buffer[]; + /** Raw ResolutionScope coded value of the TypeRef row; null by default. */ + readonly typeRefScopeRaw?: number; readonly mvid?: Buffer; readonly pinvoke?: { readonly importName?: string; @@ -174,10 +184,10 @@ const typeRefRow = ( name: number, namespace: number, stringSize: MetadataIndexSize, - resolutionScopeSize: MetadataIndexSize, + resolutionScope: Buffer, ): Buffer => Buffer.concat([ - metadataIndex(0, resolutionScopeSize), + resolutionScope, metadataIndex(name, stringSize), metadataIndex(namespace, stringSize), ]); @@ -524,6 +534,8 @@ interface FixtureMetadataRowContext { readonly methodSignature: number; readonly constructorSignature: number; readonly attributeBlob: number; + readonly typeSpecSignatures: readonly number[]; + readonly methodSpecInstantiation: number; }; readonly heapSizes: { readonly strings: MetadataIndexSize; @@ -595,6 +607,37 @@ const exportedTypeTable = ( ], ]; +const specTables = ( + options: ManagedPeFixtureOptions, + indexes: FixtureMetadataRowContext["indexes"], + heapSizes: FixtureMetadataRowContext["heapSizes"], + widths: FixtureIndexWidths, +): readonly (readonly [number, readonly Buffer[]])[] => [ + ...(indexes.typeSpecSignatures.length === 0 + ? [] + : ([ + [ + 27, + indexes.typeSpecSignatures.map((signature) => + metadataIndex(signature, heapSizes.blobs), + ), + ], + ] as const)), + ...(options.methodSpecMethods === undefined + ? [] + : ([ + [ + 43, + options.methodSpecMethods.map((method) => + Buffer.concat([ + metadataIndex(method, widths.coded([6, 10], 1)), + metadataIndex(indexes.methodSpecInstantiation, heapSizes.blobs), + ]), + ), + ], + ] as const)), +]; + const fixtureIndexWidths = ({ options, referenceStringIndexes, @@ -616,10 +659,12 @@ const fixtureIndexWidths = ({ ], ...(indexes.pinvokeModuleName === null ? [] : ([[26, 1]] as const)), ...(indexes.pinvokeImportName === null ? [] : ([[28, 1]] as const)), + [27, indexes.typeSpecSignatures.length], [32, options.assemblyRowCount ?? 1], [35, referenceStringIndexes.length], [39, indexes.exportedTypes.length], [40, 1], + [43, options.methodSpecMethods?.length ?? 0], ]); const count = (table: number): number => rowCounts.get(table) ?? 0; const table = (index: number): MetadataIndexSize => indexSize(count(index)); @@ -673,7 +718,7 @@ const fixtureMetadataRows = ({ indexes.attributeName, indexes.attributeNamespace, heapSizes.strings, - widths.resolutionScope, + metadataIndex(options.typeRefScopeRaw ?? 0, widths.resolutionScope), ), ], ], @@ -780,6 +825,7 @@ const fixtureMetadataRows = ({ ), ], ...exportedTypeTable(options, indexes, heapSizes, widths), + ...specTables(options, indexes, heapSizes, widths), [ 40, [ @@ -795,6 +841,29 @@ const fixtureMetadataRows = ({ return rows; }; +/** Blob indexes for the MemberRef and optional spec rows; defaults leave the heap unchanged. */ +const referenceBlobIndexes = ( + blobs: BlobHeap, + options: ManagedPeFixtureOptions, +): Pick< + FixtureMetadataRowContext["indexes"], + "constructorSignature" | "typeSpecSignatures" | "methodSpecInstantiation" +> => ({ + constructorSignature: blobs.add( + options.memberRefSignature ?? Buffer.from([0x20, 0x01, 0x01, 0x0e]), + ), + typeSpecSignatures: (options.typeSpecSignatures ?? []).map((signature) => + blobs.add(signature), + ), + // Defaults to GENERICINST with one string type argument. + methodSpecInstantiation: + options.methodSpecMethods === undefined + ? 0 + : blobs.add( + options.methodSpecInstantiation ?? Buffer.from([0x0a, 0x01, 0x0e]), + ), +}); + const buildManagedFixtureMetadata = ( options: ManagedPeFixtureOptions, methodRva: number, @@ -805,7 +874,7 @@ const buildManagedFixtureMetadata = ( const assemblyName = strings.add("Fixture.Managed"); const attributeName = strings.add("TargetFrameworkAttribute"); const attributeNamespace = strings.add("System.Runtime.Versioning"); - const constructorName = strings.add(".ctor"); + const constructorName = strings.add(options.memberRefName ?? ".ctor"); const typeName = strings.add(options.typeName ?? "Program"); const additionalTypeNames = Array.from( { length: options.additionalEmptyTypes ?? 0 }, @@ -840,7 +909,7 @@ const buildManagedFixtureMetadata = ( const methodSignature = blobs.add( options.methodSignature ?? Buffer.from([0x00, 0x00, 0x01]), ); - const constructorSignature = blobs.add(Buffer.from([0x20, 0x01, 0x01, 0x0e])); + const referenceBlobs = referenceBlobIndexes(blobs, options); const attributeBlob = blobs.add( fixedStringAttributeBlob( options.targetFramework === null @@ -873,8 +942,8 @@ const buildManagedFixtureMetadata = ( tokenBlob, fieldSignature, methodSignature, - constructorSignature, attributeBlob, + ...referenceBlobs, }, heapSizes: { strings: stringSize, blobs: blobSize, guids: guidSize }, }); diff --git a/src/dotnet/ManagedPeReader.ts b/src/dotnet/ManagedPeReader.ts index 2f9bc3151..9a30c007b 100644 --- a/src/dotnet/ManagedPeReader.ts +++ b/src/dotnet/ManagedPeReader.ts @@ -31,6 +31,20 @@ export interface ManagedCliHeader { readonly readyToRunSignature: boolean; } +/** + * How the CLI header constrains process bitness. 32BITREQUIRED and + * 32BITPREFERRED are one two-bit field: required alone is x86-only, both set + * is platform-neutral code that prefers a 32-bit process, and preferred alone + * is reserved and constrains nothing. + */ +export const managedCli32BitRequirement = ( + flags: number, +): "required" | "preferred" | "none" => { + const pair = flags & 0x0002_0002; + if (pair === 0x0000_0002) return "required"; + return pair === 0x0002_0002 ? "preferred" : "none"; +}; + export interface ManagedPeLayout { readonly machine: number; readonly architecture: "x86" | "x86_64" | "arm" | "arm64"; diff --git a/src/dotnet/ManagedTypeNames.ts b/src/dotnet/ManagedTypeNames.ts new file mode 100644 index 000000000..ab3b34c8f --- /dev/null +++ b/src/dotnet/ManagedTypeNames.ts @@ -0,0 +1,137 @@ +import { managedDecodeBudget } from "./ManagedDecodeBudget.js"; +import type { ManagedParseIssue } from "../domain/managed/managedArtifact.js"; +import { + fullName, + readCompressed, + specSignature, + type TypeRange, +} from "./ManagedMemberInspectorCore.js"; +import { + metadataRowOffset, + type ManagedMetadataLayout, +} from "./ManagedMetadataLayout.js"; +import { + metadataCodedTokenInvalidReason, + metadataRowCursor, + metadataToken, + readMetadataBlob, + readMetadataString, +} from "./ManagedMetadataHeaps.js"; +import { managedFailure, readManagedValue } from "./ManagedReaderFailure.js"; + +/** + * Name the type a TypeDef, TypeRef, or generic-instantiation TypeSpec token + * denotes, in the namespace-qualified form TypeDef `full_name` uses, once per + * token. Other decoded TypeSpecs stay null: the type is unknown. Malformed or + * unsupported rows and blobs are reported to `issues` and also stay null, + * except that a TypeRef with an invalid ResolutionScope keeps its readable name. + */ +export const typeNameLookup = ( + bytes: Buffer, + layout: ManagedMetadataLayout, + ranges: readonly TypeRange[], + issues: ManagedParseIssue[], +): ((token: string | null) => string | null) => { + const names = new Map(); + const checkedScopes = new Set(); + const validateScopeChain = (row: number): void => { + const path = new Set(); + let current = row; + while (!checkedScopes.has(current)) { + const cursor = metadataRowCursor(bytes, layout, 1, current); + if (path.has(current)) { + issues.push({ + code: "invalid-row", + scope: `metadata.TypeRef:${metadataToken(1, current)}`, + offset: cursor.start, + detail: "TypeRef ResolutionScope chain contains a cycle", + }); + break; + } + // Admit the chain's visited entry and retained validation-cache entry. + managedDecodeBudget(layout).reserve(128, cursor.start); + path.add(current); + const raw = cursor.readIndex(layout.codedIndexSize("ResolutionScope")); + const reason = metadataCodedTokenInvalidReason( + raw, + 2, + [0, 26, 35, 1], + layout.rowCounts, + ); + if (reason !== null) { + issues.push({ + code: "invalid-row", + scope: `metadata.TypeRef:${metadataToken(1, current)}`, + offset: cursor.start, + detail: `TypeRef ResolutionScope coded index 0x${raw.toString(16)} is invalid: ${reason}`, + }); + break; + } + // Null scopes are legal; non-TypeRef scopes terminate local validation. + if (raw === 0 || (raw & 3) !== 3) break; + current = Math.floor(raw / 4); + } + for (const visited of path) checkedScopes.add(visited); + }; + const definitionName = (table: 1 | 2, row: number): string | null => { + if (table === 2) { + metadataRowOffset(layout, 2, row); + return ranges[row - 1]?.fullName ?? null; + } + const cursor = metadataRowCursor(bytes, layout, 1, row); + cursor.readIndex(layout.codedIndexSize("ResolutionScope")); + readManagedValue(() => validateScopeChain(row), issues); + const name = cursor.readIndex(layout.stringIndexSize); + const namespace = cursor.readIndex(layout.stringIndexSize); + return fullName( + readMetadataString(bytes, layout, namespace, layout.strings.size), + readMetadataString(bytes, layout, name, layout.strings.size), + ); + }; + const specName = (row: number): string | null => { + const cursor = metadataRowCursor(bytes, layout, 27, row); + const spec = readMetadataBlob( + bytes, + layout, + cursor.readIndex(layout.blobIndexSize), + layout.blob.size, + ); + const owner = { + form: "type", + scope: `metadata.TypeSpec:${metadataToken(27, row)}`, + offset: cursor.start, + } as const; + specSignature(layout, spec, owner); + if (spec[0] !== 0x15) return null; + // GENERICINST (CLASS | VALUETYPE) TypeDefOrRefEncoded: never a TypeSpec, + // so naming never recurses. + const encoded = readCompressed(spec, 2).value; + const table = ([2, 1] as const)[encoded & 3]; + if (table === undefined) + throw managedFailure( + "invalid-blob", + owner.scope, + "Generic instantiation does not name a TypeDef or TypeRef", + owner.offset, + ); + return definitionName(table, Math.floor(encoded / 4)); + }; + return (token) => { + if (token === null) return null; + if (names.has(token)) return names.get(token) ?? null; + const table = Number.parseInt(token.slice(2, 4), 16); + const row = Number.parseInt(token.slice(4), 16); + const name = + readManagedValue( + () => + table === 27 + ? specName(row) + : table === 1 || table === 2 + ? definitionName(table, row) + : null, + issues, + ) ?? null; + names.set(token, name); + return name; + }; +}; diff --git a/src/evaluation/CodexAgentEval.ts b/src/evaluation/CodexAgentEval.ts index bac8d8191..55ec8333e 100644 --- a/src/evaluation/CodexAgentEval.ts +++ b/src/evaluation/CodexAgentEval.ts @@ -1,10 +1,11 @@ import { - deliveredCodexEvidence, evaluateKnownAnswers, + parseCodexToolCall, type FixtureClaimExpectation, type KnownAnswerAssessment, + type ParsedCodexToolCall, } from "./KnownAnswerEvaluation.js"; -import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js"; +import { canonicalJson } from "../domain/comparisonSemantics.js"; import { TOOL_CONTRACTS } from "../contracts/toolContracts.js"; /** One REA MCP invocation observed in a Codex JSONL transcript. */ @@ -57,17 +58,6 @@ const textValue = (value: unknown): string | undefined => const numberValue = (value: unknown): number => typeof value === "number" && Number.isFinite(value) ? value : 0; -const canonicalValue = (value: unknown): unknown => { - if (Array.isArray(value)) return value.map(canonicalValue); - const object = record(value); - if (object === undefined) return value; - return Object.fromEntries( - Object.entries(object) - .sort(([left], [right]) => compareUnicodeCodePoints(left, right)) - .map(([key, child]) => [key, canonicalValue(child)]), - ); -}; - const toolResultError = ( item: Record, ): Record | undefined => { @@ -113,29 +103,18 @@ const toolResultFailed = ( ); }; -const callFromEvent = ( - event: Record, +const callFromParsed = ( + parsed: ParsedCodexToolCall | undefined, ): CodexMcpCall | undefined => { - const item = record( - event.type === "item.completed" - ? event.item - : event.type === "mcp_tool_call" - ? event - : undefined, - ); - if (item?.type !== "mcp_tool_call") return undefined; - const tool = textValue(item.tool) ?? textValue(item.name); - if (tool === undefined) return undefined; - const error = toolResultError(item); + if (parsed === undefined) return undefined; + const error = toolResultError(parsed.item); return { - id: textValue(item.id) ?? null, - server: textValue(item.server) ?? textValue(item.server_name) ?? null, - tool, - arguments: item.arguments ?? item.input ?? {}, - evidenceIds: deliveredCodexEvidence(event).map( - ({ evidence_id }) => evidence_id, - ), - error: toolResultFailed(item, error), + id: parsed.id, + server: parsed.server, + tool: parsed.tool, + arguments: parsed.arguments, + evidenceIds: parsed.evidence.map(({ evidence_id }) => evidence_id), + error: toolResultFailed(parsed.item, error), errorCode: textValue(error?.code) ?? null, }; }; @@ -163,16 +142,20 @@ interface EvaluationState { outputTokens: number; } -const consumeEvent = (state: EvaluationState, value: unknown): void => { +const consumeEvent = ( + state: EvaluationState, + value: unknown, + parsedCall: ParsedCodexToolCall | undefined, +): void => { const event = record(value); if (event === undefined) return; const completedItem = event.type === "item.completed" ? event.item : undefined; - const call = callFromEvent(event); + const call = callFromParsed(parsedCall); if (call !== undefined) { const identity = call.id ?? - `${call.server ?? ""}:${call.tool}:${JSON.stringify(canonicalValue(call.arguments))}`; + `${call.server ?? ""}:${call.tool}:${canonicalJson(call.arguments)}`; if (!state.seenCallIds.has(identity)) { state.seenCallIds.add(identity); state.calls.push(call); @@ -216,7 +199,10 @@ export const evaluateCodexEvents = ( cachedInputTokens: 0, outputTokens: 0, }; - for (const event of events) consumeEvent(state, event); + const parsedCalls = events.map(parseCodexToolCall); + events.forEach((event, index) => + consumeEvent(state, event, parsedCalls[index]), + ); const reaCalls = state.calls.filter( ({ server }) => server === null || server.toLowerCase() === "rea", @@ -228,7 +214,7 @@ export const evaluateCodexEvents = ( // useful verification, not a duplicate analysis or unchanged failed retry. const context = call.tool === "binary_session" ? `${String(targetRevision)}:` : ""; - const signature = `${context}${call.tool}:${JSON.stringify(canonicalValue(call.arguments))}`; + const signature = `${context}${call.tool}:${canonicalJson(call.arguments)}`; signatures.set(signature, (signatures.get(signature) ?? 0) + 1); if ( !call.error && @@ -271,7 +257,7 @@ export const evaluateCodexEvents = ( finalMessage, ); const factualAssessment = evaluateKnownAnswers( - events, + parsedCalls, finalMessage, options.fixtureClaims, ); diff --git a/src/evaluation/KnownAnswerEvaluation.ts b/src/evaluation/KnownAnswerEvaluation.ts index 807308b59..c6e4af72b 100644 --- a/src/evaluation/KnownAnswerEvaluation.ts +++ b/src/evaluation/KnownAnswerEvaluation.ts @@ -3,7 +3,7 @@ import { z } from "zod"; import { parseEvidence, type Evidence } from "../domain/evidence.js"; import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js"; -import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js"; +import { canonicalJson } from "../domain/comparisonSemantics.js"; /** One closed fixture assertion and the producer facts that must support it. */ export interface FixtureClaimExpectation { @@ -104,23 +104,13 @@ const isRecord = (value: unknown): value is Record => typeof value === "object" && value !== null && !Array.isArray(value); const record = (value: unknown): Record | undefined => isRecord(value) ? value : undefined; -const canonicalValue = (value: JsonValue): JsonValue => { - if (Array.isArray(value)) return value.map(canonicalValue); - if (value === null || typeof value !== "object") return value; - return Object.fromEntries( - Object.entries(value) - .sort(([left], [right]) => compareUnicodeCodePoints(left, right)) - .map(([key, child]) => [key, canonicalValue(child)]), - ); -}; const sameJson = (left: unknown, right: unknown): boolean => { const parsedLeft = jsonValueSchema.safeParse(left); const parsedRight = jsonValueSchema.safeParse(right); return ( parsedLeft.success && parsedRight.success && - JSON.stringify(canonicalValue(parsedLeft.data)) === - JSON.stringify(canonicalValue(parsedRight.data)) + canonicalJson(parsedLeft.data) === canonicalJson(parsedRight.data) ); }; @@ -168,6 +158,16 @@ interface DeliveredEvidence { readonly conflicts: Set; } +/** One completed tool call, with aliases normalized and successful Evidence authenticated. */ +export interface ParsedCodexToolCall { + readonly item: Record; + readonly id: string | null; + readonly server: string | null; + readonly tool: string; + readonly arguments: JsonValue; + readonly evidence: readonly Evidence[]; +} + const consistentAliases = ( value: Record, left: string, @@ -177,30 +177,14 @@ const consistentAliases = ( value[right] === undefined || value[left] === value[right]; -const completedCall = (event: unknown): Record | undefined => { - const value = record(event); - const item = - value?.type === "item.completed" - ? record(value.item) - : value?.type === "mcp_tool_call" && value.status === "completed" - ? value - : undefined; - if (item?.type !== "mcp_tool_call") return undefined; - const server = item.server ?? item.server_name; - if ( - typeof server !== "string" || - server.toLowerCase() !== "rea" || - (item.status !== undefined && item.status !== "completed") || - (item.error !== undefined && item.error !== null) - ) - return undefined; - if ( - !consistentAliases(item, "server", "server_name") || - !consistentAliases(item, "tool", "name") - ) - return undefined; - return item; -}; +const consistentJsonFields = ( + value: Record, + left: string, + right: string, +): boolean => + value[left] === undefined || + value[right] === undefined || + sameJson(value[left], value[right]); const consistentJsonAliases = ( value: Record, @@ -215,6 +199,7 @@ const consistentJsonAliases = ( const evidenceFromCall = ( item: Record, + tool: string, ): Evidence | undefined => { if (!consistentJsonAliases(item, "result", "output")) return undefined; const result = record(item.result ?? item.output); @@ -254,22 +239,42 @@ const evidenceFromCall = ( // Transcript output is untrusted; invalid Evidence cannot support a claim. return undefined; } - if ((item.tool ?? item.name) !== evidence.operation) return undefined; + if (tool !== evidence.operation) return undefined; return evidence; }; -/** Authenticate completed REA deliveries for both citations and fixture grading. */ -export const deliveredCodexEvidence = (event: unknown): readonly Evidence[] => { - const item = completedCall(event); - if (item === undefined) return []; - const evidence = evidenceFromCall(item); - if (evidence === undefined) return []; +const completedToolItem = ( + event: unknown, +): Record | undefined => { + const value = record(event); + const wrapped = value?.type === "item.completed"; + const item = wrapped + ? record(value.item) + : value?.type === "mcp_tool_call" + ? value + : undefined; + if (item?.type !== "mcp_tool_call") return undefined; + const terminalStatus = + item.status === "completed" || item.status === "failed"; + if ( + (wrapped + ? item.status !== undefined && !terminalStatus + : !terminalStatus) || + !consistentAliases(item, "server", "server_name") || + !consistentAliases(item, "tool", "name") || + !consistentJsonFields(item, "arguments", "input") + ) + return undefined; + return item; +}; + +const artifactDelivery = (evidence: Evidence): readonly Evidence[] => { const records = [evidence]; - // Artifact inspection delivers its inventory Evidence inline. Bare links and - // arbitrary IDs in normalized facts are references, not delivered records. if (evidence.operation !== "inspect_artifact") return records; const substeps = record(evidence.normalized_result)?.substeps; if (!Array.isArray(substeps)) return records; + // Artifact inspection delivers linked inventory Evidence inline; bare IDs in + // normalized facts are references, not authenticated deliveries. for (const value of substeps) { const substep = record(value); if ( @@ -293,13 +298,56 @@ export const deliveredCodexEvidence = (event: unknown): readonly Evidence[] => { return records; }; +/** Parse one completed transcript call and authenticate any successful REA Evidence. */ +export const parseCodexToolCall = ( + event: unknown, +): ParsedCodexToolCall | undefined => { + const item = completedToolItem(event); + if (item === undefined) return undefined; + const status = item.status; + const server = item.server ?? item.server_name; + const tool = item.tool ?? item.name; + if ( + ((item.server !== undefined || item.server_name !== undefined) && + typeof server !== "string") || + typeof tool !== "string" || + tool.length === 0 + ) + return undefined; + const argumentsValue = + item.arguments !== undefined + ? item.arguments + : item.input !== undefined + ? item.input + : {}; + const parsedArguments = jsonValueSchema.safeParse(argumentsValue); + if (!parsedArguments.success) return undefined; + const parsed: ParsedCodexToolCall = { + item, + id: typeof item.id === "string" && item.id.length > 0 ? item.id : null, + server: typeof server === "string" ? server : null, + tool, + arguments: parsedArguments.data, + evidence: [], + }; + if ( + parsed.server?.toLowerCase() !== "rea" || + status === "failed" || + (item.error !== undefined && item.error !== null) + ) + return parsed; + const evidence = evidenceFromCall(item, tool); + if (evidence === undefined) return parsed; + return { ...parsed, evidence: artifactDelivery(evidence) }; +}; + const collectDeliveredEvidence = ( - events: readonly unknown[], + calls: readonly (ParsedCodexToolCall | undefined)[], ): DeliveredEvidence => { const records = new Map(); const conflicts = new Set(); - for (const event of events) { - for (const evidence of deliveredCodexEvidence(event)) { + for (const call of calls) { + for (const evidence of call?.evidence ?? []) { const previous = records.get(evidence.evidence_id); if (previous !== undefined && !sameJson(previous, evidence)) conflicts.add(evidence.evidence_id); @@ -401,7 +449,7 @@ const assessClaim = ( /** Grade closed answers only against configured fixture truth and delivered authenticated Evidence. */ export const evaluateKnownAnswers = ( - events: readonly unknown[], + calls: readonly (ParsedCodexToolCall | undefined)[], finalMessage: string, expectations: readonly FixtureClaimExpectation[] | undefined, ): KnownAnswerAssessment => { @@ -441,7 +489,7 @@ export const evaluateKnownAnswers = ( manifestFailures.push(`unknown_claim:${claim.id}`); submitted.set(claim.id, claim); } - const delivered = collectDeliveredEvidence(events); + const delivered = collectDeliveredEvidence(calls); const claims = expectations.map((expected) => assessClaim(expected, submitted.get(expected.id), delivered), ); diff --git a/src/evm/EvmoleFailures.ts b/src/evm/EvmoleFailures.ts index 2677a3e71..63c6578aa 100644 --- a/src/evm/EvmoleFailures.ts +++ b/src/evm/EvmoleFailures.ts @@ -39,6 +39,9 @@ export const evmInterfaceFailure = ( limiter = "/usr/bin/prlimit", fileFailure?: EvmFileSizeFailureEvidence, ): AnalysisError => { + const cleanup = + cause instanceof ArtifactReaderFailure ? cause.cleanup : undefined; + const cleanupOptions = cleanup === undefined ? {} : { cleanup }; if (cause instanceof AnalysisError) return cause; if (cause instanceof OwnedCommandFailure) { const outputOptions = @@ -64,12 +67,15 @@ export const evmInterfaceFailure = ( { operation: "inspect_evm_interface", cause }, ); if (cause.reason === "cancelled") - return new AnalysisCancelledError("inspect_evm_interface", outputOptions); + return new AnalysisCancelledError("inspect_evm_interface", { + ...outputOptions, + ...cleanupOptions, + }); if (cause.reason === "timeout") return new AnalysisTimeoutError( "inspect_evm_interface", EVM_INTERFACE_LIMITS.timeoutMs, - outputOptions, + { ...outputOptions, ...cleanupOptions }, ); if (cause.reason === "output-limit") return new AnalysisOutputError( @@ -122,25 +128,33 @@ export const evmInterfaceFailure = ( } if (cause instanceof ArtifactReaderFailure) { if (cause.reason === "cancelled") - return new AnalysisCancelledError("inspect_evm_interface"); + return new AnalysisCancelledError("inspect_evm_interface", { + cause, + ...cleanupOptions, + }); if (cause.reason === "integrity") return new AnalysisArtifactChangedError( "inspect_evm_interface", path, cause.message, - { cause }, + { cause, ...cleanupOptions }, ); if ( phase === "artifact-read" && ["limit", "path", "format"].includes(cause.reason) ) - return new AnalysisInputError("inspect_evm_interface", { cause }, [ - { - path: ["path"], - reason: cause.reason === "limit" ? "out_of_range" : "invalid_format", - message: cause.message, - }, - ]); + return new AnalysisInputError( + "inspect_evm_interface", + { cause, ...cleanupOptions }, + [ + { + path: ["path"], + reason: + cause.reason === "limit" ? "out_of_range" : "invalid_format", + message: cause.message, + }, + ], + ); } if (phase === "artifact-read" && cause instanceof Error && "code" in cause) { if (cause.code === "EACCES" || cause.code === "EPERM") @@ -148,22 +162,27 @@ export const evmInterfaceFailure = ( "inspect_evm_interface", path, cause.code, - { cause }, + { cause, ...cleanupOptions }, ); if (cause.code === "ENOENT" || cause.code === "ENOTDIR") - return new AnalysisInputError("inspect_evm_interface", { cause }, [ - { - path: ["path"], - reason: "invalid_value", - message: `Selected bytecode carrier could not be read (${String(cause.code)}): ${path}.`, - }, - ]); + return new AnalysisInputError( + "inspect_evm_interface", + { cause, ...cleanupOptions }, + [ + { + path: ["path"], + reason: "invalid_value", + message: `Selected bytecode carrier could not be read (${String(cause.code)}): ${path}.`, + }, + ], + ); } return new ProviderAdapterError( EVMOLE_PROVIDER_IDENTITY.id, "inspect_evm_interface", { cause, + ...cleanupOptions, diagnostics: { phase, path, diff --git a/src/evm/EvmoleInterfaceProvider.ts b/src/evm/EvmoleInterfaceProvider.ts index 03731a92a..c59427e6a 100644 --- a/src/evm/EvmoleInterfaceProvider.ts +++ b/src/evm/EvmoleInterfaceProvider.ts @@ -6,11 +6,15 @@ import { fileURLToPath } from "node:url"; import { z } from "zod"; import { createAnalysisExecution, - type AnalysisExecution, type ExecutionOptions, } from "../application/AnalysisProvider.js"; -import type { EvmInterfacePort } from "../application/evm/EvmInterfacePort.js"; +import type { + EvmInterfaceExecution, + EvmInterfacePort, +} from "../application/evm/EvmInterfacePort.js"; import { readStableArtifact } from "../artifacts/readStableArtifact.js"; +import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js"; +import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js"; import { OwnedCommandFailure } from "../process/OwnedCommand.js"; import type { AnalysisError, @@ -76,6 +80,7 @@ type Launcher = NonNullable[2]>["launcher"]; /** Run unchanged EVMole/WASM in an owned worker; no target, chain or network execution. */ export class EvmoleInterfaceProvider implements EvmInterfacePort { + readonly #resources = new ArtifactResourceScope(); #tail: Promise = Promise.resolve(); #closing = false; #closePromise: Promise | undefined; @@ -98,11 +103,16 @@ export class EvmoleInterfaceProvider implements EvmInterfacePort { inspect( input: InspectEvmInterfaceInput, options?: ExecutionOptions, - ): Promise> { + ): Promise> { if (this.#closing) return Promise.resolve(err(this.#closedError())); - const operation = this.#tail.then(() => - this.#closing ? err(this.#closedError()) : this.#inspect(input, options), - ); + const operation = this.#tail.then(async () => { + if (this.#closing) return err(this.#closedError()); + try { + return await this.#resources.run(() => this.#inspect(input, options)); + } catch (cause: unknown) { + return err(evmInterfaceFailure(cause, "artifact-read", input.path)); + } + }); this.#tail = operation.then( () => undefined, () => undefined, @@ -113,7 +123,7 @@ export class EvmoleInterfaceProvider implements EvmInterfacePort { async #inspect( input: InspectEvmInterfaceInput, options?: ExecutionOptions, - ): Promise> { + ): Promise> { const pending = await this.#retire(); if (pending !== undefined) return err(pending); let root: Pick | undefined; @@ -123,7 +133,7 @@ export class EvmoleInterfaceProvider implements EvmInterfacePort { let workerLimits: EvmWorkerLimits | undefined; const limiter = this.environment.REA_EVM_PRLIMIT_COMMAND ?? "/usr/bin/prlimit"; - let result: Result; + let result: Result; try { if (options?.signal?.aborted) throw new AnalysisCancelledError(OPERATION); if (process.platform !== "linux" || process.arch !== "x64") @@ -178,6 +188,8 @@ export class EvmoleInterfaceProvider implements EvmInterfacePort { input.path, EVM_INTERFACE_LIMITS.inputBytes, options?.signal, + undefined, + this.#resources, ); phase = "worker"; root = await this.createRuntime(); @@ -241,15 +253,27 @@ export class EvmoleInterfaceProvider implements EvmInterfacePort { replyPath, EVM_INTERFACE_LIMITS.outputBytes, options?.signal, + undefined, + this.#resources, ); reply = replySchema.parse(JSON.parse(file.bytes.toString("utf8"))); } catch (cause: unknown) { + const cleanup = + cause instanceof ArtifactReaderFailure ? cause.cleanup : undefined; if (options?.signal?.aborted) - throw new AnalysisCancelledError(OPERATION, { capturedOutput }); + throw new AnalysisCancelledError(OPERATION, { + cause, + capturedOutput, + ...(cleanup === undefined ? {} : { cleanup }), + }); throw new AnalysisOutputError( OPERATION, `Owned interface worker reply failed for ${input.path}: ${cause instanceof Error ? cause.message : String(cause)}`, - { cause, capturedOutput }, + { + cause, + capturedOutput, + ...(cleanup === undefined ? {} : { cleanup }), + }, ); } if (!reply.ok) { @@ -353,6 +377,7 @@ export class EvmoleInterfaceProvider implements EvmInterfacePort { if (cause instanceof OwnedCommandFailure) processOwner = cause.cleanupOwner; let fileFailure: EvmFileSizeFailureEvidence | undefined; + let markerCleanupFailure: ArtifactReaderFailure | undefined; if ( cause instanceof OwnedCommandFailure && cause.snapshot?.exitCode === EVM_FILE_SIZE_FAILURE_EXIT && @@ -363,6 +388,9 @@ export class EvmoleInterfaceProvider implements EvmInterfacePort { const marker = await readStableArtifact( join(root.path, "resource.failure"), 1, + undefined, + undefined, + this.#resources, ); if (!marker.bytes.equals(Buffer.from("F"))) throw new Error( @@ -370,6 +398,14 @@ export class EvmoleInterfaceProvider implements EvmInterfacePort { ); fileFailure = { verified: true, failure: null }; } catch (markerFailure: unknown) { + if ( + markerFailure instanceof ArtifactReaderFailure && + markerFailure.cleanup !== undefined + ) + markerCleanupFailure = ArtifactReaderFailure.withCleanup( + cause, + markerFailure.cleanup, + ); fileFailure = { verified: false, failure: @@ -380,23 +416,36 @@ export class EvmoleInterfaceProvider implements EvmInterfacePort { } } result = err( - options?.signal?.aborted && - (cause === options.signal.reason || - (cause instanceof Error && cause.name === "AbortError")) - ? new AnalysisCancelledError( - OPERATION, - retainedOutput === undefined - ? undefined - : { capturedOutput: retainedOutput }, - ) - : evmInterfaceFailure( - cause, - phase, + markerCleanupFailure !== undefined + ? evmInterfaceFailure( + markerCleanupFailure, + "artifact-read", input.path, workerLimits, limiter, fileFailure, - ), + ) + : options?.signal?.aborted && + (cause === options.signal.reason || + (cause instanceof Error && cause.name === "AbortError")) + ? new AnalysisCancelledError(OPERATION, { + cause, + ...(retainedOutput === undefined + ? {} + : { capturedOutput: retainedOutput }), + ...(cause instanceof ArtifactReaderFailure && + cause.cleanup !== undefined + ? { cleanup: cause.cleanup } + : {}), + }) + : evmInterfaceFailure( + cause, + phase, + input.path, + workerLimits, + limiter, + fileFailure, + ), ); } if (processOwner !== undefined) { @@ -445,8 +494,37 @@ export class EvmoleInterfaceProvider implements EvmInterfacePort { this.#closing = true; this.#closePromise ??= this.#tail .then(async () => { - const failure = await this.#retire(); - if (failure !== undefined) throw failure; + const outcomes = await Promise.allSettled([ + this.#retire(), + this.#resources.close(), + ]); + const failures = outcomes.flatMap((outcome) => + outcome.status === "rejected" + ? [outcome.reason] + : outcome.value === undefined + ? [] + : [outcome.value], + ); + if (failures.length > 0) { + const resources = failures.flatMap((failure) => + failure instanceof ProviderCleanupError + ? failure.cleanupResources + : failure instanceof ArtifactReaderFailure && + failure.cleanup !== undefined + ? failure.cleanup.resources + : ["artifact-resources"], + ); + throw new ProviderCleanupError( + EVMOLE_PROVIDER_IDENTITY.id, + [...new Set(resources)], + { + failures: failures.map((failure) => + failure instanceof Error ? failure.message : String(failure), + ), + }, + { operation: OPERATION, cause: new AggregateError(failures) }, + ); + } }) .catch((cause: unknown) => { this.#closePromise = undefined; diff --git a/src/filesystem/RegularFile.ts b/src/filesystem/RegularFile.ts index a3037c728..4942e2e46 100644 --- a/src/filesystem/RegularFile.ts +++ b/src/filesystem/RegularFile.ts @@ -1,6 +1,6 @@ import { constants } from "node:fs"; import type { BigIntStats, Stats } from "node:fs"; -import { open, type FileHandle } from "node:fs/promises"; +import { lstat, open, type FileHandle } from "node:fs/promises"; import { OwnedFileHandle } from "./OwnedFileHandle.js"; @@ -74,6 +74,16 @@ export const openRegularFile = async ( options.signal?.throwIfAborted(); if (!metadata.isFile()) throw new NonRegularFileReadError(path, metadata.isDirectory()); + // O_NOFOLLOW is unavailable on Windows. Recheck the name only on hosts + // where open could have followed a symlink, while the descriptor is owned. + if (options.symlinks === "reject" && (constants.O_NOFOLLOW ?? 0) === 0) { + const named = await lstat(path); + options.signal?.throwIfAborted(); + if (named.isSymbolicLink()) + throw new NonRegularFileReadError(path, false); + if (!sameRegularFileState(metadata, named)) + throw new RegularFileChangedError(path); + } return handle; } catch (cause: unknown) { try { diff --git a/src/firmware/FirmwareCommand.ts b/src/firmware/FirmwareCommand.ts index b5db919f5..1604de901 100644 --- a/src/firmware/FirmwareCommand.ts +++ b/src/firmware/FirmwareCommand.ts @@ -176,11 +176,11 @@ export const runFirmwareCommand = async (context: { }; let failure: unknown; try { - const deadline = Date.now() + FIRMWARE_LIMITS.timeoutMs; + const deadline = performance.now() + FIRMWARE_LIMITS.timeoutMs; while (!(await supervisor.waitForOutputClose(100))) { checkDiagnostics(); if (signal.aborted) throw new AnalysisCancelledError(operation); - if (Date.now() >= deadline) + if (performance.now() >= deadline) throw new AnalysisTimeoutError(operation, FIRMWARE_LIMITS.timeoutMs); if (context.outputBudget !== undefined) await inventoryFirmwareOutput( diff --git a/src/firmware/FirmwareFiles.ts b/src/firmware/FirmwareFiles.ts index 10201905e..d20226e76 100644 --- a/src/firmware/FirmwareFiles.ts +++ b/src/firmware/FirmwareFiles.ts @@ -213,19 +213,15 @@ export const inventoryFirmwareOutput = async ( }); if (handle === undefined) continue; const names: string[] = []; - try { - for await (const entry of handle) { - throwIfFirmwareInventoryCancelled(signal, operation); - count++; - if (count > maxEntries) - throw new AnalysisOutputError( - operation, - `Staging entry budget exceeded (${maxEntries})`, - ); - names.push(entry.name); - } - } finally { - await handle.close().catch(() => undefined); + for await (const entry of handle) { + throwIfFirmwareInventoryCancelled(signal, operation); + count++; + if (count > maxEntries) + throw new AnalysisOutputError( + operation, + `Staging entry budget exceeded (${maxEntries})`, + ); + names.push(entry.name); } for (const name of names.sort()) { const path = join(directory, name); diff --git a/src/firmware/FirmwarePublication.ts b/src/firmware/FirmwarePublication.ts index bc53084b1..b2bc4ddd5 100644 --- a/src/firmware/FirmwarePublication.ts +++ b/src/firmware/FirmwarePublication.ts @@ -96,7 +96,7 @@ export const publishFirmwareExtraction = async (context: { published_bytes: files.reduce((sum, file) => sum + file.size, 0), }); context.signal.throwIfAborted(); - await tree.commit(); + await tree.commit(context.signal); return result; } catch (cause: unknown) { if (cause instanceof SafeOutputTreeCreationFailure) tree = cause.tree; diff --git a/src/flutter/FlutterBuildProvider.ts b/src/flutter/FlutterBuildProvider.ts index 2c088b2c9..9a6e6fe50 100644 --- a/src/flutter/FlutterBuildProvider.ts +++ b/src/flutter/FlutterBuildProvider.ts @@ -23,6 +23,7 @@ import { AnalysisUnsupportedTargetError, } from "../domain/analysisErrorCore.js"; import { AnalysisProtocolError } from "../domain/analysisErrorCore.js"; +import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js"; import { AnalysisError } from "../domain/analysisErrorBase.js"; import { err, ok, type Result } from "../domain/result.js"; import { @@ -759,7 +760,7 @@ export class FlutterBuildProvider { toolchain_capped: value.libflutter.toolchainCapped, }, })) - .sort((left, right) => left.abi.localeCompare(right.abi)); + .sort((left, right) => compareUnicodeCodePoints(left.abi, right.abi)); const flutterDetected = abis.some((entry) => entry.libapp.present); const partial = abis.some( @@ -871,7 +872,7 @@ export class FlutterBuildProvider { (left, right) => (preference.indexOf(left.abi) + 1 || preference.length + 1) - (preference.indexOf(right.abi) + 1 || preference.length + 1) || - left.abi.localeCompare(right.abi), + compareUnicodeCodePoints(left.abi, right.abi), )[0]! : candidates.find((candidate) => candidate.abi === abi); if (selected === undefined) diff --git a/src/frida/FridaInstrumentationManager.ts b/src/frida/FridaInstrumentationManager.ts index 406258e5a..22b38c442 100644 --- a/src/frida/FridaInstrumentationManager.ts +++ b/src/frida/FridaInstrumentationManager.ts @@ -83,8 +83,8 @@ interface ManagedSession extends ManagedRemoteConnection { readonly pid: number; readonly mode: "attach" | "spawn"; readonly scripts: Map; - readonly messages: JsonValue[]; - readonly messagesByScript: Map; + readonly messages: MessageFifo; + readonly messagesByScript: Map>; state: "paused" | "running" | "detached"; targetResumeObserved: boolean; resumeAfterDetachPending: boolean; @@ -277,7 +277,7 @@ export class FridaInstrumentationManager implements FridaInstrumentationPort { pid, mode: input.mode, scripts: new Map(), - messages: [], + messages: new MessageFifo(), messagesByScript: new Map(), state: input.mode === "spawn" ? "paused" : "running", targetResumeObserved: false, @@ -373,7 +373,7 @@ export class FridaInstrumentationManager implements FridaInstrumentationPort { .digest("hex"); const scriptId = randomUUID(); const scriptName = `rea-${scriptId}`; - managed.messagesByScript.set(scriptId, []); + managed.messagesByScript.set(scriptId, new MessageFifo()); let nativeScript: Script | undefined; const onMessage = (message: FridaMessage, data: Buffer | null): void => { this.#recordMessage(managed, scriptId, message, data); @@ -397,7 +397,7 @@ export class FridaInstrumentationManager implements FridaInstrumentationPort { sourceKind: source.sourceKind, sourcePath: source.sourceKind === "file" ? source.path : null, sourceSha256, - messages: [...(managed.messagesByScript.get(scriptId) ?? [])], + messages: managed.messagesByScript.get(scriptId)?.toArray() ?? [], messagesTruncated: managed.messagesTruncated, }); } catch (cause: unknown) { @@ -517,7 +517,9 @@ export class FridaInstrumentationManager implements FridaInstrumentationPort { scriptId: id, name, })), - messages: [...managed.messages], + messages: managed.messages + .toArray() + .map(({ observation }) => observation), messagesTruncated: managed.messagesTruncated, }; } @@ -851,30 +853,21 @@ export class FridaInstrumentationManager implements FridaInstrumentationPort { return; } while ( - managed.messages.length > 0 && + managed.messages.size > 0 && managed.capturedBytes + charge > MAX_SESSION_CAPTURE_BYTES ) { const removed = managed.messages.shift(); if (removed !== undefined) { - managed.capturedBytes -= Buffer.byteLength(JSON.stringify(removed)); - if ( - typeof removed === "object" && - removed !== null && - !Array.isArray(removed) && - typeof removed.script_id === "string" - ) { - const scriptMessages = managed.messagesByScript.get( - removed.script_id, - ); - const index = scriptMessages?.indexOf(removed) ?? -1; - if (scriptMessages !== undefined && index >= 0) { - scriptMessages.splice(index, 1); - } - } + managed.capturedBytes -= removed.charge; + // Both queues are in arrival order, so the globally oldest message is + // also the oldest one retained for its script. + const scriptMessages = managed.messagesByScript.get(removed.scriptId); + if (scriptMessages?.peek() === removed.observation) + scriptMessages.shift(); } managed.messagesTruncated = true; } - managed.messages.push(observation); + managed.messages.push({ observation, scriptId, charge }); managed.messagesByScript.get(scriptId)?.push(observation); managed.capturedBytes += charge; } @@ -1240,3 +1233,49 @@ const providerError = ( const errorMessage = (cause: unknown): string => cause instanceof Error ? cause.message : String(cause); + +interface CapturedMessage { + readonly observation: JsonValue; + readonly scriptId: string; + /** Serialized bytes charged against the session capture budget. */ + readonly charge: number; +} + +/** + * Arrival-ordered queue whose eviction advances a head offset. A full capture + * budget evicts on every message, and `Array#shift` would copy every retained + * message each time. + */ +class MessageFifo { + #items: (T | undefined)[] = []; + #head = 0; + + get size(): number { + return this.#items.length - this.#head; + } + + push(item: T): void { + this.#items.push(item); + } + + peek(): T | undefined { + return this.#items[this.#head]; + } + + shift(): T | undefined { + if (this.#head >= this.#items.length) return undefined; + const item = this.#items[this.#head]; + this.#items[this.#head] = undefined; + this.#head += 1; + // Compacting once half the slots are evicted keeps eviction amortized O(1). + if (this.#head * 2 >= this.#items.length) { + this.#items = this.#items.slice(this.#head); + this.#head = 0; + } + return item; + } + + toArray(): T[] { + return this.#items.slice(this.#head) as T[]; + } +} diff --git a/src/generatedPackageMetadata.ts b/src/generatedPackageMetadata.ts index 1af9203ad..bada60663 100644 --- a/src/generatedPackageMetadata.ts +++ b/src/generatedPackageMetadata.ts @@ -5,5 +5,5 @@ export const PACKAGE_METADATA = { serverSdkVersion: "2.3.1", clientSdkVersion: "2.3.1", coreSdkVersion: "2.3.1", - skillVersion: "40", + skillVersion: "41", } as const; diff --git a/src/ghidra/GhidraAnalysisProfile.ts b/src/ghidra/GhidraAnalysisProfile.ts index 60edb0971..81defabd3 100644 --- a/src/ghidra/GhidraAnalysisProfile.ts +++ b/src/ghidra/GhidraAnalysisProfile.ts @@ -123,9 +123,11 @@ export const resolveGhidraAnalysisProfile = async ( call_relationship_coverage: "resolved-call-reference-diagnostics-v1", string_inventory_evidence: "defined-data-coverage-v1", location_resolution: "explicit-address-exact-entry-symbol-first-v3", + procedure_file_offset_resolution: + "address-first-unique-target-source-mapping-v2", instruction_flow_evidence: "decoded-return-pcode-v1", instruction_references: "exact-address-incoming-v1", - function_boundary_observations: "ghidra-terminal-call-limitations-v1", + function_boundary_observations: "ghidra-terminal-call-limitations-v2", no_return_repair: installation.platform === "win32" ? "disabled-windows-p0" @@ -134,7 +136,7 @@ export const resolveGhidraAnalysisProfile = async ( installation.platform === "win32" ? "official-headless-script-v1" : "inspected-jvm-launch-support-v1", - load_image_evidence: "independent-raw-mapping-context-v2", + load_image_evidence: "independent-raw-mapping-context-v4", jump_table_evidence: "typed-case-default-blocks-v1", decompiler_jump_loads: true, loader: "BinaryLoader", @@ -212,9 +214,11 @@ export const resolveGhidraAnalysisProfile = async ( call_relationship_coverage: "resolved-call-reference-diagnostics-v1", string_inventory_evidence: "defined-data-coverage-v1", location_resolution: "explicit-address-exact-entry-symbol-first-v3", + procedure_file_offset_resolution: + "address-first-unique-target-source-mapping-v2", instruction_flow_evidence: "decoded-return-pcode-v1", instruction_references: "exact-address-incoming-v1", - function_boundary_observations: "ghidra-terminal-call-limitations-v1", + function_boundary_observations: "ghidra-terminal-call-limitations-v2", no_return_repair: installation.platform === "win32" ? "disabled-windows-p0" diff --git a/src/ghidra/GhidraAnalysisSeeds.test.ts b/src/ghidra/GhidraAnalysisSeeds.test.ts index 8af76a4da..37437ebf7 100644 --- a/src/ghidra/GhidraAnalysisSeeds.test.ts +++ b/src/ghidra/GhidraAnalysisSeeds.test.ts @@ -196,7 +196,7 @@ describe("Ghidra seed profile", () => { const host: GhidraInstallationHost = { platform: "linux", architecture: "x64", - readText: () => "application.version=12.1.4\n", + readText: () => "application.version=12.1.4\napplication.java.min=21\n", executable: () => true, probeJava: () => ({ version: "21.0.11", diff --git a/src/ghidra/GhidraDoctor.test.ts b/src/ghidra/GhidraDoctor.test.ts index e1ea7fe37..69b9acb62 100644 --- a/src/ghidra/GhidraDoctor.test.ts +++ b/src/ghidra/GhidraDoctor.test.ts @@ -1,9 +1,10 @@ -import { describe, expect, it } from "vitest"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { runDoctor, type DoctorHost } from "../application/Doctor.js"; import { createDoctorHostFixture } from "../application/Doctor.fixture.js"; import { providerRegistrationEnvironment } from "../application/SetupRegistrationEnvironment.js"; import { WINDOWS_NATIVE_AUTHORITY_UNAVAILABLE_REASON } from "../process/WindowsAuthority.js"; +import * as windowsAuthority from "../process/WindowsAuthority.js"; import { inspectGhidraInstallation, type GhidraInstallationHost, @@ -15,7 +16,7 @@ const INSTALL = "/opt/ghidra_12.1.4_PUBLIC"; const installationHost = ( overrides: Partial = {}, ): GhidraInstallationHost => ({ - readText: () => "application.version=12.1.4\n", + readText: () => "application.version=12.1.4\napplication.java.min=21\n", executable: () => true, probeJava: () => ({ version: "21.0.11", @@ -111,7 +112,9 @@ describe("Ghidra doctor integration", () => { "wrong version", inspection( { installDir: INSTALL }, - { readText: () => "application.version=11.4\n" }, + { + readText: () => "application.version=11.4\napplication.java.min=21\n", + }, ), "ghidra-version", ], @@ -136,6 +139,14 @@ describe("Ghidra doctor integration", () => { ), "ghidra-java", ], + [ + "missing Java minimum", + inspection( + { installDir: INSTALL }, + { readText: () => "application.version=12.1.4\n" }, + ), + "ghidra-java", + ], [ "unsupported architecture", inspection({ installDir: INSTALL, architecture: "arm" }), @@ -163,6 +174,16 @@ describe("Ghidra doctor integration", () => { }); describe("Windows Ghidra doctor authority", () => { + beforeEach(() => { + vi.spyOn(windowsAuthority, "hasWindowsNativeAuthority").mockReturnValue( + false, + ); + vi.spyOn( + windowsAuthority, + "windowsNativeAuthorityUnavailableReason", + ).mockReturnValue(WINDOWS_NATIVE_AUTHORITY_UNAVAILABLE_REASON); + }); + afterEach(() => vi.restoreAllMocks()); it("distinguishes a valid Windows installation from unavailable native authority", async () => { const ghidra = inspection({ installDir: "C:\\tools\\ghidra_12.1.4_PUBLIC", diff --git a/src/ghidra/GhidraFunctionValues.test.ts b/src/ghidra/GhidraFunctionValues.test.ts index 662db0a26..e54692840 100644 --- a/src/ghidra/GhidraFunctionValues.test.ts +++ b/src/ghidra/GhidraFunctionValues.test.ts @@ -23,6 +23,8 @@ import { const observationOperations: ReadonlySet = new Set([ "analyze_function", "procedure_assembly", + "procedure_callees", + "procedure_callers", "procedure_info", "procedure_pseudo_code", "read_function_instructions", @@ -42,6 +44,30 @@ const parseFunctionValueFixture = ( }; describe("Ghidra function-analysis result values", () => { + it("preserves file-offset resolution in Evidence locations and rejects inexact offsets", () => { + const selection = { + file_offset: 0x210, + mapped_address: "0x140001010", + procedure_address: "0x140001010", + }; + const wire = { value: "return 42;", limitations: [], selection }; + expect(parseGhidraFunctionResult("procedure_assembly", wire)).toMatchObject( + { ok: false, error: { _tag: "AnalysisOutputError" } }, + ); + const parsed = parseGhidraFunctionResult("procedure_pseudo_code", wire); + if (!parsed.ok) throw parsed.error; + expect(parsed.value.locations).toEqual([ + { kind: "file-offset", offset: 0x210 }, + { kind: "address", address: "0x140001010" }, + { kind: "address", address: "0x140001010" }, + ]); + expect( + parseGhidraFunctionResult("procedure_pseudo_code", { + ...wire, + selection: { ...selection, file_offset: Number.MAX_SAFE_INTEGER + 1 }, + }), + ).toMatchObject({ ok: false, error: { _tag: "AnalysisOutputError" } }); + }); it.each([...observationOperations])( "requires the function-observation limitations envelope for %s", (operation) => { @@ -69,6 +95,29 @@ describe("Ghidra function-analysis result values", () => { }); }); + it.each(["procedure_callees", "procedure_callers"] as const)( + "requires the function-observation envelope to carry terminal-call limitations for %s", + (operation) => { + // A bare address list is the pre-fix bridge shape; without the envelope + // a truncated function body reports no warning, so it must be rejected. + expect(parseGhidraFunctionResult(operation, ["0x2000"])).toMatchObject({ + ok: false, + error: { _tag: "AnalysisOutputError" }, + }); + const limitation = + "Ghidra Listing reports a terminal call at 0x401000 to 0x402000 abort; fallthrough is excluded by Ghidra's flow model."; + expect( + parseGhidraFunctionResult(operation, { + value: ["0x2000"], + limitations: [limitation], + }), + ).toEqual({ + ok: true, + value: { value: ["0x2000"], limitations: [limitation] }, + }); + }, + ); + it.each(["reference_kinds_available", "unresolved_calls"])( "rejects omitted %s instead of inventing complete reference observations", (field) => { @@ -171,6 +220,51 @@ describe("Ghidra function-analysis result values", () => { }); }); +describe("Ghidra Swift String literals", () => { + it("keeps a small Swift literal without storage apart from a large literal's bytes", () => { + const dossier = ghidraFunctionDossier(); + if ( + typeof dossier !== "object" || + dossier === null || + Array.isArray(dossier) + ) + throw new TypeError("Ghidra dossier fixture is invalid"); + const literals = [ + { + source_address: "0x401004", + value: "The user name ", + representation: "small", + address: null, + registers: ["x0", "x1"], + words: ["0x7265737520656854", "0xee0020656d616e20"], + }, + { + source_address: "0x401006", + value: "", + representation: "small", + address: null, + registers: ["x0", "x1"], + words: ["0x0", "0xe000000000000000"], + }, + { + source_address: "0x401008", + value: " cannot be written into a sudoers rule safely.", + representation: "large", + address: "0x402270", + registers: ["x0", "x1"], + words: ["0xd00000000000002e", "0x8000000000402250"], + }, + ]; + expect( + parseFunctionValueFixture("analyze_function", { + ...dossier, + swift_string_literals: literals, + swift_string_literal_omissions: [], + }), + ).toMatchObject({ ok: true, value: { swift_string_literals: literals } }); + }); +}); + describe("Ghidra compact reference provenance", () => { it("distinguishes incoming references, examined emptiness and unavailable coverage at data addresses", () => { const instruction = { @@ -399,6 +493,34 @@ describe("Ghidra function-analysis malformed results", () => { ); }); +const malformedSwiftStringLiterals = ( + dossier: Record, +): Array<[string, GhidraFunctionOperation, JsonValue]> => + ( + [ + ["non-canonical Swift literal address", "large", "402240"], + ["small Swift literal with storage", "small", "0x402270"], + ["large Swift literal without storage", "large", null], + ] as const + ).map(([name, representation, address]) => [ + name, + "analyze_function", + { + ...dossier, + swift_string_literal_omissions: [], + swift_string_literals: [ + { + source_address: "0x401004", + value: "The power setting could not be changed.", + representation, + address, + registers: ["x0", "x1"], + words: ["0xd000000000000027", "0x8000000000402220"], + }, + ], + }, + ]); + const malformedOutputs = (): Array< [string, GhidraFunctionOperation, JsonValue] > => { @@ -413,6 +535,11 @@ const malformedOutputs = (): Array< throw new TypeError("Ghidra native API fixture is invalid"); return [ ["non-canonical address", "procedure_callers", ["401000"]], + [ + "Swift literals without their omission record", + "analyze_function", + { ...dossier, swift_string_literals: [] }, + ], [ "missing classification", "procedure_info", @@ -443,6 +570,26 @@ const malformedOutputs = (): Array< Object.entries(dossier).filter(([key]) => key !== "native_api"), ), ], + ...malformedSwiftStringLiterals(dossier), + [ + "non-canonical Swift literal omission address", + "analyze_function", + { + ...dossier, + swift_string_literals: [], + swift_string_literal_omissions: [ + { + source_address: "0x401004", + representation: "large", + address: "402240", + byte_length: 39, + reason: "capacity", + registers: ["x0", "x1"], + words: ["0xd000000000000027", "0x8000000000402220"], + }, + ], + }, + ], [ "non-canonical native API target", "analyze_function", diff --git a/src/ghidra/GhidraFunctionValues.ts b/src/ghidra/GhidraFunctionValues.ts index 8876e31ae..a6e62c93c 100644 --- a/src/ghidra/GhidraFunctionValues.ts +++ b/src/ghidra/GhidraFunctionValues.ts @@ -26,6 +26,7 @@ import { } from "../domain/jsonValue.js"; import { nativeApiBoundarySchema } from "../domain/native/nativeApiBoundary.js"; import { nativeValueFlowSchema } from "../domain/native/nativeValueFlow.js"; +import type { EvidenceLocation } from "../domain/evidence.js"; import { err, ok, type Result } from "../domain/result.js"; import { analysisInputErrorFromIssues } from "../domain/inputIssueProjection.js"; import { @@ -292,6 +293,15 @@ const ghidraFunctionDossier = functionDossierSchema }) .strict() .superRefine((value, context) => { + if ( + (value.swift_string_literals === undefined) !== + (value.swift_string_literal_omissions === undefined) + ) + context.addIssue({ + code: "custom", + message: + "Ghidra dossier must record Swift literals and their omissions together", + }); const addresses = [ value.procedure.address, ...value.comments.map(({ address }) => address), @@ -307,6 +317,14 @@ const ghidraFunctionDossier = functionDossierSchema address, source_address, ]), + ...(value.swift_string_literals ?? []).flatMap( + ({ address, source_address }) => + address === null ? [source_address] : [address, source_address], + ), + ...(value.swift_string_literal_omissions ?? []).flatMap( + ({ address, source_address }) => + address === null ? [source_address] : [address, source_address], + ), ...value.basic_blocks.flatMap(({ start, end, successors }) => [ start, end, @@ -391,6 +409,8 @@ const functionBoundaryObservationOperations: ReadonlySet["selection"]; }; if (functionBoundaryObservationOperations.has(operation)) { const envelope = functionObservationEnvelope.safeParse(value); - if (!envelope.success) + if ( + !envelope.success || + (operation !== "procedure_pseudo_code" && + envelope.data.selection !== undefined) + ) return err( new AnalysisOutputError( operation, "Ghidra bridge omitted the function observation envelope", - { cause: envelope.error }, + envelope.success ? undefined : { cause: envelope.error }, ), ); observation = envelope.data; @@ -434,6 +469,24 @@ export const parseGhidraFunctionResult = ( ? ok({ value: jsonValueSchema.parse(parsed.data), limitations: observation.limitations, + ...(observation.selection === undefined + ? {} + : { + locations: [ + { + kind: "file-offset" as const, + offset: observation.selection.file_offset, + }, + { + kind: "address" as const, + address: observation.selection.mapped_address, + }, + { + kind: "address" as const, + address: observation.selection.procedure_address, + }, + ], + }), }) : err( new AnalysisOutputError( diff --git a/src/ghidra/GhidraInstallation.test.ts b/src/ghidra/GhidraInstallation.test.ts index 21c4195ad..776c10308 100644 --- a/src/ghidra/GhidraInstallation.test.ts +++ b/src/ghidra/GhidraInstallation.test.ts @@ -12,6 +12,8 @@ import { import { SUPPORTED_GHIDRA_JAVA_MAJOR, SUPPORTED_GHIDRA_VERSION, + ghidraApplicationProperties, + ghidraJavaRequirement, } from "./GhidraInstallationPolicy.js"; const INSTALL = "/opt/ghidra"; @@ -42,7 +44,7 @@ const host = ( ): GhidraInstallationHost => ({ readText: (path) => path === PROPERTIES - ? `application.version=${SUPPORTED_GHIDRA_VERSION}\n` + ? `application.version=${SUPPORTED_GHIDRA_VERSION}\napplication.java.min=${String(SUPPORTED_GHIDRA_JAVA_MAJOR)}\n` : undefined, executable: (path) => path === HEADLESS || path === MAC_DECOMPILER || path === LINUX_DECOMPILER, @@ -190,7 +192,7 @@ describe("Ghidra installation inspection", () => { architecture: "x64", readText: (path) => path === WINDOWS_PROPERTIES - ? `application.version=${SUPPORTED_GHIDRA_VERSION}\n` + ? `application.version=${SUPPORTED_GHIDRA_VERSION}\napplication.java.min=21\n` : undefined, executable: (path) => path === WINDOWS_HEADLESS || path === WINDOWS_DECOMPILER, @@ -275,7 +277,9 @@ describe("Ghidra installation rejection diagnostics", () => { platform: "linux" as const, architecture: "x64" as const, }, - override: { readText: () => "application.version=11.4\n" }, + override: { + readText: () => "application.version=11.4\napplication.java.min=21\n", + }, failed: "version", code: "unsupported_version", }, @@ -350,6 +354,167 @@ describe("Ghidra installation rejection diagnostics", () => { }); describe("Ghidra release line and JDK bounds", () => { + it("uses Java Properties last-key, delimiter, escape, and continuation rules", () => { + const properties = ghidraApplicationProperties( + [ + "application.version=11.4", + "# comment ending with a slash " + "\\", + "application.\\u0076ersion : 12.\\", + " 1.\\u0034", + "! another comment " + "\\", + "application.java.\\u006din\t21", + "application.java.max=", + ].join("\n"), + ); + expect(properties).toEqual({ + providerVersion: "12.1.4", + javaMinimum: { status: "major", major: 21 }, + javaMaximum: { status: "absent" }, + parseError: null, + }); + expect( + ghidraApplicationProperties("application.version=12.1.4").javaMinimum, + ).toEqual({ + status: "absent", + }); + expect( + ghidraApplicationProperties( + "application.version=12.1.4\napplication.java.min=\n", + ).javaMinimum, + ).toEqual({ status: "unresolved", raw: "" }); + expect( + ghidraApplicationProperties("application.version=12.1.4\\") + .providerVersion, + ).toBe("12.1.4"); + expect( + ghidraApplicationProperties("application.version=12.\\\n #.1") + .providerVersion, + ).toBe("12.#.1"); + expect( + ghidraApplicationProperties( + "application.version=12.\\\n\napplication.version=12.1.4", + ).providerVersion, + ).toBe("12.1.4"); + }); + + it("matches Ghidra integer Java bounds and zero maximum semantics", () => { + const properties = ghidraApplicationProperties( + "application.java.min=21.0\napplication.java.max=0\n", + ); + expect(properties.javaMinimum).toEqual({ + status: "unresolved", + raw: "21.0", + }); + expect(properties.javaMaximum).toEqual({ status: "absent" }); + expect( + ghidraApplicationProperties("application.java.max=+0").javaMaximum, + ).toEqual({ + status: "absent", + }); + expect( + ghidraApplicationProperties("application.java.max=+0").javaMaximum, + ).toEqual({ status: "absent" }); + expect( + ghidraApplicationProperties("application.java.min=+21").javaMinimum, + ).toEqual({ + status: "major", + major: 21, + }); + expect( + ghidraApplicationProperties("application.java.min=٢١").javaMinimum, + ).toEqual({ + status: "major", + major: 21, + }); + expect( + ghidraApplicationProperties("application.java.min=21").javaMinimum, + ).toEqual({ + status: "major", + major: 21, + }); + expect( + ghidraApplicationProperties("application.java.max=+0").javaMaximum, + ).toEqual({ + status: "unresolved", + raw: "+0", + }); + expect( + ghidraApplicationProperties("application.java.min=②").javaMinimum, + ).toEqual({ + status: "unresolved", + raw: "②", + }); + expect( + ghidraApplicationProperties("application.java.min=2147483648") + .javaMinimum, + ).toEqual({ status: "unresolved", raw: "2147483648" }); + }); + + it("keeps an unavailable properties file unresolved for Java admission", () => { + expect(ghidraJavaRequirement(undefined).unresolved).toBe( + "application.properties is unavailable", + ); + }); + + it("preserves property value trailing whitespace instead of normalizing it", () => { + const properties = ghidraApplicationProperties( + "application.version=12.1.4 \napplication.java.min=21 \n", + ); + expect(properties.providerVersion).toBe("12.1.4 "); + expect(properties.javaMinimum).toEqual({ + status: "unresolved", + raw: "21 ", + }); + expect(properties.parseError).toBeNull(); + }); +}); + +describe("Ghidra malformed property admission", () => { + it("fails installation and doctor admission closed on malformed Unicode escapes", () => { + const result = inspectGhidraInstallation( + { + environment: {}, + installDir: INSTALL, + platform: "linux", + architecture: "x64", + }, + host({ + readText: () => + "application.version=12.1.4\nbroken=\\u12G4\napplication.java.min=17\n", + }), + ); + expect(result.status).toBe("unavailable"); + expect(result.providerVersion).toBeNull(); + expect(result.checks).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + name: "version", + status: "failed", + code: "version_unresolved", + detail: expect.stringContaining("Malformed Unicode escape"), + }), + expect.objectContaining({ + name: "java", + status: "failed", + code: "version_unresolved", + remediation: expect.stringContaining( + "application.properties parse failed", + ), + }), + ]), + ); + expect(projectGhidraDoctorInspection(result)).toMatchObject({ + available: false, + registrationEnvironment: {}, + checks: expect.arrayContaining([ + expect.objectContaining({ name: "version", ok: false }), + expect.objectContaining({ name: "java", ok: false }), + ]), + }); + }); +}); + +describe("Ghidra installation version admission", () => { it.each([ { name: "JDK above the installation maximum", @@ -367,16 +532,23 @@ describe("Ghidra release line and JDK bounds", () => { failed: "java", code: "version_unresolved", }, + { + name: "missing Java minimum", + readText: () => "application.version=12.1.4\n", + probeJava: () => JAVA, + failed: "java", + code: "version_unresolved", + }, { name: "unparseable Ghidra version", - readText: () => "application.version=latest\n", + readText: () => "application.version=latest\napplication.java.min=21\n", probeJava: () => JAVA, failed: "version", code: "version_unresolved", }, { name: "different Ghidra line", - readText: () => "application.version=12.2.0\n", + readText: () => "application.version=12.2.0\napplication.java.min=21\n", probeJava: () => JAVA, failed: "version", code: "unsupported_version", @@ -422,7 +594,7 @@ describe("Ghidra release line and JDK bounds", () => { }); }); - it("honors an installation that declares a lower Java minimum", () => { + it("accepts a lower Java minimum declared by the installation", () => { expect( inspectGhidraInstallation( { diff --git a/src/ghidra/GhidraInstallation.ts b/src/ghidra/GhidraInstallation.ts index 713fe820f..9675d133f 100644 --- a/src/ghidra/GhidraInstallation.ts +++ b/src/ghidra/GhidraInstallation.ts @@ -72,6 +72,15 @@ const NATIVE_ARCHITECTURES: readonly { /** Bound installation discovery reads; larger headers remain unknown. */ const EXECUTABLE_HEADER_PROBE_BYTES = 4096; +/** + * Bound the Java version probe so a chatty or hostile JVM cannot force an + * unbounded in-memory capture. Well-behaved JVMs print a few kilobytes for + * `-XshowSettings:properties`; the same 1 MiB budget used for other Java + * listings leaves ample headroom. A probe that exceeds the budget yields no + * observation and is reported through the existing failed-probe path. + */ +const JAVA_PROBE_MAX_OUTPUT_BYTES = 1024 * 1024; + /** Compatibility of one present native decompiler with the inspected host. */ type NativeDecompilerCompatibility = | { @@ -641,14 +650,20 @@ const versionCheck = ( properties: GhidraApplicationProperties | undefined, ): GhidraInstallationCheck => { const raw = properties?.providerVersion; + const parseError = properties?.parseError; return installationCheck({ name: "version", - passed: ghidraReleaseAccepted(raw), + passed: parseError == null && ghidraReleaseAccepted(raw), code: - ghidraReleaseCompatibility(raw).status === "unsupported" - ? "unsupported_version" - : "version_unresolved", - detail: ghidraVersionDetail(raw), + parseError != null + ? "version_unresolved" + : ghidraReleaseCompatibility(raw).status === "unsupported" + ? "unsupported_version" + : "version_unresolved", + detail: + parseError == null + ? ghidraVersionDetail(raw) + : `application.properties could not be parsed: ${parseError}`, remediation: `Install a Ghidra ${ghidraReleaseLine()} release (verified with ${SUPPORTED_GHIDRA_VERSION}), or update REA for another Ghidra line.`, }); }; @@ -701,7 +716,9 @@ const javaRemediation = ( const systemGhidraInstallationHost = (): GhidraInstallationHost => ({ readText(path) { try { - return readFileSync(path, "utf8"); + // Ghidra loads this file with Properties.load(InputStream), which maps + // bytes as ISO-8859-1 rather than decoding UTF-8. + return readFileSync(path).toString("latin1"); } catch (cause: unknown) { // best-effort cleanup: optional install probing; unreadable means unknown. void cause; @@ -755,7 +772,7 @@ const systemGhidraInstallationHost = (): GhidraInstallationHost => ({ encoding: "utf8", env: environment, timeout: 5_000, - maxBuffer: Number.POSITIVE_INFINITY, + maxBuffer: JAVA_PROBE_MAX_OUTPUT_BYTES, windowsHide: true, }, ); diff --git a/src/ghidra/GhidraInstallationPolicy.ts b/src/ghidra/GhidraInstallationPolicy.ts index bd19dc1d1..8a35c66c3 100644 --- a/src/ghidra/GhidraInstallationPolicy.ts +++ b/src/ghidra/GhidraInstallationPolicy.ts @@ -1,10 +1,8 @@ +import { parseGhidraJavaProperties } from "./GhidraJavaProperties.js"; + /** Ghidra build whose bridge behavior this REA release verifies. */ export const SUPPORTED_GHIDRA_VERSION = "12.1.4"; -/** - * JDK major used when an installation omits `application.java.min`. - * Admission uses that installation's `application.java.min` and - * `application.java.max`. An empty maximum means the release sets no upper bound. - */ +/** Verified JDK major; admission uses the installation's declared bounds. */ export const SUPPORTED_GHIDRA_JAVA_MAJOR = 21; /** Numeric Ghidra release read from `application.version`. */ @@ -40,30 +38,49 @@ type GhidraJavaBound = | { readonly status: "unresolved"; readonly raw: string }; /** JDK majors this installation admits, after reading its own properties. */ -export interface GhidraJavaRequirement { - readonly minimumMajor: number; - readonly maximumMajor: number | null; - readonly unresolved: string | null; -} +export type GhidraJavaRequirement = + | { + readonly minimumMajor: number; + readonly maximumMajor: number | null; + readonly unresolved: null; + } + | { readonly unresolved: string }; /** Version and Java bounds read from one `application.properties` file. */ export interface GhidraApplicationProperties { readonly providerVersion: string | null; readonly javaMinimum: GhidraJavaBound; readonly javaMaximum: GhidraJavaBound; + /** A malformed Java properties escape invalidates the entire file. */ + readonly parseError: string | null; } const GHIDRA_RELEASE_PATTERN = /^(\d+)\.(\d+)(?:\.(\d+))?([-+].*)?$/u; -const JAVA_MAJOR_PATTERN = /^(\d+)(?:\.\d+)*$/u; /** Read the version and Java bounds Ghidra publishes for one installation. */ export const ghidraApplicationProperties = ( content: string, -): GhidraApplicationProperties => ({ - providerVersion: propertyValue(content, "application.version"), - javaMinimum: parseJavaBound(propertyValue(content, "application.java.min")), - javaMaximum: parseJavaBound(propertyValue(content, "application.java.max")), -}); +): GhidraApplicationProperties => { + const parsed = parseGhidraJavaProperties(content); + if (!parsed.ok) + return { + providerVersion: null, + javaMinimum: { status: "unresolved", raw: parsed.error }, + javaMaximum: { status: "unresolved", raw: parsed.error }, + parseError: parsed.error, + }; + const providerVersion = parsed.value.get("application.version"); + const minimum = parsed.value.get("application.java.min"); + const maximum = parsed.value.get("application.java.max"); + return { + providerVersion: providerVersion ?? null, + javaMinimum: + minimum === undefined ? { status: "absent" } : parseJavaBound(minimum), + // Ghidra uses an explicitly empty maximum to mean no upper bound. + javaMaximum: parseJavaMaximum(maximum), + parseError: null, + }; +}; /** Classify `application.version` against the verified release line. */ export const ghidraReleaseCompatibility = ( @@ -130,14 +147,21 @@ export const ghidraReleaseAccepted = ( export const ghidraJavaRequirement = ( properties: GhidraApplicationProperties | undefined, ): GhidraJavaRequirement => { - const minimum = properties?.javaMinimum ?? { status: "absent" }; - const maximum = properties?.javaMaximum ?? { status: "absent" }; + if (properties?.parseError !== null && properties?.parseError !== undefined) + return unresolvedJavaRequirement( + `application.properties parse failed: ${properties.parseError}`, + ); + if (properties === undefined) + return unresolvedJavaRequirement("application.properties is unavailable"); + const minimum = properties.javaMinimum; + const maximum = properties.javaMaximum; if (minimum.status === "unresolved") return unresolvedJavaRequirement(`application.java.min=${minimum.raw}`); + if (minimum.status === "absent") + return unresolvedJavaRequirement("application.java.min is missing"); if (maximum.status === "unresolved") return unresolvedJavaRequirement(`application.java.max=${maximum.raw}`); - const minimumMajor = - minimum.status === "major" ? minimum.major : SUPPORTED_GHIDRA_JAVA_MAJOR; + const minimumMajor = minimum.major; const maximumMajor = maximum.status === "major" ? maximum.major : null; if (maximumMajor !== null && maximumMajor < minimumMajor) return unresolvedJavaRequirement( @@ -187,14 +211,56 @@ const parseGhidraRelease = (raw: string): GhidraReleaseVersion | undefined => { return patch === undefined ? undefined : { major, minor, patch }; }; -const parseJavaBound = (raw: string | null): GhidraJavaBound => { - if (raw === null) return { status: "absent" }; - const major = integerGroup(JAVA_MAJOR_PATTERN.exec(raw)?.[1]); +const parseJavaBound = (raw: string): GhidraJavaBound => { + const major = parseJavaInteger(raw); return major === undefined ? { status: "unresolved", raw } : { status: "major", major }; }; +const parseJavaMaximum = (raw: string | undefined): GhidraJavaBound => { + if (raw === undefined || raw === "") return { status: "absent" }; + const bound = parseJavaBound(raw); + return bound.status === "major" && bound.major === 0 + ? { status: "absent" } + : bound; +}; + +const parseJavaInteger = (raw: string): number | undefined => { + let index = 0; + let sign = 1; + if (raw[index] === "+" || raw[index] === "-") { + if (raw[index] === "-") sign = -1; + index += 1; + } + if (index === raw.length) return undefined; + const limit = sign < 0 ? 2_147_483_648 : 2_147_483_647; + let value = 0; + for (; index < raw.length; index += 1) { + const character = raw[index]; + if (character === undefined) return undefined; + const digit = javaDecimalDigit(character); + if (digit === undefined) return undefined; + if (value > Math.floor((limit - digit) / 10)) return undefined; + value = value * 10 + digit; + } + return sign * value; +}; + +/** Match Integer.parseInt(charAt) for BMP Decimal_Number characters. */ +const javaDecimalDigit = (character: string): number | undefined => { + const code = character.charCodeAt(0); + if (code >= 48 && code <= 57) return code - 48; + if (!/^\p{Decimal_Number}$/u.test(character)) return undefined; + let blockStart = code; + while ( + blockStart > 0 && + /^\p{Decimal_Number}$/u.test(String.fromCharCode(blockStart - 1)) + ) + blockStart -= 1; + return (code - blockStart) % 10; +}; + const integerGroup = (value: string | undefined): number | undefined => { if (value === undefined) return undefined; const parsed = Number(value); @@ -204,17 +270,7 @@ const integerGroup = (value: string | undefined): number | undefined => { const unresolvedJavaRequirement = ( unresolved: string, ): GhidraJavaRequirement => ({ - minimumMajor: SUPPORTED_GHIDRA_JAVA_MAJOR, - maximumMajor: null, unresolved, }); -const propertyValue = (content: string, name: string): string | null => { - for (const line of content.split(/\r?\n/u)) { - const separator = line.indexOf("="); - if (separator < 0 || line.slice(0, separator).trim() !== name) continue; - const value = line.slice(separator + 1).trim(); - return value.length === 0 ? null : value; - } - return null; -}; +/** Parse Java Properties.load(InputStream) syntax from its ISO-8859-1 text. */ diff --git a/src/ghidra/GhidraJavaProperties.ts b/src/ghidra/GhidraJavaProperties.ts new file mode 100644 index 000000000..ab3b7bead --- /dev/null +++ b/src/ghidra/GhidraJavaProperties.ts @@ -0,0 +1,115 @@ +import { err, ok, type Result } from "../domain/result.js"; + +/** Parse Java Properties.load(InputStream) syntax from its ISO-8859-1 text. */ +export const parseGhidraJavaProperties = ( + content: string, +): Result, string> => { + const values = new Map(); + for (const line of logicalPropertyLines(content)) { + let start = 0; + while (isPropertyWhitespace(line[start])) start += 1; + if (start === line.length || line[start] === "#" || line[start] === "!") + continue; + let end = start; + let escaped = false; + while (end < line.length) { + const character = line[end]; + if ( + !escaped && + (character === "=" || + character === ":" || + isPropertyWhitespace(character)) + ) + break; + if (character === "\\") escaped = !escaped; + else escaped = false; + end += 1; + } + let valueStart = end; + while (isPropertyWhitespace(line[valueStart])) valueStart += 1; + if (line[valueStart] === "=" || line[valueStart] === ":") valueStart += 1; + while (isPropertyWhitespace(line[valueStart])) valueStart += 1; + const key = decodePropertyEscapes(line.slice(start, end)); + if (!key.ok) return err(key.error); + const value = decodePropertyEscapes(line.slice(valueStart)); + if (!value.ok) return err(value.error); + values.set(key.value, value.value); + } + return ok(values); +}; + +const isPropertyWhitespace = (value: string | undefined): boolean => + value === " " || value === "\t" || value === "\f"; + +/** Join odd-backslash continuations while stripping leading continuation space. */ +function* logicalPropertyLines(content: string): Generator { + let physicalStart = 0; + let logical = ""; + let continuing = false; + while (physicalStart <= content.length) { + let physicalEnd = physicalStart; + while ( + physicalEnd < content.length && + content[physicalEnd] !== "\n" && + content[physicalEnd] !== "\r" + ) + physicalEnd += 1; + let line = content.slice(physicalStart, physicalEnd); + let leading = 0; + while (isPropertyWhitespace(line[leading])) leading += 1; + const comment = + !continuing && (line[leading] === "#" || line[leading] === "!"); + if (continuing) line = line.slice(leading); + let continued = false; + if (!comment) { + let slashes = 0; + for ( + let index = line.length - 1; + index >= 0 && line[index] === "\\"; + index -= 1 + ) + slashes += 1; + continued = slashes % 2 === 1; + logical += continued ? line.slice(0, -1) : line; + } + if (!continued) { + yield logical; + logical = ""; + } + continuing = continued; + if (physicalEnd === content.length) break; + physicalStart = + content[physicalEnd] === "\r" && content[physicalEnd + 1] === "\n" + ? physicalEnd + 2 + : physicalEnd + 1; + } + // Properties.load still consumes the accumulated line when EOF follows a + // continuation marker; the marker itself was already removed above. + if (continuing) yield logical; +} + +const decodePropertyEscapes = (value: string): Result => { + let result = ""; + for (let index = 0; index < value.length; index += 1) { + const character = value[index]; + if (character !== "\\") { + result += character; + continue; + } + const escaped = value[++index]; + if (escaped === undefined) + return err("Trailing escape in Java properties value"); + if (escaped === "t") result += "\t"; + else if (escaped === "n") result += "\n"; + else if (escaped === "r") result += "\r"; + else if (escaped === "f") result += "\f"; + else if (escaped === "u") { + const digits = value.slice(index + 1, index + 5); + if (digits.length !== 4 || !/^[0-9a-f]{4}$/iu.test(digits)) + return err("Malformed Unicode escape in Java properties file"); + result += String.fromCharCode(Number.parseInt(digits, 16)); + index += 4; + } else result += escaped; + } + return ok(result); +}; diff --git a/src/ghidra/GhidraLanguageCatalog.ts b/src/ghidra/GhidraLanguageCatalog.ts index 5da5943dc..6373f187f 100644 --- a/src/ghidra/GhidraLanguageCatalog.ts +++ b/src/ghidra/GhidraLanguageCatalog.ts @@ -75,8 +75,10 @@ export const readGhidraLanguageCatalog = async ( if (!entry.endsWith(".ldefs")) continue; const bytes = await readFile(join(directory, entry)); if (bytes.length > MAX_LDEFS_BYTES) continue; + // XML 1.0 section 4.3.3: a leading UTF-8 BOM is an encoding signature, + // not document content; TextDecoder consumes it, Buffer#toString does not. definitions.push( - ...parseGhidraLanguageDefinitions(bytes.toString("utf8")), + ...parseGhidraLanguageDefinitions(new TextDecoder().decode(bytes)), ); } } diff --git a/src/ghidra/GhidraLanguageOverride.test.ts b/src/ghidra/GhidraLanguageOverride.test.ts index a355b39fa..02c5bf245 100644 --- a/src/ghidra/GhidraLanguageOverride.test.ts +++ b/src/ghidra/GhidraLanguageOverride.test.ts @@ -2,6 +2,7 @@ import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js"; import { parseConfig } from "../config/parseConfig.js"; import type { BinaryTarget } from "../domain/binaryTargetTypes.js"; import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; @@ -42,7 +43,7 @@ const X86_LDEFS = ` const host: GhidraInstallationHost = { platform: "linux", architecture: "x64", - readText: () => "application.version=12.1.4\n", + readText: () => "application.version=12.1.4\napplication.java.min=21\n", executable: () => true, probeJava: () => ({ version: "21.0.11", @@ -190,6 +191,28 @@ describe("Ghidra language catalog", () => { ); }); + it("reads a UTF-8 signed definition beside an unsigned sibling", async () => { + const root = await createTestTempDirectory("rea-ghidra-bom-"); + const module = (name: string) => + join(root, "Ghidra/Processors", name, "data/languages"); + await mkdir(module("probe"), { recursive: true }); + await mkdir(module("x86"), { recursive: true }); + await writeFile( + join(module("probe"), "probe.ldefs"), + Buffer.concat([ + Buffer.from([0xef, 0xbb, 0xbf]), + Buffer.from( + '\n', + ), + ]), + ); + await writeFile(join(module("x86"), "x86.ldefs"), X86_LDEFS); + expect(await readGhidraLanguageCatalog(root)).toEqual([ + { id: "probe:LE:32:default", compilerSpecIds: [] }, + ...parseGhidraLanguageDefinitions(X86_LDEFS), + ]); + }); + it("admits only languages and compiler specs the installation defines", () => { const catalog = parseGhidraLanguageDefinitions(X86_LDEFS); expect( diff --git a/src/ghidra/GhidraLauncher.ts b/src/ghidra/GhidraLauncher.ts index 59c555606..e1527ea81 100644 --- a/src/ghidra/GhidraLauncher.ts +++ b/src/ghidra/GhidraLauncher.ts @@ -1,3 +1,4 @@ +import { parseGhidraJavaProperties } from "./GhidraJavaProperties.js"; import { GHIDRA_SEED_SCRIPT, snapshotGhidraAnalysisSeeds, @@ -5,7 +6,7 @@ import { } from "./GhidraAnalysisSeeds.js"; import type { GhidraLanguageOverride } from "../config/ghidraLanguageOverride.js"; import { snapshotEnvironment } from "../process/snapshotEnvironment.js"; -import { mkdir, realpath } from "node:fs/promises"; +import { mkdir, readFile, realpath } from "node:fs/promises"; import { basename, dirname, join, posix, win32 } from "node:path"; import writeFileAtomic from "write-file-atomic"; @@ -219,7 +220,11 @@ export class GhidraHeadlessLauncher implements GhidraLauncher { "Installed Allegrex content changed since profile resolution; reopen the target to resolve a fresh profile.", ); } - await createGhidraRuntimeDirectories(paths, platform); + await createGhidraRuntimeDirectories( + paths, + platform, + await ghidraConfigReleaseDirectory(this.options.analyzeHeadlessPath), + ); const seedPath = this.options.analysisSeeds === undefined ? undefined @@ -721,9 +726,44 @@ const ghidraRuntimePaths = (runtimeRoot: string) => ({ scriptLogPath: join(runtimeRoot, "script.log"), }); +/** + * Ghidra's launch support resolves its JDK before the headless run and then + * persists that choice under `/ghidra/`. Inside + * REA's Windows private runtime a child JVM cannot create those directories + * itself, so startup aborted with "Failed to create directory" and reported + * that no supported JDK was found. Create the release directory while this + * process still owns the runtime, leaving Ghidra only a file write inside it. + */ +const ghidraConfigReleaseDirectory = async ( + analyzeHeadlessPath: string, +): Promise => { + const installRoot = dirname(dirname(analyzeHeadlessPath)); + try { + const properties = await readFile( + join(installRoot, "Ghidra", "application.properties"), + "latin1", + ); + const parsed = parseGhidraJavaProperties(properties); + if (!parsed.ok) return basename(installRoot); + const version = parsed.value.get("application.version"); + const release = parsed.value.get("application.release.name"); + return version === undefined || + release === undefined || + release.length === 0 + ? basename(installRoot) + : `ghidra_${version}_${release}`; + } catch (cause: unknown) { + // An unreadable properties file leaves the release coordinate unknown; the + // conventional extraction directory carries the same name. + void cause; + return basename(installRoot); + } +}; + const createGhidraRuntimeDirectories = async ( paths: ReturnType, platform: NodeJS.Platform, + configReleaseDirectory: string, ): Promise => { if (platform === "win32") { const runtime = windowsPrivateRuntime(dirname(paths.projectRoot)); @@ -736,6 +776,11 @@ const createGhidraRuntimeDirectories = async ( paths.dataRoot, ]) runtime.mkdir(basename(path)); + // Each level needs its parent first: the native owner opens every existing + // component and creates only the final one. + const configRoot = basename(paths.configRoot); + runtime.mkdir(win32.join(configRoot, "ghidra")); + runtime.mkdir(win32.join(configRoot, "ghidra", configReleaseDirectory)); return; } await Promise.all( @@ -746,6 +791,7 @@ const createGhidraRuntimeDirectories = async ( paths.cacheRoot, paths.configRoot, paths.dataRoot, + join(paths.configRoot, "ghidra", configReleaseDirectory), ].map((path) => mkdir(path, { recursive: true, mode: 0o700 })), ); }; diff --git a/src/ghidra/GhidraLoadImageValues.ts b/src/ghidra/GhidraLoadImageValues.ts index 1c80bb635..f06f4fdaa 100644 --- a/src/ghidra/GhidraLoadImageValues.ts +++ b/src/ghidra/GhidraLoadImageValues.ts @@ -361,7 +361,10 @@ export const attestGhidraRawBinaryLoadImage = ( }); }; const source = observations.source_files[0]; - const mapping = observations.mappings[0]; + const fileMappings = observations.mappings.filter( + (item) => item.source_file_index !== null, + ); + const mapping = fileMappings[0]; check("executable_format", "Raw Binary", observations.executable_format); check("language_id", language.languageId, observations.language_id); check( @@ -371,11 +374,34 @@ export const attestGhidraRawBinaryLoadImage = ( ); // BinaryLoader's block base is independent of the Program image base. check("image_base", "0x0", observations.image_base); + check( + "default_address_space", + language.defaultAddressSpace, + observations.default_address_space, + ); check("source_file_count", 1, observations.source_files.length); check("source_file_size", snapshot.bytes, source?.size ?? null); check("original_file_sha256", target.sha256, source?.original_sha256 ?? null); check("modified_file_sha256", target.sha256, source?.modified_sha256 ?? null); - check("mapping_count", 1, observations.mappings.length); + // Language-defined uninitialized zero page/stack blocks are observations, not file bytes. + check("mapping_count", 1, fileMappings.length); + check( + "initialized_unbacked_mappings", + 0, + observations.mappings.filter( + (item) => item.initialized && item.source_file_index === null, + ).length, + ); + check("mapping.source_file_index", 0, mapping?.source_file_index ?? null); + // Matching bytes alone do not establish the admitted default-space load. + check("mapping.initialized", true, mapping?.initialized ?? null); + check("mapping.loaded", true, mapping?.loaded ?? null); + check("mapping.overlay", false, mapping?.overlay ?? null); + check( + "mapping.address_space", + observations.default_address_space, + mapping?.address_space ?? null, + ); check( "mapping.file_offset", 0, @@ -427,8 +453,9 @@ export const attestGhidraRawBinaryLoadImage = ( checks, limitations: [ "Raw binary is a headerless interpretation explicitly selected by the caller, not format identification from file bytes.", - "Verification covers complete original/modified source identity, mapped file bytes, the caller-supplied entry, and the bound little-endian Ghidra language. File offset 0 is mapped at the entry address.", + "Verification covers complete original/modified source identity, initialized and loaded non-overlay file bytes in the default address space, the caller-supplied entry, and the bound little-endian Ghidra language. File offset 0 is mapped at the entry address.", "Ghidra may report additional language-defined external entries outside the mapped image. They are preserved as observations; verification requires the caller-supplied entry and does not attest other entries.", + "Language-defined uninitialized memory, such as 6502 zero page and stack, is preserved as observations but is not attested as file-backed bytes.", "Reset vectors, memory-mapped I/O, big-endian and Thumb-only variants, and runtime boot behavior are not modeled or verified. No original executable bytes are changed.", "Mapping ends are inclusive; a mismatching digest identifies its source range, not the first differing byte.", ], diff --git a/src/ghidra/GhidraMipsProfile.test.ts b/src/ghidra/GhidraMipsProfile.test.ts index 31a121cd1..ae72a3069 100644 --- a/src/ghidra/GhidraMipsProfile.test.ts +++ b/src/ghidra/GhidraMipsProfile.test.ts @@ -19,7 +19,7 @@ import { ghidraProfileLanguageOverride } from "./GhidraAnalysisProfile.js"; const host: GhidraInstallationHost = { platform: "linux", architecture: "x64", - readText: () => "application.version=12.1.4\n", + readText: () => "application.version=12.1.4\napplication.java.min=21\n", executable: () => true, probeJava: () => ({ version: "21.0.11", diff --git a/src/ghidra/GhidraProvider.behavior.test.ts b/src/ghidra/GhidraProvider.behavior.test.ts index 8b9c25461..c9084905b 100644 --- a/src/ghidra/GhidraProvider.behavior.test.ts +++ b/src/ghidra/GhidraProvider.behavior.test.ts @@ -8,7 +8,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { fileURLToPath } from "node:url"; import { promisify } from "node:util"; -import { describe, expect, it } from "vitest"; +import { describe, expect, it, onTestFinished, vi } from "vitest"; import { fixtureDosLoadImage } from "./GhidraLoadImage.fixture.js"; import { fixtureRawBinaryLoadImage } from "./GhidraRawBinaryLoadImage.fixture.js"; import { @@ -30,6 +30,8 @@ import { err, ok } from "../domain/result.js"; import type { Result } from "../domain/result.js"; import { GhidraSessionError } from "./GhidraSessionError.js"; import { silentLogger } from "../logger.js"; +import * as WindowsNativeLoader from "../windows/WindowsNativeLoader.js"; +import { WINDOWS_NATIVE_AUTHORITY_UNAVAILABLE_REASON } from "../process/WindowsAuthority.js"; import { ProviderCleanupError } from "../domain/providerCleanupError.js"; import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; import { ProviderStartupDeadline } from "../process/ProviderDeadline.js"; @@ -38,7 +40,7 @@ const INSTALL = "/opt/ghidra_12.1.4_PUBLIC"; const installationHost = (): GhidraInstallationHost => ({ platform: "linux", architecture: "x64", - readText: () => "application.version=12.1.4\n", + readText: () => "application.version=12.1.4\napplication.java.min=21\n", executable: () => true, probeJava: () => ({ version: "21.0.11", @@ -190,6 +192,36 @@ describe("Ghidra provider", () => { }); describe("Ghidra raw-binary profile", () => { + it("admits raw 6502 on Windows and binds its 16-bit language without mutation authority", async () => { + const ghidra = provider({ ...installationHost(), platform: "win32" }); + const target: BinaryTarget = { + path: "/tmp/6502.bin", + sha256: "a".repeat(64), + kind: "executable", + format: "raw-binary", + architecture: "6502", + availableArchitectures: ["6502"], + entryPoint: "0x8001", + entryPointSource: "0x008001", + }; + expect(ghidra.inspectTargetSupport(target).status).toBe("supported"); + const resolved = await ghidra.resolveAnalysisProfile(target); + if (!resolved.ok) throw resolved.error; + expect(resolved.value.profile?.parameters).toMatchObject({ + architecture: "6502", + loader: "BinaryLoader", + language_id: "6502:LE:16:default", + compiler_spec_id: "default", + entry_point: "0x8001", + language_selection: "caller-supplied-v1", + load_image_evidence: "independent-raw-mapping-context-v4", + }); + expect( + ghidra + .capabilities() + .some(({ available, effects }) => available && effects.mutatesArtifact), + ).toBe(false); + }); it("commits BinaryLoader language and caller-supplied entry", async () => { const ghidra = provider(); const target: BinaryTarget = { @@ -324,6 +356,13 @@ describe("Ghidra platform support", () => { }); it("keeps Windows P0 unavailable until native isolation authority exists", () => { + const native = vi + .spyOn(WindowsNativeLoader, "systemWindowsNativeAuthority") + .mockReturnValue({ + available: false, + reason: WINDOWS_NATIVE_AUTHORITY_UNAVAILABLE_REASON, + }); + onTestFinished(() => native.mockRestore()); const ghidra = provider({ ...installationHost(), platform: "win32" }); const nativeApplication = peTarget("x86_64"); @@ -789,7 +828,11 @@ describe("Ghidra relationship coverage", () => { start: () => Promise.resolve(ok(sessionInfo())), callTool: (operation) => Promise.resolve( - ok(operation === "procedure_callees" ? ["0x2000"] : []), + ok( + operation === "procedure_callees" + ? { value: ["0x2000"], limitations: [] } + : { value: [], limitations: [] }, + ), ), close: () => Promise.resolve(ok(null)), })); @@ -812,11 +855,22 @@ describe("Ghidra relationship coverage", () => { }); }); -describe("Ghidra measured load-image projection", () => { - it.each(["verified", "changed-snapshot", "cancelled"] as const)( +describe("Ghidra measured raw load-image projection", () => { + it.each([ + "verified", + "mapping-mismatch", + "changed-snapshot", + "cancelled", + ] as const)( "streams raw snapshot identity and preserves %s outcomes", async (state) => { const fixture = fixtureRawBinaryLoadImage(); + if (state === "mapping-mismatch") { + const mapping = fixture.observation.mappings[0]; + if (mapping === undefined) + throw new Error("Missing source mapping fixture"); + mapping.overlay = true; + } const controller = new AbortController(); const signals: (AbortSignal | undefined)[] = []; const ghidra = provider(installationHost(), () => ({ @@ -872,12 +926,23 @@ describe("Ghidra measured load-image projection", () => { { signal: controller.signal }, ); expect(signals).toEqual([controller.signal]); - if (state === "verified") { + if (state === "verified" || state === "mapping-mismatch") { if (!result.ok) throw result.error; expect(result.value.result).toMatchObject({ - status: "verified", + status: state === "verified" ? "verified" : "mismatch", observations: fixture.observation, }); + if (state === "mapping-mismatch") { + expect(result.value.result).toMatchObject({ + checks: expect.arrayContaining([ + expect.objectContaining({ + name: "mapping.overlay", + observed: true, + matched: false, + }), + ]), + }); + } } else { expect(result).toMatchObject({ ok: false, @@ -894,7 +959,9 @@ describe("Ghidra measured load-image projection", () => { } }, ); +}); +describe("Ghidra measured DOS load-image projection", () => { it.each([ "verified", "mismatch", diff --git a/src/ghidra/GhidraProvider.ts b/src/ghidra/GhidraProvider.ts index c48f58957..0a1cde0f0 100644 --- a/src/ghidra/GhidraProvider.ts +++ b/src/ghidra/GhidraProvider.ts @@ -66,6 +66,7 @@ const SUPPORTED_ARCHITECTURES = new Set([ "arm", "arm64", "mips", + "6502", "riscv", ]); @@ -184,7 +185,7 @@ export class GhidraProvider implements AnalysisProviderCandidate { status: "unsupported", code: "architecture_unsupported", reason: - "Ghidra v1 requires a supported x86, x86_64, arm, arm64, or MIPS target profile.", + "Ghidra v1 requires a supported x86, x86_64, arm, arm64, MIPS, RISC-V, or raw 6502 target profile.", diagnostics, }; return { @@ -304,11 +305,15 @@ const inspectWindowsP0TargetSupport = ( target: BinaryTarget, diagnostics: Readonly>, ): ProviderTargetSupport => { + // The raw 6502 lane uses the same immutable snapshot and native ownership boundary. + if (target.format === "raw-binary" && target.architecture === "6502") + return { status: "supported", code: null, reason: null, diagnostics }; if (target.format !== "pe") return { status: "unsupported", code: "target_format_unsupported", - reason: "Windows Ghidra P0 accepts PE targets only.", + reason: + "Windows Ghidra P0 accepts native PE and explicitly selected raw 6502 targets only.", diagnostics, }; if (target.architecture !== "x86" && target.architecture !== "x86_64") diff --git a/src/ghidra/GhidraProviderCapabilities.ts b/src/ghidra/GhidraProviderCapabilities.ts index ea863e8a7..68c3045ce 100644 --- a/src/ghidra/GhidraProviderCapabilities.ts +++ b/src/ghidra/GhidraProviderCapabilities.ts @@ -33,7 +33,7 @@ export const healthLimitations = Object.freeze([ /** Additional limitations applied to the experimental Windows x64 P0 boundary. */ export const windowsP0Limitations = Object.freeze([ - "Windows Ghidra P0 accepts approved native x86 and x86-64 PE applications and DLLs; managed, hostile, sensitive, and mutable-path targets are unsupported.", + "Windows Ghidra P0 accepts native x86 and x86-64 PE applications and DLLs, and explicitly selected raw 6502 images; managed, hostile, sensitive, and mutable-path targets are unsupported.", "The Windows bridge uses authenticated IPv4 loopback because Node path-based IPC does not expose Java AF_UNIX sockets; the endpoint file contains no bearer token.", "Windows sessions require the matching packaged Windows x64 native addon, local NTFS targets and runtimes, and Windows 10 or later. Native handles enforce path admission, private DACLs, and Job Object ownership automatically.", ]); diff --git a/src/ghidra/GhidraProviderClient.ts b/src/ghidra/GhidraProviderClient.ts index 5a4f4eafd..3a15dbc4e 100644 --- a/src/ghidra/GhidraProviderClient.ts +++ b/src/ghidra/GhidraProviderClient.ts @@ -32,6 +32,7 @@ import { AnalysisTimeoutError, } from "../domain/analysisErrorCore.js"; import type { AnalysisError } from "../domain/analysisErrorBase.js"; +import type { EvidenceLocation } from "../domain/evidence.js"; import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js"; import { ProviderAdapterError } from "../domain/providerAdapterError.js"; import { err, ok, type Result } from "../domain/result.js"; @@ -515,7 +516,7 @@ export const createGhidraProviderClient = (input: { ...(target.format === "raw-binary" ? [ "Headerless raw-binary analysis uses the caller-supplied architecture and entry address; file offset 0 is mapped at that address. Headers are not discovered from file bytes.", - "Little-endian generic Ghidra languages are used for x86, x86_64, arm, and arm64. Big-endian, Thumb-only, Cortex-M, and other processor variants are unsupported in this increment.", + "Little-endian generic Ghidra languages are used for x86, x86_64, arm, arm64, and NMOS 6502. Big-endian, Thumb-only, Cortex-M, 65C02, 65816, and other processor variants are unsupported in this increment.", "Static analysis does not emulate reset vectors, memory-mapped I/O, or firmware boot sequences.", ] : []), @@ -678,10 +679,12 @@ export const createGhidraProviderClient = (input: { normalized: JsonValue, rawResult: JsonValue, observationLimitations: readonly string[] = [], + locations: readonly EvidenceLocation[] = [], ) => { const relationshipCoverage = relationshipCoverageFor(operation); return createAnalysisExecution(normalized, committedProfile.provider, { rawResult, + locations, analysisProfile: committedProfile, ...(relationshipCoverage === undefined ? {} : { relationshipCoverage }), limitations: [ @@ -776,11 +779,13 @@ export const createGhidraProviderClient = (input: { ); let normalized: JsonValue; let observationLimitations: readonly string[] = []; + let observationLocations: readonly EvidenceLocation[] = []; if (isGhidraFunctionOperation(operation)) { const result = parseGhidraFunctionResult(operation, called.value); if (!result.ok) return result; normalized = result.value.value; observationLimitations = result.value.limitations; + observationLocations = result.value.locations ?? []; } else { const result = parseGhidraInventoryResult(operation, called.value); if (!result.ok) return result; @@ -875,6 +880,7 @@ export const createGhidraProviderClient = (input: { normalized, called.value, observationLimitations, + observationLocations, ), ); }, diff --git a/src/ghidra/GhidraPspExtension.ts b/src/ghidra/GhidraPspExtension.ts index 1a3769378..9c017f367 100644 --- a/src/ghidra/GhidraPspExtension.ts +++ b/src/ghidra/GhidraPspExtension.ts @@ -13,6 +13,7 @@ import { import type { AnalysisError } from "../domain/analysisErrorBase.js"; import { err, ok, type Result } from "../domain/result.js"; import type { GhidraInstallationInspection } from "./GhidraInstallation.js"; +import { parseGhidraJavaProperties } from "./GhidraJavaProperties.js"; /** Installed processor code and data, distinct from REA's analysis-extension ABI. */ export const ghidraPspExtensionSchema = z.strictObject({ @@ -88,7 +89,7 @@ export const inspectGhidraPspExtension = async ( throw new Error("The installed Allegrex extension has no loader JAR."); const hash = createHash("sha256"); let bytesRead = 0; - let properties = ""; + let properties = Buffer.alloc(0); for (const relative of files) { signal?.throwIfAborted(); const path = join(root, relative); @@ -135,22 +136,22 @@ export const inspectGhidraPspExtension = async ( ); bytesRead += before.size; if (relative === "extension.properties") - properties = Buffer.concat(chunks).toString("utf8"); + properties = Buffer.concat(chunks); } finally { await handle.close(); } } // Ghidra's extension.properties version is the compatible Ghidra version, // not the extension release (e.g. v21.4); content identity supplies the latter. - const property = (name: string): string => { - const values = properties.split(/\r?\n/u).flatMap((line) => { - const match = /^\s*([A-Za-z][A-Za-z0-9]*)\s*=\s*(.*?)\s*$/u.exec(line); - return match?.[1] === name ? [match[2] ?? ""] : []; - }); - if (values.length !== 1) - throw new Error(`Allegrex extension must declare exactly one ${name}.`); - return values[0] ?? ""; - }; + const parsedProperties = parseGhidraJavaProperties( + properties.toString("latin1"), + ); + if (!parsedProperties.ok) + throw new Error( + `Invalid Allegrex extension properties: ${parsedProperties.error}`, + ); + const property = (name: string): string | undefined => + parsedProperties.value.get(name); if ( property("name") !== "ghidra-allegrex" || property("version") !== ghidraVersion diff --git a/src/ghidra/GhidraRawBinaryLanguage.ts b/src/ghidra/GhidraRawBinaryLanguage.ts index ea024bdba..e9046975d 100644 --- a/src/ghidra/GhidraRawBinaryLanguage.ts +++ b/src/ghidra/GhidraRawBinaryLanguage.ts @@ -1,28 +1,51 @@ -import type { HeaderArchitecture } from "../domain/binaryTargetTypes.js"; +import type { RawBinaryAdmission } from "../domain/rawBinary.js"; /** Bound Ghidra language for a caller-supplied raw-binary architecture. */ export interface GhidraRawBinaryLanguage { readonly languageId: string; readonly compilerSpecId: "default" | "gcc"; + readonly defaultAddressSpace: "ram" | "RAM"; } /** - * Map the four caller-visible CPU families to little-endian generic Ghidra + * Map caller-visible CPU families to little-endian generic Ghidra * languages with a supported generic compiler specification. These are * analysis conventions, not evidence of the target's original compiler. * Big-endian, Thumb-only, and Cortex-M variants are out of scope. */ export const ghidraRawBinaryLanguage = ( - architecture: HeaderArchitecture, + architecture: RawBinaryAdmission["architecture"], ): GhidraRawBinaryLanguage => { switch (architecture) { + case "6502": + return { + languageId: "6502:LE:16:default", + compilerSpecId: "default", + defaultAddressSpace: "RAM", + }; case "x86": - return { languageId: "x86:LE:32:default", compilerSpecId: "gcc" }; + return { + languageId: "x86:LE:32:default", + compilerSpecId: "gcc", + defaultAddressSpace: "ram", + }; case "x86_64": - return { languageId: "x86:LE:64:default", compilerSpecId: "gcc" }; + return { + languageId: "x86:LE:64:default", + compilerSpecId: "gcc", + defaultAddressSpace: "ram", + }; case "arm": - return { languageId: "ARM:LE:32:v7", compilerSpecId: "default" }; + return { + languageId: "ARM:LE:32:v7", + compilerSpecId: "default", + defaultAddressSpace: "ram", + }; case "arm64": - return { languageId: "AARCH64:LE:64:v8A", compilerSpecId: "default" }; + return { + languageId: "AARCH64:LE:64:v8A", + compilerSpecId: "default", + defaultAddressSpace: "ram", + }; } }; diff --git a/src/ghidra/GhidraRawBinaryLoadImage.test.ts b/src/ghidra/GhidraRawBinaryLoadImage.test.ts index 2fc11502e..d09b3a504 100644 --- a/src/ghidra/GhidraRawBinaryLoadImage.test.ts +++ b/src/ghidra/GhidraRawBinaryLoadImage.test.ts @@ -9,23 +9,161 @@ describe("raw-binary Ghidra language binding", () => { expect(ghidraRawBinaryLanguage("x86")).toEqual({ languageId: "x86:LE:32:default", compilerSpecId: "gcc", + defaultAddressSpace: "ram", }); expect(ghidraRawBinaryLanguage("x86_64")).toEqual({ languageId: "x86:LE:64:default", compilerSpecId: "gcc", + defaultAddressSpace: "ram", }); expect(ghidraRawBinaryLanguage("arm")).toEqual({ languageId: "ARM:LE:32:v7", compilerSpecId: "default", + defaultAddressSpace: "ram", }); expect(ghidraRawBinaryLanguage("arm64")).toEqual({ languageId: "AARCH64:LE:64:v8A", compilerSpecId: "default", + defaultAddressSpace: "ram", + }); + }); +}); + +describe("raw-binary mapped-block state", () => { + it.each(["RAM", "ram"])( + "requires the bound 6502 address space for %s observations", + (space) => { + const f = fixtureRawBinaryLoadImage(); + const target = { + ...f.target, + architecture: "6502" as const, + availableArchitectures: ["6502" as const], + entryPoint: "0x8001", + entryPointSource: "0x8001", + }; + f.observation.language_id = "6502:LE:16:default"; + f.observation.default_address_space = space; + f.observation.entry_points = [target.entryPoint]; + const mapping = f.observation.mappings[0]; + if (mapping === undefined) + throw new Error("Missing source mapping fixture"); + mapping.start = target.entryPoint; + mapping.end = "0x8004"; + mapping.address_space = space; + expect( + attestGhidraRawBinaryLoadImage(f.snapshot, target, f.observation), + ).toMatchObject({ + ok: true, + value: { + status: space === "RAM" ? "verified" : "mismatch", + checks: expect.arrayContaining([ + expect.objectContaining({ + name: "default_address_space", + expected: "RAM", + observed: space, + matched: space === "RAM", + }), + ]), + }, + }); + }, + ); + it.each([ + ["initialized", false], + ["loaded", false], + ["overlay", true], + ["address_space", "overlay"], + ] as const)( + "retains a mismatch when the file mapping declares %s=%s", + (property, value) => { + const f = fixtureRawBinaryLoadImage(); + const mapping = f.observation.mappings[0]; + if (mapping === undefined) + throw new Error("Missing source mapping fixture"); + Object.assign(mapping, { [property]: value }); + expect( + attestGhidraRawBinaryLoadImage(f.snapshot, f.target, f.observation), + ).toMatchObject({ + ok: true, + value: { + status: "mismatch", + observations: f.observation, + checks: expect.arrayContaining([ + expect.objectContaining({ + name: `mapping.${property}`, + matched: false, + observed: value, + }), + ]), + }, + }); + }, + ); + it("does not verify a foreign default address space even when its mapping agrees", () => { + const f = fixtureRawBinaryLoadImage(); + f.observation.default_address_space = "foreign"; + const mapping = f.observation.mappings[0]; + if (mapping === undefined) + throw new Error("Missing source mapping fixture"); + mapping.address_space = "foreign"; + expect( + attestGhidraRawBinaryLoadImage(f.snapshot, f.target, f.observation), + ).toMatchObject({ + ok: true, + value: { + status: "mismatch", + observations: f.observation, + checks: expect.arrayContaining([ + expect.objectContaining({ + name: "default_address_space", + matched: false, + observed: "foreign", + }), + ]), + }, }); }); }); describe("raw-binary independent load-image checks", () => { + it("retains uninitialized language memory without treating it as file-backed bytes", () => { + const f = fixtureRawBinaryLoadImage(); + const zeroPage = { + block: "ZERO_PAGE", + start: "0x0", + end: "0xff", + address_space: "ram", + initialized: false, + loaded: true, + overlay: false, + length: 256, + source_file_index: null, + file_offset: null, + sha256: null, + }; + f.observation.mappings.unshift(zeroPage); + expect( + attestGhidraRawBinaryLoadImage(f.snapshot, f.target, f.observation), + ).toMatchObject({ + ok: true, + value: { status: "verified", observations: f.observation }, + }); + zeroPage.initialized = true; + expect( + attestGhidraRawBinaryLoadImage(f.snapshot, f.target, f.observation), + ).toMatchObject({ + ok: true, + value: { + status: "mismatch", + checks: expect.arrayContaining([ + expect.objectContaining({ + name: "initialized_unbacked_mappings", + matched: false, + }), + ]), + }, + }); + }); it("verifies mapped bytes at the caller-supplied entry", () => { const f = fixtureRawBinaryLoadImage(); const result = attestGhidraRawBinaryLoadImage( diff --git a/src/ghidra/GhidraRiscvProfile.test.ts b/src/ghidra/GhidraRiscvProfile.test.ts index 9bbb65cf3..718fe8d46 100644 --- a/src/ghidra/GhidraRiscvProfile.test.ts +++ b/src/ghidra/GhidraRiscvProfile.test.ts @@ -1,6 +1,6 @@ import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; -import { join } from "node:path"; +import { join, parse, relative } from "node:path"; import { afterAll, beforeAll, describe, expect, it } from "vitest"; import { parseConfig } from "../config/parseConfig.js"; import type { @@ -53,7 +53,7 @@ const RISCV_LDEFS = ` const host: GhidraInstallationHost = { platform: "linux", architecture: "x64", - readText: () => "application.version=12.1.4\n", + readText: () => "application.version=12.1.4\napplication.java.min=21\n", executable: () => true, probeJava: () => ({ version: "21.0.11", @@ -92,7 +92,14 @@ const target = (change: Partial = {}): BinaryTarget => ({ let installDir = ""; beforeAll(async () => { - installDir = await mkdtemp(join(tmpdir(), "rea-ghidra-riscv-languages-")); + const root = + process.platform === "win32" ? join(process.cwd(), "build") : tmpdir(); + await mkdir(root, { recursive: true }); + installDir = await mkdtemp(join(root, "rea-ghidra-riscv-languages-")); + // The simulated Linux host needs a POSIX absolute path. A current-drive root + // path also reaches this real fixture on Windows without changing admission. + if (process.platform === "win32") + installDir = `/${relative(parse(installDir).root, installDir).replaceAll("\\", "/")}`; const languages = join(installDir, "Ghidra/Processors/RISCV/data/languages"); await mkdir(languages, { recursive: true }); await writeFile(join(languages, "riscv.ldefs"), RISCV_LDEFS); diff --git a/src/ghidra/GhidraSnapshotCache.test.ts b/src/ghidra/GhidraSnapshotCache.test.ts index 703bb4906..9cf5c24ad 100644 --- a/src/ghidra/GhidraSnapshotCache.test.ts +++ b/src/ghidra/GhidraSnapshotCache.test.ts @@ -39,7 +39,8 @@ const fixture = (version = "12.1.4") => { { platform: "linux", architecture: "x64", - readText: () => `application.version=${version}\n`, + readText: () => + `application.version=${version}\napplication.java.min=21\n`, executable: () => true, probeJava: () => ({ version: "21.0.11", diff --git a/src/go/GoBinaryImage.ts b/src/go/GoBinaryImage.ts index 03d0866ab..4a845c197 100644 --- a/src/go/GoBinaryImage.ts +++ b/src/go/GoBinaryImage.ts @@ -8,6 +8,10 @@ import { type GoBinaryContainer, type GoFileMapping, } from "./GoBinaryContainer.js"; +import { + classifyFatContainerIdentity, + readFatContainerHeader, +} from "../domain/apple/machoContainer.js"; import { readGoElfImage } from "./GoElfImage.js"; import { readGoMachoImage } from "./GoMachoImage.js"; import { readGoPeImage } from "./GoPeImage.js"; @@ -187,11 +191,27 @@ const readContainer = (bytes: Buffer): GoBinaryContainer => { return readGoPeImage(bytes); if (bytes.length >= 4) { const magic = bytes.readUInt32BE(); - if ([0xcafebabe, 0xcafebabf, 0xbebafeca, 0xbfbafeca].includes(magic)) - throw new GoBinaryFormatFailure( - "unsupported", - "Universal/fat Mach-O containers require selecting a thin slice", - ); + if ([0xcafebabe, 0xcafebabf, 0xbebafeca, 0xbfbafeca].includes(magic)) { + const fatHeader = readFatContainerHeader(bytes, bytes.length); + if (fatHeader.state === "malformed") + throw new GoBinaryFormatFailure("malformed", fatHeader.reason); + const identity = + fatHeader.state === "parsed" + ? classifyFatContainerIdentity(fatHeader, (offset) => { + if (offset >= bytes.length) return undefined; + if (offset + 4 > bytes.length) return null; + return bytes.readUInt32BE(offset); + }) + : undefined; + // An invalid FAT-shaped table is not proof of a Mach-O container: JVM + // constant-pool bytes can share both the magic and a known CPU value. + const confirmedFatMachO = identity?.state === "mach-o"; + if (confirmedFatMachO) + throw new GoBinaryFormatFailure( + "unsupported", + "Universal/fat Mach-O containers require selecting a thin slice", + ); + } if ([0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe].includes(magic)) return readGoMachoImage(bytes); } diff --git a/src/go/GoBinaryProvider.ts b/src/go/GoBinaryProvider.ts index a1c0cc6a3..466136f3f 100644 --- a/src/go/GoBinaryProvider.ts +++ b/src/go/GoBinaryProvider.ts @@ -1,5 +1,7 @@ import { readStableArtifact } from "../artifacts/readStableArtifact.js"; +import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js"; import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js"; +import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js"; import type { ExecutionOptions } from "../application/AnalysisProvider.js"; import type { GoBinaryPort } from "../application/go/GoBinaryPort.js"; import { AnalysisError } from "../domain/analysisErrorBase.js"; @@ -18,6 +20,7 @@ import { type InspectGoBinaryInput, } from "../domain/go/goBinary.js"; import { ProviderAdapterError } from "../domain/providerAdapterError.js"; +import { ProviderCleanupError } from "../domain/providerCleanupError.js"; import { err, ok, type Result } from "../domain/result.js"; import { readGoBinaryImage, @@ -36,13 +39,19 @@ const OPERATION = "inspect_go_binary"; const INPUT_BYTES = 256 * 1024 * 1024; const OUTPUT_BYTES = 8 * 1024 * 1024; -const failureFor = ( +const failureForPrimary = ( cause: unknown, path: string, signal?: AbortSignal, ): AnalysisError => { if (cause instanceof AnalysisError) return cause; - if (signal?.aborted) return new AnalysisCancelledError(OPERATION); + if (cause instanceof ArtifactReaderFailure && cause.reason === "integrity") + return new AnalysisArtifactChangedError(OPERATION, path, cause.message, { + cause, + }); + if (cause instanceof ArtifactReaderFailure && cause.reason === "cancelled") + return new AnalysisCancelledError(OPERATION, { cause }); + if (signal?.aborted) return new AnalysisCancelledError(OPERATION, { cause }); if (cause instanceof GoBinaryResourceFailure) return new AnalysisResourceConstraintError( OPERATION, @@ -64,12 +73,6 @@ const failureFor = ( }, ]); if (cause instanceof ArtifactReaderFailure) { - if (cause.reason === "integrity") - return new AnalysisArtifactChangedError(OPERATION, path, cause.message, { - cause, - }); - if (cause.reason === "cancelled") - return new AnalysisCancelledError(OPERATION); if (cause.reason === "limit") return new AnalysisResourceConstraintError( OPERATION, @@ -110,6 +113,23 @@ const failureFor = ( }); }; +const failureFor = ( + cause: unknown, + path: string, + signal?: AbortSignal, +): AnalysisError => { + const primary = failureForPrimary(cause, path, signal); + if (!(cause instanceof ArtifactReaderFailure) || cause.cleanup === undefined) + return primary; + const cleanup = new ProviderCleanupError( + GO_BINARY_PROVIDER_IDENTITY.id, + cause.cleanup.resources, + { reason: cause.cleanup.reason }, + { operation: OPERATION, cause }, + ); + return analysisErrorWithCleanupFailure(primary, cleanup, OPERATION); +}; + const outputLimit = (path: string) => new AnalysisResourceConstraintError( OPERATION, @@ -121,6 +141,11 @@ const outputLimit = (path: string) => /** Stable in-process metadata inspection without subprocesses, temporary files or Go installation. */ export class GoBinaryProvider implements GoBinaryPort { readonly identity = GO_BINARY_PROVIDER_IDENTITY; + readonly #resources = new ArtifactResourceScope(); + + close(): Promise { + return this.#resources.close(); + } /** Read complete compiler/module metadata while preserving original byte identity and source ranges. */ async inspect( @@ -128,10 +153,14 @@ export class GoBinaryProvider implements GoBinaryPort { options?: ExecutionOptions, ): Promise> { try { - const snapshot = await readStableArtifact( - input.path, - INPUT_BYTES, - options?.signal, + const snapshot = await this.#resources.run(() => + readStableArtifact( + input.path, + INPUT_BYTES, + options?.signal, + undefined, + this.#resources, + ), ); options?.signal?.throwIfAborted(); const image = readGoBinaryImage(snapshot.bytes); diff --git a/src/hopper/BridgeLauncher.ts b/src/hopper/BridgeLauncher.ts index 8201ee2df..043d394a4 100644 --- a/src/hopper/BridgeLauncher.ts +++ b/src/hopper/BridgeLauncher.ts @@ -413,6 +413,8 @@ const launchHopperProcess = async (input: { runId: input.session.runId, expectedCommand: ownershipCommand, hostEnvironment: input.options.environment, + // The helper observes EOF when this REA process exits, including SIGKILL. + stdin: "pipe", ...(input.signal === undefined ? {} : { signal: input.signal }), }); return spawn(input.argumentsForTarget); @@ -434,6 +436,7 @@ export const linuxDemoLaunch = ( return undefined; const helperArguments = [ options.demoHelperPath, + "--owner-stdin", "--strategy", strategy ?? "direct", ...(strategy === "user-mount-namespace" ? ["--mount-private-x11"] : []), diff --git a/src/hopper/HopperAnalysisProfile.ts b/src/hopper/HopperAnalysisProfile.ts index 299862a97..a4848533f 100644 --- a/src/hopper/HopperAnalysisProfile.ts +++ b/src/hopper/HopperAnalysisProfile.ts @@ -123,6 +123,8 @@ export const hopperLoaderArgsForTarget = ( if (target.kind === "database") return ok([]); if (target.kind !== "executable") return err(new ProviderAdapterError("hopper", "resolve_analysis_profile")); + if (target.format === "raw-binary") + return err(new ProviderAdapterError("hopper", "resolve_analysis_profile")); const architecture = target.architecture; if (isElfMetadataArchitecture(architecture)) return err( @@ -155,7 +157,6 @@ export const hopperLoaderArgsForTarget = ( return ok(["-l", "WinPE", flag]); case "dos-mz": case "dos-com": - case "raw-binary": return err( new ProviderAdapterError("hopper", "resolve_analysis_profile"), ); diff --git a/src/hopper/HopperMachOImage.ts b/src/hopper/HopperMachOImage.ts index 20a53105c..ed81ea749 100644 --- a/src/hopper/HopperMachOImage.ts +++ b/src/hopper/HopperMachOImage.ts @@ -2,11 +2,7 @@ import { createHash } from "node:crypto"; import { open, type FileHandle } from "node:fs/promises"; import { z } from "zod"; -import { - isMachOFormat, - isElfMetadataArchitecture, - type BinaryTarget, -} from "../domain/binaryTargetTypes.js"; +import type { BinaryTarget } from "../domain/binaryTargetTypes.js"; import { BinaryTargetError } from "../domain/configurationErrors.js"; import { AnalysisCancelledError } from "../domain/analysisErrorCore.js"; import { @@ -48,10 +44,7 @@ export const resolveHopperMachOImage = async ( ), ); if (signal?.aborted) return err(new AnalysisCancelledError("open_binary")); - if ( - !isMachOFormat(target.format) || - isElfMetadataArchitecture(target.architecture) - ) + if (target.format !== "mach-o" && target.format !== "mach-o-universal") return invalid("Hopper FAT64 image preparation requires a Mach-O target"); let file: FileHandle; try { diff --git a/src/ida/IdaBusyHealth.test.ts b/src/ida/IdaBusyHealth.test.ts index 512cf78a6..b997b2591 100644 --- a/src/ida/IdaBusyHealth.test.ts +++ b/src/ida/IdaBusyHealth.test.ts @@ -18,6 +18,7 @@ const scripted = (answers: readonly JsonValue[]) => { connect: async () => [], serverInfo: () => null, close: async () => undefined, + detach: () => undefined, call: async () => { reads.push(reads.length); const answer = answers[Math.min(reads.length - 1, answers.length - 1)]; diff --git a/src/ida/IdaContainment.test.ts b/src/ida/IdaContainment.test.ts new file mode 100644 index 000000000..069388df8 --- /dev/null +++ b/src/ida/IdaContainment.test.ts @@ -0,0 +1,623 @@ +import { rm } from "node:fs/promises"; +import { afterEach, describe, expect, it } from "vitest"; +import type { JsonValue } from "../domain/jsonValue.js"; +import { + createIdaTarget, + RecordingIdaMcp, +} from "../../tests/fixtures/idaMcp.js"; +import { decodeIdaToolResult } from "./IdaMcpConnection.js"; +import { idaCapabilities } from "./IdaProviderCapabilities.js"; +import { IdaSessionClient } from "./IdaSessionClient.js"; + +const roots: string[] = []; +afterEach(async () => { + for (const root of roots.splice(0)) + await rm(root, { recursive: true, force: true }); +}); + +const segment = (name: string, start: string, end: string) => ({ + name, + start, + end, + size: `0x${(BigInt(end) - BigInt(start)).toString(16)}`, + permissions: "r--", +}); + +/** Adjacent .text and .data, a gap, then a segment above 2^53. */ +const SEGMENTS = [ + segment(".text", "0x1000", "0x2000"), + segment(".data", "0x2000", "0x3000"), + segment(".far", "0x5000", "0x6000"), + segment(".huge", "0x20000000000001", "0x20000000000002"), +]; + +/** The same segments in descending order, which a scan may not rely on. */ +const UNSORTED = [...SEGMENTS].reverse(); + +const survey = (segments: JsonValue = SEGMENTS) => ({ + metadata: { path: "/db/target.i64", module: "target", arch: "metapc" }, + statistics: { total_segments: Array.isArray(segments) ? segments.length : 0 }, + segments, + entrypoints: [], +}); + +const function_ = { addr: "0x1000", name: "main", size: "0x20" }; + +const MODERN_WITHOUT_SURVEY = [ + "idb_open", + "idb_list", + "idb_close", + "server_health", + "list_funcs", + "lookup_funcs", + "find_regex", + "decompile", + "disasm", + "callees", + "xrefs_to", +]; + +interface Upstream { + /** Answer for the lookup of the queried address; default is "Not a function". */ + readonly lookup?: (query: string) => JsonValue; + readonly survey?: JsonValue; + readonly tools?: readonly string[]; + readonly surveyFailure?: "error" | "preview"; + /** A distinct source alias of the admitted target, as for an app bundle. */ + readonly sourcePath?: string; + /** Report a different input from the second health read on. */ + readonly identityChangesAfterOperation?: boolean; +} + +const session = async ( + upstream: Upstream = {}, + mode: "headless" | "attached" = "headless", +) => { + const created = await createIdaTarget("x86_64"); + const { root } = created; + const target = + upstream.sourcePath === undefined + ? created.target + : { ...created.target, sourcePath: upstream.sourcePath }; + roots.push(root); + const producer = new RecordingIdaMcp(target, mode); + if (upstream.tools !== undefined) producer.tools = upstream.tools; + producer.beforeCall = async (name) => { + if (name !== "survey_binary") return; + if (upstream.surveyFailure === "error") + decodeIdaToolResult({ + content: [{ type: "text", text: "survey failed" }], + isError: true, + }); + if (upstream.surveyFailure === "preview") + decodeIdaToolResult({ + content: [{ type: "text", text: "preview" }], + _meta: { ida_mcp: { output_truncated: true } }, + }); + }; + let healthReads = 0; + producer.overrideCall = (name, args) => { + if (name === "server_health" && upstream.identityChangesAfterOperation) { + healthReads += 1; + if (healthReads === 2) + return { + status: "ok", + input_path: "/different/target.elf", + idb_path: "/different/target.elf.i64", + module: "target.elf", + imagebase: "0x1000", + auto_analysis_ready: true, + }; + } + if (name === "lookup_funcs") { + const queries = Array.isArray(args.queries) ? args.queries : []; + return queries.map((query) => + upstream.lookup === undefined + ? { query, fn: null, error: "Not a function" } + : upstream.lookup(String(query)), + ); + } + if (name === "survey_binary") return upstream.survey ?? survey(); + return undefined; + }; + const client = new IdaSessionClient( + { + command: "fixture", + args: [], + env: {}, + mode, + timeoutMs: 1000, + workspaceRoot: root, + }, + target, + producer, + ); + return { client, producer, target }; +}; + +const containing = async ( + client: IdaSessionClient, + address: string, + options?: Parameters[2], +) => client.execute("resolve_containing_procedure", { address }, options); + +const found = (query: string, fn: typeof function_) => ({ + query, + fn, + error: null, +}); + +describe("IDA containment capabilities", () => { + it("declares both operations for the headless profile only", () => { + const declared = (mode: "attached" | "headless", native = false) => + idaCapabilities(mode, false, native) + .map(({ operation }) => operation) + .filter( + (operation) => + operation === "resolve_containing_procedure" || + operation === "list_documents", + ); + + expect( + idaCapabilities("headless", false) + .filter( + ({ operation }) => + operation === "resolve_containing_procedure" || + operation === "list_documents", + ) + .map(({ available }) => available), + "both are declared available, not merely listed", + ).toEqual([true, true]); + expect(declared("headless")).toEqual([ + "resolve_containing_procedure", + "list_documents", + ]); + expect(declared("attached")).toEqual([]); + expect(declared("attached", true)).toEqual([]); + expect( + idaCapabilities("attached", false).map(({ operation }) => operation), + "attached profiles do not even declare them", + ).not.toContain("list_documents"); + }); + + it.each(["resolve_containing_procedure", "list_documents"] as const)( + "refuses %s on the legacy attached profile without calling upstream", + async (operation) => { + const { client, producer } = await session({}, "attached"); + try { + const result = await client.execute( + operation, + operation === "list_documents" ? {} : { address: "0x1000" }, + ); + + expect(result.ok).toBe(false); + if (!result.ok) + expect(result.error._tag).toBe("AnalysisCapabilityUnavailableError"); + expect(producer.calls.some(({ name }) => name === "lookup_funcs")).toBe( + false, + ); + } finally { + expect((await client.close()).ok).toBe(true); + } + }, + ); +}); + +describe("headless IDA containing procedure", () => { + it("returns the canonical entry for an interior address without reading segments", async () => { + const { client, producer } = await session({ + lookup: (query) => found(query, function_), + }); + try { + const result = await containing(client, "1ffe"); + + if (!result.ok) throw result.error; + expect(result.value.result).toEqual({ + query_address: "0x1ffe", + found: true, + procedure: { + address: "0x1000", + name: "main", + classification: null, + body: { + available: false, + reason: "IDA MCP does not report complete function body ranges.", + }, + }, + }); + expect( + producer.calls.filter(({ name }) => name === "lookup_funcs"), + "the normalized address is the only query", + ).toMatchObject([{ args: { queries: ["0x1ffe"] } }]); + expect( + producer.calls.some(({ name }) => name === "survey_binary"), + "a positive answer needs no segment facts", + ).toBe(false); + } finally { + expect((await client.close()).ok).toBe(true); + } + }); + + it.each([ + ["inside the first segment", "0x1000", "not_in_procedure"], + ["inside an adjacent segment", "0x2000", "not_in_procedure"], + [ + "at the exclusive end of the last adjacent segment", + "0x3000", + "outside_segments", + ], + ["in the gap between segments", "0x4000", "outside_segments"], + ["one below a segment start", "0xfff", "outside_segments"], + ["the last byte of a segment", "0x5fff", "not_in_procedure"], + ["above every segment", "0x6000", "outside_segments"], + [ + "inside a one-byte segment above 2^53", + "0x20000000000001", + "not_in_procedure", + ], + [ + "one below that segment, which a rounded number cannot tell apart", + "0x20000000000000", + "outside_segments", + ], + [ + "at the exclusive end of that segment", + "0x20000000000002", + "outside_segments", + ], + ] as const)( + "reports an address that is not a function %s as %s", + async (_description, address, reason) => { + const { client, producer } = await session(); + try { + const result = await containing(client, address); + + if (!result.ok) throw result.error; + expect(result.value.result).toEqual({ + query_address: address, + found: false, + procedure: null, + reason, + }); + expect( + producer.calls.find(({ name }) => name === "survey_binary")?.args, + "the minimal survey is the only segment source", + ).toMatchObject({ detail_level: "minimal" }); + expect( + producer.calls.filter(({ name }) => name === "survey_binary"), + ).toHaveLength(1); + } finally { + expect((await client.close()).ok).toBe(true); + } + }, + ); +}); + +describe("headless IDA containment failures", () => { + it.each([ + ["no segments field", { metadata: {}, statistics: {} }], + ["segments that are null", survey(null)], + ["an empty segment list", survey([])], + [ + "a row with a malformed address", + survey([ + { + name: ".x", + start: "0xzz", + end: "0x2000", + size: "0x1000", + permissions: "r--", + }, + ]), + ], + [ + "a reversed segment", + survey([ + { name: ".x", start: "0x2000", end: "0x1000", permissions: "r--" }, + ]), + ], + [ + "a size that disagrees with the bounds", + survey([{ ...segment(".x", "0x1000", "0x2000"), size: "0x10" }]), + ], + [ + "a total that disagrees with the rows", + { ...survey(), statistics: { total_segments: 9 } }, + ], + ["a malformed row after valid ones", survey([...SEGMENTS, { name: ".y" }])], + ] as const)( + "fails instead of guessing a reason when the survey has %s", + async (_description, reply) => { + const { client } = await session({ survey: reply as JsonValue }); + try { + const result = await containing(client, "0x4000"); + + expect(result.ok).toBe(false); + if (!result.ok) expect(result.error._tag).toBe("AnalysisOutputError"); + } finally { + expect((await client.close()).ok).toBe(true); + } + }, + ); + + it.each(["error", "preview"] as const)( + "fails when the survey tool reports %s", + async (surveyFailure) => { + const { client } = await session({ surveyFailure }); + try { + const result = await containing(client, "0x4000"); + + expect(result.ok).toBe(false); + } finally { + expect((await client.close()).ok).toBe(true); + } + }, + ); + + it("fails without a segment source when the server does not advertise the survey", async () => { + const { client, producer } = await session({ + tools: MODERN_WITHOUT_SURVEY, + }); + try { + const negative = await containing(client, "0x4000"); + + expect(negative.ok).toBe(false); + expect( + producer.calls.some(({ name }) => name === "survey_binary"), + "an unadvertised tool is never requested", + ).toBe(false); + } finally { + expect((await client.close()).ok).toBe(true); + } + }); + + it("still answers a positive lookup when the server does not advertise the survey", async () => { + const { client } = await session({ + lookup: (query) => found(query, function_), + tools: MODERN_WITHOUT_SURVEY, + }); + try { + const result = await containing(client, "0x1010"); + + expect(result.ok).toBe(true); + } finally { + expect((await client.close()).ok).toBe(true); + } + }); +}); + +describe("headless IDA containment lookup failures", () => { + it.each([ + ["an answer for another query", () => found("0x9999", function_)], + [ + "a function together with an error", + (query: string) => ({ query, fn: function_, error: "Not found" }), + ], + [ + "neither a function nor an error", + (query: string) => ({ query, fn: null, error: null }), + ], + [ + "an unknown error text", + (query: string) => ({ query, fn: null, error: "boom" }), + ], + [ + "a padded missing answer", + (query: string) => ({ query, fn: null, error: " Not a function" }), + ], + ] as const)( + "treats %s as a producer failure and reads no segments", + async (_description, lookup) => { + const { client, producer } = await session({ lookup }); + try { + const result = await containing(client, "0x4000"); + + expect(result.ok).toBe(false); + if (!result.ok) expect(result.error._tag).toBe("AnalysisProtocolError"); + expect( + producer.calls.some(({ name }) => name === "survey_binary"), + ).toBe(false); + } finally { + expect((await client.close()).ok).toBe(true); + } + }, + ); + + it('treats "Not found" for an address like any expected missing answer', async () => { + const { client } = await session({ + lookup: (query) => ({ query, fn: null, error: "Not found" }), + }); + try { + const result = await containing(client, "0x4000"); + + if (!result.ok) throw result.error; + expect(result.value.result).toMatchObject({ + found: false, + reason: "outside_segments", + }); + } finally { + expect((await client.close()).ok).toBe(true); + } + }); + + it("stops before the survey when the request is cancelled after the lookup", async () => { + const controller = new AbortController(); + const { client, producer } = await session(); + const inner = producer.beforeCall; + producer.beforeCall = async (name, args) => { + await inner?.(name, args); + if (name === "lookup_funcs") controller.abort(); + }; + try { + const result = await containing(client, "0x4000", { + signal: controller.signal, + }); + + expect(result.ok).toBe(false); + if (!result.ok) expect(result.error._tag).toBe("AnalysisCancelledError"); + expect( + producer.calls.some(({ name }) => name === "survey_binary"), + "nothing is dispatched after cancellation", + ).toBe(false); + } finally { + expect((await client.close()).ok).toBe(true); + } + }); + + it("does not depend on the order of the surveyed segments", async () => { + const { client } = await session({ survey: survey(UNSORTED) }); + try { + const inside = await containing(client, "0x1000"); + const gap = await containing(client, "0x4000"); + + if (!inside.ok) throw inside.error; + if (!gap.ok) throw gap.error; + expect(inside.value.result).toMatchObject({ reason: "not_in_procedure" }); + expect(gap.value.result).toMatchObject({ reason: "outside_segments" }); + } finally { + expect((await client.close()).ok).toBe(true); + } + }); + + it("retains the completed replies when the survey is rejected", async () => { + const { client } = await session({ + survey: survey([ + { name: ".x", start: "0x2000", end: "0x1000", permissions: "r--" }, + ]), + }); + try { + const result = await containing(client, "0x4000"); + + expect(result.ok).toBe(false); + if (!result.ok) + expect(result.error.partialObservation).toMatchObject({ + normalized_result: { + status: "partial", + observations: [ + expect.objectContaining({ tool: "lookup_funcs" }), + expect.objectContaining({ tool: "survey_binary" }), + ], + }, + }); + } finally { + expect((await client.close()).ok).toBe(true); + } + }); + + it("rejects an address that is not hexadecimal and a document that is not the admitted target", async () => { + const { client, producer } = await session(); + try { + const badAddress = await containing(client, "main"); + const badDocument = await client.execute("resolve_containing_procedure", { + address: "0x1000", + document: "/somewhere/else", + }); + + for (const result of [badAddress, badDocument]) { + expect(result.ok).toBe(false); + if (!result.ok) expect(result.error._tag).toBe("AnalysisInputError"); + } + expect( + producer.calls.some(({ name }) => name === "lookup_funcs"), + "invalid input never reaches upstream", + ).toBe(false); + } finally { + expect((await client.close()).ok).toBe(true); + } + }); +}); + +describe("headless IDA document identity", () => { + it("accepts an explicit admitted document and rejects an unknown one for the same operations", async () => { + const { client, target, producer } = await session(); + try { + const admitted = await client.execute("resolve_containing_procedure", { + address: "0x4000", + document: target.path, + }); + const unknownContainment = await client.execute( + "resolve_containing_procedure", + { address: "0x4000", document: "/somewhere/else" }, + ); + const unknownListing = await client.execute("list_documents", { + document: "/somewhere/else", + }); + + expect(admitted.ok, "the admitted program is accepted").toBe(true); + for (const rejected of [unknownContainment, unknownListing]) { + expect(rejected.ok).toBe(false); + if (!rejected.ok) + expect(rejected.error._tag).toBe("AnalysisInputError"); + } + expect( + producer.calls.filter(({ name }) => name === "lookup_funcs"), + "only the admitted request reached upstream", + ).toHaveLength(1); + } finally { + expect((await client.close()).ok).toBe(true); + } + }); + + it("lists the admitted path, not its source alias or the private copy", async () => { + const { client, target, producer } = await session({ + sourcePath: "/Applications/Fixture.app", + }); + try { + const result = await client.execute("list_documents", {}); + + if (!result.ok) throw result.error; + expect(result.value.result).toEqual([target.path]); + expect( + producer.calls.filter(({ name }) => name === "server_health"), + "the target identity is checked before and after the local answer", + ).toHaveLength(2); + } finally { + expect((await client.close()).ok).toBe(true); + } + }); + + it("fails when the target identity changes during the operation", async () => { + const { client } = await session({ identityChangesAfterOperation: true }); + try { + const result = await client.execute("list_documents", {}); + + expect(result.ok).toBe(false); + if (!result.ok) { + expect(result.error.message).toContain("identity changed"); + } + } finally { + expect((await client.close()).ok).toBe(true); + } + }); + + it("stops without dispatching when the request is already cancelled", async () => { + const controller = new AbortController(); + controller.abort(); + const { client, producer } = await session(); + try { + const result = await client.execute( + "list_documents", + {}, + { signal: controller.signal }, + ); + + expect(result.ok).toBe(false); + if (!result.ok) expect(result.error._tag).toBe("AnalysisCancelledError"); + expect(producer.calls).toEqual([]); + } finally { + expect((await client.close()).ok).toBe(true); + } + }); + + it("lists exactly the admitted program", async () => { + const { client, target } = await session(); + try { + const result = await client.execute("list_documents", {}); + + if (!result.ok) throw result.error; + expect(result.value.result).toEqual([target.path]); + } finally { + expect((await client.close()).ok).toBe(true); + } + }); +}); diff --git a/src/ida/IdaMcpConnection.ts b/src/ida/IdaMcpConnection.ts index ee7861389..d532a66e8 100644 --- a/src/ida/IdaMcpConnection.ts +++ b/src/ida/IdaMcpConnection.ts @@ -1,3 +1,5 @@ +import { ChildProcess } from "node:child_process"; +import { Socket } from "node:net"; import { snapshotEnvironment } from "../process/snapshotEnvironment.js"; import { Client, @@ -22,6 +24,12 @@ export interface IdaMcpConnection { readResource?(uri: string): Promise; serverInfo(): JsonValue; close(): Promise; + /** + * Keep a connection retained after failed cleanup from holding the host + * process open. A long-lived MCP server can still retry through it; a + * one-shot CLI exits, and the upstream proxy observes end of input. + */ + detach(): void; } const toolResultSchema = z.object({ @@ -205,5 +213,14 @@ export const createIdaMcpConnection = ( }, serverInfo: () => jsonValueSchema.parse(client.getServerVersion() ?? null), close: () => client.close().catch(transportFailure), + detach() { + if (!(transport instanceof StdioClientTransport)) return; + // The SDK exposes only the child's pid, not the handles it holds. + const child: unknown = Reflect.get(transport, "_process"); + if (!(child instanceof ChildProcess)) return; + child.unref(); + for (const stream of [child.stdin, child.stdout, child.stderr]) + if (stream instanceof Socket) stream.unref(); + }, }; }; diff --git a/src/ida/IdaNativeProtocol.test.ts b/src/ida/IdaNativeProtocol.test.ts index d59f598dd..dc95727f6 100644 --- a/src/ida/IdaNativeProtocol.test.ts +++ b/src/ida/IdaNativeProtocol.test.ts @@ -90,6 +90,7 @@ const fixture = async () => { }, serverInfo: () => producer.serverInfo(), close: () => producer.close(), + detach: () => producer.detach(), }; const client = new IdaSessionClient( { @@ -758,3 +759,33 @@ describe("native IDA malformed observations", () => { await client.close(); }); }); + +describe("native IDA attached operation admission", () => { + it("forwards the native producer's explicit reference cap flag", async () => { + const { client } = await fixture(); + const result = await client.execute("xrefs", { address: "0x1000" }); + + if (!result.ok) throw result.error; + expect(result.value.referencesTruncated).toBe(false); + await client.close(); + }); + + it("does not admit the headless-only containment and document operations", async () => { + const { client, producer } = await fixture(); + for (const [operation, parameters] of [ + ["resolve_containing_procedure", { address: "0x1000" }], + ["list_documents", {}], + ] as const) { + const result = await client.execute(operation, parameters); + + expect(result.ok).toBe(false); + if (!result.ok) + expect(result.error._tag).toBe("AnalysisCapabilityUnavailableError"); + } + expect( + producer.calls.some(({ name }) => name === "lookup_funcs"), + "nothing is dispatched upstream", + ).toBe(false); + await client.close(); + }); +}); diff --git a/src/ida/IdaOperationRunner.ts b/src/ida/IdaOperationRunner.ts index c30ec81bc..25a49c268 100644 --- a/src/ida/IdaOperationRunner.ts +++ b/src/ida/IdaOperationRunner.ts @@ -45,6 +45,7 @@ import { modernStringsSchema, modernXrefsSchema, nextOffset, + surveySegmentsSchema, type IdaDisassembly, type IdaFunction, } from "./IdaProtocolValues.js"; @@ -86,6 +87,8 @@ export class IdaOperationRunner { readonly locations: EvidenceLocation[] = []; readonly limitations: string[] = []; relationshipCoverage: RelationshipCoverage | undefined; + /** The upstream cap flag of the last `xrefs_to` read, when the producer reported one. */ + referencesTruncated: boolean | undefined; constructor( private readonly connection: IdaMcpConnection, private readonly protocol: "legacy" | "modern" | "native", @@ -97,6 +100,10 @@ export class IdaOperationRunner { readonly x86Only?: boolean; /** False when the upstream server does not advertise `callgraph`. */ readonly callGraphAvailable?: boolean; + /** False when the upstream server does not advertise `survey_binary`. */ + readonly surveyAvailable?: boolean; + /** The program identity REA admitted, reported by `list_documents`. */ + readonly admittedDocument?: string; } = {}, ) {} @@ -177,41 +184,89 @@ export class IdaOperationRunner { return entry; } - private lookupFunction( + /** Parse one lookup row: a function, an expected "missing" answer, or a producer failure. */ + private lookupOutcome( query: string, item: z.infer[number] | undefined, - requestedBy?: string, - ): IdaFunction { + ): { readonly fn: IdaFunction } | { readonly missing: string } { if (item?.query !== query) throw new AnalysisProtocolError( `IDA function lookup answered for a different query than ${JSON.stringify(query)}.`, ); if ( - requestedBy !== undefined && item.fn === null && item.error !== null && MISSING_PROCEDURE_ANSWERS.has(item.error) ) - throw new AnalysisInputError( - requestedBy, - { - cause: new AnalysisProtocolError( - `IDA function lookup failed for ${query}: ${item.error}`, - ), - }, - [ - { - path: [], - reason: "invalid_value", - message: `IDA could not resolve procedure ${JSON.stringify(query)}: ${item.error}.`, - }, - ], - ); + return { missing: item.error }; if (item.fn === null || item.error !== null) throw new AnalysisProtocolError( `IDA function lookup failed for ${query}: ${item.error ?? "missing function"}`, ); - return item.fn; + return { fn: item.fn }; + } + + private lookupFunction( + query: string, + item: z.infer[number] | undefined, + requestedBy?: string, + ): IdaFunction { + const outcome = this.lookupOutcome(query, item); + if ("fn" in outcome) return outcome.fn; + if (requestedBy === undefined) + throw new AnalysisProtocolError( + `IDA function lookup failed for ${query}: ${outcome.missing}`, + ); + throw new AnalysisInputError( + requestedBy, + { + cause: new AnalysisProtocolError( + `IDA function lookup failed for ${query}: ${outcome.missing}`, + ), + }, + [ + { + path: [], + reason: "invalid_value", + message: `IDA could not resolve procedure ${JSON.stringify(query)}: ${outcome.missing}.`, + }, + ], + ); + } + + /** The containing function of an address, or why none contains it. */ + private async containingProcedure(address: string): Promise { + const item = modernLookupSchema.parse( + await this.call("lookup_funcs", { queries: [address] }), + )[0]; + const outcome = this.lookupOutcome(address, item); + if ("fn" in outcome) + return { + query_address: address, + found: true, + procedure: identity(outcome.fn), + }; + return { + query_address: address, + found: false, + procedure: null, + reason: (await this.addressInSegment(address)) + ? "not_in_procedure" + : "outside_segments", + }; + } + + /** Segment membership from the database-scoped minimal survey; a missing fact is a failure. */ + private async addressInSegment(address: string): Promise { + if (this.options.surveyAvailable === false) + throw new AnalysisProtocolError( + "The upstream server does not advertise survey_binary, so segment membership cannot be established.", + ); + const { segments } = surveySegmentsSchema.parse( + await this.call("survey_binary", { detail_level: "minimal" }), + ); + const value = BigInt(address); + return segments.some(({ start, end }) => start <= value && value < end); } /** True only for modern callee reads on targets not proven x86-only; x86 reads never reach the graph code. */ @@ -390,6 +445,7 @@ export class IdaOperationRunner { throw new AnalysisProtocolError( `IDA xrefs failed: ${page?.error ?? "missing references"}`, ); + if (page.more !== undefined) this.referencesTruncated = page.more; if (page.more === true) this.limitations.push( "Upstream xrefs_to capped results at 1000 references; additional references remain unknown.", @@ -536,6 +592,16 @@ export class IdaOperationRunner { address === idaAddressSchema.parse(parameters.address), ); } + if (operation === "list_documents") { + if (this.options.admittedDocument === undefined) + throw new AnalysisProtocolError("The admitted IDA program is unknown."); + return [this.options.admittedDocument]; + } + if (operation === "resolve_containing_procedure") { + const queried = idaAddressSchema.parse(parameters.address); + this.locations.push({ kind: "address", address: queried }); + return this.containingProcedure(queried); + } if (operation === "xrefs") { const inspectedAddress = idaAddressSchema.parse(parameters.address); this.locations.push({ kind: "address", address: inspectedAddress }); diff --git a/src/ida/IdaProtocolValues.ts b/src/ida/IdaProtocolValues.ts index 93ddbef88..f18e85ee0 100644 --- a/src/ida/IdaProtocolValues.ts +++ b/src/ida/IdaProtocolValues.ts @@ -218,6 +218,14 @@ export const openDatabaseSchema = z .passthrough(), }) .passthrough(); +/** Only an explicit completed rejection preserves legacy no-worker cleanup. */ +export const rejectedOpenDatabaseSchema = z + .object({ + error: z.string().min(1), + success: z.never().optional(), + }) + .passthrough(); + export const databaseListSchema = z .object({ sessions: z.array( @@ -247,3 +255,53 @@ export const absentDatabaseCloseSchema = (sessionId: string) => z .object({ error: z.literal(`Session not found: ${sessionId}`) }) .passthrough(); + +// Piped, so a malformed coordinate is a schema issue and never reaches BigInt. +const hexCoordinate = z + .string() + .regex(/^0x[0-9a-f]+$/iu) + .pipe(z.transform((value) => BigInt(value))); + +/** + * The minimal `survey_binary` reply, reduced to validated segment intervals + * (exclusive ends). Any malformed, reversed or inconsistent fact rejects the + * whole reply, because a membership answer needs every segment. + */ +export const surveySegmentsSchema = z + .object({ + statistics: z + .object({ total_segments: z.number().int().nonnegative().optional() }) + .passthrough() + .optional(), + segments: z + .array( + z + .object({ + start: hexCoordinate, + end: hexCoordinate, + size: hexCoordinate.optional(), + }) + .passthrough(), + ) + .min(1), + }) + .passthrough() + .superRefine(({ statistics, segments }, context) => { + if ( + statistics?.total_segments !== undefined && + statistics.total_segments !== segments.length + ) + context.addIssue({ + code: "custom", + message: "survey segment count disagrees with its statistics", + }); + for (const { start, end, size } of segments) { + // A malformed row has already reported its own issue; refinements still run. + if (typeof start !== "bigint" || typeof end !== "bigint") continue; + if (start >= end || (size !== undefined && size !== end - start)) + context.addIssue({ + code: "custom", + message: "survey segment bounds are reversed or inconsistent", + }); + } + }); diff --git a/src/ida/IdaProviderCapabilities.ts b/src/ida/IdaProviderCapabilities.ts index d52c949e0..0ccc016d2 100644 --- a/src/ida/IdaProviderCapabilities.ts +++ b/src/ida/IdaProviderCapabilities.ts @@ -24,9 +24,23 @@ export const IDA_OPERATIONS = Object.freeze([ "read_function_instructions", "xrefs", "analyze_function", + "resolve_containing_procedure", + "list_documents", ] as const satisfies readonly Exclude[]); export type IdaOperation = (typeof IDA_OPERATIONS)[number]; +/** Operations only the modern headless profile implements. */ +const HEADLESS_ONLY_OPERATIONS: ReadonlySet = new Set([ + "resolve_containing_procedure", + "list_documents", +]); + +/** Whether the configured profile admits an operation. */ +export const idaProfileAdmits = ( + operation: IdaOperation, + mode: "attached" | "headless", +): boolean => mode === "headless" || !HEADLESS_ONLY_OPERATIONS.has(operation); + /** Shared limitations of observations from an externally mutable analysis database. */ export const IDA_LIMITATIONS = Object.freeze([ "Provider version identifies the REA IDA adapter. IDA engine and upstream distribution versions are unknown unless separately observed; MCP serverInfo is reported independently.", @@ -58,34 +72,37 @@ export const idaCapabilities = ( stdio: boolean, native = false, ): readonly CapabilityDescriptor[] => - IDA_OPERATIONS.map((operation) => ({ - provider: IDA_PROVIDER_IDENTITY, - operation, - ...((mode === "headless" || native) && operation === "procedure_callers" - ? ({ - available: false, - reason: native - ? "The native upstream profile reports code references without proving direct call edges." - : "The modern upstream profile reports code references without proving direct call edges.", - } as const) - : ({ available: true, reason: null } as const)), - cachePolicy: "live", - effects: { - mutatesArtifact: false, - launchesProcess: stdio || mode === "headless", - mayShowUi: - mode === "attached" && - (native - ? operation === "procedure_pseudo_code" || - operation === "analyze_function" - : ATTACHED_GUI_NAVIGATION_OPERATIONS.has(operation)), - mayAccessNetwork: true, - mayWriteFilesystem: stdio || mode === "headless", - changesPermissions: false, - requiresRoot: false, - }, - limitations: - native && (operation === "list_strings" || operation === "search_strings") - ? [...IDA_LIMITATIONS, IDA_NATIVE_STRING_LIMITATION] - : IDA_LIMITATIONS, - })); + IDA_OPERATIONS.filter((operation) => idaProfileAdmits(operation, mode)).map( + (operation) => ({ + provider: IDA_PROVIDER_IDENTITY, + operation, + ...((mode === "headless" || native) && operation === "procedure_callers" + ? ({ + available: false, + reason: native + ? "The native upstream profile reports code references without proving direct call edges." + : "The modern upstream profile reports code references without proving direct call edges.", + } as const) + : ({ available: true, reason: null } as const)), + cachePolicy: "live", + effects: { + mutatesArtifact: false, + launchesProcess: stdio || mode === "headless", + mayShowUi: + mode === "attached" && + (native + ? operation === "procedure_pseudo_code" || + operation === "analyze_function" + : ATTACHED_GUI_NAVIGATION_OPERATIONS.has(operation)), + mayAccessNetwork: true, + mayWriteFilesystem: stdio || mode === "headless", + changesPermissions: false, + requiresRoot: false, + }, + limitations: + native && + (operation === "list_strings" || operation === "search_strings") + ? [...IDA_LIMITATIONS, IDA_NATIVE_STRING_LIMITATION] + : IDA_LIMITATIONS, + }), + ); diff --git a/src/ida/IdaReferenceCoverage.test.ts b/src/ida/IdaReferenceCoverage.test.ts new file mode 100644 index 000000000..930c3a88a --- /dev/null +++ b/src/ida/IdaReferenceCoverage.test.ts @@ -0,0 +1,96 @@ +import { rm } from "node:fs/promises"; +import { afterEach, describe, expect, it } from "vitest"; +import { + createIdaTarget, + RecordingIdaMcp, +} from "../../tests/fixtures/idaMcp.js"; +import { + callGraphSession as session, + removeCallGraphSessions, +} from "../../tests/fixtures/idaCallGraph.js"; + +import { IdaSessionClient } from "./IdaSessionClient.js"; + +const attachedRoots: string[] = []; +afterEach(async () => { + await removeCallGraphSessions(); + for (const root of attachedRoots.splice(0)) + await rm(root, { recursive: true, force: true }); +}); + +const xrefs = async (more: boolean | undefined) => { + const { client, producer } = await session({ callees: [] }, "x86_64"); + const inner = producer.overrideCall; + producer.overrideCall = (name, args) => + name === "xrefs_to" + ? [{ xrefs: [], ...(more === undefined ? {} : { more }) }] + : inner?.(name, args); + try { + const result = await client.execute("xrefs", { address: "0x1000" }); + if (!result.ok) throw result.error; + return result.value; + } finally { + expect((await client.close()).ok).toBe(true); + } +}; + +describe("headless IDA reference coverage", () => { + it("reports a capped xrefs read as truncated", async () => { + const execution = await xrefs(true); + + expect(execution.referencesTruncated).toBe(true); + expect( + execution.limitations, + "the diagnostic limitation is still reported", + ).toContain( + "Upstream xrefs_to capped results at 1000 references; additional references remain unknown.", + ); + }); + + it("reports an explicitly uncapped xrefs read as not truncated", async () => { + expect((await xrefs(false)).referencesTruncated).toBe(false); + }); + + it("reports nothing when the producer does not say whether the read was capped", async () => { + expect((await xrefs(undefined)).referencesTruncated).toBeUndefined(); + }); + + it("does not report reference coverage for other operations", async () => { + const { client } = await session({ callees: [] }, "x86_64"); + try { + const result = await client.execute("procedure_callees", { + procedure: "main", + }); + + if (!result.ok) throw result.error; + expect(result.value.referencesTruncated).toBeUndefined(); + } finally { + expect((await client.close()).ok).toBe(true); + } + }); + + it("reports nothing for the legacy profile, whose producer has no cap flag", async () => { + const { root, target } = await createIdaTarget("x86_64"); + attachedRoots.push(root); + const client = new IdaSessionClient( + { + command: "fixture", + args: [], + env: {}, + mode: "attached", + timeoutMs: 1000, + workspaceRoot: root, + }, + target, + new RecordingIdaMcp(target, "attached"), + ); + try { + const result = await client.execute("xrefs", { address: "0x1000" }); + + if (!result.ok) throw result.error; + expect(result.value.referencesTruncated).toBeUndefined(); + } finally { + expect((await client.close()).ok).toBe(true); + } + }); +}); diff --git a/src/ida/IdaSessionClient.test.ts b/src/ida/IdaSessionClient.test.ts index e43f18cf7..7fa366bce 100644 --- a/src/ida/IdaSessionClient.test.ts +++ b/src/ida/IdaSessionClient.test.ts @@ -16,6 +16,18 @@ afterEach(async () => { for (const root of roots.splice(0)) await rm(root, { recursive: true, force: true }); }); +/** A failed close whose diagnostic names exactly these unverified sessions. */ +const reportsSessions = (...sessions: readonly object[]) => ({ + error: { + diagnostics: { upstream_sessions: { state: "unverified", sessions } }, + }, +}); +const LIVE_PEER = { + session_id: "unrelated-live-peer", + backend: "worker", + owned: false, +} as const; + const fixture = async ( mode: "attached" | "headless" = "attached", timeoutMs = 1000, @@ -515,7 +527,7 @@ describe("headless IDA transient busy health", () => { }); it.each([ - ["free-form text", "token=abc123 /private/home/user secret"], + ["free-form text", "token=abc123 /private secret"], ["an over-long name", "x".repeat(200)], ])( "reports %s from the worker as unknown instead of echoing it", @@ -792,10 +804,19 @@ describe("headless IDA session ownership and cleanup", () => { it("retains the private workspace and reports cleanup failure when worker release is unverified", async () => { const { producer, client, root } = await fixture("headless"); await client.execute("health", {}); + const sessionId = producer.database; producer.failClose = true; const result = await client.close(); expect(result.ok).toBe(false); if (!result.ok) expect(result.error.cleanupIncomplete).toBe(true); + expect(result).toMatchObject( + reportsSessions({ + session_id: sessionId, + backend: "worker", + owned: true, + input_path: producer.inputPath, + }), + ); expect( (await readdir(root)).some((name) => name.startsWith("rea-ida-")), ).toBe(true); @@ -843,10 +864,20 @@ describe("headless IDA reaped worker cleanup", () => { const result = await client.close(); expect(result.ok).toBe(false); if (!result.ok) expect(result.error.cleanupIncomplete).toBe(true); + // The released session must not be named as the remaining worker. + expect(result).toMatchObject(reportsSessions(LIVE_PEER)); expect(producer.closes).toBe(0); expect( (await readdir(root)).some((name) => name.startsWith("rea-ida-")), ).toBe(true); + + // A retry whose inventory is unavailable keeps the observed blocker. + producer.beforeCall = async (name) => { + if (name === "idb_list") throw new Error("inventory unavailable"); + }; + const retried = await client.close(); + expect(retried.ok).toBe(false); + expect(retried).toMatchObject(reportsSessions(LIVE_PEER)); }); it("retries independent inventory verification after an absent-session response", async () => { @@ -899,6 +930,305 @@ describe("headless IDA reaped worker cleanup", () => { }); }); +/** A rejected open response, with helpers for separately inventoried workers. */ +const rejectedOpenFixture = async () => { + const { producer, client } = await fixture("headless"); + producer.overrideCall = (name) => + name === "idb_open" ? { error: "open did not complete" } : undefined; + expect((await client.execute("health", {})).ok).toBe(false); + const open = producer.calls.find(({ name }) => name === "idb_open")?.args; + const entry = (session_id: string, owned: boolean) => ({ + session_id, + input_path: String(open?.input_path), + owned, + backend: "worker", + is_active: true, + }); + return { + producer, + client, + requested: String(open?.preferred_session_id), + entry, + }; +}; + +describe("headless IDA cleanup diagnostics", () => { + it("does not name a worker whose close was confirmed when inventory then fails", async () => { + const { producer, client } = await fixture("headless"); + expect((await client.execute("health", {})).ok).toBe(true); + producer.beforeCall = async (name) => { + if (name === "idb_list") throw new Error("inventory unavailable"); + }; + + const result = await client.close(); + expect(result.ok).toBe(false); + expect(producer.calls.map(({ name }) => name)).toContain("idb_close"); + expect(result).toMatchObject(reportsSessions()); + }); + + it("retains peers and confirmed ownership from successful startup when close fails", async () => { + const { producer, client } = await fixture("headless"); + producer.overrideCall = (name) => { + if (name === "idb_list") + return { + sessions: [ + { + session_id: String(producer.database), + input_path: producer.inputPath, + owned: true, + backend: "worker", + is_active: true, + }, + { + session_id: "startup-peer", + input_path: producer.inputPath, + owned: false, + backend: "worker", + is_active: false, + }, + ], + }; + if (name === "idb_close") return { error: "close rejected" }; + return undefined; + }; + expect((await client.execute("health", {})).ok).toBe(true); + const result = await client.close(); + expect(result).toMatchObject( + reportsSessions( + { + session_id: producer.database, + backend: "worker", + owned: true, + input_path: producer.inputPath, + }, + { + session_id: "startup-peer", + backend: "worker", + owned: false, + input_path: producer.inputPath, + }, + ), + ); + }); + + it("preserves an observed conflicting input binding after close", async () => { + const { producer, client } = await fixture("headless"); + expect((await client.execute("health", {})).ok).toBe(true); + const requested = String(producer.database); + producer.overrideCall = (name) => + name === "idb_list" + ? { + sessions: [ + { + session_id: requested, + input_path: "/different/input", + owned: false, + backend: "worker", + is_active: true, + }, + ], + } + : undefined; + const result = await client.close(); + expect(result).toMatchObject( + reportsSessions({ + session_id: requested, + backend: "worker", + owned: false, + input_path: "/different/input", + }), + ); + }); + + it("keeps peers seen beside the requested worker when its close fails", async () => { + const { producer, client, requested, entry } = await rejectedOpenFixture(); + producer.overrideCall = (name) => { + if (name === "idb_list") + return { + sessions: [ + entry(requested, true), + entry("unrelated-live-peer", false), + ], + }; + if (name === "idb_close") return { error: "close rejected" }; + return undefined; + }; + + const result = await client.close(); + expect(result.ok).toBe(false); + expect(result).toMatchObject( + reportsSessions( + { session_id: requested, backend: "worker", owned: true }, + LIVE_PEER, + ), + ); + }); +}); + +describe("headless IDA cleanup diagnostics for observed peers", () => { + it("drops the requested worker from blockers once its close is confirmed", async () => { + const { producer, client, requested, entry } = await rejectedOpenFixture(); + let inventories = 0; + producer.beforeCall = async (name) => { + if (name === "idb_list" && ++inventories > 1) + throw new Error("inventory unavailable"); + }; + producer.overrideCall = (name) => { + if (name === "idb_list") + return { + sessions: [ + entry(requested, true), + entry("unrelated-live-peer", false), + ], + }; + if (name === "idb_close") + return { + success: true, + session_id: requested, + saved: false, + owned: true, + backend: "worker", + }; + return undefined; + }; + + const result = await client.close(); + expect(result.ok).toBe(false); + expect(result).toMatchObject(reportsSessions(LIVE_PEER)); + }); + + it("reports a blocking GUI session by its backend", async () => { + const { producer, client } = await fixture("headless"); + expect((await client.execute("health", {})).ok).toBe(true); + const sessionId = producer.database; + producer.overrideCall = (name) => { + if (name === "idb_close") + return { error: `Session not found: ${sessionId}` }; + if (name === "idb_list") + return { + sessions: [ + { + session_id: "", + input_path: producer.inputPath, + owned: false, + backend: "gui", + is_active: true, + }, + ], + }; + return undefined; + }; + + const result = await client.close(); + expect(result.ok).toBe(false); + expect(result).toMatchObject( + reportsSessions({ session_id: "", backend: "gui", owned: false }), + ); + }); +}); + +/** A completed open response is not proof that its private Worker was ever released. */ +describe("headless IDA unresolved successful-open ownership", () => { + it.each(["matching", "foreign-input", "malformed"] as const)( + "retains the private input after %s success with incomplete initial discovery", + async (kind) => { + const { producer, client, root } = await fixture("headless"); + let hidden = true; + let ownedId: string | undefined; + let privateInput: string | undefined; + producer.beforeCall = async (name, args) => { + if (name !== "idb_open") return; + ownedId = z.string().parse(args.preferred_session_id); + privateInput = z.string().parse(args.input_path); + producer.database = ownedId; + producer.inputPath = privateInput; + }; + producer.overrideCall = (name) => { + if (name === "idb_open") { + const originalId = z.string().parse(ownedId); + const originalInput = z.string().parse(privateInput); + if (kind === "foreign-input") + return { + success: true, + session: { + session_id: "foreign-session", + input_path: "/foreign/input", + }, + }; + if (kind === "malformed") + return { success: true, session: { session_id: originalId } }; + return { + success: true, + session: { session_id: originalId, input_path: originalInput }, + }; + } + if (name === "idb_list" && hidden) return { sessions: [] }; + return undefined; + }; + + expect((await client.execute("health", {})).ok).toBe(false); + const incomplete = await client.close(); + expect(incomplete.ok).toBe(false); + if (!incomplete.ok) expect(incomplete.error.cleanupIncomplete).toBe(true); + expect(producer.closes).toBe(0); + expect(producer.database).toBe(ownedId); + expect( + (await readdir(root)).some((name) => name.startsWith("rea-ida-")), + ).toBe(true); + expect(producer.calls.some(({ name }) => name === "idb_close")).toBe( + false, + ); + + // A later positive private ownership record permits ordinary save=false + // cleanup, without ever adopting the foreign open-response identity. + hidden = false; + expect((await client.close()).ok).toBe(true); + expect(producer.calls.filter(({ name }) => name === "idb_close")).toEqual( + [{ name: "idb_close", args: { database: ownedId, save: false } }], + ); + expect(producer.closes).toBe(1); + expect(await readdir(root)).toEqual(["target.elf"]); + }, + ); + + it("preserves cleanup after a completed, explicit idb_open rejection", async () => { + const { producer, client, root } = await fixture("headless"); + producer.overrideCall = (name) => + name === "idb_open" + ? { error: "Input was rejected before creating a session" } + : undefined; + + expect((await client.execute("health", {})).ok).toBe(false); + expect((await client.close()).ok).toBe(true); + expect(producer.calls.some(({ name }) => name === "idb_close")).toBe(false); + expect(producer.closes).toBe(1); + expect(await readdir(root)).toEqual(["target.elf"]); + }); + + it("retains cleanup when a rejected open leaves the requested ID on another input", async () => { + const { producer, client, requested } = await rejectedOpenFixture(); + producer.overrideCall = (name) => + name === "idb_list" + ? { + sessions: [ + { + session_id: requested, + input_path: "/different/input", + owned: false, + backend: "worker", + is_active: true, + }, + ], + } + : undefined; + + const result = await client.close(); + expect(result.ok).toBe(false); + if (!result.ok) expect(result.error.cleanupIncomplete).toBe(true); + expect(producer.closes).toBe(0); + }); +}); + describe("headless IDA cleanup verification", () => { it("retries failed worker release through the retained connection before removing its workspace", async () => { const { producer, client, root } = await fixture("headless"); @@ -943,23 +1273,6 @@ describe("headless IDA cleanup verification", () => { ).toHaveLength(1); expect(await readdir(root)).toEqual(["target.elf"]); }); - it("never closes an unrelated database when startup returns a different input identity", async () => { - const { producer, client, root } = await fixture("headless"); - producer.overrideCall = (name) => - name === "idb_open" - ? { - success: true, - session: { - session_id: "user-database", - input_path: "/unrelated/input", - }, - } - : undefined; - expect((await client.execute("health", {})).ok).toBe(false); - expect((await client.close()).ok).toBe(true); - expect(producer.calls.some(({ name }) => name === "idb_close")).toBe(false); - expect(await readdir(root)).toEqual(["target.elf"]); - }); it("releases a privately owned worker after malformed open output using verified session inventory", async () => { const { producer, client, root } = await fixture("headless"); producer.beforeCall = async (name, args) => { diff --git a/src/ida/IdaSessionClient.ts b/src/ida/IdaSessionClient.ts index 7d3ec066c..dc0ae5d6f 100644 --- a/src/ida/IdaSessionClient.ts +++ b/src/ida/IdaSessionClient.ts @@ -33,6 +33,7 @@ import { IDA_LIMITATIONS, IDA_OPERATIONS, IDA_PROVIDER_IDENTITY, + idaProfileAdmits, type IdaOperation, } from "./IdaProviderCapabilities.js"; import { @@ -42,6 +43,7 @@ import { legacyMetadataSchema, modernHealthSchema, openDatabaseSchema, + rejectedOpenDatabaseSchema, } from "./IdaProtocolValues.js"; import { idaFileDigest, @@ -91,11 +93,25 @@ interface IdaOperationFailureContext { parameters: Readonly>; } +/** Upstream session ids of inventory entries, omitting unreported ones. */ +interface UpstreamSession { + readonly session_id: string; + readonly backend: "worker" | "gui"; + /** Upstream ownership; null when no inventory has reported it. */ + readonly owned: boolean | null; + /** Last inventoried input binding; null until upstream reports it. */ + readonly input_path: string | null; +} + /** A failed cleanup retains its private workspace and reports the remaining resource. */ class IdaCleanupError extends ProviderAdapterError { override readonly cleanupIncomplete = true; override readonly cleanupResources: readonly string[]; - constructor(path: string, cause: unknown) { + constructor( + path: string, + cause: unknown, + sessions?: readonly UpstreamSession[], + ) { super("ida", "close", { cause, diagnostics: { @@ -103,6 +119,23 @@ class IdaCleanupError extends ProviderAdapterError { "IDA cleanup could not be verified; the remaining connection or private workspace is retained for retry.", failure_reason: cause instanceof Error ? cause.message : String(cause), workspace: path, + ...(sessions === undefined + ? {} + : { + upstream_sessions: { + state: "unverified", + sessions: sessions.map( + ({ session_id, backend, owned, input_path }) => ({ + session_id, + backend, + owned, + input_path, + }), + ), + reason: + "Release of upstream sessions matching the requested session ID or private input was not confirmed; input_path preserves the last observed binding and is null when no inventory has reported it. A worker outlives its supervisor and may keep running until it finishes and reaches its idle timeout; exiting REA does not stop it. A gui session belongs to an IDA instance REA does not control. An empty session_id means upstream reported none; owned is null when no inventory has reported ownership; an empty list means the requested worker's close was confirmed but no inventory has identified any remaining session.", + }, + }), }, }); this.cleanupResources = [path]; @@ -121,7 +154,10 @@ export class IdaSessionClient implements AnalysisClient { #databaseCloseCompleted = false; #openAttempted = false; #openCompleted = false; + #blockingSessions: readonly UpstreamSession[] | undefined; + #openRejected = false; #callGraphAdvertised = false; + #surveyAdvertised = false; #initialIdentity: string | undefined; #metadata: JsonValue = null; #lifecycle: JsonValue[] = []; @@ -194,6 +230,17 @@ export class IdaSessionClient implements AnalysisClient { "This integration admits read-only analysis operations only.", ), ); + if ( + operation !== "health" && + !idaProfileAdmits(operation, this.config.mode) + ) + return err( + new AnalysisCapabilityUnavailableError( + "ida", + operation, + "This operation is available for the modern headless profile only.", + ), + ); if ( operation === "procedure_callers" && idaProtocol(this.config) !== "legacy" @@ -269,6 +316,9 @@ export class IdaSessionClient implements AnalysisClient { ...(runner.relationshipCoverage === undefined ? {} : { relationshipCoverage: runner.relationshipCoverage }), + ...(runner.referencesTruncated !== undefined + ? { referencesTruncated: runner.referencesTruncated } + : {}), locations: [ { kind: "artifact-path", path: this.target.path }, ...runner.locations, @@ -293,6 +343,8 @@ export class IdaSessionClient implements AnalysisClient { selectorOrigin: options?.selectorOrigin, x86Only: isX86OnlyTarget(this.target), callGraphAvailable: this.#callGraphAdvertised, + surveyAvailable: this.#surveyAdvertised, + admittedDocument: this.target.path, }, ); } @@ -423,6 +475,7 @@ export class IdaSessionClient implements AnalysisClient { async #initialize(): Promise { const tools = new Set(await this.connection.connect()); this.#callGraphAdvertised = tools.has("callgraph"); + this.#surveyAdvertised = tools.has("survey_binary"); const expected = this.config.protocol === "native" ? nativeTools @@ -450,6 +503,9 @@ export class IdaSessionClient implements AnalysisClient { preferred_session_id: this.#workspace.sessionId, }); this.#openCompleted = true; + // A completed but malformed/positive open may still own a live Worker. + // Only an explicit producer rejection keeps the old no-worker branch. + this.#openRejected = rejectedOpenDatabaseSchema.safeParse(raw).success; this.#lifecycle.push(raw); const opened = openDatabaseSchema.parse(raw); if ( @@ -466,6 +522,14 @@ export class IdaSessionClient implements AnalysisClient { await this.connection.call("idb_list", {}), ); this.#lifecycle.push(jsonValueSchema.parse(inventory)); + const workspace = this.#workspace; + this.#recordBlockingSessions( + inventory.sessions.filter( + (entry) => + entry.session_id === workspace.sessionId || + sameIdaHostPath(entry.input_path, workspace.inputPath), + ), + ); const entry = inventory.sessions.find( ({ session_id }) => session_id === opened.session.session_id, ); @@ -565,6 +629,52 @@ export class IdaSessionClient implements AnalysisClient { return shared; } + /** + * Upstream sessions whose release cleanup has not confirmed: the entries + * the latest successful inventory found blocking release; else the + * requested private worker until its close is confirmed, and then none + * known. + */ + #unreleasedSessions(): readonly UpstreamSession[] | undefined { + if (this.#blockingSessions !== undefined) return this.#blockingSessions; + if (!this.#openAttempted || this.#workspace === undefined) return undefined; + return this.#databaseCloseCompleted + ? [] + : [ + { + session_id: this.#workspace.sessionId, + backend: "worker", + owned: null, + input_path: null, + }, + ]; + } + + /** Drop a session whose release upstream confirmed, keeping other blockers. */ + #forgetReleasedSession(sessionId: string): void { + const remaining = this.#blockingSessions?.filter( + ({ session_id }) => session_id !== sessionId, + ); + this.#blockingSessions = + remaining !== undefined && remaining.length > 0 ? remaining : undefined; + } + + /** Replace blocker evidence from a successful inventory; true when blocked. */ + #recordBlockingSessions( + entries: readonly (UpstreamSession & { readonly owned: boolean })[], + ): boolean { + this.#blockingSessions = + entries.length > 0 + ? entries.map(({ session_id, backend, owned, input_path }) => ({ + session_id, + backend, + owned, + input_path, + })) + : undefined; + return entries.length > 0; + } + async #closeDatabase(database: string): Promise { const response = await this.connection.call("idb_close", { database, @@ -605,34 +715,47 @@ export class IdaSessionClient implements AnalysisClient { entry.session_id === workspace.sessionId && sameIdaHostPath(entry.input_path, workspace.inputPath), )?.session_id; - if ( - this.#database === undefined && - inventory.sessions.some((entry) => - sameIdaHostPath(entry.input_path, workspace.inputPath), - ) - ) + // Every same-input session and any entry under the requested ID, + // whatever its input, stays reported until a later successful + // inventory confirms release. + const blocked = this.#recordBlockingSessions( + inventory.sessions.filter( + (entry) => + entry.session_id === workspace.sessionId || + sameIdaHostPath(entry.input_path, workspace.inputPath), + ), + ); + if (this.#database === undefined && blocked) throw new AnalysisProtocolError( "An unowned worker still uses the private workspace; it was retained.", ); + if ( + this.#database === undefined && + this.#openCompleted && + !this.#openRejected + ) + throw new AnalysisProtocolError( + "IDA open returned without a verified private worker binding. An empty session inventory does not prove that its Worker exited; the private workspace was retained.", + ); } if (this.#database !== undefined) { if (!this.#databaseCloseCompleted) { await this.#closeDatabase(this.#database); this.#databaseCloseCompleted = true; + this.#forgetReleasedSession(this.#database); } const inventory = databaseListSchema.parse( await this.connection.call("idb_list", {}), ); - if ( - inventory.sessions.some( - (entry) => - entry.session_id === this.#database || - sameIdaHostPath( - entry.input_path, - this.#workspace?.inputPath ?? "", - ), - ) - ) + const remaining = inventory.sessions.filter( + (entry) => + entry.session_id === this.#database || + sameIdaHostPath( + entry.input_path, + this.#workspace?.inputPath ?? "", + ), + ); + if (this.#recordBlockingSessions(remaining)) throw new AnalysisProtocolError( "IDA worker remains registered after close.", ); @@ -649,10 +772,12 @@ export class IdaSessionClient implements AnalysisClient { this.#closed = true; return ok(null); } catch (cause: unknown) { + this.connection.detach(); return err( new IdaCleanupError( this.#workspace?.directory ?? "IDA MCP connection", cause, + this.#unreleasedSessions(), ), ); } diff --git a/src/ida/IdaWorkspace.ts b/src/ida/IdaWorkspace.ts index 0b3c36231..c7d9e642c 100644 --- a/src/ida/IdaWorkspace.ts +++ b/src/ida/IdaWorkspace.ts @@ -85,8 +85,10 @@ export class IdaWorkspace { slice === undefined ? position + bytes.length : Math.min(position + bytes.length, slice.offset + slice.size); - const admitted = bytes.subarray(start - position, end - position); - if (admitted.length > 0) { + // A chunk after the slice has end < position, and a negative subarray + // bound would count from the chunk's end instead of being empty. + if (end > start) { + const admitted = bytes.subarray(start - position, end - position); inputHash.update(admitted); await input.write(admitted); } diff --git a/src/ida/IdaWorkspaceOwnership.test.ts b/src/ida/IdaWorkspaceOwnership.test.ts index b39c6e916..1f4a10d65 100644 --- a/src/ida/IdaWorkspaceOwnership.test.ts +++ b/src/ida/IdaWorkspaceOwnership.test.ts @@ -98,11 +98,17 @@ describe("IDA workspace ownership before input preparation", () => { expect(closed.error.cleanupResources).toEqual([ expect.stringContaining(workspace ?? "missing workspace"), ]); + // No open was attempted, so there is no worker to report. + expect(closed.error).not.toHaveProperty( + "diagnostics.upstream_sessions", + ); } + expect(producer.detaches).toBe(1); } finally { await chmod(root, 0o700); } expect((await client.close()).ok).toBe(true); + expect(producer.detaches).toBe(1); expect(await readdir(root)).toEqual(["target.elf"]); }, ); diff --git a/src/ida/IdaWorkspaceThinning.test.ts b/src/ida/IdaWorkspaceThinning.test.ts index 7feca3ddc..baba6e145 100644 --- a/src/ida/IdaWorkspaceThinning.test.ts +++ b/src/ida/IdaWorkspaceThinning.test.ts @@ -5,7 +5,7 @@ import { afterEach, describe, expect, it } from "vitest"; import { parseBinaryTarget } from "../application/BinaryTargetResolver.js"; import { parseExecutableHeader } from "../domain/binaryTarget.js"; -import { fat } from "../domain/binaryTarget.fixture.js"; +import { fat, thinMach } from "../domain/binaryTarget.fixture.js"; import { createIdaTarget, RecordingIdaMcp, @@ -35,6 +35,46 @@ const universalTarget = async (hostArchitecture: "arm64" | "x64") => { return { bytes, target: resolved.value }; }; +/** One slice of a synthetic universal file: CPU type, offset and size. */ +interface Layout { + readonly cpu: number; + readonly offset: number; + readonly size: number; + readonly alignmentExponent?: number; +} + +/** + * Write a universal file whose slices span more than one 64 KiB read, each + * filled with its own byte so bytes outside the selected slice are visible. + */ +const largeUniversalTarget = async ( + layout: readonly Layout[], + hostArchitecture: "arm64" | "x64", +) => { + const directory = await createTestTempDirectory("rea-ida-thin-large-"); + roots.push(directory); + const path = join(directory, "universal"); + const total = Math.max(...layout.map(({ offset, size }) => offset + size)); + const bytes = Buffer.alloc(total); + bytes.writeUInt32BE(0xcafebabe, 0); + bytes.writeUInt32BE(layout.length, 4); + layout.forEach(({ cpu, offset, size, alignmentExponent = 14 }, index) => { + const entry = 8 + index * 20; + bytes.writeUInt32BE(cpu, entry); + bytes.writeUInt32BE(offset, entry + 8); + bytes.writeUInt32BE(size, entry + 12); + bytes.writeUInt32BE(alignmentExponent, entry + 16); + bytes.fill(0x11 * (index + 1), offset, offset + size); + thinMach(0xfeedfacf, cpu).copy(bytes, offset); + }); + await writeFile(path, bytes); + const resolved = await parseBinaryTarget(path, { hostArchitecture }); + if (!resolved.ok) throw resolved.error; + if (resolved.value.format !== "mach-o-universal") + throw new Error("the universal fixture did not resolve as a FAT container"); + return { bytes, target: resolved.value }; +}; + describe("IDA workspace input thinning", () => { it.skipIf(process.platform === "win32")( "thins the selected arm64 slice into the private headless input", @@ -78,6 +118,63 @@ describe("IDA workspace input thinning", () => { }, ); + it.skipIf(process.platform === "win32")( + "copies exactly the selected slice when later slices follow it across read boundaries", + async () => { + const { bytes, target } = await largeUniversalTarget( + [ + { + cpu: 0x01000007, + offset: 4096, + size: 8480, + alignmentExponent: 12, + }, + { cpu: 0x0100000c, offset: 16384, size: 33464 }, + { cpu: 0x00000007, offset: 65536, size: 33464 }, + ], + "arm64", + ); + const workspace = await IdaWorkspace.allocate(target); + roots.push(workspace.directory); + await workspace.copyInput(); + + const input = await readFile(workspace.inputPath); + const { offset, size } = target.universalSlice; + expect(size, "the arm64 slice is selected").toBe(33464); + expect(input.length, "no byte after the slice is copied").toBe(size); + expect(input.equals(bytes.subarray(offset, offset + size))).toBe(true); + expect(workspace.inputSha256).toBe( + createHash("sha256") + .update(bytes.subarray(offset, offset + size)) + .digest("hex"), + ); + await workspace.remove(); + }, + ); + + it.skipIf(process.platform === "win32")( + "copies exactly the first slice when it is selected and others follow", + async () => { + const { bytes, target } = await largeUniversalTarget( + [ + { cpu: 0x01000007, offset: 16384, size: 37456 }, + { cpu: 0x0100000c, offset: 65536, size: 33464 }, + ], + "x64", + ); + const workspace = await IdaWorkspace.allocate(target); + roots.push(workspace.directory); + await workspace.copyInput(); + + const input = await readFile(workspace.inputPath); + const { offset, size } = target.universalSlice; + expect(size, "the x86_64 slice is selected").toBe(37456); + expect(input.length, "no byte after the slice is copied").toBe(size); + expect(input.equals(bytes.subarray(offset, offset + size))).toBe(true); + await workspace.remove(); + }, + ); + it("keeps a whole-file copy for thin targets", async () => { const { root, target } = await createIdaTarget("arm64"); roots.push(root); diff --git a/src/javascript/analysis/JavaScriptAnalysisWorker.ts b/src/javascript/analysis/JavaScriptAnalysisWorker.ts index c02ddaa3e..6a587915b 100644 --- a/src/javascript/analysis/JavaScriptAnalysisWorker.ts +++ b/src/javascript/analysis/JavaScriptAnalysisWorker.ts @@ -27,10 +27,8 @@ import { ProviderAdapterError } from "../../domain/providerAdapterError.js"; import { ProviderCleanupError } from "../../domain/providerCleanupError.js"; import { ok, err, type Result } from "../../domain/result.js"; import { javaScriptRuntime } from "../../domain/runtimeVersion.js"; -import { - PrivateRuntimeRoot, - PrivateRuntimeRootUnavailableError, -} from "../../process/PrivateRuntimeRoot.js"; +import { PrivateRuntimeRoot } from "../../process/PrivateRuntimeRoot.js"; +import { PrivateRuntimeRootUnavailableError } from "../../process/PrivateRuntimeRootUnavailableError.js"; import { javaScriptWorkerMemoryLimitBytes } from "../../process/JavaScriptWorkerMemoryLimit.js"; import { ProviderProcessSupervisor, @@ -735,7 +733,7 @@ export class JavaScriptAnalysisWorker { PROVIDER, OPERATION, cause.message, - { cause }, + { cause, retryable: cause.retryable }, ) : new ProviderAdapterError(PROVIDER, OPERATION, { cause, diff --git a/src/javascript/analysis/JavaScriptAnalysisWorkerSchemas.ts b/src/javascript/analysis/JavaScriptAnalysisWorkerSchemas.ts index b666cfa9b..84a63d79b 100644 --- a/src/javascript/analysis/JavaScriptAnalysisWorkerSchemas.ts +++ b/src/javascript/analysis/JavaScriptAnalysisWorkerSchemas.ts @@ -475,7 +475,9 @@ const origin = z .strictObject({ specifier: z.string(), importedPath: strings, - resolutionMode: z.enum(["import", "require"]).optional(), + resolutionMode: z + .enum(["import", "require", "compiler-dependent"]) + .optional(), }) .transform(({ resolutionMode, ...rest }) => ({ ...rest, @@ -530,7 +532,9 @@ const coverage = z }); const moduleLink = z .strictObject({ - resolutionMode: z.enum(["import", "require"]).optional(), + resolutionMode: z + .enum(["import", "require", "compiler-dependent"]) + .optional(), kind: z.enum([ "import", "type-import", diff --git a/src/javascript/recovery/RecoveryFiles.ts b/src/javascript/recovery/RecoveryFiles.ts index 0503a83e1..c64f011b4 100644 --- a/src/javascript/recovery/RecoveryFiles.ts +++ b/src/javascript/recovery/RecoveryFiles.ts @@ -279,6 +279,6 @@ export const checkRecoveryDeadline = ( signal?: AbortSignal, ): void => { checkRecoveryCancellation(signal); - if (Date.now() >= deadline) + if (performance.now() >= deadline) throw new AnalysisTimeoutError(OPERATION, RECOVERY_LIMITS.timeoutMs); }; diff --git a/src/javascript/recovery/WakaruCommand.test.ts b/src/javascript/recovery/WakaruCommand.test.ts index 08e43e44a..8dcb74be4 100644 --- a/src/javascript/recovery/WakaruCommand.test.ts +++ b/src/javascript/recovery/WakaruCommand.test.ts @@ -33,7 +33,7 @@ it("rejects an exhausted shared execution budget before launching the next comma arguments: ["--version"], cwd: "/tmp/unused", environment: {}, - deadline: Date.now() - 1, + deadline: performance.now() - 1, launcher: async () => { throw new Error("An expired budget must not launch"); }, @@ -52,7 +52,7 @@ it("preserves caller cancellation before acquiring an expired execution budget", arguments: ["--version"], cwd: "/tmp/unused", environment: {}, - deadline: Date.now() - 1, + deadline: performance.now() - 1, signal: AbortSignal.abort(), launcher: async () => { throw new Error("A cancelled request must not launch"); @@ -86,7 +86,7 @@ const commandFixture = ( arguments: ["--version"], cwd: "/tmp/unused", environment: {}, - deadline: Date.now() + 5_000, + deadline: performance.now() + 5_000, launcher, }, }; @@ -140,7 +140,7 @@ it("enforces the shared deadline while exited output streams remain open", async }); try { await expect( - runWakaruCommand({ ...context, deadline: Date.now() + 50 }), + runWakaruCommand({ ...context, deadline: performance.now() + 50 }), ).rejects.toMatchObject({ _tag: "AnalysisTimeoutError", operation: "recover_javascript_sources", diff --git a/src/javascript/recovery/WakaruProvider.ts b/src/javascript/recovery/WakaruProvider.ts index 7a51c8bd5..e25812663 100644 --- a/src/javascript/recovery/WakaruProvider.ts +++ b/src/javascript/recovery/WakaruProvider.ts @@ -9,7 +9,11 @@ import type { import { SafeOutputTree } from "../../artifacts/SafeOutputTree.js"; import { SafeOutputTreeCreationFailure } from "../../artifacts/SafeOutputTreeCreationFailure.js"; import { AnalysisError } from "../../domain/analysisErrorBase.js"; -import { AnalysisOutputError } from "../../domain/analysisErrorCore.js"; +import { + AnalysisCancelledError, + AnalysisOutputError, +} from "../../domain/analysisErrorCore.js"; +import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js"; import { javascriptRecoveryResultSchema, type JavaScriptRecoveryInput, @@ -100,7 +104,7 @@ export class WakaruProvider implements JavaScriptRecoveryPort { let tree: SafeOutputTree | undefined; let failed: { readonly cause: unknown } | undefined; let execution: AnalysisExecution | undefined; - const deadline = Date.now() + RECOVERY_LIMITS.timeoutMs; + const deadline = performance.now() + RECOVERY_LIMITS.timeoutMs; try { checkRecoveryCancellation(options?.signal); const engine = await resolveWakaruCommand(this.environment); @@ -139,12 +143,16 @@ export class WakaruProvider implements JavaScriptRecoveryPort { await rm(root, { recursive: true, force: true }); root = undefined; checkRecoveryDeadline(deadline, options?.signal); - await tree.commit(); + await tree.commit(options?.signal); } catch (cause: unknown) { failed = { cause }; } if (failed !== undefined) { - const failure = failed.cause; + const failure = + failed.cause instanceof ArtifactReaderFailure && + failed.cause.reason === "cancelled" + ? new AnalysisCancelledError(OPERATION, { cause: failed.cause }) + : failed.cause; const residuals: string[] = []; const cleanupFailures: { resource: string; reason: string }[] = []; const cleanupOwner = diff --git a/src/javascript/sourceMaps/DecodedSourceMap.ts b/src/javascript/sourceMaps/DecodedSourceMap.ts index 60bfcfc0e..e9ed304e1 100644 --- a/src/javascript/sourceMaps/DecodedSourceMap.ts +++ b/src/javascript/sourceMaps/DecodedSourceMap.ts @@ -19,31 +19,6 @@ export const isBeforeSourceMapLeafStop = ( ); }; -/** Resolve one declaration with the same pinned upstream implementation used by full maps. */ -export const resolveSourceMapSource = ( - source: string | null, - sourceRoot: string | null | undefined, - mapUrl: string, -): string => { - const map = new TraceMap( - { - version: 3, - names: [], - sources: [source ?? ""], - ...(sourceRoot == null ? {} : { sourceRoot }), - mappings: "", - }, - mapUrl, - ); - const resolved = map.resolvedSources[0]; - if (resolved === undefined) - throw new SourceMapFormatFailure( - "format", - "Source-map resolver did not return the requested source identity.", - ); - return resolved; -}; - /** Decode leaf mappings and validate producer indices before any consumer publishes them. */ export const decodeValidatedSourceMapLeaves = ( leaves: readonly SourceMapLeaf[], diff --git a/src/javascript/sourceMaps/SourceMapCommand.ts b/src/javascript/sourceMaps/SourceMapCommand.ts index e86954106..9e8837952 100644 --- a/src/javascript/sourceMaps/SourceMapCommand.ts +++ b/src/javascript/sourceMaps/SourceMapCommand.ts @@ -16,10 +16,8 @@ import { ProviderCleanupError } from "../../domain/providerCleanupError.js"; import type { JsonValue } from "../../domain/jsonValue.js"; import { err, ok, type Result } from "../../domain/result.js"; import { WEB_SOURCE_MAP_LIMITS } from "../../domain/webSourceLocation.js"; -import { - PrivateRuntimeRoot, - PrivateRuntimeRootUnavailableError, -} from "../../process/PrivateRuntimeRoot.js"; +import { PrivateRuntimeRoot } from "../../process/PrivateRuntimeRoot.js"; +import { PrivateRuntimeRootUnavailableError } from "../../process/PrivateRuntimeRootUnavailableError.js"; import { ProviderStartupDeadline } from "../../process/ProviderDeadline.js"; import { javaScriptWorkerMemoryLimitBytes } from "../../process/JavaScriptWorkerMemoryLimit.js"; import { cleanupOwnedProcessGroup } from "../../process/ProcessOwnership.js"; @@ -261,7 +259,7 @@ class OwnedCodec implements SourceMapCleanupOwner { PROVIDER, OPERATION, cause.message, - { cause }, + { cause, retryable: cause.retryable }, ) : new ProviderAdapterError(PROVIDER, OPERATION, { cause, diff --git a/src/mcpDoctor.ts b/src/mcpDoctor.ts index 62a588280..d7bf1e366 100644 --- a/src/mcpDoctor.ts +++ b/src/mcpDoctor.ts @@ -1,6 +1,7 @@ import { Client } from "@modelcontextprotocol/client"; import { StdioClientTransport } from "@modelcontextprotocol/client/stdio"; import { Formatter, Help, z } from "incur"; +import { performance } from "node:perf_hooks"; import { resolve } from "node:path"; import { CATALOG_IDENTITY } from "./catalogIdentity.js"; @@ -74,14 +75,15 @@ interface McpDoctorCheck { /** Diagnose the actual production stdio adapter with one startup-bounded SDK session. */ export const runProductionMcpDoctor = async (options: McpDoctorOptions) => { const deadline = - Date.now() + (options.deadlineMs ?? MCP_STARTUP_POLICY.doctorDeadlineMs); + performance.now() + + (options.deadlineMs ?? MCP_STARTUP_POLICY.doctorDeadlineMs); const controller = new AbortController(); const timer = setTimeout( () => controller.abort(new Error("Production MCP doctor deadline expired")), - Math.max(1, deadline - Date.now()), + Math.max(1, Math.ceil(deadline - performance.now())), ); const stderr = captureStderr(); - const transport = new TrackedStdioClientTransport({ + const transport = new OwnedStdioClientTransport({ command: options.command, args: [...options.args], cwd: options.cwd, @@ -90,21 +92,20 @@ export const runProductionMcpDoctor = async (options: McpDoctorOptions) => { }); transport.stderr?.on("data", stderr.append); const client = new Client({ name: "rea-mcp-doctor", version: "1" }); + let result: ProductionMcpDoctorResult; + let cleanupFailure: { readonly cause: unknown } | undefined; try { await client.connect( transport, requestOptions(deadline, controller.signal), ); - return { - ...(await inspectProductionMcpSession( - client, - deadline, - controller.signal, - )), - diagnostics: stderr.value(), - }; + result = await inspectProductionMcpSession( + client, + deadline, + controller.signal, + ); } catch (cause: unknown) { - return { + result = { healthy: false, adapter: "production-stdio" as const, server: null, @@ -118,14 +119,71 @@ export const runProductionMcpDoctor = async (options: McpDoctorOptions) => { detail: errorMessage(cause), }, ], - diagnostics: stderr.value(), }; } finally { clearTimeout(timer); - await closeMcpChild(client, transport); + try { + await client.close(); + } catch (cause: unknown) { + cleanupFailure = { cause }; + } + try { + // An interrupted connect can clear the client's transport reference + // before its startup child is closed. The transport retains that child. + await transport.close(); + } catch (cause: unknown) { + cleanupFailure = { + cause: + cleanupFailure === undefined || cleanupFailure.cause === cause + ? cause + : new AggregateError( + [cleanupFailure.cause, cause], + `MCP client and transport cleanup failed: ${errorMessage(cleanupFailure.cause)}; ${errorMessage(cause)}`, + ), + }; + } } + if (cleanupFailure !== undefined) + result = { + ...result, + healthy: false, + checks: [ + ...result.checks, + { + name: "cleanup", + ok: false, + detail: `MCP stdio cleanup failed: ${errorMessage(cleanupFailure.cause)}`, + }, + ], + }; + return { ...result, diagnostics: stderr.value() }; }; +type ProductionMcpDoctorResult = + | Awaited> + | { + readonly healthy: false; + readonly adapter: "production-stdio"; + readonly server: null; + readonly inventory: null; + readonly identity: null; + readonly request_flow: { + readonly tool: "binary_session"; + readonly ok: false; + }; + readonly checks: readonly McpDoctorCheck[]; + }; + +/** Join SDK-triggered and caller-triggered close calls for the same child. */ +class OwnedStdioClientTransport extends StdioClientTransport { + #closePromise: Promise | undefined; + + override close(): Promise { + this.#closePromise ??= super.close(); + return this.#closePromise; + } +} + const inspectProductionMcpSession = async ( client: Client, deadline: number, @@ -385,7 +443,7 @@ const isOutputFormat = (value: string | undefined): value is OutputFormat => const requestOptions = (deadline: number, signal: AbortSignal) => ({ signal, - timeout: Math.max(1, deadline - Date.now()), + timeout: Math.max(1, Math.ceil(deadline - performance.now())), }); const targetFreeEnvironment = ( @@ -428,61 +486,3 @@ const isRecord = (value: unknown): value is Readonly> => const errorMessage = (cause: unknown): string => cause instanceof Error ? cause.message : String(cause); - -class TrackedStdioClientTransport extends StdioClientTransport { - spawnedPid: number | null = null; - - override async start(): Promise { - await super.start(); - this.spawnedPid = this.pid; - } -} - -const closeMcpChild = async ( - client: Client, - transport: TrackedStdioClientTransport, -): Promise => { - const pid = transport.spawnedPid ?? transport.pid; - await Promise.allSettled([client.close()]); - await Promise.allSettled([transport.close()]); - if (pid === null || !(await processExists(pid))) return; - await terminateProcess(pid, "SIGTERM"); - if (await waitForProcessExit(pid, 1_000)) return; - await terminateProcess(pid, "SIGKILL"); - await waitForProcessExit(pid, 1_000); -}; - -const terminateProcess = async ( - pid: number, - signal: NodeJS.Signals, -): Promise => { - try { - process.kill(pid, signal); - } catch (cause: unknown) { - if (!(cause instanceof Error && "code" in cause && cause.code === "ESRCH")) - throw cause; - } -}; - -const waitForProcessExit = async ( - pid: number, - timeoutMs: number, -): Promise => { - const deadline = Date.now() + timeoutMs; - do { - if (!(await processExists(pid))) return true; - await new Promise((resolve) => setTimeout(resolve, 20)); - } while (Date.now() < deadline); - return !(await processExists(pid)); -}; - -const processExists = async (pid: number): Promise => { - try { - process.kill(pid, 0); - return true; - } catch (cause: unknown) { - if (cause instanceof Error && "code" in cause && cause.code === "ESRCH") - return false; - throw cause; - } -}; diff --git a/src/native/AppleDispatchCoverage.ts b/src/native/AppleDispatchCoverage.ts index c49e35acc..a6c68ca51 100644 --- a/src/native/AppleDispatchCoverage.ts +++ b/src/native/AppleDispatchCoverage.ts @@ -42,14 +42,62 @@ export const pushDispatchCoverage = (input: { vtables, swift, } = input; + const classOwnedMethods = result.objc_dispatch_implementations.filter( + ({ category }) => category === undefined, + ); + const classRecords = [ + ...result.objc_classes.flatMap((record) => + record.decode?.status === "decoded" + ? [] + : [ + `class ${record.name}: ${record.decode?.reason ?? record.decode?.status ?? "decode_status_missing"}`, + ], + ), + ...result.objc_protocol_records.flatMap((record) => + record.decode.status === "decoded" + ? [] + : [ + `protocol ${record.name}: ${record.decode.reason ?? record.decode.status}`, + ], + ), + ...classOwnedMethods.flatMap((record) => + record.decode.status === "decoded" + ? [] + : [ + `method ${record.class_name} ${record.method_type} ${record.selector}: ${record.decode.reason ?? record.decode.status}`, + ], + ), + ...result.objc_ivars.flatMap((record) => + record.decode.status === "decoded" + ? [] + : [ + `ivar ${record.class_name}.${record.name}: ${record.decode.reason ?? record.decode.status}`, + ], + ), + ]; + const classReasons = [ + ...failures, + ...classRecords, + ...(truncated ? ["max_records_reached"] : []), + ]; + const categoryMethods = result.objc_dispatch_implementations.filter( + ({ category }) => category !== undefined, + ); + const categoryMethodReasons = categoryMethods.flatMap((record) => + record.decode.status === "decoded" + ? [] + : [ + `category method ${record.category} ${record.class_name} ${record.method_type} ${record.selector}: ${record.decode.reason ?? record.decode.status}`, + ], + ); result.coverage.push({ facet: "objc_class_method_ivar_metadata", - status: failures.length > 0 || truncated ? "partial" : "complete", - reason: - [...failures, ...(truncated ? ["max_records_reached"] : [])].join("; ") || - null, + status: classReasons.length > 0 ? "partial" : "complete", + reason: classReasons.join("; ") || null, examined, - decoded: result.objc_classes.length, + decoded: result.objc_classes.filter( + ({ decode }) => decode?.status === "decoded", + ).length, }); result.coverage.push({ facet: "binary_relative_pointers", @@ -79,13 +127,18 @@ export const pushDispatchCoverage = (input: { status: truncated || categoryFailures.length > 0 || - result.objc_categories.some(({ decode }) => decode.status !== "decoded") + result.objc_categories.some( + ({ decode }) => decode.status !== "decoded", + ) || + categoryMethodReasons.length > 0 ? "partial" : "complete", reason: - [...categoryFailures, ...(truncated ? ["max_records_reached"] : [])].join( - "; ", - ) || null, + [ + ...categoryFailures, + ...categoryMethodReasons, + ...(truncated ? ["max_records_reached"] : []), + ].join("; ") || null, examined: categoriesExamined, decoded: result.objc_categories.filter( ({ decode }) => decode.status === "decoded", @@ -110,7 +163,7 @@ export const pushDispatchCoverage = (input: { : "complete", reason: fixupReason.length > 0 ? fixupReason : null, examined: fixups.examined, - decoded: fixups.issues.length > 0 ? 0 : fixups.examined, + decoded: fixups.decoded, }); result.coverage.push({ facet: "swift_generic_resilient_witnesses_overrides_async_coroutines", diff --git a/src/native/AppleDispatchMetadata.test.ts b/src/native/AppleDispatchMetadata.test.ts index 6791ba1b0..244a35aeb 100644 --- a/src/native/AppleDispatchMetadata.test.ts +++ b/src/native/AppleDispatchMetadata.test.ts @@ -2,7 +2,10 @@ import { describe, expect, it } from "vitest"; import { decodeAppleDispatchMetadata } from "./AppleDispatchMetadata.js"; const base = 0x100000000n; -const fixture = (relative = false) => { +const fixture = ( + relative = false, + category?: "partial-method" | "unmapped-pointer", +) => { const bytes = Buffer.alloc(4096); const u32 = (offset: number, value: number) => bytes.writeUInt32LE(value, offset); @@ -11,19 +14,26 @@ const fixture = (relative = false) => { u32(0, 0xfeedfacf); u32(4, 0x0100000c); u32(16, 1); - u32(20, 152); + u32(20, category === undefined ? 152 : 232); u32(32, 0x19); - u32(36, 152); + u32(36, category === undefined ? 152 : 232); bytes.write("__DATA", 40); ptr(56, 0); bytes.writeBigUInt64LE(4096n, 64); bytes.writeBigUInt64LE(4096n, 80); u32(92, 5); - u32(96, 1); + u32(96, category === undefined ? 1 : 2); bytes.write("__objc_classlist", 104); bytes.write("__DATA", 120); ptr(136, 0x180); bytes.writeBigUInt64LE(8n, 144); + if (category !== undefined) { + bytes.write("__objc_catlist", 184); + bytes.write("__DATA", 200); + ptr(216, 0xb00); + bytes.writeBigUInt64LE(8n, 224); + u32(232, 0xb00); + } u32(152, 0x180); ptr(0x180, 0x200); ptr(0x200, 0x280); @@ -86,10 +96,58 @@ const fixture = (relative = false) => { bytes.write("FixtureProtocol\0", 0x900); bytes.write("requiredAction:\0", 0x920); bytes.write("optionalAction:\0", 0x940); + if (category !== undefined) { + if (category === "unmapped-pointer") { + bytes.writeBigUInt64LE(0xffffffffffffffffn, 0xb00); + } else { + ptr(0xb00, 0xb40); + ptr(0xb40, 0xbc0); + ptr(0xb48, 0x200); + ptr(0xb50, 0xb80); + u32(0xb80, 24); + u32(0xb84, 1); + ptr(0xb88, 0x460); + ptr(0xb90, 0x5c0); + bytes.writeBigUInt64LE(0xffffffffffffffffn, 0xb98); + bytes.write("FixtureCategory\0", 0xbc0); + } + } return bytes; }; const provenance = { path: "/fixture/app", sha256: "a".repeat(64) }; -describe("chained Apple dispatch binary metadata", () => { +describe("Apple dispatch pointer fixup metadata", () => { + it("keeps decoded bindings and their coverage when an independent stream fails", () => { + const bytes = fixture(); + bytes.writeUInt32LE(2, 16); + bytes.writeUInt32LE(200, 20); + bytes.writeUInt32LE(0x80000022, 184); + bytes.writeUInt32LE(48, 188); + const bindings = Buffer.from([ + 0x40, + ...Buffer.from("_OBJC_CLASS_$_NSObject\0"), + 0x70, + 0x88, + 0x04, + 0x90, + 0x00, + ]); + bindings.copy(bytes, 0xc00); + bytes[0xc80] = 0xd0; + bytes.writeUInt32LE(0xc00, 200); + bytes.writeUInt32LE(bindings.length, 204); + bytes.writeUInt32LE(0xc80, 208); + bytes.writeUInt32LE(1, 212); + + const result = decodeAppleDispatchMetadata(bytes, 100, provenance); + expect(result.objc_classes[0]).toMatchObject({ + name: "Fixture", + super_class: "NSObject", + }); + expect( + result.coverage.find(({ facet }) => facet === "pointer_fixups"), + ).toMatchObject({ status: "partial", examined: 2, decoded: 1 }); + }); + it("keeps absolute metadata pointers between declared chained fixups", () => { const bytes = fixture(); bytes.writeUInt32LE(2, 16); @@ -120,7 +178,7 @@ describe("chained Apple dispatch binary metadata", () => { }); expect( result.coverage.find(({ facet }) => facet === "pointer_fixups"), - ).toMatchObject({ status: "complete" }); + ).toMatchObject({ status: "complete", examined: 1, decoded: 1 }); }); }); describe("Apple dispatch binary metadata", () => { @@ -221,6 +279,9 @@ describe("Apple dispatch binary metadata", () => { }); }, ); +}); + +describe("Apple dispatch coverage ownership", () => { it("keeps unsupported pointers and record truncation explicit", () => { const bytes = fixture(); bytes.writeBigUInt64LE(0xffffffffffffffffn, 0x498); @@ -229,6 +290,14 @@ describe("Apple dispatch binary metadata", () => { implementation_address: null, decode: { status: "partial" }, }); + expect( + result.coverage.find( + ({ facet }) => facet === "objc_class_method_ivar_metadata", + ), + ).toMatchObject({ + status: "partial", + reason: expect.stringContaining("method Fixture instance performAction:"), + }); const truncated = decodeAppleDispatchMetadata(fixture(), 1, provenance); expect(truncated.coverage[0]?.reason).toContain("max_records_reached"); expect( @@ -237,6 +306,67 @@ describe("Apple dispatch binary metadata", () => { ), ).toMatchObject({ status: "partial", reason: "max_records_reached" }); }); + it("keeps partial category methods in category coverage only", () => { + const baseline = decodeAppleDispatchMetadata(fixture(), 100, provenance); + const result = decodeAppleDispatchMetadata( + fixture(false, "partial-method"), + 100, + provenance, + ); + + expect(result.objc_dispatch_implementations).toContainEqual( + expect.objectContaining({ + category: "FixtureCategory", + decode: { + status: "partial", + reason: "implementation_pointer_unmapped_or_encoded", + }, + }), + ); + const baselineClassCoverage = baseline.coverage.find( + ({ facet }) => facet === "objc_class_method_ivar_metadata", + ); + const classCoverage = result.coverage.find( + ({ facet }) => facet === "objc_class_method_ivar_metadata", + ); + expect(baselineClassCoverage).toBeDefined(); + expect(classCoverage).toEqual(baselineClassCoverage); + expect( + result.coverage.find( + ({ facet }) => facet === "objc_properties_categories", + ), + ).toMatchObject({ + status: "partial", + reason: expect.stringContaining("category method FixtureCategory"), + }); + }); + it("keeps malformed category pointers out of class failure coverage", () => { + const baseline = decodeAppleDispatchMetadata(fixture(), 100, provenance); + const result = decodeAppleDispatchMetadata( + fixture(false, "unmapped-pointer"), + 100, + provenance, + ); + const baselineClassCoverage = baseline.coverage.find( + ({ facet }) => facet === "objc_class_method_ivar_metadata", + ); + const classCoverage = result.coverage.find( + ({ facet }) => facet === "objc_class_method_ivar_metadata", + ); + expect(baselineClassCoverage).toBeDefined(); + expect(classCoverage).toEqual(baselineClassCoverage); + expect( + result.coverage.find( + ({ facet }) => facet === "objc_properties_categories", + ), + ).toMatchObject({ + status: "partial", + reason: expect.stringContaining("category_decode_failed"), + }); + }); +}); + +describe("malformed Apple dispatch metadata", () => { it("rejects malformed command and section boundaries", () => { const bytes = fixture(); bytes.writeUInt32LE(7, 36); diff --git a/src/native/AppleMachoFixups.test.ts b/src/native/AppleMachoFixups.test.ts index 40c50c59b..bf4ae348f 100644 --- a/src/native/AppleMachoFixups.test.ts +++ b/src/native/AppleMachoFixups.test.ts @@ -149,6 +149,64 @@ const SEGMENTS: Segment[] = [ { address: BASE + 0x4000n, size: 0x4000n, offset: 0x4000, executable: false }, ]; +describe("independent chained segment coverage", () => { + it("counts successfully decoded chained segments alongside an unsupported segment", () => { + const starts = 28; + const startsSize = 4 + 2 * 4; + const firstInfo = starts + startsSize; + const secondInfo = firstInfo + 24; + const imports = secondInfo + 24; + const data = Buffer.alloc(0x3000); + data.writeUInt32LE(starts, 4); + data.writeUInt32LE(imports, 8); + data.writeUInt32LE(imports, 12); + data.writeUInt32LE(1, 20); + data.writeUInt32LE(2, starts); + data.writeUInt32LE(firstInfo - starts, starts + 4); + data.writeUInt32LE(secondInfo - starts, starts + 8); + for (const [info, format, segmentOffset] of [ + [firstInfo, 6, 0], + [secondInfo, 99, 0x1000], + ] as const) { + data.writeUInt32LE(24, info); + data.writeUInt16LE(0x1000, info + 4); + data.writeUInt16LE(format, info + 6); + data.writeBigUInt64LE(BigInt(segmentOffset), info + 8); + data.writeUInt16LE(1, info + 20); + data.writeUInt16LE(8, info + 22); + } + const segments: Segment[] = [ + { address: BASE, size: 0x1000n, offset: 0x1000, executable: false }, + { + address: BASE + 0x1000n, + size: 0x1000n, + offset: 0x2000, + executable: false, + }, + ]; + const fixups = parsePointerFixups( + data, + { + chained: { offset: 0, size: imports }, + binds: [], + rebases: [], + dylibs: [], + }, + segments, + BASE, + ); + + expect(fixups).toMatchObject({ examined: 2, decoded: 1 }); + expect(fixups.issues).toMatchObject([ + { code: "unsupported_pointer_format" }, + ]); + expect(fixups.decode(BASE + 8n, 0n)).toEqual({ + kind: "rebase", + target: BASE, + }); + }); +}); + describe("pointer fixup tables", () => { it("reports unsupported pointer formats even when no pointer is visited", () => { const data = chainedData(1); @@ -196,20 +254,27 @@ describe("pointer fixup tables", () => { { chained: { offset: 0, size: 128 }, binds: [], - rebases: [], + rebases: [{ offset: 8, size: 12 }], dylibs: [], }, SEGMENTS, BASE, ); + expect(fixups).toMatchObject({ + examined: 1, + decoded: 1, + unreadRebaseBytes: 12, + }); expect(fixups.decode(BASE + 0x4008n, field(1n, 63))).toMatchObject({ kind: "bind", symbol: "_OBJC_CLASS_$_NSObject", }); expect(fixups.issues).toEqual([]); + expect(fixups.decoded).toBe(1); expect(fixups.decode(BASE + 0x4008n, 1n | field(1n, 63))).toMatchObject({ kind: "unsupported", }); + expect(fixups.decoded).toBe(0); expect(fixups.issues).toMatchObject([ { code: "chained_pointer_decode_failed" }, ]); diff --git a/src/native/AppleMachoFixups.ts b/src/native/AppleMachoFixups.ts index 2ec711cc8..09080ec1d 100644 --- a/src/native/AppleMachoFixups.ts +++ b/src/native/AppleMachoFixups.ts @@ -42,6 +42,8 @@ export interface PointerFixups { readonly issues: readonly DecodeIssue[]; /** Fixup segments or bind streams this reader walked. */ readonly examined: number; + /** Fully decoded fixup segments or bind streams, in the same units as `examined`. */ + readonly decoded: number; /** Rebase opcode bytes present in the load command and not decoded. */ readonly unreadRebaseBytes: number; /** Decode the pointer stored at `address` whose raw 64-bit value is `raw`. */ @@ -260,6 +262,7 @@ const readChainedPage = ( const chainedFixups = ( image: FixupImage, range: { readonly offset: number; readonly size: number }, + unreadRebaseBytes: number, ): PointerFixups => { const { bytes, segments, dylibs, baseAddress } = image; const base = range.offset; @@ -400,19 +403,25 @@ const chainedFixups = ( ] : [], ); - const examinedViews = new Set(); - for (const chain of ranges) { + const examinedViews = new Map(); + const decodedRanges = new Set(); + for (const [chainIndex, chain] of ranges.entries()) { // Unknown formats retain their compact starts table; only a queried page // needs its NONE/presence value. Do not expand unsupported declarations. if (POINTER_FORMATS[chain.format] === undefined) continue; - const view = `${chain.pageStarts}:${chain.segment.offset}:${chain.segment.size}:${chain.segment.address}`; - if (examinedViews.has(view)) continue; - examinedViews.add(view); + const view = `${chain.pageStarts}:${chain.segment.offset}:${chain.segment.size}:${chain.segment.address}:${chain.format}:${chain.pageSize}:${chain.pageCount}`; + const prior = examinedViews.get(view); + if (prior !== undefined) { + if (prior) decodedRanges.add(chainIndex); + continue; + } + let complete = true; for (let page = 0; page < chain.pageCount; page++) { const start = bytes.readUInt16LE(chain.pageStarts + page * 2); if (start === 0xffff) continue; const facts = pageFacts(chain, page, start); if (facts.failure !== undefined) { + complete = false; issues.push( issue( "pointer_fixups", @@ -427,6 +436,8 @@ const chainedFixups = ( break; } } + examinedViews.set(view, complete); + if (complete) decodedRanges.add(chainIndex); } const failures = issues.map(({ message }) => message); return { @@ -435,13 +446,17 @@ const chainedFixups = ( failures, issues, examined: Math.max(ranges.length, 1), - unreadRebaseBytes: 0, + get decoded() { + return ranges.length === 0 ? 1 : decodedRanges.size; + }, + unreadRebaseBytes, decode(address, raw) { - const chain = ranges.find( + const chainIndex = ranges.findIndex( ({ segment }) => address >= segment.address && address < segment.address + segment.size, ); + const chain = ranges[chainIndex]; // Pointers outside fixup segments, such as in __TEXT, are stored plainly. if (chain === undefined) return { kind: "rebase", target: raw }; const within = Number(address - chain.segment.address); @@ -486,6 +501,7 @@ const chainedFixups = ( ); failures.push(reason); } + decodedRanges.delete(chainIndex); return { kind: "unsupported", reason }; } }, @@ -684,10 +700,7 @@ export const parsePointerFixups = ( 0, ); if (commands.chained !== null) { - const chained = chainedFixups(image, commands.chained); - return unreadRebaseBytes === 0 - ? chained - : { ...chained, unreadRebaseBytes }; + return chainedFixups(image, commands.chained, unreadRebaseBytes); } if (commands.binds.length === 0) return { @@ -696,14 +709,17 @@ export const parsePointerFixups = ( failures: [], issues: [], examined: 0, + decoded: 0, unreadRebaseBytes, decode: (_address, raw) => ({ kind: "rebase", target: raw }), }; const binds: BindRun[] = []; const failures: string[] = []; + let decoded = 0; for (const stream of commands.binds) try { readBindStream(image, stream, binds); + decoded++; } catch (cause: unknown) { failures.push( `${stream.stream} bind opcodes: ${cause instanceof Error ? cause.message : String(cause)}`, @@ -717,6 +733,7 @@ export const parsePointerFixups = ( issue("pointer_fixups", "bind_stream_decode_failed", message), ), examined: commands.binds.length, + decoded, unreadRebaseBytes, decode(address, raw) { let match: BindRun | undefined; diff --git a/src/native/AppleMachoSelection.test.ts b/src/native/AppleMachoSelection.test.ts index 5c19026e0..af2d0be55 100644 --- a/src/native/AppleMachoSelection.test.ts +++ b/src/native/AppleMachoSelection.test.ts @@ -7,9 +7,13 @@ import { fatImage, machoImage, } from "../artifacts/apple/MachoImage.fixture.js"; -import { parseMachoLayout } from "./AppleMachoSelection.js"; +import { parseMachoLayout, selectMachoSlice } from "./AppleMachoSelection.js"; import { decodeAppleDispatchMetadata } from "./AppleDispatchMetadata.js"; +/** macOS 27 system binaries ship CPU_SUBTYPE_ARM64E_X1 (12) with the PAC ABI bit. */ +const ARM64E_X1 = { type: 0x0100000c, subtype: 0x8000000c } as const; +const X86_64H = { type: 0x01000007, subtype: 8 } as const; + describe("native Mach-O slice selection", () => { it("preserves high-bit section-name bytes instead of ASCII-folding metadata names", () => { const segment = Buffer.alloc(152); @@ -68,6 +72,48 @@ describe("native Mach-O slice selection", () => { /not aligned/u, ); }); + + it.each([ + { layout: "arm64e, arm64e.x1", cpus: [CPU.arm64e, ARM64E_X1], index: 0 }, + { layout: "arm64e.x1, arm64e", cpus: [ARM64E_X1, CPU.arm64e], index: 1 }, + { layout: "arm64, arm64e.x1", cpus: [CPU.arm64, ARM64E_X1], index: 0 }, + { layout: "arm64e.x1, arm64", cpus: [ARM64E_X1, CPU.arm64], index: 1 }, + { + layout: "x86_64, arm64e, arm64e.x1", + cpus: [CPU.x86_64, CPU.arm64e, ARM64E_X1], + index: 1, + }, + { + layout: "x86_64h, x86_64", + cpus: [X86_64H, CPU.x86_64], + index: 1, + architecture: "x86_64", + }, + ])( + "selects one slice when FAT slices $layout share a CPU type", + ({ cpus, index, architecture = "arm64" }) => { + const fat = Buffer.from( + fatImage(cpus.map((cpu) => ({ cpu, bytes: machoImage({ cpu }) }))), + ); + expect(selectMachoSlice(fat, architecture).fatDeclaration).toMatchObject({ + index, + cpuSubtype: cpus[index]?.subtype, + }); + expect(parseMachoLayout(fat, architecture).segments).toEqual([]); + }, + ); + + it("rejects slices of the requested CPU type that share a subtype", () => { + const fat = fatImage( + [CPU.arm64e, CPU.arm64e].map((cpu) => ({ + cpu, + bytes: machoImage({ cpu }), + })), + ); + expect(() => parseMachoLayout(Buffer.from(fat), "arm64")).toThrow( + /Ambiguous FAT architecture slice/u, + ); + }); }); describe("native Mach-O fixup load command boundaries", () => { diff --git a/src/native/AppleMachoSelection.ts b/src/native/AppleMachoSelection.ts index 9ee93f995..c0c68068d 100644 --- a/src/native/AppleMachoSelection.ts +++ b/src/native/AppleMachoSelection.ts @@ -1,7 +1,13 @@ import type { FixupCommands } from "./AppleMachoFixups.js"; +import { + MACHO_ARCHITECTURE_POLICIES, + machoSubtypeRank, +} from "./MachoArchitectureSelection.js"; +import { machoCpuSubtypeBase } from "../domain/apple/machoCpuIdentity.js"; import { readFatSliceDeclarations, validateFatSlice, + type FatSliceDeclaration, } from "../domain/apple/machoContainer.js"; /** One mapped Mach-O segment with file-backed VA range. */ @@ -90,32 +96,44 @@ export const selectMachoSlice = ( const headerEnd = 8 + count * stride; if (count > 128 || headerEnd > bytes.length) throw malformed("Malformed FAT architecture table"); - const cpu = MACHO_CPU_TYPE[architecture]; - if (cpu === undefined) throw invalid("Requested FAT architecture is absent"); + const policy = MACHO_ARCHITECTURE_POLICIES[architecture]; + if (policy === undefined) + throw invalid("Requested FAT architecture is absent"); const declarations = readFatSliceDeclarations(bytes.subarray(8, headerEnd), { count, wide: fat64, littleEndian, }); if (declarations.status !== "parsed") throw malformed(declarations.reason); - let selected: MachoSlice | undefined; + let best: FatSliceDeclaration | undefined; + let bestRank = Number.POSITIVE_INFINITY; + let ambiguous = false; for (const declaration of declarations.slices) { const issue = validateFatSlice(declaration, bytes.length, headerEnd); if (issue !== null) throw malformed(issue); - if (declaration.cpuType !== cpu) continue; - if (selected !== undefined) - throw invalid("Ambiguous FAT architecture slice"); - selected = { - slice: declaration.offset, - sliceEnd: declaration.offset + declaration.size, - fatDeclaration: { - ...declaration, - tableEnd: headerEnd, - }, - }; + if (declaration.cpuType !== policy.cpuType) continue; + const rank = machoSubtypeRank(policy, declaration.cpuSubtype); + if (best !== undefined && rank >= bestRank) { + // Equal ranks keep FAT table order unless the slices share a subtype. + if ( + rank === bestRank && + machoCpuSubtypeBase(declaration.cpuSubtype) === + machoCpuSubtypeBase(best.cpuSubtype) + ) + ambiguous = true; + continue; + } + best = declaration; + bestRank = rank; + ambiguous = false; } - if (selected === undefined) - throw invalid("Requested FAT architecture is absent"); + if (best === undefined) throw invalid("Requested FAT architecture is absent"); + if (ambiguous) throw invalid("Ambiguous FAT architecture slice"); + const selected: MachoSlice = { + slice: best.offset, + sliceEnd: best.offset + best.size, + fatDeclaration: { ...best, tableEnd: headerEnd }, + }; if (selected.slice + 4 > bytes.length) throw malformed("Truncated Mach-O header"); const sliceMagic = bytes.readUInt32LE(selected.slice); @@ -126,13 +144,6 @@ export const selectMachoSlice = ( return selected; }; -/** CPU types admitted by `machArchitecture` in the domain binary target parser. */ -const MACHO_CPU_TYPE: Readonly> = { - x86: 7, - x86_64: 0x01000007, - arm: 12, - arm64: 0x0100000c, -}; const MH_MAGIC = 0xfeedface; const MH_CIGAM = 0xcefaedfe; diff --git a/src/native/AppleObjcCategories.ts b/src/native/AppleObjcCategories.ts index bd936e0ce..7beb7fc03 100644 --- a/src/native/AppleObjcCategories.ts +++ b/src/native/AppleObjcCategories.ts @@ -23,7 +23,6 @@ export const decodeObjcCategories = (input: { readonly sections: readonly Section[]; readonly read: ObjcDispatchReaders; readonly result: ObjcSwiftMetadata; - readonly failures: string[]; readonly issues: DecodeIssue[]; readonly admit: () => boolean; /** Decode a method list into implementations attributed to the category. */ @@ -37,7 +36,7 @@ export const decodeObjcCategories = (input: { /** Name of a class defined in this image, from its class_ro_t. */ readonly localClassName: (address: bigint) => string; }): number => { - const { read, result, failures } = input; + const { read, result } = input; let examined = 0; for (const section of input.sections.filter( ({ name }) => name === "__objc_catlist", @@ -93,7 +92,6 @@ export const decodeObjcCategories = (input: { }); } catch (cause: unknown) { const message = cause instanceof Error ? cause.message : String(cause); - failures.push(`Category at 0x${field.toString(16)}: ${message}`); input.issues.push( issue( "objc_properties_categories", diff --git a/src/native/AppleObjcDispatchFacets.ts b/src/native/AppleObjcDispatchFacets.ts index 14f0b42b8..b52545840 100644 --- a/src/native/AppleObjcDispatchFacets.ts +++ b/src/native/AppleObjcDispatchFacets.ts @@ -335,7 +335,6 @@ export const decodeObjcDispatchFacets = (input: { sections, read, result, - failures, issues: categoryIssues, admit: () => budget.admit(), methods, diff --git a/src/native/AppleSwiftVtables.test.ts b/src/native/AppleSwiftVtables.test.ts index dbb1c12b5..fb1615820 100644 --- a/src/native/AppleSwiftVtables.test.ts +++ b/src/native/AppleSwiftVtables.test.ts @@ -67,6 +67,7 @@ describe("Swift vtable coverage facts: unresolved Swift dispatch slots", () => { issues: [], decode: (_address, raw) => ({ kind: "rebase", target: raw }), examined: 0, + decoded: 0, unreadRebaseBytes: 0, }, swift: { @@ -181,6 +182,7 @@ describe("Swift vtable coverage facts: unsupported pointer fixup coverage", () = reason: "Unsupported chained pointer format 99", }), examined: 0, + decoded: 0, unreadRebaseBytes: 0, }, swift: { diff --git a/src/native/CommandRunner.ts b/src/native/CommandRunner.ts index e9775d1b7..ccd49c570 100644 --- a/src/native/CommandRunner.ts +++ b/src/native/CommandRunner.ts @@ -4,6 +4,7 @@ import { createReadStream } from "node:fs"; import { realpath } from "node:fs/promises"; import { err, ok, type Result } from "../domain/result.js"; +import { DarwinProcessOwnershipInspectionError } from "../process/DarwinProcessRunTokenReader.js"; import { OwnedCommandFailure, runOwnedCommand, @@ -39,6 +40,7 @@ export interface NativeCommandCapture { export class NativeCommandFailure extends Error { readonly cleanupOwner: ProviderProcessSupervisor | undefined; readonly cleanupFailures: readonly NativeCommandCleanupFailure[]; + readonly detail: string | null; constructor( readonly tool: string, @@ -58,6 +60,7 @@ export class NativeCommandFailure extends Error { super(`Native command ${tool} failed: ${reason}`, options); this.cleanupOwner = options?.cleanupOwner; this.cleanupFailures = options?.cleanupFailures ?? []; + this.detail = options?.detail ?? null; } } @@ -67,6 +70,8 @@ export interface NativeCommandFailureOptions extends ErrorOptions { readonly cleanupOwner?: ProviderProcessSupervisor; /** All owners still incomplete after a preflight retry blocked this command. */ readonly cleanupFailures?: readonly NativeCommandCleanupFailure[]; + /** Host-side reason the command could not be supervised, such as a missing ownership helper compiler. */ + readonly detail?: string; } /** One failed cleanup attempt; the runner retains `owner` until it is verified stopped. */ @@ -310,6 +315,9 @@ export const resolveXcrunTool = async ( ...(found.error.cleanupFailures.length === 0 ? {} : { cleanupFailures: found.error.cleanupFailures }), + ...(found.error.detail === null + ? {} + : { detail: found.error.detail }), }, found.error.capture, found.error.cleanupFailure, @@ -449,7 +457,16 @@ const captureProcess = async ( ), ); } - return err(new NativeCommandFailure(tool, "io", null, { cause })); + return err( + new NativeCommandFailure(tool, "io", null, { + cause, + // Process supervision needs the Darwin ownership helper; its typed + // message names the compiler or SDK prerequisite to fix. + ...(cause instanceof DarwinProcessOwnershipInspectionError + ? { detail: cause.message } + : {}), + }), + ); } }; diff --git a/src/native/MachoArchitectureSelection.ts b/src/native/MachoArchitectureSelection.ts new file mode 100644 index 000000000..14c5f696f --- /dev/null +++ b/src/native/MachoArchitectureSelection.ts @@ -0,0 +1,80 @@ +import { machoCpuSubtypeBase } from "../domain/apple/machoCpuIdentity.js"; +import type { LipoArchitecture } from "./parsers/lipo.js"; + +interface MachoArchitecturePolicy { + readonly cpuType: number; + readonly subtypes: readonly { + readonly subtype: number; + readonly name: string; + }[]; + readonly variantName?: RegExp; +} + +/** One preference order for FAT subtype identities and Apple tool slice names. */ +export const MACHO_ARCHITECTURE_POLICIES: Readonly< + Record +> = { + x86: { cpuType: 7, subtypes: [{ subtype: 3, name: "i386" }] }, + x86_64: { + cpuType: 0x01000007, + subtypes: [{ subtype: 3, name: "x86_64" }], + }, + arm: { cpuType: 12, subtypes: [], variantName: /^arm$/u }, + arm64: { + cpuType: 0x0100000c, + subtypes: [ + { subtype: 0, name: "arm64" }, + { subtype: 2, name: "arm64e" }, + { subtype: 1, name: "arm64v8" }, + ], + variantName: /^arm64e\.[A-Za-z0-9_]+$/u, + }, +}; + +/** Unlisted subtypes follow the preferred slices in FAT table order. */ +export const machoSubtypeRank = ( + policy: MachoArchitecturePolicy, + subtype: number, +): number => { + const rank = policy.subtypes.findIndex( + (candidate) => candidate.subtype === machoCpuSubtypeBase(subtype), + ); + return rank < 0 ? policy.subtypes.length : rank; +}; + +/** Select a tool spelling using the same preference order as FAT selection. */ +export const selectMachoArchitectureName = ( + policy: MachoArchitecturePolicy, + available: readonly LipoArchitecture[], +): string | undefined => { + let selected: string | undefined; + let bestRank = Number.POSITIVE_INFINITY; + for (const architecture of available) { + if ( + architecture.cpu_type_code !== null && + architecture.cpu_type_code !== policy.cpuType + ) + continue; + const namedSubtype = policy.subtypes.find( + ({ name }) => name === architecture.name, + ); + const recognizedName = + namedSubtype !== undefined || policy.variantName?.test(architecture.name); + if ( + !recognizedName && + (architecture.cpu_type_code === null || + architecture.cpu_subtype_base === null) + ) + continue; + // Parsed identity is authoritative. Names support thin/partial lipo records. + const subtype = architecture.cpu_subtype_base ?? namedSubtype?.subtype; + const rank = + subtype === undefined + ? policy.subtypes.length + : machoSubtypeRank(policy, subtype); + if (rank >= bestRank) continue; + selected = architecture.name; + bestRank = rank; + } + return selected; +}; diff --git a/src/native/NativeMacOSProvider.ts b/src/native/NativeMacOSProvider.ts index 1d1c2a36c..333569e47 100644 --- a/src/native/NativeMacOSProvider.ts +++ b/src/native/NativeMacOSProvider.ts @@ -73,7 +73,7 @@ import { } from "./SignatureTargetBinding.js"; import { parseCodeSignature } from "./parsers/codesign.js"; import { parseDemangledSymbols } from "./parsers/demangle.js"; -import { parseLipoArchitectures } from "./parsers/lipo.js"; +import { parseLipoOutput } from "./parsers/lipo.js"; import { parsePlistJson, parsePlistXml, @@ -507,7 +507,10 @@ class NativeMacOSClient implements AnalysisClient { ); if (!capture.ok) return capture; const provenance = [invocation(capture.value, this.target.path)]; - const architectures = parseLipoArchitectures(capture.value.stdout); + const { architectures } = parseLipoOutput( + capture.value.stdout, + this.target.path, + ); const result = listArchitecturesSchema.parse({ architectures: { items: architectures, @@ -561,6 +564,19 @@ class NativeMacOSClient implements AnalysisClient { async #inspectSignature( signal?: AbortSignal, ): Promise> { + // This provider's signature inspection covers Mach-O code signatures + // only. On another container, codesign's "not signed" verdict cannot + // see a format-level signature such as a xar pkg signature, so + // reporting it would claim unsigned for a layer that was never + // evaluated. + if (this.target.kind !== "executable" || !isMachOFormat(this.target.format)) + return err( + new AnalysisCapabilityUnavailableError( + IDENTITY.id, + "inspect_signature", + "Active artifact is not Mach-O; its container-level signature layer is not evaluated by codesign.", + ), + ); return withSignatureTarget( this.target, this.#signatureResources, @@ -966,6 +982,7 @@ const translateCommandFailure = ( signal: failure.capture?.signal ?? null, stdout_bytes: failure.capture?.stdoutBytes ?? 0, stderr_bytes: failure.capture?.stderrBytes ?? 0, + ...(failure.detail === null ? {} : { detail: failure.detail }), }, }); }; diff --git a/src/native/NativeMachoInspection.test.ts b/src/native/NativeMachoInspection.test.ts index 4ed533c2e..640e540be 100644 --- a/src/native/NativeMachoInspection.test.ts +++ b/src/native/NativeMachoInspection.test.ts @@ -1,7 +1,18 @@ import { describe, expect, it } from "vitest"; -import { AnalysisCancelledError } from "../domain/analysisErrorCore.js"; +import { + CPU, + LC, + dylibCommand, + fatImage, + machoImage, +} from "../artifacts/apple/MachoImage.fixture.js"; +import { + AnalysisCancelledError, + AnalysisOutputError, +} from "../domain/analysisErrorCore.js"; import { err, ok } from "../domain/result.js"; +import { parseMachoLayout, selectMachoSlice } from "./AppleMachoSelection.js"; import type { NativeCommandCapture } from "./CommandRunner.js"; import { inspectNativeMacho } from "./NativeMachoInspection.js"; import { @@ -75,6 +86,62 @@ describe("Mach-O tool architecture selection", () => { "/fixture/app", ]); }); + + it.each([ + { names: ["arm64v8", "arm64e"], selected: "arm64e" }, + { names: ["arm64e", "arm64v8"], selected: "arm64e" }, + { names: ["arm64v8", "arm64e", "arm64"], selected: "arm64" }, + { names: ["arm64v8"], selected: "arm64v8" }, + { names: ["arm64v8", "arm64e.x1"], selected: "arm64v8" }, + { names: ["arm64e.x1", "arm64v8"], selected: "arm64v8" }, + { names: ["arm64v8", "arm64e.v1"], selected: "arm64e.v1" }, + { names: ["arm64e.v1", "arm64v8"], selected: "arm64e.v1" }, + ] as const)( + "keeps inspection and dispatch on $selected for FAT order $names", + async ({ names, selected }) => { + const cpus = { + arm64: CPU.arm64, + arm64v8: { type: CPU.arm64.type, subtype: 1 }, + arm64e: CPU.arm64e, + "arm64e.x1": { type: CPU.arm64.type, subtype: 0x8000000c }, + "arm64e.v1": { type: CPU.arm64.type, subtype: 0x81000002 }, + }; + const slices = names.map((name) => { + const cpu = cpus[name]; + return { + cpu, + bytes: machoImage({ + cpu, + commands: [dylibCommand(LC.LOAD_DYLIB, `/fixture/${name}.dylib`)], + }), + }; + }); + const fat = Buffer.from(fatImage(slices)); + const lipo = names + .map( + (name, index) => + `architecture ${name}\n cputype ${cpus[name].type}\n cpusubtype ${cpus[name].subtype}\n offset ${(index + 1) * 4096}\n size ${slices[index]?.bytes.length}\n align 2^12\n`, + ) + .join(""); + const { commands } = await inspect(lipo, "arm64"); + expect(commands.find(({ tool }) => tool === "otool")?.args).toEqual([ + "-h", + "-l", + "-arch", + selected.split(".")[0], + "/fixture/app", + ]); + expect(commands.find(({ tool }) => tool === "dyld_info")?.args).toContain( + selected, + ); + expect(selectMachoSlice(fat, "arm64").slice).toBe( + (names.findIndex((name) => name === selected) + 1) * 4096, + ); + expect(parseMachoLayout(fat, "arm64").fixups.dylibs).toEqual([ + `/fixture/${selected}.dylib`, + ]); + }, + ); }); describe("Mach-O UUID reconciliation", () => { @@ -186,3 +253,64 @@ describe("Mach-O UUID reconciliation", () => { ); }); }); + +describe("Mach-O slice command scheduling", () => { + const SLICE_TOOLS = ["otool", "nm", "dyld_info", "dyld_info", "dwarfdump"]; + + /** Finish later-launched slice commands first and record overlap. */ + const reorderingRunner = (failing: ReadonlySet = new Set()) => { + const fixtures = new NativeFixtureRunner(); + let inFlight = 0; + let maxInFlight = 0; + let launched = 0; + const run = async (tool: string, args: readonly string[]) => { + if (tool === "file" || tool === "lipo") return fixtures.run(tool, args); + const order = launched++; + inFlight += 1; + maxInFlight = Math.max(maxInFlight, inFlight); + for (let turn = 0; turn < 12 - order * 2; turn += 1) + await new Promise((resolve) => setImmediate(resolve)); + inFlight -= 1; + if (failing.has(tool)) + return err(new AnalysisOutputError("inspect_macho", `${tool} failed`)); + return fixtures.run(tool, args); + }; + return { run, maxInFlight: () => maxInFlight }; + }; + + const inspectWith = (run: ReturnType["run"]) => + inspectNativeMacho({ + target: nativeMachoTarget("/fixture/app"), + run, + invocation: (capture) => ({ + tool: capture.tool, + command: [capture.executable, ...capture.arguments], + tool_version: capture.toolVersion, + version_reason: capture.versionReason, + executable_sha256: capture.executableSha256, + exit: { code: capture.exitCode, signal: capture.signal }, + stdout_bytes: capture.stdoutBytes, + stderr_bytes: capture.stderrBytes, + }), + }); + + it("runs independent slice commands together and keeps provenance in declared order", async () => { + const runner = reorderingRunner(); + const result = await inspectWith(runner.run); + if (!result.ok) throw result.error; + expect(runner.maxInFlight()).toBe(SLICE_TOOLS.length + 1); + expect(result.value.result).toMatchObject({ + provenance: ["file", "lipo", ...SLICE_TOOLS, "vtool"].map((tool) => ({ + tool, + })), + }); + }); + + it("reports the first failing slice command in declared order", async () => { + const runner = reorderingRunner(new Set(["nm", "dwarfdump"])); + const result = await inspectWith(runner.run); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(result.error).toMatchObject({ reason: "nm failed" }); + }); +}); diff --git a/src/native/NativeMachoInspection.ts b/src/native/NativeMachoInspection.ts index 2ec6cd4a9..bb1966c3d 100644 --- a/src/native/NativeMachoInspection.ts +++ b/src/native/NativeMachoInspection.ts @@ -9,8 +9,12 @@ import { import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js"; import { err, ok, type Result } from "../domain/result.js"; import type { NativeCommandCapture } from "./CommandRunner.js"; +import { + MACHO_ARCHITECTURE_POLICIES, + selectMachoArchitectureName, +} from "./MachoArchitectureSelection.js"; import { parseDyldSymbols } from "./parsers/dyldInfo.js"; -import { parseLipoArchitectures } from "./parsers/lipo.js"; +import { parseLipoOutput, type LipoArchitecture } from "./parsers/lipo.js"; import { withoutEchoedPathHeader } from "./parsers/echoedPath.js"; import { parseOtoolLoadCommands } from "./parsers/otool.js"; import { reconcileMachoUuid } from "./parsers/machoUuid.js"; @@ -67,9 +71,11 @@ export const inspectNativeMacho = async ( } const lipo = captures.find(({ tool }) => tool === "lipo"); if (lipo === undefined) throw new TypeError("Missing lipo capture"); - const architectures = parseLipoArchitectures(lipo.stdout); - const availableArchitectures = architectures.map(({ name }) => name); - const selection = selectArchitecture(context.target, availableArchitectures); + const { format, architectures } = parseLipoOutput( + lipo.stdout, + context.target.path, + ); + const selection = selectArchitecture(context.target, architectures); if (selection.status === "absent") return err( new AnalysisOutputError( @@ -79,8 +85,11 @@ export const inspectNativeMacho = async ( ); const selectedArchitecture = selection.status === "selected" ? selection.architecture : null; - for (const [tool, prefix] of REQUIRED_SLICE_COMMANDS) { - const captured = await context.run( + const runSliceCommand = ([tool, prefix]: readonly [ + string, + readonly string[], + ]) => + context.run( tool, [ ...prefix, @@ -89,20 +98,18 @@ export const inspectNativeMacho = async ( ], context.signal, ); + // Slice commands only read the selected artifact, so launch them together. + // Every launch settles before return; declared order fixes provenance and + // which failure is reported. + const [required, optional] = await Promise.all([ + Promise.all(REQUIRED_SLICE_COMMANDS.map(runSliceCommand)), + runSliceCommand(OPTIONAL_VTOOL_COMMAND), + ]); + for (const captured of required) { if (!captured.ok) return err(captured.error); captures.push(captured.value); } const limitations: string[] = []; - const [tool, prefix] = OPTIONAL_VTOOL_COMMAND; - const optional = await context.run( - tool, - [ - ...prefix, - ...selectedArchitectureArguments(tool, selectedArchitecture), - context.target.path, - ], - context.signal, - ); if (optional.ok) captures.push(optional.value); else if (optional.error._tag === "AnalysisCapabilityUnavailableError") limitations.push(VTOOL_UNAVAILABLE_LIMITATION); @@ -138,7 +145,7 @@ export const inspectNativeMacho = async ( const containerOffset = selectedArchitectureName === null ? undefined - : /^Non-fat file:/mu.test(lipo.stdout) + : format === "thin" ? 0 : (architectures.find(({ name }) => name === selectedArchitectureName) ?.file_offset ?? undefined); @@ -178,7 +185,7 @@ const normalizeMacho = ( toInvocation: (capture: NativeCommandCapture) => NativeCommandInvocation, additionalLimitations: readonly string[], load: ReturnType, - architectures: ReturnType, + architectures: readonly LipoArchitecture[], artifactPath: string, selectedArchitecture: { readonly name: string | null; @@ -396,25 +403,24 @@ type ArchitectureSelection = const selectArchitecture = ( target: BinaryTarget, - available: readonly string[], + available: readonly LipoArchitecture[], ): ArchitectureSelection => { - if (target.kind !== "executable" || target.availableArchitectures.length < 2) + // BinaryTarget lists CPU families and lipo lists slices. A universal file + // can hold several slices of one family (arm64e and arm64e.x1), and slice + // tools without -arch print every slice. + if ( + target.kind !== "executable" || + (target.availableArchitectures.length < 2 && available.length < 2) + ) return { status: "unfiltered" }; const normalized = target.architecture === "x86" ? "i386" : target.architecture; - if (available.includes(normalized)) - return { status: "selected", architecture: normalized }; - // BinaryTarget intentionally normalizes ARM64 CPU subtypes. Recover the - // concrete arm64e slice name from lipo before invoking slice-aware tools. - if (normalized === "arm64") { - const arm64e = available.find((architecture) => architecture === "arm64e"); - if (arm64e !== undefined) - return { status: "selected", architecture: arm64e }; - const variant = available.find((architecture) => - /^arm64e\.[A-Za-z0-9_]+$/u.test(architecture), - ); - if (variant !== undefined) - return { status: "selected", architecture: variant }; - } + const policy = MACHO_ARCHITECTURE_POLICIES[target.architecture]; + const selected = + policy === undefined + ? available.find(({ name }) => name === normalized)?.name + : selectMachoArchitectureName(policy, available); + if (selected !== undefined) + return { status: "selected", architecture: selected }; return { status: "absent", architecture: normalized }; }; diff --git a/src/native/NativeUiObservation.test.ts b/src/native/NativeUiObservation.test.ts index d2238bd01..527afd8db 100644 --- a/src/native/NativeUiObservation.test.ts +++ b/src/native/NativeUiObservation.test.ts @@ -35,6 +35,37 @@ const validPng = Buffer.from( "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+jv4sAAAAASUVORK5CYII=", "base64", ); +it.each([ + { offset: 8, value: 12 }, + { offset: 16, value: 0x8000_0000 }, + { offset: 20, value: 0x8000_0000 }, +])( + "rejects invalid IHDR metadata with a matching digest", + async ({ offset, value }) => { + const bytes = Buffer.from(validPng); + bytes.writeUInt32BE(value, offset); + const screenshot = { + mime_type: "image/png", + base64: bytes.toString("base64"), + sha256: createHash("sha256").update(bytes).digest("hex"), + width: bytes.readUInt32BE(16), + height: bytes.readUInt32BE(20), + }; + const result = await observeNativeUi( + target, + "observe_native_ui", + { ...scope, screenshot: true }, + { + environment: {}, + invoke: async () => ({ + ok: true, + result: { ...snapshot, screenshot }, + }), + }, + ); + expect(result.ok).toBe(false); + }, +); describe("native UI screenshot validation", () => { it("rejects malformed or mismatched screenshot bytes from the helper", async () => { const screenshot = { diff --git a/src/native/parsers/codesign.ts b/src/native/parsers/codesign.ts index 3a39a389e..74a472f33 100644 --- a/src/native/parsers/codesign.ts +++ b/src/native/parsers/codesign.ts @@ -16,14 +16,11 @@ export const parseCodeSignature = ( unsigned: boolean, executablePaths: readonly string[] = [], ): Omit => { - const { identifier, fields } = splitEchoedValues( - output.replaceAll("\r\n", "\n"), - executablePaths, - ); + const { identifier, fields } = splitEchoedValues(output, executablePaths); const values = new Map(); const authorities: string[] = []; const cdhashes: string[] = []; - for (const line of fields.split("\n")) { + for (const line of fields.split(/\r?\n/u)) { if (line.startsWith("CodeDirectory ")) { values.set("CodeDirectory", line.slice("CodeDirectory ".length)); continue; @@ -65,12 +62,20 @@ const splitEchoedValues = ( text: string, executablePaths: readonly string[], ): { readonly identifier: string | undefined; readonly fields: string } => { - const executable = executablePaths - .map((path) => `Executable=${path}\n`) - .find((line) => text.startsWith(line)); let rest = text; - if (executable !== undefined) rest = text.slice(executable.length); - else if (text.startsWith("Executable=")) { + // Frame the exact source path before interpreting record separators. A CRLF + // inside that path or the signing identifier is payload, not a line ending. + let newline: string | undefined; + for (const path of executablePaths) { + const prefix = `Executable=${path}`; + if (!text.startsWith(prefix)) continue; + const separator = /^\r?\n/u.exec(text.slice(prefix.length))?.[0]; + if (separator === undefined) continue; + newline = separator; + rest = text.slice(prefix.length + separator.length); + break; + } + if (rest === text && text.startsWith("Executable=")) { const identifierLine = text.indexOf("\nIdentifier="); rest = identifierLine < 0 ? text : text.slice(identifierLine + 1); } @@ -79,8 +84,17 @@ const splitEchoedValues = ( // Later fields come from the signature itself, so the last `Format=` line // is the one codesign printed after the identifier. const format = rest.lastIndexOf("\nFormat="); - const lineEnd = rest.indexOf("\n"); - const end = format >= 0 ? format : lineEnd >= 0 ? lineEnd : rest.length; + // Without an exact path, use the producer-owned Format record's terminator. + // Looking inside Identifier instead would confuse literal CRLF payload with + // record framing, or strip a literal trailing CR from an LF-delimited value. + if (newline === undefined && format >= 0) + newline = /\r?\n/u.exec(rest.slice(format + 1))?.[0]; + const boundary = format >= 0 ? format : rest.indexOf("\n"); + const crlf = newline === undefined || newline === "\r\n"; + const end = + boundary < 0 + ? rest.length + : boundary - (crlf && rest[boundary - 1] === "\r" ? 1 : 0); return { identifier: rest.slice("Identifier=".length, end), fields: rest.slice(end), diff --git a/src/native/parsers/lipo.test.ts b/src/native/parsers/lipo.test.ts index 43f08d8b8..b5c97a415 100644 --- a/src/native/parsers/lipo.test.ts +++ b/src/native/parsers/lipo.test.ts @@ -1,10 +1,28 @@ import { describe, expect, it } from "vitest"; -import { parseLipoArchitectures } from "./lipo.js"; +import { parseLipoOutput } from "./lipo.js"; + +const fatRecords = `fat_magic 0xcafebabe +nfat_arch 2 +architecture arm64 + cputype CPU_TYPE_ARM64 + cpusubtype CPU_SUBTYPE_ARM64_ALL + capabilities 0x0 + offset 4096 + size 32 + align 2^12 (4096) +architecture x86_64 + cputype CPU_TYPE_X86_64 + cpusubtype CPU_SUBTYPE_X86_64_ALL + capabilities 0x0 + offset 8192 + size 32 + align 2^12 (4096) +`; describe("lipo architecture facts", () => { it("normalizes symbolic CPU fields and arm64e capability bits", () => { - const architectures = parseLipoArchitectures(`Fat header in: /bin/ls + const { format, architectures } = parseLipoOutput(`Fat header in: /bin/ls fat_magic 0xcafebabe nfat_arch 2 architecture x86_64 @@ -22,6 +40,7 @@ architecture arm64e size 89088 align 2^14 (16384) `); + expect(format).toBe("fat"); expect(architectures).toEqual([ expect.objectContaining({ name: "x86_64", @@ -50,15 +69,16 @@ architecture arm64e it("keeps unknown symbolic CPU values as source text without a code", () => { expect( - parseLipoArchitectures( + parseLipoOutput( "architecture future\n cputype CPU_TYPE_FUTURE\n cpusubtype CPU_SUBTYPE_FUTURE\n offset 4096\n size 64\n align 2^12\n", - ), + ).architectures, ).toMatchObject([ { cpu_type: "CPU_TYPE_FUTURE", cpu_type_code: null, cpu_subtype: "CPU_SUBTYPE_FUTURE", cpu_subtype_code: null, + cpu_subtype_base: null, }, ]); }); @@ -69,24 +89,121 @@ architecture arm64e ["0x0", 3], ])("applies printed x86_64 capabilities %s", (capabilities, code) => { expect( - parseLipoArchitectures( + parseLipoOutput( `architecture x86_64\n cpusubtype CPU_SUBTYPE_X86_64_ALL\n capabilities ${capabilities}\n`, - )[0]?.cpu_subtype_code, + ).architectures[0]?.cpu_subtype_code, ).toBe(code); }); it("leaves the subtype code unknown when lipo prints no capabilities", () => { // macOS 26 lipo prints this blank line for LIB64 and plain x86_64 slices. expect( - parseLipoArchitectures( + parseLipoOutput( "architecture x86_64\n cputype CPU_TYPE_X86_64\n cpusubtype CPU_SUBTYPE_X86_ALL\n capabilities \n offset 16384\n", - ), + ).architectures, ).toMatchObject([ { cpu_subtype: "CPU_SUBTYPE_X86_ALL", cpu_subtype_code: null, + cpu_subtype_base: 3, capabilities: null, }, ]); }); + + it.each([ + { capabilities: "PTR_AUTH_VERSION USERSPACE 1", code: 0x81000002 }, + { capabilities: "", code: null }, + ])( + "retains an arm64e base subtype with capabilities '$capabilities'", + ({ capabilities, code }) => { + expect( + parseLipoOutput( + `architecture arm64e.v1\n cputype CPU_TYPE_ARM64\n cpusubtype CPU_SUBTYPE_ARM64E\n capabilities ${capabilities}\n`, + ).architectures, + ).toMatchObject([ + { + cpu_type_code: 0x0100000c, + cpu_subtype_code: code, + cpu_subtype_base: 2, + }, + ]); + }, + ); +}); + +describe("lipo FAT operand framing", () => { + it.each([ + "/fixture/\narchitecture forged\n offset 123\n size 64", + "/fixture/\nfat_magic 0xcafebabe\nnfat_arch 1\narchitecture forged", + "/fixture/\nNon-fat file: fake is architecture: forged", + "/fixture/first\nsecond\nthird", + "/fixture/first\r\narchitecture forged\r\n offset 123", + "/fixture/[app](arm64).*+$^?\\name", + ])( + "keeps the complete echoed operand out of FAT slice records: %j", + (path) => { + const output = `Fat header in: ${path}\n${fatRecords}`; + + expect(parseLipoOutput(output, path)).toEqual( + parseLipoOutput(fatRecords), + ); + expect( + parseLipoOutput(output, path).architectures.map(({ name }) => name), + ).toEqual(["arm64", "x86_64"]); + }, + ); + + it("keeps a CRLF pathname separate from a CRLF record terminator", () => { + const path = "/fixture/first\r\narchitecture forged\r\n size 64"; + const output = `Fat header in: ${path}\r\n${fatRecords.replaceAll("\n", "\r\n")}`; + + expect(parseLipoOutput(output, path)).toEqual(parseLipoOutput(fatRecords)); + }); + + it("preserves ordinary FAT metadata with and without an operand", () => { + const path = "/fixture/app"; + const output = `Fat header in: ${path}\n${fatRecords}`; + + expect(parseLipoOutput(output, path)).toEqual(parseLipoOutput(output)); + }); + + it("does not require a FAT path header in captured fixtures", () => { + expect(parseLipoOutput(fatRecords, "/fixture/app")).toEqual( + parseLipoOutput(fatRecords), + ); + }); + + it("does not strip a header for a different operand that shares a prefix", () => { + const path = "/fixture/app-extra\narchitecture forged"; + const output = `Fat header in: ${path}\n${fatRecords}`; + + expect(parseLipoOutput(output, "/fixture/app")).toEqual( + parseLipoOutput(output), + ); + }); + + it("preserves thin-file parsing when the operand contains FAT-like lines", () => { + const path = "/fixture/\nFat header in: fake\narchitecture forged"; + const output = `Non-fat file: ${path} is architecture: arm64\n`; + + expect(parseLipoOutput(output, path)).toEqual({ + format: "thin", + architectures: [ + { + name: "arm64", + cpu_type: null, + cpu_subtype: null, + cpu_type_code: null, + cpu_subtype_code: null, + cpu_subtype_base: null, + capabilities: null, + file_offset: null, + size: null, + alignment: null, + }, + ], + }); + expect(parseLipoOutput(output, path)).toEqual(parseLipoOutput(output)); + }); }); diff --git a/src/native/parsers/lipo.ts b/src/native/parsers/lipo.ts index 4f56d8b6b..7f74a51b9 100644 --- a/src/native/parsers/lipo.ts +++ b/src/native/parsers/lipo.ts @@ -1,4 +1,5 @@ import { z } from "zod"; +import { machoCpuSubtypeBase } from "../../domain/apple/machoCpuIdentity.js"; const architectureSchema = z.object({ name: z.string().min(1), @@ -6,6 +7,7 @@ const architectureSchema = z.object({ cpu_subtype: z.string().nullable(), cpu_type_code: z.number().int().nonnegative().nullable(), cpu_subtype_code: z.number().int().nonnegative().nullable(), + cpu_subtype_base: z.number().int().nonnegative().nullable(), capabilities: z.string().nullable(), file_offset: z.number().int().min(0).nullable(), size: z.number().int().min(0).nullable(), @@ -14,8 +16,26 @@ const architectureSchema = z.object({ export type LipoArchitecture = z.infer; -/** Parse `lipo -detailed_info` into deterministic slice metadata. */ -export const parseLipoArchitectures = (output: string): LipoArchitecture[] => { +interface LipoOutput { + readonly format: "thin" | "fat"; + readonly architectures: LipoArchitecture[]; +} + +/** Parse `lipo -detailed_info` framing and deterministic slice metadata. */ +export const parseLipoOutput = ( + output: string, + operandPath?: string, +): LipoOutput => { + // The echoed FAT operand is one value, even when its pathname contains lines. + if (operandPath !== undefined) { + const header = `Fat header in: ${operandPath}`; + if (output.startsWith(header)) { + if (output.startsWith("\r\n", header.length)) + output = output.slice(header.length + 2); + else if (output.startsWith("\n", header.length)) + output = output.slice(header.length + 1); + } + } // lipo echoes a thin file's pathname verbatim, including embedded newlines. // Match its final architecture field before splitting records into lines. const thinArchitecture = @@ -23,19 +43,23 @@ export const parseLipoArchitectures = (output: string): LipoArchitecture[] => { output, )?.[1]; if (thinArchitecture !== undefined) - return [ - architectureSchema.parse({ - name: thinArchitecture, - cpu_type: null, - cpu_subtype: null, - cpu_type_code: null, - cpu_subtype_code: null, - capabilities: null, - file_offset: null, - size: null, - alignment: null, - }), - ]; + return { + format: "thin", + architectures: [ + architectureSchema.parse({ + name: thinArchitecture, + cpu_type: null, + cpu_subtype: null, + cpu_type_code: null, + cpu_subtype_code: null, + cpu_subtype_base: null, + capabilities: null, + file_offset: null, + size: null, + alignment: null, + }), + ], + }; const architectures: LipoArchitecture[] = []; let current: Record | undefined; const flush = (): void => { @@ -54,6 +78,7 @@ export const parseLipoArchitectures = (output: string): LipoArchitecture[] => { current.cpusubtype, current.capabilities, ), + cpu_subtype_base: cpuSubtypeBase(current.cpusubtype), capabilities: current.capabilities ?? null, file_offset: integer(current.offset), size: integer(current.size), @@ -86,7 +111,7 @@ export const parseLipoArchitectures = (output: string): LipoArchitecture[] => { flush(); if (architectures.length === 0) throw new TypeError("lipo output contained no architectures"); - return architectures; + return { format: "fat", architectures }; }; const integer = (value: string | undefined): number | null => { @@ -158,10 +183,10 @@ const cpuSubtypeCode = ( }; /** The subtype bits a lipo record names, without its capability byte. */ -export const lipoSubtypeBits = (value: string | null): number | null => { - if (value === null) return null; +const cpuSubtypeBase = (value: string | undefined): number | null => { + if (value === undefined) return null; const subtype = unsigned32(value) ?? CPU_SUBTYPES[value]; - return subtype === undefined ? null : subtype & 0x00ffffff; + return subtype === undefined ? null : machoCpuSubtypeBase(subtype); }; /** lipo names the LIB64 capability and prints other capability bits in hex. */ diff --git a/src/native/pe/PeResourcesProvider.ts b/src/native/pe/PeResourcesProvider.ts index 4326cb3b7..4f31256d9 100644 --- a/src/native/pe/PeResourcesProvider.ts +++ b/src/native/pe/PeResourcesProvider.ts @@ -1,5 +1,7 @@ import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js"; import { readStableArtifact } from "../../artifacts/readStableArtifact.js"; +import { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js"; +import { analysisErrorWithCleanupFailure } from "../../domain/analysisErrorCleanup.js"; import type { PeResourcesPort } from "../../application/binaryDiagnostics/PeResourcesPort.js"; import type { ExecutionOptions } from "../../application/AnalysisProvider.js"; import { ArtifactOperationError } from "../../domain/artifactOperationError.js"; @@ -12,19 +14,29 @@ import { } from "../../domain/analysisErrorCore.js"; import type { InspectPeResourcesInput } from "../../domain/native/peResources.js"; import { err, ok } from "../../domain/result.js"; +import { ProviderCleanupError } from "../../domain/providerCleanupError.js"; import { PE_RESOURCES_PROVIDER } from "../../application/InvestigationProviders.js"; import { parsePeResources } from "./PeResourceParser.js"; /** Portable parser over one bounded, identity-checked regular file. */ export class PeResourcesProvider implements PeResourcesPort { readonly identity = PE_RESOURCES_PROVIDER; + readonly #resources = new ArtifactResourceScope(); + + close(): Promise { + return this.#resources.close(); + } async inspect(input: InspectPeResourcesInput, options?: ExecutionOptions) { try { - const snapshot = await readStableArtifact( - input.path, - input.max_file_bytes, - options?.signal, + const snapshot = await this.#resources.run(() => + readStableArtifact( + input.path, + input.max_file_bytes, + options?.signal, + undefined, + this.#resources, + ), ); return ok( await parsePeResources( @@ -44,12 +56,21 @@ export class PeResourcesProvider implements PeResourcesPort { } } -const peResourcesFailure = ( +const peResourcesFailurePrimary = ( cause: unknown, input: InspectPeResourcesInput, signal: AbortSignal | undefined, ) => { const operation = "inspect_pe_resources"; + if (cause instanceof ArtifactReaderFailure && cause.reason === "integrity") + return new AnalysisArtifactChangedError( + operation, + input.path, + cause.message, + { cause }, + ); + if (cause instanceof ArtifactReaderFailure && cause.reason === "cancelled") + return new AnalysisCancelledError(operation, { cause }); if (signal?.aborted) return new AnalysisCancelledError(operation, { cause }); if (cause instanceof ArtifactReaderFailure) { @@ -64,13 +85,6 @@ const peResourcesFailure = ( }, { cause }, ); - if (cause.reason === "integrity") - return new AnalysisArtifactChangedError( - operation, - input.path, - cause.message, - { cause }, - ); if (cause.reason === "path" || cause.reason === "format") return new AnalysisInputError(operation, { cause }, [ { @@ -79,8 +93,6 @@ const peResourcesFailure = ( message: cause.message, }, ]); - if (cause.reason === "cancelled") - return new AnalysisCancelledError(operation, { cause }); } if (cause instanceof Error && "code" in cause) { @@ -106,3 +118,24 @@ const peResourcesFailure = ( { cause }, ); }; + +const peResourcesFailure = ( + cause: unknown, + input: InspectPeResourcesInput, + signal: AbortSignal | undefined, +) => { + const primary = peResourcesFailurePrimary(cause, input, signal); + if (!(cause instanceof ArtifactReaderFailure) || cause.cleanup === undefined) + return primary; + const cleanup = new ProviderCleanupError( + PE_RESOURCES_PROVIDER.id, + cause.cleanup.resources, + { reason: cause.cleanup.reason }, + { operation: "inspect_pe_resources", cause }, + ); + return analysisErrorWithCleanupFailure( + primary, + cleanup, + "inspect_pe_resources", + ); +}; diff --git a/src/native/pwndbg/PwndbgCoreContext.ts b/src/native/pwndbg/PwndbgCoreContext.ts index 0ef29cd9b..db8386d26 100644 --- a/src/native/pwndbg/PwndbgCoreContext.ts +++ b/src/native/pwndbg/PwndbgCoreContext.ts @@ -17,6 +17,7 @@ import { type RecordedCrash, } from "../../domain/native/recordedCrash.js"; import { readStableArtifact } from "../../artifacts/readStableArtifact.js"; +import type { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js"; import { AnalysisError } from "../../domain/analysisErrorBase.js"; import { capturedPwntoolsOutput, @@ -46,6 +47,7 @@ const replySchema = z.discriminatedUnion("ok", [ export const inspectPwndbgCore = async ({ environment, rootPath, + resources, diagnostics, options, runCommand, @@ -53,6 +55,7 @@ export const inspectPwndbgCore = async ({ }: { readonly environment: Readonly; readonly rootPath: string; + readonly resources: ArtifactResourceScope; readonly diagnostics: AnalysisCapturedOutput; readonly options: ExecutionOptions | undefined; readonly runCommand: PwntoolsCommand; @@ -178,11 +181,14 @@ export const inspectPwndbgCore = async ({ replyPath, PWNTOOLS_LIMITS.outputBytes, options?.signal, + undefined, + resources, ); reply = replySchema.parse(JSON.parse(raw.bytes.toString("utf8"))); } catch (cause: unknown) { if (options?.signal?.aborted) throw new AnalysisCancelledError(OPERATION, { + cause, capturedOutput: recordedCrashStageOutput(diagnostics, output), }); throw new AnalysisOutputError( @@ -226,22 +232,22 @@ export const inspectPwndbgCore = async ({ return { ...reply.value, diagnostics: output }; } catch (cause: unknown) { if (cause instanceof AnalysisError && cause.cleanupIncomplete) throw cause; - if ( + const primary = options?.signal?.aborted && (cause === options.signal.reason || (cause instanceof Error && cause.name === "AbortError")) - ) - throw new AnalysisCancelledError(OPERATION, { - capturedOutput: recordedCrashStageOutput(diagnostics, output), - }); - throw pwntoolsDecoderFailure( - cause, - "debugger", - gdb, - environment.REA_PWNTOOLS_PYTHON ?? "", - await readPwntoolsFailureEvidence(cause, stage), - failureContext, - ); + ? new AnalysisCancelledError(OPERATION, { + capturedOutput: recordedCrashStageOutput(diagnostics, output), + }) + : pwntoolsDecoderFailure( + cause, + "debugger", + gdb, + environment.REA_PWNTOOLS_PYTHON ?? "", + await readPwntoolsFailureEvidence(cause, stage, resources), + failureContext, + ); + throw primary; } }; diff --git a/src/native/pwntools/PwntoolsDecoder.ts b/src/native/pwntools/PwntoolsDecoder.ts index 3a28edbfa..6aa9a3a96 100644 --- a/src/native/pwntools/PwntoolsDecoder.ts +++ b/src/native/pwntools/PwntoolsDecoder.ts @@ -16,6 +16,8 @@ import type { AnalysisPartialObservation, } from "../../domain/analysisErrorBase.js"; import { readStableArtifact } from "../../artifacts/readStableArtifact.js"; +import { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js"; +import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js"; import { AnalysisError } from "../../domain/analysisErrorBase.js"; import { AnalysisCancelledError, @@ -30,7 +32,10 @@ import { analysisErrorWithCleanupFailure } from "../../domain/analysisErrorClean import { ProviderAdapterError } from "../../domain/providerAdapterError.js"; import { ProviderCleanupError } from "../../domain/providerCleanupError.js"; import { err, ok, type Result } from "../../domain/result.js"; -import { runOwnedCommand } from "../../process/OwnedCommand.js"; +import { + ownedCommandResources, + runOwnedCommand, +} from "../../process/OwnedCommand.js"; import type { ProviderProcessSupervisor, OwnedProviderProcessSpawnOptions, @@ -46,6 +51,7 @@ import { PWNTOOLS_PROVIDER_IDENTITY, PWNTOOLS_LIMITS, } from "./PwntoolsRelease.js"; +import { pwntoolsInputBytes } from "./PwntoolsInputLimit.js"; const HOST_REQUIREMENT = "The initial real-verified pwntools artifact profile supports Linux x64 only."; @@ -78,6 +84,7 @@ export interface PwntoolsDecoderProfile< readonly payload: Payload; readonly snapshot: { readonly bytes: Buffer; readonly sha256: string }; readonly rootPath: string; + readonly resources: ArtifactResourceScope; readonly diagnostics: AnalysisCapturedOutput; readonly options: ExecutionOptions | undefined; readonly runCommand: PwntoolsCommand; @@ -96,6 +103,7 @@ export class PwntoolsDecoder< symbol, "admitted" | "cancelled" >(); + readonly #resources = new ArtifactResourceScope(); #closing = false; #closePromise: Promise | undefined; #pendingCleanup: @@ -238,12 +246,26 @@ export class PwntoolsDecoder< undefined, context, ); + const inputLimit = pwntoolsInputBytes(this.environment); + if (!inputLimit.configured) + throw pwntoolsUnavailable( + inputLimit.error, + selectedPath, + undefined, + undefined, + { + ...context, + configurationKey: "REA_PWNTOOLS_MAX_INPUT_BYTES", + }, + ); phase = "artifact-read"; selectedPath = input.path; const snapshot = await readStableArtifact( input.path, - PWNTOOLS_LIMITS.inputBytes, + inputLimit.bytes, options?.signal, + undefined, + this.#resources, ); phase = "decoder"; root = await this.createRuntime(); @@ -298,11 +320,16 @@ export class PwntoolsDecoder< replyPath, PWNTOOLS_LIMITS.outputBytes, options?.signal, + undefined, + this.#resources, ); reply = replySchema.parse(JSON.parse(file.bytes.toString("utf8"))); } catch (cause: unknown) { if (options?.signal?.aborted) - throw new AnalysisCancelledError(OPERATION, { capturedOutput }); + throw new AnalysisCancelledError(OPERATION, { + cause, + capturedOutput, + }); throw new AnalysisOutputError( OPERATION, `Owned pwntools decoder reply failed for ${input.path}: ${cause instanceof z.ZodError ? cause.issues[0]?.message + " at " + (cause.issues[0]?.path.map(String).join(".") ?? "root") : cause instanceof Error ? cause.message : String(cause)}`, @@ -369,6 +396,7 @@ export class PwntoolsDecoder< payload: reply.value, snapshot, rootPath: root.path, + resources: this.#resources, diagnostics: capturedOutput, options, runCommand: (spawn, classifyFailure) => @@ -385,12 +413,18 @@ export class PwntoolsDecoder< } return err(cause); } - const evidence = await readPwntoolsFailureEvidence(cause, root?.path); - result = err( + const evidence = await readPwntoolsFailureEvidence( + cause, + root?.path, + this.#resources, + ); + const primary = options?.signal?.aborted && - (cause === options.signal.reason || - (cause instanceof Error && cause.name === "AbortError")) - ? new AnalysisCancelledError(OPERATION) + (cause === options.signal.reason || + (cause instanceof Error && cause.name === "AbortError") || + (cause instanceof ArtifactReaderFailure && + cause.cleanup !== undefined)) + ? new AnalysisCancelledError(OPERATION, { cause }) : pwntoolsDecoderFailure( cause, phase, @@ -398,8 +432,8 @@ export class PwntoolsDecoder< executablePath, evidence, context, - ), - ); + ); + result = err(primary); } if (this.#pendingCleanup !== undefined) { this.#pendingCleanup.failure = result.ok ? undefined : result.error; @@ -483,7 +517,39 @@ export class PwntoolsDecoder< this.#closing = true; this.#closePromise ??= this.#tail .then(async () => { - const failure = await this.#retire(); + let failure: AnalysisError | undefined; + try { + failure = await this.#retire(); + } catch (cause: unknown) { + failure = new ProviderCleanupError( + this.identity.id, + [this.#pendingCleanup?.root?.path ?? "pwntools decoder"], + { reason: cause instanceof Error ? cause.message : String(cause) }, + { operation: this.profile.operation, cause }, + ); + } + try { + await this.#resources.close(); + } catch (cause: unknown) { + const observation = ArtifactReaderFailure.cleanupObservation( + cause, + "pwntools artifact read", + ); + const resourceFailure = new ProviderCleanupError( + this.identity.id, + observation.resources, + { reason: observation.reason }, + { operation: this.profile.operation, cause }, + ); + failure = + failure === undefined + ? resourceFailure + : analysisErrorWithCleanupFailure( + failure, + resourceFailure, + this.profile.operation, + ); + } if (failure !== undefined) throw failure; }) .catch((cause: unknown) => { @@ -502,8 +568,8 @@ export class PwntoolsDecoder< } async #completeCleanup( - retire: () => Promise, - ): Promise { + retire: () => Promise, + ): Promise { const failure = await retire(); return failure !== undefined && this.#closing ? retire() : failure; } @@ -512,13 +578,16 @@ export class PwntoolsDecoder< const pending = this.#pendingCleanup; if (pending?.process === undefined) return undefined; const stopped = await pending.process.stop(); + const ownership = pending.process.launch.ownership; if (stopped.status === "incomplete") return new ProviderCleanupError( this.identity.id, - [ - pending.process.launch.ownership?.runId ?? "owned-decoder", - ...(pending.root === undefined ? [] : [pending.root.path]), - ], + ownership === undefined + ? [ + "owned-decoder", + ...(pending.root === undefined ? [] : [pending.root.path]), + ] + : ownedCommandResources(ownership, pending.root?.path), { reason: stopped.reason, captured_output: { @@ -541,18 +610,46 @@ export class PwntoolsDecoder< return undefined; } - async #retire(): Promise { - const failure = await this.#retireProcess(); - if (failure !== undefined) return failure; + async #retire(): Promise { + let failure: AnalysisError | undefined; + try { + await this.#resources.retryCleanup(); + } catch (cause: unknown) { + const observation = ArtifactReaderFailure.cleanupObservation( + cause, + "pwntools artifact read", + ); + failure = new ProviderCleanupError( + this.identity.id, + observation.resources, + { reason: observation.reason }, + { + operation: this.profile.operation, + cause, + ...(this.#pendingCleanup?.observation === undefined + ? {} + : { partialObservation: this.#pendingCleanup.observation }), + }, + ); + } + const processFailure = await this.#retireProcess(); + if (processFailure !== undefined) { + return failure === undefined + ? processFailure + : analysisErrorWithCleanupFailure( + failure, + processFailure, + this.profile.operation, + ); + } const pending = this.#pendingCleanup; - if (pending === undefined) return undefined; - if (pending.root !== undefined) { + if (pending?.root !== undefined) { const root = pending.root; try { await root.close(); pending.root = undefined; } catch (cause: unknown) { - return new ProviderCleanupError( + const rootFailure = new ProviderCleanupError( this.identity.id, [root.path], { @@ -573,9 +670,18 @@ export class PwntoolsDecoder< : { partialObservation: pending.observation }), }, ); + failure = + failure === undefined + ? rootFailure + : analysisErrorWithCleanupFailure( + failure, + rootFailure, + this.profile.operation, + ); } } - this.#pendingCleanup = undefined; - return undefined; + if (!this.#resources.hasPending && pending?.root === undefined) + this.#pendingCleanup = undefined; + return failure; } } diff --git a/src/native/pwntools/PwntoolsFailures.ts b/src/native/pwntools/PwntoolsFailures.ts index f5e1b2654..e36ae9503 100644 --- a/src/native/pwntools/PwntoolsFailures.ts +++ b/src/native/pwntools/PwntoolsFailures.ts @@ -30,6 +30,7 @@ export interface PwntoolsFailureContext { readonly operation: string; readonly providerId: string; readonly precedingOutput?: AnalysisCapturedOutput; + readonly configurationKey?: string; } const DEFAULT_CONTEXT = { operation: "inspect_binary_layout", @@ -274,7 +275,7 @@ export const pwntoolsUnavailable = ( code: "provider_unavailable", reason, diagnostics: { - configuration_key: "REA_PWNTOOLS_PYTHON", + configuration_key: context.configurationKey ?? "REA_PWNTOOLS_PYTHON", executable_path: path, ...(systemCode === undefined ? {} : { system_code: systemCode }), }, diff --git a/src/native/pwntools/PwntoolsInputLimit.test.ts b/src/native/pwntools/PwntoolsInputLimit.test.ts new file mode 100644 index 000000000..d60c27b80 --- /dev/null +++ b/src/native/pwntools/PwntoolsInputLimit.test.ts @@ -0,0 +1,96 @@ +import { describe, expect, it } from "vitest"; + +import { pwntoolsInputBytes } from "./PwntoolsInputLimit.js"; +import { pwntoolsUnavailable } from "./PwntoolsFailures.js"; +import { + PWNTOOLS_LIMITS, + PWNTOOLS_PROVIDER_IDENTITY, +} from "./PwntoolsRelease.js"; + +const CONSTRAINT = + "REA_PWNTOOLS_MAX_INPUT_BYTES must be a decimal integer from 1 to " + + "1073741824 bytes, and REA never installs Python or packages."; + +describe("pwntools input bound", () => { + it("keeps the shipped default when the override is absent", () => { + expect(pwntoolsInputBytes({})).toEqual({ + configured: true, + bytes: PWNTOOLS_LIMITS.inputBytes, + }); + expect(PWNTOOLS_LIMITS.inputBytes).toBe(32 * 1024 * 1024); + }); + + it.each([ + ["1", 1], + ["67108864", 64 * 1024 * 1024], + ["172879872", 172_879_872], + ["1073741824", 1024 * 1024 * 1024], + ] as const)("accepts %s as %s bytes", (value, expected) => { + expect(pwntoolsInputBytes({ REA_PWNTOOLS_MAX_INPUT_BYTES: value })).toEqual( + { configured: true, bytes: expected }, + ); + }); + + it.each([ + "", + "0", + "-1", + "33554432.0", + "1e8", + " 67108864", + "67_108_864", + "0x2000000", + "1073741825", + "99999999999999999999", + "not-a-number", + ])("rejects %s instead of widening the bound silently", (value) => { + expect(pwntoolsInputBytes({ REA_PWNTOOLS_MAX_INPUT_BYTES: value })).toEqual( + { + configured: false, + error: CONSTRAINT, + }, + ); + }); + + it("rejects a value above the ceiling that a float would round", () => { + const ceiling = pwntoolsInputBytes({ + REA_PWNTOOLS_MAX_INPUT_BYTES: "1073741824", + }); + const above = pwntoolsInputBytes({ + REA_PWNTOOLS_MAX_INPUT_BYTES: "10737418241", + }); + expect(ceiling.configured).toBe(true); + expect(above).toEqual({ configured: false, error: CONSTRAINT }); + }); + + it("never narrows the bound below the shipped default implicitly", () => { + const accepted = [1, 1024, PWNTOOLS_LIMITS.inputBytes].map((bytes) => + pwntoolsInputBytes({ REA_PWNTOOLS_MAX_INPUT_BYTES: String(bytes) }), + ); + expect(accepted.every((limit) => limit.configured)).toBe(true); + }); +}); + +describe("pwntools unavailability configuration key", () => { + const context = { + operation: "inspect_binary_layout", + providerId: PWNTOOLS_PROVIDER_IDENTITY.id, + }; + const reportedKey = (configurationKey?: string) => + pwntoolsUnavailable( + CONSTRAINT, + "/caller-selected/python", + undefined, + undefined, + configurationKey === undefined + ? context + : { ...context, configurationKey }, + ).rejections[0]?.diagnostics["configuration_key"]; + + it("keeps the engine key unless the failure is about the bound", () => { + expect(reportedKey()).toBe("REA_PWNTOOLS_PYTHON"); + expect(reportedKey("REA_PWNTOOLS_MAX_INPUT_BYTES")).toBe( + "REA_PWNTOOLS_MAX_INPUT_BYTES", + ); + }); +}); diff --git a/src/native/pwntools/PwntoolsInputLimit.ts b/src/native/pwntools/PwntoolsInputLimit.ts new file mode 100644 index 000000000..e6867e80f --- /dev/null +++ b/src/native/pwntools/PwntoolsInputLimit.ts @@ -0,0 +1,40 @@ +import { z } from "zod"; + +import { PWNTOOLS_LIMITS } from "./PwntoolsRelease.js"; + +// The artifact is read into one in-memory buffer and staged again as a private +// snapshot before the owned bridge parses it under a 3 GiB address-space +// reservation, so an override stays within a third of that reservation. +const MAXIMUM_INPUT_BYTES = 1024 * 1024 * 1024; + +/** Optional caller-supplied input bound, in bytes, without a decimal point. */ +export const pwntoolsInputBytesSchema = z + .string() + .regex(/^[1-9]\d*$/u, "must be a positive decimal integer") + .transform(Number) + .pipe(z.number().int().safe().max(MAXIMUM_INPUT_BYTES)); + +const CONSTRAINT = + `REA_PWNTOOLS_MAX_INPUT_BYTES must be a decimal integer from 1 to ` + + `${String(MAXIMUM_INPUT_BYTES)} bytes, and REA never installs Python or packages.`; + +/** Resolved input bound, or the reason the caller's override cannot be used. */ +export type PwntoolsInputBytes = + | { readonly configured: true; readonly bytes: number } + | { readonly configured: false; readonly error: string }; + +/** + * Resolve the admitted artifact size for the pwntools adapters. An unset value + * keeps the shipped default, so the bound is only ever widened deliberately. + */ +export const pwntoolsInputBytes = ( + environment: Readonly, +): PwntoolsInputBytes => { + const value = environment.REA_PWNTOOLS_MAX_INPUT_BYTES; + if (value === undefined) + return { configured: true, bytes: PWNTOOLS_LIMITS.inputBytes }; + const parsed = pwntoolsInputBytesSchema.safeParse(value); + return parsed.success + ? { configured: true, bytes: parsed.data } + : { configured: false, error: CONSTRAINT }; +}; diff --git a/src/native/pwntools/PwntoolsRecordedCrashProvider.ts b/src/native/pwntools/PwntoolsRecordedCrashProvider.ts index ffc10d13b..24cee7314 100644 --- a/src/native/pwntools/PwntoolsRecordedCrashProvider.ts +++ b/src/native/pwntools/PwntoolsRecordedCrashProvider.ts @@ -48,6 +48,7 @@ export class PwntoolsRecordedCrashProvider payload, snapshot, rootPath, + resources, diagnostics, options, runCommand, @@ -120,6 +121,7 @@ export class PwntoolsRecordedCrashProvider const context = await inspectPwndbgCore({ environment, rootPath, + resources, diagnostics, options, runCommand, diff --git a/src/native/pwntools/PwntoolsResourceLimits.ts b/src/native/pwntools/PwntoolsResourceLimits.ts index dcbe91d5a..3b379779f 100644 --- a/src/native/pwntools/PwntoolsResourceLimits.ts +++ b/src/native/pwntools/PwntoolsResourceLimits.ts @@ -7,6 +7,7 @@ import { } from "./PwntoolsRelease.js"; import { z } from "zod"; import { PWNTOOLS_LIMITS } from "./PwntoolsRelease.js"; +import type { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js"; /** Actual limits reported by the owned bridge after lowering inherited soft limits. */ export const pwntoolsResourceLimitsSchema = z.strictObject({ @@ -42,6 +43,7 @@ export interface PwntoolsFailureEvidence { export const readPwntoolsFailureEvidence = async ( cause: unknown, rootPath?: string, + resources?: ArtifactResourceScope, ): Promise => { let limitReport: PwntoolsLimitReport | undefined; let marker: PwntoolsFailureEvidence["marker"]; @@ -56,6 +58,9 @@ export const readPwntoolsFailureEvidence = async ( const reported = await readStableArtifact( join(rootPath, "resource.failure"), 1, + undefined, + undefined, + resources, ); const expected = cause.snapshot.exitCode === PWNTOOLS_MEMORY_FAILURE_EXIT ? "M" : "F"; @@ -90,6 +95,9 @@ export const readPwntoolsFailureEvidence = async ( const reported = await readStableArtifact( join(rootPath, "limits.json"), 4096, + undefined, + undefined, + resources, ); limitReport = { limits: pwntoolsResourceLimitsSchema.parse( diff --git a/src/pcap/PcapCommand.ts b/src/pcap/PcapCommand.ts new file mode 100644 index 000000000..0d28a3d75 --- /dev/null +++ b/src/pcap/PcapCommand.ts @@ -0,0 +1,205 @@ +import { randomUUID } from "node:crypto"; +import { AnalysisError } from "../domain/analysisErrorBase.js"; +import { + AnalysisCancelledError, + AnalysisProtocolError, + AnalysisResourceConstraintError, + AnalysisTimeoutError, + AnalysisUnsupportedTargetError, +} from "../domain/analysisErrorCore.js"; +import { + OwnedCommandFailure, + runOwnedCommand, +} from "../process/OwnedCommand.js"; +import type { ProviderProcessSupervisor } from "../process/ProviderProcess.js"; +import { prepareProcessOwnershipInspection } from "../process/ProcessOwnershipObservation.js"; +import { + isWindowsBatchFile, + windowsBatchCommand, + WindowsBatchCommandError, +} from "../process/WindowsBatchCommand.js"; +import { pcapToolUnavailable } from "./PcapConfiguration.js"; + +export interface PcapCommandLimits { + readonly timeoutMs: number; + readonly maxOutputBytes: number; +} + +/** Owned process execution shared by version probes and capture reads. */ +export const runPcapCommand = async (options: { + readonly operation: string; + readonly command: string; + readonly arguments: readonly string[]; + readonly limits: PcapCommandLimits; + readonly environment: Readonly>; + readonly signal: AbortSignal | undefined; + readonly inputPath?: string; + readonly nativeBound?: { + readonly limiter?: { command: string; arguments: readonly string[] }; + readonly memoryLimitBytes?: number; + }; + readonly retain: ( + owner: ProviderProcessSupervisor, + resources: readonly string[], + ) => void; +}): Promise<{ stdout: string; stderr: string }> => { + const { operation, command, limits, signal } = options; + if (signal?.aborted) + throw new AnalysisCancelledError(operation, { cause: signal.reason }); + const hostEnvironment = { + ...process.env, + ...options.environment, + LC_ALL: "C", + LANG: "C", + }; + try { + const batch = process.platform === "win32" && isWindowsBatchFile(command); + const invocation = batch + ? windowsBatchCommand({ + environment: hostEnvironment, + batchPath: command, + arguments: options.arguments, + }) + : { command, arguments: options.arguments }; + const boundedInvocation = + options.nativeBound?.limiter === undefined + ? invocation + : { + command: options.nativeBound.limiter.command, + arguments: [ + ...options.nativeBound.limiter.arguments, + invocation.command, + ...invocation.arguments, + ], + }; + // Native ownership preparation has its own bounded compiler lifecycle; + // do not spend the tool's execution deadline rebuilding a cold SDK cache. + await prepareProcessOwnershipInspection(signal); + const output = await runOwnedCommand( + { + ...boundedInvocation, + runId: randomUUID(), + hostEnvironment, + windowsVerbatimArguments: batch, + expectedCommand: null, + ...(options.nativeBound?.memoryLimitBytes === undefined + ? {} + : { memoryLimitBytes: options.nativeBound.memoryLimitBytes }), + }, + { timeoutMs: limits.timeoutMs, diagnosticBytes: limits.maxOutputBytes }, + signal === undefined ? {} : { signal }, + ); + return { stdout: output.stdout.text, stderr: output.stderr.text }; + } catch (cause) { + if (cause instanceof AnalysisError) throw cause; + if ( + cause instanceof OwnedCommandFailure && + cause.cleanupOwner !== undefined + ) + options.retain(cause.cleanupOwner, cause.resources); + const details = { + cause, + ...(cause instanceof OwnedCommandFailure && cause.snapshot !== null + ? { + capturedOutput: { + stdout: "", + stderr: cause.snapshot.stderr.text, + truncated: + cause.snapshot.stdout.observedBytes > 0 || + (cause.snapshot.diagnosticTruncated ?? false), + exit_code: cause.snapshot.exitCode ?? null, + signal: cause.snapshot.signal ?? null, + }, + } + : {}), + ...(cause instanceof OwnedCommandFailure && cause.cleanupFailure !== null + ? { + cleanup: { + reason: cause.cleanupFailure, + resources: cause.resources, + }, + } + : {}), + }; + if ( + signal?.aborted || + (cause instanceof OwnedCommandFailure && cause.reason === "cancelled") + ) + throw new AnalysisCancelledError(operation, details); + if ( + cause instanceof OwnedCommandFailure && + cause.snapshot?.memoryLimitReached === true + ) + throw new AnalysisResourceConstraintError( + operation, + "memory", + "The owned Windows tool tree reached its enforced committed-memory limit.", + { + process_memory_limit_bytes: + options.nativeBound?.memoryLimitBytes ?? null, + }, + details, + ); + if ( + cause instanceof OwnedCommandFailure && + (cause.snapshot?.signal === "SIGXCPU" || + cause.snapshot?.signal === "SIGXFSZ") + ) + throw new AnalysisResourceConstraintError( + operation, + cause.snapshot.signal === "SIGXCPU" ? "cpu" : "file-size", + `Producer terminated with ${cause.snapshot.signal}; its exact cause is not independently established.`, + null, + details, + ); + if (cause instanceof OwnedCommandFailure && cause.reason === "timeout") + throw new AnalysisTimeoutError(operation, limits.timeoutMs, details); + if (cause instanceof OwnedCommandFailure && cause.reason === "output-limit") + throw new AnalysisResourceConstraintError( + operation, + "memory", + "Tool output exceeded its complete-output budget", + { max_output_bytes: limits.maxOutputBytes }, + details, + ); + if ( + cause instanceof WindowsBatchCommandError || + (cause instanceof Error && + "code" in cause && + (cause.code === "ENOENT" || cause.code === "EACCES")) + ) + throw pcapToolUnavailable( + operation, + `Cannot run selected tool ${command}: ${cause.message}`, + details, + ); + const diagnostic = + cause instanceof OwnedCommandFailure + ? cause.snapshot?.stderr.text.trim() || cause.message + : cause instanceof Error + ? cause.message + : String(cause); + if (options.inputPath === undefined) + throw pcapToolUnavailable( + operation, + `The selected tool ${command} cannot report its version: ${diagnostic}`, + details, + ); + if ( + options.inputPath !== undefined && + /not a capture file|file.*isn't a capture file|file format.*not supported/iu.test( + diagnostic, + ) + ) + throw new AnalysisUnsupportedTargetError( + operation, + options.inputPath, + diagnostic, + details, + ); + throw new AnalysisProtocolError( + `Packet capture tool ${command} failed: ${diagnostic}`, + details, + ); + } +}; diff --git a/src/pcap/PcapConfiguration.test.ts b/src/pcap/PcapConfiguration.test.ts new file mode 100644 index 000000000..edbd223b2 --- /dev/null +++ b/src/pcap/PcapConfiguration.test.ts @@ -0,0 +1,170 @@ +import { describe, expect, it } from "vitest"; + +import { + CAPINFOS_COMMAND_ENV, + TSHARK_COMMAND_ENV, + resolvePcapTools, +} from "./PcapConfiguration.js"; +import { parseCapinfosOutput, parseTsharkJsonOutput } from "./PcapOutput.js"; + +describe("resolvePcapTools", () => { + it("prefers absolute environment paths for both tools", () => { + expect( + resolvePcapTools({ + [TSHARK_COMMAND_ENV]: "/usr/bin/tshark", + [CAPINFOS_COMMAND_ENV]: "/usr/bin/capinfos", + }), + ).toEqual({ + tshark: { command: "/usr/bin/tshark", source: "environment" }, + capinfos: { command: "/usr/bin/capinfos", source: "environment" }, + }); + }); + + it("falls back to PATH resolution without the environment keys", () => { + expect(resolvePcapTools({})).toEqual({ + tshark: { command: "tshark", source: "path" }, + capinfos: { command: "capinfos", source: "path" }, + }); + }); + + it("rejects a relative tshark path explicitly", () => { + expect(() => + resolvePcapTools({ [TSHARK_COMMAND_ENV]: "bin/tshark" }), + ).toThrow(/must be an absolute path/u); + }); + + it("rejects a relative capinfos path explicitly", () => { + expect(() => + resolvePcapTools({ [CAPINFOS_COMMAND_ENV]: "capinfos" }), + ).toThrow(/must be an absolute path/u); + }); +}); + +const REAL_CAPINFOS = [ + "File name: /work/rich-capture.pcap", + "File type: Wireshark/tcpdump/... - pcap", + "File encapsulation: Linux cooked-mode capture v2", + "Packet size limit: file hdr: 262144 bytes", + "Number of packets: 50", + "File size: 23279 bytes", + "Capture duration: 0.373473 seconds", + "Earliest packet time: 2026-10-11 14:13:40.608505", + "Latest packet time: 2026-10-11 14:13:40.981978", + "SHA256: c94f94742b3288b9a979711509ed5eab3028576301f8938319de3c22a6f5c40b", +].join("\n"); + +describe("parseCapinfosOutput", () => { + it("projects metadata from real capinfos output", () => { + expect(parseCapinfosOutput(REAL_CAPINFOS)).toEqual({ + fileType: "Wireshark/tcpdump/... - pcap", + encapsulation: "Linux cooked-mode capture v2", + packetCount: 50, + captureStart: "2026-10-11 14:13:40.608505", + captureEnd: "2026-10-11 14:13:40.981978", + captureDurationSeconds: 0.373473, + snapshotLength: 262144, + sha256: + "c94f94742b3288b9a979711509ed5eab3028576301f8938319de3c22a6f5c40b", + }); + }); + + it("rejects empty metadata rather than reporting complete unknown facts", () => { + expect(() => parseCapinfosOutput("")).toThrow(/machine-readable/u); + }); +}); + +const REAL_TSHARK_JSON = JSON.stringify([ + { + _source: { + layers: { + "frame.number": ["1"], + "frame.protocols": ["sll:ethertype:ip:udp:dns"], + "ip.src": ["192.168.1.1"], + "ip.dst": ["8.8.8.8"], + "dns.qry.name": ["example.com"], + "dns.flags.response": ["1"], + "dns.a": ["93.184.216.34"], + }, + }, + }, + { + _source: { + layers: { + "frame.number": ["2"], + "frame.protocols": ["sll:ethertype:ip:tcp:tls"], + "ip.src": ["192.168.1.1"], + "ip.dst": ["93.184.216.34"], + "tls.handshake.extensions_server_name": ["example.com"], + }, + }, + }, + { + _source: { + layers: { + "frame.number": ["3"], + "frame.protocols": ["sll:ethertype:ip:tcp:http"], + "ip.src": ["192.168.1.1"], + "ip.dst": ["93.184.216.34"], + "http.request.full_uri": ["http://example.com/page"], + }, + }, + }, +]); + +describe("parseTsharkJsonOutput", () => { + it("projects endpoints, DNS, TLS SNI, protocols, and HTTP URIs", async () => { + const result = await parseTsharkJsonOutput(REAL_TSHARK_JSON); + expect(result.endpoints).toEqual( + expect.arrayContaining([ + { address: "192.168.1.1", port: null, packet_count: 3 }, + { address: "8.8.8.8", port: null, packet_count: 1 }, + { address: "93.184.216.34", port: null, packet_count: 2 }, + ]), + ); + expect(result.dnsQueries).toEqual([ + { + query: "example.com", + response_addresses: ["93.184.216.34"], + response_address_count: 1, + }, + ]); + expect(result.tlsSniValues).toEqual(["example.com"]); + expect(result.httpRequestUris).toEqual(["http://example.com/page"]); + expect(result.protocolDistribution).toEqual( + expect.arrayContaining([ + { protocol: "sll:ethertype:ip:udp:dns", packet_count: 1 }, + { protocol: "sll:ethertype:ip:tcp:tls", packet_count: 1 }, + { protocol: "sll:ethertype:ip:tcp:http", packet_count: 1 }, + ]), + ); + }); + + it("retains every endpoint within the aggregate resource budget", async () => { + const packets = Array.from({ length: 10 }, (_, index) => ({ + _source: { + layers: { + "frame.number": [String(index + 1)], + "frame.protocols": ["ip:tcp"], + "ip.src": [`10.0.0.${index}`], + "ip.dst": ["10.0.1.1"], + }, + }, + })); + const result = await parseTsharkJsonOutput(JSON.stringify(packets)); + expect(result.endpoints).toHaveLength(11); + expect(result.endpointCount).toBe(11); // 10 src + 1 dst + }); + + it.each([ + "", + "not json", + "{}", + "null", + "[null]", + '[{"_source":{"layers":{"ip.src":[3]}}}]', + ])("rejects malformed producer output %s", async (text) => { + await expect(parseTsharkJsonOutput(text)).rejects.toMatchObject({ + _tag: "AnalysisProtocolError", + }); + }); +}); diff --git a/src/pcap/PcapConfiguration.ts b/src/pcap/PcapConfiguration.ts new file mode 100644 index 000000000..b2a6212d1 --- /dev/null +++ b/src/pcap/PcapConfiguration.ts @@ -0,0 +1,172 @@ +import { isAbsolute } from "node:path"; + +import type { ProviderAvailability } from "../application/AnalysisProvider.js"; +import type { ProviderRejectionCode } from "../contracts/providerSelection.js"; +import type { AnalysisErrorOptions } from "../domain/analysisErrorBase.js"; +import { AnalysisCapabilityUnavailableError } from "../domain/analysisErrorCore.js"; +import type { JsonValue } from "../domain/jsonValue.js"; + +/** Environment keys selecting the caller's tshark and capinfos binaries. */ +export const TSHARK_COMMAND_ENV = "REA_TSHARK_COMMAND"; +export const CAPINFOS_COMMAND_ENV = "REA_CAPINFOS_COMMAND"; + +const TOOL_REMEDIATION = `Supply Wireshark 4.x tshark and capinfos on PATH or through absolute ${TSHARK_COMMAND_ENV} and ${CAPINFOS_COMMAND_ENV}; REA does not install them.`; + +/** + * Host-tool refusals concern the caller's Wireshark installation, not the + * target, so they carry their own guidance instead of the generic advice to + * choose another target. + */ +export const pcapToolUnavailable = ( + operation: string, + reason: string, + options?: AnalysisErrorOptions, +): AnalysisCapabilityUnavailableError => + new AnalysisCapabilityUnavailableError("pcap", operation, reason, { + ...options, + userMessage: `${/[.!?]$/u.test(reason) ? reason : `${reason}.`} ${TOOL_REMEDIATION}`, + }); + +/** How each tool was selected for this provider instance. */ +export interface PcapToolSelection { + readonly tshark: { command: string; source: "environment" | "path" }; + readonly capinfos: { command: string; source: "environment" | "path" }; +} + +/** + * Resolve the caller-supplied tshark and capinfos binaries. Each accepts an + * absolute environment path or falls back to PATH resolution, mirroring the + * mitmdump pattern. REA never installs the Wireshark suite. + */ +export const resolvePcapTools = ( + environment: Readonly>, +): PcapToolSelection => { + const tsharkEnv = environment[TSHARK_COMMAND_ENV]; + const capinfosEnv = environment[CAPINFOS_COMMAND_ENV]; + if ( + tsharkEnv !== undefined && + tsharkEnv.trim() !== "" && + !isAbsolute(tsharkEnv) + ) + throw new PcapConfigurationFailure( + "tshark_not_absolute", + `${TSHARK_COMMAND_ENV} must be an absolute path to the tshark binary`, + { configured: tsharkEnv }, + ); + if ( + capinfosEnv !== undefined && + capinfosEnv.trim() !== "" && + !isAbsolute(capinfosEnv) + ) + throw new PcapConfigurationFailure( + "capinfos_not_absolute", + `${CAPINFOS_COMMAND_ENV} must be an absolute path to the capinfos binary`, + { configured: capinfosEnv }, + ); + return { + tshark: { + command: + tsharkEnv === undefined || tsharkEnv.trim() === "" + ? "tshark" + : tsharkEnv, + source: + tsharkEnv === undefined || tsharkEnv.trim() === "" + ? "path" + : "environment", + }, + capinfos: { + command: + capinfosEnv === undefined || capinfosEnv.trim() === "" + ? "capinfos" + : capinfosEnv, + source: + capinfosEnv === undefined || capinfosEnv.trim() === "" + ? "path" + : "environment", + }, + }; +}; + +export type PcapConfigurationFailureCode = + | "tshark_not_absolute" + | "capinfos_not_absolute" + | "tshark_missing" + | "capinfos_missing" + | "tshark_version_unresolved" + | "capinfos_version_unresolved" + | "tshark_version_unsupported" + | "capinfos_version_unsupported"; + +export class PcapConfigurationFailure extends Error { + constructor( + readonly code: PcapConfigurationFailureCode, + message: string, + readonly diagnostics: Readonly> = {}, + options?: ErrorOptions, + ) { + super(message, options); + } +} + +/** Recognize the selected Wireshark tool's version banner. */ +const parseVersion = ( + stdout: string, + tool: "tshark" | "capinfos", +): string | null => { + const line = stdout + .split(/\r?\n/u) + .map((entry) => entry.trim()) + .find((entry) => entry !== ""); + const parsed = + line === undefined + ? null + : new RegExp(`^${tool} \\(Wireshark\\) (\\d+)\\.\\d+\\.\\d+`, "iu").exec( + line, + ); + if (parsed === null || line === undefined) return null; + if (parsed[1] !== "4") + throw new PcapConfigurationFailure( + `${tool}_version_unsupported`, + "Packet summary inspection currently requires Wireshark 4.x tools; select an installation with the admitted output profile.", + { version: line }, + ); + return line; +}; + +export interface PcapClientIdentity { + readonly tsharkVersion: string | null; + readonly capinfosVersion: string | null; +} + +/** Probe both tools without reading any capture file. */ +export const inspectPcapClient = async ( + selection: PcapToolSelection, + probe: (command: string) => Promise, +): Promise => ({ + tsharkVersion: parseVersion(await probe(selection.tshark.command), "tshark"), + capinfosVersion: parseVersion( + await probe(selection.capinfos.command), + "capinfos", + ), +}); + +/** Availability in the shape shared by every provider. */ +const rejectionCodeOf = ( + code: PcapConfigurationFailureCode, +): ProviderRejectionCode => + code.includes("missing") + ? "executable_missing" + : code.includes("unsupported") + ? "unsupported_version" + : code.includes("not_absolute") + ? "not_configured" + : "version_unresolved"; + +export const pcapConfigurationAvailability = ( + cause: PcapConfigurationFailure, +): ProviderAvailability => ({ + status: "unavailable", + code: rejectionCodeOf(cause.code), + reason: cause.message, + diagnostics: { ...cause.diagnostics }, +}); diff --git a/src/pcap/PcapNativeBound.ts b/src/pcap/PcapNativeBound.ts new file mode 100644 index 000000000..dc98bfca0 --- /dev/null +++ b/src/pcap/PcapNativeBound.ts @@ -0,0 +1,78 @@ +import { access, realpath } from "node:fs/promises"; +import { constants } from "node:fs"; +import { isAbsolute } from "node:path"; +import { AnalysisError } from "../domain/analysisErrorBase.js"; +import { AnalysisCapabilityUnavailableError } from "../domain/analysisErrorCore.js"; +import type { ArtifactAnalysisResources } from "../process/ArtifactAnalysisResources.js"; +import type { runPcapCommand } from "./PcapCommand.js"; + +const PRLIMIT_COMMAND_ENV = "REA_TSHARK_PRLIMIT_COMMAND"; + +/** Own native capture admission and the exact enforced resource profile. */ +export const preparePcapNativeBound = async ( + environment: Readonly>, + resources: ArtifactAnalysisResources, +): Promise<{ + nativeBound: NonNullable[0]["nativeBound"]>; + limitations: readonly string[]; +}> => { + if (process.platform === "win32") { + return { + nativeBound: { memoryLimitBytes: 1024 ** 3 }, + limitations: [ + "Each owned Windows tool tree has a 1 GiB Job Object committed-memory limit; this is not a machine-wide quota.", + ], + }; + } + if (process.platform !== "linux") { + const reason = + "Packet summaries require an enforceable native memory bound: Linux with util-linux prlimit, or Windows with owned Job Objects. This host's capture boundary remains unsupported."; + throw new AnalysisCapabilityUnavailableError( + "pcap", + "inspect_pcap_capture", + reason, + { userMessage: reason }, + ); + } + const configured = environment[PRLIMIT_COMMAND_ENV] ?? "/usr/bin/prlimit"; + try { + if (!isAbsolute(configured)) + throw new Error(`${PRLIMIT_COMMAND_ENV} must be absolute.`); + const limiter = await realpath(configured); + await access(limiter, constants.X_OK); + const limits = await resources.readLimits({ + address_space_bytes: 1024 ** 3, + file_bytes: 64 * 1024 ** 2, + cpu_seconds: 300, + }); + const nativeBound = { + limiter: { + command: limiter, + arguments: [ + `--as=${limits.address_space_bytes}:${limits.address_space_bytes}`, + `--fsize=${limits.file_bytes}:${limits.file_bytes}`, + `--cpu=${limits.cpu_seconds}:${limits.cpu_seconds}`, + "--core=0:0", + "--", + ], + }, + }; + const limitations = [ + `Linux prlimit preserves stricter inherited ceilings: ${limits.address_space_bytes} bytes of per-process address space, ${limits.file_bytes} bytes per file and ${limits.cpu_seconds} CPU seconds; core dumps are disabled. These are not aggregate machine quotas.`, + ]; + return { nativeBound, limitations }; + } catch (cause) { + if (cause instanceof AnalysisError) throw cause; + const reason = + "Cannot establish the selected Linux resource limiter before processing capture bytes."; + throw new AnalysisCapabilityUnavailableError( + "pcap", + "inspect_pcap_capture", + reason, + { + cause, + userMessage: `${reason} (${cause instanceof Error ? cause.message : String(cause)}) Check that ${PRLIMIT_COMMAND_ENV} (default /usr/bin/prlimit) names an absolute, executable util-linux prlimit.`, + }, + ); + } +}; diff --git a/src/pcap/PcapOutput.test.ts b/src/pcap/PcapOutput.test.ts new file mode 100644 index 000000000..c0e158438 --- /dev/null +++ b/src/pcap/PcapOutput.test.ts @@ -0,0 +1,167 @@ +import { expect, it } from "vitest"; +import { parseCapinfosOutput, parseTsharkJsonOutput } from "./PcapOutput.js"; + +const capture = (layers: ReadonlyArray>) => + JSON.stringify( + layers.map((fields, index) => ({ + _source: { + layers: { + "frame.number": [String(index + 1)], + "frame.protocols": ["ip:udp:dns"], + ...fields, + }, + }, + })), + ); + +it("binds interleaved DNS answers to the question in their response packet", async () => { + const result = await parseTsharkJsonOutput( + capture([ + { "dns.qry.name": ["alpha.example"], "dns.flags.response": ["False"] }, + { "dns.qry.name": ["beta.example"], "dns.flags.response": ["False"] }, + { + "dns.qry.name": ["alpha.example"], + "dns.flags.response": ["True"], + "dns.a": ["192.0.2.10"], + }, + { + "dns.qry.name": ["beta.example"], + "dns.flags.response": ["True"], + "dns.a": ["192.0.2.20"], + }, + ]), + ); + expect(result.dnsQueries).toEqual([ + { + query: "alpha.example", + response_addresses: ["192.0.2.10"], + response_address_count: 1, + }, + { + query: "beta.example", + response_addresses: ["192.0.2.20"], + response_address_count: 1, + }, + ]); + expect(result.partial).toBe(false); +}); + +it("retains unassignable DNS addresses without inventing a query relationship", async () => { + const result = await parseTsharkJsonOutput( + capture([ + { + "dns.qry.name": ["alpha.example", "beta.example"], + "dns.flags.response": ["True"], + "dns.a": ["192.0.2.9"], + }, + { "dns.flags.response": ["True"], "dns.aaaa": ["2001:db8::9"] }, + ]), + ); + expect( + result.dnsQueries.every((query) => query.response_addresses.length === 0), + ).toBe(true); + expect(result.unattributedDnsAddresses).toEqual(["192.0.2.9", "2001:db8::9"]); + expect(result.unattributedDnsPackets).toBe(2); + expect(result.partial).toBe(true); +}); + +it("counts IPv6 endpoints once per packet and preserves every examined fact", async () => { + const packets = Array.from({ length: 51 }, (_, index) => ({ + "ipv6.src": ["2001:db8::1"], + "ipv6.dst": ["2001:db8::1"], + "tls.handshake.extensions_server_name": [`host${String(index)}.example`], + "frame.protocols": [`ipv6:protocol${String(index)}`], + })); + const result = await parseTsharkJsonOutput(capture(packets)); + expect(result.endpoints).toEqual([ + { address: "2001:db8::1", port: null, packet_count: 51 }, + ]); + expect(result.tlsSniCount).toBe(51); + expect(result.tlsSniValues).toHaveLength(51); + expect(result.protocolCount).toBe(51); + expect(result.protocolDistribution).toHaveLength(51); + expect(result.partial).toBe(false); +}); + +it("orders equal-count endpoints and protocols by code point, not host locale", async () => { + // Locale collation puts "fe80::1" first and, under da-DK, "aarp" after "zz". + const result = await parseTsharkJsonOutput( + capture([ + { "ipv6.src": ["fe80::1"], "frame.protocols": ["eth:llc:zz"] }, + { "ipv6.src": ["fe80:1::1"], "frame.protocols": ["eth:llc:aarp"] }, + ]), + ); + expect(result.endpoints.map(({ address }) => address)).toEqual([ + "fe80:1::1", + "fe80::1", + ]); + expect(result.protocolDistribution.map(({ protocol }) => protocol)).toEqual([ + "eth:llc:aarp", + "eth:llc:zz", + ]); +}); + +it("preserves all DNS addresses and their exact examined count", async () => { + const result = await parseTsharkJsonOutput( + capture( + Array.from({ length: 101 }, (_, index) => ({ + "dns.qry.name": ["rotating.example"], + "dns.flags.response": ["True"], + "dns.a": [`192.0.2.${String(index)}`], + })), + ), + ); + expect(result.dnsQueries[0]?.response_addresses).toHaveLength(101); + expect(result.dnsQueries[0]?.response_address_count).toBe(101); + expect(result.partial).toBe(false); +}); + +it("accepts exact machine packet counts and rejects scaled human counts", () => { + const prefix = + "File type: pcap\nFile encapsulation: Ethernet\nNumber of packets: "; + expect(parseCapinfosOutput(`${prefix}20010`).packetCount).toBe(20010); + expect(() => parseCapinfosOutput(`${prefix}20 k`)).toThrow( + /machine-readable/u, + ); +}); + +it("excludes transport credentials without normalizing observed URL bytes", async () => { + const result = await parseTsharkJsonOutput( + capture([ + { + "http.request.full_uri": [ + "http://user:password@Example.com/a/../b?token=local-evidence#keep", + ], + }, + { + "http.request.full_uri": [ + "http://other:secret@Example.com/a/../b?token=local-evidence#keep", + ], + }, + ]), + ); + expect(result.httpRequestUris).toEqual([ + "http://Example.com/a/../b?token=local-evidence#keep", + ]); + expect(result.httpRequestUriCount).toBe(2); + expect(result.httpUriRedactionCount).toBe(2); + expect(result.partial).toBe(true); +}); + +it("refuses dense JSON before its expanded graph exceeds the decoding budget", async () => { + await expect( + parseTsharkJsonOutput(`[${"{},".repeat(600_000)}{}]`), + ).rejects.toMatchObject({ + _tag: "AnalysisResourceConstraintError", + }); +}); + +it("permits cancellation while validating and aggregating producer output", async () => { + const controller = new AbortController(); + const result = parseTsharkJsonOutput( + capture([{ "ip.src": ["192.0.2.1"] }]), + controller.signal, + ); + controller.abort(new Error("stopped")); + await expect(result).rejects.toThrow("stopped"); +}); diff --git a/src/pcap/PcapOutput.ts b/src/pcap/PcapOutput.ts new file mode 100644 index 000000000..b14d7ba3c --- /dev/null +++ b/src/pcap/PcapOutput.ts @@ -0,0 +1,316 @@ +import { setImmediate as yieldToEventLoop } from "node:timers/promises"; +import { + AnalysisProtocolError, + AnalysisResourceConstraintError, +} from "../domain/analysisErrorCore.js"; +import { parseRecordedTransportUrl } from "../domain/recordedTransportUrl.js"; +import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js"; + +const OPERATION = "inspect_pcap_capture"; +/** Bounds JSON decoding plus its expanded object graph before JSON.parse. */ +export const TSHARK_JSON_MAX_BYTES = 8 * 1024 * 1024; +const MAX_RETAINED_BYTES = 16 * 1024 * 1024; +const MAX_DECODED_BYTES = 128 * 1024 * 1024; + +const protocolFailure = (reason: string): never => { + throw new AnalysisProtocolError(`Invalid packet-capture output: ${reason}`); +}; + +export interface CapinfosMetadata { + readonly fileType: string; + readonly encapsulation: string; + readonly packetCount: number; + readonly captureStart: string | null; + readonly captureEnd: string | null; + readonly captureDurationSeconds: number | null; + readonly snapshotLength: number | null; + readonly sha256: string | null; +} + +/** capinfos -M with a C locale provides exact numbers instead of "20 k". */ +export const parseCapinfosOutput = (stdout: string): CapinfosMetadata => { + const lines = new Map(); + for (const line of stdout.split(/\r?\n/u)) { + const at = line.indexOf(":"); + if (at >= 0 && !/^\s/u.test(line)) + lines.set(line.slice(0, at), line.slice(at + 1).trim()); + } + const fileType = lines.get("File type"); + const encapsulation = lines.get("File encapsulation"); + const countText = lines.get("Number of packets"); + if ( + !fileType || + !encapsulation || + countText === undefined || + !/^\d+$/u.test(countText) + ) + return protocolFailure( + "capinfos did not provide machine-readable type, encapsulation and packet count", + ); + const packetCount = Number(countText); + if (!Number.isSafeInteger(packetCount)) + return protocolFailure("packet count is not representable"); + const durationText = lines.get("Capture duration"); + const duration = + durationText === undefined + ? null + : /^(\d+(?:\.\d+)?)(?:\s+seconds?)?$/u.exec(durationText); + const snapshot = /^file hdr:\s*(\d+) bytes$/u.exec( + lines.get("Packet size limit") ?? "", + ); + const snapshotLength = snapshot === null ? null : Number(snapshot[1]); + const sha256 = lines.get("SHA256") ?? null; + if (sha256 !== null && !/^[0-9a-f]{64}$/u.test(sha256)) + return protocolFailure("capinfos SHA256 is malformed"); + return { + fileType, + encapsulation, + packetCount, + captureStart: lines.get("Earliest packet time") ?? null, + captureEnd: lines.get("Latest packet time") ?? null, + captureDurationSeconds: duration === null ? null : Number(duration[1]), + snapshotLength: + snapshotLength !== null && + Number.isSafeInteger(snapshotLength) && + snapshotLength > 0 + ? snapshotLength + : null, + sha256, + }; +}; + +export interface TsharkFields { + readonly endpoints: { address: string; port: null; packet_count: number }[]; + readonly endpointCount: number; + readonly dnsQueries: { + query: string; + response_addresses: string[]; + response_address_count: number; + }[]; + readonly dnsQueryCount: number; + readonly tlsSniValues: string[]; + readonly tlsSniCount: number; + readonly protocolDistribution: { protocol: string; packet_count: number }[]; + readonly protocolCount: number; + readonly httpRequestUris: string[]; + readonly httpRequestUriCount: number; + readonly httpUriRedactionCount: number; + readonly unattributedDnsAddresses: string[]; + readonly unattributedDnsAddressCount: number; + readonly unattributedDnsPackets: number; + readonly packetCount: number; + readonly partial: boolean; +} + +const object = (value: unknown): value is Record => + typeof value === "object" && value !== null && !Array.isArray(value); + +/** Validate the fixed JSON field representation before retaining aggregate facts. */ +export const parseTsharkJsonOutput = async ( + stdout: string, + signal?: AbortSignal, +): Promise => { + signal?.throwIfAborted(); + if (Buffer.byteLength(stdout) > TSHARK_JSON_MAX_BYTES) + throw new AnalysisResourceConstraintError( + OPERATION, + "memory", + "tshark JSON exceeds the bounded decoding budget", + { max_json_bytes: TSHARK_JSON_MAX_BYTES }, + ); + // The selected-field protocol has five container levels. Reject deeper + // graphs before materializing them; quoted brackets are string data. + let depth = 0, + quoted = false, + escaped = false; + let decodedBytes = stdout.length * 2; + for (let index = 0; index < stdout.length; index += 1) { + if (index % 65536 === 0) { + await yieldToEventLoop(); + signal?.throwIfAborted(); + } + const char = stdout[index]; + if (quoted) { + if (escaped) escaped = false; + else if (char === "\\") escaped = true; + else if (char === '"') quoted = false; + } else if (char === '"') { + quoted = true; + decodedBytes += 64; + } else if (char === "{" || char === "[") { + decodedBytes += 128; + if (++depth > 5) protocolFailure("unexpected JSON nesting"); + } else if (char === "}" || char === "]") depth -= 1; + else if (char === ":" || char === ",") decodedBytes += 128; + if (decodedBytes > MAX_DECODED_BYTES) + throw new AnalysisResourceConstraintError( + OPERATION, + "memory", + "tshark JSON expansion exceeds the decoding budget", + { max_decoded_bytes: MAX_DECODED_BYTES }, + ); + } + let packets: unknown; + try { + packets = JSON.parse(stdout) as unknown; + } catch (cause) { + throw new AnalysisProtocolError("tshark emitted malformed JSON", { cause }); + } + if (!Array.isArray(packets)) + return protocolFailure("tshark packet root must be an array"); + const endpoints = new Map(); + const dns = new Map>(); + const sni = new Set(); + const protocols = new Map(); + const uris = new Set(); + const observedUris = new Set(); + let uriRedactions = 0; + const unattributed = new Set(); + let unattributedPackets = 0; + let retainedBytes = 0; + const charge = (value: string): void => { + retainedBytes += 192 + 2 * Buffer.byteLength(value); + if (retainedBytes > MAX_RETAINED_BYTES) + throw new AnalysisResourceConstraintError( + OPERATION, + "memory", + "Packet inventory exceeds the retained-fact budget", + { max_retained_bytes: MAX_RETAINED_BYTES }, + ); + }; + const add = (set: Set, value: string): void => { + if (!set.has(value)) { + charge(value); + set.add(value); + } + }; + for (let index = 0; index < packets.length; index += 1) { + if (index % 256 === 0) { + await yieldToEventLoop(); + signal?.throwIfAborted(); + } + const packet: unknown = packets[index]; + if ( + !object(packet) || + !object(packet._source) || + !object(packet._source.layers) + ) + return protocolFailure( + `packet ${String(index + 1)} lacks _source.layers`, + ); + const layers = packet._source.layers; + for (const [key, value] of Object.entries(layers)) + if ( + !Array.isArray(value) || + !value.every((item) => typeof item === "string") + ) + return protocolFailure( + `packet ${String(index + 1)} field ${key} must be an array of strings`, + ); + const field = (name: string): readonly string[] => + (layers[name] as string[] | undefined) ?? []; + const frame = field("frame.number"); + if ( + frame.length !== 1 || + !/^\d+$/u.test(frame[0]!) || + Number(frame[0]) !== index + 1 + ) + return protocolFailure( + "frame inventory is missing, duplicated or out of order", + ); + if ( + field("frame.protocols").length !== 1 || + field("frame.protocols")[0] === "" + ) + return protocolFailure( + "packet protocol inventory is missing or ambiguous", + ); + // Count one occurrence per packet even for loopback or nested IP layers. + for (const address of new Set([ + ...field("ip.src"), + ...field("ip.dst"), + ...field("ipv6.src"), + ...field("ipv6.dst"), + ])) { + if (address === "") continue; + if (!endpoints.has(address)) charge(address); + endpoints.set(address, (endpoints.get(address) ?? 0) + 1); + } + const questions = new Set( + field("dns.qry.name").filter((value) => value !== ""), + ); + for (const question of questions) + if (!dns.has(question)) { + charge(question); + dns.set(question, new Set()); + } + const addresses = [...field("dns.a"), ...field("dns.aaaa")].filter( + (value) => value !== "", + ); + const response = field("dns.flags.response"); + if (addresses.length > 0) { + if ( + questions.size === 1 && + response.length > 0 && + response.every((value) => value === "True" || value === "1") + ) { + const values = dns.get(questions.values().next().value!)!; + for (const address of addresses) add(values, address); + } else { + unattributedPackets += 1; + for (const address of addresses) add(unattributed, address); + } + } + for (const value of field("tls.handshake.extensions_server_name")) + if (value !== "") add(sni, value); + for (const value of new Set(field("frame.protocols"))) { + if (value === "") continue; + if (!protocols.has(value)) charge(value); + protocols.set(value, (protocols.get(value) ?? 0) + 1); + } + for (const value of field("http.request.full_uri")) { + if (value === "" || observedUris.has(value)) continue; + add(observedUris, value); + const { credentials } = parseRecordedTransportUrl(value); + if (credentials !== null) uriRedactions += 1; + if (credentials === "ambiguous") continue; + add( + uris, + credentials === null + ? value + : value.slice(0, credentials.start) + value.slice(credentials.end), + ); + } + } + signal?.throwIfAborted(); + return { + endpoints: [...endpoints] + .sort((a, b) => b[1] - a[1] || compareUnicodeCodePoints(a[0], b[0])) + .map(([address, packet_count]) => ({ + address, + port: null, + packet_count, + })), + endpointCount: endpoints.size, + dnsQueries: [...dns].map(([query, addresses]) => ({ + query, + response_addresses: [...addresses], + response_address_count: addresses.size, + })), + dnsQueryCount: dns.size, + tlsSniValues: [...sni], + tlsSniCount: sni.size, + protocolDistribution: [...protocols] + .sort((a, b) => b[1] - a[1] || compareUnicodeCodePoints(a[0], b[0])) + .map(([protocol, packet_count]) => ({ protocol, packet_count })), + protocolCount: protocols.size, + httpRequestUris: [...uris], + httpRequestUriCount: observedUris.size, + httpUriRedactionCount: uriRedactions, + unattributedDnsAddresses: [...unattributed], + unattributedDnsAddressCount: unattributed.size, + unattributedDnsPackets: unattributedPackets, + packetCount: packets.length, + partial: observedUris.size > uris.size || unattributedPackets > 0, + }; +}; diff --git a/src/pcap/PcapProvider.ts b/src/pcap/PcapProvider.ts new file mode 100644 index 000000000..be2c12ed2 --- /dev/null +++ b/src/pcap/PcapProvider.ts @@ -0,0 +1,491 @@ +import type { + ExecutionOptions, + ProviderAvailability, +} from "../application/AnalysisProvider.js"; +import { + createAnalysisExecution, + type AnalysisExecution, +} from "../application/AnalysisProvider.js"; +import { + pcapResultSchemas, + type PcapRequest, + type PcapOperation, +} from "../domain/pcap/pcapAnalysis.js"; +import { + AnalysisCancelledError, + AnalysisCapabilityUnavailableError, + AnalysisProtocolError, +} from "../domain/analysisErrorCore.js"; +import { AnalysisError } from "../domain/analysisErrorBase.js"; +import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js"; +import { err, ok, type Result } from "../domain/result.js"; +import type { ProviderProcessSupervisor } from "../process/ProviderProcess.js"; +import { + PcapConfigurationFailure, + inspectPcapClient, + resolvePcapTools, + pcapConfigurationAvailability, + pcapToolUnavailable, + type PcapToolSelection, +} from "./PcapConfiguration.js"; +import { + parseCapinfosOutput, + parseTsharkJsonOutput, + TSHARK_JSON_MAX_BYTES, +} from "./PcapOutput.js"; +import { runPcapCommand, type PcapCommandLimits } from "./PcapCommand.js"; +import { ProviderCleanupError } from "../domain/providerCleanupError.js"; +import { createPcapEvidence } from "../domain/pcap/pcapEvidence.js"; +import { preparePcapNativeBound } from "./PcapNativeBound.js"; +import { PcapSnapshot } from "./PcapSnapshot.js"; +import { readStableArtifact } from "../artifacts/readStableArtifact.js"; +import { ArtifactAnalysisResources } from "../process/ArtifactAnalysisResources.js"; +import { readLinuxResourceLimits } from "../process/LinuxResourceLimits.js"; + +export const PCAP_PROVIDER_IDENTITY = { + id: "pcap", + name: "Packet capture analysis", + version: null, +} as const; +const FIELDS = [ + "frame.number", + "frame.protocols", + "ip.src", + "ip.dst", + "ipv6.src", + "ipv6.dst", + "dns.flags.response", + "dns.qry.name", + "dns.a", + "dns.aaaa", + "tls.handshake.extensions_server_name", + "http.request.full_uri", +]; + +export interface PcapProviderOptions { + readonly environment?: Readonly>; +} + +/** Inspect saved captures through bounded, owned tool processes and one stable input copy. */ +export class PcapProvider { + readonly #resources = new ArtifactAnalysisResources( + "pcap", + "inspect_pcap_capture", + readStableArtifact, + readLinuxResourceLimits, + { + message: "A packet summary is already active; retry after it completes.", + details: { max_concurrent_inspections: 1 }, + }, + ); + private readonly environment: Readonly>; + private readonly snapshots = new Set(); + private readonly commands = new Map< + ProviderProcessSupervisor, + { snapshot: PcapSnapshot | undefined; resources: readonly string[] } + >(); + + constructor(options: PcapProviderOptions = {}) { + this.environment = options.environment ?? process.env; + } + + private run( + operation: PcapOperation, + command: string, + args: readonly string[], + limits: PcapCommandLimits, + signal: AbortSignal | undefined, + snapshot?: PcapSnapshot, + inputPath?: string, + ) { + return runPcapCommand({ + operation, + command, + arguments: args, + limits, + signal, + environment: + snapshot === undefined + ? this.environment + : { + ...this.environment, + WIRESHARK_CONFIG_DIR: snapshot.configurationPath, + }, + ...(inputPath === undefined ? {} : { inputPath }), + ...(inputPath === undefined || this.nativeBound === undefined + ? {} + : { nativeBound: this.nativeBound }), + retain: (owner, resources) => { + this.commands.set(owner, { snapshot, resources }); + }, + }); + } + + private nativeBound: Parameters[0]["nativeBound"]; + private nativeLimitations: readonly string[] = []; + + private async prepareCaptureBound(): Promise { + const prepared = await preparePcapNativeBound( + this.environment, + this.#resources, + ); + this.nativeBound = prepared.nativeBound; + this.nativeLimitations = prepared.limitations; + } + + private async client(operation: PcapOperation, signal?: AbortSignal) { + const selection = resolvePcapTools(this.environment); + const identity = await inspectPcapClient( + selection, + async (command) => + ( + await this.run( + operation, + command, + ["--version"], + { timeoutMs: 20_000, maxOutputBytes: 64 * 1024 }, + signal, + ) + ).stdout, + ); + for (const [name, version] of [ + ["tshark", identity.tsharkVersion], + ["capinfos", identity.capinfosVersion], + ] as const) { + if (version === null) + throw new PcapConfigurationFailure( + `${name}_version_unresolved`, + `The selected ${name} command did not report a version. Set REA_${name.toUpperCase()}_COMMAND to its absolute executable path.`, + ); + } + return { + selection, + client: { + tshark_command: selection.tshark.command, + tshark_source: selection.tshark.source, + tshark_version: identity.tsharkVersion, + capinfos_command: selection.capinfos.command, + capinfos_source: selection.capinfos.source, + capinfos_version: identity.capinfosVersion, + }, + }; + } + + async inspectAvailability( + signal?: AbortSignal, + ): Promise { + const result = await this.#resources.run( + async () => ok(await this.inspectAvailabilityAdmitted(signal)), + signal, + ); + if (!result.ok) throw result.error; + return result.value; + } + + private async inspectAvailabilityAdmitted( + signal?: AbortSignal, + ): Promise { + try { + const { client } = await this.client("inspect_pcap_client", signal); + if (process.platform !== "linux" && process.platform !== "win32") + return { + status: "unavailable", + code: "unsupported_host", + reason: + "No enforceable native memory boundary is available for capture summaries on this host.", + diagnostics: client, + }; + await this.prepareCaptureBound(); + return { + status: "available", + code: null, + reason: null, + diagnostics: client, + }; + } catch (cause) { + if (signal?.aborted || cause instanceof AnalysisCancelledError) + throw new AnalysisCancelledError("inspect_pcap_client", { cause }); + if (cause instanceof PcapConfigurationFailure) + return pcapConfigurationAvailability(cause); + return { + status: "unavailable", + code: + cause instanceof AnalysisCapabilityUnavailableError + ? "executable_missing" + : "version_unresolved", + reason: cause instanceof Error ? cause.message : String(cause), + diagnostics: {}, + }; + } + } + + private async discard( + snapshot: PcapSnapshot, + ): Promise { + try { + if ( + [...this.commands.values()].some( + (command) => command.snapshot === snapshot, + ) + ) + throw new Error( + "An unreleased command still owns the capture snapshot", + ); + await snapshot.close(); + this.snapshots.delete(snapshot); + return undefined; + } catch (cause) { + return new ProviderCleanupError( + "pcap", + [snapshot.path], + { reason: String(cause) }, + { operation: "inspect_pcap_capture", cause }, + ); + } + } + + async close(): Promise { + const failures: unknown[] = []; + try { + await this.#resources.close(); + } catch (cause) { + failures.push(cause); + } + for (const owner of this.commands.keys()) { + try { + const result = await owner.stop(); + if (result.status === "incomplete") failures.push(result.reason); + else if (result.status === "not-owned") + failures.push("Command ownership remains unresolved"); + else this.commands.delete(owner); + } catch (cause) { + failures.push(cause); + } + } + for (const snapshot of this.snapshots) { + const failure = await this.discard(snapshot); + if (failure !== undefined) failures.push(failure); + } + if (failures.length > 0) + throw new ProviderCleanupError( + "pcap", + [...this.snapshots] + .map((snapshot) => snapshot.path) + .concat( + [...this.commands.values()].flatMap((command) => command.resources), + ), + { reason: "Owned process or snapshot cleanup failed" }, + { operation: "pcap.close", cause: new AggregateError(failures) }, + ); + } + + async execute( + request: PcapRequest, + options?: ExecutionOptions, + ): Promise> { + return this.#resources.run( + () => this.executeAdmitted(request, options), + options?.signal, + ); + } + + private async executeAdmitted( + request: PcapRequest, + options?: ExecutionOptions, + ): Promise> { + const signal = options?.signal; + let snapshot: PcapSnapshot | undefined; + let result: Result; + try { + signal?.throwIfAborted(); + const { selection, client } = await this.client( + request.operation, + signal, + ); + const provider = { + ...PCAP_PROVIDER_IDENTITY, + version: client.tshark_version, + }; + if (request.operation === "inspect_pcap_client") + result = ok( + createAnalysisExecution( + pcapResultSchemas.inspect_pcap_client.parse({ client }), + provider, + ), + ); + else { + await this.prepareCaptureBound(); + snapshot = await PcapSnapshot.create(); + this.snapshots.add(snapshot); + const identity = await snapshot.capture(request.input.path, signal); + const captured = await this.inspectCapture( + selection, + snapshot, + request.input.path, + identity, + signal, + ); + const data = pcapResultSchemas.inspect_pcap_capture.parse({ + client, + ...captured.result, + }); + result = ok( + createAnalysisExecution(data, provider, { + subject: { + path: request.input.path, + format: "file", + sha256: identity.sha256, + }, + limitations: captured.limitations, + }), + ); + } + } catch (cause) { + result = err( + cause instanceof AnalysisError + ? cause + : signal?.aborted + ? new AnalysisCancelledError(request.operation, { cause }) + : cause instanceof PcapConfigurationFailure + ? pcapToolUnavailable(request.operation, cause.message, { + cause, + }) + : new AnalysisProtocolError( + `Packet-capture analysis failed: ${cause instanceof Error ? cause.message : String(cause)}`, + { cause }, + ), + ); + } + if (signal?.aborted && result.ok) + result = err( + new AnalysisCancelledError(request.operation, { + cause: signal.reason, + partialObservation: createPcapEvidence(request, result.value), + }), + ); + if (snapshot !== undefined) { + const cleanup = await this.discard(snapshot); + if (signal?.aborted && result.ok) + result = err( + new AnalysisCancelledError(request.operation, { + cause: signal.reason, + partialObservation: createPcapEvidence(request, result.value), + }), + ); + if (cleanup !== undefined) + result = result.ok + ? err( + cleanup.retainPartialObservation( + createPcapEvidence(request, result.value), + ), + ) + : err( + analysisErrorWithCleanupFailure( + result.error, + cleanup, + request.operation, + ), + ); + } + return result; + } + + private async inspectCapture( + selection: PcapToolSelection, + snapshot: PcapSnapshot, + path: string, + identity: { bytes: number; sha256: string }, + signal: AbortSignal | undefined, + ) { + const info = await this.run( + "inspect_pcap_capture", + selection.capinfos.command, + ["-M", "--", snapshot.path], + { timeoutMs: 60_000, maxOutputBytes: 64 * 1024 }, + signal, + snapshot, + path, + ); + const metadata = parseCapinfosOutput(info.stdout); + if (metadata.sha256 !== null && metadata.sha256 !== identity.sha256) + throw new AnalysisProtocolError( + "capinfos SHA256 disagrees with the owned capture snapshot", + ); + const dissected = await this.run( + "inspect_pcap_capture", + selection.tshark.command, + [ + "-n", + "-r", + snapshot.path, + "-T", + "json", + "--no-duplicate-keys", + `--temp-dir=${snapshot.configurationPath}`, + ...FIELDS.flatMap((field) => ["-e", field]), + ], + { timeoutMs: 300_000, maxOutputBytes: TSHARK_JSON_MAX_BYTES }, + signal, + snapshot, + path, + ); + const fields = await parseTsharkJsonOutput(dissected.stdout, signal); + await snapshot.verify(identity, path, signal); + if (fields.packetCount !== metadata.packetCount) + throw new AnalysisProtocolError( + `capinfos counted ${String(metadata.packetCount)} packets but tshark returned ${String(fields.packetCount)}`, + ); + return { + result: { + target: { path, ...identity }, + file_type: metadata.fileType, + encapsulation: metadata.encapsulation, + packet_count: metadata.packetCount, + capture_start: metadata.captureStart, + capture_end: metadata.captureEnd, + capture_duration_seconds: metadata.captureDurationSeconds, + diagnostics: { + capinfos_stderr: info.stderr, + tshark_stderr: dissected.stderr, + }, + snapshot_length: metadata.snapshotLength, + endpoints: fields.endpoints, + endpoint_count: fields.endpointCount, + dns_queries: fields.dnsQueries, + dns_query_count: fields.dnsQueryCount, + tls_sni_values: fields.tlsSniValues, + tls_sni_count: fields.tlsSniCount, + protocol_distribution: fields.protocolDistribution, + protocol_count: fields.protocolCount, + http_request_uris: fields.httpRequestUris, + http_request_uri_count: fields.httpRequestUriCount, + http_uri_redaction_count: fields.httpUriRedactionCount, + unattributed_dns_addresses: fields.unattributedDnsAddresses, + unattributed_dns_address_count: fields.unattributedDnsAddressCount, + unattributed_dns_packet_count: fields.unattributedDnsPackets, + coverage: fields.partial ? "partial" : "complete", + }, + limitations: [ + ...this.nativeLimitations, + "Both tools read one owned snapshot; target identity comes from its admitted source descriptor", + "DNS address values are associated only with a single question in the same response packet; values may include additional DNS records, not only final resolution answers", + "Only bytes present in the saved capture are examined; encryption and capture truncation can hide higher-level fields", + ...(fields.partial + ? [ + "One or more projected categories were capped, credential-bearing URIs were omitted or coalesced, or DNS address attribution remains unresolved; counts retain the examined totals", + ] + : []), + ...(fields.httpUriRedactionCount > 0 + ? [ + "HTTP transport credentials were excluded; ambiguous credential spans cause the URI to be omitted", + ] + : []), + ...(fields.unattributedDnsPackets > 0 + ? [ + `DNS address attribution was unresolved in ${String(fields.unattributedDnsPackets)} packet(s); unattributed addresses are retained separately`, + ] + : []), + ], + }; + } +} diff --git a/src/pcap/PcapSnapshot.ts b/src/pcap/PcapSnapshot.ts new file mode 100644 index 000000000..2e9082ccd --- /dev/null +++ b/src/pcap/PcapSnapshot.ts @@ -0,0 +1,269 @@ +import { createHash } from "node:crypto"; +import { chmod, open, stat, mkdir } from "node:fs/promises"; +import { join } from "node:path"; +import { AnalysisError } from "../domain/analysisErrorBase.js"; +import { + AnalysisAccessDeniedError, + AnalysisArtifactChangedError, + AnalysisCancelledError, + AnalysisCapabilityUnavailableError, + AnalysisInputError, + AnalysisResourceConstraintError, +} from "../domain/analysisErrorCore.js"; +import { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js"; +import { + openRegularFile, + RegularFileAdmissionFailure, + sameRegularFileState, +} from "../filesystem/RegularFile.js"; +import { PrivateRuntimeRoot } from "../process/PrivateRuntimeRoot.js"; +import { PrivateRuntimeRootUnavailableError } from "../process/PrivateRuntimeRootUnavailableError.js"; + +const OPERATION = "inspect_pcap_capture"; +const MAX_CAPTURE_BYTES = 2 * 1024 ** 3; + +/** An owned capture copy and every descriptor needed to release it safely. */ +export class PcapSnapshot { + private readonly handles = new Set(); + readonly path: string; + readonly configurationPath: string; + private constructor(private readonly root: PrivateRuntimeRoot) { + this.path = join(root.path, "capture.pcap"); + this.configurationPath = join(root.path, "configuration"); + } + + static async create(): Promise { + try { + return new PcapSnapshot( + await PrivateRuntimeRoot.create({ prefix: "rea-pcap-" }), + ); + } catch (cause) { + if (cause instanceof PrivateRuntimeRootUnavailableError) + throw new AnalysisCapabilityUnavailableError( + "pcap", + OPERATION, + cause.message, + { cause, retryable: cause.retryable }, + ); + throw new AnalysisResourceConstraintError( + OPERATION, + "file-size", + "Cannot allocate the owned capture workspace", + null, + { cause }, + ); + } + } + + private own(handle: OwnedFileHandle): OwnedFileHandle { + this.handles.add(handle); + return handle; + } + private async release(handle: OwnedFileHandle): Promise { + await handle.close(); + this.handles.delete(handle); + } + + async capture( + path: string, + signal?: AbortSignal, + ): Promise<{ bytes: number; sha256: string }> { + const sourceRead = async (read: () => Promise): Promise => { + try { + return await read(); + } catch (cause) { + if (signal?.aborted) + throw new AnalysisCancelledError(OPERATION, { cause }); + if (cause instanceof AnalysisError) throw cause; + if ( + cause instanceof Error && + "code" in cause && + (cause.code === "EACCES" || cause.code === "EPERM") + ) + throw new AnalysisAccessDeniedError(OPERATION, path, cause.code, { + cause, + }); + throw new AnalysisInputError(OPERATION, { cause }, [ + { + path: ["path"], + reason: "invalid_value", + message: `Cannot read selected capture ${path}: ${cause instanceof Error ? cause.message : String(cause)}`, + }, + ]); + } + }; + try { + signal?.throwIfAborted(); + await mkdir(this.configurationPath, { mode: 0o700 }); + const source = await sourceRead(async () => { + try { + return this.own( + new OwnedFileHandle( + await openRegularFile(path, { symlinks: "follow", signal }), + ), + ); + } catch (cause) { + if (cause instanceof RegularFileAdmissionFailure) { + this.own(cause.owner); + throw cause.cause; + } + throw cause; + } + }); + const before = await sourceRead(() => + source.handle.stat({ bigint: true }), + ); + if (before.size > BigInt(MAX_CAPTURE_BYTES)) + throw new AnalysisResourceConstraintError( + OPERATION, + "file-size", + "Capture exceeds the owned-snapshot disk budget", + { max_capture_bytes: MAX_CAPTURE_BYTES }, + ); + const destination = this.own( + new OwnedFileHandle(await open(this.path, "wx", 0o600)), + ); + const buffer = Buffer.alloc(64 * 1024); + const hash = createHash("sha256"); + let copied = 0; + while (copied <= Number(before.size)) { + signal?.throwIfAborted(); + const { bytesRead } = await sourceRead(() => + source.handle.read( + buffer, + 0, + Math.min(buffer.length, Number(before.size) - copied + 1), + copied, + ), + ); + signal?.throwIfAborted(); + if (bytesRead === 0) break; + copied += bytesRead; + if (BigInt(copied) > before.size) + throw new AnalysisArtifactChangedError( + OPERATION, + path, + "Capture grew during acquisition", + ); + const chunk = buffer.subarray(0, bytesRead); + await destination.handle.writeFile(chunk); + hash.update(chunk); + } + const after = await sourceRead(() => + source.handle.stat({ bigint: true }), + ); + const selected = await stat(path, { bigint: true }).catch( + (cause: unknown) => { + if ( + cause instanceof Error && + "code" in cause && + (cause.code === "EACCES" || cause.code === "EPERM") + ) + throw new AnalysisAccessDeniedError(OPERATION, path, cause.code, { + cause, + }); + throw new AnalysisArtifactChangedError( + OPERATION, + path, + "Selected path became unavailable during acquisition", + { cause }, + ); + }, + ); + if ( + BigInt(copied) !== before.size || + !sameRegularFileState(before, after) || + !sameRegularFileState(before, selected) + ) + throw new AnalysisArtifactChangedError( + OPERATION, + path, + "Capture identity or contents changed during acquisition", + ); + await this.release(source); + await this.release(destination); + await chmod(this.path, 0o400); + signal?.throwIfAborted(); + return { bytes: copied, sha256: hash.digest("hex") }; + } catch (cause) { + if (signal?.aborted) + throw new AnalysisCancelledError(OPERATION, { cause }); + if (cause instanceof AnalysisError) throw cause; + throw new AnalysisResourceConstraintError( + OPERATION, + "file-size", + "Owned capture snapshot I/O failed", + null, + { cause }, + ); + } + } + + /** Recheck byte identity after both tools; a read-only mode alone is not proof. */ + async verify( + expected: { bytes: number; sha256: string }, + selectedPath: string, + signal?: AbortSignal, + ): Promise { + try { + const file = this.own( + new OwnedFileHandle( + await openRegularFile(this.path, { symlinks: "reject", signal }), + ), + ); + const before = await file.handle.stat({ bigint: true }); + const buffer = Buffer.alloc(64 * 1024); + const hash = createHash("sha256"); + let bytes = 0; + while (bytes <= expected.bytes) { + signal?.throwIfAborted(); + const read = await file.handle.read( + buffer, + 0, + Math.min(buffer.length, expected.bytes - bytes + 1), + bytes, + ); + if (read.bytesRead === 0) break; + bytes += read.bytesRead; + hash.update(buffer.subarray(0, read.bytesRead)); + } + const after = await file.handle.stat({ bigint: true }); + const selected = await stat(this.path, { bigint: true }); + if ( + bytes !== expected.bytes || + hash.digest("hex") !== expected.sha256 || + !sameRegularFileState(before, after) || + !sameRegularFileState(before, selected) + ) + throw new AnalysisArtifactChangedError( + OPERATION, + selectedPath, + "Owned packet snapshot changed during tool analysis.", + ); + await this.release(file); + } catch (cause) { + if (signal?.aborted) + throw new AnalysisCancelledError(OPERATION, { cause }); + if (cause instanceof RegularFileAdmissionFailure) this.own(cause.owner); + if (cause instanceof AnalysisError) throw cause; + throw new AnalysisArtifactChangedError( + OPERATION, + selectedPath, + "Cannot establish the owned packet snapshot's post-analysis identity.", + { cause }, + ); + } + } + + async close(): Promise { + const results = await Promise.allSettled( + [...this.handles].map((handle) => this.release(handle)), + ); + const failures = results.flatMap((result) => + result.status === "rejected" ? [result.reason as unknown] : [], + ); + if (failures.length > 0) + throw new AggregateError(failures, "Capture descriptor cleanup failed"); + await this.root.close(); + } +} diff --git a/src/process/ArtifactAnalysisResources.ts b/src/process/ArtifactAnalysisResources.ts index 5eb90b8e2..b0083e05b 100644 --- a/src/process/ArtifactAnalysisResources.ts +++ b/src/process/ArtifactAnalysisResources.ts @@ -36,7 +36,10 @@ export class ArtifactAnalysisResources { readonly #artifacts = new ArtifactResourceScope(); readonly #pending = new Map< ArtifactAnalysisRuntime, - ProviderProcessSupervisor | undefined + { + readonly owner: ProviderProcessSupervisor | undefined; + readonly operation: string; + } >(); #tail: Promise = Promise.resolve(); #busy = false; @@ -56,14 +59,15 @@ export class ArtifactAnalysisResources { run( work: () => Promise>, signal?: AbortSignal, + operation = this.operation, ): Promise> { if (this.#closed || signal?.aborted) - return Promise.resolve(err(new AnalysisCancelledError(this.operation))); + return Promise.resolve(err(new AnalysisCancelledError(operation))); if (this.#busy) return Promise.resolve( err( new AnalysisResourceConstraintError( - this.operation, + operation, "capacity", this.capacity.message, this.capacity.details, @@ -73,7 +77,7 @@ export class ArtifactAnalysisResources { this.#busy = true; const running = this.#tail .then(() => this.#artifacts.run(work)) - .catch((cause: unknown) => err(this.#artifactFailure(cause))) + .catch((cause: unknown) => err(this.#artifactFailure(cause, operation))) .finally(() => { this.#busy = false; }); @@ -101,8 +105,9 @@ export class ArtifactAnalysisResources { retain( root: ArtifactAnalysisRuntime, owner?: ProviderProcessSupervisor, + operation = this.operation, ): void { - this.#pending.set(root, owner); + this.#pending.set(root, { owner, operation }); } get pendingPaths(): readonly string[] { @@ -110,12 +115,13 @@ export class ArtifactAnalysisResources { } async release(): Promise { - for (const [root, owner] of this.#pending) { + for (const [root, retained] of this.#pending) { try { + const { owner } = retained; if (owner !== undefined) { const stopped = await owner.stop(); if (stopped.status === "incomplete") throw new Error(stopped.reason); - this.#pending.set(root, undefined); + this.#pending.set(root, { ...retained, owner: undefined }); } await root.close(); this.#pending.delete(root); @@ -124,7 +130,7 @@ export class ArtifactAnalysisResources { this.providerId, [root.path], { reason: String(cause) }, - { operation: this.operation, cause }, + { operation: retained.operation, cause }, ); } } @@ -134,6 +140,7 @@ export class ArtifactAnalysisResources { observation: NonNullable, cause: unknown, capturedOutput?: AnalysisCapturedOutput, + operation = this.operation, ): ProviderCleanupError { return new ProviderCleanupError( this.providerId, @@ -144,7 +151,7 @@ export class ArtifactAnalysisResources { ? {} : { captured_output: { ...capturedOutput } }), }, - { operation: this.operation, cause }, + { operation, cause }, ); } @@ -155,6 +162,7 @@ export class ArtifactAnalysisResources { readonly output_bytes: number; }, effectiveLimits: LinuxResourceLimits | undefined, + operation = this.operation, ): AnalysisError { const capturedOutput = cause.snapshot === null @@ -171,16 +179,16 @@ export class ArtifactAnalysisResources { ...(capturedOutput === undefined ? {} : { capturedOutput }), }; if (cause.reason === "cancelled") - return new AnalysisCancelledError(this.operation, detail); + return new AnalysisCancelledError(operation, detail); if (cause.reason === "timeout") return new AnalysisTimeoutError( - this.operation, + operation, limits.command_timeout_ms, detail, ); if (cause.reason === "output-limit") return new AnalysisResourceConstraintError( - this.operation, + operation, "memory", cause.message, { output_bytes: limits.output_bytes }, @@ -192,18 +200,19 @@ export class ArtifactAnalysisResources { cause.snapshot?.signal === "SIGXCPU") ) return new AnalysisResourceConstraintError( - this.operation, + operation, cause.snapshot.signal === "SIGXFSZ" ? "file-size" : "cpu", `Producer terminated with ${cause.snapshot.signal}; the signal's exact cause is not independently established.`, effectiveLimits === undefined ? null : { ...effectiveLimits }, detail, ); - return new AnalysisOutputError(this.operation, cause.message, detail); + return new AnalysisOutputError(operation, cause.message, detail); } async finish( result: Result, observation: (value: T) => AnalysisPartialObservation, + operation = this.operation, ): Promise> { try { await this.release(); @@ -223,7 +232,7 @@ export class ArtifactAnalysisResources { : projectAnalysisError(result.error), }, { - operation: this.operation, + operation, cause, ...(partialObservation === undefined ? {} : { partialObservation }), }, @@ -260,11 +269,11 @@ export class ArtifactAnalysisResources { if (descriptorFailure !== undefined) throw descriptorFailure; } - #artifactFailure(cause: unknown): AnalysisError { + #artifactFailure(cause: unknown, operation = this.operation): AnalysisError { if (cause instanceof AnalysisError) return cause; if (cause instanceof ArtifactReaderFailure && cause.cleanup !== undefined) - return this.cleanupFailure(cause.cleanup, cause); - return new ProviderAdapterError(this.providerId, this.operation, { + return this.cleanupFailure(cause.cleanup, cause, undefined, operation); + return new ProviderAdapterError(this.providerId, operation, { cause, diagnostics: { stage: "artifact-resources", reason: String(cause) }, }); diff --git a/src/process/DarwinProcessRunTokenReader.ts b/src/process/DarwinProcessRunTokenReader.ts index 5aafcd787..335e72fbe 100644 --- a/src/process/DarwinProcessRunTokenReader.ts +++ b/src/process/DarwinProcessRunTokenReader.ts @@ -74,7 +74,9 @@ const sourceFiles = [ export const SWIFTC_INTERRUPT = "SIGINT"; /** Deadline for the one-shot helper compilation. */ -const COMPILER_TIMEOUT_MS = 60_000; +// A fresh Swift SDK cache took 88 seconds on the supported Intel macOS VM. +// Keep compilation bounded independently of short provider version probes. +const COMPILER_TIMEOUT_MS = 120_000; /** Bound on compiler diagnostics embedded in the inspection failure text. */ const STDERR_EXCERPT_LIMIT = 512; @@ -253,7 +255,7 @@ export type DarwinProcessReaderCompiler = export const createDarwinProcessRunTokenReader = ( options: { readonly compiler?: DarwinProcessReaderCompiler; - /** Deadline for the helper compilation; the default keeps verification fast. */ + /** Deadline for helper compilation, independent of provider execution. */ readonly compileTimeoutMs?: number; /** * Caller-owned directory holding the compiler's module cache. Unset keeps diff --git a/src/process/OwnedCommand.ts b/src/process/OwnedCommand.ts index 02fb4defc..d375971ae 100644 --- a/src/process/OwnedCommand.ts +++ b/src/process/OwnedCommand.ts @@ -9,6 +9,7 @@ import { } from "./ProviderProcess.js"; import { cleanupOwnedProcessGroup, + type OwnedProcessGroup, type ProcessCleanupFailure, } from "./ProcessOwnership.js"; import { @@ -34,6 +35,17 @@ export class OwnedCommandFailure extends Error { } } +/** Report recorded launch identities; these labels do not authorize signaling. */ +export const ownedCommandResources = ( + ownership: OwnedProcessGroup, + directory?: string, +): readonly string[] => [ + ownership.runId, + ...(directory === undefined ? [] : [directory]), + `pid:${ownership.leaderPid}`, + `process-group:${ownership.processGroupId}`, +]; + /** Execute a single owned process with independent cleanup and bounded diagnostic retention. */ export const runOwnedCommand = async ( spawn: OwnedProviderProcessSpawnOptions, @@ -208,12 +220,7 @@ const collectOwnedCommand = async ( ? {} : { cleanupResult: stopped.cleanupResult }), }, - [ - spawn.runId, - ...(spawn.cwd === undefined ? [] : [spawn.cwd]), - `pid:${launched.ownership.leaderPid}`, - `process-group:${launched.ownership.processGroupId}`, - ], + ownedCommandResources(launched.ownership, spawn.cwd), supervisor, ); if (failure !== undefined) diff --git a/src/process/PrivateRuntimeRoot.ts b/src/process/PrivateRuntimeRoot.ts index 7002e2f65..c319df0de 100644 --- a/src/process/PrivateRuntimeRoot.ts +++ b/src/process/PrivateRuntimeRoot.ts @@ -3,6 +3,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { WindowsPrivateRuntime } from "../windows/WindowsPrivateRuntime.js"; +import { PrivateRuntimeRootUnavailableError } from "./PrivateRuntimeRootUnavailableError.js"; import { windowsNativeCapabilities, @@ -34,16 +35,6 @@ export const privateRuntimeRootCapability = ( ? windowsNativeCapabilities(platform).private_runtime_dacl : { available: true, reason: null, proof: "posix-mkdtemp-owner-only" }; -/** Expected failure when a runtime root's privacy boundary is unavailable. */ -export class PrivateRuntimeRootUnavailableError extends Error { - readonly code = "private-runtime-root-authority-unavailable"; - - constructor(message: string) { - super(message); - this.name = "PrivateRuntimeRootUnavailableError"; - } -} - /** * Owns a private temporary runtime root and removes it idempotently. * @@ -66,7 +57,9 @@ export class PrivateRuntimeRoot { const platform = options.platform ?? process.platform; const capability = privateRuntimeRootCapability(platform); if (!capability.available) - throw new PrivateRuntimeRootUnavailableError(capability.reason); + throw new PrivateRuntimeRootUnavailableError(capability.reason, { + retryable: capability.retryable === true, + }); if (platform === "win32") { const runtime = WindowsPrivateRuntime.create( options.parent ?? tmpdir(), diff --git a/src/process/PrivateRuntimeRootUnavailableError.ts b/src/process/PrivateRuntimeRootUnavailableError.ts new file mode 100644 index 000000000..27e7ae49d --- /dev/null +++ b/src/process/PrivateRuntimeRootUnavailableError.ts @@ -0,0 +1,14 @@ +/** Expected failure when a runtime root's privacy boundary is unavailable. */ +export class PrivateRuntimeRootUnavailableError extends Error { + readonly code = "private-runtime-root-authority-unavailable"; + readonly retryable: boolean; + + constructor( + message: string, + options?: ErrorOptions & { readonly retryable?: boolean }, + ) { + super(message, options); + this.name = "PrivateRuntimeRootUnavailableError"; + this.retryable = options?.retryable ?? false; + } +} diff --git a/src/process/ProviderDeadline.test.ts b/src/process/ProviderDeadline.test.ts new file mode 100644 index 000000000..b43ce6654 --- /dev/null +++ b/src/process/ProviderDeadline.test.ts @@ -0,0 +1,35 @@ +import { afterEach, expect, it, vi } from "vitest"; +import { ProviderStartupDeadline } from "./ProviderDeadline.js"; + +afterEach(() => { + vi.restoreAllMocks(); +}); + +it("keeps remaining time and cancellation classification stable across wall-clock jumps", () => { + const dateNow = vi.spyOn(Date, "now").mockReturnValue(10_000); + const caller = new AbortController(); + const deadline = new ProviderStartupDeadline(10_000, caller.signal); + try { + dateNow.mockReturnValue(9_000_000_000_000); + const remaining = deadline.remainingMs(); + expect(remaining).toBeGreaterThan(0); + expect(remaining).toBeLessThanOrEqual(10_000); + expect(Number.isInteger(remaining)).toBe(true); + expect(deadline.interruption).toBeUndefined(); + caller.abort(new Error("caller stopped startup")); + expect(deadline.interruption).toBe("cancelled"); + } finally { + deadline.dispose(); + } +}); + +it("aborts a bounded wait when its real monotonic deadline expires", async () => { + const deadline = new ProviderStartupDeadline(40); + try { + await expect(deadline.wait(500)).resolves.toBe("aborted"); + expect(deadline.interruption).toBe("timeout"); + expect(deadline.remainingMs()).toBe(0); + } finally { + deadline.dispose(); + } +}); diff --git a/src/process/ProviderDeadline.ts b/src/process/ProviderDeadline.ts index 3081f469f..33cb3a8d0 100644 --- a/src/process/ProviderDeadline.ts +++ b/src/process/ProviderDeadline.ts @@ -42,11 +42,11 @@ export class ProviderStartupDeadline { ) { if (!Number.isFinite(timeoutMs) || timeoutMs <= 0) throw new RangeError("Provider startup timeout must be positive"); - this.#deadlineAt = Date.now() + timeoutMs; + this.#deadlineAt = performance.now() + timeoutMs; this.#externalSignal = signal; this.#onExternalAbort = () => { if (this.#controller.signal.aborted) return; - if (Date.now() >= this.#deadlineAt) { + if (performance.now() >= this.#deadlineAt) { this.#interruption = "timeout"; this.#abort( new DOMException("Provider startup deadline elapsed", "TimeoutError"), @@ -79,13 +79,13 @@ export class ProviderStartupDeadline { get interruption(): "cancelled" | "timeout" | undefined { return ( this.#interruption ?? - (Date.now() >= this.#deadlineAt ? "timeout" : undefined) + (performance.now() >= this.#deadlineAt ? "timeout" : undefined) ); } /** Milliseconds still available to the next startup phase. */ remainingMs(): number { - return Math.max(0, this.#deadlineAt - Date.now()); + return Math.max(0, Math.ceil(this.#deadlineAt - performance.now())); } /** Wait for at most one polling interval under the shared deadline signal. */ diff --git a/src/process/WindowsAuthority.ts b/src/process/WindowsAuthority.ts index ccd041312..b25a22863 100644 --- a/src/process/WindowsAuthority.ts +++ b/src/process/WindowsAuthority.ts @@ -11,6 +11,7 @@ export type WindowsNativeCapability = readonly available: false; readonly reason: string; readonly proof: "not-proven" | "not-applicable"; + readonly retryable?: true; }; /** Security controls required before REA may claim Windows isolation. */ @@ -52,6 +53,7 @@ export const windowsNativeCapabilities = ( available: false, reason: loaded.reason, proof: "not-proven", + ...(loaded.retryable === true ? { retryable: true as const } : {}), } as const); return { job_object_process_ownership: capability, diff --git a/src/process/capture/ProcessCaptureClock.ts b/src/process/capture/ProcessCaptureClock.ts new file mode 100644 index 000000000..1fe9fda0a --- /dev/null +++ b/src/process/capture/ProcessCaptureClock.ts @@ -0,0 +1,3 @@ +/** Keep wall-clock timestamps separate from capture-relative elapsed time. */ +export const monotonicProcessCaptureClock = (): number => + Math.floor(performance.now()); diff --git a/src/process/capture/ProcessCaptureJournal.ts b/src/process/capture/ProcessCaptureJournal.ts index b8a0874d9..07c1f0351 100644 --- a/src/process/capture/ProcessCaptureJournal.ts +++ b/src/process/capture/ProcessCaptureJournal.ts @@ -13,6 +13,7 @@ import type { ProcessCaptureEventJournalEntry, RecordProcessCaptureEvent, } from "../../domain/process/processCapture.js"; +import { monotonicProcessCaptureClock } from "./ProcessCaptureClock.js"; /** Mutable observation-order ledger whose writer is shared by capture producers. */ export interface ProcessCaptureJournal { readonly entries: readonly ProcessCaptureEventJournalEntry[]; @@ -55,7 +56,10 @@ export const scheduleScenarioInteractions = ( const timer = scheduleProcessDelay(event.at_ms, () => { options.dispatchedEventIndexes.add(eventIndex); const terminal = getTerminal(); - const dispatchedAt = Math.max(0, Date.now() - started); + const dispatchedAt = Math.max( + 0, + monotonicProcessCaptureClock() - started, + ); let outcome: InteractionEvent["outcome"] = "dispatched"; if (terminal === undefined) outcome = "target_exited"; else { diff --git a/src/process/capture/ProcessCaptureLifecycle.test.ts b/src/process/capture/ProcessCaptureLifecycle.test.ts index 765635ccc..abad38ac2 100644 --- a/src/process/capture/ProcessCaptureLifecycle.test.ts +++ b/src/process/capture/ProcessCaptureLifecycle.test.ts @@ -5,6 +5,7 @@ import { parseProcessScenario } from "../../domain/process/processScenario.js"; import type { ProcessCaptureFinalization } from "../../domain/process/processCapture.js"; import { awaitTerminalExit } from "./ProcessCaptureLifecycle.js"; import type { ProcessTimer } from "./ProcessTimer.js"; +import { monotonicProcessCaptureClock } from "./ProcessCaptureClock.js"; type Delivery = ProcessCaptureFinalization["signals"][number]["delivery"]; @@ -79,8 +80,8 @@ const start = ( idle_timeout_ms: 10_000, ...scenario, }), - started: Date.now(), - lastOutput: () => Date.now(), + started: monotonicProcessCaptureClock(), + lastOutput: monotonicProcessCaptureClock, signal, signalTarget: fake.signalTarget, recordFinalization: (snapshot) => { diff --git a/src/process/capture/ProcessCaptureLifecycle.ts b/src/process/capture/ProcessCaptureLifecycle.ts index 2d2fcec00..ab9271dda 100644 --- a/src/process/capture/ProcessCaptureLifecycle.ts +++ b/src/process/capture/ProcessCaptureLifecycle.ts @@ -48,6 +48,7 @@ import { type ProcessCaptureTruncationDetails, } from "../../domain/process/processCaptureCoverage.js"; import { scheduleProcessInterval, type ProcessTimer } from "./ProcessTimer.js"; +import { monotonicProcessCaptureClock } from "./ProcessCaptureClock.js"; import type { FilesystemCheckpoint, @@ -266,10 +267,10 @@ export const settleProcessCaptureJournal = async ( quietMs = 25, maxWaitMs = 500, ): Promise => { - const deadline = Date.now() + maxWaitMs; + const deadline = monotonicProcessCaptureClock() + maxWaitMs; let observed = journal.length; - let lastChangeAt = Date.now(); - while (Date.now() < deadline) { + let lastChangeAt = monotonicProcessCaptureClock(); + while (monotonicProcessCaptureClock() < deadline) { await new Promise((resolve) => setTimeout(resolve, 4)); // A stalled event loop can leave timer and poll callbacks that were already // due queued behind this wake-up. Drain them before judging silence so a @@ -277,8 +278,8 @@ export const settleProcessCaptureJournal = async ( await setImmediate(); if (journal.length !== observed) { observed = journal.length; - lastChangeAt = Date.now(); - } else if (Date.now() - lastChangeAt >= quietMs) return; + lastChangeAt = monotonicProcessCaptureClock(); + } else if (monotonicProcessCaptureClock() - lastChangeAt >= quietMs) return; } }; @@ -337,6 +338,12 @@ export const buildCaptureResult = ( truncation_details: options.truncationDetails, limitations: [ "The executable digest is a prelaunch file sample; matching path metadata immediately after spawn does not prove an atomic operating-system image binding.", + ...(Date.parse(options.manifest.completed_at) < + Date.parse(options.manifest.started_at) + ? [ + "Wall-clock completion time precedes start time; elapsed capture timing is recorded separately and remains monotonic.", + ] + : []), "Process trees are sampled and may omit short-lived descendants.", ...(options.samplingPartial ? ["Process-tree sampling ended with an incomplete observation."] @@ -660,7 +667,7 @@ export const observeSettlement = async ( recordEvent("lifecycle", 1); return { state: "unverifiable", elapsed_ms: 0 }; } - const started = Date.now(); + const started = monotonicProcessCaptureClock(); let consecutiveEmpty = 0; let sawLive = false; let deadlineReached = false; @@ -678,19 +685,25 @@ export const observeSettlement = async ( [...observations.values()].some(({ state }) => state === "unverifiable") ) { recordEvent("lifecycle", 1); - return { state: "unverifiable", elapsed_ms: Date.now() - started }; + return { + state: "unverifiable", + elapsed_ms: monotonicProcessCaptureClock() - started, + }; } if ([...observations.values()].every(({ state }) => state === "empty")) { consecutiveEmpty += 1; if (consecutiveEmpty >= 2) { recordEvent("lifecycle", 1); - return { state: "quiesced", elapsed_ms: Date.now() - started }; + return { + state: "quiesced", + elapsed_ms: monotonicProcessCaptureClock() - started, + }; } } else { consecutiveEmpty = 0; sawLive = true; } - deadlineReached = Date.now() - started >= settleMs; + deadlineReached = monotonicProcessCaptureClock() - started >= settleMs; if (deadlineReached) break; await delay(50, undefined, { signal }); } @@ -698,8 +711,14 @@ export const observeSettlement = async ( // A lone empty observation proves neither liveness nor quiescence; only an // observed live member justifies alive_at_deadline. return sawLive - ? { state: "alive_at_deadline", elapsed_ms: Date.now() - started } - : { state: "unverifiable", elapsed_ms: Date.now() - started }; + ? { + state: "alive_at_deadline", + elapsed_ms: monotonicProcessCaptureClock() - started, + } + : { + state: "unverifiable", + elapsed_ms: monotonicProcessCaptureClock() - started, + }; }; /** Name why an attempt is unverified; a blank message never reaches the capture schema. */ @@ -786,7 +805,10 @@ export const awaitTerminalExit = async ({ if (finalizationStartedAt === undefined) return; const record: (typeof finalizationSignals)[number] = { signal: finalizationSignal, - sent_at_ms: Math.round(performance.now() - finalizationStartedAt), + sent_at_ms: Math.max( + 0, + monotonicProcessCaptureClock() - finalizationStartedAt, + ), delivery: "unverified" as FinalizationDelivery, reason: "delivery had not settled when the record was published", }; @@ -856,7 +878,7 @@ export const awaitTerminalExit = async ({ terminal.kill("SIGKILL"); return; } - finalizationStartedAt = performance.now(); + finalizationStartedAt = monotonicProcessCaptureClock(); attempt("SIGTERM"); }; terminal.onExit((exit) => { @@ -867,7 +889,7 @@ export const awaitTerminalExit = async ({ const finalizationElapsedMs = finalizationStartedAt === undefined ? undefined - : Math.round(performance.now() - finalizationStartedAt); + : Math.max(0, monotonicProcessCaptureClock() - finalizationStartedAt); recordEvent("lifecycle", 0); for (const [eventIndex, event] of scenario.events.entries()) { if (dispatchedEventIndexes.has(eventIndex)) continue; @@ -875,7 +897,10 @@ export const awaitTerminalExit = async ({ interactions.push({ sequence, scheduled_at_ms: event.at_ms, - dispatched_at_ms: Math.max(0, Date.now() - started), + dispatched_at_ms: Math.max( + 0, + monotonicProcessCaptureClock() - started, + ), type: event.type, data: event.type === "input" @@ -935,14 +960,20 @@ export const awaitTerminalExit = async ({ forceKill(); } else if (finalizationStartedAt !== undefined) { if ( - performance.now() - finalizationStartedAt >= + monotonicProcessCaptureClock() - finalizationStartedAt >= scenario.finalization_ms ) forceKill(); - } else if (Date.now() - started >= scenario.timeout_ms) { + } else if ( + monotonicProcessCaptureClock() - started >= + scenario.timeout_ms + ) { reason = "timeout"; terminateAtDeadline(); - } else if (Date.now() - lastOutput() >= scenario.idle_timeout_ms) { + } else if ( + monotonicProcessCaptureClock() - lastOutput() >= + scenario.idle_timeout_ms + ) { reason = "idle_timeout"; terminateAtDeadline(); } @@ -971,10 +1002,11 @@ const PROCESS_CLEANUP_VERIFICATION_INTERVAL_MS = 25; const verifyTokenOwnedProcessesUntilSettled = async ( verify: () => Promise, ): Promise => { - const deadline = Date.now() + PROCESS_CLEANUP_VERIFICATION_GRACE_MS; + const deadline = + monotonicProcessCaptureClock() + PROCESS_CLEANUP_VERIFICATION_GRACE_MS; let result = await verify(); while (!result.cleaned) { - const remaining = deadline - Date.now(); + const remaining = deadline - monotonicProcessCaptureClock(); if (remaining <= 0) return result; await new Promise((resolve) => setTimeout( @@ -1288,7 +1320,7 @@ export const captureTerminalFrames = (options: { } outputBytes += bytes; const atMs = normalizeProcessElapsedTime( - Date.now() - options.started, + monotonicProcessCaptureClock() - options.started, options.scenario.normalization.time_bucket_ms, ); const normalized = normalizeProcessText( diff --git a/src/process/capture/ProcessCaptureProgress.ts b/src/process/capture/ProcessCaptureProgress.ts index 83d5a51f5..205334db7 100644 --- a/src/process/capture/ProcessCaptureProgress.ts +++ b/src/process/capture/ProcessCaptureProgress.ts @@ -1,4 +1,5 @@ import type { ProcessCaptureCleanupReport } from "../../domain/process/processCapture.js"; +import { monotonicProcessCaptureClock } from "./ProcessCaptureClock.js"; /** * Progress port accepted by process capture. @@ -61,7 +62,7 @@ const formatProgressMessage = ( /** Serialize observed capture status through one rate-limited reporter. */ export const createProcessCaptureProgressTracker = ( progress: ProcessCaptureProgress | undefined, - now: () => number = Date.now, + now: () => number = monotonicProcessCaptureClock, ) => { const started = now(); let completed = 0; diff --git a/src/process/capture/ProcessHarness.ts b/src/process/capture/ProcessHarness.ts index 1033e68cd..847e512f8 100644 --- a/src/process/capture/ProcessHarness.ts +++ b/src/process/capture/ProcessHarness.ts @@ -70,6 +70,7 @@ import { observeProcessStartIdentity, signalProcessWithStartIdentity, } from "../ProcessOwnershipObservation.js"; +import { monotonicProcessCaptureClock } from "./ProcessCaptureClock.js"; interface StartedCaptureRuntime { readonly renderer: TerminalRenderer; @@ -268,7 +269,7 @@ const startCaptureRuntime = async ( options.signal, ); const startedAt = new Date(); - const started = Date.now(); + const started = monotonicProcessCaptureClock(); let lastOutput = started; renderer = createTerminalRenderer( scenario, @@ -296,7 +297,7 @@ const startCaptureRuntime = async ( ...options, terminal, started, - onOutput: () => (lastOutput = Date.now()), + onOutput: () => (lastOutput = monotonicProcessCaptureClock()), renderer, }); scheduleScenarioInteractions({ @@ -541,7 +542,7 @@ const completeCapture = async (options: { }, { name: "after_settlement", - at_ms: Math.max(0, Date.now() - runtime.started), + at_ms: Math.max(0, monotonicProcessCaptureClock() - runtime.started), files: after.files, effects: classifyFilesystemEffects(options.before, after), truncated: after.truncated, diff --git a/src/process/capture/ProcessNormalization.test.ts b/src/process/capture/ProcessNormalization.test.ts index 9c9ba1951..e5ef61062 100644 --- a/src/process/capture/ProcessNormalization.test.ts +++ b/src/process/capture/ProcessNormalization.test.ts @@ -266,3 +266,47 @@ it("normalizes representative process identities and preserves null, zero, and u "/program child 42020 parent 41010 group 43030 session 44040 listen: selected", ); }); + +it("normalizes nested paths once using the most specific selected identity", () => { + const declared = parseProcessScenario({ + executable: "/workspace/project/bin/program", + working_directory: "/workspace/project", + filesystem_observation_paths: [ + "/workspace/project/data", + "/workspace/project/data/cache", + ], + normalization: { paths: true, pids: false, ports: false, patterns: [] }, + }); + expect( + normalizeProcessText( + "/workspace/project/bin/program /workspace/project/data/cache/item /workspace/project/data/result /workspace/project/log /workspace/tmp", + declared, + "/workspace", + rootPid, + ), + ).toBe( + " /item /result /log /tmp", + ); +}); + +it("treats path metacharacters literally while preferring nested observed roots", () => { + const declared = parseProcessScenario({ + executable: "/selected/[tool]+", + working_directory: "/selected", + filesystem_observation_paths: [ + "/selected/(data)", + "/selected/(data)/cache", + ], + normalization: { paths: true, pids: false, ports: false, patterns: [] }, + }); + expect( + normalizeProcessText( + "/selected/[tool]+ /selected/(data)/cache/item /selected/(data)/file /selected/log /selected/toolll", + declared, + "/tmp", + rootPid, + ), + ).toBe( + " /item /file /log /toolll", + ); +}); diff --git a/src/process/capture/ProcessNormalization.ts b/src/process/capture/ProcessNormalization.ts index 9364ec95b..74db31b1c 100644 --- a/src/process/capture/ProcessNormalization.ts +++ b/src/process/capture/ProcessNormalization.ts @@ -17,19 +17,8 @@ export const normalizeProcessText = ( pid: number, ): string => { let normalized = value; - if (scenario.normalization.paths) { - normalized = normalized.replaceAll(temporaryRoot, ""); - normalized = normalized.replaceAll( - scenario.working_directory, - "", - ); - normalized = normalized.replaceAll(scenario.executable, ""); - for (const [index, root] of scenario.filesystem_observation_paths.entries()) - normalized = normalized.replaceAll( - root, - ``, - ); - } + if (scenario.normalization.paths) + normalized = normalizePathTokens(normalized, scenario, temporaryRoot); if (scenario.normalization.ports) normalized = normalizePortTokens(normalized); if (scenario.normalization.pids) @@ -44,6 +33,39 @@ export const normalizeProcessText = ( return normalized; }; +/** Match original paths once, preferring the most specific selected identity. */ +const normalizePathTokens = ( + value: string, + scenario: ProcessScenario, + temporaryRoot: string, +): string => { + const aliases = new Map(); + const selected: readonly (readonly [string, string])[] = [ + [temporaryRoot, ""], + [scenario.working_directory, ""], + [scenario.executable, ""], + ...scenario.filesystem_observation_paths.map( + (root, index): readonly [string, string] => [ + root, + ``, + ], + ), + ]; + for (const [path, alias] of selected) + if (path !== "" && !aliases.has(path)) aliases.set(path, alias); + const paths = [...aliases.keys()].sort( + (left, right) => right.length - left.length, + ); + if (paths.length === 0) return value; + const pattern = paths + .map((path) => path.replace(/[.*+?^${}()|[\]\\]/gu, "\\$&")) + .join("|"); + return value.replace( + new RegExp(pattern, "gu"), + (path) => aliases.get(path) ?? path, + ); +}; + // A port may end a sentence, but a period followed by a letter, digit, or // underscore in any script continues a decimal, version, or host name and is // not a port boundary. diff --git a/src/process/capture/ProcessSampling.ts b/src/process/capture/ProcessSampling.ts index 69e7dd343..59b67f577 100644 --- a/src/process/capture/ProcessSampling.ts +++ b/src/process/capture/ProcessSampling.ts @@ -6,6 +6,7 @@ import type { } from "../../domain/process/processCapture.js"; import { execFileOutput } from "../ExecFileOutput.js"; import { readProcessRunId } from "../ProcessOwnership.js"; +import { monotonicProcessCaptureClock } from "./ProcessCaptureClock.js"; const PROC_READ_CONCURRENCY = 64; interface ProcessRow { @@ -491,7 +492,7 @@ export const startProcessSampler = (options: { } return sampleProcesses({ rootPid, - elapsedMs: Date.now() - started, + elapsedMs: monotonicProcessCaptureClock() - started, limit: limit + 1, identities, signal: controller.signal, diff --git a/src/process/capture/ProcessScenarioRuntimeValidation.test.ts b/src/process/capture/ProcessScenarioRuntimeValidation.test.ts index 1277b753f..7e5cb7e30 100644 --- a/src/process/capture/ProcessScenarioRuntimeValidation.test.ts +++ b/src/process/capture/ProcessScenarioRuntimeValidation.test.ts @@ -17,6 +17,28 @@ import { resolveProcessScenarioRuntimePaths, } from "./ProcessScenarioRuntimeValidation.js"; import { parseProcessScenario } from "../../domain/process/processScenario.js"; +import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js"; + +it.skipIf(process.platform === "win32")( + "resolves a literal backslash in a POSIX executable basename through PATH", + async () => { + const root = await createTestTempDirectory("rea-process-posix-basename-"); + const bin = join(root, "bin"); + await mkdir(bin); + const command = "selected\\node"; + const selected = join(bin, command); + await symlink(process.execPath, selected); + await expect(resolveExecutable(command, root, bin)).resolves.toBe(selected); + const resolved = await resolveProcessScenarioRuntimePaths( + parseProcessScenario({ + executable: command, + working_directory: root, + environment: { PATH: bin }, + }), + ); + expect(resolved.executable).toBe(selected); + }, +); it("preserves the caller-selected executable symlink for process invocation", async () => { const root = await mkdtemp(join(tmpdir(), "rea-process-executable-alias-")); diff --git a/src/process/capture/ProcessScenarioRuntimeValidation.ts b/src/process/capture/ProcessScenarioRuntimeValidation.ts index 1ac9d2228..6d2978b5b 100644 --- a/src/process/capture/ProcessScenarioRuntimeValidation.ts +++ b/src/process/capture/ProcessScenarioRuntimeValidation.ts @@ -1,6 +1,6 @@ import { constants as fsConstants } from "node:fs"; import { access, realpath, stat } from "node:fs/promises"; -import { delimiter, isAbsolute, join, resolve } from "node:path"; +import { delimiter, isAbsolute, join, resolve, sep } from "node:path"; import type { ProcessScenario } from "../../domain/process/processScenario.js"; import { AnalysisInputError } from "../../domain/analysisErrorCore.js"; @@ -71,7 +71,9 @@ export const resolveExecutable = async ( : resolve(workingDirectory, directory), ); const candidates = - isAbsolute(command) || command.includes("/") || command.includes("\\") + isAbsolute(command) || + command.includes("/") || + (sep === "\\" && command.includes("\\")) ? [resolve(workingDirectory, command)] : directories.map((directory) => join(directory, command)); const failures: string[] = []; diff --git a/src/server/createServer.ts b/src/server/createServer.ts index 12ad6418b..c73071bea 100644 --- a/src/server/createServer.ts +++ b/src/server/createServer.ts @@ -9,6 +9,7 @@ import { wasmArtifactModule } from "../composition/wasm.js"; import { registerWasmTools } from "./registerWasmTools.js"; import { snapshotEnvironment } from "../process/snapshotEnvironment.js"; import { createPeResourcesService } from "../composition/binaryDiagnostics.js"; +import type { PeResourcesService } from "../application/binaryDiagnostics/PeResourcesService.js"; import { registerPeResourcesTool } from "./registerPeResourcesTool.js"; import type { EvmInterfaceService } from "../application/evm/EvmInterfaceService.js"; import { evmInterfaceModule } from "../composition/evm.js"; @@ -54,6 +55,7 @@ import { silentLogger } from "../logger.js"; import type { Logger } from "pino"; import { registerApplicationTools } from "./registerApplicationTools.js"; import { registerArtifactTools } from "./registerArtifactTools.js"; +import { registerNativeTools } from "./registerNativeTools.js"; import { registerBrowserScenarioTool } from "./registerBrowserScenarioTool.js"; import { registerBrowserTools } from "./registerBrowserTools.js"; import { registerWebScriptTool } from "./registerWebScriptTool.js"; @@ -84,6 +86,7 @@ import type { FirmwareAnalysisPort } from "../application/firmware/FirmwareAnaly import { createFirmwareAnalysisProvider } from "../composition/firmware.js"; import { registerAndroidTools } from "./registerAndroidTools.js"; import { registerApktoolTools } from "./registerApktoolTools.js"; +import { registerPcapTools } from "./registerPcapTools.js"; import { registerFlutterTools } from "./registerFlutterTools.js"; import { registerJebTools } from "./registerJebTools.js"; import { registerAdbTools } from "./registerAdbTools.js"; @@ -91,11 +94,14 @@ import { AndroidAnalysisService } from "../application/android/AndroidAnalysisSe import { JebAnalysisService } from "../application/jeb/JebAnalysisService.js"; import type { AndroidAnalysisPort } from "../application/android/AndroidAnalysisPort.js"; import type { ApktoolResourceAnalysisPort } from "../application/apktool/ApktoolResourceAnalysisPort.js"; +import type { PcapAnalysisPort } from "../application/pcap/PcapAnalysisPort.js"; +import { PcapAnalysisService } from "../application/pcap/PcapAnalysisService.js"; import { ApktoolResourceAnalysisService } from "../application/apktool/ApktoolResourceAnalysisService.js"; import type { FlutterBuildAnalysisPort } from "../application/flutter/FlutterBuildAnalysisPort.js"; import { FlutterBuildAnalysisService } from "../application/flutter/FlutterBuildAnalysisService.js"; import { createAndroidAnalysisProvider } from "../composition/android.js"; import { createApktoolResourceAnalysisProvider } from "../composition/apktool.js"; +import { createPcapAnalysisProvider } from "../composition/pcap.js"; import { createFlutterBuildAnalysisProvider } from "../composition/flutter.js"; import type { JebAnalysisPort } from "../application/jeb/JebAnalysisPort.js"; import { createJebAnalysisProvider } from "../composition/jeb.js"; @@ -103,8 +109,6 @@ import type { AdbDeviceAnalysisPort } from "../application/adb/AdbDeviceAnalysis import { AdbDeviceAnalysisService } from "../application/adb/AdbDeviceAnalysisService.js"; import { createAdbDeviceAnalysisProvider } from "../composition/adb.js"; import { registerManagedWorkflowTools } from "./registerManagedWorkflowTools.js"; -import { NATIVE_TOOL_CONTRACTS } from "../contracts/native/nativeToolContracts.js"; -import { registerEvidenceTools } from "./registerEvidenceTools.js"; import { registerOfficialTools } from "./registerOfficialTools.js"; import { registerGuidedPrompts } from "./registerPrompts.js"; import { registerSessionTools } from "./registerSessionTools.js"; @@ -146,6 +150,7 @@ export interface CreateServerOptions { readonly webNetworkCapture?: WebNetworkCaptureService; readonly androidAnalysis?: AndroidAnalysisPort; readonly apktoolAnalysis?: ApktoolResourceAnalysisPort; + readonly pcapAnalysis?: PcapAnalysisPort; readonly flutterAnalysis?: FlutterBuildAnalysisPort; readonly jebAnalysis?: JebAnalysisPort; readonly adbDeviceAnalysis?: AdbDeviceAnalysisPort; @@ -276,6 +281,10 @@ export const createServer = ( options.webSourceLocation ?? createWebSourceLocationService(environment); const webModuleTrace = options.webModuleTrace ?? createWebModuleTraceService(environment); + const goBinary = options.goBinary ?? createGoBinaryService(); + const peResources = createPeResourcesService(); + const webNetworkCapture = + options.webNetworkCapture ?? createWebNetworkCaptureService(environment); const javascriptRecovery = options.javascriptRecovery ?? createJavaScriptRecoveryProvider(environment); const artifactResources = new ArtifactResourceScope(); @@ -303,6 +312,7 @@ export const createServer = ( ); const android = options.androidAnalysis ?? createAndroidAnalysisProvider(environment); + const pcap = options.pcapAnalysis ?? createPcapAnalysisProvider(environment); const flutter = options.flutterAnalysis ?? createFlutterBuildAnalysisProvider(); const fridaInstrumentation = @@ -333,12 +343,16 @@ export const createServer = ( jeb.close(), adbDevice.close(), flutter.close(), + pcap.close(), evmInterface.close(), sqliteDatabase.close(), binaryLayout.close(), recordedCrash.close(), webSourceLocation.close(), webModuleTrace.close(), + goBinary.close(), + peResources.close(), + webNetworkCapture.close(), javascriptRecovery.close?.() ?? Promise.resolve(), artifactResources.close(), firmwareAnalysis.close(), @@ -382,6 +396,9 @@ export const createServer = ( recordEvidenceWithUnknown, withAdmittedAnalysis: analysisAdmission, artifactResources, + goBinary, + peResources, + webNetworkCapture, }; registerBinaryAnalysisTools(toolContext); registerFridaTools( @@ -438,12 +455,21 @@ export const createServer = ( recordEvidence, analysisAdmission, ); + registerPcapTools( + server, + new PcapAnalysisService(pcap), + toolLogger, + recordEvidence, + analysisAdmission, + evidenceById, + ); registerFlutterTools( server, new FlutterBuildAnalysisService(flutter), toolLogger, recordEvidence, analysisAdmission, + evidenceById, ); registerApktoolTools( server, @@ -502,6 +528,9 @@ const registerConfiguredAnalysisTools = ( evidenceById, recordEvidence, withAdmittedAnalysis, + goBinary, + peResources, + webNetworkCapture, }: ServerToolContext, android: AndroidAnalysisPort, wasmArtifact: Pick, @@ -515,6 +544,7 @@ const registerConfiguredAnalysisTools = ( toolLogger, recordEvidence, withAdmittedAnalysis, + evidenceById, ); registerJebTools( server, @@ -540,14 +570,14 @@ const registerConfiguredAnalysisTools = ( ); registerPeResourcesTool( server, - createPeResourcesService(), + peResources, toolLogger, recordEvidence, withAdmittedAnalysis, ); registerGoTools( server, - options.goBinary ?? createGoBinaryService(), + goBinary, toolLogger, recordEvidence, withAdmittedAnalysis, @@ -612,10 +642,11 @@ const registerConfiguredAnalysisTools = ( ); registerWebNetworkCaptureTool( server, - options.webNetworkCapture ?? createWebNetworkCaptureService(environment), + webNetworkCapture, toolLogger, recordEvidence, withAdmittedAnalysis, + evidenceById, ); registerJavaScriptRecoveryTool( server, @@ -656,6 +687,9 @@ const createSessionRecorders = ( }); interface ServerToolContext extends ReturnType { + readonly goBinary: GoBinaryService; + readonly peResources: PeResourcesService; + readonly webNetworkCapture: WebNetworkCaptureService; readonly artifactResources: ArtifactResourceScope; readonly server: EvidenceMcpServer; readonly session: BinarySessionPort | undefined; @@ -707,7 +741,7 @@ const registerBinaryAnalysisTools = ({ recordEvidence, withAdmittedAnalysis, }; - registerEvidenceTools(server, NATIVE_TOOL_CONTRACTS, evidenceOptions); + registerNativeTools(server, { ...evidenceOptions, evidenceById }); registerArtifactTools(server, { ...evidenceOptions, evidenceById }); registerManagedTools(server, { ...evidenceOptions, evidenceById }); if (session !== undefined) diff --git a/src/server/registerAnalysisViewTool.ts b/src/server/registerAnalysisViewTool.ts index ebd204edd..d427c0cb4 100644 --- a/src/server/registerAnalysisViewTool.ts +++ b/src/server/registerAnalysisViewTool.ts @@ -7,7 +7,10 @@ import { toolContract } from "../contracts/toolContracts.js"; import type { Logger } from "pino"; import { createEvidenceToolHandler } from "./contractToolHandler.js"; import type { WithAdmittedAnalysis } from "./analysisAdmission.js"; -import { toolRegistrationOptions } from "./toolRegistrationOptions.js"; +import { + toolRegistrationOptions, + applicationValidatedInputSchema, +} from "./toolRegistrationOptions.js"; /** Bind selected-view projection to its named contract and session Evidence owner. */ export const registerAnalysisViewTool = ( @@ -18,6 +21,7 @@ export const registerAnalysisViewTool = ( withAdmittedAnalysis?: WithAdmittedAnalysis, ): void => { const contract = toolContract("inspect_analysis_view"); + const registration = toolRegistrationOptions(contract); const handler = createEvidenceToolHandler( server, (input) => Promise.resolve(inspectAnalysisView(input, evidenceById)), @@ -27,7 +31,10 @@ export const registerAnalysisViewTool = ( ); server.registerTool( contract.name, - toolRegistrationOptions(contract), + { + ...registration, + inputSchema: applicationValidatedInputSchema(registration.inputSchema), + }, handler(contract), ); }; diff --git a/src/server/registerAndroidTools.ts b/src/server/registerAndroidTools.ts index 722bd6695..a8c29a187 100644 --- a/src/server/registerAndroidTools.ts +++ b/src/server/registerAndroidTools.ts @@ -1,6 +1,8 @@ import type { EvidenceMcpServer } from "./EvidenceMcpServer.js"; import type { EvidenceWriter } from "../application/investigation/InvestigationRecordPort.js"; import { AndroidAnalysisService } from "../application/android/AndroidAnalysisService.js"; +import { inspectAndroidPackage } from "../application/android/AndroidPackageInspection.js"; +import type { EvidenceLookup } from "../application/EvidenceInputResolver.js"; import { summarizeRetainedAndroidSearch } from "../application/android/AndroidSearchEvidenceSummary.js"; import { toolContract } from "../contracts/toolContracts.js"; import type { Logger } from "pino"; @@ -18,6 +20,7 @@ export const registerAndroidTools = ( logger: Logger, recordEvidence?: EvidenceWriter["recordEvidence"], withAdmittedAnalysis?: WithAdmittedAnalysis, + evidenceById?: EvidenceLookup, ): void => { const handler = createContractToolHandler( server, @@ -44,10 +47,26 @@ export const registerAndroidTools = ( withAdmittedAnalysis, ); const packageContract = toolContract("inspect_android_package"); + const packageService: ContractToolService<"inspect_android_package"> = { + execute: (_operation, input, options) => + inspectAndroidPackage( + service, + input, + { recordEvidence, evidenceById }, + options, + ), + }; + const packageHandler = createContractToolHandler( + server, + packageService, + logger, + recordEvidence, + withAdmittedAnalysis, + ); server.registerTool( packageContract.name, toolRegistrationOptions(packageContract), - handler(packageContract), + packageHandler(packageContract), ); const searchContract = toolContract("search_android_classes"); server.registerTool( diff --git a/src/server/registerArtifactTools.ts b/src/server/registerArtifactTools.ts index cf95fea41..1217664e6 100644 --- a/src/server/registerArtifactTools.ts +++ b/src/server/registerArtifactTools.ts @@ -6,7 +6,10 @@ import { artifactInspectionInputSchema, } from "../contracts/artifactToolContracts.js"; import type { EvidenceLookup } from "../application/EvidenceInputResolver.js"; -import { summarizeRetainedAnalysis } from "../application/analysisView/AnalysisViewService.js"; +import { + requireAnalysisViewRetention, + summarizeRetainedAnalysis, +} from "../application/analysisView/AnalysisViewService.js"; import { createArtifactExtractionDestination } from "../application/artifacts/ArtifactExtractionDestination.js"; import { ok } from "../domain/result.js"; import type { BinaryTarget } from "../domain/binaryTargetTypes.js"; @@ -25,6 +28,10 @@ export const registerArtifactTools = ( readonly withAdmittedAnalysis: import("./analysisAdmission.js").WithAdmittedAnalysis; }, ): void => { + const retention = { + recordEvidence: options.recordEvidence, + evidenceById: options.evidenceById, + }; registerEvidenceTools(server, ARTIFACT_TOOL_CONTRACTS, { ...options, parametersForExecution: (operation, parameters) => { @@ -37,12 +44,13 @@ export const registerArtifactTools = ( ? { ...parameters, output_root: createArtifactExtractionDestination() } : parameters; }, + preflight: (operation, parameters) => + operation === "inspect_artifact" && parameters.detail === "summary" + ? requireAnalysisViewRetention(operation, retention) + : ok(null), projectEvidence: (operation, parameters, evidence) => operation === "inspect_artifact" && parameters.detail === "summary" - ? summarizeRetainedAnalysis(evidence, { - recordEvidence: options.recordEvidence, - evidenceById: options.evidenceById, - }) + ? summarizeRetainedAnalysis(evidence, retention) : ok(evidence), sourceEvidence: (operation, result) => operation === "inspect_artifact" diff --git a/src/server/registerCaptureTools.ts b/src/server/registerCaptureTools.ts index c097f89be..296faba5c 100644 --- a/src/server/registerCaptureTools.ts +++ b/src/server/registerCaptureTools.ts @@ -28,4 +28,17 @@ export const registerCaptureTools = ( toolRegistrationOptions(contract), handler(contract), ); + const discovery = toolContract("inspect_packet_fields"); + const discover = createEvidenceToolHandler( + server, + (input, options) => service.inspectFields(input, options), + logger, + recordEvidence, + withAdmittedAnalysis, + ); + server.registerTool( + discovery.name, + toolRegistrationOptions(discovery), + discover(discovery), + ); }; diff --git a/src/server/registerEvidenceTools.ts b/src/server/registerEvidenceTools.ts index a8efac6de..b3b2358d1 100644 --- a/src/server/registerEvidenceTools.ts +++ b/src/server/registerEvidenceTools.ts @@ -33,6 +33,16 @@ interface EvidenceToolRegistration { operation: Exclude, parameters: Readonly>, ) => Readonly>; + /** Exclude adapter-only selections, such as delivery detail, from Evidence identity. */ + readonly parametersForEvidence?: ( + operation: Exclude, + parameters: Readonly>, + ) => Readonly>; + /** Refuse an unsatisfiable delivery selection before any analysis runs. */ + readonly preflight?: ( + operation: Exclude, + parameters: Readonly>, + ) => Result; readonly projectEvidence?: ( operation: Exclude, parameters: Readonly>, @@ -51,6 +61,12 @@ export const registerEvidenceTools = ( contract.name, toolRegistrationOptions(contract), async (input, context) => { + const ready = options.preflight?.( + contract.name, + jsonObjectSchema.parse(input), + ); + if (ready !== undefined && !ready.ok) + return server.delivery.toCallToolResult(ready, contract); const admitted = await options.withAdmittedAnalysis( contract.name, context.mcpReq.signal, @@ -95,7 +111,9 @@ export const registerEvidenceTools = ( execution.value.provider, { operation: contract.name, - parameters, + parameters: + options.parametersForEvidence?.(contract.name, parameters) ?? + parameters, result: execution.value.result, ...(execution.value.analysisProfile === undefined ? {} diff --git a/src/server/registerFlutterTools.ts b/src/server/registerFlutterTools.ts index 5b601c5f7..cd57e12a9 100644 --- a/src/server/registerFlutterTools.ts +++ b/src/server/registerFlutterTools.ts @@ -1,10 +1,15 @@ import type { EvidenceMcpServer } from "./EvidenceMcpServer.js"; import type { EvidenceWriter } from "../application/investigation/InvestigationRecordPort.js"; import { FlutterBuildAnalysisService } from "../application/flutter/FlutterBuildAnalysisService.js"; +import { inspectDartAot } from "../application/flutter/DartAotInspection.js"; +import type { EvidenceLookup } from "../application/EvidenceInputResolver.js"; import { toolContract } from "../contracts/toolContracts.js"; import type { WithAdmittedAnalysis } from "./analysisAdmission.js"; import type { Logger } from "pino"; -import { createContractToolHandler } from "./contractToolHandler.js"; +import { + createContractToolHandler, + type ContractToolService, +} from "./contractToolHandler.js"; import { toolRegistrationOptions } from "./toolRegistrationOptions.js"; /** Bind Flutter handlers to their exact named contracts through admission. */ @@ -14,6 +19,7 @@ export const registerFlutterTools = ( logger: Logger, recordEvidence?: EvidenceWriter["recordEvidence"], withAdmittedAnalysis?: WithAdmittedAnalysis, + evidenceById?: EvidenceLookup, ): void => { const handler = createContractToolHandler( server, @@ -22,14 +28,23 @@ export const registerFlutterTools = ( recordEvidence, withAdmittedAnalysis, ); - for (const contract of [ - toolContract("identify_flutter_build"), - toolContract("inspect_dart_aot"), - ] as const) { - server.registerTool( - contract.name, - toolRegistrationOptions(contract), - handler(contract), - ); - } + const identify = toolContract("identify_flutter_build"); + server.registerTool( + identify.name, + toolRegistrationOptions(identify), + handler(identify), + ); + const aotService: ContractToolService<"inspect_dart_aot"> = { + execute: (_operation, input, options) => + inspectDartAot(service, input, { recordEvidence, evidenceById }, options), + }; + const aotHandler = createContractToolHandler( + server, + aotService, + logger, + recordEvidence, + withAdmittedAnalysis, + ); + const aot = toolContract("inspect_dart_aot"); + server.registerTool(aot.name, toolRegistrationOptions(aot), aotHandler(aot)); }; diff --git a/src/server/registerInvestigationTools.ts b/src/server/registerInvestigationTools.ts index d5a2a4706..02e1c52d0 100644 --- a/src/server/registerInvestigationTools.ts +++ b/src/server/registerInvestigationTools.ts @@ -21,7 +21,10 @@ import { RECONSTRUCTION_PROVIDER, STATIC_RUNTIME_PROVIDER, } from "../application/InvestigationProviders.js"; -import { toolRegistrationOptions } from "./toolRegistrationOptions.js"; +import { + applicationValidatedInputSchema, + toolRegistrationOptions, +} from "./toolRegistrationOptions.js"; import { verifyReconstruction } from "../domain/reconstructionVerification.js"; /** Register Evidence-composed differential investigation workflows. */ @@ -52,30 +55,29 @@ const registerChangedBehavior = ( session: BinarySessionPort, contract: ReturnType>, ): void => { + const registration = toolRegistrationOptions(contract); server.registerTool( contract.name, - toolRegistrationOptions(contract), + { + ...registration, + inputSchema: applicationValidatedInputSchema(registration.inputSchema), + }, async (input, context) => { - const closure = evidenceClosure( - session, - comparisonClosure(input.comparisons), - ); - if (!closure.ok) - return server.delivery.toCallToolResult(closure, contract); - const links = closure.value; const computed = await runDerivedOperation(context, contract.name, () => - findChangedBehavior(input.comparisons), + findChangedBehavior(input), ); if (!computed.ok) return server.delivery.toCallToolResult(computed, contract); - const result = computed.value; + const { result, comparisonEvidenceIds } = computed.value; + const closure = evidenceClosure(session, result.evidence_links); + if (!closure.ok) + return server.delivery.toCallToolResult(closure, contract); + const links = closure.value; const evidence = createEvidence(undefined, CHANGED_BEHAVIOR_PROVIDER, { predicateType: "rea.changed-behavior", operation: contract.name, parameters: { - comparison_evidence_ids: input.comparisons.map( - ({ evidence_id: id }) => id, - ), + comparison_evidence_ids: [...comparisonEvidenceIds], }, result: jsonValueSchema.parse(result), confidence: "derived", diff --git a/src/server/registerNativeTools.ts b/src/server/registerNativeTools.ts new file mode 100644 index 000000000..036389d16 --- /dev/null +++ b/src/server/registerNativeTools.ts @@ -0,0 +1,54 @@ +import type { EvidenceMcpServer } from "./EvidenceMcpServer.js"; +import type { EvidenceWriter } from "../application/investigation/InvestigationRecordPort.js"; + +import { NATIVE_TOOL_CONTRACTS } from "../contracts/native/nativeToolContracts.js"; +import type { EvidenceLookup } from "../application/EvidenceInputResolver.js"; +import { + requireAnalysisViewRetention, + summarizeRetainedAnalysis, +} from "../application/analysisView/AnalysisViewService.js"; +import type { JsonValue } from "../domain/jsonValue.js"; +import { ok } from "../domain/result.js"; +import type { BinaryTarget } from "../domain/binaryTargetTypes.js"; +import type { Logger } from "pino"; +import { registerEvidenceTools } from "./registerEvidenceTools.js"; + +/** Delivery detail selects a response shape; it never reaches the tools or Evidence identity. */ +const withoutDelivery = ( + operation: string, + parameters: Readonly>, +): Readonly> => { + if (operation !== "inspect_macho") return parameters; + const { detail: _detail, ...selected } = parameters; + return selected; +}; + +/** Register macOS native contracts; Mach-O inspection can deliver a retained summary. */ +export const registerNativeTools = ( + server: EvidenceMcpServer, + options: { + readonly logger: Logger; + readonly activeTarget: (() => BinaryTarget | undefined) | undefined; + readonly recordEvidence: EvidenceWriter["recordEvidence"] | undefined; + readonly evidenceById?: EvidenceLookup | undefined; + readonly withAdmittedAnalysis: import("./analysisAdmission.js").WithAdmittedAnalysis; + }, +): void => { + const retention = { + recordEvidence: options.recordEvidence, + evidenceById: options.evidenceById, + }; + registerEvidenceTools(server, NATIVE_TOOL_CONTRACTS, { + ...options, + parametersForExecution: withoutDelivery, + parametersForEvidence: withoutDelivery, + preflight: (operation, parameters) => + operation === "inspect_macho" && parameters.detail === "summary" + ? requireAnalysisViewRetention(operation, retention) + : ok(null), + projectEvidence: (operation, parameters, evidence) => + operation === "inspect_macho" && parameters.detail === "summary" + ? summarizeRetainedAnalysis(evidence, retention) + : ok(evidence), + }); +}; diff --git a/src/server/registerPcapTools.ts b/src/server/registerPcapTools.ts new file mode 100644 index 000000000..b2ebd239a --- /dev/null +++ b/src/server/registerPcapTools.ts @@ -0,0 +1,83 @@ +import type { EvidenceMcpServer } from "./EvidenceMcpServer.js"; +import type { EvidenceWriter } from "../application/investigation/InvestigationRecordPort.js"; +import { PcapAnalysisService } from "../application/pcap/PcapAnalysisService.js"; +import { toolContract } from "../contracts/toolContracts.js"; +import type { WithAdmittedAnalysis } from "./analysisAdmission.js"; +import type { Logger } from "pino"; +import { + createContractToolHandler, + createEvidenceToolHandler, +} from "./contractToolHandler.js"; +import { + toolRegistrationOptions, + applicationValidatedInputSchema, +} from "./toolRegistrationOptions.js"; +import type { EvidenceLookup } from "../application/EvidenceInputResolver.js"; +import { + requireAnalysisViewRetention, + summarizeRetainedAnalysis, +} from "../application/analysisView/AnalysisViewService.js"; +import { analysisInputErrorFromIssues } from "../domain/inputIssueProjection.js"; +import { err } from "../domain/result.js"; + +/** Bind pcap handlers to their exact named contracts through admission. */ +export const registerPcapTools = ( + server: EvidenceMcpServer, + service: PcapAnalysisService, + logger: Logger, + recordEvidence?: EvidenceWriter["recordEvidence"], + withAdmittedAnalysis?: WithAdmittedAnalysis, + evidenceById?: EvidenceLookup, +): void => { + const handler = createContractToolHandler( + server, + service, + logger, + recordEvidence, + withAdmittedAnalysis, + ); + const clientContract = toolContract("inspect_pcap_client"); + server.registerTool( + clientContract.name, + toolRegistrationOptions(clientContract), + handler(clientContract), + ); + const contract = toolContract("inspect_pcap_capture"); + const registration = toolRegistrationOptions(contract); + const capture = createEvidenceToolHandler( + server, + async (rawInput, options) => { + const parsed = contract.inputSchema.safeParse(rawInput); + if (!parsed.success) + return err( + analysisInputErrorFromIssues( + contract.name, + parsed.error.issues, + rawInput, + { cause: parsed.error }, + ), + ); + const { detail, ...input } = parsed.data; + const retention = { recordEvidence, evidenceById }; + if (detail === "summary") { + const ready = requireAnalysisViewRetention(contract.name, retention); + if (!ready.ok) return ready; + } + const result = await service.execute(contract.name, input, options); + return detail === "summary" && result.ok + ? summarizeRetainedAnalysis(result.value, retention) + : result; + }, + logger, + recordEvidence, + withAdmittedAnalysis, + ); + server.registerTool( + contract.name, + { + ...registration, + inputSchema: applicationValidatedInputSchema(registration.inputSchema), + }, + capture(contract), + ); +}; diff --git a/src/server/registerSessionRecordTools.ts b/src/server/registerSessionRecordTools.ts index a41d60cac..68daab23b 100644 --- a/src/server/registerSessionRecordTools.ts +++ b/src/server/registerSessionRecordTools.ts @@ -12,6 +12,8 @@ import { err, ok } from "../domain/result.js"; import { AnalysisCancelledError } from "../domain/analysisErrorCore.js"; import { inspectEvidenceBundle } from "../application/investigation/InspectEvidenceBundle.js"; import { toolRegistrationOptions } from "./toolRegistrationOptions.js"; +import { withAdmittedAnalysis } from "./analysisAdmission.js"; +import { runAdmittedToolOperation } from "./admittedToolOperation.js"; interface EvidenceToolRegistration { readonly server: EvidenceMcpServer; @@ -92,40 +94,54 @@ const registerImportEvidenceTool = ({ resources, importContract, }: EvidenceToolRegistration): void => { + const admission = withAdmittedAnalysis({ kind: "session", session }); server.registerTool( importContract.name, toolRegistrationOptions(importContract), - async (input, context) => { - const path = input.path; - const signal = context.mcpReq.signal; - const loaded = await readImmutableEvidenceBundle(path, signal, resources); - if (!loaded.ok) - return server.delivery.toCallToolResult(loaded, importContract); - const retainedUnknownRevisions = new Set( - bundleForSerialization(session).unknowns.map((unknown) => - unknownRevisionKey(unknown), - ), - ); - if (signal.aborted) - return server.delivery.toCallToolResult( - err(new AnalysisCancelledError("import_evidence_bundle")), - importContract, - ); - const imported = session.importEvidenceBundle(loaded.value); - return imported.ok - ? server.delivery.toCallToolResult( - ok({ - imported: imported.value, - unknowns_added: loaded.value.unknowns.filter( - (unknown) => - !retainedUnknownRevisions.has(unknownRevisionKey(unknown)), - ).length, - total: bundleForSerialization(session).records.length, - }), - importContract, - ) - : server.delivery.toCallToolResult(imported, importContract); - }, + async (input, context) => + runAdmittedToolOperation( + server, + admission, + importContract.name, + context.mcpReq.signal, + async () => { + const path = input.path; + const signal = context.mcpReq.signal; + const loaded = await readImmutableEvidenceBundle( + path, + signal, + resources, + ); + if (!loaded.ok) + return server.delivery.toCallToolResult(loaded, importContract); + const retainedUnknownRevisions = new Set( + bundleForSerialization(session).unknowns.map((unknown) => + unknownRevisionKey(unknown), + ), + ); + if (signal.aborted) + return server.delivery.toCallToolResult( + err(new AnalysisCancelledError("import_evidence_bundle")), + importContract, + ); + const imported = session.importEvidenceBundle(loaded.value); + return imported.ok + ? server.delivery.toCallToolResult( + ok({ + imported: imported.value, + unknowns_added: loaded.value.unknowns.filter( + (unknown) => + !retainedUnknownRevisions.has( + unknownRevisionKey(unknown), + ), + ).length, + total: bundleForSerialization(session).records.length, + }), + importContract, + ) + : server.delivery.toCallToolResult(imported, importContract); + }, + ), ); }; diff --git a/src/server/registerSignalTools.ts b/src/server/registerSignalTools.ts index 87a67361f..f833ffbd5 100644 --- a/src/server/registerSignalTools.ts +++ b/src/server/registerSignalTools.ts @@ -28,4 +28,17 @@ export const registerSignalTools = ( toolRegistrationOptions(contract), handler(contract), ); + const discovery = toolContract("inspect_signal_decoders"); + const discover = createEvidenceToolHandler( + server, + (input, options) => service.inspectDecoders(input, options), + logger, + recordEvidence, + withAdmittedAnalysis, + ); + server.registerTool( + discovery.name, + toolRegistrationOptions(discovery), + discover(discovery), + ); }; diff --git a/src/server/registerWebNetworkCaptureTool.ts b/src/server/registerWebNetworkCaptureTool.ts index 9f0c9dec8..3e9759504 100644 --- a/src/server/registerWebNetworkCaptureTool.ts +++ b/src/server/registerWebNetworkCaptureTool.ts @@ -1,12 +1,23 @@ import type { EvidenceMcpServer } from "./EvidenceMcpServer.js"; -import type { StandardSchemaWithJSON } from "@modelcontextprotocol/server"; +import type { EvidenceLookup } from "../application/EvidenceInputResolver.js"; +import { + requireAnalysisViewRetention, + summarizeRetainedAnalysis, +} from "../application/analysisView/AnalysisViewService.js"; +import { analysisInputErrorFromIssues } from "../domain/inputIssueProjection.js"; +import { redactExplicitFailure } from "../domain/explicitSensitiveFailure.js"; +import { captureSensitiveValues } from "../domain/webNetworkCapture.js"; +import { err } from "../domain/result.js"; import type { WebNetworkCaptureService } from "../application/WebNetworkCaptureService.js"; import type { EvidenceWriter } from "../application/investigation/InvestigationRecordPort.js"; import { toolContract } from "../contracts/toolContracts.js"; import type { Logger } from "pino"; import type { WithAdmittedAnalysis } from "./analysisAdmission.js"; import { createEvidenceToolHandler } from "./contractToolHandler.js"; -import { toolRegistrationOptions } from "./toolRegistrationOptions.js"; +import { + toolRegistrationOptions, + applicationValidatedInputSchema, +} from "./toolRegistrationOptions.js"; /** Bind historical inspection to its named contract and caller-owned Evidence writer. */ export const registerWebNetworkCaptureTool = ( @@ -15,22 +26,41 @@ export const registerWebNetworkCaptureTool = ( logger: Logger, recordEvidence?: EvidenceWriter["recordEvidence"], withAdmittedAnalysis?: WithAdmittedAnalysis, + evidenceById?: EvidenceLookup, ): void => { const contract = toolContract("inspect_web_network_capture"); const registration = toolRegistrationOptions(contract); // Keep the exact advertised contract; the shared service validates raw input // before effects so caller-declared sensitivity also covers invalid arguments. // SDK pre-validation otherwise emits unprojected issue paths as plain text. - const inputSchema: StandardSchemaWithJSON = { - "~standard": { - ...registration.inputSchema["~standard"], - vendor: "rea-application", - validate: (value: unknown) => ({ value }), - }, - }; + const inputSchema = applicationValidatedInputSchema(registration.inputSchema); const handler = createEvidenceToolHandler( server, - (input, options) => service.inspect(input, options), + async (rawInput, options) => { + const parsed = contract.inputSchema.safeParse(rawInput); + if (!parsed.success) + return err( + redactExplicitFailure( + analysisInputErrorFromIssues( + contract.name, + parsed.error.issues, + rawInput, + { cause: parsed.error }, + ), + captureSensitiveValues(rawInput), + ), + ); + const { detail, ...input } = parsed.data; + const retention = { recordEvidence, evidenceById }; + if (detail === "summary") { + const ready = requireAnalysisViewRetention(contract.name, retention); + if (!ready.ok) return ready; + } + const inspected = await service.inspect(input, options); + return detail === "summary" && inspected.ok + ? summarizeRetainedAnalysis(inspected.value, retention) + : inspected; + }, logger, recordEvidence, withAdmittedAnalysis, diff --git a/src/server/toolRegistrationOptions.ts b/src/server/toolRegistrationOptions.ts index 4615971d4..c943be3f2 100644 --- a/src/server/toolRegistrationOptions.ts +++ b/src/server/toolRegistrationOptions.ts @@ -1,9 +1,21 @@ import type { ToolContract } from "../contracts/toolContractTypes.js"; +import type { StandardSchemaWithJSON } from "@modelcontextprotocol/server"; import { toolInputSchemaWithMetadata, toolOutputSchemaWithMetadata, } from "../contracts/toolSchemaMetadata.js"; +/** Preserve advertised contracts while the application owns structured input failures. */ +export const applicationValidatedInputSchema = ( + schema: StandardSchemaWithJSON, +): StandardSchemaWithJSON => ({ + "~standard": { + ...schema["~standard"], + vendor: "rea-application", + validate: (value: unknown) => ({ value }), + }, +}); + /** Project the canonical Zod contracts directly into SDK registration. */ export const toolRegistrationOptions = ( contract: Contract, diff --git a/src/signals/SigrokRecordingProvider.ts b/src/signals/SigrokRecordingProvider.ts index b0eb5c4ba..2e903d4c6 100644 --- a/src/signals/SigrokRecordingProvider.ts +++ b/src/signals/SigrokRecordingProvider.ts @@ -1,3 +1,11 @@ +import { + type InspectSignalDecodersInput, + signalDecodersSchema, +} from "../domain/signals/signalDecoders.js"; +import { + DecoderDiscoveryBudgetError, + parseDecoderDiscovery, +} from "./parseDecoderDiscovery.js"; import { readLinuxResourceLimits, type LinuxResourceLimits, @@ -81,7 +89,7 @@ export class SigrokRecordingProvider implements SignalRecordingPort { readLimits, { message: - "A signal decoding operation is already active; retry after it completes.", + "A sigrok operation is already active; retry after it completes.", details: { max_concurrent_decodes: 1 }, }, ); @@ -91,17 +99,28 @@ export class SigrokRecordingProvider implements SignalRecordingPort { options?: ExecutionOptions, ): Promise> { return this.#resources.run( - () => this.#decode(input, options), + () => this.#executeRequest(input, options), options?.signal, ); } + inspectDecoders( + input: InspectSignalDecodersInput, + options?: ExecutionOptions, + ): Promise> { + return this.#resources.run( + () => this.#executeRequest(input, options), + options?.signal, + "inspect_signal_decoders", + ); + } close(): Promise { return this.#resources.close(); } - async #decode( - input: DecodeSignalRecordingInput, + async #executeRequest( + input: DecodeSignalRecordingInput | InspectSignalDecodersInput, options?: ExecutionOptions, ): Promise> { + const operation = "path" in input ? OPERATION : "inspect_signal_decoders"; let phase: | "configuration" | "artifact" @@ -109,12 +128,12 @@ export class SigrokRecordingProvider implements SignalRecordingPort { | "producer" | "integrity" = "configuration"; let capturedOutput: AnalysisCapturedOutput | undefined; - let resourcePath = input.path; + let resourcePath = "path" in input ? input.path : "sigrok decoder catalog"; let effectiveLimits: LinuxResourceLimits | undefined; let root: ArtifactAnalysisRuntime | undefined; let result: Result; try { - if (options?.signal?.aborted) throw new AnalysisCancelledError(OPERATION); + if (options?.signal?.aborted) throw new AnalysisCancelledError(operation); await this.#resources.release(); if (this.platform !== "linux") throw new Error( @@ -154,7 +173,7 @@ export class SigrokRecordingProvider implements SignalRecordingPort { ]; phase = "workspace"; root = await this.createRuntime(); - this.#resources.retain(root, undefined); + this.#resources.retain(root, undefined, operation); const snapshotPath = join(root.path, "recording"); const execute = async (args: readonly string[]) => { const output = await runOwnedCommand( @@ -197,24 +216,24 @@ export class SigrokRecordingProvider implements SignalRecordingPort { ); if (currentLimiter.sha256 !== limiterIdentity.sha256) throw new AnalysisArtifactChangedError( - OPERATION, + operation, limiter, "Configured resource limiter changed during analysis.", { capturedOutput }, ); if (current.sha256 !== tool.sha256) throw new AnalysisArtifactChangedError( - OPERATION, + operation, command, "Configured decoder executable changed during analysis.", { capturedOutput }, ); phase = previousPhase; - resourcePath = input.path; + resourcePath = "path" in input ? input.path : command; if (output.exitCode !== 0) throw new ProviderAdapterError( SIGROK_PROVIDER_IDENTITY.id, - OPERATION, + operation, { diagnostics: { exit_code: output.exitCode ?? null, @@ -234,65 +253,15 @@ export class SigrokRecordingProvider implements SignalRecordingPort { throw new Error( "Expected sigrok-cli 0.7.2 with libsigrokdecode; inspect the selected installation.", ); - phase = "artifact"; - const selected = await this.#resources.read( - input.path, - LIMITS.input_bytes, - options?.signal, - ); - phase = "workspace"; - await writeFile(snapshotPath, selected.bytes, { - flag: "wx", - mode: 0o400, - }); - phase = "producer"; - const args = [ - "--dont-scan", - "--input-file=recording", - `--protocol-decoders=${input.decoders}`, - "--protocol-decoder-samplenum", - ...(input.input_format === null - ? [] - : [`--input-format=${input.input_format}`]), - ...(input.annotations === null - ? [] - : [`--protocol-decoder-annotations=${input.annotations}`]), - ]; - const output = await execute(args); - phase = "integrity"; - resourcePath = snapshotPath; - if ( - (await this.#resources.read(snapshotPath, LIMITS.input_bytes)) - .sha256 !== selected.sha256 - ) - throw new AnalysisArtifactChangedError( - OPERATION, - snapshotPath, - "Decoder changed the private recording copy.", - capturedOutput === undefined ? {} : { capturedOutput }, - ); - phase = "producer"; - resourcePath = input.path; - const report = signalRecordingSchema.parse({ - artifact: { - path: input.path, - sha256: selected.sha256, - bytes: selected.bytes.length, - }, - selection: input, - ...windowAnnotationText( - output.stdout.text, - input.line_offset, - input.line_limit, - ), - diagnostics: output.stderr.text, + const metadata = (args: readonly string[], diagnostics: string) => ({ + diagnostics, tool: { configured_path: configured, path: command, sha256: tool.sha256, version_banner: version.stdout.text, version_diagnostics: version.stderr.text, - arguments: args, + arguments: [...args], }, resource_limiter: { configured_path: configuredLimiter, @@ -301,31 +270,108 @@ export class SigrokRecordingProvider implements SignalRecordingPort { arguments: limitArguments, }, limits: { ...LIMITS, ...effectiveLimits }, - hardware_acquisition: "not-performed", + hardware_acquisition: "not-performed" as const, }); - result = ok( - createAnalysisExecution(report, SIGROK_PROVIDER_IDENTITY, { - subject: { + if (!("path" in input)) { + phase = "producer"; + const args = + input.decoder === null + ? ["--dont-scan", "--list-supported"] + : ["--dont-scan", `--protocol-decoders=${input.decoder}`, "--show"]; + const output = await execute(args); + const report = signalDecodersSchema.parse({ + selection: input, + ...parseDecoderDiscovery(output.stdout.text, input), + ...metadata(args, output.stderr.text), + }); + result = ok( + createAnalysisExecution(report, SIGROK_PROVIDER_IDENTITY, { + subject: { path: command, sha256: tool.sha256, format: "file" }, + limitations: [ + "Catalog and help are observed from the selected sigrok installation. Help lines preserve producer descriptions, not a separately validated parameter schema.", + "Each request probes the installation again; pages do not share a frozen catalog. Decoder modules and libraries are trusted and not fully fingerprinted.", + "Hardware discovery, recording reads and acquisition are not requested. Linux per-process limits and bounded output apply; this is not an OS sandbox.", + ], + }), + ); + } else { + phase = "artifact"; + const selected = await this.#resources.read( + input.path, + LIMITS.input_bytes, + options?.signal, + ); + phase = "workspace"; + await writeFile(snapshotPath, selected.bytes, { + flag: "wx", + mode: 0o400, + }); + phase = "producer"; + const args = [ + "--dont-scan", + "--input-file=recording", + `--protocol-decoders=${input.decoders}`, + "--protocol-decoder-samplenum", + ...(input.input_format === null + ? [] + : [`--input-format=${input.input_format}`]), + ...(input.annotations === null + ? [] + : [`--protocol-decoder-annotations=${input.annotations}`]), + ]; + const output = await execute(args); + phase = "integrity"; + resourcePath = snapshotPath; + if ( + (await this.#resources.read(snapshotPath, LIMITS.input_bytes)) + .sha256 !== selected.sha256 + ) + throw new AnalysisArtifactChangedError( + operation, + snapshotPath, + "Decoder changed the private recording copy.", + capturedOutput === undefined ? {} : { capturedOutput }, + ); + phase = "producer"; + resourcePath = "path" in input ? input.path : command; + const report = signalRecordingSchema.parse({ + artifact: { path: input.path, sha256: selected.sha256, - format: "file", + bytes: selected.bytes.length, }, - locations: [{ kind: "artifact-path", path: input.path }], - limitations: [ - "Annotations are the selected decoder's interpretation of the recorded samples and caller-selected settings, not independently verified protocol semantics. Empty output does not establish absence of traffic.", - "Sample-number ranges and display text are preserved verbatim, not converted to timestamps or parsed as a lossless byte stream. All producer stderr is retained, including warnings on exit zero.", - "Only a private copy of the selected file is decoded. Hardware discovery/acquisition and target execution are not requested. Caller-supplied tools, plugins and decoder modules are trusted; this is not an OS sandbox.", - "Linux prlimit enforces soft and hard ceilings before execution, preserving stricter inherited limits: at most 1 GiB address space per process, 64 MiB per file, 30 CPU seconds and no core dumps. Effective limits are reported. Resident input is capped at 32 MiB, each command at 30 wall-clock seconds and combined output at 4 MiB. Output overflow fails. These are per-process/per-file limits, not aggregate memory or filesystem quotas; caller-supplied tools and the limiter are trusted.", - ], - }), - ); + selection: input, + ...windowAnnotationText( + output.stdout.text, + input.line_offset, + input.line_limit, + ), + ...metadata(args, output.stderr.text), + }); + result = ok( + createAnalysisExecution(report, SIGROK_PROVIDER_IDENTITY, { + subject: { + path: input.path, + sha256: selected.sha256, + format: "file", + }, + locations: [{ kind: "artifact-path", path: input.path }], + limitations: [ + "Annotations are the selected decoder's interpretation of the recorded samples and caller-selected settings, not independently verified protocol semantics. Empty output does not establish absence of traffic.", + "Sample-number ranges and display text are preserved verbatim, not converted to timestamps or parsed as a lossless byte stream. All producer stderr is retained, including warnings on exit zero.", + "Only a private copy of the selected file is decoded. Hardware discovery/acquisition and target execution are not requested. Caller-supplied tools, plugins and decoder modules are trusted; this is not an OS sandbox.", + "Linux prlimit enforces soft and hard ceilings before execution, preserving stricter inherited limits: at most 1 GiB address space per process, 64 MiB per file, 30 CPU seconds and no core dumps. Effective limits are reported. Resident input is capped at 32 MiB, each command at 30 wall-clock seconds and combined output at 4 MiB. Output overflow fails. These are per-process/per-file limits, not aggregate memory or filesystem quotas; caller-supplied tools and the limiter are trusted.", + ], + }), + ); + } } catch (cause) { if ( root !== undefined && cause instanceof OwnedCommandFailure && cause.cleanupOwner !== undefined ) - this.#resources.retain(root, cause.cleanupOwner); + this.#resources.retain(root, cause.cleanupOwner, operation); result = err( this.#failure( cause, @@ -334,18 +380,22 @@ export class SigrokRecordingProvider implements SignalRecordingPort { root?.path ?? null, effectiveLimits, capturedOutput, + operation, ), ); } - return this.#resources.finish(result, (execution) => - createEvidence(execution.subject ?? undefined, execution.provider, { - operation: OPERATION, - parameters: input, - result: execution.result, - confidence: "observed", - locations: execution.locations, - limitations: execution.limitations, - }), + return this.#resources.finish( + result, + (execution) => + createEvidence(execution.subject ?? undefined, execution.provider, { + operation, + parameters: input, + result: execution.result, + confidence: "observed", + locations: execution.locations, + limitations: execution.limitations, + }), + operation, ); } #failure( @@ -360,6 +410,7 @@ export class SigrokRecordingProvider implements SignalRecordingPort { workspacePath: string | null, effectiveLimits: LinuxResourceLimits | undefined, observedOutput: AnalysisCapturedOutput | undefined, + operation = OPERATION, ): AnalysisError { if (cause instanceof AnalysisError) return cause; if (cause instanceof ArtifactReaderFailure && cause.cleanup !== undefined) @@ -367,15 +418,34 @@ export class SigrokRecordingProvider implements SignalRecordingPort { cause.cleanup, cause, observedOutput, + operation, ); if ( (cause instanceof Error && cause.name === "AbortError") || (cause instanceof ArtifactReaderFailure && cause.reason === "cancelled") ) - return new AnalysisCancelledError(OPERATION, { cause }); + return new AnalysisCancelledError(operation, { cause }); + if (cause instanceof DecoderDiscoveryBudgetError) + return new AnalysisResourceConstraintError( + operation, + "memory", + cause.message, + { line_budget: 65536 }, + { + cause, + ...(observedOutput === undefined + ? {} + : { capturedOutput: observedOutput }), + }, + ); const reason = cause instanceof Error ? cause.message : String(cause); if (cause instanceof OwnedCommandFailure) - return this.#resources.commandFailure(cause, LIMITS, effectiveLimits); + return this.#resources.commandFailure( + cause, + LIMITS, + effectiveLimits, + operation, + ); if (phase === "workspace") { const code = cause instanceof Error && @@ -385,13 +455,13 @@ export class SigrokRecordingProvider implements SignalRecordingPort { : null; if (code === "ENOSPC" || code === "EDQUOT" || code === "EFBIG") return new AnalysisResourceConstraintError( - OPERATION, + operation, "file-size", `Private analysis workspace failed (${code}): ${reason}`, { workspace_path: workspacePath, system_code: code }, { cause }, ); - return new ProviderAdapterError(SIGROK_PROVIDER_IDENTITY.id, OPERATION, { + return new ProviderAdapterError(SIGROK_PROVIDER_IDENTITY.id, operation, { cause, userMessage: `Cannot prepare the private analysis workspace: ${reason}`, diagnostics: { @@ -414,12 +484,12 @@ export class SigrokRecordingProvider implements SignalRecordingPort { ["limit", "integrity", "path"].includes(cause.reason) ) return new AnalysisArtifactChangedError( - OPERATION, + operation, path, `Post-run identity verification failed for ${path}: ${reason}`, detail, ); - return new ProviderAdapterError(SIGROK_PROVIDER_IDENTITY.id, OPERATION, { + return new ProviderAdapterError(SIGROK_PROVIDER_IDENTITY.id, operation, { ...detail, diagnostics: { stage: "integrity", resource_path: path, reason }, }); @@ -427,10 +497,13 @@ export class SigrokRecordingProvider implements SignalRecordingPort { if (phase === "configuration") return new AnalysisCapabilityUnavailableError( SIGROK_PROVIDER_IDENTITY.id, - OPERATION, + operation, reason, { cause, + ...(observedOutput === undefined + ? {} + : { capturedOutput: observedOutput }), userMessage: `${reason} On Linux, configure sigrok-cli 0.7.2 through REA_SIGROK_COMMAND and util-linux prlimit through REA_SIGROK_PRLIMIT_COMMAND (default /usr/bin/prlimit).`, }, ); @@ -440,14 +513,14 @@ export class SigrokRecordingProvider implements SignalRecordingPort { cause.reason === "limit" ) return new AnalysisResourceConstraintError( - OPERATION, + operation, "file-size", reason, { input_bytes: LIMITS.input_bytes }, { cause }, ); if (cause instanceof ArtifactReaderFailure && cause.reason === "integrity") - return new AnalysisArtifactChangedError(OPERATION, path, reason, { + return new AnalysisArtifactChangedError(operation, path, reason, { cause, }); if ( @@ -456,13 +529,18 @@ export class SigrokRecordingProvider implements SignalRecordingPort { "code" in cause && (cause.code === "EACCES" || cause.code === "EPERM") ) - return new AnalysisAccessDeniedError(OPERATION, path, cause.code, { + return new AnalysisAccessDeniedError(operation, path, cause.code, { cause, }); if (phase === "artifact") - return new AnalysisInputError(OPERATION, { cause }, [ + return new AnalysisInputError(operation, { cause }, [ { path: ["path"], reason: "invalid_value", message: reason }, ]); - return new AnalysisOutputError(OPERATION, reason, { cause }); + return new AnalysisOutputError(operation, reason, { + cause, + ...(observedOutput === undefined + ? {} + : { capturedOutput: observedOutput }), + }); } } diff --git a/src/signals/parseDecoderDiscovery.ts b/src/signals/parseDecoderDiscovery.ts new file mode 100644 index 000000000..230abe2b0 --- /dev/null +++ b/src/signals/parseDecoderDiscovery.ts @@ -0,0 +1,79 @@ +import type { InspectSignalDecodersInput } from "../domain/signals/signalDecoders.js"; + +export class DecoderDiscoveryBudgetError extends Error {} + +const copyText = (value: string): string => + Buffer.from(value, "utf16le").toString("utf16le"); + +/** Parse only the 0.7.2 catalog grammar; help remains literal producer text. */ +export const parseDecoderDiscovery = ( + text: string, + input: InspectSignalDecodersInput, +) => { + // Bound record-object expansion before splitting the already byte-bounded output. + // 65,536 records leaves an 8 MiB budget at 128 bytes of bookkeeping per record. + let separators = 0; + for (const character of text) + if (character === "\n" && ++separators >= 65536) + throw new DecoderDiscoveryBudgetError( + "Decoder discovery exceeds the 65,536-line representation budget", + ); + const lines = text.split(/\r?\n/u); + if (lines.at(-1) === "") lines.pop(); + const query = input.query.toLowerCase(); + const decoders: { id: string; name: string }[] = []; + const help: { line_number: number; text: string }[] = []; + if (input.decoder === null) { + const start = lines.indexOf("Supported protocol decoders:"); + if (start < 0) throw new Error("sigrok decoder catalog heading is missing"); + const seen = new Set(); + for (const line of lines.slice(start + 1)) { + if (line === "") continue; + const match = /^ {2}([a-zA-Z0-9_+-]+) +(.+)$/u.exec(line); + if ( + match?.[1] === undefined || + match[2] === undefined || + seen.has(match[1]) + ) + throw new Error("Malformed or duplicate sigrok decoder catalog record"); + seen.add(match[1]); + decoders.push({ id: match[1], name: match[2] }); + } + if (decoders.length === 0) + throw new Error("sigrok returned an empty decoder catalog"); + } else { + if (lines[0] !== `ID: ${input.decoder}`) + throw new Error("Decoder help does not identify the selected decoder"); + lines.forEach((line, i) => help.push({ line_number: i + 1, text: line })); + } + const catalog = input.decoder === null; + const matching = catalog + ? decoders.filter(({ id, name }) => + `${id} ${name}`.toLowerCase().includes(query), + ) + : help.filter(({ text: line }) => line.toLowerCase().includes(query)); + const start = Math.min(input.offset, matching.length); + const count = Math.min( + input.limit ?? matching.length, + matching.length - start, + ); + // Keep retained pages independent of the complete producer-output allocation. + const page = matching + .slice(start, start + count) + .map((record) => + "id" in record + ? { id: copyText(record.id), name: copyText(record.name) } + : { line_number: record.line_number, text: copyText(record.text) }, + ); + return { + decoders: catalog ? page : [], + help_lines: catalog ? [] : page, + output_window: { + unit: catalog ? "decoders" : "help-lines", + examined: catalog ? decoders.length : help.length, + matching: matching.length, + returned: page.length, + has_more: start + count < matching.length, + }, + }; +}; diff --git a/src/sqlite/SqliteDatabaseInspection.ts b/src/sqlite/SqliteDatabaseInspection.ts index f5f92bdf4..510fbaafc 100644 --- a/src/sqlite/SqliteDatabaseInspection.ts +++ b/src/sqlite/SqliteDatabaseInspection.ts @@ -394,7 +394,7 @@ const readRows = ( for (const name of columns) if (!available.has(name)) throw new SqliteInspectionFailure( - "selection", + "column-selection", `Selected column does not exist with that exact name in table ${input.table}: ${name}`, ); const projection = columns diff --git a/src/sqlite/SqliteDatabaseLimits.ts b/src/sqlite/SqliteDatabaseLimits.ts index 99df7ed9d..408fc4bc3 100644 --- a/src/sqlite/SqliteDatabaseLimits.ts +++ b/src/sqlite/SqliteDatabaseLimits.ts @@ -12,6 +12,7 @@ export class SqliteInspectionFailure extends Error { readonly reason: | "format" | "selection" + | "column-selection" | "unsupported" | "unavailable" | "output-limit" diff --git a/src/sqlite/SqliteDatabaseWorkerExecution.ts b/src/sqlite/SqliteDatabaseWorkerExecution.ts index 9474a6d27..cba180399 100644 --- a/src/sqlite/SqliteDatabaseWorkerExecution.ts +++ b/src/sqlite/SqliteDatabaseWorkerExecution.ts @@ -36,6 +36,7 @@ const replySchema = z.discriminatedUnion("ok", [ reason: z.enum([ "format", "selection", + "column-selection", "unsupported", "unavailable", "output-limit", @@ -49,6 +50,14 @@ export type SqliteWorkerLauncher = NonNullable< Parameters[2] >["launcher"]; +// Each input rejection names the caller argument that requires correction. +const rejectedArgument = { + format: "path", + unsupported: "path", + selection: "table", + "column-selection": "columns", +} as const; + const rejectedReply = ( reply: z.output & { ok: false }, capturedOutput: AnalysisCapturedOutput, @@ -83,7 +92,7 @@ const rejectedReply = ( ); return new AnalysisInputError(operation, { capturedOutput }, [ { - path: [reply.reason === "selection" ? "table" : "path"], + path: [rejectedArgument[reply.reason]], reason: "invalid_format", message: reply.message, }, diff --git a/src/wasm/WabtArtifactProvider.test.ts b/src/wasm/WabtArtifactProvider.test.ts index 74d481de3..428cf8cbd 100644 --- a/src/wasm/WabtArtifactProvider.test.ts +++ b/src/wasm/WabtArtifactProvider.test.ts @@ -1,14 +1,40 @@ import { access, chmod, readFile, writeFile } from "node:fs/promises"; import { join } from "node:path"; -import { expect, it } from "vitest"; +import { expect, it, vi } from "vitest"; import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js"; import { WabtArtifactProvider } from "./WabtArtifactProvider.js"; -import { - PrivateRuntimeRoot, - PrivateRuntimeRootUnavailableError, -} from "../process/PrivateRuntimeRoot.js"; +import { PrivateRuntimeRoot } from "../process/PrivateRuntimeRoot.js"; +import { PrivateRuntimeRootUnavailableError } from "../process/PrivateRuntimeRootUnavailableError.js"; import { spawnOwnedProviderProcess } from "../process/ProviderProcess.js"; import { OwnedCommandFailure } from "../process/OwnedCommand.js"; +import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; + +const fileCloseFailure = vi.hoisted(() => ({ + selectedPath: "", + attempts: 0, + failFirst: false, +})); + +vi.mock("node:fs/promises", async (importOriginal) => { + const fs = await importOriginal(); + return { + ...fs, + open: async (...args: Parameters) => { + const handle = await fs.open(...args); + if (String(args[0]) !== fileCloseFailure.selectedPath) return handle; + const close = handle.close.bind(handle); + handle.close = async () => { + fileCloseFailure.attempts += 1; + if (fileCloseFailure.failFirst) { + fileCloseFailure.failFirst = false; + throw new Error("injected WABT input close failure"); + } + return close(); + }; + return handle; + }, + }; +}); const failureTags: Readonly> = { invalid: "AnalysisInputError", @@ -21,6 +47,82 @@ const failureTags: Readonly> = { "wat-warning": "AnalysisOutputError", }; +const assertCleanupRecovery = async (options: { + readonly provider: WabtArtifactProvider; + readonly result: Awaited>; + readonly path: string; + readonly runtimePath: string; + readonly cleanupAttempts: () => number; + readonly runtimeCreations: () => number; +}): Promise => { + expect(options.result).toMatchObject({ + error: { + cleanupIncomplete: true, + cleanupResources: [options.runtimePath], + partialObservation: { normalized_result: { validation: "valid" } }, + }, + }); + await expect(access(options.runtimePath)).resolves.toBeUndefined(); + const blocked = await options.provider.inspect({ + path: options.path, + glue_paths: [], + candidate_paths: [], + }); + expect(blocked).toMatchObject({ + ok: false, + error: { + cleanupIncomplete: true, + cleanupResources: [options.runtimePath], + }, + }); + expect(options.cleanupAttempts()).toBe(2); + expect(options.runtimeCreations()).toBe(1); + await options.provider.close(); + expect(options.cleanupAttempts()).toBe(3); + await expect(access(options.runtimePath)).rejects.toMatchObject({ + code: "ENOENT", + }); +}; + +const assertModeOutcome = ( + mode: string, + result: Awaited>, + path: string, +): void => { + if (mode === "success") + expect(result).toMatchObject({ + ok: true, + value: { + result: { + artifact: { path, bytes: 76 }, + validation: "valid", + runtime_execution: "not-performed", + }, + }, + }); + else + expect(result).toMatchObject({ + ok: false, + error: { _tag: failureTags[mode] }, + }); + if (mode === "invalid") + expect(result).toMatchObject({ + error: { capturedOutput: { stderr: "module.wasm: invalid bytes\n" } }, + }); + if (mode === "malformed-wat" || mode === "wat-warning") + expect(result).toMatchObject({ + error: { + capturedOutput: { + stdout: + mode === "malformed-wat" + ? "unexpected decoder output\n" + : '(module (@custom "test" "payload"))\n', + stderr: mode === "wat-warning" ? "decoder warning\n" : "", + }, + }, + }); +}; + it("returns actionable unavailable without creating a workspace when tools are absent", async () => { const provider = new WabtArtifactProvider({}, undefined, () => { throw new Error("Must not allocate"); @@ -40,40 +142,133 @@ it("returns actionable unavailable without creating a workspace when tools are a }); }); -it("reports an unavailable private runtime root without a misleading WABT remediation", async () => { - const toolRoot = await createTestTempDirectory("rea-wabt-tool-root-"); +it("retries a retained input descriptor before admitting the next WABT operation", async () => { + const root = await createTestTempDirectory("rea-wabt-read-cleanup-"); + const path = join(root, "selected.wasm"); + await writeFile(path, Buffer.alloc(76, 1)); for (const tool of ["wasm-validate", "wasm-objdump", "wasm2wat"]) - await writeFile( - join(toolRoot, tool + (process.platform === "win32" ? ".exe" : "")), - "source-owned executable seam", - { mode: 0o700 }, - ); + await writeFile(join(root, tool), "source-owned executable seam", { + mode: 0o700, + }); + const dump = await readFile( + new URL("../../tests/fixtures/wasm/objdump.txt", import.meta.url), + "utf8", + ); const provider = new WabtArtifactProvider( - { REA_WABT_BIN_DIRECTORY: toolRoot }, - undefined, - () => { - throw new PrivateRuntimeRootUnavailableError( - "REA Windows native controls are unavailable: probe failure", - ); + { REA_WABT_BIN_DIRECTORY: root }, + async (input) => { + const version = input.arguments[0] === "--version"; + const stdout = version + ? "1.0.42\n" + : input.command.includes("wasm-objdump") + ? dump + : input.command.includes("wasm2wat") + ? '(module (@custom "test" "payload"))\n' + : ""; + const launched = await spawnOwnedProviderProcess({ + ...input, + command: process.execPath, + arguments: [ + "-e", + `process.stdout.write(${JSON.stringify(stdout)}); process.exit(0);`, + ], + expectedCommand: null, + }); + return launched; }, ); - expect( - await provider.inspect({ + fileCloseFailure.selectedPath = path; + fileCloseFailure.attempts = 0; + fileCloseFailure.failFirst = true; + try { + const first = await provider.inspect({ + path, + candidate_paths: [], + glue_paths: [], + }); + expect(first).toMatchObject({ + ok: false, + error: { + cleanupIncomplete: true, + cleanupResources: [path], + cause: { message: "injected WABT input close failure" }, + }, + }); + expect(fileCloseFailure.attempts).toBe(1); + + const retried = await provider.inspect({ + path, + candidate_paths: [], + glue_paths: [], + }); + expect(retried.ok).toBe(true); + expect(fileCloseFailure.attempts).toBeGreaterThan(1); + + fileCloseFailure.failFirst = true; + const closeRetry = await provider.inspect({ + path, + candidate_paths: [], + glue_paths: [], + }); + expect(closeRetry).toMatchObject({ + ok: false, + error: { cleanupIncomplete: true, cleanupResources: [path] }, + }); + const beforeClose = fileCloseFailure.attempts; + await provider.close(); + expect(fileCloseFailure.attempts).toBeGreaterThan(beforeClose); + } finally { + fileCloseFailure.selectedPath = ""; + fileCloseFailure.failFirst = false; + await provider.close(); + } +}); + +it.each([false, true])( + "reports an unavailable private runtime root with retryable=%s without a misleading WABT remediation", + async (retryable) => { + const toolRoot = await createTestTempDirectory("rea-wabt-tool-root-"); + for (const tool of ["wasm-validate", "wasm-objdump", "wasm2wat"]) + await writeFile( + join(toolRoot, tool + (process.platform === "win32" ? ".exe" : "")), + "source-owned executable seam", + { mode: 0o700 }, + ); + const provider = new WabtArtifactProvider( + { REA_WABT_BIN_DIRECTORY: toolRoot }, + undefined, + () => { + throw new PrivateRuntimeRootUnavailableError( + "REA Windows native controls are unavailable: probe failure", + { retryable }, + ); + }, + ); + const result = await provider.inspect({ path: "/module.wasm", glue_paths: [], candidate_paths: [], - }), - ).toMatchObject({ - ok: false, - error: { - _tag: "AnalysisCapabilityUnavailableError", - userMessage: undefined, - message: expect.stringContaining( - "REA Windows native controls are unavailable", - ), - }, - }); -}); + }); + expect(result).toMatchObject({ + ok: false, + error: { + _tag: "AnalysisCapabilityUnavailableError", + retryable, + userMessage: undefined, + message: expect.stringContaining( + "REA Windows native controls are unavailable", + ), + }, + }); + if (result.ok) + throw new Error("Unavailable runtime must refuse inspection"); + expect(projectAnalysisError(result.error)).toMatchObject({ + code: "capability_unavailable", + retryable, + category: retryable ? "unavailable" : "unsupported_provider", + }); + }, +); it.each([ "success", "invalid", @@ -100,6 +295,7 @@ it.each([ "utf8", ); let runtimePath = ""; + let runtimeCreations = 0; let acquiredPid: number | undefined; let cleanupAttempts = 0; const controller = new AbortController(); @@ -158,6 +354,7 @@ it.each([ return launched; }, async () => { + runtimeCreations += 1; const root = await PrivateRuntimeRoot.create({ prefix: "rea-wabt-owned-test-", }); @@ -176,53 +373,17 @@ it.each([ { path, glue_paths: [], candidate_paths: [] }, { signal: controller.signal }, ); - if (mode === "success") - expect(result).toMatchObject({ - ok: true, - value: { - result: { - artifact: { path, bytes: 76 }, - validation: "valid", - runtime_execution: "not-performed", - }, - }, - }); - else - expect(result).toMatchObject({ - ok: false, - error: { _tag: failureTags[mode] }, - }); - if (mode === "invalid") - expect(result).toMatchObject({ - error: { capturedOutput: { stderr: "module.wasm: invalid bytes\n" } }, - }); + assertModeOutcome(mode, result, path); if (mode === "cleanup") { - expect(result).toMatchObject({ - error: { - cleanupIncomplete: true, - cleanupResources: [runtimePath], - partialObservation: { normalized_result: { validation: "valid" } }, - }, + await assertCleanupRecovery({ + provider, + result, + path, + runtimePath, + cleanupAttempts: () => cleanupAttempts, + runtimeCreations: () => runtimeCreations, }); - await expect(access(runtimePath)).resolves.toBeUndefined(); - await expect(provider.close()).rejects.toMatchObject({ - cleanupIncomplete: true, - }); - await expect(access(runtimePath)).resolves.toBeUndefined(); - await provider.close(); } - if (mode === "malformed-wat" || mode === "wat-warning") - expect(result).toMatchObject({ - error: { - capturedOutput: { - stdout: - mode === "malformed-wat" - ? "unexpected decoder output\n" - : '(module (@custom "test" "payload"))\n', - stderr: mode === "wat-warning" ? "decoder warning\n" : "", - }, - }, - }); await expect(access(runtimePath)).rejects.toMatchObject({ code: "ENOENT" }); expect(await readFile(path)).toEqual(bytes); if (acquiredPid !== undefined) { diff --git a/src/wasm/WabtArtifactProvider.ts b/src/wasm/WabtArtifactProvider.ts index ffed3871b..c5f0aef53 100644 --- a/src/wasm/WabtArtifactProvider.ts +++ b/src/wasm/WabtArtifactProvider.ts @@ -2,13 +2,15 @@ import { randomUUID, createHash } from "node:crypto"; import { access, realpath, writeFile } from "node:fs/promises"; import { constants } from "node:fs"; import { isAbsolute, join } from "node:path"; -import type { - AnalysisExecution, - ExecutionOptions, -} from "../application/AnalysisProvider.js"; +import type { ExecutionOptions } from "../application/AnalysisProvider.js"; import { createAnalysisExecution } from "../application/AnalysisProvider.js"; -import type { WasmArtifactPort } from "../application/wasm/WasmArtifactPort.js"; +import type { + WasmArtifactExecution, + WasmArtifactPort, +} from "../application/wasm/WasmArtifactPort.js"; import { readStableArtifact } from "../artifacts/readStableArtifact.js"; +import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js"; +import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js"; import type { AnalysisError, AnalysisCapturedOutput, @@ -16,6 +18,7 @@ import type { import { AnalysisCancelledError, AnalysisArtifactChangedError, + AnalysisCapabilityUnavailableError, AnalysisInputError, AnalysisOutputError, } from "../domain/analysisErrorCore.js"; @@ -41,9 +44,7 @@ const digest = (text: string) => createHash("sha256").update(text).digest("hex"); /** Run caller-supplied WABT against private copies of exactly selected bytes. */ export class WabtArtifactProvider implements WasmArtifactPort { - readonly #pendingRoots = new Set< - Pick - >(); + readonly #resources = new ArtifactResourceScope(); constructor( readonly environment: Readonly = process.env, readonly launcher?: Launcher, @@ -53,37 +54,58 @@ export class WabtArtifactProvider implements WasmArtifactPort { ) {} /** Retry retained cleanup without discarding roots that still cannot be removed. */ async close(): Promise { - const failures: { path: string; reason: string }[] = []; - for (const root of this.#pendingRoots) { - try { - await root.close(); - this.#pendingRoots.delete(root); - } catch (cause: unknown) { - failures.push({ - path: root.path, - reason: cause instanceof Error ? cause.message : String(cause), - }); - } - } - if (failures.length > 0) + try { + await this.#resources.close(); + } catch (cause: unknown) { + const cleanup = + cause instanceof ArtifactReaderFailure ? cause.cleanup : undefined; throw new ProviderCleanupError( WABT_PROVIDER_IDENTITY.id, - failures.map(({ path }) => path), - { failures }, - { operation: OPERATION }, + cleanup?.resources ?? ["artifact-resources"], + { + reason: + cleanup?.reason ?? + (cause instanceof Error ? cause.message : String(cause)), + }, + { operation: OPERATION, cause }, ); + } } async inspect( input: InspectWasmArtifactInput, options?: ExecutionOptions, - ): Promise> { + ): Promise> { + try { + return await this.#resources.run(() => this.#inspect(input, options)); + } catch (cause: unknown) { + if ( + cause instanceof ArtifactReaderFailure && + cause.reason === "unavailable" && + cause.cleanup === undefined + ) + return err(this.#closedError()); + return err(wabtFailure(cause, "artifact", input.path)); + } + } + + #closedError(): AnalysisCapabilityUnavailableError { + return new AnalysisCapabilityUnavailableError( + WABT_PROVIDER_IDENTITY.id, + OPERATION, + "Provider is closing", + ); + } + + async #inspect( + input: InspectWasmArtifactInput, + options?: ExecutionOptions, + ): Promise> { let root: Pick | undefined; let phase: "configuration" | "artifact" | "producer" = "configuration"; - let result: Result; + let result: Result; let capturedOutput: AnalysisCapturedOutput | undefined; try { if (options?.signal?.aborted) throw new AnalysisCancelledError(OPERATION); - await this.close(); const directory = this.environment.REA_WABT_BIN_DIRECTORY ?? ""; if (!isAbsolute(directory)) throw new Error( @@ -108,6 +130,8 @@ export class WabtArtifactProvider implements WasmArtifactPort { path, WABT_LIMITS.toolBytes, options?.signal, + undefined, + this.#resources, ); commands.push({ tool, @@ -130,17 +154,24 @@ export class WabtArtifactProvider implements WasmArtifactPort { snapshotPath, WABT_LIMITS.inputBytes, options?.signal, + undefined, + this.#resources, ); if (bytes.sha256 !== snapshotHash) throw new Error("Private WASM snapshot bytes changed."); } catch (cause: unknown) { + const cleanup = + cause instanceof ArtifactReaderFailure ? cause.cleanup : undefined; if (options?.signal?.aborted) - throw new AnalysisCancelledError(OPERATION); + throw new AnalysisCancelledError(OPERATION, { + cause, + ...(cleanup === undefined ? {} : { cleanup }), + }); throw new AnalysisArtifactChangedError( OPERATION, input.path, "Private WASM snapshot no longer matches selected byte identity.", - { cause }, + { cause, ...(cleanup === undefined ? {} : { cleanup }) }, ); } }; @@ -175,6 +206,8 @@ export class WabtArtifactProvider implements WasmArtifactPort { command.path, WABT_LIMITS.toolBytes, options?.signal, + undefined, + this.#resources, ); if (current.sha256 !== command.sha256) throw new AnalysisOutputError( @@ -200,6 +233,8 @@ export class WabtArtifactProvider implements WasmArtifactPort { input.path, WABT_LIMITS.inputBytes, options?.signal, + undefined, + this.#resources, ); const artifact = { path: input.path, @@ -215,6 +250,8 @@ export class WabtArtifactProvider implements WasmArtifactPort { path, WABT_LIMITS.inputBytes, options?.signal, + undefined, + this.#resources, ); candidates.push({ path, @@ -228,6 +265,8 @@ export class WabtArtifactProvider implements WasmArtifactPort { path, WABT_LIMITS.inputBytes, options?.signal, + undefined, + this.#resources, ); const source = new TextDecoder("utf-8", { fatal: true }).decode( bytes.bytes, @@ -395,10 +434,13 @@ export class WabtArtifactProvider implements WasmArtifactPort { ); } if (root !== undefined) { - try { - await root.close(); - } catch (cause: unknown) { - this.#pendingRoots.add(root); + const cleanup = await this.#resources.release({ + kind: "private-runtime-root", + resource: root.path, + root, + }); + if (cleanup.kind === "failed") { + const cause = cleanup.cause; const partialObservation = result.ok ? createEvidence( result.value.subject ?? undefined, diff --git a/src/wasm/WabtFailures.ts b/src/wasm/WabtFailures.ts index ef5692f1b..7b6875203 100644 --- a/src/wasm/WabtFailures.ts +++ b/src/wasm/WabtFailures.ts @@ -11,7 +11,7 @@ import { } from "../domain/analysisErrorCore.js"; import { ProviderCleanupError } from "../domain/providerCleanupError.js"; import { OwnedCommandFailure } from "../process/OwnedCommand.js"; -import { PrivateRuntimeRootUnavailableError } from "../process/PrivateRuntimeRoot.js"; +import { PrivateRuntimeRootUnavailableError } from "../process/PrivateRuntimeRootUnavailableError.js"; import { WABT_LIMITS, WABT_PROVIDER_IDENTITY } from "./WabtRelease.js"; /** Preserve lifecycle failures and diagnostic bytes instead of returning partial success. */ export const wabtFailure = ( @@ -20,12 +20,19 @@ export const wabtFailure = ( path: string, ): AnalysisError => { if (cause instanceof AnalysisError) return cause; + const artifactCleanup = + cause instanceof ArtifactReaderFailure ? cause.cleanup : undefined; + const cleanupOptions = + artifactCleanup === undefined ? {} : { cleanup: artifactCleanup }; const reason = cause instanceof Error ? cause.message : String(cause); if ( (cause instanceof ArtifactReaderFailure && cause.reason === "cancelled") || (cause instanceof Error && cause.name === "AbortError") ) - return new AnalysisCancelledError("inspect_wasm_artifact", { cause }); + return new AnalysisCancelledError("inspect_wasm_artifact", { + cause, + ...cleanupOptions, + }); if (cause instanceof OwnedCommandFailure) { const capturedOutput = cause.snapshot === null @@ -76,7 +83,7 @@ export const wabtFailure = ( WABT_PROVIDER_IDENTITY.id, "inspect_wasm_artifact", reason, - { cause }, + { cause, retryable: cause.retryable }, ); if (phase === "configuration") return new AnalysisCapabilityUnavailableError( @@ -85,6 +92,7 @@ export const wabtFailure = ( reason, { cause, + ...cleanupOptions, userMessage: `${reason} Supply WABT 1.0.42 via absolute REA_WABT_BIN_DIRECTORY; REA does not install tools.`, }, ); @@ -97,7 +105,7 @@ export const wabtFailure = ( "inspect_wasm_artifact", path, reason, - { cause }, + { cause, ...cleanupOptions }, ); if ( phase === "artifact" && @@ -109,11 +117,16 @@ export const wabtFailure = ( "inspect_wasm_artifact", path, cause.code, - { cause }, + { cause, ...cleanupOptions }, ); if (phase === "artifact") - return new AnalysisInputError("inspect_wasm_artifact", { cause }, [ - { path: ["path"], reason: "invalid_value", message: reason }, - ]); - return new AnalysisOutputError("inspect_wasm_artifact", reason, { cause }); + return new AnalysisInputError( + "inspect_wasm_artifact", + { cause, ...cleanupOptions }, + [{ path: ["path"], reason: "invalid_value", message: reason }], + ); + return new AnalysisOutputError("inspect_wasm_artifact", reason, { + cause, + ...cleanupOptions, + }); }; diff --git a/src/windows/WindowsNativeLoader.test.ts b/src/windows/WindowsNativeLoader.test.ts index 1aa8d7375..c0fee8273 100644 --- a/src/windows/WindowsNativeLoader.test.ts +++ b/src/windows/WindowsNativeLoader.test.ts @@ -143,6 +143,22 @@ describe("packaged Windows native admission", () => { }, ); + it.each([ + { signature: "MZ", offset: 0 }, + { signature: "PE", offset: 64 }, + ])( + "rejects digest-valid high-bit $signature lookalikes before loading", + ({ offset }) => { + const value = fixture(); + value.bytes[offset] = (value.bytes[offset] ?? 0) | 0x80; + value.manifest.artifactSha256 = createHash("sha256") + .update(value.bytes) + .digest("hex"); + expect(loadWindowsNativeAuthority(value.host).available).toBe(false); + expect(value.loads()).toBe(0); + }, + ); + it("rejects a digest-valid artifact for another machine", () => { const value = fixture(); value.bytes.writeUInt16LE(0xaa64, 68); diff --git a/src/windows/WindowsNativeLoader.ts b/src/windows/WindowsNativeLoader.ts index 7719a0a85..2ab08e1ed 100644 --- a/src/windows/WindowsNativeLoader.ts +++ b/src/windows/WindowsNativeLoader.ts @@ -132,12 +132,12 @@ export const loadWindowsNativeAuthority = ( }; const assertWindowsX64Artifact = (bytes: Buffer): void => { - if (bytes.length < 64 || bytes.toString("ascii", 0, 2) !== "MZ") + if (bytes.length < 64 || bytes.toString("latin1", 0, 2) !== "MZ") throw new Error("Windows native artifact is not a PE image"); const offset = bytes.readUInt32LE(0x3c); if ( offset > bytes.length - 26 || - bytes.toString("ascii", offset, offset + 4) !== "PE\0\0" || + bytes.toString("latin1", offset, offset + 4) !== "PE\0\0" || bytes.readUInt16LE(offset + 4) !== 0x8664 || bytes.readUInt16LE(offset + 24) !== 0x20b ) diff --git a/src/windows/WindowsPrivateRuntime.ts b/src/windows/WindowsPrivateRuntime.ts index e17544a49..2c361a6d0 100644 --- a/src/windows/WindowsPrivateRuntime.ts +++ b/src/windows/WindowsPrivateRuntime.ts @@ -1,6 +1,7 @@ import { win32 } from "node:path"; import { z } from "zod"; +import { PrivateRuntimeRootUnavailableError } from "../process/PrivateRuntimeRootUnavailableError.js"; import { requireWindowsNativeAuthority, @@ -23,6 +24,13 @@ const fileSchema = windowsFileIdentitySchema.extend({ handle: windowsHandleSchema, }); const runtimes = new Map(); +// Retry only a proven pre-allocation sharing/lock conflict, never opaque text +// or an operation that may already own an unpublished runtime directory. +const transientRuntimeAdmissionSchema = z.object({ + code: z.literal("ERR_REA_WINDOWS_NATIVE"), + win32Code: z.union([z.literal(32), z.literal(33)]), + runtimeAllocation: z.literal("not-started"), +}); /** Retain private DACL and path handles until native recursive cleanup settles. */ export class WindowsPrivateRuntime { @@ -40,9 +48,21 @@ export class WindowsPrivateRuntime { /** Create a new directory with its private DACL installed at creation. */ static create(parent: string, prefix: string): WindowsPrivateRuntime { const authority = requireWindowsNativeAuthority(); - const observation = runtimeSchema.parse( - authority.call("runtime_create", [parent, prefix]), - ); + let observed: unknown; + try { + observed = authority.call("runtime_create", [parent, prefix]); + } catch (cause: unknown) { + if ( + cause instanceof Error && + transientRuntimeAdmissionSchema.safeParse(cause).success + ) + throw new PrivateRuntimeRootUnavailableError(cause.message, { + cause, + retryable: true, + }); + throw cause; + } + const observation = runtimeSchema.parse(observed); const runtime = new WindowsPrivateRuntime(observation, authority); runtimes.set(windowsRuntimeIdentityKey(observation), runtime); return runtime; diff --git a/tests/acceptance/applications/JavaScriptModuleLookupCli.test.ts b/tests/acceptance/applications/JavaScriptModuleLookupCli.test.ts index cd045a00a..3a7fad35f 100644 --- a/tests/acceptance/applications/JavaScriptModuleLookupCli.test.ts +++ b/tests/acceptance/applications/JavaScriptModuleLookupCli.test.ts @@ -29,6 +29,14 @@ const createModuleLookupFixture = async () => { 'import "ancestor-empty";', 'import "ancestor-blocked";', 'import "ancestor-sub/parser";', + 'import "./.hidden.js";', + 'import "./..pkg.js";', + 'import "./exact.js/.";', + 'import "./exact.js/child/..";', + 'import ".hidden.js";', + 'import "..pkg.js";', + 'import ".";', + 'import "..";', "void fallback;", ].join("\n"), "esm/extensionless.js": "module.exports = 1;", @@ -36,6 +44,10 @@ const createModuleLookupFixture = async () => { "esm/explicit.js.js": "module.exports = 3;", "esm/exact.js": "module.exports = 4;", "esm/actual.cjs": "module.exports = 5;", + "esm/.hidden.js": "module.exports = 'local hidden';", + "esm/..pkg.js": "module.exports = 'local dot package';", + "node_modules/.hidden.js/index.js": "module.exports = 'hidden package';", + "node_modules/..pkg.js/index.js": "module.exports = 'dot package';", "esm/typed.mts": 'import value from "./typed-dep.js"; void value;', "esm/typed-dep.ts": "export default 42;", "cjs/entry.cjs": [ @@ -54,8 +66,15 @@ const createModuleLookupFixture = async () => { 'require("ancestor-empty");', 'require("ancestor-blocked");', 'require("ancestor-sub/parser");', + 'require(".hidden.js");', + 'require("./.hidden.js");', + 'require(".");', + 'require("..");', ].join("\n"), "cjs/directory/index.cjs": "module.exports = 6;", + "cjs/index.js": "module.exports = 'current directory';", + "index.js": "module.exports = 'parent directory';", + "cjs/.hidden.js": "module.exports = 'local hidden';", "cjs/extensionless.cjs": "module.exports = 1;", "cjs/feature.cjs": "module.exports = 2;", "cjs/module-only.mjs": "export default 42;", @@ -79,7 +98,7 @@ const createModuleLookupFixture = async () => { "node_modules/ancestor-blocked/index.js": "module.exports = 9;", "node_modules/ancestor-sub/parser.js": "module.exports = 10;", "esm/runtime-control.mjs": [ - 'const specifiers = ["./extensionless", "./directory", "./explicit.js", "./exact.js", "./actual.cjs?cache=1", "empty-main", "nested-main", "file-only", "ancestor-empty", "ancestor-blocked", "ancestor-sub/parser"];', + 'const specifiers = ["./extensionless", "./directory", "./explicit.js", "./exact.js", "./actual.cjs?cache=1", "empty-main", "nested-main", "file-only", "ancestor-empty", "ancestor-blocked", "ancestor-sub/parser", "./.hidden.js", "./..pkg.js", "./exact.js/.", "./exact.js/child/..", ".hidden.js", "..pkg.js", ".", ".."];', "const results = [];", "for (const specifier of specifiers) {", " try { const value = (await import(specifier)).default; results.push({ specifier, ok: true, value }); }", @@ -88,7 +107,7 @@ const createModuleLookupFixture = async () => { "console.log(JSON.stringify(results));", ].join("\n"), "cjs/runtime-control.cjs": [ - 'const specifiers = ["./extensionless", "./directory", "./feature", "./feature.cjs", "./module-only", "./valid-directory", "./default-js", "./default-json", "empty-main", "nested-main", "file-only", "ancestor-empty", "ancestor-blocked", "ancestor-sub/parser"];', + 'const specifiers = ["./extensionless", "./directory", "./feature", "./feature.cjs", "./module-only", "./valid-directory", "./default-js", "./default-json", "empty-main", "nested-main", "file-only", "ancestor-empty", "ancestor-blocked", "ancestor-sub/parser", ".hidden.js", "./.hidden.js", ".", ".."];', "const results = [];", "for (const specifier of specifiers) {", " try { const value = require(specifier); results.push({ specifier, ok: true, value }); }", @@ -109,6 +128,110 @@ const createModuleLookupFixture = async () => { return root; }; +type RuntimeControlResult = { + readonly specifier: string; + readonly ok: boolean; + readonly code?: string; + readonly value?: unknown; +}; + +const expectNodeRuntimeControls = ( + esmOutput: string, + cjsOutput: string, +): void => { + const esmResults = JSON.parse(esmOutput) as RuntimeControlResult[]; + const cjsResults = JSON.parse(cjsOutput) as RuntimeControlResult[]; + expect(esmResults).toEqual([ + { + specifier: "./extensionless", + ok: false, + code: "ERR_MODULE_NOT_FOUND", + }, + { + specifier: "./directory", + ok: false, + code: "ERR_UNSUPPORTED_DIR_IMPORT", + }, + { + specifier: "./explicit.js", + ok: false, + code: "ERR_MODULE_NOT_FOUND", + }, + { specifier: "./exact.js", ok: true, value: 4 }, + { specifier: "./actual.cjs?cache=1", ok: true, value: 5 }, + { specifier: "empty-main", ok: true, value: 42 }, + { specifier: "nested-main", ok: false, code: "ERR_MODULE_NOT_FOUND" }, + { specifier: "file-only", ok: false, code: "ERR_MODULE_NOT_FOUND" }, + { specifier: "ancestor-empty", ok: false, code: "ERR_MODULE_NOT_FOUND" }, + { + specifier: "ancestor-blocked", + ok: false, + code: "ERR_MODULE_NOT_FOUND", + }, + { + specifier: "ancestor-sub/parser", + ok: false, + code: "ERR_MODULE_NOT_FOUND", + }, + { specifier: "./.hidden.js", ok: true, value: "local hidden" }, + { specifier: "./..pkg.js", ok: true, value: "local dot package" }, + { + specifier: "./exact.js/.", + ok: false, + code: "ERR_UNSUPPORTED_DIR_IMPORT", + }, + { + specifier: "./exact.js/child/..", + ok: false, + code: "ERR_UNSUPPORTED_DIR_IMPORT", + }, + { + specifier: ".hidden.js", + ok: false, + code: "ERR_INVALID_MODULE_SPECIFIER", + }, + { + specifier: "..pkg.js", + ok: false, + code: "ERR_INVALID_MODULE_SPECIFIER", + }, + { specifier: ".", ok: false, code: "ERR_UNSUPPORTED_DIR_IMPORT" }, + { specifier: "..", ok: false, code: "ERR_UNSUPPORTED_DIR_IMPORT" }, + ]); + expect(cjsResults).toEqual([ + { + specifier: "./extensionless", + ok: false, + code: "MODULE_NOT_FOUND", + }, + { + specifier: "./directory", + ok: false, + code: "MODULE_NOT_FOUND", + }, + { + specifier: "./feature", + ok: false, + code: "MODULE_NOT_FOUND", + }, + { specifier: "./feature.cjs", ok: true, value: 2 }, + { specifier: "./module-only", ok: false, code: "MODULE_NOT_FOUND" }, + { specifier: "./valid-directory", ok: true, value: 42 }, + { specifier: "./default-js", ok: true, value: 3 }, + { specifier: "./default-json", ok: true, value: { value: 4 } }, + { specifier: "empty-main", ok: true, value: 42 }, + { specifier: "nested-main", ok: false, code: "MODULE_NOT_FOUND" }, + { specifier: "file-only", ok: true, value: 7 }, + { specifier: "ancestor-empty", ok: true, value: 8 }, + { specifier: "ancestor-blocked", ok: false, code: "MODULE_NOT_FOUND" }, + { specifier: "ancestor-sub/parser", ok: true, value: 10 }, + { specifier: ".hidden.js", ok: true, value: "hidden package" }, + { specifier: "./.hidden.js", ok: true, value: "local hidden" }, + { specifier: ".", ok: true, value: "current directory" }, + { specifier: "..", ok: true, value: "parent directory" }, + ]); +}; + cliTest.for(["CLI", "stdio MCP"] as const)( "matches Node module lookup for ESM and CommonJS through %s", async (surface, { cli }) => { @@ -123,79 +246,7 @@ cliTest.for(["CLI", "stdio MCP"] as const)( timeout: 5_000, }), ]); - const esmResults = JSON.parse(esmControl.stdout) as Array<{ - readonly specifier: string; - readonly ok: boolean; - readonly code?: string; - readonly value?: unknown; - }>; - const cjsResults = JSON.parse(cjsControl.stdout) as Array<{ - readonly specifier: string; - readonly ok: boolean; - readonly code?: string; - readonly value?: unknown; - }>; - expect(esmResults).toEqual([ - { - specifier: "./extensionless", - ok: false, - code: "ERR_MODULE_NOT_FOUND", - }, - { - specifier: "./directory", - ok: false, - code: "ERR_UNSUPPORTED_DIR_IMPORT", - }, - { - specifier: "./explicit.js", - ok: false, - code: "ERR_MODULE_NOT_FOUND", - }, - { specifier: "./exact.js", ok: true, value: 4 }, - { specifier: "./actual.cjs?cache=1", ok: true, value: 5 }, - { specifier: "empty-main", ok: true, value: 42 }, - { specifier: "nested-main", ok: false, code: "ERR_MODULE_NOT_FOUND" }, - { specifier: "file-only", ok: false, code: "ERR_MODULE_NOT_FOUND" }, - { specifier: "ancestor-empty", ok: false, code: "ERR_MODULE_NOT_FOUND" }, - { - specifier: "ancestor-blocked", - ok: false, - code: "ERR_MODULE_NOT_FOUND", - }, - { - specifier: "ancestor-sub/parser", - ok: false, - code: "ERR_MODULE_NOT_FOUND", - }, - ]); - expect(cjsResults).toEqual([ - { - specifier: "./extensionless", - ok: false, - code: "MODULE_NOT_FOUND", - }, - { - specifier: "./directory", - ok: false, - code: "MODULE_NOT_FOUND", - }, - { - specifier: "./feature", - ok: false, - code: "MODULE_NOT_FOUND", - }, - { specifier: "./feature.cjs", ok: true, value: 2 }, - { specifier: "./module-only", ok: false, code: "MODULE_NOT_FOUND" }, - { specifier: "./valid-directory", ok: true, value: 42 }, - { specifier: "./default-js", ok: true, value: 3 }, - { specifier: "./default-json", ok: true, value: { value: 4 } }, - { specifier: "empty-main", ok: true, value: 42 }, - { specifier: "nested-main", ok: false, code: "MODULE_NOT_FOUND" }, - { specifier: "file-only", ok: true, value: 7 }, - { specifier: "ancestor-empty", ok: true, value: 8 }, - { specifier: "ancestor-blocked", ok: false, code: "MODULE_NOT_FOUND" }, - { specifier: "ancestor-sub/parser", ok: true, value: 10 }, - ]); + expectNodeRuntimeControls(esmControl.stdout, cjsControl.stdout); const { graph } = await analyzeJavaScriptDirectory(surface, cli, root); const expectations = [ @@ -211,6 +262,14 @@ cliTest.for(["CLI", "stdio MCP"] as const)( ["import", "ancestor-empty", null, "not-found"], ["import", "ancestor-blocked", null, "not-found"], ["import", "ancestor-sub/parser", null, "not-found"], + ["import", "./.hidden.js", "esm/.hidden.js", "resolved"], + ["import", "./..pkg.js", "esm/..pkg.js", "resolved"], + ["import", ".hidden.js", null, "rejected"], + ["import", "..pkg.js", null, "rejected"], + ["import", ".", null, "not-found"], + ["import", "..", null, "not-found"], + ["import", "./exact.js/.", null, "not-found"], + ["import", "./exact.js/child/..", null, "not-found"], ["require", "./extensionless", null, "not-found"], ["require", "./directory", null, "not-found"], ["require", "./feature", null, "not-found"], @@ -242,19 +301,30 @@ cliTest.for(["CLI", "stdio MCP"] as const)( "node_modules/ancestor-sub/parser.js", "resolved", ], + ["require", ".hidden.js", "node_modules/.hidden.js/index.js", "resolved"], + ["require", "./.hidden.js", "cjs/.hidden.js", "resolved"], + ["require", ".", "cjs/index.js", "resolved"], + ["require", "..", "index.js", "resolved"], ] as const; - for (const [kind, specifier, resolvedPath, status] of expectations) - expect(graph.edges).toContainEqual( + for (const [kind, specifier, resolvedPath, status] of expectations) { + const matchingProperties = graph.edges + .filter((edge) => edge.relation === "imports") + .map((edge) => edge.properties) + .filter( + (properties) => + "module_link_kind" in properties && + properties.module_link_kind === kind && + properties.specifier === specifier, + ); + expect(matchingProperties).toContainEqual( expect.objectContaining({ - relation: "imports", - properties: expect.objectContaining({ - module_link_kind: kind, - specifier, - resolved_path: resolvedPath, - resolution_status: status, - }), + module_link_kind: kind, + specifier, + resolved_path: resolvedPath, + resolution_status: status, }), ); + } }, ); diff --git a/tests/acceptance/applications/bundlerCallbackMetadataCli.test.ts b/tests/acceptance/applications/bundlerCallbackMetadataCli.test.ts new file mode 100644 index 000000000..1f3987237 --- /dev/null +++ b/tests/acceptance/applications/bundlerCallbackMetadataCli.test.ts @@ -0,0 +1,61 @@ +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect } from "vitest"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; +import { analyzeJavaScriptDirectory } from "../../support/cli/javascriptApplication.js"; + +cliTest.for(["CLI", "stdio MCP"] as const)( + "retains both repeated chunk callback observations through %s", + async (surface, { cli }) => { + const root = await createTestTempDirectory("rea-bundler-callbacks-"); + const source = [ + '(globalThis.webpackChunkFixture = globalThis.webpackChunkFixture || []).push([[1], {a: function() {}}, function(r) { r(7); r.e("chunk-a"); }]);', + '(globalThis.webpackChunkFixture = globalThis.webpackChunkFixture || []).push([[1], {a: function() {}}, function(r) { r(8); r.e("chunk-b"); }]);', + ].join("\n"); + await writeFile(join(root, "bundle.js"), source); + const { graph } = await analyzeJavaScriptDirectory(surface, cli, root); + const chunk = graph.nodes.find(({ kind }) => kind === "javascript-chunk"); + for (const [index, entry, asyncChunk] of [ + [0, "7", "chunk-a"], + [1, "8", "chunk-b"], + ] as const) { + expect(chunk?.observations).toContainEqual( + expect.objectContaining({ + properties: expect.objectContaining({ + entry_module_keys: [entry], + async_chunk_keys: [asyncChunk], + }), + evidence: expect.objectContaining({ + location: expect.objectContaining({ + value: expect.objectContaining({ + source: "bundle.js", + start: { line: index + 1, column: 0 }, + }), + }), + }), + }), + ); + expect(graph.edges).toContainEqual( + expect.objectContaining({ + source_node_id: chunk?.node_id, + properties: expect.objectContaining({ + kind: "bundler-entry-module", + module_key: entry, + resolution_status: "not-found", + }), + }), + ); + expect(graph.edges).toContainEqual( + expect.objectContaining({ + source_node_id: chunk?.node_id, + properties: expect.objectContaining({ + kind: "bundler-async-chunk", + chunk_key: asyncChunk, + resolution_status: "not-found", + }), + }), + ); + } + }, +); diff --git a/tests/acceptance/applications/getterLocalAliasesCli.test.ts b/tests/acceptance/applications/getterLocalAliasesCli.test.ts new file mode 100644 index 000000000..2b2b0196c --- /dev/null +++ b/tests/acceptance/applications/getterLocalAliasesCli.test.ts @@ -0,0 +1,85 @@ +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect } from "vitest"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; +import { analyzeJavaScriptDirectory } from "../../support/cli/javascriptApplication.js"; + +cliTest.for(["CLI", "stdio MCP"] as const)( + "retains getter-local alias mutations in exported Evidence through %s", + async (surface, { cli }) => { + const root = await createTestTempDirectory("rea-getter-local-aliases-"); + const cases = [ + ["initialized", "const self = this; return self.s;", true], + ["assigned", "let self; self = this; return self.s;", true], + ["destructured", "const { s: value } = this; return value;", true], + ["readOnly", "const self = this; return self.s;", false], + [ + "cyclic", + "let self = this.start; while (flag) self = self.child; return self;", + true, + "start: { child: { child: shared } }", + ], + ] as const; + const privateCases = [ + ["privateProjection", "shared", true, "unknown"], + ["privateUnrelated", "{ peer: shared }", true, "literal"], + ["privateReadOnly", "shared", false, "literal"], + ] as const; + await writeFile( + join(root, "app.mjs"), + cases + .map( + ([ + name, + getter, + write, + members = "s: shared", + ]) => `export function ${name}() { + const shared = { x: 1 }; + const box = { ${members}, get v() { ${getter} } }; + ${write ? "box.v.x = 2;" : "const saved = box.v;"} + return shared.x; + }`, + ) + .join("\n") + + privateCases + .map( + ([name, value, write]) => `export function ${name}() { + const shared = { x: 1 }; + class Box { + #s = ${value}; + peer = this; + get v() { const { peer: self } = this; return self.#s; } + } + const box = new Box(); + ${write ? "box.v.x = 2;" : "const saved = box.v;"} + return shared.x; + }`, + ) + .join("\n"), + ); + const { graph } = await analyzeJavaScriptDirectory(surface, cli, root); + for (const [name, status] of [ + ...cases.map(([name, , write]) => [name, write ? "unknown" : "literal"]), + ...privateCases.map(([name, , , status]) => [name, status]), + ]) { + const observation = graph.nodes + .flatMap(({ observations }) => observations) + .find( + ({ properties }) => + properties.semantic_role === "export-return-shapes" && + properties.exported_name === name, + ); + expect(observation, name).toBeDefined(); + expect(observation?.properties.static_return_shapes).toMatchObject([ + { + value_status: status, + fields: [ + { path: "", state: status, value: status === "unknown" ? null : 1 }, + ], + }, + ]); + } + }, +); diff --git a/tests/acceptance/applications/getterReceiverMutationsCli.test.ts b/tests/acceptance/applications/getterReceiverMutationsCli.test.ts index a5ba0bfeb..95b6e4862 100644 --- a/tests/acceptance/applications/getterReceiverMutationsCli.test.ts +++ b/tests/acceptance/applications/getterReceiverMutationsCli.test.ts @@ -39,6 +39,47 @@ export function superclass() { new B().v.x = 2; return shared.x; } +export function arrayReceiver() { + const arr = [{ x: 1, get self() { return this; } }]; + arr[0].self.x = 2; + return arr[0].x; +} +export function nestedArrayReceiver() { + const outer = { arr: [{ x: 1, get self() { return this; } }] }; + outer.arr[0].self.x = 2; + return outer.arr[0].x; +} +export function assignedReceiver() { + let box; + box = { x: 1, get self() { return this; } }; + box.self.x = 2; + return box.x; +} +export function savedArrayReceiver() { + const arr = [{ x: 1, get self() { return this; } }]; + const saved = arr[0].self; + saved.x = 2; + return arr[0].x; +} +export function arrayReadOnly() { + const arr = [{ x: 1, get self() { return this; } }]; + const saved = arr[0].self; + return arr[0].x; +} +export function replacementKeepsOldObject() { + const old = { x: 1 }; + let box = old; + box = { x: 1, old: box, get self() { return this; } }; + box.self.x = 2; + return old.x; +} +export function replacementMutatesCapturedObject() { + const old = { x: 1 }; + let box = old; + box = { old: box, get self() { return this; } }; + box.self.old.x = 2; + return old.x; +} export function untouched() { const box = { x: 1, y: 1, get self() { return this; } }; box.self.y = 2; @@ -66,13 +107,20 @@ export function untouched() { const { graph } = z .object({ normalized_result: javascriptApplicationAnalysisResultSchema }) .parse(document).normalized_result; - for (const name of [ - "self", - "property", - "privateField", - "superclass", - "untouched", - ]) { + for (const [name, status] of [ + ["self", "unknown"], + ["property", "unknown"], + ["privateField", "unknown"], + ["superclass", "unknown"], + ["arrayReceiver", "unknown"], + ["nestedArrayReceiver", "unknown"], + ["assignedReceiver", "unknown"], + ["savedArrayReceiver", "unknown"], + ["arrayReadOnly", "literal"], + ["replacementKeepsOldObject", "literal"], + ["replacementMutatesCapturedObject", "unknown"], + ["untouched", "literal"], + ] as const) { const projection = graph.nodes .flatMap(({ observations }) => observations) .find( @@ -82,7 +130,16 @@ export function untouched() { ); expect(projection, name).toBeDefined(); expect(projection?.properties.static_return_shapes).toMatchObject([ - { value_status: name === "untouched" ? "literal" : "unknown" }, + { + value_status: status, + fields: [ + { + path: "", + state: status, + ...(status === "literal" ? { value: 1 } : {}), + }, + ], + }, ]); } }, diff --git a/tests/acceptance/applications/nibFieldIdentity.test.ts b/tests/acceptance/applications/nibFieldIdentity.test.ts new file mode 100644 index 000000000..5af16f890 --- /dev/null +++ b/tests/acceptance/applications/nibFieldIdentity.test.ts @@ -0,0 +1,110 @@ +import { Client } from "@modelcontextprotocol/client"; +import { StdioClientTransport } from "@modelcontextprotocol/client/stdio"; +import { createHash } from "node:crypto"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { join, resolve } from "node:path"; +import { expect } from "vitest"; + +import { encodeNibArchiveFixture } from "../../../src/artifacts/apple/NibArchive.fixture.js"; +import { toolContract } from "../../../src/contracts/toolContracts.js"; +import { interfaceBuilderAnalysisSchema } from "../../../src/domain/apple/interfaceBuilderGraph.js"; +import { thinMach } from "../../../src/domain/binaryTarget.fixture.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; + +cliTest( + "preserves literal NIB fields and UTF-8 names through CLI and stdio MCP", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-nib-field-identity-"); + const app = join(root, "Fixture.app"); + const contents = join(app, "Contents"); + await mkdir(join(contents, "Resources"), { recursive: true }); + await mkdir(join(contents, "MacOS")); + await writeFile( + join(contents, "MacOS/App"), + thinMach(0xfeedfacf, 0x0100000c), + ); + await writeFile( + join(contents, "Info.plist"), + "CFBundleExecutableAppCFBundleIdentifierdev.rea.nib-fields", + ); + const archive = encodeNibArchiveFixture({ + classes: ["UIView", "\uFEFFCustomView"], + objects: [ + { + classIndex: 0, + values: { + ["__proto__"]: { ref: 1 }, + constructor: 7, + enabled: true, + "\uFEFFenabled": false, + }, + }, + { classIndex: 1, values: { ["__proto__"]: 23 } }, + ], + }); + const path = join(contents, "Resources/Panel.nib"); + await writeFile(path, archive); + const result = await cli.run({ + arguments: ["decode-interface-builder", app, "--json"], + environment: { REA_LOG_LEVEL: "silent", REA_ANALYSIS_PROVIDER: "auto" }, + }); + expect(result.exitCode, result.stdout + result.stderr).toBe(0); + const contract = toolContract("decode_interface_builder"); + const evidence = contract.outputSchema.parse(result.json); + const transport = new StdioClientTransport({ + command: process.execPath, + args: [resolve("scripts/rea.mjs"), "mcp"], + env: { + PATH: process.env.PATH ?? "", + REA_LOG_LEVEL: "silent", + REA_ANALYSIS_PROVIDER: "auto", + }, + stderr: "pipe", + }); + const client = new Client({ name: "nib-field-identity", version: "1" }); + try { + await client.connect(transport); + const opened = await client.callTool({ + name: "open_binary", + arguments: { path: app }, + }); + expect(opened.isError, JSON.stringify(opened)).not.toBe(true); + const response = await client.callTool({ + name: contract.name, + arguments: {}, + }); + expect(response.isError, JSON.stringify(response)).not.toBe(true); + const mcp = contract.outputSchema.parse(response.structuredContent); + expect(mcp.normalized_result).toEqual(evidence.normalized_result); + } finally { + try { + await client.close(); + } finally { + await transport.close(); + } + } + const report = interfaceBuilderAnalysisSchema.parse( + evidence.normalized_result, + ); + expect(report.documents[0]?.archive_sha256).toBe( + createHash("sha256").update(archive).digest("hex"), + ); + const objects = report.graph.nodes.filter( + ({ attributes }) => attributes.interface_builder_object_id !== undefined, + ); + expect(objects).toHaveLength(2); + expect(objects[0]?.attributes.nibValues).toEqual({ + ["__proto__"]: { objectID: "1" }, + constructor: 7, + enabled: true, + "\uFEFFenabled": false, + }); + expect(objects[1]?.attributes).toMatchObject({ + class_name: "\uFEFFCustomView", + nibValues: { ["__proto__"]: 23 }, + }); + expect(await readFile(path)).toEqual(archive); + }, + 60_000, +); diff --git a/tests/acceptance/applications/packageSubpathsCli.test.ts b/tests/acceptance/applications/packageSubpathsCli.test.ts index 15f734e22..869a8deba 100644 --- a/tests/acceptance/applications/packageSubpathsCli.test.ts +++ b/tests/acceptance/applications/packageSubpathsCli.test.ts @@ -141,6 +141,13 @@ cliTest.for(["CLI", "stdio MCP"] as const)( ).toBe("node_modules/fi%xture/actual.cjs"); const { graph } = await analyzeJavaScriptDirectory(surface, cli, root); + expect(graph.coverage.status).toBe("complete"); + const requireFileUrl = graph.edges.find( + ({ properties }) => + properties.module_link_kind === "require" && + properties.specifier === "file:///src/uri-dep.cjs", + ); + expect(requireFileUrl?.evidence.limitations.length).toBeGreaterThan(0); const imports = graph.edges.filter( ({ properties }) => properties.module_link_kind === "require", ); diff --git a/tests/acceptance/artifactClassification.test.ts b/tests/acceptance/artifactClassification.test.ts new file mode 100644 index 000000000..7436a4967 --- /dev/null +++ b/tests/acceptance/artifactClassification.test.ts @@ -0,0 +1,111 @@ +import { createHash } from "node:crypto"; +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { Uint8ArrayReader, Uint8ArrayWriter, ZipWriter } from "@zip.js/zip.js"; +import { expect } from "vitest"; + +import { + CPU, + fatImage, + machoImage, +} from "../../src/artifacts/apple/MachoImage.fixture.js"; +import { artifactInventoryResultSchema } from "../../src/domain/artifactGraph.js"; +import { artifactInspectionResultSchema } from "../../src/domain/artifactInspection.js"; +import { parseEvidence } from "../../src/domain/evidence.js"; +import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js"; +import { cliTest } from "../support/cli/cliFixture.js"; + +cliTest.for(["jar", "zip"])( + "distinguishes JVM classes from universal Mach-O files in %s inventory", + async (extension, { cli }) => { + const root = await createTestTempDirectory("rea-class-inventory-"); + const path = join(root, `mixed.${extension}`); + // The large class can hold the entire 65-entry table suggested by its + // version bytes. Checking only the table length does not distinguish it. + const classes = [ + ["Hello.class", javaClass()], + ["Big.class", javaClass("x".repeat(4_000))], + ["renamed.bin", javaClass()], + ] as const; + const writer = new ZipWriter(new Uint8ArrayWriter()); + for (const [name, bytes] of classes) + await writer.add(name, new Uint8ArrayReader(bytes)); + const nativePaths: string[] = []; + for (const wide of [false, true]) { + for (const littleEndian of [false, true]) { + const name = `fat${wide ? 64 : 32}-${littleEndian ? "le" : "be"}`; + nativePaths.push(name); + await writer.add( + name, + new Uint8ArrayReader( + fatImage( + [{ cpu: CPU.arm64, bytes: machoImage({ cpu: CPU.arm64 }) }], + wide, + littleEndian, + ), + ), + ); + } + } + await writeFile(path, await writer.close()); + + const reply = await cli.run({ + arguments: ["inspect-artifact", path, "--json"], + }); + expect(reply.exitCode, reply.stdout + reply.stderr).toBe(0); + const inspection = artifactInspectionResultSchema.parse( + parseEvidence(reply.json).normalized_result, + ); + const inventory = artifactInventoryResultSchema.parse( + inspection.substeps[0]?.evidence.normalized_result, + ); + for (const name of nativePaths) { + expect(inventory.occurrences).toContainEqual( + expect.objectContaining({ + logical_path: name, + artifact_kind: "executable", + artifact_format: "mach-o-universal", + limitations: [], + }), + ); + } + for (const [name, bytes] of classes) { + const occurrence = inventory.occurrences.find( + (item) => item.logical_path === name, + ); + expect(occurrence).toBeDefined(); + expect.soft(occurrence?.artifact_kind, name).not.toBe("executable"); + expect.soft(occurrence?.artifact_format, name).not.toMatch(/^mach-o/u); + const sha256 = createHash("sha256").update(bytes).digest("hex"); + const node = inventory.nodes.find((item) => item.sha256 === sha256); + expect(node).toBeDefined(); + expect.soft(node?.format, name).not.toMatch(/^mach-o/u); + } + }, +); + +/** Minimal valid Java 21 class, with an optional unused UTF-8 constant. */ +const javaClass = (extra?: string): Buffer => { + const utf8 = (value: string): Buffer => { + const text = Buffer.from(value); + const header = Buffer.alloc(3); + header[0] = 1; + header.writeUInt16BE(text.length, 1); + return Buffer.concat([header, text]); + }; + const constants = [ + Buffer.from([7, 0, 2]), + utf8("Hello"), + Buffer.from([7, 0, 4]), + utf8("java/lang/Object"), + ...(extra === undefined ? [] : [utf8(extra)]), + ]; + const header = Buffer.from("cafebabe000000410000", "hex"); + header.writeUInt16BE(constants.length + 1, 8); + return Buffer.concat([ + header, + ...constants, + Buffer.from("0021000100030000000000000000", "hex"), + ]); +}; diff --git a/tests/acceptance/flutterAnalysis.test.ts b/tests/acceptance/flutterAnalysis.test.ts index 457dce96b..11671785e 100644 --- a/tests/acceptance/flutterAnalysis.test.ts +++ b/tests/acceptance/flutterAnalysis.test.ts @@ -2,6 +2,7 @@ import { createHash } from "node:crypto"; import { expect } from "vitest"; import { toolContract } from "../../src/contracts/toolContracts.js"; import { parseEvidence } from "../../src/domain/evidence.js"; +import { flutterResultSchemas } from "../../src/domain/flutter/flutterBuildAnalysis.js"; import { aotFixtureImage, engineFixtureImage, @@ -59,6 +60,9 @@ cliTest( }); expect(result.exitCode, result.stderr).toBe(0); const evidence = toolContract(name).outputSchema.parse(result.json); + const complete = flutterResultSchemas[name].parse( + evidence.normalized_result, + ); const response = await call(name, { path, ...(abi === undefined ? {} : { abi }), @@ -70,7 +74,7 @@ cliTest( expect(mcp.subject).toMatchObject({ local_path: path, format: "apk", - digest: { sha256: evidence.normalized_result.target.sha256 }, + digest: { sha256: complete.target.sha256 }, }); expect(mcp.evidence_id).toBe(evidence.evidence_id); if (name === "inspect_dart_aot") { @@ -134,6 +138,9 @@ cliTest( const evidence = toolContract("inspect_dart_aot").outputSchema.parse( result.json, ); + const complete = flutterResultSchemas.inspect_dart_aot.parse( + evidence.normalized_result, + ); const { call } = await connectLocalToolsMcp(); const response = await call("inspect_dart_aot", { path }); expect(response.isError, JSON.stringify(response)).not.toBe(true); @@ -146,7 +153,7 @@ cliTest( string_pool: { coverage: "complete" }, }); expect( - evidence.normalized_result.libapp.sections + complete.libapp.sections .filter((section) => section.name.endsWith("Data")) .map((section) => section.kind), ).toEqual([4, 4]); diff --git a/tests/acceptance/harmonyArtifactJourney.test.ts b/tests/acceptance/harmonyArtifactJourney.test.ts new file mode 100644 index 000000000..51178d744 --- /dev/null +++ b/tests/acceptance/harmonyArtifactJourney.test.ts @@ -0,0 +1,135 @@ +import { createHash } from "node:crypto"; +import { readFile, writeFile } from "node:fs/promises"; +import { resolve } from "node:path"; + +import { Client } from "@modelcontextprotocol/client"; +import { StdioClientTransport } from "@modelcontextprotocol/client/stdio"; +import { TextReader, Uint8ArrayWriter, ZipWriter } from "@zip.js/zip.js"; +import { expect } from "vitest"; + +import { artifactInspectionResultSchema } from "../../src/domain/artifactInspection.js"; +import { parseEvidence } from "../../src/domain/evidence.js"; +import { harmonyApplicationProjectionResultSchema } from "../../src/domain/harmony/harmonyApplication.js"; +import { workspaceCliTest as test } from "../support/cli/workspaceCliFixture.js"; + +test("preserves Harmony archive formats through the public CLI/MCP lifecycle", async ({ + cli, + workspace, +}) => { + const writer = new ZipWriter(new Uint8ArrayWriter()); + await writer.add( + "module.json", + new TextReader('{"module":{"name":"fixture"}}'), + ); + await writer.add( + "ets/modules.abc", + new TextReader("opaque bytecode fixture"), + ); + const bytes = await writer.close(); + const environment = { + PATH: process.env.PATH ?? "", + HOME: workspace.root, + USERPROFILE: workspace.root, + XDG_CONFIG_HOME: workspace.path("config"), + XDG_CACHE_HOME: workspace.path("cache"), + XDG_STATE_HOME: workspace.path("state"), + NODE_OPTIONS: "--max-old-space-size=512", + REA_LOG_LEVEL: "silent", + }; + const client = new Client({ name: "rea-harmony-acceptance", version: "1" }); + const transport = new StdioClientTransport({ + command: process.execPath, + args: [resolve("scripts/rea.mjs"), "mcp"], + env: environment, + stderr: "pipe", + }); + try { + await client.connect(transport); + await client.listTools(); + for (const { suffix, format } of [ + { suffix: "hap", format: "hap" }, + { suffix: "hsp", format: "hsp" }, + { suffix: "app", format: "app-pack" }, + ]) { + const path = workspace.path(`fixture.${suffix}`); + await writeFile(path, bytes); + const sha256 = createHash("sha256").update(bytes).digest("hex"); + const cliReply = await cli.run({ + arguments: ["inspect-artifact", path, "--format", "json"], + environment, + }); + expect(cliReply.exitCode, cliReply.stderr).toBe(0); + const cliEvidence = parseEvidence(cliReply.json); + const cliInventory = artifactInspectionResultSchema.parse( + cliEvidence.normalized_result, + ); + expect(cliInventory.subject).toMatchObject({ + root_format: format, + root_sha256: sha256, + }); + const opened = await client.callTool({ + name: "open_binary", + arguments: { path }, + }); + expect(opened.isError, JSON.stringify(opened)).not.toBe(true); + expect(opened.structuredContent).toMatchObject({ + result: { kind: "archive", format, sha256 }, + }); + const status = await client.callTool({ + name: "binary_session", + arguments: {}, + }); + expect(status.isError, JSON.stringify(status)).not.toBe(true); + expect(status.structuredContent).toMatchObject({ + result: { open: true, kind: "archive", format, sha256 }, + }); + const inspected = await client.callTool({ + name: "inspect_artifact", + arguments: {}, + }); + expect(inspected.isError, JSON.stringify(inspected)).not.toBe(true); + const inventoryEvidence = parseEvidence(inspected.structuredContent); + expect( + artifactInspectionResultSchema.parse( + inventoryEvidence.normalized_result, + ), + ).toEqual(cliInventory); + + const projected = await client.callTool({ + name: "project_harmony_application_graph", + arguments: { inventory_evidence: [inventoryEvidence] }, + }); + expect(projected.isError, JSON.stringify(projected)).not.toBe(true); + const graph = harmonyApplicationProjectionResultSchema.parse( + parseEvidence(projected.structuredContent).normalized_result, + ); + expect(graph).toMatchObject({ + root_format: format, + root_sha256: sha256, + coverage: { status: "complete-within-inventory" }, + }); + expect( + graph.components.manifests.map(({ path: member }) => member), + ).toEqual(["module.json"]); + expect( + graph.components.bytecode.map(({ path: member }) => member), + ).toEqual(["ets/modules.abc"]); + const closed = await client.callTool({ + name: "close_binary", + arguments: {}, + }); + expect(closed.isError, JSON.stringify(closed)).not.toBe(true); + const empty = await client.callTool({ + name: "binary_session", + arguments: {}, + }); + expect(empty.isError, JSON.stringify(empty)).not.toBe(true); + expect(empty.structuredContent).toMatchObject({ + result: { open: false }, + }); + expect(await readFile(path)).toEqual(Buffer.from(bytes)); + } + } finally { + await client.close(); + } +}, 120_000); diff --git a/tests/acceptance/setup/clientConfigurationDiagnosticsCli.test.ts b/tests/acceptance/setup/clientConfigurationDiagnosticsCli.test.ts index d9363988f..425e0e610 100644 --- a/tests/acceptance/setup/clientConfigurationDiagnosticsCli.test.ts +++ b/tests/acceptance/setup/clientConfigurationDiagnosticsCli.test.ts @@ -6,6 +6,17 @@ import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { cliTest } from "../../support/cli/cliFixture.js"; const cases = [ + { + client: "cursor", + path: ".cursor/mcp.json", + text: "malformed UTF-8", + bytes: Buffer.concat([ + Buffer.from('{"label":"'), + Buffer.from([0xff]), + Buffer.from('"}'), + ]), + diagnostic: "not valid UTF-8", + }, { client: "cursor", path: ".cursor/mcp.json", @@ -92,7 +103,9 @@ for (const scenario of cases) ); const config = join(home, scenario.path); await mkdir(dirname(config), { recursive: true }); - await writeFile(config, scenario.text); + const original = + "bytes" in scenario ? scenario.bytes : Buffer.from(scenario.text); + await writeFile(config, original); const environment = { HOME: home, USERPROFILE: home, @@ -146,7 +159,7 @@ for (const scenario of cases) }); if (!("syntax" in scenario)) expect(uninstall.stdout).not.toContain("not valid JSON"); - expect(await readFile(config, "utf8")).toBe(scenario.text); + expect(await readFile(config)).toEqual(original); expect(await readdir(dirname(config))).toEqual([ scenario.path.split("/").at(-1), ]); diff --git a/tests/acceptance/setup/dshClientSetup.test.ts b/tests/acceptance/setup/dshClientSetup.test.ts index 4010369cd..b544c5816 100644 --- a/tests/acceptance/setup/dshClientSetup.test.ts +++ b/tests/acceptance/setup/dshClientSetup.test.ts @@ -1,5 +1,5 @@ import { mkdir, readFile, writeFile } from "node:fs/promises"; -import { join } from "node:path"; +import { dirname, join } from "node:path"; import { Client } from "@modelcontextprotocol/client"; import { StdioClientTransport } from "@modelcontextprotocol/client/stdio"; import { expect } from "vitest"; @@ -8,6 +8,111 @@ import { z } from "zod"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { cliTest } from "../../support/cli/cliFixture.js"; +cliTest.for([ + { + label: "blank defaults", + harness: "", + agents: " \t ", + dshRoot: "home/.dsh", + agentsRoot: "home/.agents", + }, + { + label: "tilde roots", + harness: "~/selected harness", + agents: "~\\shared agents", + dshRoot: "home/selected harness", + agentsRoot: "home/shared agents", + }, + { + label: "relative roots", + harness: "./selected harness", + agents: "../shared agents", + dshRoot: "workspace/selected harness", + agentsRoot: "shared agents", + }, +])( + "DSH CLI uses Harness path semantics through setup, doctor and uninstall for $label", + async ({ harness, agents, dshRoot, agentsRoot }, { cli }) => { + const root = await createTestTempDirectory("rea-dsh-paths-"); + const home = join(root, "home"); + const cwd = join(root, "workspace"); + const config = join(root, dshRoot, "cordis.patch.yml"); + const skillRoot = join( + root, + agentsRoot, + "skills", + "reverse-engineer-anything", + ); + const skill = join(skillRoot, "SKILL.md"); + await mkdir(home); + await mkdir(cwd); + await mkdir(dirname(config), { recursive: true }); + const original = + "# caller patch\n- insert: [{id: caller, name: caller-plugin}]\n"; + await writeFile(config, original); + const run = (args: readonly string[]) => + cli.run({ + arguments: [...args, "--json"], + cwd, + environment: { + HOME: home, + USERPROFILE: home, + DSH_HOME: harness, + DSH_AGENTS_HOME: agents, + }, + }); + const planned = await run(["setup", "--client", "dsh", "--dry-run"]); + expect(planned.exitCode).toBe(0); + expect(planned.json).toMatchObject({ + status: "planned", + appliedActions: [], + plannedActions: expect.arrayContaining([ + expect.objectContaining({ id: "configure_client:dsh", target: config }), + expect.objectContaining({ kind: "install_skill", target: skillRoot }), + ]), + }); + expect(await readFile(config, "utf8")).toBe(original); + await expect(readFile(skill)).rejects.toMatchObject({ code: "ENOENT" }); + const configured = await run(["setup", "--client", "dsh", "--yes"]); + expect(configured.exitCode).toBe(0); + expect(configured.json).toMatchObject({ + status: "ready", + clients: { dsh: { status: "configured" } }, + }); + const text = await readFile(config, "utf8"); + expect(text).toContain("# caller patch"); + expect(text).toContain("rea-mcp"); + expect(await readFile(`${config}.rea.backup`, "utf8")).toBe(original); + expect(await readFile(skill, "utf8")).toContain( + "name: reverse-engineer-anything", + ); + const doctor = await run(["doctor", "--client", "dsh", "--skill"]); + expect(doctor.exitCode).toBe(0); + expect(doctor.json).toMatchObject({ healthy: true }); + const repeated = await run(["setup", "--client", "dsh", "--yes"]); + expect(repeated.exitCode).toBe(0); + expect(repeated.json).toMatchObject({ + status: "ready", + plannedActions: [], + appliedActions: [], + }); + expect(await readFile(config, "utf8")).toBe(text); + const removed = await run(["uninstall"]); + expect(removed.exitCode).toBe(0); + expect(removed.json).toMatchObject({ + status: "complete", + items: expect.arrayContaining([ + expect.objectContaining({ name: "dsh", status: "removed" }), + ]), + }); + expect(parse(await readFile(config, "utf8"))).toEqual([ + { insert: [{ id: "caller", name: "caller-plugin" }] }, + ]); + expect(await readFile(`${config}.rea.backup`, "utf8")).toBe(original); + await expect(readFile(skill)).rejects.toMatchObject({ code: "ENOENT" }); + }, +); + cliTest( "DSH CLI previews, approves, repairs, launches MCP, repeats and uninstalls while preserving caller YAML", async ({ cli }) => { diff --git a/tests/acceptance/tools/historicalCaptureResources.test.ts b/tests/acceptance/tools/historicalCaptureResources.test.ts index d4c56d644..64a7b6638 100644 --- a/tests/acceptance/tools/historicalCaptureResources.test.ts +++ b/tests/acceptance/tools/historicalCaptureResources.test.ts @@ -5,7 +5,7 @@ import { fileURLToPath } from "node:url"; import { promisify } from "node:util"; import { expect, it } from "vitest"; -import { HAR_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES } from "../../../src/browser/history/CaptureRelease.js"; +import { NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES } from "../../../src/browser/history/CaptureRelease.js"; import { WEB_NETWORK_CAPTURE_LIMITS } from "../../../src/domain/webNetworkCapture.js"; import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js"; import { parseMcpToolError } from "../../fixtures/mcpToolError.js"; @@ -46,7 +46,7 @@ const capture = (count: number) => // Node.js enforces the decoder's V8 heap flags. Bun ignores them; on Windows // the decoder's Job Object commit limit is the enforced bound instead, and // Bun elsewhere has no bound this capture can reach. -const jobLimit = HAR_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES; +const jobLimit = NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES; it.skipIf(process.versions.bun !== undefined && jobLimit === null)( "reports real HAR memory exhaustion through CLI and MCP and preserves server usability", async () => { diff --git a/tests/acceptance/tools/qlogCapture.test.ts b/tests/acceptance/tools/qlogCapture.test.ts new file mode 100644 index 000000000..be67c3a34 --- /dev/null +++ b/tests/acceptance/tools/qlogCapture.test.ts @@ -0,0 +1,457 @@ +import { execFile } from "node:child_process"; +import { writeFile, truncate, rm } from "node:fs/promises"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { promisify } from "node:util"; +import { expect, it as runtimeTest } from "vitest"; +import { + NODE_CAPTURE_HEAP_ENFORCED, + NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES, +} from "../../../src/browser/history/CaptureRelease.js"; +import { webNetworkCaptureSummaryViewResultSchema } from "../../../src/domain/analysisView/analysisView.js"; +import { parseEvidence } from "../../../src/domain/evidence.js"; +import { + webNetworkCaptureSchema, + WEB_NETWORK_CAPTURE_LIMITS, +} from "../../../src/domain/webNetworkCapture.js"; +import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js"; +import { parseMcpToolError } from "../../fixtures/mcpToolError.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +const execute = promisify(execFile); +const hasMemoryBound = + NODE_CAPTURE_HEAP_ENFORCED || + NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES !== null; +const it = runtimeTest.skipIf(!hasMemoryBound); +const cli = fileURLToPath(new URL("../../../scripts/rea.mjs", import.meta.url)); +const event = { + name: "transport:packet_sent", + time: 1791819895000.25, + data: { + header: { packet_type: "1RTT", packet_number: 5 }, + frames: [ + { frame_type: "stream", stream_id: 0, offset: 10, length: 5, fin: true }, + ], + }, +}; +const qlog = (events: unknown[]) => + JSON.stringify({ + qlog_format: "JSON", + qlog_version: "0.3", + traces: [ + { + vantage_point: { name: "aioquic", type: "server" }, + common_fields: { ODCID: "trace-local" }, + events, + }, + ], + }); + +const assertAdditionalQlogFacets = async ( + call: Awaited>["call"], + retained: { kind: string; evidence_id: string }, +): Promise => { + for (const [view, expected] of [ + [ + { kind: "qlog", facet: "traces", offset: 1, limit: 1 }, + [{ trace_index: 1, event_count: 1, stream_count: 1 }], + ], + [ + { kind: "qlog", facet: "loss-declarations", trace_index: 0 }, + [{ record_ordinal: 1, packet_number: { state: "reported", value: "5" } }], + ], + [{ kind: "qlog", facet: "incomplete-records", trace_index: 0 }, []], + [ + { + kind: "qlog", + facet: "ranges", + trace_index: 0, + stream_id: "0", + direction: "sent", + }, + [{ start: "10", end: "15" }], + ], + [ + { + kind: "qlog", + facet: "final-sizes", + trace_index: 0, + stream_id: "0", + direction: "sent", + }, + ["15"], + ], + [{ kind: "qlog", facet: "traces", offset: 99 }, []], + ] as const) { + const response = await call("inspect_analysis_view", { + source: retained, + view, + }); + expect(response.isError, JSON.stringify(response)).not.toBe(true); + expect( + parseEvidence(response.structuredContent).normalized_result, + ).toMatchObject({ + page: { items: expected }, + coverage: { exhausted: true }, + }); + } +}; + +it("pages original trace-scoped facts after source deletion without conflating selected events or identical stream IDs", async () => { + const root = await createTestTempDirectory("rea-qlog-pages-"); + const path = join(root, "input.qlog"); + const source = JSON.parse( + qlog([ + event, + { + name: "recovery:packet_lost", + time: 1791819895001, + data: { packet_number: 5, type: "1RTT" }, + }, + ]), + ); + source.traces.push({ + ...source.traces[0], + events: [ + { + ...event, + data: { + frames: [ + { + frame_type: "stream", + stream_id: 0, + offset: 100, + length: 3, + fin: true, + }, + ], + }, + }, + ], + }); + await writeFile(path, JSON.stringify(source)); + const { call } = await connectLocalToolsMcp(); + const response = await call("inspect_web_network_capture", { + capture_path: path, + format: "qlog", + detail: "summary", + record_ordinals: [2], + }); + expect(response.isError, JSON.stringify(response)).not.toBe(true); + const summary = webNetworkCaptureSummaryViewResultSchema.parse( + parseEvidence(response.structuredContent).normalized_result, + ); + expect(summary).toMatchObject({ + summary: { + total_records: 3, + retained_records: 1, + qlog: { + trace_count: 2, + retained_stream_count: 2, + loss_declaration_count: 1, + }, + }, + }); + expect(JSON.stringify(summary)).not.toContain('"contributions"'); + if ( + !("parent_evidence_id" in summary) || + typeof summary.parent_evidence_id !== "string" + ) + throw new Error("Missing retained parent"); + const retained = { + kind: "retained-evidence", + evidence_id: summary.parent_evidence_id, + }; + const parent = parseEvidence( + ( + await call("inspect_web_network_capture", { + capture_path: path, + format: "qlog", + record_ordinals: [2], + }) + ).structuredContent, + ); + expect(parent.evidence_id).toBe(summary.parent_evidence_id); + const portablePath = join(root, "parent.json"); + await rm(path); + const pages = []; + for (const trace_index of [0, 1]) { + const page = await call("inspect_analysis_view", { + source: retained, + view: { kind: "qlog", facet: "streams", trace_index, limit: 1 }, + }); + expect(page.isError, JSON.stringify(page)).not.toBe(true); + pages.push(parseEvidence(page.structuredContent).normalized_result); + } + expect(pages[0]).toMatchObject({ + page: { items: [{ stream_id: "0", unique_bytes: "5" }] }, + context: { scope: "all-recorded-events-before-selection" }, + }); + expect(pages[1]).toMatchObject({ + page: { items: [{ stream_id: "0", unique_bytes: "3" }] }, + }); + await assertAdditionalQlogFacets(call, retained); + const view = { + kind: "qlog", + facet: "contributions", + trace_index: 0, + stream_id: "0", + direction: "sent", + offset: 0, + limit: 1, + }; + const contribution = parseEvidence( + (await call("inspect_analysis_view", { source: retained, view })) + .structuredContent, + ); + expect(contribution.normalized_result).toMatchObject({ + page: { + items: [{ record_ordinal: 0, frame_index: 0, offset: "10", length: "5" }], + }, + coverage: { total: 1, exhausted: true }, + }); + await writeFile( + portablePath, + JSON.stringify({ source: { kind: "inline", evidence: parent }, view }), + ); + const cliResult = await execute(process.execPath, [ + cli, + "inspect-analysis-view", + portablePath, + "--json", + ]); + expect(parseEvidence(JSON.parse(cliResult.stdout)).normalized_result).toEqual( + contribution.normalized_result, + ); + for (const invalid of [ + { kind: "qlog", facet: "streams" }, + { kind: "qlog", facet: "streams", trace_index: 99 }, + { ...view, stream_id: "99" }, + { kind: "qlog", facet: "traces", trace_index: 0 }, + ]) + expect( + parseMcpToolError( + await call("inspect_analysis_view", { + source: retained, + view: invalid, + }), + ), + ).toMatchObject({ error: { code: "invalid_request" } }); +}, 60_000); + +runtimeTest.runIf(!hasMemoryBound)( + "refuses qlog when this runtime cannot enforce a decoder memory bound", + async () => { + const { call } = await connectLocalToolsMcp(); + const root = await createTestTempDirectory("rea-qlog-unbounded-"); + const path = join(root, "input.qlog"); + await writeFile(path, qlog([event])); + expect( + parseMcpToolError( + await call("inspect_web_network_capture", { + capture_path: path, + format: "qlog", + }), + ), + ).toMatchObject({ error: { code: "capability_unavailable" } }); + }, +); + +it("runs built CLI and real stdio SDK with exact evidence parity, selection, retained references and strict schemas", async () => { + const root = await createTestTempDirectory("rea-qlog-public-"); + const path = join(root, "input.qlog"); + await writeFile( + path, + qlog([ + event, + { + name: "recovery:packet_lost", + time: 1791819895001.125, + data: { packet_number: 5, type: "1RTT" }, + }, + ]), + ); + const { call, client } = await connectLocalToolsMcp(); + const input = { capture_path: path, format: "qlog", record_ordinals: [1, 0] }; + const response = await call("inspect_web_network_capture", input); + expect(response.isError, JSON.stringify(response)).not.toBe(true); + const mcp = parseEvidence(response.structuredContent); + const result = await execute(process.execPath, [ + cli, + "inspect-web-network-capture", + path, + "qlog", + "--record", + "1", + "--record", + "0", + "--json", + ]); + const cliEvidence = parseEvidence(JSON.parse(result.stdout)); + expect(cliEvidence.evidence_id).toBe(mcp.evidence_id); + expect(cliEvidence.normalized_result).toEqual(mcp.normalized_result); + expect(mcp.analysis_profile?.parameters).toMatchObject({ + qlog_profile: "aioquic-1.5.0-json-0.3", + }); + expect(mcp.authority).toBe("historical-reference"); + const decoded = webNetworkCaptureSchema.parse(mcp.normalized_result); + expect(decoded.records.map((record) => record.ordinal)).toEqual([1, 0]); + if (decoded.format !== "qlog") throw new Error("qlog format required"); + expect(decoded.qlog.traces[0]?.streams[0]).toMatchObject({ + unique_bytes: "5", + covered_ranges: [{ start: "10", end: "15" }], + }); + const retained = await call("inspect_analysis_view", { + source: { kind: "retained-evidence", evidence_id: mcp.evidence_id }, + view: { kind: "api-surface" }, + }); + // An unsupported view over an existing retained parent proves lookup without pretending qlog is HAR. + expect(parseMcpToolError(retained)).toMatchObject({ + error: { code: "unsupported_target" }, + }); + await client.ping(); + for (const record_ordinals of [[2], [0, 0]]) { + expect( + parseMcpToolError( + await call("inspect_web_network_capture", { + ...input, + record_ordinals, + }), + ), + ).toMatchObject({ error: { code: "invalid_request" } }); + } + const hidden = await call("inspect_web_network_capture", { + capture_path: path, + format: "qlog", + sensitive_values: ["trace-local", "/traces/0", "input.qlog"], + }); + expect(JSON.stringify(hidden)).not.toContain("trace-local"); + expect(JSON.stringify(hidden)).not.toContain("/traces/0"); + expect(JSON.stringify(hidden)).not.toContain("input.qlog"); + const hiddenEvidence = parseEvidence(hidden.structuredContent); + expect( + webNetworkCaptureSchema.parse(hiddenEvidence.normalized_result).artifact + .path, + ).toBe(""); + const invalid = await call("inspect_web_network_capture", { + ...input, + mysecret: true, + sensitive_values: ["mysecret"], + }); + expect(parseMcpToolError(invalid)).toMatchObject({ + error: { code: "invalid_request" }, + }); + expect(JSON.stringify(invalid)).not.toContain("mysecret"); +}, 60_000); + +it("fails whole-capture validation before selection and survives malformed, unsupported and oversized qlog", async () => { + const root = await createTestTempDirectory("rea-qlog-errors-"); + const path = join(root, "input.qlog"); + const { call, client } = await connectLocalToolsMcp(); + for (const [text, code] of [ + [qlog([event, { ...event, time: "invalid" }]), "invalid_request"], + [qlog([event]).slice(0, -1), "invalid_request"], + [ + qlog([event]).replace('"qlog_version":"0.3"', '"qlog_version":"0.4"'), + "capability_unavailable", + ], + [ + qlog([ + { + ...event, + extension: JSON.parse("[".repeat(65) + "0" + "]".repeat(65)), + }, + ]), + "invalid_request", + ], + ] as const) { + await writeFile(path, text); + expect( + parseMcpToolError( + await call("inspect_web_network_capture", { + capture_path: path, + format: "qlog", + record_ordinals: [0], + }), + ), + ).toMatchObject({ error: { code } }); + const failed: unknown = await execute(process.execPath, [ + cli, + "inspect-web-network-capture", + path, + "qlog", + "--json", + ]).catch((cause: unknown) => cause); + expect(failed).toMatchObject({ + code: 1, + stdout: expect.stringContaining(`"code":"${code}"`), + }); + } + await truncate(path, WEB_NETWORK_CAPTURE_LIMITS.inputBytes + 1); + expect( + parseMcpToolError( + await call("inspect_web_network_capture", { + capture_path: path, + format: "qlog", + }), + ), + ).toMatchObject({ error: { code: "invalid_request" } }); + await writeFile(path, qlog([event])); + await client.ping(); + expect( + ( + await call("inspect_web_network_capture", { + capture_path: path, + format: "qlog", + }) + ).isError, + ).not.toBe(true); +}, 60_000); + +runtimeTest.skipIf(process.versions.bun !== undefined || !hasMemoryBound)( + "contains actual qlog decoder heap exhaustion and keeps MCP usable", + async () => { + const root = await createTestTempDirectory("rea-qlog-capacity-"); + const path = join(root, "dense.qlog"); + const dense = qlog(Array(100_000).fill(event)); + expect(Buffer.byteLength(dense)).toBeLessThan( + WEB_NETWORK_CAPTURE_LIMITS.inputBytes, + ); + await writeFile(path, dense); + const { call, client } = await connectLocalToolsMcp(); + const response = await call("inspect_web_network_capture", { + capture_path: path, + format: "qlog", + record_ordinals: [0], + }); + expect(parseMcpToolError(response)).toMatchObject({ + error: { + code: "resource_constraint", + details: { + resource: "memory", + reported_limits: { old_generation_heap_mib: 192 }, + }, + }, + }); + const failed: unknown = await execute( + process.execPath, + [cli, "inspect-web-network-capture", path, "qlog", "--json"], + { timeout: 40_000, maxBuffer: 1024 * 1024 }, + ).catch((cause: unknown) => cause); + expect(failed).toMatchObject({ + code: 1, + stdout: expect.stringContaining('"code":"resource_constraint"'), + }); + await writeFile(path, qlog([event])); + await client.ping(); + expect( + ( + await call("inspect_web_network_capture", { + capture_path: path, + format: "qlog", + }) + ).isError, + ).not.toBe(true); + }, + 60_000, +); diff --git a/tests/boundary/android/jadxMcpTransport.test.ts b/tests/boundary/android/jadxMcpTransport.test.ts index afe79fc09..fe05f14b4 100644 --- a/tests/boundary/android/jadxMcpTransport.test.ts +++ b/tests/boundary/android/jadxMcpTransport.test.ts @@ -26,6 +26,26 @@ class FixtureProcess extends EventEmitter implements ProviderProcessHandle { } } +const fixtureLauncher = + (process: FixtureProcess): JadxLauncher => + async ({ runId }) => ({ + process, + ownership: { + runId, + leaderPid: process.pid, + processGroupId: process.pid, + expectedParentPid: process.pid, + }, + cleanup: async () => { + process.emit("exit", 0, null); + process.emit("close", 0, null); + process.stdout.end(); + process.stderr.end(); + process.stdin.end(); + return { cleaned: true, signaled: false }; + }, + }); + it("joins pending input errors before detaching the owned protocol stream", async () => { let completeWrite: ((cause: Error) => void) | undefined; const failure = new Error("owned pipe closed during cancellation"); @@ -70,26 +90,9 @@ it("joins pending input errors before detaching the owned protocol stream", asyn it("decodes split UTF-8 and newline bytes, then routes multiple complete frames", async () => { const process = new FixtureProcess(); - const launcher: JadxLauncher = async ({ runId }) => ({ - process, - ownership: { - runId, - leaderPid: process.pid, - processGroupId: process.pid, - expectedParentPid: process.pid, - }, - cleanup: async () => { - process.emit("exit", 0, null); - process.emit("close", 0, null); - process.stdout.end(); - process.stderr.end(); - process.stdin.end(); - return { cleaned: true, signaled: false }; - }, - }); const transport = new JadxMcpTransport( { command: "fixture", arguments: [] }, - launcher, + fixtureLauncher(process), ); const received: JSONRPCMessage[] = []; transport.onmessage = (message) => received.push(message); @@ -98,7 +101,9 @@ it("decodes split UTF-8 and newline bytes, then routes multiple complete frames" const first = { jsonrpc: "2.0", id: 1, - result: { content: [{ type: "text", text: "snow 雪 🦊" }] }, + result: { + content: [{ type: "text", text: "snow 雪 🦊 and literal �" }], + }, } satisfies JSONRPCMessage; const second = { jsonrpc: "2.0", @@ -120,28 +125,41 @@ it("decodes split UTF-8 and newline bytes, then routes multiple complete frames" await transport.close(); }); -it("accepts a complete JSON-RPC frame at the 8 MiB boundary", async () => { +it.each([ + ["invalid UTF-8 byte", Buffer.from([0xff])], + ["truncated UTF-8 sequence", Buffer.from([0xe2, 0x82])], +])("rejects %s in a complete JSON-RPC frame", async (_label, invalid) => { const process = new FixtureProcess(); - const launcher: JadxLauncher = async ({ runId }) => ({ - process, - ownership: { - runId, - leaderPid: process.pid, - processGroupId: process.pid, - expectedParentPid: process.pid, - }, - cleanup: async () => { - process.emit("exit", 0, null); - process.emit("close", 0, null); - process.stdout.end(); - process.stderr.end(); - process.stdin.end(); - return { cleaned: true, signaled: false }; - }, - }); const transport = new JadxMcpTransport( { command: "fixture", arguments: [] }, - launcher, + fixtureLauncher(process), + ); + const received: JSONRPCMessage[] = []; + const errors: Error[] = []; + transport.onmessage = (message) => received.push(message); + transport.onerror = (cause) => errors.push(cause); + + await transport.start(); + process.stdout.write( + Buffer.concat([ + Buffer.from('{"jsonrpc":"2.0","id":1,"result":{"text":"'), + invalid, + Buffer.from('"}}\n'), + ]), + ); + await new Promise((resolve) => setImmediate(resolve)); + await transport.close(); + + expect(received).toEqual([]); + expect(errors).toHaveLength(1); + expect(errors[0]?.message).toMatch(/encoded data was not valid/i); +}); + +it("accepts a complete JSON-RPC frame at the 8 MiB boundary", async () => { + const process = new FixtureProcess(); + const transport = new JadxMcpTransport( + { command: "fixture", arguments: [] }, + fixtureLauncher(process), ); const received: JSONRPCMessage[] = []; transport.onmessage = (message) => received.push(message); diff --git a/tests/boundary/binaryDiagnostics/pwntoolsInputBoundAdmission.test.ts b/tests/boundary/binaryDiagnostics/pwntoolsInputBoundAdmission.test.ts new file mode 100644 index 000000000..ca9f0bc2a --- /dev/null +++ b/tests/boundary/binaryDiagnostics/pwntoolsInputBoundAdmission.test.ts @@ -0,0 +1,89 @@ +import { open } from "node:fs/promises"; +import { join } from "node:path"; + +import { expect, it, onTestFinished } from "vitest"; + +import { PwntoolsLayoutProvider } from "../../../src/native/pwntools/PwntoolsLayoutProvider.js"; +import { projectAnalysisError } from "../../../src/domain/analysisErrorProjection.js"; +import { spawnOwnedProviderProcess } from "../../../src/process/ProviderProcess.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +const unsupportedHost = process.platform !== "linux" || process.arch !== "x64"; +const DEFAULT_INPUT_BYTES = 33_554_432; +// One byte above the shipped bound, so the admission decision is the only +// difference between the first two cases. +const ARTIFACT_BYTES = DEFAULT_INPUT_BYTES + 1; + +/** Sparse regular file just larger than the shipped input bound. */ +const oversizedArtifact = async (): Promise => { + const root = await createTestTempDirectory("rea-pwntools-input-bound-"); + const path = join(root, "recorded-core"); + const handle = await open(path, "w"); + await handle.truncate(ARTIFACT_BYTES); + await handle.close(); + return path; +}; + +/** One admission attempt, reported as the projected caller-visible failure. */ +const inspect = async (override?: string) => { + const path = await oversizedArtifact(); + const provider = new PwntoolsLayoutProvider( + { + REA_PWNTOOLS_PYTHON: process.execPath, + ...(override === undefined + ? {} + : { REA_PWNTOOLS_MAX_INPUT_BYTES: override }), + }, + async (spawn) => + spawnOwnedProviderProcess({ + ...spawn, + command: process.execPath, + arguments: ["-e", "process.stdout.write('no reply')"], + }), + ); + onTestFinished(async () => { + await provider.close(); + }); + const result = await provider.inspect({ path }); + if (result.ok) throw new Error("Expected the bounded decoder to fail"); + const projected = projectAnalysisError(result.error); + return { projected, text: JSON.stringify(projected) }; +}; + +it.runIf(!unsupportedHost)( + "refuses an artifact above the shipped bound when no override is set", + async () => { + const { projected, text } = await inspect(); + expect(projected.code).toBe("invalid_request"); + expect(text).toContain( + `Selected artifact exceeds ${String(DEFAULT_INPUT_BYTES)} bytes`, + ); + }, +); + +it.runIf(!unsupportedHost)( + "admits the same artifact once the override widens the bound", + async () => { + const { projected, text } = await inspect(String(ARTIFACT_BYTES)); + // Admission passed, so the only remaining failure is that the owned decoder + // never wrote a reply, which also proves the process really ran. + expect(text).not.toContain("Selected artifact exceeds"); + expect(projected.code).toBe("unreadable_output"); + expect(text).toContain("no reply"); + }, +); + +it.runIf(!unsupportedHost)( + "refuses an unusable override before reading the artifact", + async () => { + const { projected, text } = await inspect("0"); + expect(projected.code).toBe("provider_unavailable"); + expect(projected.details).toMatchObject({ + rejections: [ + { diagnostics: { configuration_key: "REA_PWNTOOLS_MAX_INPUT_BYTES" } }, + ], + }); + expect(text).toContain("REA_PWNTOOLS_MAX_INPUT_BYTES must be a decimal"); + expect(text).not.toContain("Selected artifact exceeds"); + }, +); diff --git a/tests/boundary/browser/cdpDomSvgIdentity.test.ts b/tests/boundary/browser/cdpDomSvgIdentity.test.ts new file mode 100644 index 000000000..7d42ae336 --- /dev/null +++ b/tests/boundary/browser/cdpDomSvgIdentity.test.ts @@ -0,0 +1,315 @@ +import { describe, expect, it } from "vitest"; + +import { CdpBrowserProvider } from "../../../src/browser/CdpBrowserProvider.js"; +import { inspectWebPageInputSchema } from "../../../src/domain/browserObservation.js"; +import { + startFakeCdpBrowser, + type FakeCdpBrowser, + type FakeOptions, +} from "../../fixtures/fakeCdpBrowser.js"; +import { trackBrowser } from "./cdpBrowserProvider.support.js"; + +const startBrowser = async (options: FakeOptions = {}) => { + const browser = await startFakeCdpBrowser({ + ...options, + commandResult(command, origin, frameReads) { + const override = options.commandResult?.(command, origin, frameReads); + if (override !== undefined) return override; + if (command.method === "DOMSnapshot.captureSnapshot") { + const nodes = { + nodeType: [9, 1, 1, 1, 1], + nodeName: [1, 2, 3, 3, 3], + nodeValue: [4, 4, 4, 4, 4], + parentIndex: [-1, 0, 1, 1, 1], + backendNodeId: [1, 2, 3, 4, 5], + attributes: [[], [], [5, 6], [5, 7], [5, 6, 8, 9]], + }; + return { + strings: [ + `${origin}/app`, + "#document", + "HTML", + "a", + "", + "xlink:href", + "/first", + "/second", + "href", + "/ordinary", + "https://excluded.example/page", + ], + documents: [ + { documentURL: 0, nodes }, + { + documentURL: 10, + nodes: { ...nodes, backendNodeId: [11, 12, 13, 14, 15] }, + }, + ], + }; + } + if (command.method === "DOM.describeNode") + return { + node: { + backendNodeId: command.params.backendNodeId, + nodeType: 1, + nodeName: "a", + isSVG: true, + }, + }; + return undefined; + }, + }); + trackBrowser(browser); + return browser; +}; + +const inspect = (browser: FakeCdpBrowser, signal?: AbortSignal) => + new CdpBrowserProvider().inspectPage( + inspectWebPageInputSchema.parse({ + cdp_endpoint: browser.endpoint, + target_id: "allowed-page", + allowed_origins: [browser.allowedOrigin], + observation_ms: 0, + }), + signal === undefined ? undefined : { signal }, + ); + +describe("CDP SVG identity at the snapshot boundary", () => { + it("describes only in-scope XLink candidates without a winning href", async () => { + const browser = await startBrowser(); + const result = await inspect(browser); + if (!result.ok) throw result.error; + expect(result.value.metadata.dom_urls.map(({ url }) => url)).toEqual( + ["/first", "/second", "/ordinary"].map( + (path) => browser.allowedOrigin + path, + ), + ); + expect( + browser.commands + .filter(({ method }) => method === "DOM.describeNode") + .map(({ params }) => params), + ).toEqual([ + { backendNodeId: 3, depth: 0 }, + { backendNodeId: 4, depth: 0 }, + ]); + }); + + it.each(["removed", "mismatched"])( + "retains partial metadata when a candidate is %s", + async (failure) => { + const browser = await startBrowser({ + commandError: (command) => + failure === "removed" && + command.method === "DOM.describeNode" && + command.params.backendNodeId === 3 + ? { code: -32_000, message: "No node found for given backend id" } + : undefined, + commandResult: (command) => + failure === "mismatched" && + command.method === "DOM.describeNode" && + command.params.backendNodeId === 3 + ? { + node: { + backendNodeId: 99, + nodeType: 1, + nodeName: "a", + isSVG: true, + }, + } + : undefined, + }); + const result = await inspect(browser); + if (!result.ok) throw result.error; + expect(result.value.metadata.dom_urls.map(({ url }) => url)).toEqual( + ["/second", "/ordinary"].map((path) => browser.allowedOrigin + path), + ); + expect(result.value.completeness.excluded).toContainEqual({ + section: "metadata", + reason: "provider_unavailable", + count: 1, + }); + expect(result.value.limitations).toContain( + "SVG identity was unavailable for 1 snapshotted XLink candidate nodes; their URL destinations remain unknown.", + ); + }, + ); + + it("stops querying after the first batch reports an unsupported method", async () => { + const browser = await startBrowser({ + commandResult: manyCandidates, + unsupportedMethods: ["DOM.describeNode"], + }); + const result = await inspect(browser); + if (!result.ok) throw result.error; + expect(result.value.metadata.dom_urls).toEqual([]); + expect(result.value.completeness.excluded).toContainEqual({ + section: "metadata", + reason: "provider_unavailable", + count: 25, + }); + expect(descriptions(browser).length).toBeGreaterThan(0); + expect(descriptions(browser).length).toBeLessThanOrEqual(8); + }); + + it.each([ + { options: { closeOnMethod: "DOM.describeNode" }, reason: "disconnected" }, + { + options: { + commandError: (command: { method: string }) => + command.method === "DOM.describeNode" + ? { code: -32_602, message: "Invalid parameters" } + : undefined, + }, + reason: "protocol_error", + }, + ])( + "propagates $reason during node description", + async ({ options, reason }) => { + const browser = await startBrowser(options); + expect(await inspect(browser)).toMatchObject({ + ok: false, + error: { _tag: "BrowserObservationError", reason }, + }); + }, + ); + + it("cancels a pending description and detaches without closing the target", async () => { + const browser = await startBrowser({ hangOnMethod: "DOM.describeNode" }); + const controller = new AbortController(); + const pending = inspect(browser, controller.signal); + await expect.poll(() => descriptions(browser).length).toBeGreaterThan(1); + controller.abort(); + expect(await pending).toMatchObject({ + ok: false, + error: { _tag: "AnalysisCancelledError", operation: "inspect_web_page" }, + }); + const methods = browser.commands.map(({ method }) => method); + expect(methods).toContain("Target.detachFromTarget"); + expect(methods).not.toContain("Target.closeTarget"); + expect(methods).not.toContain("Browser.close"); + }); +}); + +const manyCandidates: NonNullable = ( + command, + origin, +) => { + if (command.method !== "DOMSnapshot.captureSnapshot") return undefined; + const count = 25; + return { + strings: [ + origin, + "#document", + "HTML", + "a", + "", + "xlink:href", + ...Array.from({ length: count }, (_, index) => `/link-${String(index)}`), + ], + documents: [ + { + documentURL: 0, + nodes: { + nodeType: [9, 1, ...Array(count).fill(1)], + nodeName: [1, 2, ...Array(count).fill(3)], + nodeValue: Array(count + 2).fill(4), + parentIndex: [-1, 0, ...Array(count).fill(1)], + backendNodeId: Array.from( + { length: count + 2 }, + (_, index) => index + 1, + ), + attributes: [ + [], + [], + ...Array.from({ length: count }, (_, index) => [5, index + 6]), + ], + }, + }, + ], + }; +}; + +const descriptions = (browser: FakeCdpBrowser) => + browser.commands.filter(({ method }) => method === "DOM.describeNode"); + +describe("CDP SVG query concurrency", () => { + it("bounds in-flight reads and preserves metadata order across reversed replies", async () => { + const browser = await startBrowser({ + commandResult: manyCandidates, + hangOnMethod: "DOM.describeNode", + }); + const controller = new AbortController(); + const pending = inspect(browser, controller.signal); + try { + await expect.poll(() => descriptions(browser).length).toBeGreaterThan(1); + let replied = 0; + while (replied < 25) { + await expect + .poll(() => descriptions(browser).length) + .toBeGreaterThan(replied); + const batch = descriptions(browser).slice(replied); + expect(batch.length).toBeLessThanOrEqual(8); + for (const command of [...batch].reverse()) { + const id = Number(command.params.backendNodeId); + browser.emitRawMessage( + JSON.stringify({ + id: command.id, + result: { + node: { + backendNodeId: id, + nodeType: 1, + nodeName: "a", + isSVG: id % 2 === 1, + }, + }, + }), + ); + } + replied += batch.length; + } + const result = await pending; + if (!result.ok) throw result.error; + expect(result.value.metadata.dom_urls.map(({ url }) => url)).toEqual( + Array.from( + { length: 13 }, + (_, index) => `${browser.allowedOrigin}/link-${String(index * 2)}`, + ), + ); + expect(descriptions(browser)).toHaveLength(25); + } finally { + controller.abort(); + await pending; + } + }); + + it("cancels sibling reads on a fatal reply without waiting for hung nodes", async () => { + const browser = await startBrowser({ + commandResult: manyCandidates, + hangOnMethod: "DOM.describeNode", + }); + const controller = new AbortController(); + const pending = inspect(browser, controller.signal); + try { + await expect.poll(() => descriptions(browser).length).toBeGreaterThan(1); + const batch = descriptions(browser); + browser.emitRawMessage( + JSON.stringify({ + id: batch.at(-1)?.id, + error: { code: -32_602, message: "Invalid parameters" }, + }), + ); + await expect(pending).resolves.toMatchObject({ + ok: false, + error: { _tag: "BrowserObservationError", reason: "protocol_error" }, + }); + expect(descriptions(browser)).toHaveLength(batch.length); + const methods = browser.commands.map(({ method }) => method); + expect(methods).toContain("Target.detachFromTarget"); + expect(methods).not.toContain("Target.closeTarget"); + expect(methods).not.toContain("Browser.close"); + } finally { + controller.abort(); + await pending; + } + }); +}); diff --git a/tests/boundary/browser/cdpScreenshotDocumentIdentity.test.ts b/tests/boundary/browser/cdpScreenshotDocumentIdentity.test.ts index 012ad9663..b3fcbc592 100644 --- a/tests/boundary/browser/cdpScreenshotDocumentIdentity.test.ts +++ b/tests/boundary/browser/cdpScreenshotDocumentIdentity.test.ts @@ -3,6 +3,37 @@ import { CdpBrowserProvider } from "../../../src/browser/CdpBrowserProvider.js"; import { captureWebScreenshotInputSchema } from "../../../src/domain/webScreenshot.js"; import { startFakeCdpBrowser } from "../../fixtures/fakeCdpBrowser.js"; describe("screenshot committed document identity", () => { + it.each([ + { offset: 8, value: 14 }, + { offset: 16, value: 0x8000_0000 }, + { offset: 20, value: 0x8000_0000 }, + ])("rejects invalid screenshot IHDR metadata", async ({ offset, value }) => { + const bytes = Buffer.from( + "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+jv4sAAAAASUVORK5CYII=", + "base64", + ); + bytes.writeUInt32BE(value, offset); + const browser = await startFakeCdpBrowser({ + commandResult: ({ method }) => + method === "Page.captureScreenshot" + ? { data: bytes.toString("base64") } + : undefined, + }); + onTestFinished(async () => { + await browser.close(); + }); + const result = await new CdpBrowserProvider().captureScreenshot( + captureWebScreenshotInputSchema.parse({ + cdp_endpoint: browser.endpoint, + allowed_origins: [browser.allowedOrigin], + target_id: "allowed-page", + }), + ); + expect(result).toMatchObject({ + ok: false, + error: { reason: "protocol_error" }, + }); + }); it.each(["loader-main", "loader-reloaded"])( "distinguishes %s for the same URL", async (screenshotDocumentLoader) => { diff --git a/tests/boundary/browser/cdpSvgXlinkHref.test.ts b/tests/boundary/browser/cdpSvgXlinkHref.test.ts index 56c3fc3cf..38048861b 100644 --- a/tests/boundary/browser/cdpSvgXlinkHref.test.ts +++ b/tests/boundary/browser/cdpSvgXlinkHref.test.ts @@ -45,8 +45,38 @@ const pageHtml = `x out + move desc + move title + + not an SVG link + HTML link + nested + + + <a xlink:href="/xl-html-title">not an SVG link</a> + <a href="/html-title">HTML link</a> + <svg><a xlink:href="/xl-nested-title"><text>nested</text></a></svg> + html + `; it.skipIf(chromeExecutable === undefined)( @@ -143,17 +173,36 @@ it.skipIf(chromeExecutable === undefined)( url: null, destination_scope: "outside_policy", }), + ...[ + "/html-desc", + "/html-title", + "/xl-nested-desc", + "/xl-nested-title", + "/xl-moved-desc", + "/xl-moved-title", + "/xl-created-desc", + "/xl-created-title", + "/xl-created-nested-desc", + "/xl-created-nested-title", + ].map((path) => + expect.objectContaining({ + attribute: "href", + url: `${origin}${path}`, + destination_scope: "approved", + }), + ), ]), ); expect( urls.some( (url) => - url.url?.endsWith("/xl-html") === true || + url.url?.includes("/xl-html") === true || + url.url?.includes("/xl-math") === true || url.url?.endsWith("/xlink-loses") === true, ), ).toBe(false); expect(urls.every((url) => url.attribute === "href")).toBe(true); - expect(urls).toHaveLength(5); + expect(urls).toHaveLength(15); } finally { child.kill("SIGTERM"); if (!(await waitForExit(child, 2_000))) child.kill("SIGKILL"); @@ -173,6 +222,8 @@ const xhtmlPage = ` p + desc + title no no @@ -202,6 +253,8 @@ for (const { format, page, contentType, nodeNames, link, destinations } of [ destinations: [ "/xl-pa1", "/xl-image", + "/xl-xml-desc", + "/xl-xml-title", "/xl-fo-svg", "/xl-orphan", "/xl-custom", diff --git a/tests/boundary/browser/historicalCaptureDecoder.test.ts b/tests/boundary/browser/historicalCaptureDecoder.test.ts index 92b26d5df..e02524f8d 100644 --- a/tests/boundary/browser/historicalCaptureDecoder.test.ts +++ b/tests/boundary/browser/historicalCaptureDecoder.test.ts @@ -11,7 +11,7 @@ import { } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { expect, it, onTestFinished } from "vitest"; +import { expect, it, onTestFinished, vi } from "vitest"; import { HistoricalCaptureDecoder } from "../../../src/browser/history/HistoricalCaptureDecoder.js"; import { MitmproxyCaptureAdapter } from "../../../src/browser/history/MitmproxyCaptureAdapter.js"; import { HAR_CAPTURE_PROVIDER_IDENTITY } from "../../../src/browser/history/CaptureRelease.js"; @@ -22,6 +22,7 @@ import { import { historicalHar } from "../../fixtures/historicalHar.js"; import { decodeHarCapture } from "../../../src/browser/history/HarCapture.js"; import { projectAnalysisError } from "../../../src/domain/analysisErrorProjection.js"; +import { OwnedFileHandle } from "../../../src/filesystem/OwnedFileHandle.js"; const linux = it.skipIf(process.platform !== "linux"); @@ -138,6 +139,66 @@ it("classifies an oversized selected capture before creating an owned decoder", }); }); +it("retains a failed capture reader and retries it before a fresh load", async () => { + const root = await mkdtemp(join(tmpdir(), "rea-capture-reader-close-")); + onTestFinished(() => rm(root, { recursive: true, force: true })); + const path = join(root, "input.har"); + await writeFile(path, JSON.stringify(historicalHar())); + const originalClose = OwnedFileHandle.prototype.close; + const close = vi.spyOn(OwnedFileHandle.prototype, "close"); + let allowClose = false; + let closeAttempts = 0; + close.mockImplementation(async function (this: OwnedFileHandle) { + closeAttempts += 1; + if (!allowClose) throw new Error("injected capture reader close failure"); + await originalClose.call(this); + }); + let decoderCalls = 0; + const decoder = new HistoricalCaptureDecoder( + [ + { + format: "har", + identity: HAR_CAPTURE_PROVIDER_IDENTITY, + command: async () => { + decoderCalls += 1; + return { + command: process.execPath, + arguments: ["-e", "process.exit(0)"], + }; + }, + }, + ], + process.env, + ); + const input = inspectWebNetworkCaptureInputSchema.parse({ + capture_path: path, + format: "har", + }); + try { + const first = await decoder.inspect(input); + expect(first.ok).toBe(false); + if (first.ok) throw new Error("Reader close failure must fail inspection"); + expect(first.error.cleanupResources).toContain(path); + expect(decoderCalls).toBe(0); + + const refused = await decoder.inspect(input); + expect(refused.ok).toBe(false); + if (refused.ok) throw new Error("Fresh read must wait for owner cleanup"); + expect(refused.error.cleanupIncomplete).toBe(true); + expect(refused.error.cleanupResources).toContain(path); + expect(decoderCalls).toBe(0); + expect(closeAttempts).toBe(2); + + allowClose = true; + await decoder.close(); + expect(closeAttempts).toBe(3); + } finally { + allowClose = true; + await decoder.close().catch(() => undefined); + close.mockRestore(); + } +}); + it("does not misclassify decoder filesystem failures as missing selected input", async () => { const root = await mkdtemp(join(tmpdir(), "rea-historical-command-")); onTestFinished(() => rm(root, { recursive: true, force: true })); diff --git a/tests/boundary/browser/historicalWebCapture.test.ts b/tests/boundary/browser/historicalWebCapture.test.ts index a0a9f10e2..781172cd3 100644 --- a/tests/boundary/browser/historicalWebCapture.test.ts +++ b/tests/boundary/browser/historicalWebCapture.test.ts @@ -7,6 +7,9 @@ import { execFile } from "node:child_process"; import { fileURLToPath } from "node:url"; import { promisify } from "node:util"; import { parseEvidence } from "../../../src/domain/evidence.js"; +import { webNetworkCaptureSchema } from "../../../src/domain/webNetworkCapture.js"; +import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; const entrypoint = fileURLToPath( new URL("../../../scripts/rea.mjs", import.meta.url), ); @@ -41,6 +44,91 @@ const cli = async ( }; import { historicalHar } from "../../fixtures/historicalHar.js"; +it("excludes declared binary lengths through CLI, MCP and retained capture pages", async () => { + const root = await createTestTempDirectory("rea-har-byte-length-"); + const path = join(root, "capture.har"); + const fixture = historicalHar(); + const entry = fixture.log.entries[0]; + if (entry === undefined) throw new Error("fixture missing"); + const body = Buffer.alloc(12_345, "A"); + const content = body.toString("base64"); + const digest = createHash("sha256").update(body).digest("hex"); + await writeFile( + path, + JSON.stringify({ + log: { + ...fixture.log, + entries: [ + { + ...entry, + response: { + ...entry.response, + content: { + size: 99, + mimeType: "text/plain", + encoding: "base64", + text: content, + }, + }, + }, + ], + }, + }), + ); + const { call } = await connectLocalToolsMcp(); + const args = { capture_path: path, format: "har" }; + const control = await call("inspect_web_network_capture", args); + expect(control.isError).not.toBe(true); + const retained = parseEvidence(control.structuredContent); + const controlCapture = webNetworkCaptureSchema.parse( + retained.normalized_result, + ); + expect(controlCapture.records[0]?.binary_fields[0]).toMatchObject({ + state: "retained", + bytes: body.length, + content_base64: content, + sha256: digest, + }); + for (const literal of ["12345", "2345"]) { + // Only the derived count matches, so this cannot pass via another exclusion. + expect(content).not.toContain(literal); + expect(digest).not.toContain(literal); + const fromCli = await cli(path, "har", ["--sensitive-value", literal]); + expect(fromCli.ok, fromCli.stdout).toBe(true); + const evidence = parseEvidence(JSON.parse(fromCli.stdout)); + const response = await call("inspect_web_network_capture", { + ...args, + sensitive_values: [literal], + }); + expect(response.isError).not.toBe(true); + const fromMcp = parseEvidence(response.structuredContent); + expect(fromMcp.normalized_result).toEqual(evidence.normalized_result); + const capture = webNetworkCaptureSchema.parse(evidence.normalized_result); + expect(capture.records[0]?.reported).toMatchObject({ + response: { content: { size: 99, text: null } }, + }); + expect(capture.records[0]?.binary_fields[0]).toEqual({ + pointer: "/response/content/text", + representation: "har-base64-content", + state: "redacted", + bytes: null, + content_base64: null, + sha256: null, + }); + for (const detail of ["preview", "complete"]) { + const page = await call("inspect_analysis_view", { + source: { kind: "retained-evidence", evidence_id: fromMcp.evidence_id }, + view: { kind: "capture", detail }, + }); + expect(page.isError).not.toBe(true); + const result = parseEvidence(page.structuredContent); + expect(result.normalized_result).toMatchObject({ + items: [{ binary_fields: [{ state: "redacted", bytes: null }] }], + }); + } + } +}, 60_000); + it("reads actual upstream HAR packages in an owned process and retains selected producer order", async () => { const root = await mkdtemp(join(tmpdir(), "rea-history-har-")); onTestFinished(() => rm(root, { recursive: true, force: true })); diff --git a/tests/boundary/browser/sazWebCapture.test.ts b/tests/boundary/browser/sazWebCapture.test.ts new file mode 100644 index 000000000..69eca33b8 --- /dev/null +++ b/tests/boundary/browser/sazWebCapture.test.ts @@ -0,0 +1,499 @@ +import { createHash } from "node:crypto"; +import { execFile } from "node:child_process"; +import { readFile, unlink, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { promisify } from "node:util"; +import { fileURLToPath } from "node:url"; +import { gzipSync } from "node:zlib"; +import { Uint8ArrayReader, Uint8ArrayWriter, ZipWriter } from "@zip.js/zip.js"; +import { Ajv2020 } from "ajv/dist/2020.js"; +import { expect, it as runtimeTest } from "vitest"; +import { z } from "zod"; +import { parseEvidence } from "../../../src/domain/evidence.js"; +import { webNetworkCaptureSchema } from "../../../src/domain/webNetworkCapture.js"; +import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +import { + NODE_CAPTURE_HEAP_ENFORCED, + NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES, +} from "../../../src/browser/history/CaptureRelease.js"; +const hasMemoryBound = + NODE_CAPTURE_HEAP_ENFORCED || + NODE_CAPTURE_PROCESS_MEMORY_LIMIT_BYTES !== null; + +const it = runtimeTest.skipIf(!hasMemoryBound); + +// Synthetic member fixtures based on Telerik's SAZ layout; not a real-producer golden. +const compressedBody = gzipSync(Buffer.from("stored compressed body")); +const members = (): [string, Buffer][] => [ + [ + "raw/0042_s.txt", + Buffer.concat([ + Buffer.from( + "HTTP/1.1 200 OK\r\nContent-Encoding: gzip\r\nSet-Cookie: server-credential\r\n\r\n", + ), + compressedBody, + ]), + ], + [ + "raw/7_c.txt", + Buffer.concat([ + Buffer.from("POST /bytes HTTP/2.0\r\n:authority: example.test\r\n\r\n"), + Buffer.from([0xff, 0, 0x80]), + ]), + ], + [ + "raw/0042_c.txt", + Buffer.from( + "GET https://user:password@example.test/a HTTP/1.1\r\nAuthorization: Bearer client-credential\r\nCookie: cookie-credential\r\nX-Duplicate: one\r\nX-Duplicate: two\r\nX-Text: unicode-\u03b2\r\n\r\n", + ), + ], + [ + "raw/0042_m.xml", + Buffer.from( + '', + ), + ], + ["raw/0042_w.txt", Buffer.from("opaque websocket message file")], + ["raw/7_g.txt", Buffer.from([0, 1, 2])], + ["[Content_Types].xml", Buffer.from("")], +]; +const archive = async (entries = members(), password?: string, level = 6) => { + const writer = new ZipWriter(new Uint8ArrayWriter(), { + useWebWorkers: false, + level, + ...(password === undefined ? {} : { password }), + }); + for (const [name, bytes] of entries) + await writer.add(name, new Uint8ArrayReader(bytes)); + return Buffer.from(await writer.close()); +}; +const fixture = async (bytes: Buffer) => { + const root = await createTestTempDirectory("rea-saz-capture-"); + const path = join(root, "capture.saz"); + await writeFile(path, bytes); + return path; +}; +const entrypoint = fileURLToPath( + new URL("../../../scripts/rea.mjs", import.meta.url), +); +const cliResponse = async (path: string, extra: string[] = []) => { + try { + const { stdout } = await promisify(execFile)( + process.execPath, + [ + entrypoint, + "inspect-web-network-capture", + path, + "saz", + "--json", + ...extra, + ], + { timeout: 40_000, maxBuffer: 4 * 1024 * 1024 }, + ); + return { ok: true, stdout }; + } catch (cause: unknown) { + if ( + cause instanceof Error && + "stdout" in cause && + typeof cause.stdout === "string" + ) + return { ok: false, stdout: cause.stdout }; + throw cause; + } +}; +const cli = async (path: string, extra: string[] = []) => { + const result = await cliResponse(path, extra); + expect(result.ok, result.stdout).toBe(true); + return parseEvidence(JSON.parse(result.stdout)); +}; + +it("inspects native SAZ through the public CLI and MCP with equal evidence, bytes and archive coordinates", async () => { + const bytes = await archive(); + const path = await fixture(bytes); + const evidence = await cli(path, ["--record", "1", "--record", "0"]); + const capture = webNetworkCaptureSchema.parse(evidence.normalized_result); + expect(capture).toMatchObject({ + format: "saz", + total_records: 2, + runtime_attribution: "unknown", + records: [ + { ordinal: 1, reported: { session_id: "7", response: null } }, + { + ordinal: 0, + reported: { + session_id: "0042", + request: { url: "https://example.test/a" }, + response: { status: "200" }, + }, + }, + ], + }); + expect(capture.records[1]?.location).toEqual({ + kind: "archive-members", + members: [ + "raw/0042_s.txt", + "raw/0042_c.txt", + "raw/0042_m.xml", + "raw/0042_w.txt", + ], + }); + expect(capture.records[1]?.binary_fields).toContainEqual({ + pointer: "/response/body", + representation: "producer-bytes", + state: "retained", + content_base64: compressedBody.toString("base64"), + bytes: compressedBody.length, + sha256: createHash("sha256").update(compressedBody).digest("hex"), + }); + expect(capture.records[0]?.binary_fields).toContainEqual( + expect.objectContaining({ + pointer: "/request/body", + content_base64: "/wCA", + }), + ); + expect(capture.records[1]?.binary_fields).toContainEqual( + expect.objectContaining({ + pointer: "/request/body", + bytes: 0, + content_base64: "", + }), + ); + expect(capture.records[1]?.reported).toMatchObject({ + request: { + headers: [ + { name: "Authorization", value: null }, + { name: "Cookie", value: null }, + { name: "X-Duplicate", value: " one" }, + { name: "X-Duplicate", value: " two" }, + { name: "X-Text" }, + ], + }, + }); + expect(JSON.stringify(capture)).toContain("retained metadata"); + for (const secret of [ + "user:password", + "client-credential", + "server-credential", + "cookie-credential", + ]) + expect(JSON.stringify(capture)).not.toContain(secret); + expect(capture.artifact.sha256).toBe( + createHash("sha256").update(bytes).digest("hex"), + ); + expect(await readFile(path)).toEqual(bytes); + + const { client, call } = await connectLocalToolsMcp(); + const tool = (await client.listTools()).tools.find( + (item) => item.name === "inspect_web_network_capture", + ); + if (tool?.outputSchema === undefined) + throw new Error("Missing capture schema"); + const args = { capture_path: path, format: "saz", record_ordinals: [1, 0] }; + const ajv = new Ajv2020({ strict: false, validateFormats: false }); + expect(ajv.validate(tool.inputSchema, args)).toBe(true); + const response = await call("inspect_web_network_capture", args); + expect(response.isError, JSON.stringify(response)).not.toBe(true); + const fromMcp = parseEvidence(response.structuredContent); + expect(fromMcp.normalized_result).toEqual(evidence.normalized_result); + const bundle = await client.callTool({ + name: "get_evidence_bundle", + arguments: { detail: "complete" }, + }); + const records = z + .object({ result: z.object({ records: z.array(z.unknown()) }) }) + .parse(bundle.structuredContent) + .result.records.map(parseEvidence); + expect( + records.find((record) => record.evidence_id === fromMcp.evidence_id), + ).toEqual(fromMcp); +}, 60_000); + +it("pages retained SAZ Evidence through public MCP and portable CLI after source deletion", async () => { + const path = await fixture(await archive()); + const { call } = await connectLocalToolsMcp(); + const args = { capture_path: path, format: "saz", record_ordinals: [1, 0] }; + const full = await call("inspect_web_network_capture", args); + const fromMcp = parseEvidence(full.structuredContent); + const capture = webNetworkCaptureSchema.parse(fromMcp.normalized_result); + const summaryArgs = { ...args, detail: "summary" }; + const summaryResponse = await call( + "inspect_web_network_capture", + summaryArgs, + ); + expect(summaryResponse.isError, JSON.stringify(summaryResponse)).not.toBe( + true, + ); + const summary = parseEvidence(summaryResponse.structuredContent); + expect(summary.normalized_result).toMatchObject({ + kind: "summary", + parent_evidence_id: fromMcp.evidence_id, + summary: { format: "saz", total_records: 2, retained_records: 2 }, + }); + expect(JSON.stringify(summary)).not.toContain( + compressedBody.toString("base64"), + ); + + // Retained pages must keep their snapshot semantics after the input disappears. + await unlink(path); + const source = { + kind: "retained-evidence", + evidence_id: fromMcp.evidence_id, + }; + const previewArgs = { + source, + view: { kind: "capture", offset: 0, limit: 1 }, + }; + const previewResponse = await call("inspect_analysis_view", previewArgs); + expect(previewResponse.isError, JSON.stringify(previewResponse)).not.toBe( + true, + ); + const preview = parseEvidence(previewResponse.structuredContent); + expect(preview.normalized_result).toMatchObject({ + items: [ + { + ordinal: 1, + reported: { + session_id: "7", + request: { method: "POST", url: "/bytes" }, + }, + }, + ], + coverage: { examined: 1, total: 2, next_offset: 1, exhausted: false }, + }); + expect(JSON.stringify(preview)).not.toContain("content_base64"); + const completeResponse = await call("inspect_analysis_view", { + source, + view: { kind: "capture", detail: "complete", offset: 1, limit: 1 }, + }); + expect(completeResponse.isError, JSON.stringify(completeResponse)).not.toBe( + true, + ); + const complete = parseEvidence(completeResponse.structuredContent); + expect(complete.normalized_result).toMatchObject({ + items: [capture.records[1]], + coverage: { next_offset: null, exhausted: true }, + }); + + const workflow = join(path, "..", "summary-view.json"); + await writeFile( + workflow, + JSON.stringify({ + source: { kind: "inline", evidence: fromMcp }, + view: { kind: "summary" }, + }), + ); + const standalone = await promisify(execFile)(process.execPath, [ + entrypoint, + "inspect-analysis-view", + workflow, + "--json", + ]); + expect( + parseEvidence(JSON.parse(standalone.stdout)).normalized_result, + ).toEqual(summary.normalized_result); +}, 60_000); + +it("keeps CRLF separators inside a body after LF-only HTTP headers", async () => { + const body = Buffer.from("\ufefffirst\r\n\r\nsecond\n\nthird"); + const bytes = await archive([ + [ + "raw/1_c.txt", + Buffer.concat([ + Buffer.from("POST https://example.test/ HTTP/1.1\nX-Test: request\n\n"), + body, + ]), + ], + [ + "raw/1_s.txt", + Buffer.concat([ + Buffer.from("HTTP/1.1 200 OK\nX-Test: response\n\n"), + body, + ]), + ], + ]); + const path = await fixture(bytes); + const capture = webNetworkCaptureSchema.parse( + (await cli(path)).normalized_result, + ); + expect(capture.records[0]?.reported).toMatchObject({ + request: { + headers: [{ name: "X-Test", value: " request" }], + body: body.toString("utf8"), + }, + response: { + headers: [{ name: "X-Test", value: " response" }], + body: body.toString("utf8"), + }, + }); + for (const pointer of ["/request/body", "/response/body"]) + expect(capture.records[0]?.binary_fields).toContainEqual({ + pointer, + representation: "producer-bytes", + state: "retained", + content_base64: body.toString("base64"), + bytes: body.length, + sha256: createHash("sha256").update(body).digest("hex"), + }); + expect(await readFile(path)).toEqual(bytes); +}); + +it("excludes explicit literals from XML, HTTP byte views, member coordinates and binary sidecars", async () => { + const path = await fixture(await archive()); + const result = await cli(path, [ + "--sensitive-value", + "0042", + "--sensitive-value", + "unicode-\u03b2", + "--sensitive-value", + compressedBody.toString("base64"), + "--sensitive-value", + "retained metadata", + ]); + const capture = webNetworkCaptureSchema.parse(result.normalized_result); + expect(capture.records[0]?.location).toEqual({ + kind: "unknown", + reason: "explicit-sensitive-value", + }); + expect(capture.records[0]?.binary_fields).toContainEqual({ + pointer: "/response/body", + representation: "producer-bytes", + state: "redacted", + bytes: null, + sha256: null, + content_base64: null, + }); + const text = JSON.stringify(result); + for (const secret of [ + "0042", + "unicode-", + "retained metadata", + compressedBody.toString("base64"), + createHash("sha256").update(compressedBody).digest("hex"), + ]) + expect(text).not.toContain(secret); +}); + +it("retains folded headers, stored chunk framing and BOM-marked UTF-16 metadata", async () => { + const body = "3\r\none\r\n0\r\n\r\n"; + const path = await fixture( + await archive([ + [ + "raw/1_s.txt", + Buffer.from( + `HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\nX-Folded: first\r\n second\r\n\r\n${body}`, + ), + ], + [ + "raw/1_m.xml", + Buffer.from( + '\ufeff\u03b2]]>', + "utf16le", + ), + ], + [ + "raw/2_c.txt", + Buffer.from( + "CONNECT example.test:443 HTTP/1.1\nHost: example.test:443\n\n", + ), + ], + ]), + ); + const evidence = await cli(path); + const capture = webNetworkCaptureSchema.parse(evidence.normalized_result); + expect(capture.records[0]?.reported).toMatchObject({ + request: null, + response: { + body, + headers: [ + { name: "Transfer-Encoding", value: " chunked" }, + { name: "X-Folded", value: " first\r\n second" }, + ], + }, + }); + expect(capture.records[0]?.binary_fields).toContainEqual( + expect.objectContaining({ + pointer: "/response/body", + content_base64: Buffer.from(body).toString("base64"), + }), + ); + expect(JSON.stringify(capture.records[0]?.reported)).toContain("\u03b2"); + expect(capture.records[1]?.reported).toMatchObject({ + request: { method: "CONNECT", url: "example.test:443", body: "" }, + }); +}); + +it.each([ + "bad-http", + "bad-xml", + "dtd", + "unknown-member", + "encrypted", + "duplicate", + "crc", + "expanded-limit", +])("rejects %s before returning a selected subset", async (variant) => { + let entries = members(); + if (variant === "bad-http") + entries.push(["raw/8_c.txt", Buffer.from("malformed-private-input")]); + if (variant === "bad-xml") + entries.push(["raw/8_m.xml", Buffer.from("")]); + if (variant === "dtd") + entries.push([ + "raw/8_m.xml", + Buffer.from( + ']>&external;', + ), + ]); + if (variant === "unknown-member") + entries.push(["raw/8_unknown.txt", Buffer.from("unknown extension")]); + if (variant === "duplicate") + entries = [ + ["raw/8_c.txt", Buffer.from("GET /a HTTP/1.1\r\n\r\n")], + ["raw/9_c.txt", Buffer.from("GET /b HTTP/1.1\r\n\r\n")], + ]; + let bytes = await archive( + entries, + variant === "encrypted" ? "fixture-password" : undefined, + variant === "crc" ? 0 : 6, + ); + if (variant === "duplicate") + bytes = Buffer.from( + bytes.toString("latin1").replaceAll("raw/9_c.txt", "raw/8_c.txt"), + "latin1", + ); + if (variant === "crc") { + const offset = bytes.indexOf(Buffer.from("HTTP/1.1 200 OK")); + expect(offset).toBeGreaterThan(0); + bytes[offset + 9] = 0x35; + } + if (variant === "expanded-limit") { + const central = bytes.indexOf(Buffer.from([0x50, 0x4b, 0x01, 0x02])); + bytes.writeUInt32LE(33 * 1024 * 1024, central + 24); + } + const path = await fixture(bytes); + const result = await cliResponse(path, ["--record", "0"]); + expect(result.ok, result.stdout).toBe(false); + expect(JSON.parse(result.stdout).category).toBe( + ["encrypted", "unknown-member", "dtd"].includes(variant) + ? "unsupported_provider" + : variant === "expanded-limit" + ? "execution_failure" + : "invalid_input", + ); + if (variant === "expanded-limit") + expect(JSON.parse(result.stdout).code).toBe("unreadable_output"); + expect(JSON.stringify(result)).not.toContain("malformed-private-input"); +}); + +runtimeTest.runIf(!hasMemoryBound)( + "reports SAZ unavailable when its runtime cannot enforce decoder memory", + async () => { + const response = await cliResponse(await fixture(await archive())); + expect(response.ok).toBe(false); + expect(JSON.parse(response.stdout)).toMatchObject({ + code: "capability_unavailable", + }); + }, +); diff --git a/tests/boundary/captures/captureProvider.test.ts b/tests/boundary/captures/captureProvider.test.ts index 9f5fb4afc..2428b39d4 100644 --- a/tests/boundary/captures/captureProvider.test.ts +++ b/tests/boundary/captures/captureProvider.test.ts @@ -1,3 +1,4 @@ +import { packetFieldsCatalogSchema } from "../../../src/domain/captures/packetFields.js"; import { OwnedCommandFailure } from "../../../src/process/OwnedCommand.js"; import { it, expect, onTestFinished } from "vitest"; import { writeFile, readdir, chmod, symlink } from "node:fs/promises"; @@ -20,14 +21,15 @@ const fixture = async (mode = "ok", failCleanup = false) => { await writeFile( script, `import {readFileSync, writeFileSync, chmodSync, truncateSync} from 'node:fs'; -if (process.argv.includes('--version')) { console.log('TShark (Wireshark) 4.6.9 (fixture)'); console.error('version warning'); } +if (process.argv.includes('--version')) { console.log(${JSON.stringify(mode)} === 'bad-version' ? 'TShark (Wireshark) 3.0.0' : 'TShark (Wireshark) 4.6.9 (fixture)'); console.error('version warning'); } else if (${JSON.stringify(mode)} === 'nonzero') { console.error('invalid decoder'); process.exitCode = 1; } else if (${JSON.stringify(mode)} === 'malformed-copy') { chmodSync('capture', 0o600); writeFileSync('capture', 'changed'); } else if (${JSON.stringify(mode)} === 'oversized-copy') { chmodSync('capture', 0o600); truncateSync('capture', 32 * 1024 * 1024 + 1); console.log('[]'); } -else if (${JSON.stringify(mode)} === 'overflow') process.stdout.write('x'.repeat(5 * 1024 * 1024)); +else if (${JSON.stringify(mode)} === 'overflow') process.stdout.write('x'.repeat((process.argv.includes('-G') ? 33 : 5) * 1024 * 1024)); else if (${JSON.stringify(mode)} === 'wait') { console.log('ready'); setInterval(() => {}, 1000); } else if (${JSON.stringify(mode)} === 'invalid-json') console.log('not json'); else if (${JSON.stringify(mode)} === 'many') console.log(JSON.stringify(Array.from({length:90},(_,i)=>({_source:{layers:{"frame.number":[String(i+1)],"ip.addr":["192.0.2.1"]}}})))); +else if (process.argv.includes('-G')) { console.log('P\\tUniversal Serial Bus\\tusb\\nF\\tVendor ID\\tusb.idVendor\\tFT_UINT16\\tusb\\tBASE_HEX\\t0x0\\t\\nF\\tOther Vendor\\tusb.idVendor\\tFT_UINT32\\tusb\\tBASE_HEX\\t0x0\\tSecond registration\\nP\\tInternet Protocol\\tip\\nF\\tAddress\\tip.addr\\tFT_IPv4\\tip\\t\\t0x0\\tSource or destination'); console.error('glossary warning'); } else { if(readFileSync('capture', 'utf8') !== 'recording bytes') throw Error('wrong copy'); console.log(JSON.stringify([{_source:{layers:{"frame.number":["1"],"ip.addr":["192.0.2.1","192.0.2.2"]}}}])); console.error('producer warning'); } `, ); @@ -301,6 +303,9 @@ it("retains a completed observation when cancellation arrives at the service bou const execution = await f.service.provider.inspect(input); if (!execution.ok) throw execution.error; const service = new PacketCaptureService({ + inspectFields: async () => { + throw new Error("Unexpected field discovery call"); + }, inspect: async () => { f.controller.abort(); return execution; @@ -465,3 +470,158 @@ it("rejects oversized selector lists and UTF-8 argv expansion before provider wo } expect(f.calls()).toBe(0); }); + +it("discovers installed protocols and field definitions without a capture", async () => { + const f = await fixture(); + for (const [input, expected] of [ + [ + { query: "USB" }, + { + records: [{ abbreviation: "usb", kind: "protocol", field_type: null }], + output_window: { examined: 2, matching: 1, returned: 1 }, + }, + ], + [ + { kind: "fields", protocol: "usb", limit: 1 }, + { + records: [ + { + abbreviation: "usb.idVendor", + field_type: "FT_UINT16", + source_line: 2, + }, + ], + output_window: { + examined: 3, + matching: 2, + returned: 1, + has_more: true, + }, + }, + ], + [ + { kind: "fields", protocol: "usb", offset: 1, limit: null }, + { + records: [ + { + abbreviation: "usb.idVendor", + field_type: "FT_UINT32", + source_line: 3, + }, + ], + output_window: { matching: 2, returned: 1, has_more: false }, + }, + ], + [ + { kind: "fields", query: "destination" }, + { + records: [ + { + abbreviation: "ip.addr", + description: "Source or destination", + display_base: "", + }, + ], + }, + ], + [ + { query: "no-such-protocol" }, + { records: [], output_window: { matching: 0, returned: 0 } }, + ], + ] as const) { + const result = await f.service.inspectFields(input); + if (!result.ok) throw result.error; + const report = packetFieldsCatalogSchema.parse( + result.value.normalized_result, + ); + expect(report).toMatchObject(expected); + expect(report.tool.arguments).toEqual(["-G", "fields"]); + expect(report.tool.version_diagnostics).toBe("version warning\n"); + expect(report.diagnostics).toBe("glossary warning\n"); + expect(report.limits.output_bytes).toBe(32 * 1024 * 1024); + } + expect( + (await readdir(f.directory)).filter((name) => name.startsWith("decoder-")), + ).toEqual([]); +}); + +it("shares discovery admission with inspection and refuses work after shutdown", async () => { + const f = await fixture("gated"); + const active = f.service.inspect({ path: f.path, fields: ["ip.addr"] }); + await f.acquired; + const busy = await f.service.inspectFields({}); + f.release(); + if (!(await active).ok) throw new Error("Inspection failed"); + expect(busy).toMatchObject({ + ok: false, + error: { operation: "inspect_packet_fields", resource: "capacity" }, + }); + await f.service.close(); + expect(await f.service.inspectFields({})).toMatchObject({ + ok: false, + error: { + operation: "inspect_packet_fields", + _tag: "AnalysisCancelledError", + }, + }); +}); + +it.each([ + "bad-version", + "nonzero", + "overflow", + "wait", + "oversized-tool", + "invalid-json", +])("retains field discovery failure facts: %s", async (mode) => { + const f = await fixture(mode); + const result = await f.service.inspectFields( + {}, + { signal: f.controller.signal }, + ); + if (result.ok) throw new Error("Expected failure"); + expect(result.error).toMatchObject({ operation: "inspect_packet_fields" }); + if (mode === "bad-version") + expect(result.error.capturedOutput).toMatchObject({ + stdout: "TShark (Wireshark) 3.0.0\n", + stderr: "version warning\n", + }); + if (mode === "invalid-json") + expect(result.error.capturedOutput?.stdout).toBe("not json\n"); + if (mode === "overflow") + expect(result.error).toMatchObject({ + resource: "memory", + reportedLimits: { output_bytes: 32 * 1024 * 1024 }, + }); +}); + +it("retains completed discovery on late cancellation and cleanup failure", async () => { + for (const mode of ["post-run-abort", "cleanup"]) { + const f = await fixture(mode, mode === "cleanup"); + const result = await f.service.inspectFields( + { query: "usb" }, + { signal: f.controller.signal }, + ); + expect(result).toMatchObject({ + ok: false, + error: { + partialObservation: { + operation: "inspect_packet_fields", + normalized_result: { records: [{ abbreviation: "usb" }] }, + }, + }, + }); + if (mode === "cleanup") { + await f.service.inspectFields({}); + await f.service.inspect({ path: f.path, fields: ["ip.addr"] }); + expect(f.calls()).toBe(2); + for (let attempt = 0; attempt < 2; attempt++) + await expect(f.service.close()).rejects.toMatchObject({ + operation: "inspect_packet_fields", + cleanupIncomplete: true, + }); + } + f.allowCleanup(); + await f.service.close(); + } +}); diff --git a/tests/boundary/captures/fieldDiscovery.test.ts b/tests/boundary/captures/fieldDiscovery.test.ts new file mode 100644 index 000000000..79a27f7bf --- /dev/null +++ b/tests/boundary/captures/fieldDiscovery.test.ts @@ -0,0 +1,67 @@ +import { expect, it } from "vitest"; +import { inspectPacketFieldsInputSchema } from "../../../src/domain/captures/packetFields.js"; +import { parseFieldDiscovery } from "../../../src/captures/parseFieldDiscovery.js"; + +it("keeps duplicate registrations and source ordinals instead of overwriting abbreviations", () => { + const text = + "P\tUSB\tusb\r\nF\tVendor\tusb.idVendor\tFT_UINT16\tusb\tBASE_HEX\t0x0\t\r\nF\tVendor\tusb.idVendor\tFT_UINT32\tusb\tBASE_HEX\t0x0\t"; + const result = parseFieldDiscovery( + text, + inspectPacketFieldsInputSchema.parse({ + kind: "fields", + query: "idVendor", + offset: 1, + }), + ); + expect(result.records).toMatchObject([ + { + source_line: 3, + abbreviation: "usb.idVendor", + field_type: "FT_UINT32", + description: "", + }, + ]); + expect(result.output_window).toEqual({ + examined: 2, + matching: 2, + returned: 1, + has_more: false, + }); +}); + +it("rejects damaged records even outside the selected page", () => { + for (const text of [ + "", + "P\tmissing", + "F\tname\tabbr", + "P\tname\t", + "P\tUSB\tusb\nBAD", + ]) + expect(() => + parseFieldDiscovery( + text, + inspectPacketFieldsInputSchema.parse({ limit: 1 }), + ), + ).toThrow(); +}); + +it("bounds explicit full-result expansion while allowing a page from the same large glossary", () => { + const text = "P\t" + "Long name ".repeat(30) + "\tabbr\n"; + const glossary = text.repeat(6000); + expect(() => + parseFieldDiscovery( + glossary, + inspectPacketFieldsInputSchema.parse({ limit: null }), + ), + ).toThrow("representation budget"); + const result = parseFieldDiscovery( + glossary, + inspectPacketFieldsInputSchema.parse({ offset: 5999, limit: 1 }), + ); + expect(result.output_window).toEqual({ + examined: 6000, + matching: 6000, + returned: 1, + has_more: false, + }); +}); diff --git a/tests/boundary/cli/analysisSelectorCli.test.ts b/tests/boundary/cli/analysisSelectorCli.test.ts index 404cad6a2..5e89daa2b 100644 --- a/tests/boundary/cli/analysisSelectorCli.test.ts +++ b/tests/boundary/cli/analysisSelectorCli.test.ts @@ -12,6 +12,38 @@ const selectors = [ ] as const; describe("CLI analysis selector boundaries", () => { + cliTest( + "accepts file offsets and rejects mixed coordinate selectors before opening a target", + async ({ cli }) => { + const selected = await cli.run({ + arguments: [ + "decompile", + "does-not-exist", + "--file-offset=0x210", + "--json", + ], + environment: { REA_LOG_LEVEL: "silent" }, + }); + expect(selected.json).toMatchObject({ code: "target_unavailable" }); + const conflicting = await cli.run({ + arguments: [ + "decompile", + "does-not-exist", + "0x140001010", + "--file-offset=0x210", + "--json", + ], + environment: { REA_LOG_LEVEL: "silent" }, + }); + expect(conflicting.json).toMatchObject({ + code: "invalid_request", + details: { + operation: "procedure_pseudo_code", + issues: [{ path: ["file-offset"], reason: "invalid_value" }], + }, + }); + }, + ); cliTest( "rejects missing and conflicting selectors before opening a target", async ({ cli }) => { diff --git a/tests/boundary/cli/apktoolBooleanOptions.test.ts b/tests/boundary/cli/apktoolBooleanOptions.test.ts new file mode 100644 index 000000000..5c5578c18 --- /dev/null +++ b/tests/boundary/cli/apktoolBooleanOptions.test.ts @@ -0,0 +1,85 @@ +import { expect, it } from "vitest"; +import { Cli } from "incur"; +import pino from "pino"; + +import type { ApktoolResourceAnalysisPort } from "../../../src/application/apktool/ApktoolResourceAnalysisPort.js"; +import { createAnalysisExecution } from "../../../src/application/AnalysisProvider.js"; +import { registerApktoolCommands } from "../../../src/cli/apktoolCommands.js"; +import { presentOmittableDefaults } from "../../../src/cli/omittableDefaults.js"; +import type { ApktoolRequest } from "../../../src/domain/apktool/apktoolResourceAnalysis.js"; +import { ok } from "../../../src/domain/result.js"; + +const invoke = async (argv: readonly string[]) => { + const requests: ApktoolRequest[] = []; + const provider: ApktoolResourceAnalysisPort = { + async inspectAvailability() { + throw new Error("unexpected provider probe"); + }, + async close() {}, + async execute(request) { + requests.push(request); + return ok( + createAnalysisExecution( + { strings: [] }, + { id: "apktool", name: "Apktool", version: null }, + ), + ); + }, + }; + const cli = Cli.create("rea", { sync: false }); + registerApktoolCommands(cli, pino({ level: "silent" }), {}, provider); + presentOmittableDefaults(cli); + + let stdout = ""; + let exitCode = 0; + await cli.serve([...argv], { + env: {}, + exit: (code) => { + exitCode = code; + }, + stdout: (text) => { + stdout += text; + }, + }); + return { exitCode, stdout, requests }; +}; + +it("dispatches --no-strings as include_strings false", async () => { + const result = await invoke([ + "decode-android-resources", + "/targets/app.apk", + "--no-strings", + "--json", + ]); + + expect(result.exitCode).toBe(0); + expect(result.requests).toEqual([ + { + operation: "decode_android_resources", + input: { path: "/targets/app.apk", include_strings: false }, + }, + ]); +}); + +it("keeps string projection enabled when the flag is omitted", async () => { + const result = await invoke([ + "decode-android-resources", + "/targets/app.apk", + "--json", + ]); + + expect(result.exitCode).toBe(0); + expect(result.requests).toEqual([ + { + operation: "decode_android_resources", + input: { path: "/targets/app.apk", include_strings: true }, + }, + ]); +}); + +it("documents the supported negative flag in command help", async () => { + const result = await invoke(["decode-android-resources", "--help"]); + + expect(result.exitCode).toBe(0); + expect(result.stdout).toContain("--no-strings"); +}); diff --git a/tests/boundary/cli/captureCleanup.test.ts b/tests/boundary/cli/captureCleanup.test.ts index 1ec6adedd..6e5882cd4 100644 --- a/tests/boundary/cli/captureCleanup.test.ts +++ b/tests/boundary/cli/captureCleanup.test.ts @@ -5,6 +5,7 @@ import { silentLogger } from "../../../src/logger.js"; import { createEvidence } from "../../../src/domain/evidence.js"; import { ProviderCleanupError } from "../../../src/domain/providerCleanupError.js"; import { err } from "../../../src/domain/result.js"; +import { AnalysisInputError } from "../../../src/domain/analysisErrorCore.js"; it.each([false, true])( "awaits cleanup retry and preserves completed output when retry failure is %s", @@ -32,6 +33,9 @@ it.each([false, true])( silentLogger, {}, { + inspectFields: async () => { + throw new Error("Unexpected field discovery call"); + }, inspect: async () => err(original), close: async () => { await Promise.resolve(); @@ -79,6 +83,111 @@ it.each([false, true])( }, ); +it("keeps the primary CLI failure when close throws an untyped error", async () => { + const cli = Cli.create("capture-cleanup-primary-test", { sync: false }); + registerCaptureCommands( + cli, + silentLogger, + {}, + { + inspectFields: async () => { + throw new Error("Unexpected field discovery call"); + }, + inspect: async () => + err( + new AnalysisInputError("inspect_packet_capture", { + userMessage: "The selected capture input was rejected.", + }), + ), + close: async () => { + throw new Error("raw close failure"); + }, + }, + ); + let stdout = ""; + await cli.serve( + [ + "inspect-packet-capture", + "recording.pcap", + "--fields", + "ip.addr", + "--format", + "json", + ], + { + stdout: (text) => { + stdout += text; + }, + exit: () => {}, + }, + ); + expect(JSON.parse(stdout)).toMatchObject({ + code: "cleanup_incomplete", + message: "The selected capture input was rejected.", + details: { + diagnostics: { + primary_error: { code: "invalid_request" }, + }, + resources: [], + }, + }); +}); + +it("closes the service and retains an unexpected producer rejection", async () => { + const cli = Cli.create("capture-producer-rejection-test", { sync: false }); + let closed = false; + registerCaptureCommands( + cli, + silentLogger, + {}, + { + inspectFields: async () => { + throw new Error("Unexpected field discovery call"); + }, + inspect: async () => { + throw new Error("raw producer failure"); + }, + close: async () => { + closed = true; + throw new Error("raw close failure"); + }, + }, + ); + let stdout = ""; + await cli.serve( + [ + "inspect-packet-capture", + "recording.pcap", + "--fields", + "ip.addr", + "--format", + "json", + ], + { + stdout: (text) => { + stdout += text; + }, + exit: () => {}, + }, + ); + expect(closed).toBe(true); + expect(JSON.parse(stdout)).toMatchObject({ + code: "cleanup_incomplete", + details: { + diagnostics: { + primary_error: { + code: "execution_failure", + details: { + operation: "inspect_packet_capture", + provider_id: "tshark-capture", + }, + }, + cleanup_error: { code: "cleanup_incomplete" }, + }, + }, + }); +}); + it("rejects an oversized selector budget before the CLI invokes its service", async () => { const cli = Cli.create("capture-selector-test", { sync: false }); let called = false; @@ -87,6 +196,9 @@ it("rejects an oversized selector budget before the CLI invokes its service", as silentLogger, {}, { + inspectFields: async () => { + throw new Error("Unexpected field discovery call"); + }, inspect: async () => { called = true; throw new Error("Must not inspect"); @@ -125,6 +237,9 @@ it("rejects conflicting output options before the CLI invokes its service", asyn silentLogger, {}, { + inspectFields: async () => { + throw new Error("Unexpected field discovery call"); + }, inspect: async () => { called = true; throw new Error("Must not inspect"); diff --git a/tests/boundary/cli/cliOutput.test.ts b/tests/boundary/cli/cliOutput.test.ts index 43d517e16..44b3ac234 100644 --- a/tests/boundary/cli/cliOutput.test.ts +++ b/tests/boundary/cli/cliOutput.test.ts @@ -58,6 +58,95 @@ describe("CLI output argument boundary", () => { }); }); +describe("unclosed CLI output filter slices", () => { + it.each([ + "providers[", + "providers[0,1].name[", + "name,providers[", + "][,]", + "providers],name[0,1]", + ])("rejects %s before the filter parser runs", (filter) => { + expect( + validateCliOutputArguments([ + "providers", + "--json", + "--filter-output", + filter, + ]), + ).toMatchObject({ ok: false, code: "INVALID_OUTPUT_FILTER" }); + }); + + it("preserves effective-filter and earlier flag-error precedence", () => { + for (const arguments_ of [ + [ + "--filter-output", + "providers[", + "--filter-output", + "capabilities[0,1].provider.id,open", + ], + ["--filter-output", "capabilities[-1,].provider.id"], + ["--filter-output", "capabilities[[0,1]"], + ["--format", "invalid", "--filter-output", "providers["], + ["--token-limit", "invalid", "--filter-output", "providers["], + ]) + expect(validateCliOutputArguments(arguments_)).toEqual({ ok: true }); + }); + + workspaceCliTest( + "fails promptly across output formats without invoking the command", + async ({ cli }) => { + for (const format of ["json", "jsonl", "yaml", "toon", "md"]) { + const result = await cli.run({ + arguments: [ + "analyze", + "/absent/target", + "--format", + format, + "--filter-output", + "result[0,1].name[", + ], + timeoutMs: 3000, + }); + expect(result.exitCode).toBe(1); + expect(result.stdout).toContain("INVALID_OUTPUT_FILTER"); + expect(result.stdout).toContain("closing ]"); + expect(result.stderr).toBe(""); + } + const splitPath = await cli.run({ + arguments: ["providers", "--json", "--filter-output", "][,]"], + timeoutMs: 3000, + }); + expect(splitPath.exitCode).toBe(1); + expect(splitPath.json).toMatchObject({ + ok: false, + error: { code: "INVALID_OUTPUT_FILTER" }, + }); + }, + CLI_INTEGRATION_TIMEOUT_MS, + ); + + workspaceCliTest( + "retains closed slices and multiple filter paths", + async ({ cli }) => { + const result = await cli.run({ + arguments: [ + "providers", + "--json", + "--filter-output", + "capabilities[0,1].provider.id,open", + ], + timeoutMs: 3000, + }); + expect(result.exitCode).toBe(0); + expect(result.json).toEqual({ + capabilities: [{ provider: { id: expect.any(String) } }], + open: false, + }); + }, + CLI_INTEGRATION_TIMEOUT_MS, + ); +}); + describe("CLI output valued-flag parsing", () => { workspaceCliTest( "preserves complete JSON errors when native builtin flags follow unusual arguments", diff --git a/tests/boundary/cli/developmentTests.test.ts b/tests/boundary/cli/developmentTests.test.ts index 21d8c5127..2987206c8 100644 --- a/tests/boundary/cli/developmentTests.test.ts +++ b/tests/boundary/cli/developmentTests.test.ts @@ -1,100 +1,175 @@ -import { execFile } from "node:child_process"; -import { mkdir, symlink, writeFile } from "node:fs/promises"; -import { join, resolve } from "node:path"; -import { promisify } from "node:util"; +import { + copyFile, + mkdir, + readFile, + symlink, + writeFile, +} from "node:fs/promises"; +import { join } from "node:path"; import { describe, expect, it } from "vitest"; +import { execFileOutput } from "../../../src/process/ExecFileOutput.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; -const execute = promisify(execFile); -const runner = resolve("scripts/run-development-tests.mjs"); - -describe("development test runner", () => { - it("rejects nonexistent explicit paths instead of returning a zero-test success", async () => { - await expect( - execute(process.execPath, [ - runner, - "focused", - "src/missing.test.ts", - "--dry-run", - ]), - ).rejects.toMatchObject({ code: 1 }); +const run = (root: string, command: string, arguments_: readonly string[]) => + execFileOutput(command, arguments_, { + cwd: root, + stopSignal: "SIGTERM", + timeout: 30_000, + maxBuffer: 1024 * 1024, }); - it("reports actionable remediation when the comparison base is unavailable", async () => { - await expect( - execute(process.execPath, [ - runner, - "changed", - "--base", - "missing-test-base", - "--dry-run", +const git = (root: string, ...arguments_: string[]) => + run(root, "git", arguments_); + +const commit = (root: string, message: string) => + git( + root, + "-c", + "user.name=REA fixture", + "-c", + "user.email=fixture@example.test", + "-c", + "commit.gpgSign=false", + "commit", + "--quiet", + "-m", + message, + ); + +const fixture = async () => { + const root = await createTestTempDirectory("rea-development-tests-"); + await mkdir(join(root, "scripts", "lib"), { recursive: true }); + await mkdir(join(root, "src")); + for (const path of [ + "scripts/run-development-tests.mjs", + "scripts/lib/development-tests.mjs", + ]) + await copyFile(join(process.cwd(), path), join(root, path)); + await symlink( + join(process.cwd(), "node_modules"), + join(root, "node_modules"), + process.platform === "win32" ? "junction" : undefined, + ); + await writeFile(join(root, "package.json"), '{"type":"module"}\n'); + await writeFile(join(root, ".gitignore"), "node_modules\nselected.txt\n"); + await writeFile( + join(root, "vitest.config.mjs"), + 'export default { test: { projects: [{ test: { name: "domain", include: ["src/**/*.test.ts"] } }] } };\n', + ); + await writeFile(join(root, "src/value.ts"), "export const value = 1;\n"); + await writeFile( + join(root, "src/value.test.ts"), + 'import { expect, it } from "vitest"; import { writeFileSync } from "node:fs"; import { value } from "./value.js"; it("checks the selected source", () => { expect(value).toBe(1); writeFileSync("selected.txt", String(value)); });\n', + ); + await git(root, "init", "--quiet", "-b", "main"); + await git(root, "add", "."); + await commit(root, "fixture"); + return root; +}; + +const testCommand = (root: string, ...arguments_: string[]) => + run(root, process.execPath, [ + "scripts/run-development-tests.mjs", + ...arguments_, + ]); + +describe("development test runner public CLI", () => { + it("runs an explicitly selected source test on a clean tree without build artifacts", async () => { + const root = await fixture(); + const { stdout: status } = await git(root, "status", "--porcelain"); + expect(status).toBe(""); + await testCommand(root, "local", "src/value.test.ts"); + expect(await readFile(join(root, "selected.txt"), "utf8")).toBe("1"); + }); + + it("plans an exact local test without building", async () => { + const root = await fixture(); + const { stdout } = await testCommand( + root, + "local", + "src/value.test.ts", + "--dry-run", + ); + expect(JSON.parse(stdout)).toMatchObject({ + needsBuild: false, + artifactTasks: [], + vitestArguments: expect.arrayContaining([ + "run", + "--project", + "domain", + "--project", + "services", + "--project", + "adapters", + "--project", + "composition", + "--project", + "conformance", + "--project", + "evaluation", + "src/value.test.ts", ]), + }); + }); + + it("uses committed dependency changes to select source tests and reports failures", async () => { + const root = await fixture(); + const { stdout: base } = await git(root, "rev-parse", "HEAD"); + await writeFile(join(root, "src/value.ts"), "export const value = 2;\n"); + await git(root, "add", "src/value.ts"); + await commit(root, "change dependency"); + await expect( + testCommand(root, "changed", "--base", base.trim()), + ).rejects.toMatchObject({ + code: 1, + stdout: expect.stringContaining("src/value.test.ts"), + stderr: expect.stringContaining("expected 2 to be 1"), + }); + }); + + it("runs changed and focused tests from a committed branch regression", async () => { + const root = await fixture(); + const { stdout: baseline } = await git(root, "rev-parse", "HEAD"); + await git(root, "update-ref", "refs/remotes/origin/main", baseline.trim()); + await git(root, "switch", "--quiet", "-c", "regression"); + await writeFile(join(root, "src/value.ts"), "export const value = 2;\n"); + await writeFile( + join(root, "src/value.test.ts"), + 'import { expect, it } from "vitest"; import { value } from "./value.js"; it("checks current source", () => expect(value).toBe(2));\n', + ); + await git(root, "add", "src"); + await commit(root, "regression fix"); + expect((await git(root, "status", "--porcelain")).stdout).toBe(""); + + const changed = await testCommand(root, "changed"); + expect(changed.stdout).toContain("1 passed"); + const focused = await testCommand(root, "focused", "src/value.test.ts"); + expect(focused.stdout).toContain("1 passed"); + }, 60_000); + + it("reports a missing comparison base and rejects missing or compiled-boundary paths", async () => { + const root = await fixture(); + await expect( + testCommand(root, "changed", "--base", "missing-test-base", "--dry-run"), ).rejects.toMatchObject({ code: 1, stderr: expect.stringContaining( "Fetch origin/main or pass --base REVISION", ), }); - }); - - it("runs a committed branch regression through real Git and Vitest without dist", async () => { - const root = await createTestTempDirectory("rea-development-tests-"); - await mkdir(join(root, "src")); - await symlink( - resolve("node_modules"), - join(root, "node_modules"), - "junction", - ); - await writeFile( - join(root, "package.json"), - JSON.stringify({ type: "module" }), - ); - await writeFile(join(root, ".gitignore"), "node_modules\n"); - await writeFile( - join(root, "vitest.config.ts"), - `export default { test: { projects: [{ test: { name: "domain", include: ["src/**/*.test.ts"] } }] } };`, - ); - await writeFile(join(root, "src/value.ts"), "export const value = 1;\n"); - await writeFile( - join(root, "src/value.test.ts"), - `import { it, expect } from "vitest"; import { value } from "./value"; it("checks current source", () => expect(value).toBe(1));\n`, - ); - const git = (arguments_: string[]) => - execute("git", arguments_, { cwd: root }); - await git(["init", "-b", "main"]); - await git(["add", ".gitignore", "package.json", "vitest.config.ts", "src"]); - const commit = [ - "-c", - "user.name=REA fixture", - "-c", - "user.email=fixture@example.test", - "commit", - "-m", - "fixture", - ]; - await git(commit); - const { stdout: baseline } = await git(["rev-parse", "HEAD"]); - await git(["update-ref", "refs/remotes/origin/main", baseline.trim()]); - await git(["switch", "-c", "regression"]); - await writeFile(join(root, "src/value.ts"), "export const value = 2;\n"); - await writeFile( - join(root, "src/value.test.ts"), - `import { it, expect } from "vitest"; import { value } from "./value"; it("checks current source", () => expect(value).toBe(2));\n`, - ); - await git(["add", "src"]); - await git(commit); - expect((await git(["status", "--porcelain"])).stdout).toBe(""); - const { stdout } = await execute(process.execPath, [runner, "changed"], { - cwd: root, + await expect( + testCommand(root, "focused", "src/missing.test.ts"), + ).rejects.toMatchObject({ + code: 1, + stderr: expect.stringContaining("ENOENT"), }); - expect(stdout).toContain("1 passed"); - const explicit = await execute( - process.execPath, - [runner, "focused", "src/value.test.ts"], - { cwd: root }, - ); - expect(explicit.stdout).toContain("1 passed"); - }, 30_000); + await expect( + testCommand(root, "local", "tests/acceptance/runtime.test.ts"), + ).rejects.toMatchObject({ + code: 1, + stderr: expect.stringContaining("Use test:focused"), + }); + }); }); diff --git a/tests/boundary/cli/dylibResolutionCli.test.ts b/tests/boundary/cli/dylibResolutionCli.test.ts index b74dae71e..21ea99a9c 100644 --- a/tests/boundary/cli/dylibResolutionCli.test.ts +++ b/tests/boundary/cli/dylibResolutionCli.test.ts @@ -4,11 +4,13 @@ import { dirname, join } from "node:path"; import { describe, expect } from "vitest"; import { + CPU, FILE_TYPE, LC, buildVersionCommand, dylibCommand, dyldEnvironmentCommand, + fatImage, machoImage, rpathCommand, } from "../../../src/artifacts/apple/MachoImage.fixture.js"; @@ -178,14 +180,17 @@ describe("trace-dylib-resolution CLI", () => { // A root below a symlinked directory that points outside the bundle. const outside = await createTestTempDirectory("rea-dylib-outside-"); await writeFiles(outside, { Tool: machoImage({}) }); - await symlink(outside, join(app, "Contents/Linked")); + await symlink(outside, join(app, "Contents/Linked"), "dir"); for (const [root, reason, message] of [ ["../outside", "invalid_format", "Use a normalized path"], ["Contents/Info.plist", "invalid_value", "is not a Mach-O image"], [ "Contents/Linked/Tool", "invalid_value", - "resolves outside the analyzed root", + // Windows link text is outside the POSIX dyld path representation. + process.platform === "win32" + ? "is not a regular file in the analyzed root" + : "resolves outside the analyzed root", ], ] as const) { await expectRootInputError(cli, { app, root, reason, message }); @@ -376,7 +381,8 @@ cliTest( }, ); -describe.skipIf(process.getuid?.() === 0)( +// This permission fixture requires POSIX mode bits and a non-root caller. +describe.skipIf(process.platform === "win32" || process.getuid?.() === 0)( "trace-dylib-resolution permissions", () => { cliTest( @@ -471,6 +477,64 @@ cliTest( }, ); +cliTest( + "reports each same-named slice's own embedded dyld environment", + async ({ cli }) => { + // arm64e and arm64e.x1 slices both report the architecture name arm64e. + // A plain x86_64 slice keeps the bundle openable on x64 hosts. + for (const [settings, expected] of [ + [[null, "DYLD_PRINT_RPATHS=1"], ["DYLD_PRINT_RPATHS=1"]], + [ + ["DYLD_PRINT_LIBRARIES=1", "DYLD_PRINT_RPATHS=1"], + ["DYLD_PRINT_LIBRARIES=1", "DYLD_PRINT_RPATHS=1"], + ], + ] as const) { + const app = await fixtureApp(); + await writeFiles(app, { + "Contents/MacOS/App": fatImage([ + ...[CPU.arm64e, CPU.arm64eX1].map((cpu, index) => { + const setting = settings[index]; + return { + cpu, + bytes: machoImage({ + cpu, + commands: + setting === null || setting === undefined + ? [] + : [dyldEnvironmentCommand(setting)], + }), + }; + }), + { cpu: CPU.x86_64, bytes: machoImage({ cpu: CPU.x86_64 }) }, + ]), + }); + const result = await cli.run({ + arguments: [ + "trace-dylib-resolution", + app, + "--root", + "Contents/MacOS/App", + "--architecture", + "arm64e", + "--json", + ], + environment: ENVIRONMENT, + }); + expect(result.exitCode, JSON.stringify(result.json)).toBe(0); + const trace = dylibResolutionResultSchema.parse( + parseEvidence(result.json).normalized_result, + ); + expect(trace.roots).toHaveLength(2); + const environments = trace.findings + .filter(({ kind }) => kind === "dyld-environment-present") + .map(({ explanation }) => + expected.filter((setting) => explanation.includes(setting)), + ); + expect(environments).toEqual(expected.map((setting) => [setting])); + } + }, +); + cliTest( "retains definitive load findings for an explicitly selected app library root", async ({ cli }) => { diff --git a/tests/boundary/cli/idaRetainedCleanupCli.test.ts b/tests/boundary/cli/idaRetainedCleanupCli.test.ts new file mode 100644 index 000000000..b31903489 --- /dev/null +++ b/tests/boundary/cli/idaRetainedCleanupCli.test.ts @@ -0,0 +1,92 @@ +import { describe, expect } from "vitest"; + +import { workspaceCliTest } from "../../support/cli/workspaceCliFixture.js"; + +import { elf } from "../../../src/domain/binaryTarget.fixture.js"; + +/** + * A headless IDA supervisor that completes the MCP handshake and then leaves + * every tool call unanswered, as an `idb_open` still running auto-analysis + * would. It exits when REA closes its input. + */ +const UNRESPONSIVE_SUPERVISOR = `import { createInterface } from "node:readline"; +const tools = ["idb_open", "idb_list", "idb_close", "server_health", "list_funcs", + "lookup_funcs", "find_regex", "decompile", "disasm", "callees", "xrefs_to"]; +const send = (message) => + process.stdout.write(JSON.stringify({ jsonrpc: "2.0", ...message }) + "\\n"); +for await (const line of createInterface({ input: process.stdin })) { + const request = JSON.parse(line); + if (request.method === "initialize") + send({ id: request.id, result: { + protocolVersion: request.params.protocolVersion, + capabilities: { tools: {} }, + serverInfo: { name: "unresponsive-ida-fixture", version: "1" }, + } }); + else if (request.method === "tools/list") + send({ id: request.id, result: { + tools: tools.map((name) => ({ name, inputSchema: { type: "object" } })), + } }); +} +`; + +describe("headless IDA timeout through the one-shot CLI", () => { + workspaceCliTest( + "exits after a retained-cleanup failure instead of waiting on the supervisor", + async ({ cli, workspace }) => { + const supervisor = await workspace.write( + "supervisor.mjs", + UNRESPONSIVE_SUPERVISOR, + ); + const workspaceRoot = await workspace.mkdir("ida-workspaces"); + const registration = await workspace.write( + "ida-mcp.json", + JSON.stringify({ + command: process.execPath, + args: [supervisor], + mode: "headless", + timeoutMs: 500, + workspaceRoot, + }), + ); + const target = await workspace.write("target.elf", elf(2, 1, 62)); + + const result = await cli.run({ + arguments: ["decompile", target, "main", "--json"], + environment: { + ...workspace.environment, + REA_ANALYSIS_PROVIDER: "ida", + REA_IDA_MCP_CONFIG: registration, + REA_LOG_LEVEL: "silent", + }, + timeoutMs: 30_000, + }); + + expect(result.exitCode).toBe(1); + expect(result.json).toMatchObject({ + code: "cleanup_incomplete", + details: { + provider_id: "ida", + cleanup: "incomplete", + resources: [expect.stringContaining(workspaceRoot)], + cleanup_error: { + details: { + diagnostics: { + upstream_sessions: { + state: "unverified", + sessions: [ + { + session_id: expect.stringMatching(/^rea-/), + backend: "worker", + owned: null, + input_path: null, + }, + ], + }, + }, + }, + }, + }, + }); + }, + ); +}); diff --git a/tests/boundary/cli/jsonOutput.test.ts b/tests/boundary/cli/jsonOutput.test.ts index 5e8e86e7f..0d1ee5a59 100644 --- a/tests/boundary/cli/jsonOutput.test.ts +++ b/tests/boundary/cli/jsonOutput.test.ts @@ -561,3 +561,35 @@ describe("CLI result output surface", () => { }); }, 60_000); }); + +describe("streamed CLI filtered selections", () => { + it.each([ + ["raw_result", "json", "null\n"], + ["raw_result", "jsonl", "null\n"], + ["nested.value", "json", '{"nested":{"value":null}}\n'], + ["missing", "json", "{}\n"], + ["count", "json", "0\n"], + ["enabled", "json", "false\n"], + ] as const)( + "keeps the selected value of --filter-output %s in %s", + async (filter, format, expected) => { + const chunks: Buffer[] = []; + const destination = new Writable({ + write(chunk: Buffer, _encoding, callback) { + chunks.push(chunk); + callback(); + }, + }); + const output = createCliResultOutput( + ["--format", format, "--filter-output", filter], + destination, + ); + if (output === undefined) throw new Error("Missing JSON output surface"); + await output.write( + { raw_result: null, nested: { value: null }, count: 0, enabled: false }, + { command: "fixture", duration: "1ms", format }, + ); + expect(Buffer.concat(chunks).toString("utf8")).toBe(expected); + }, + ); +}); diff --git a/tests/boundary/cli/nativeMachoEchoedPathCli.test.ts b/tests/boundary/cli/nativeMachoEchoedPathCli.test.ts index 0c362d263..a3118c1f0 100644 --- a/tests/boundary/cli/nativeMachoEchoedPathCli.test.ts +++ b/tests/boundary/cli/nativeMachoEchoedPathCli.test.ts @@ -1,9 +1,13 @@ import { copyFile } from "node:fs/promises"; -import { join } from "node:path"; +import { join, resolve } from "node:path"; import { describe, expect } from "vitest"; import { z } from "zod"; +import { + inspectMachoSchema, + listArchitecturesSchema, +} from "../../../src/domain/native/nativeInspection.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { cliTest } from "../../support/cli/cliFixture.js"; @@ -34,27 +38,68 @@ const resultSchema = z.object({ describe.skipIf(process.platform !== "darwin")( "native Mach-O CLI with echoed path text", () => { - cliTest( - "does not read load commands or symbols from the file name", - async ({ cli }) => { - const directory = await createTestTempDirectory("rea-macho-echo-"); - const inspect = async (name: string) => { - const path = join(directory, name); - await copyFile("/usr/bin/true", path); - const result = await cli.run({ - arguments: ["inspect-macho", path, "--json"], - environment, - }); - expect(result.exitCode).toBe(0); - const { provenance: _provenance, ...facts } = resultSchema.parse( - result.json, - ).normalized_result; - return facts; - }; - const forged = await inspect(FORGED_NAME); - expect(forged).toEqual(await inspect("tool")); - expect(JSON.stringify(forged)).not.toMatch(/forged|4242/u); - }, - ); + const lipoName = "tool\narchitecture forged\n offset 123\n size 32"; + for (const [facts, command, filename] of [ + ["load commands or symbols", "inspect-macho", FORGED_NAME], + ["FAT architecture records", "inspect-macho", lipoName], + ["FAT architecture inventory", "list-architectures", lipoName], + ["thin-file framing", "inspect-macho", "tool\nNon-fat file: forged"], + ] as const) { + cliTest( + `does not read ${facts} from the file name`, + async ({ cli, processes }) => { + const directory = await createTestTempDirectory( + "rea-native-cli-echo-", + ); + const fixture = join(directory, "fat-fixture"); + const compiled = await processes.run( + "/usr/bin/xcrun", + [ + "clang", + "-arch", + "arm64", + "-arch", + "x86_64", + resolve("tests/conformance/c/fixture.c"), + "-o", + fixture, + ], + { timeoutMs: 30_000 }, + ); + expect(compiled.exitCode, compiled.stderr).toBe(0); + const inspect = async (name: string) => { + const path = join(directory, name); + await copyFile(fixture, path); + const result = await cli.run({ + arguments: [command, path, "--json"], + environment, + }); + expect(result.exitCode).toBe(0); + const { provenance: _provenance, ...facts } = resultSchema.parse( + result.json, + ).normalized_result; + return facts; + }; + const ordinary = await inspect("tool"); + // Prove this comparison exercised FAT framing, independent of host binaries. + const architectures = + listArchitecturesSchema.shape.architectures.parse( + ordinary.architectures, + ); + expect(architectures.total).toBeGreaterThan(1); + if (command === "inspect-macho") { + const selected = + inspectMachoSchema.shape.selected_architecture.parse( + ordinary.selected_architecture, + ); + expect(selected.container_file_offset).toBeGreaterThan(0); + } + const forged = await inspect(filename); + expect(forged).toEqual(ordinary); + expect(JSON.stringify(forged)).not.toMatch(/forged|4242/u); + }, + 60_000, + ); + } }, ); diff --git a/tests/boundary/cli/nativeSignatureCli.test.ts b/tests/boundary/cli/nativeSignatureCli.test.ts index 9302d9c24..e30581db4 100644 --- a/tests/boundary/cli/nativeSignatureCli.test.ts +++ b/tests/boundary/cli/nativeSignatureCli.test.ts @@ -21,14 +21,14 @@ const adHocProgram = async (directory: string, name: string) => { describe.skipIf(process.platform !== "darwin")( "native signature CLI with echoed path text", () => { - cliTest( - "does not read signature fields from the file name codesign echoes", - async ({ cli }) => { + cliTest.for(["\n", "\r\n"])( + "does not read signature fields from the file name codesign echoes (%j)", + async (newline, { cli }) => { const directory = await createTestTempDirectory("rea-signature-"); const forged = "Developer ID Application: Example (ABCDE12345)"; const program = await adHocProgram( directory, - `tool\nAuthority=${forged}\nTeamIdentifier=ABCDE12345`, + `tool${newline}Authority=${forged}${newline}Identifier=forged${newline}TeamIdentifier=ABCDE12345`, ); const unsignedName = await adHocProgram( directory, @@ -75,7 +75,7 @@ describe.skipIf(process.platform !== "darwin")( normalized_result: { identifier: expect.stringMatching( new RegExp( - `^tool\\nAuthority=${forged.replace(/[()]/gu, "\\$&")}\\nTeamIdentifier=ABCDE12345-`, + `^tool${newline}Authority=${forged.replace(/[()]/gu, "\\$&")}${newline}Identifier=forged${newline}TeamIdentifier=ABCDE12345-`, "u", ), ), diff --git a/tests/boundary/cli/pcapCli.test.ts b/tests/boundary/cli/pcapCli.test.ts new file mode 100644 index 000000000..1eb61cdcf --- /dev/null +++ b/tests/boundary/cli/pcapCli.test.ts @@ -0,0 +1,37 @@ +import { expect, it } from "vitest"; + +import { expectConfigurationRefusal, serveCli } from "./cliServeHarness.js"; + +it.each(["inspect-pcap-client", "inspect-pcap-capture"])( + "%s advertises a schema through the public CLI", + async (command) => { + const result = await serveCli([command, "--schema", "--json"]); + expect(result.exitCode).toBe(0); + expect(JSON.parse(result.stdout)).toBeTruthy(); + }, +); + +it.each([ + ["inspect-pcap-client"], + ["inspect-pcap-capture", "/nonexistent/capture.pcap"], +])("%s names the missing Wireshark tool to configure", async (...argv) => { + const result = await serveCli([...argv, "--json"], { + REA_TSHARK_COMMAND: "/nonexistent/tshark", + REA_CAPINFOS_COMMAND: "/nonexistent/capinfos", + }); + expectConfigurationRefusal( + result, + "Cannot run selected tool /nonexistent/tshark", + ); + expectConfigurationRefusal(result, "through absolute REA_TSHARK_COMMAND"); +}); + +it("names a relative tool selection as configuration to fix", async () => { + const result = await serveCli(["inspect-pcap-client", "--json"], { + REA_TSHARK_COMMAND: "tshark", + }); + expectConfigurationRefusal( + result, + "REA_TSHARK_COMMAND must be an absolute path to the tshark binary", + ); +}); diff --git a/tests/boundary/cli/rawBinaryCli.test.ts b/tests/boundary/cli/rawBinaryCli.test.ts index 1cff4cac6..f12312d02 100644 --- a/tests/boundary/cli/rawBinaryCli.test.ts +++ b/tests/boundary/cli/rawBinaryCli.test.ts @@ -38,6 +38,36 @@ cliTest( }, ); +cliTest( + "rejects a 6502 image crossing the 16-bit boundary before provider startup", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-6502-cli-overflow-"); + const path = join(root, "image.bin"); + await writeFile(path, Buffer.from("a92a60", "hex")); + const result = await cli.run({ + arguments: [ + "inspect", + path, + "--target-format", + "raw-binary", + "--architecture", + "6502", + "--entry-point", + "0xffff", + "--provider", + "ghidra", + "--json", + ], + environment: { HOME: root, XDG_CONFIG_HOME: root, XDG_CACHE_HOME: root }, + }); + expect(result.exitCode).toBe(1); + expect(result.json).toMatchObject({ + code: "invalid_request", + details: { reason: expect.stringContaining("6502 address width") }, + }); + }, +); + cliTest( "rejects an unaligned AArch64 entry before provider startup", async ({ cli }) => { diff --git a/tests/boundary/cli/referenceSourceDirectoryImports.test.ts b/tests/boundary/cli/referenceSourceDirectoryImports.test.ts new file mode 100644 index 000000000..6270855cb --- /dev/null +++ b/tests/boundary/cli/referenceSourceDirectoryImports.test.ts @@ -0,0 +1,118 @@ +import { mkdir, writeFile } from "node:fs/promises"; +import { createRequire } from "node:module"; +import { dirname, join } from "node:path"; + +import { describe, expect } from "vitest"; + +import { historicalSourceGraphSchema } from "../../../src/domain/referenceSourceGraph.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; + +// Reference-source reads require no-follow opens; Windows reports unsupported. +describe.skipIf(process.platform === "win32")( + "compiled reference-source directory imports", + () => { + cliTest( + "resolves directory spellings without selecting similarly named files", + async ({ cli }) => { + const workspace = await createTestTempDirectory( + "rea-reference-directory-imports-", + ); + const root = join(workspace, "source"); + const imports = [ + ["slash.cjs", "./lib/", "lib/index.js"], + ["dot.cjs", "./lib/.", "lib/index.js"], + ["self.cjs", ".", "index.js"], + ["self-slash.cjs", "./", "index.js"], + ["nested/parent.cjs", "..", "index.js"], + ["nested/parent-slash.cjs", "../", "index.js"], + ["direct.cjs", "./lib.js", "lib.js"], + ["fallback.cjs", "./fallback", "fallback/index.js"], + ["legacy-order.cjs", "./dep", "dep/index.js"], + ] as const; + const unresolved = [ + ["missing.cjs", "./missing/", "missing/"], + ["escape.cjs", "../outside/", "../outside/"], + ["main.ts", "./folder.js/.", "folder.js"], + ] as const; + const files = new Map([ + ["index.js", 'module.exports = "root index";\n'], + ["lib/index.js", 'module.exports = "library index";\n'], + ["lib.js", 'module.exports = "sibling JavaScript file";\n'], + ["lib.ts", 'export const value = "sibling TypeScript file";\n'], + ["fallback/index.js", 'module.exports = "fallback index";\n'], + ["dep/index.js", 'module.exports = "existing index priority";\n'], + ["dep.cjs", 'module.exports = "later CommonJS candidate";\n'], + [ + "folder.ts", + 'export const value = "emitted filename counterpart";\n', + ], + ["missing.js", 'module.exports = "not a directory";\n'], + ...[...imports, ...unresolved].map( + ([path, specifier]) => + [path, `require(${JSON.stringify(specifier)});\n`] as const, + ), + ]); + await Promise.all( + [...files].map(async ([path, contents]) => { + const target = join(root, path); + await mkdir(dirname(target), { recursive: true }); + await writeFile(target, contents); + }), + ); + const outside = join(workspace, "outside", "index.js"); + await mkdir(dirname(outside), { recursive: true }); + await writeFile(outside, 'module.exports = "outside import root";\n'); + + // Node's loader establishes each fixture's meaning independently. + for (const [path, specifier, target] of imports) + expect(createRequire(join(root, path)).resolve(specifier)).toBe( + join(root, target), + ); + expect(() => + createRequire(join(root, "missing.cjs")).resolve("./missing/"), + ).toThrow(); + expect(() => + createRequire(join(root, "main.ts")).resolve("./folder.js/."), + ).toThrow(); + expect( + createRequire(join(root, "escape.cjs")).resolve("../outside/"), + ).toBe(outside); + + const result = await cli.run({ + arguments: ["import-reference-source", root, "--json"], + environment: { + HOME: workspace, + XDG_CONFIG_HOME: workspace, + XDG_CACHE_HOME: workspace, + }, + }); + expect(result.exitCode).toBe(0); + const graph = historicalSourceGraphSchema.parse(result.json); + expect(graph.parse_failures).toEqual([]); + expect(graph.relationships).toHaveLength( + imports.length + unresolved.length, + ); + for (const [from, , to] of imports) + expect.soft(graph.relationships, from).toContainEqual({ + from_path: from, + to, + kind: "requires", + resolution: "internal", + parse_state: "parsed", + }); + for (const [from, , to] of unresolved) + expect.soft(graph.relationships, from).toContainEqual({ + from_path: from, + to, + kind: "requires", + resolution: "unresolved", + parse_state: "parsed", + }); + expect( + graph.entries.some((entry) => entry.path.includes("outside")), + ).toBe(false); + }, + ); + }, +); diff --git a/tests/boundary/cli/scenarioCliInputDiagnostics.test.ts b/tests/boundary/cli/scenarioCliInputDiagnostics.test.ts index bc98d9d9f..ae03568f4 100644 --- a/tests/boundary/cli/scenarioCliInputDiagnostics.test.ts +++ b/tests/boundary/cli/scenarioCliInputDiagnostics.test.ts @@ -1,3 +1,5 @@ +import { join } from "node:path"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { describe, expect } from "vitest"; import { cliTest } from "../../support/cli/cliFixture.js"; @@ -62,3 +64,42 @@ describe("scenario CLI schema diagnostics", () => { }); }); }); + +for (const field of [ + "executable_path", + "application_path", + "application_root", +] as const) { + cliTest(`names the unresolved Electron ${field}`, async ({ cli }) => { + const root = await createTestTempDirectory("rea-electron-path-"); + const missing = join(root, "missing"); + const result = await cli.run({ + arguments: [ + "capture-electron-scenario", + JSON.stringify({ + executable_path: process.execPath, + application_path: root, + application_root: root, + actions: [], + [field]: missing, + }), + "--json", + ], + environment: { REA_LOG_LEVEL: "silent" }, + }); + expect(result.exitCode).toBe(1); + expect(result.json).toMatchObject({ + code: "invalid_request", + details: { + issues: [ + { + path: [field], + reason: "invalid_value", + message: expect.stringContaining("ENOENT"), + }, + ], + }, + }); + expect(JSON.stringify(result.json)).toContain(missing); + }); +} diff --git a/tests/boundary/cli/signalCleanup.test.ts b/tests/boundary/cli/signalCleanup.test.ts index 594a46cc0..dc706c17c 100644 --- a/tests/boundary/cli/signalCleanup.test.ts +++ b/tests/boundary/cli/signalCleanup.test.ts @@ -30,6 +30,9 @@ it.each([ silentLogger, {}, { + inspectDecoders: async () => { + throw new Error("Unexpected discovery call"); + }, decode: async () => { called = true; throw new Error("Must not decode"); @@ -84,6 +87,9 @@ it.each([false, true])( silentLogger, {}, { + inspectDecoders: async () => { + throw new Error("Unexpected discovery call"); + }, decode: async () => err(original), close: async () => { await Promise.resolve(); diff --git a/tests/boundary/filesystem/analysisSnapshotInput.test.ts b/tests/boundary/filesystem/analysisSnapshotInput.test.ts new file mode 100644 index 000000000..85bd7b7d1 --- /dev/null +++ b/tests/boundary/filesystem/analysisSnapshotInput.test.ts @@ -0,0 +1,88 @@ +import { readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { describe, expect, it } from "vitest"; + +import { readAnalysisSnapshot } from "../../../src/application/binary/AnalysisSnapshotFiles.js"; +import { projectAnalysisError } from "../../../src/domain/analysisErrorProjection.js"; +import { + snapshotBinding, + snapshotTarget, +} from "../../../src/domain/analysisSnapshot.js"; +import { + ANALYSIS_SNAPSHOT_PROFILE, + ANALYSIS_SNAPSHOT_TARGET, +} from "../../../src/domain/analysisSnapshot.fixture.js"; +import { createEvidenceBundle } from "../../../src/domain/evidenceBundle.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +describe.each([ + { mode: "native", paddingBytes: 0 }, + { mode: "streamed", paddingBytes: 9 * 1024 * 1024 }, +])("analysis snapshot input ($mode)", ({ paddingBytes }) => { + it.each(["[]", "null", "5", '"x"', "true"])( + "classifies a readable %s root as invalid content, not an I/O failure", + async (document) => { + const root = await createTestTempDirectory("rea-snapshot-root-"); + const path = join(root, "snapshot.json"); + const encoded = `${" ".repeat(paddingBytes)}${document}`; + await writeFile(path, encoded); + + const loaded = await readAnalysisSnapshot(path); + if (loaded.ok) throw new Error("Expected a non-object root rejection"); + const projected = projectAnalysisError(loaded.error); + expect(projected).toMatchObject({ + code: "evidence_integrity_mismatch", + details: { reason: "invalid-json" }, + }); + expect(projected.message).toContain("Repair or recreate"); + expect(loaded.error.cause).toMatchObject({ + message: "JSON document root must be an object", + }); + expect(await readFile(path, "utf8")).toBe(encoded); + }, + ); + + it("retains syntax and snapshot-member validation and loads valid objects", async () => { + const root = await createTestTempDirectory("rea-snapshot-validation-"); + const path = join(root, "snapshot.json"); + const prefix = " ".repeat(paddingBytes); + await writeFile(path, `${prefix}[`); + expect(await readAnalysisSnapshot(path)).toMatchObject({ + ok: false, + error: { _tag: "EvidenceFileError", reason: "invalid-json" }, + }); + await writeFile(path, `${prefix}{}`); + expect(await readAnalysisSnapshot(path)).toMatchObject({ + ok: false, + error: { _tag: "EvidenceIntegrityError" }, + }); + const snapshot = { + target: snapshotTarget(ANALYSIS_SNAPSHOT_TARGET), + binding: snapshotBinding(ANALYSIS_SNAPSHOT_PROFILE), + entries: [], + workflow_entries: [], + evidence_bundle: createEvidenceBundle([]), + }; + await writeFile(path, `${prefix}${JSON.stringify(snapshot)}`); + expect(await readAnalysisSnapshot(path)).toEqual({ + ok: true, + value: snapshot, + }); + }); +}); + +it("keeps file-selection failures distinct from invalid snapshot contents", async () => { + const root = await createTestTempDirectory("rea-snapshot-selection-"); + for (const [path, reason] of [ + [join(root, "missing.json"), "missing"], + [root, "not-file"], + ] as const) { + const loaded = await readAnalysisSnapshot(path); + if (loaded.ok) throw new Error("Expected a file-selection failure"); + expect(projectAnalysisError(loaded.error)).toMatchObject({ + code: "invalid_request", + details: { reason }, + }); + } +}); diff --git a/tests/boundary/filesystem/artifactZipIntegrity.test.ts b/tests/boundary/filesystem/artifactZipIntegrity.test.ts index 0bb475518..cf7dcd595 100644 --- a/tests/boundary/filesystem/artifactZipIntegrity.test.ts +++ b/tests/boundary/filesystem/artifactZipIntegrity.test.ts @@ -38,8 +38,19 @@ it("completes an empty ZIP extraction with an empty identity", async () => { expect(await readdir(output)).toEqual([]); }); +const legacyControlNames = [ + ["legacy ASCII control", Buffer.from("Icon\r"), false, "Icon\r"], + [ + "legacy CP437 control", + Buffer.from([0x82, 0x09, 0x7f, 0x2e, 0x61]), + false, + "é\t\u007F.a", + ], +] as const; + it.each([ ["legacy CP437", Buffer.from([0x82, 0x2e, 0x74, 0x78, 0x74]), false, "é.txt"], + ...legacyControlNames, [ "EFS with leading U+FEFF", Buffer.from([0xef, 0xbb, 0xbf, 0x61, 0x62]), @@ -68,6 +79,38 @@ it.each([ }, ); +it.skipIf(process.platform === "win32").each(legacyControlNames)( + "extracts %s ZIP members under their stored names", + async (_label, rawName, utf8Flag, logicalPath) => { + const root = await createTestTempDirectory("rea-zip-extracted-name-"); + const path = join(root, "names.zip"); + await writeSingleEntryZip(path, "x.txt"); + const archive = await readFile(path); + await writeFile( + path, + rewriteSingleEntryName(archive, "x.txt", rawName, utf8Flag), + ); + const target = await parseBinaryTarget(path); + if (!target.ok) throw target.error; + + const output = join(root, "output"); + const result = await withArtifactClient(target.value, (client) => + client.execute("extract_artifact", { output_root: output }), + ); + if (!result.ok) throw result.error; + const extraction = artifactExtractionResultSchema.parse( + result.value.result, + ); + expect( + extraction.artifacts.map(({ relative_path }) => relative_path), + ).toEqual([logicalPath]); + expect(await readdir(output)).toEqual([logicalPath]); + expect(await readFile(join(output, logicalPath), "utf8")).toBe( + "member bytes", + ); + }, +); + it("preserves valid Info-ZIP Unicode Path overrides", async () => { const root = await createTestTempDirectory("rea-zip-unicode-path-"); const path = join(root, "unicode-path.zip"); @@ -323,16 +366,11 @@ it("rejects overlapping ZIP member data ranges through artifact inventory", asyn bytes.writeUInt32LE(firstLocalOffset, second + 42); await writeFile(path, bytes); - const target = await parseBinaryTarget(path); - if (!target.ok) throw target.error; - const result = await withArtifactClient(target.value, (client) => - client.execute("inventory_artifact", {}), + await expectInventoryFailure( + path, + "format", + "ZIP archive is malformed: Overlapping entry found", ); - - expect(result).toMatchObject({ - ok: false, - error: { _tag: "ArtifactOperationError", reason: "io" }, - }); }); it("preserves encrypted-member and corrupt-CRC observations", async () => { @@ -395,16 +433,11 @@ it("preserves encrypted-member and corrupt-CRC observations", async () => { bytes[dataOffset] = firstByte ^ 1; await writeFile(corruptPath, bytes); - const corruptTarget = await parseBinaryTarget(corruptPath); - if (!corruptTarget.ok) throw corruptTarget.error; - const corruptResult = await withArtifactClient( - corruptTarget.value, - (client) => client.execute("inventory_artifact", {}), + await expectInventoryFailure( + corruptPath, + "format", + "ZIP archive is malformed: Invalid compressed data", ); - expect(corruptResult).toMatchObject({ - ok: false, - error: { _tag: "ArtifactOperationError", reason: "io" }, - }); }); it("rejects a truncated ZIP through artifact inventory", async () => { @@ -412,17 +445,51 @@ it("rejects a truncated ZIP through artifact inventory", async () => { const path = join(root, "truncated.zip"); await writeFile(path, Buffer.from([0x50, 0x4b, 0x03, 0x04])); + await expectInventoryFailure( + path, + "format", + "ZIP archive is malformed: File format is not recognized", + ); +}); + +it("reports an unsupported ZIP compression method as unavailable", async () => { + const root = await createTestTempDirectory("rea-zip-method-"); + const path = join(root, "lzma.zip"); + const writer = new ZipWriter(new Uint8ArrayWriter()); + await writer.add("data.txt", new TextReader("member bytes"), { level: 0 }); + const bytes = Buffer.from(await writer.close()); + // Declare LZMA (14) in both headers; REA reads only stored and Deflate data. + const localHeader = bytes.indexOf(Buffer.from([0x50, 0x4b, 0x03, 0x04])); + const centralHeader = bytes.indexOf(Buffer.from([0x50, 0x4b, 0x01, 0x02])); + if (localHeader < 0 || centralHeader < 0) + throw new Error("Expected ZIP local and central headers"); + bytes.writeUInt16LE(14, localHeader + 8); + bytes.writeUInt16LE(14, centralHeader + 10); + await writeFile(path, bytes); + + await expectInventoryFailure( + path, + "unavailable", + "ZIP archive uses an unsupported feature: Compression method not supported", + ); +}); + +/** Inventory one archive through the provider and expect a typed failure. */ +const expectInventoryFailure = async ( + path: string, + reason: string, + detail: string, +): Promise => { const target = await parseBinaryTarget(path); if (!target.ok) throw target.error; const result = await withArtifactClient(target.value, (client) => client.execute("inventory_artifact", {}), ); - expect(result).toMatchObject({ ok: false, - error: { _tag: "ArtifactOperationError", reason: "io" }, + error: { _tag: "ArtifactOperationError", reason, detail }, }); -}); +}; const writeSingleEntryZip = async ( path: string, diff --git a/tests/boundary/filesystem/ciChangeScope.test.ts b/tests/boundary/filesystem/ciChangeScope.test.ts index 616c44957..11ba04fa5 100644 --- a/tests/boundary/filesystem/ciChangeScope.test.ts +++ b/tests/boundary/filesystem/ciChangeScope.test.ts @@ -1,4 +1,5 @@ import { + chmod, cp, mkdir, readFile, @@ -7,6 +8,7 @@ import { writeFile, } from "node:fs/promises"; import { dirname, join, matchesGlob } from "node:path"; +import { fileURLToPath } from "node:url"; import { afterAll, beforeAll, expect, it } from "vitest"; import { parse } from "yaml"; @@ -78,12 +80,13 @@ async function classify( head: string, event = "pull_request", full = false, + skipFull = false, ) { const output = join(directory, "scope-output"); await writeFile(output, ""); const result = await execFileAsync( process.execPath, - [new URL("../../../scripts/ci/plan.mjs", import.meta.url).pathname], + [fileURLToPath(new URL("../../../scripts/ci/plan.mjs", import.meta.url))], { cwd: directory, env: { @@ -92,6 +95,7 @@ async function classify( HEAD_SHA: head, GITHUB_EVENT_NAME: event, CI_FULL: String(full), + CI_SKIP_FULL: String(skipFull), GITHUB_OUTPUT: output, }, }, @@ -99,6 +103,21 @@ async function classify( return z.record(z.string(), z.unknown()).parse(JSON.parse(result.stdout)); } +it("reuses only scheduled qualification and preserves every explicit full request", async () => { + const { directory, base } = await fixture(); + const reused = await classify(directory, base, base, "schedule", false, true); + expect(reused.tier).toBe("reused"); + const { CI_FLAGS } = await loadScopes(); + for (const flag of CI_FLAGS) expect(reused[flag]).toBe(false); + for (const event of ["workflow_dispatch", "pull_request", "schedule"]) + expect( + (await classify(directory, base, base, event, true, true)).tier, + ).toBe("full"); + expect( + (await classify(directory, base, base, "push", false, true)).tier, + ).toBe("fast"); +}); + it("keeps every translated root README and authored guide on the documentation lane", async () => { const { directory, base } = await fixture(); const readmes = (await readdir(new URL("../../../", import.meta.url))).filter( @@ -118,6 +137,64 @@ it("keeps every translated root README and authored guide on the documentation l }); }); +it.each([ + "job-timings.mjs", + "job-timings-batch.mjs", + "test-report.mjs", + "test-timing-reporter.mjs", + "test-history.mjs", +])( + "keeps portable CI reporter %s on static checks and portable tests", + async (name) => { + const { directory, base } = await fixture(); + const head = await change(directory, [`scripts/ci/${name}`]); + const plan = await classify(directory, base, head); + const { CI_FLAGS } = await loadScopes(); + expect(plan.code).toBe(true); + for (const flag of CI_FLAGS) + if (flag !== "code") expect(plan[flag]).toBe(false); + }, +); + +it("preserves a provider lane when a portable CI reporter changes with provider source", async () => { + const { directory, base } = await fixture(); + const head = await change(directory, [ + "scripts/ci/test-report.mjs", + "src/evm/Example.ts", + ]); + expect(await classify(directory, base, head)).toMatchObject({ + code: true, + evm: true, + browser: false, + package: false, + }); +}); + +it("keeps unknown CI scripts on complete qualification", async () => { + const { directory, base } = await fixture(); + const head = await change(directory, ["scripts/ci/new-qualification.mjs"]); + const plan = await classify(directory, base, head); + const { CI_FLAGS } = await loadScopes(); + for (const flag of CI_FLAGS) expect(plan[flag]).toBe(true); +}); + +it("emits an observational owner proposal without changing ordinary or full portable execution", async () => { + const { directory, base } = await fixture(); + const head = await change(directory, ["src/evm/Example.ts"]); + expect(await classify(directory, base, head)).toMatchObject({ + code: true, + testSelection: { mode: "candidate", owners: ["evm"] }, + }); + expect( + await classify(directory, base, head, "pull_request", true), + ).toMatchObject({ + code: true, + testSelection: { mode: "full", owners: [] }, + }); + const output = await readFile(join(directory, "scope-output"), "utf8"); + expect(output).toContain('test_selection={"version":1,"mode":"full"'); +}); + it("does not charge a documentation PR for unrelated implementation updates on main", async () => { const { directory, base } = await fixture(); await git(directory, ["switch", "-c", "docs/change"]); @@ -161,13 +238,148 @@ it("retains complete checks when implementation is renamed into documentation", }); }); -it("retains the complete main-push baseline even for documentation", async () => { +it("keeps main pushes scoped when a successful ancestor baseline is available", async () => { const { directory, base } = await fixture(); const head = await change(directory, ["README_ja.md"]); expect(await classify(directory, base, head, "push")).toMatchObject({ + tier: "fast", + code: false, + package: false, + docs: true, + matrix: { + include: [ + { os: "ubuntu-latest", platform: "linux", architecture: "x64" }, + ], + }, + }); +}); + +it("includes changes from earlier cancelled pushes since the last successful baseline", async () => { + const { directory, base } = await fixture(); + await change(directory, ["src/evm/client.ts"]); + const head = await change(directory, ["README.md"]); + expect(await classify(directory, base, head, "push")).toMatchObject({ + tier: "fast", + code: true, + evm: true, + runtime: true, + package: false, + }); +}); + +it.each(["", "not-a-commit"])( + "fails closed to all lanes for a missing or invalid push baseline (%s)", + async (base) => { + const { directory } = await fixture(); + const head = await change(directory, ["README.md"]); + const plan = await classify(directory, base, head, "push"); + expect(plan).toMatchObject({ + tier: "fast", + code: true, + package: true, + evm: true, + browser: true, + reasons: expect.arrayContaining([expect.stringContaining("baseline")]), + }); + const { CI_FLAGS } = await loadScopes(); + for (const flag of CI_FLAGS) expect(plan[flag]).toBe(true); + }, +); + +it("selects only a trusted successful ancestor from recent CI push runs", async () => { + const { directory, base } = await fixture(); + const head = await change(directory, ["src/evm/client.ts"]); + const bin = join(directory, "fake-bin"); + await mkdir(bin); + const gh = join(bin, "gh"); + await writeFile(gh, '#!/bin/sh\ncat "$GH_FIXTURE_RESPONSE"\n'); + await chmod(gh, 0o755); + const response = join(directory, "runs.json"); + await writeFile( + response, + JSON.stringify({ + workflow_runs: [ + { + event: "push", + conclusion: "success", + head_branch: "main", + head_repository: { full_name: "untrusted/fork" }, + head_sha: base, + }, + { + event: "push", + conclusion: "success", + head_branch: "main", + head_repository: { full_name: "owner/repo" }, + head_sha: base, + }, + ], + }), + ); + const output = join(directory, "baseline-output"); + const result = await execFileAsync( + process.execPath, + [new URL("../../../scripts/ci/baseline.mjs", import.meta.url).pathname], + { + cwd: directory, + env: { + ...process.env, + PATH: `${bin}:${process.env.PATH}`, + GH_FIXTURE_RESPONSE: response, + CI_REPOSITORY: "owner/repo", + CI_DEFAULT_BRANCH: "main", + HEAD_SHA: head, + GITHUB_OUTPUT: output, + }, + }, + ); + expect(JSON.parse(result.stdout)).toEqual({ base, reason: "" }); + expect(await readFile(output, "utf8")).toBe(`base=${base}\nreason=\n`); +}); + +it("returns an empty baseline when GitHub lookup fails", async () => { + const { directory, base } = await fixture(); + const head = await change(directory, ["src/evm/client.ts"]); + const bin = join(directory, "fake-bin"); + await mkdir(bin); + const gh = join(bin, "gh"); + await writeFile(gh, "#!/bin/sh\nexit 1\n"); + await chmod(gh, 0o755); + const result = await execFileAsync( + process.execPath, + [new URL("../../../scripts/ci/baseline.mjs", import.meta.url).pathname], + { + cwd: directory, + env: { + ...process.env, + PATH: `${bin}:${process.env.PATH}`, + CI_REPOSITORY: "owner/repo", + CI_DEFAULT_BRANCH: "main", + HEAD_SHA: head, + }, + }, + ); + expect(JSON.parse(result.stdout)).toMatchObject({ + base: "", + reason: expect.stringContaining("Could not read"), + }); + expect(head).not.toBe(base); +}); + +it("fails closed when the successful push baseline is not an ancestor", async () => { + const { directory, base } = await fixture(); + await git(directory, ["switch", "-c", "unrelated"]); + const unrelated = await change(directory, ["other.txt"]); + await git(directory, ["switch", "main"]); + const head = await change(directory, ["README.md"]); + expect(await classify(directory, unrelated, head, "push")).toMatchObject({ + tier: "fast", code: true, package: true, + evm: true, + browser: true, }); + expect(head).not.toBe(base); }); it("fails classification rather than treating an invalid Git comparison as a scope", async () => { @@ -207,7 +419,8 @@ it("retains native Inspector verification on every documented package host", asy ); expect(packageInspector.if).toBeUndefined(); expect( - (await classify((await fixture()).directory, "", "", "push")).matrix, + (await classify((await fixture()).directory, "", "", "workflow_dispatch")) + .matrix, ).toEqual({ include: [ { os: "ubuntu-latest", platform: "linux", architecture: "x64" }, @@ -227,7 +440,14 @@ it("retains native Inspector verification on every documented package host", asy it.each([ [ "src/evm/client.ts", - { code: true, runtime: true, evm: true, browser: false, package: false }, + { + code: true, + runtime: true, + evm: true, + browser: false, + package: false, + sqlite: false, + }, ], [ "src/browser/execution/evaluate.ts", @@ -299,6 +519,10 @@ it.each([ "src/windows/Example.ts", { code: true, runtime: true, package: true, apple: false, evm: false }, ], + [ + "src/domain/javascript/javascriptRuntimeStaticCandidates.ts", + { code: true, runtime: true, package: true, javascript_recovery: true }, + ], ["install.sh", { code: true, runtime: true, package: true, evm: false }], [ ".agents/skills/reverse-engineer-anything/SKILL.md", @@ -328,6 +552,38 @@ it.each([ "src/process/ProcessOwnership.test.ts", { code: true, package: true, apple: false, evm: false, browser: false }, ], + [ + "src/sqlite/SqliteDatabaseProvider.ts", + { code: true, sqlite: true, runtime: true, package: false }, + ], + [ + "src/application/sqlite/SqliteDatabaseService.test.ts", + { code: true, sqlite: true, runtime: false }, + ], + [ + "tests/boundary/sqlite/sqliteDatabaseProvider.test.ts", + { code: true, sqlite: true, runtime: false }, + ], + [ + "tests/fixtures/sqlite/database.ts", + { code: true, sqlite: true, runtime: false }, + ], + ["tests/acceptance/sqliteDatabaseCli.test.ts", { code: true, sqlite: true }], + [ + "tests/boundary/mcp/serverLifecycleMcp.test.ts", + { code: true, sqlite: true }, + ], + ["src/process/OwnedCommand.ts", { code: true, sqlite: true, runtime: true }], + ["src/server/createServer.ts", { code: true, sqlite: true, runtime: true }], + [ + "src/contracts/toolContracts.ts", + { code: true, sqlite: true, runtime: true }, + ], + [ + "src/domain/analysisErrorProjection.ts", + { code: true, sqlite: true, runtime: true }, + ], + ["src/domain/javascript/example.ts", { code: true, sqlite: false }], ])("selects the appropriate lanes for %s", async (path, expected) => { const { directory, base } = await fixture(); const head = await change(directory, [path]); @@ -370,6 +626,52 @@ it("scopes package script edits while retaining broad dependency checks", async }); }); +it.each(["test:sqlite", "test:sqlite:run"])( + "routes %s script changes to the SQLite runtime matrix", + async (script) => { + const { directory } = await fixture(); + const base = await change( + directory, + ["package.json"], + JSON.stringify({ scripts: { [script]: "vitest run" } }), + ); + const head = await change( + directory, + ["package.json"], + JSON.stringify({ + scripts: { [script]: "vitest run --reporter=default" }, + }), + ); + expect(await classify(directory, base, head)).toMatchObject({ + code: true, + sqlite: true, + evm: false, + package: false, + }); + }, +); + +it("keeps unrelated test script changes off the SQLite runtime matrix", async () => { + const { directory } = await fixture(); + const base = await change( + directory, + ["package.json"], + JSON.stringify({ scripts: { "test:domain": "vitest run src/domain" } }), + ); + const head = await change( + directory, + ["package.json"], + JSON.stringify({ + scripts: { "test:domain": "vitest run src/domain --reporter=default" }, + }), + ); + expect(await classify(directory, base, head)).toMatchObject({ + code: true, + sqlite: false, + evm: false, + }); +}); + it.each([ ["browser", "real-browser"], ["binary_layout", "real-binary-layout"], @@ -386,7 +688,9 @@ it.each([ async (scope, name) => { const workflow = z .object({ - on: z.object({ workflow_call: z.null() }).passthrough(), + on: z + .object({ workflow_call: z.object({}).passthrough().nullable() }) + .passthrough(), }) .parse( parse( @@ -423,10 +727,19 @@ it.each([ ); it("rejects skipped selected work while allowing unselected lanes to skip", async () => { - const command = new URL("../../../scripts/ci/required.mjs", import.meta.url) - .pathname; + const command = fileURLToPath( + new URL("../../../scripts/ci/required.mjs", import.meta.url), + ); + const { CI_FLAGS } = await loadScopes(); const jobs = { - changes: { result: "success", outputs: { evm: "true" } }, + changes: { + result: "success", + outputs: { + ...Object.fromEntries(CI_FLAGS.map((flag) => [flag, "false"])), + tier: "fast", + evm: "true", + }, + }, static: { result: "success" }, test: { result: "success" }, evm: { result: "success" }, @@ -462,7 +775,10 @@ it.each([ "src/process/capture/example.test.ts", { code: true, runtime: false, browser: false, package: false }, ], - [".nvmrc", { code: true, package: true, browser: true, evm: true }], + [ + ".nvmrc", + { code: true, package: true, browser: true, evm: true, sqlite: true }, + ], [ "tsconfig.build.json", { code: true, package: true, browser: true, evm: true }, @@ -599,8 +915,10 @@ it.each(["workflow_dispatch", "schedule"])( async (event) => { const { directory, base } = await fixture(); expect(await classify(directory, base, base, event)).toMatchObject({ + tier: "full", code: true, package: true, + sqlite: true, evm: true, browser: true, }); @@ -614,7 +932,14 @@ it("honors ci:full for successive PR revisions", async () => { for (const head of [first, second]) { expect( await classify(directory, base, head, "pull_request", true), - ).toMatchObject({ code: true, package: true, evm: true, browser: true }); + ).toMatchObject({ + tier: "full", + code: true, + package: true, + sqlite: true, + evm: true, + browser: true, + }); } }); @@ -654,7 +979,7 @@ const INTEL_HOST = { architecture: "x64", }; -it("keeps macOS x64 package verification for ci:full and non-PR events", async () => { +it("keeps macOS x64 package verification only for full qualification", async () => { const { directory, base } = await fixture(); const head = await change(directory, ["src/windows/Example.ts"]); const matrix = z.object({ include: z.array(z.unknown()) }); @@ -665,7 +990,8 @@ it("keeps macOS x64 package verification for ci:full and non-PR events", async ( expect(pullRequest).toHaveLength(3); expect(pullRequest).not.toContainEqual(INTEL_HOST); expect(await hosts("pull_request", true)).toContainEqual(INTEL_HOST); - for (const event of ["push", "schedule", "workflow_dispatch"]) + expect(await hosts("push", false)).not.toContainEqual(INTEL_HOST); + for (const event of ["schedule", "workflow_dispatch"]) expect(await hosts(event, false)).toContainEqual(INTEL_HOST); }); @@ -678,6 +1004,7 @@ const workflowFile = async (name: string): Promise => ); const scopesSchema = z.object({ + CI_FLAGS: z.array(z.string()), SCOPES: z.record(z.string(), z.array(z.string())), MANUAL_VERIFIERS: z.array(z.string()), }); diff --git a/tests/boundary/filesystem/ciJobTimings.test.ts b/tests/boundary/filesystem/ciJobTimings.test.ts new file mode 100644 index 000000000..85a2ddd1f --- /dev/null +++ b/tests/boundary/filesystem/ciJobTimings.test.ts @@ -0,0 +1,243 @@ +import { chmod, mkdir, readFile, writeFile } from "node:fs/promises"; +import { delimiter, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +import { expect, it } from "vitest"; + +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { + executeWorkflowFixture, + workflowGit, +} from "../../support/workflowGit.js"; + +const reporter = fileURLToPath( + new URL("../../../scripts/ci/job-timings.mjs", import.meta.url), +); + +it("merges paginated jobs, reports elapsed metrics, and preserves raw responses", async () => { + const fixture = await createFixture([ + { + total_count: 4, + jobs: [ + { + name: "Build package", + workflow_name: "CI", + status: "completed", + conclusion: "success", + created_at: "2026-10-13T00:00:00Z", + started_at: "2026-10-13T00:00:10Z", + completed_at: "2026-10-13T00:01:10Z", + steps: [ + { + name: "Install dependencies (npm ci)", + started_at: "2026-10-13T00:00:10Z", + completed_at: "2026-10-13T00:00:40Z", + }, + { + name: "Upload artifact", + started_at: "2026-10-13T00:00:40Z", + completed_at: "2026-10-13T00:00:41Z", + }, + ], + }, + ], + }, + { + total_count: 4, + jobs: [ + { + name: "Queued lane", + status: "queued", + created_at: "2026-10-13T00:00:00Z", + started_at: "2026-10-13T00:00:00Z", + completed_at: null, + steps: [], + }, + { + name: "Running lane", + status: "in_progress", + created_at: "2026-10-13T00:00:00Z", + started_at: "2026-10-13T00:00:15Z", + completed_at: null, + steps: [ + { + name: "Download artifacts", + started_at: "2026-10-13T00:00:20Z", + completed_at: "2026-10-13T00:00:22Z", + }, + ], + }, + { + name: "Skipped lane", + status: "completed", + conclusion: "skipped", + created_at: "2026-10-13T00:00:00Z", + started_at: "2026-10-13T00:00:00Z", + completed_at: "2026-10-13T00:00:00Z", + steps: [], + }, + ], + }, + ]); + const result = await runReporter(fixture); + const report = JSON.parse( + await readFile(join(fixture.outputDirectory, "report.json"), "utf8"), + ); + + expect(report.run.snapshot).toContain("partial"); + expect(report.counts).toMatchObject({ + jobs: 4, + completed: 2, + skipped: 1, + queueUnknown: 2, + executionUnknown: 3, + completedExecutionUnknown: 0, + }); + expect(report.aggregates.queueSeconds).toMatchObject({ + count: 2, + p50: 10, + p95: 15, + }); + expect(report.aggregates.completedExecutionSeconds).toMatchObject({ + count: 1, + p50: 60, + p95: 60, + }); + expect(report.aggregates.summedCompletedJobElapsedSeconds).toMatchObject({ + value: 60, + knownCount: 1, + unknownCount: 0, + label: expect.stringContaining("not CPU time"), + }); + expect(report.jobs[1].queue).toEqual({ + seconds: null, + reason: "job start is not established by its status", + }); + expect(report.jobs[3].queue).toEqual({ + seconds: null, + reason: "job was skipped", + }); + expect(report.jobs[3].execution).toEqual({ + seconds: null, + reason: "job was skipped", + }); + expect(report.stepCosts.npmCi[0].duration.seconds).toBe(30); + expect(report.stepCosts.artifactUpload[0].duration.seconds).toBe(1); + expect(report.stepCosts.artifactDownload[0].duration.seconds).toBe(2); + expect(report.stepCostInterpretation).toContain( + "not total artifact transfer costs", + ); + expect(report.cpuSeconds).toBeUndefined(); + expect(result.stdout).toContain("4 job timing records"); + expect( + await readFile(join(fixture.outputDirectory, "jobs.json"), "utf8"), + ).toBe(fixture.raw); + const summary = await readFile(fixture.summary, "utf8"); + expect(summary).toContain("point-in-time snapshot"); + expect(summary).toContain("unknown"); + expect(summary).toContain("not CPU time"); + expect(summary).toContain("60 s"); + expect(summary).toContain("not total artifact transfer costs"); + const args = JSON.parse(await readFile(fixture.args, "utf8")); + expect(args).toEqual([ + "api", + "--paginate", + "--slurp", + "repos/example/rea/actions/runs/42/attempts/2/jobs?per_page=100", + ]); +}); + +it("marks invalid and out-of-order timestamps unknown instead of emitting negative durations", async () => { + const fixture = await createFixture([ + { + jobs: [ + { + name: "Invalid dates", + status: "completed", + created_at: "not-a-date", + started_at: "2026-10-13T00:00:10Z", + completed_at: "2026-10-13T00:00:05Z", + steps: [], + }, + ], + }, + ]); + await runReporter(fixture); + const report = JSON.parse( + await readFile(join(fixture.outputDirectory, "report.json"), "utf8"), + ); + + expect(report.jobs[0].queue).toEqual({ + seconds: null, + reason: "invalid timestamp", + }); + expect(report.jobs[0].execution).toEqual({ + seconds: null, + reason: "timestamps out of order", + }); + expect(report.counts.completedExecutionUnknown).toBe(1); + expect(report.aggregates.summedCompletedJobElapsedSeconds).toMatchObject({ + value: null, + unknownCount: 1, + }); +}); + +it("fails on malformed pages or a failed GitHub API request", async () => { + const malformed = await createFixture([{ jobs: "not-an-array" }]); + await expect(runReporter(malformed)).rejects.toThrow(); + await expect( + readFile(join(malformed.outputDirectory, "jobs.json"), "utf8"), + ).resolves.toBe(malformed.raw); + + const failedApi = await createFixture([], { exitCode: 1 }); + await expect(runReporter(failedApi)).rejects.toThrow(); +}); + +async function createFixture( + pages: unknown[], + { exitCode = 0 }: { exitCode?: number } = {}, +) { + const directory = await createTestTempDirectory("rea-ci-job-timings-"); + await workflowGit(directory, ["init", "--initial-branch=main"]); + const fakeBin = join(directory, "bin"); + await mkdir(fakeBin); + const raw = JSON.stringify(pages); + const rawPath = join(directory, "gh-response.json"); + const args = join(directory, "gh-args.json"); + await writeFile(rawPath, raw); + const fakeGh = join(fakeBin, "gh"); + await writeFile( + fakeGh, + `#!/usr/bin/env node\nconst fs = require("node:fs");\nfs.writeFileSync(process.env.CI_GH_ARGS, JSON.stringify(process.argv.slice(2)));\nprocess.stdout.write(fs.readFileSync(process.env.CI_GH_RESPONSE, "utf8"));\nprocess.exitCode = ${String(exitCode)};\n`, + ); + await chmod(fakeGh, 0o755); + const outputDirectory = join(directory, ".cache", "ci-job-timings"); + const summary = join(directory, "summary.md"); + await writeFile(summary, ""); + return { + directory, + fakeBin, + raw, + rawPath, + args, + outputDirectory, + summary, + }; +} + +function runReporter(fixture: Awaited>) { + return executeWorkflowFixture(process.execPath, [reporter], { + cwd: fixture.directory, + env: { + ...process.env, + PATH: `${fixture.fakeBin}${delimiter}${process.env.PATH ?? ""}`, + CI_REPOSITORY: "example/rea", + CI_JOB_TIMINGS_DIRECTORY: fixture.outputDirectory, + CI_GH_ARGS: fixture.args, + CI_GH_RESPONSE: fixture.rawPath, + CI_TIMING_RUN_ID: "42", + CI_TIMING_RUN_ATTEMPT: "2", + GITHUB_STEP_SUMMARY: fixture.summary, + }, + }); +} diff --git a/tests/boundary/filesystem/ciJobTimingsBatch.test.ts b/tests/boundary/filesystem/ciJobTimingsBatch.test.ts new file mode 100644 index 000000000..6c2658c00 --- /dev/null +++ b/tests/boundary/filesystem/ciJobTimingsBatch.test.ts @@ -0,0 +1,288 @@ +import { chmod, mkdir, readFile, writeFile } from "node:fs/promises"; +import { delimiter, join } from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; + +import { expect, it } from "vitest"; + +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { executeWorkflowFixture } from "../../support/workflowGit.js"; + +const batchScript = fileURLToPath( + new URL("../../../scripts/ci/job-timings-batch.mjs", import.meta.url), +); + +it("collects the latest attempt for recent completed CI and Release runs into a bounded manifest", async () => { + const recent = new Date().toISOString(); + const fixture = await createFixture([ + { + workflow_runs: [ + { + id: 42, + run_attempt: 3, + status: "completed", + conclusion: "failure", + created_at: new Date(Date.parse(recent) + 1_000).toISOString(), + }, + { + id: 41, + run_attempt: 1, + status: "completed", + conclusion: "cancelled", + created_at: recent, + }, + ], + }, + { + workflow_runs: [ + { + id: 43, + run_attempt: 2, + status: "completed", + conclusion: "success", + created_at: new Date(Date.parse(recent) + 2_000).toISOString(), + }, + ], + }, + [ + { + jobs: [ + { + name: "Release lane", + status: "completed", + conclusion: "success", + created_at: recent, + started_at: recent, + completed_at: new Date(Date.parse(recent) + 50_000).toISOString(), + steps: [], + }, + ], + }, + ], + [ + { + jobs: [ + { + name: "Failed lane", + status: "completed", + conclusion: "failure", + created_at: recent, + started_at: recent, + completed_at: new Date(Date.parse(recent) + 55_000).toISOString(), + steps: [], + }, + ], + }, + ], + [ + { + jobs: [ + { + name: "Cancelled lane", + status: "completed", + conclusion: "cancelled", + created_at: recent, + started_at: recent, + completed_at: new Date(Date.parse(recent) + 30_000).toISOString(), + steps: [], + }, + ], + }, + ], + ]); + const result = await runBatch(fixture); + const manifest = JSON.parse( + await readFile( + join(fixture.outputDirectory, "batch-manifest.json"), + "utf8", + ), + ); + const calls = JSON.parse(await readFile(fixture.calls, "utf8")); + + expect(result.stdout).toContain("3 collected runs"); + expect( + await readFile(join(fixture.directory, "summary.md"), "utf8"), + ).toContain("collected 3 of 3"); + expect(manifest.selection).toMatchObject({ + lookbackHours: 24, + maxRunsPerWorkflow: 100, + exhaustive: false, + }); + expect(manifest.coverage).toMatchObject({ + workflowsQueried: ["CI", "Release"], + selectedRuns: 3, + collectedRuns: 3, + }); + expect(manifest.truncation).toEqual([]); + expect(manifest.errors).toEqual([]); + expect(manifest.runs).toMatchObject([ + { id: 43, runAttempt: 2, conclusion: "success" }, + { id: 42, runAttempt: 3, conclusion: "failure" }, + { id: 41, runAttempt: 1, conclusion: "cancelled" }, + ]); + expect(manifest.runs).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + status: "collected", + rawPath: expect.any(String), + reportPath: expect.any(String), + }), + ]), + ); + expect(calls).toContainEqual([ + "api", + "--paginate", + "--slurp", + "repos/example/rea/actions/runs/42/attempts/3/jobs?per_page=100", + ]); + expect(calls).toContainEqual([ + "api", + "--paginate", + "--slurp", + "repos/example/rea/actions/runs/41/attempts/1/jobs?per_page=100", + ]); + const failedReport = JSON.parse( + await readFile( + join(fixture.outputDirectory, "runs/42-3/report.json"), + "utf8", + ), + ); + expect(failedReport.jobs[0].conclusion).toBe("failure"); + expect(failedReport.jobs[0].execution.seconds).toBe(55); +}); + +it("records listing and per-run errors without claiming complete coverage", async () => { + const recent = new Date().toISOString(); + const fixture = await createFixture([ + { + workflow_runs: [ + { + id: 7, + run_attempt: 2, + status: "completed", + conclusion: "success", + created_at: recent, + }, + ], + }, + { error: true }, + { jobs: "malformed" }, + ]); + await runBatch(fixture); + const manifest = JSON.parse( + await readFile( + join(fixture.outputDirectory, "batch-manifest.json"), + "utf8", + ), + ); + expect(manifest.selection.exhaustive).toBe(false); + expect(manifest.errors).toMatchObject([ + { scope: "workflow:Release" }, + { scope: "run" }, + ]); + expect(manifest.runs[0]).toMatchObject({ + id: 7, + runAttempt: 2, + status: "error", + }); +}); + +it("retains a partial manifest when its collection budget is exhausted", async () => { + const fixture = await createFixture([ + { + workflow_runs: [8, 9].map((id) => ({ + id, + run_attempt: 1, + status: "completed", + created_at: new Date().toISOString(), + })), + }, + { workflow_runs: [] }, + ]); + await executeWorkflowFixture( + process.execPath, + [ + "--input-type=module", + "-e", + ` + const { collectJobTimingsBatch } = await import(${JSON.stringify(pathToFileURL(batchScript).href)}); + let clock = Date.now(); + await collectJobTimingsBatch({ + repository: "example/rea", + outputDirectory: process.env.CI_JOB_TIMINGS_DIRECTORY, + now: () => clock, + collect: async () => { + clock += 240001; + throw new Error("fixture request exhausted the remaining budget"); + }, + }); + `, + ], + { cwd: fixture.directory, env: fixtureEnvironment(fixture) }, + ); + const manifest = JSON.parse( + await readFile( + join(fixture.outputDirectory, "batch-manifest.json"), + "utf8", + ), + ); + expect(manifest.coverage).toMatchObject({ + selectedRuns: 2, + collectedRuns: 0, + }); + expect(manifest.runs).toHaveLength(1); + expect(manifest.runs[0]).toMatchObject({ status: "error" }); + expect(manifest.truncation.join(" ")).toContain("Time budget"); +}); + +async function createFixture(responses: unknown[]) { + const directory = await createTestTempDirectory("rea-ci-job-timings-batch-"); + const fakeBin = join(directory, "bin"); + await mkdir(fakeBin); + const responsePath = join(directory, "responses.json"); + const calls = join(directory, "calls.json"); + await writeFile(responsePath, JSON.stringify(responses)); + await writeFile(calls, "[]"); + const fakeGh = join(fakeBin, "gh"); + await writeFile( + fakeGh, + `#!/usr/bin/env node +const fs = require("node:fs"); +const calls = JSON.parse(fs.readFileSync(process.env.CI_GH_CALLS, "utf8")); +calls.push(process.argv.slice(2)); +fs.writeFileSync(process.env.CI_GH_CALLS, JSON.stringify(calls)); +const responses = JSON.parse(fs.readFileSync(process.env.CI_GH_RESPONSES, "utf8")); +const response = responses[calls.length - 1]; +if (response && response.error) { process.stderr.write("fixture API error"); process.exitCode = 1; } +else process.stdout.write(JSON.stringify(response)); +`, + ); + await chmod(fakeGh, 0o755); + return { + directory, + fakeBin, + responses: responsePath, + calls, + outputDirectory: join(directory, ".cache", "ci-job-timings"), + }; +} + +function runBatch(fixture: Awaited>) { + return executeWorkflowFixture(process.execPath, [batchScript], { + cwd: fixture.directory, + env: fixtureEnvironment(fixture), + }); +} + +function fixtureEnvironment( + fixture: Awaited>, +) { + return { + ...process.env, + PATH: `${fixture.fakeBin}${delimiter}${process.env.PATH ?? ""}`, + CI_REPOSITORY: "example/rea", + CI_JOB_TIMINGS_DIRECTORY: fixture.outputDirectory, + CI_GH_CALLS: fixture.calls, + CI_GH_RESPONSES: fixture.responses, + GITHUB_STEP_SUMMARY: join(fixture.directory, "summary.md"), + }; +} diff --git a/tests/boundary/filesystem/ciPrEvents.test.ts b/tests/boundary/filesystem/ciPrEvents.test.ts new file mode 100644 index 000000000..3562b5df9 --- /dev/null +++ b/tests/boundary/filesystem/ciPrEvents.test.ts @@ -0,0 +1,98 @@ +import { readFile } from "node:fs/promises"; +import { expect, it } from "vitest"; +import { parse } from "yaml"; +import { z } from "zod"; +import { executeWorkflowFixture } from "../../support/workflowGit.js"; + +async function workflow(name: string) { + return z + .object({ + on: z.record(z.string(), z.unknown()), + permissions: z.record(z.string(), z.string()), + concurrency: z.object({ group: z.string() }).optional(), + jobs: z.record( + z.string(), + z.object({ + name: z.string().optional(), + if: z.string().optional(), + uses: z.string().optional(), + with: z.record(z.string(), z.string()).optional(), + steps: z + .array( + z.object({ + run: z.string().optional(), + uses: z.string().optional(), + env: z.record(z.string(), z.string()).optional(), + }), + ) + .optional(), + }), + ), + }) + .parse( + parse( + await readFile( + new URL(`../../../.github/workflows/${name}`, import.meta.url), + "utf8", + ), + ), + ); +} + +it("refreshes title validation on edits without checking out PR code or cancelling CI", async () => { + const title = await workflow("pr-title.yml"); + expect(title.on.pull_request).toMatchObject({ + types: ["opened", "synchronize", "reopened", "edited"], + }); + expect(title.permissions).toEqual({}); + expect(title.concurrency?.group).not.toBe( + (await workflow("ci.yml")).concurrency?.group, + ); + const jobs = Object.values(title.jobs); + expect(jobs).toHaveLength(1); + expect(jobs[0]?.name).toBe("Conventional PR title"); + expect(jobs[0]?.if).toBeUndefined(); // A skipped check must not green-light an invalid title on body edits. + const steps = jobs[0]?.steps ?? []; + expect(steps).toHaveLength(1); + expect(steps[0]?.uses).toBeUndefined(); + expect(steps[0]?.env).toEqual({ + PR_TITLE: "${{ github.event.pull_request.title }}", + }); + const command = z.string().parse(steps[0]?.run); + for (const title of [ + "fix(ci): repair workflow", + "feat!: changed contract", + "docs: literal $(exit 91) `exit 92` text", + ]) { + await expect( + executeWorkflowFixture("bash", ["-c", command], { + env: { ...process.env, PR_TITLE: title }, + }), + ).resolves.toMatchObject({ stderr: "" }); + } + for (const title of ["", "Fix CI", "feat: "]) { + await expect( + executeWorkflowFixture("bash", ["-c", command], { + env: { ...process.env, PR_TITLE: title }, + }), + ).rejects.toMatchObject({ code: 1 }); + } + expect((await workflow("ci.yml")).jobs["pr-title"]).toBeUndefined(); +}); + +it("qualifies only ci:full label additions at the caller's merge revision", async () => { + const full = await workflow("ci-full-pr.yml"); + expect(full.on.pull_request).toMatchObject({ types: ["labeled"] }); + expect(full.permissions).toEqual({ contents: "read", actions: "read" }); + expect(Object.values(full.jobs)).toEqual([ + { + name: "Qualify labeled PR revision", + if: "github.event.label.name == 'ci:full'", + uses: "./.github/workflows/ci.yml", + with: { source_sha: "${{ github.sha }}" }, + }, + ]); + // Metadata events must not create skipped replacements for the main CI checks. + expect(full.jobs.required).toBeUndefined(); + expect(full.concurrency).toBeUndefined(); +}); diff --git a/tests/boundary/filesystem/ciQualification.test.ts b/tests/boundary/filesystem/ciQualification.test.ts new file mode 100644 index 000000000..b6a5a1e1a --- /dev/null +++ b/tests/boundary/filesystem/ciQualification.test.ts @@ -0,0 +1,375 @@ +import { chmod, mkdir, readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; + +import { expect, it } from "vitest"; +import { parse } from "yaml"; +import { z } from "zod"; + +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { + executeWorkflowFixture, + workflowGit, +} from "../../support/workflowGit.js"; + +const qualificationScript = fileURLToPath( + new URL("../../../scripts/ci/qualification.mjs", import.meta.url), +); +const requiredScript = fileURLToPath( + new URL("../../../scripts/ci/required.mjs", import.meta.url), +); +const now = Date.now(); +const validSha = "a".repeat(40); +const repository = "example/rea"; + +function artifact(overrides: Record = {}) { + return { + id: 42, + name: "ci-full-qualification-v1", + expired: false, + created_at: new Date(now - 60_000).toISOString(), + workflow_run: { + id: 7, + repository_id: 1, + head_repository_id: 1, + head_branch: "main", + head_sha: validSha, + }, + ...overrides, + }; +} + +function run(overrides: Record = {}) { + return { + id: 7, + status: "completed", + conclusion: "success", + path: ".github/workflows/ci.yml", + event: "schedule", + created_at: new Date(now - 120_000).toISOString(), + head_sha: validSha, + head_branch: "main", + head_repository: { full_name: repository }, + ...overrides, + }; +} + +async function admissionFixture( + candidate: Record = artifact(), + candidateRun: Record = run(), + options: { apiError?: boolean; malformed?: boolean; force?: boolean } = {}, +) { + const directory = await createTestTempDirectory("rea-ci-qualification-"); + const bin = join(directory, "bin"); + await mkdir(bin); + const fixturePath = join(directory, "api.json"); + await writeFile( + fixturePath, + options.malformed + ? "not-json" + : JSON.stringify({ artifacts: [candidate], run: candidateRun }), + ); + const gh = join(bin, "gh"); + await writeFile( + gh, + [ + "#!/bin/sh", + 'if [ "$GH_API_ERROR" = "true" ]; then exit 1; fi', + 'case "$*" in', + ' *actions/artifacts*) cat "$GH_FIXTURE_RESPONSE" ;;', + " *actions/runs/7*) node -e 'const x=require(process.env.GH_FIXTURE_RESPONSE); process.stdout.write(JSON.stringify(x.run))' ;;", + " *) exit 2 ;;", + "esac", + "", + ].join("\n"), + ); + await chmod(gh, 0o755); + const output = join(directory, "github-output"); + await writeFile(output, ""); + const env = { + ...process.env, + PATH: `${bin}:${process.env.PATH ?? ""}`, + GH_FIXTURE_RESPONSE: fixturePath, + GH_API_ERROR: String(options.apiError ?? false), + CI_REPOSITORY: repository, + CI_REPOSITORY_ID: "1", + CI_DEFAULT_BRANCH: "main", + HEAD_SHA: validSha, + GITHUB_RUN_ID: "100", + GITHUB_OUTPUT: output, + ...(options.force ? { CI_FORCE_FULL: "true" } : {}), + }; + return { directory, output, env }; +} + +async function decide( + candidate?: Record, + candidateRun?: Record, + options?: { apiError?: boolean; malformed?: boolean; force?: boolean }, +) { + const fixture = await admissionFixture(candidate, candidateRun, options); + const result = await executeWorkflowFixture( + process.execPath, + [qualificationScript], + { + cwd: fixture.directory, + env: fixture.env, + }, + ); + return { + ...fixture, + decision: z + .record(z.string(), z.unknown()) + .parse(JSON.parse(result.stdout)), + outputText: await readFile(fixture.output, "utf8"), + }; +} + +it("reuses only a recent trusted full qualification for the exact default-branch SHA", async () => { + const { decision, outputText } = await decide(); + expect(decision).toMatchObject({ run_full: false, run_id: 7 }); + expect(Object.keys(decision).sort()).toEqual([ + "reason", + "run_full", + "run_id", + ]); + expect(outputText).toContain("run_full=false\n"); + expect(outputText).toContain("run_id=7\n"); +}); + +const rejectedCandidates: Array< + [string, () => Record, () => Record] +> = [ + [ + "old", + () => + artifact({ + created_at: new Date(now - 24 * 60 * 60 * 1000 - 1).toISOString(), + }), + run, + ], + [ + "future", + () => artifact({ created_at: new Date(now + 60_000).toISOString() }), + run, + ], + ["expired", () => artifact({ expired: true }), run], + [ + "foreign repository", + () => + artifact({ + workflow_run: { ...artifact().workflow_run, repository_id: 2 }, + }), + run, + ], + [ + "foreign branch", + () => + artifact({ + workflow_run: { ...artifact().workflow_run, head_branch: "other" }, + }), + run, + ], + [ + "foreign SHA", + () => + artifact({ + workflow_run: { ...artifact().workflow_run, head_sha: "b".repeat(40) }, + }), + run, + ], + [ + "untrusted workflow", + artifact, + () => run({ path: ".github/workflows/release.yml" }), + ], + ["failed run", artifact, () => run({ conclusion: "failure" })], + [ + "old producer rerun with a fresh artifact", + artifact, + () => + run({ created_at: new Date(now - 25 * 60 * 60 * 1000).toISOString() }), + ], + ["wrong workflow event", artifact, () => run({ event: "push" })], +]; + +it.each(rejectedCandidates)( + "runs full qualification for a %s candidate", + async (_name, makeArtifact, makeRun) => { + const { decision } = await decide(makeArtifact(), makeRun()); + expect(decision.run_full).toBe(true); + }, +); + +it.each([ + ["API error", { apiError: true }], + ["malformed query response", { malformed: true }], +] as const)("fails closed on %s", async (_name, options) => { + const { decision } = await decide(undefined, undefined, options); + expect(decision.run_full).toBe(true); +}); + +it("lets the daily force-full setting bypass the GitHub API", async () => { + const { decision } = await decide(undefined, undefined, { + force: true, + apiError: true, + }); + expect(decision).toMatchObject({ run_full: true }); + expect(decision.reason).toContain("Forced"); +}); + +async function workflowContract() { + const workflowPath = new URL( + "../../../.github/workflows/ci.yml", + import.meta.url, + ); + const parsed = parse(await readFile(workflowPath, "utf8")); + return z + .object({ + jobs: z.record( + z.string(), + z.object({ needs: z.unknown().optional() }).passthrough(), + ), + }) + .parse(parsed).jobs; +} + +async function requiredRun( + changes: Record, + overrides: { failed?: string; runAttempt?: string; event?: string } = {}, +) { + const directory = await createTestTempDirectory("rea-ci-receipt-"); + await workflowGit(directory, ["init", "--initial-branch=main"]); + await writeFile(join(directory, "fixture.txt"), "qualified source\n"); + await workflowGit(directory, ["add", "."]); + await workflowGit(directory, ["commit", "-m", "fixture source"]); + const head = await workflowGit(directory, ["rev-parse", "HEAD"]); + const jobs = await workflowContract(); + const results: Record< + string, + { result: string; outputs?: Record } + > = {}; + for (const name of z.array(z.string()).parse(jobs.required?.needs)) { + results[name] = { + result: + name === overrides.failed + ? "failure" + : changes.tier === "reused" + ? "skipped" + : "success", + }; + } + results.changes = { result: "success", outputs: changes }; + const receiptDirectory = join(directory, ".cache", "receipt"); + const output = join(directory, "github-output"); + await writeFile(output, ""); + const env = { + ...process.env, + CI_JOBS: JSON.stringify(results), + CI_QUALIFICATION_DIRECTORY: receiptDirectory, + GITHUB_EVENT_NAME: overrides.event ?? "schedule", + GITHUB_SHA: head, + GITHUB_RUN_ID: "100", + GITHUB_RUN_ATTEMPT: overrides.runAttempt ?? "1", + GITHUB_REPOSITORY: repository, + GITHUB_OUTPUT: output, + }; + const result = executeWorkflowFixture(process.execPath, [requiredScript], { + cwd: directory, + env, + }); + return { directory, head, output, result }; +} + +it("writes a full receipt only after every canonical flag and workflow need pass", async () => { + const scopesPath = new URL("../../../scripts/ci/scopes.mjs", import.meta.url); + const { CI_FLAGS } = await import(scopesPath.href); + const changes = Object.fromEntries( + CI_FLAGS.map((flag: string) => [flag, "true"]), + ); + changes.tier = "full"; + const { directory, head, output, result } = await requiredRun(changes); + await expect(result).resolves.toMatchObject({ stderr: "" }); + const receiptPath = join(directory, ".cache", "receipt", "receipt.json"); + const receipt = z + .object({ + sourceSha: z.string(), + runId: z.number(), + runAttempt: z.number(), + }) + .parse(JSON.parse(await readFile(receiptPath, "utf8"))); + expect(receipt).toMatchObject({ sourceSha: head, runId: 100, runAttempt: 1 }); + expect(await readFile(output, "utf8")).toContain( + `receipt_path=${receiptPath}\n`, + ); + expect(await readFile(output, "utf8")).toContain( + "receipt_name=ci-full-qualification-v1\n", + ); +}); + +it.each([ + ["failed selected lane", { failed: "sqlite-runtime" }], + ["partial full scope", {}], +] as const)( + "does not issue a receipt after %s", + async (caseName, overrides) => { + const scopesPath = new URL( + "../../../scripts/ci/scopes.mjs", + import.meta.url, + ); + const { CI_FLAGS } = await import(scopesPath.href); + const changes = Object.fromEntries( + CI_FLAGS.map((flag: string) => [flag, "true"]), + ); + changes.tier = "full"; + if (caseName === "partial full scope") changes.sqlite = "false"; + const outcome = await requiredRun(changes, overrides); + await expect(outcome.result).rejects.toMatchObject({ code: 1 }); + await expect( + readFile(join(outcome.directory, ".cache", "receipt", "receipt.json")), + ).rejects.toMatchObject({ code: "ENOENT" }); + expect(await readFile(outcome.output, "utf8")).not.toContain("receipt_"); + }, +); + +it("never writes a fresh receipt for a reused scheduled qualification", async () => { + const scopesPath = new URL("../../../scripts/ci/scopes.mjs", import.meta.url); + const { CI_FLAGS } = await import(scopesPath.href); + const changes = Object.fromEntries( + CI_FLAGS.map((flag: string) => [flag, "false"]), + ); + changes.tier = "reused"; + changes.qualification_run_id = "7"; + const outcome = await requiredRun(changes); + const result = await outcome.result; + await expect( + readFile(join(outcome.directory, ".cache", "receipt", "receipt.json")), + ).rejects.toMatchObject({ code: "ENOENT" }); + expect(result.stdout).toContain("Qualification reused from run 7"); + expect(await readFile(outcome.output, "utf8")).not.toContain("receipt_"); +}); + +it.each([ + ["missing receipt run", { qualification_run_id: "" }, "schedule"], + ["selected validation lane", { sqlite: "true" }, "schedule"], + ["manual dispatch", {}, "workflow_dispatch"], +] as const)( + "rejects qualification reuse with %s", + async (_name, override, event) => { + const { CI_FLAGS } = await import( + new URL("../../../scripts/ci/scopes.mjs", import.meta.url).href + ); + const changes: Record = { + ...Object.fromEntries(CI_FLAGS.map((flag: string) => [flag, "false"])), + tier: "reused", + qualification_run_id: "7", + ...override, + }; + const outcome = await requiredRun(changes, { event }); + await expect(outcome.result).rejects.toMatchObject({ + code: 1, + stderr: expect.stringContaining("scheduled admission receipt"), + }); + expect(await readFile(outcome.output, "utf8")).not.toContain("receipt_"); + }, +); diff --git a/tests/boundary/filesystem/ciRequiredLanes.test.ts b/tests/boundary/filesystem/ciRequiredLanes.test.ts index 4cff44e96..da6c43186 100644 --- a/tests/boundary/filesystem/ciRequiredLanes.test.ts +++ b/tests/boundary/filesystem/ciRequiredLanes.test.ts @@ -26,7 +26,19 @@ const workflow = async (filename = "ci.yml") => ), ); +const scopeFlags = async () => + z + .object({ CI_FLAGS: z.array(z.string()) }) + .parse( + await import( + new URL("../../../scripts/ci/scopes.mjs", import.meta.url).href + ), + ).CI_FLAGS; + it("binds SQLite compatibility jobs to the build's prepared runtime artifact and root layout", async () => { + expect((await workflow()).jobs["sqlite-runtime"]?.if).toBe( + "needs.changes.outputs.sqlite == 'true'", + ); const steps = z.array( z.object({ uses: z.string().optional(), @@ -52,6 +64,77 @@ it("binds SQLite compatibility jobs to the build's prepared runtime artifact and expect(downloads[0]?.with).toEqual({ name, path: "." }); }); +it("exports every canonical scope with read authority for admission and keeps telemetry outside the gate", async () => { + const { jobs } = await workflow(); + const CI_FLAGS = await scopeFlags(); + const outputs = z.record(z.string(), z.string()).parse(jobs.changes?.outputs); + expect(jobs.changes?.permissions).toEqual({ + contents: "read", + actions: "read", + }); + expect(jobs.build?.permissions).toEqual({ + contents: "read", + actions: "read", + }); + for (const flag of CI_FLAGS) + expect(outputs[flag]).toBe(`\${{ steps.plan.outputs.${flag} }}`); + const steps = z + .array( + z.object({ + run: z.string().optional(), + if: z.string().optional(), + }), + ) + .parse(jobs.required?.steps); + const telemetry = steps.find((step) => + step.run?.includes("scripts/ci/job-timings.mjs"), + ); + expect(telemetry).toBeUndefined(); + const timings = z + .object({ + on: z.object({ + schedule: z.array(z.object({ cron: z.string() })), + workflow_dispatch: z.unknown(), + workflow_run: z.unknown().optional(), + }), + permissions: z.record(z.string(), z.string()), + jobs: z.object({ + report: z.object({ + steps: z.array( + z.object({ + uses: z.string().optional(), + run: z.string().optional(), + with: z.record(z.string(), z.unknown()).optional(), + env: z.record(z.string(), z.unknown()).optional(), + }), + ), + }), + }), + }) + .parse( + parse( + await readFile( + new URL("../../../.github/workflows/ci-timings.yml", import.meta.url), + "utf8", + ), + ), + ); + expect(timings.on.workflow_run).toBeUndefined(); + expect(timings.on.schedule).toHaveLength(1); + expect(timings.permissions).toEqual({ contents: "read", actions: "read" }); + const checkout = timings.jobs.report.steps.find((step) => + step.uses?.startsWith("actions/checkout@"), + ); + expect(checkout?.with).toEqual({ + ref: "${{ github.event.repository.default_branch }}", + "persist-credentials": false, + }); + const report = timings.jobs.report.steps.find((step) => + step.run?.includes("scripts/ci/job-timings-batch.mjs"), + ); + expect(report?.env?.CI_REPOSITORY).toBe("${{ github.repository }}"); +}); + const requiredNeeds = async () => { const { jobs } = await workflow(); return { jobs, needs: z.array(z.string()).parse(jobs.required?.needs) }; @@ -64,6 +147,240 @@ it("makes the required gate wait for every CI lane, including future additions", ); }); +it("holds native and provider fan-out behind static validation while portable feedback stays independent", async () => { + const { jobs } = await workflow(); + const earlyJobs = new Set([ + "changes", + "static", + "build", + "test-shard", + "test", + "docs", + "required", + ]); + for (const [name, job] of Object.entries(jobs)) { + if (earlyJobs.has(name)) continue; + expect(job.needs, `${name} must wait for static validation`).toContain( + "static", + ); + expect(job.if).not.toMatch(/always\(|cancelled\(/u); + } + expect(jobs["test-shard"]?.needs).not.toContain("static"); +}); + +it("binds provider build reuse to the current run while keeping standalone dispatches independent", async () => { + const { jobs } = await workflow(); + const consumers = [ + "browser", + "binary_layout", + "evm", + "javascript_recovery", + "recorded_crash", + "web_network", + "web_runtime", + "web_source_map", + ]; + const buildTerms = z + .string() + .parse(jobs.build?.if) + .split("||") + .map((term) => term.trim()); + // Provider workflow edits select their lane without code=true. The build + // producer must still run, without selecting the portable test shards. + expect(new Set(buildTerms)).toEqual( + new Set( + ["code", ...consumers].map( + (scope) => `needs.changes.outputs.${scope} == 'true'`, + ), + ), + ); + expect(jobs["test-shard"]?.if).toBe("needs.changes.outputs.code == 'true'"); + for (const name of consumers) { + const caller = jobs[name]; + expect(caller?.needs).toContain("build"); + expect(caller?.with).toEqual({ build_cache_artifact: "rea-build-cache" }); + const path = z.string().parse(caller?.uses); + const called = z + .object({ + on: z.object({ + workflow_call: z.object({ + inputs: z.record(z.string(), z.unknown()), + }), + workflow_dispatch: z.unknown(), + }), + jobs: z.record( + z.string(), + z.object({ + steps: z.array( + z.object({ + uses: z.string().optional(), + with: z.record(z.string(), z.unknown()).optional(), + }), + ), + }), + ), + }) + .parse( + parse( + await readFile(new URL(`../../../${path}`, import.meta.url), "utf8"), + ), + ); + expect(called.on.workflow_call.inputs.build_cache_artifact).toMatchObject({ + type: "string", + default: "", + }); + expect(Object.hasOwn(called.on, "workflow_dispatch")).toBe(true); + for (const job of Object.values(called.jobs)) + expect( + job.steps.find( + (step) => step.uses === "./.github/actions/portable-build-cache", + )?.with, + ).toEqual({ build_cache_artifact: "${{ inputs.build_cache_artifact }}" }); + } + const action = z + .object({ + runs: z.object({ + steps: z.array( + z.object({ + id: z.string().optional(), + uses: z.string().optional(), + if: z.string().optional(), + with: z.record(z.string(), z.unknown()).optional(), + "continue-on-error": z.boolean().optional(), + }), + ), + }), + }) + .parse( + parse( + await readFile( + new URL( + "../../../.github/actions/portable-build-cache/action.yml", + import.meta.url, + ), + "utf8", + ), + ), + ); + const download = action.runs.steps.find((step) => + step.uses?.startsWith("actions/download-artifact@"), + ); + expect(download?.with).toEqual({ + name: "${{ inputs.build_cache_artifact }}", + path: ".turbo/cache/", + }); + expect(download?.if).toBe("inputs.build_cache_artifact != ''"); + expect(download?.["continue-on-error"]).not.toBe(true); + for (const step of action.runs.steps.filter( + (step) => step.id === "plan" || step.uses?.startsWith("actions/cache"), + )) + expect(step.if).toContain("inputs.build_cache_artifact == ''"); +}); + +it("runs workflow validation inside required static checks when selected", async () => { + const { jobs } = await workflow(); + const steps = z + .array(z.object({ run: z.string().optional(), if: z.string().optional() })) + .parse(jobs.static?.steps); + const validation = steps.filter((step) => step.run?.includes("actionlint")); + expect(validation).toHaveLength(1); + expect(validation[0]?.if).toBe("needs.changes.outputs.workflows == 'true'"); + expect(jobs.workflows).toBeUndefined(); +}); + +it("leaves cancellation ownership with CI so child lanes cannot cancel another tier", async () => { + const { jobs } = await workflow(); + const children = Object.values(jobs).flatMap((job) => + typeof job.uses === "string" && job.uses.startsWith("./.github/workflows/") + ? [job.uses] + : [], + ); + expect(children.length).toBeGreaterThan(0); + for (const path of children) { + const child = z + .object({ concurrency: z.unknown().optional() }) + .parse( + parse( + await readFile(new URL(`../../../${path}`, import.meta.url), "utf8"), + ), + ); + expect( + child.concurrency, + `${path} must use caller scheduling`, + ).toBeUndefined(); + } +}); + +it("keeps fast feedback separate from complete qualification and release proof", async () => { + const ci = z + .object({ + on: z.object({ + push: z.unknown(), + pull_request: z.unknown(), + schedule: z.array(z.object({ cron: z.string() })), + workflow_call: z.object({ + inputs: z.object({ + source_sha: z.object({ + required: z.literal(true), + type: z.literal("string"), + }), + }), + }), + }), + concurrency: z.object({ + group: z.string(), + "cancel-in-progress": z.string(), + }), + jobs: z.record( + z.string(), + z.object({ + if: z.string().optional(), + steps: z + .array( + z.object({ + run: z.string().optional(), + if: z.string().optional(), + with: z.record(z.string(), z.unknown()).optional(), + }), + ) + .optional(), + }), + ), + }) + .parse( + parse( + await readFile( + new URL("../../../.github/workflows/ci.yml", import.meta.url), + "utf8", + ), + ), + ); + expect(ci.on.schedule).toEqual([ + { cron: "23 0-2,4-23 * * *" }, + { cron: "23 3 * * *" }, + ]); + expect(ci.concurrency.group).toContain("'qualification' || 'feedback'"); + expect(ci.concurrency.group).toContain("inputs.source_sha ||"); + expect(ci.concurrency.group).toContain("github.ref"); + expect(ci.concurrency["cancel-in-progress"]).toBe( + "${{ github.event_name == 'pull_request' && inputs.source_sha == '' && !contains(github.event.pull_request.labels.*.name, 'ci:full') }}", + ); + expect(ci.jobs.required?.if).toBe("${{ always() }}"); + const shards = ci.jobs["test-shard"]?.steps ?? []; + const fast = shards.find((step) => step.run?.includes("npm run test:run")); + const full = shards.find((step) => step.run?.includes("test:ci:shard:run")); + expect(fast?.if).toBe("needs.changes.outputs.tier == 'fast'"); + expect(fast?.run).toContain("--reporter=blob"); + expect(fast?.run).not.toContain("--coverage"); + expect(full?.if).toBe("needs.changes.outputs.tier == 'full'"); + const proof = ci.jobs["package-e2e"]?.steps?.find( + (step) => step.with?.name === "verified-release-package", + ); + expect(proof?.if).toContain("needs.changes.outputs.tier == 'full'"); + expect(proof?.if).toContain("matrix.platform == 'linux'"); + expect(proof?.if).toContain("matrix.architecture == 'x64'"); +}); + it.each([ ["true", "success", true], ["true", "failure", false], @@ -71,34 +388,119 @@ it.each([ ["true", "skipped", false], ["true", "missing", false], ["false", "skipped", true], + ["false", "failure", false], ] as const)( - "enforces the SQLite required gate for code=%s and result=%s", - async (code, result, passes) => { + "enforces mapped required scopes for selected=%s and result=%s", + async (selected, result, passes) => { const workflow = await requiredNeeds(); - // GitHub exposes only direct dependencies in the required job's needs context. - const needs: Record< - string, - { result: string; outputs?: Record } - > = Object.fromEntries( - workflow.needs.map((name) => [name, { result: "success" }]), - ); - needs.changes = { result: "success", outputs: { code } }; - if (result === "missing") delete needs["sqlite-runtime"]; - else if ("sqlite-runtime" in needs) needs["sqlite-runtime"] = { result }; - const execution = executeWorkflowFixture( + for (const [scope, lane] of [ + ["sqlite", "sqlite-runtime"], + ["workflows", "static"], + ] as const) { + // GitHub exposes only direct dependencies in the required job's needs context. + const needs: Record< + string, + { result: string; outputs?: Record } + > = Object.fromEntries( + workflow.needs.map((name) => [name, { result: "success" }]), + ); + needs.changes = { + result: "success", + outputs: { + ...Object.fromEntries( + (await scopeFlags()).map((flag) => [flag, "false"]), + ), + tier: "fast", + code: "true", + [scope]: selected, + }, + }; + if (result === "missing") delete needs[lane]; + else needs[lane] = { result }; + const execution = executeWorkflowFixture( + process.execPath, + [ + fileURLToPath( + new URL("../../../scripts/ci/required.mjs", import.meta.url), + ), + ], + { env: { ...process.env, CI_JOBS: JSON.stringify(needs) } }, + ); + // Static checks remain required even without workflow edits. + const expected = passes && !(lane === "static" && result !== "success"); + if (expected) + await expect(execution).resolves.toMatchObject({ stderr: "" }); + else + await expect(execution).rejects.toMatchObject({ + code: 1, + stderr: expect.stringContaining(lane), + }); + } + }, +); + +it("rejects incomplete or malformed classification even when available jobs succeeded", async () => { + const flags = await scopeFlags(); + const valid = { + ...Object.fromEntries(flags.map((flag) => [flag, "false"])), + tier: "fast", + }; + const run = (outputs: unknown) => + executeWorkflowFixture( process.execPath, [ fileURLToPath( new URL("../../../scripts/ci/required.mjs", import.meta.url), ), ], - { env: { ...process.env, CI_JOBS: JSON.stringify(needs) } }, + { + env: { + ...process.env, + CI_JOBS: JSON.stringify({ + changes: { result: "success", outputs }, + static: { result: "success" }, + test: { result: "success" }, + }), + CI_QUALIFICATION_DIRECTORY: "", + }, + }, ); - if (passes) await expect(execution).resolves.toMatchObject({ stderr: "" }); - else - await expect(execution).rejects.toMatchObject({ - code: 1, - stderr: expect.stringContaining("sqlite-runtime"), - }); - }, -); + await expect(run(valid)).resolves.toMatchObject({ stderr: "" }); + // Metadata is not a scope, even when its string happens to be "true". + await expect( + run({ ...valid, qualification_run_id: "true" }), + ).resolves.toMatchObject({ stderr: "" }); + for (const outputs of [ + undefined, + null, + {}, + { ...valid, tier: undefined }, + { ...valid, tier: "" }, + { ...valid, tier: "unknown" }, + ]) { + await expect(run(outputs)).rejects.toMatchObject({ + code: 1, + stderr: expect.stringContaining("valid tier"), + }); + } + for (const flag of flags) { + await expect(run({ ...valid, [flag]: undefined })).rejects.toMatchObject({ + code: 1, + stderr: expect.stringContaining(`scope outputs: ${flag}`), + }); + } + for (const value of ["", "TRUE", "yes", true, false, 0, null, [], {}]) { + await expect(run({ ...valid, sqlite: value })).rejects.toMatchObject({ + code: 1, + stderr: expect.stringContaining("scope outputs: sqlite"), + }); + } + for (const tier of ["full", "reused"]) { + await expect( + run({ ...valid, tier, sqlite: undefined }), + ).rejects.toMatchObject({ + code: 1, + stderr: expect.stringContaining("scope outputs: sqlite"), + }); + } +}); diff --git a/tests/boundary/filesystem/ciTestHistory.test.ts b/tests/boundary/filesystem/ciTestHistory.test.ts new file mode 100644 index 000000000..411e9ece9 --- /dev/null +++ b/tests/boundary/filesystem/ciTestHistory.test.ts @@ -0,0 +1,188 @@ +import { chmod, mkdir, readFile, writeFile } from "node:fs/promises"; +import { delimiter, dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +import { expect, it } from "vitest"; + +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { executeWorkflowFixture } from "../../support/workflowGit.js"; + +const script = fileURLToPath( + new URL("../../../scripts/ci/test-history.mjs", import.meta.url), +); +const repository = "example/rea"; +const repositoryId = 1; +const now = Date.now(); + +function artifact(id: number, overrides: Record = {}) { + return { + id: id + 100, + name: "test-timings", + expired: false, + created_at: new Date(now - 60_000).toISOString(), + workflow_run: { + id, + repository_id: repositoryId, + head_repository_id: repositoryId, + head_branch: "main", + }, + ...overrides, + }; +} + +function run(id: number, overrides: Record = {}) { + return { + id, + status: "completed", + conclusion: "success", + path: ".github/workflows/ci.yml", + event: "schedule", + created_at: new Date(now - 120_000).toISOString(), + head_branch: "main", + head_repository: { full_name: repository }, + ...overrides, + }; +} + +async function invoke( + artifacts: unknown[], + runs: Record, + options: { + apiError?: boolean; + repository?: string; + repositoryId?: string; + } = {}, +) { + const directory = await createTestTempDirectory("rea-ci-test-history-"); + const fakeBin = join(directory, "bin"); + await mkdir(fakeBin); + const fixture = join(directory, "api.json"); + const calls = join(directory, "api-calls"); + const output = join(directory, "github-output"); + await writeFile(fixture, JSON.stringify({ artifacts, runs })); + await writeFile(output, ""); + await writeFile(calls, ""); + const fakeGh = join(fakeBin, "gh"); + await writeFile( + fakeGh, + [ + "#!/bin/sh", + 'printf \'%s\\n\' "$*" >> "$GH_API_CALLS"', + 'if [ "$GH_API_ERROR" = "true" ]; then exit 1; fi', + 'if [ "$4" = "repos/example/rea/actions/artifacts" ]; then', + ' node -e \'const fs=require("node:fs"); const x=JSON.parse(fs.readFileSync(process.env.GH_FIXTURE_RESPONSE,"utf8")); process.stdout.write(JSON.stringify({artifacts:x.artifacts}));\'', + "else", + ' run_id="${4##*/}"', + ' node -e \'const fs=require("node:fs"); const x=JSON.parse(fs.readFileSync(process.env.GH_FIXTURE_RESPONSE,"utf8")); process.stdout.write(JSON.stringify(x.runs[Number(process.argv[1])] ?? {}));\' "$run_id"', + "fi", + "", + ].join("\n"), + ); + await chmod(fakeGh, 0o755); + const node24 = dirname(process.execPath); + const result = await executeWorkflowFixture(process.execPath, [script], { + cwd: directory, + env: { + ...process.env, + PATH: [fakeBin, node24, process.env.PATH ?? ""].join(delimiter), + GH_FIXTURE_RESPONSE: fixture, + GH_API_CALLS: calls, + GH_API_ERROR: String(options.apiError ?? false), + GH_TOKEN: "fixture-token", + CI_REPOSITORY: options.repository ?? repository, + CI_REPOSITORY_ID: options.repositoryId ?? String(repositoryId), + CI_DEFAULT_BRANCH: "main", + GITHUB_OUTPUT: output, + }, + }); + return { + result, + output: await readFile(output, "utf8"), + calls: await readFile(calls, "utf8"), + }; +} + +it("selects the latest trusted full run that actually published timing history", async () => { + const { result, output, calls } = await invoke( + [ + artifact(15), + artifact(14), + artifact(13, { + workflow_run: { ...artifact(13).workflow_run, head_repository_id: 2 }, + }), + artifact(12, { + workflow_run: { ...artifact(12).workflow_run, head_branch: "feature" }, + }), + artifact(11), + ], + { + 15: run(15, { conclusion: "failure" }), + 14: run(14, { path: ".github/workflows/release.yml" }), + 11: run(11, { event: "workflow_dispatch" }), + }, + ); + + expect(JSON.parse(result.stdout)).toEqual({ run_id: "11" }); + expect(output).toBe("run-id=11\n"); + expect(calls).toContain("name=test-timings"); + expect(calls).toContain("per_page=100"); + expect(calls.match(/actions\/runs\//gu)).toHaveLength(3); + expect(calls).not.toContain("actions/workflows"); +}); + +it("rejects expired, foreign, old, and non-full run candidates", async () => { + const { result, output } = await invoke( + [ + artifact(15, { expired: true }), + artifact(14, { + workflow_run: { ...artifact(14).workflow_run, head_repository_id: 2 }, + }), + artifact(13, { + created_at: new Date(now - 31 * 24 * 60 * 60 * 1000).toISOString(), + }), + artifact(12), + ], + { + 12: run(12, { event: "pull_request" }), + }, + ); + + expect(JSON.parse(result.stdout)).toEqual({ run_id: "" }); + expect(output).toBe("run-id=\n"); +}); + +it("writes an empty run id when GitHub history is unavailable", async () => { + const { result, output } = await invoke([], {}, { apiError: true }); + expect(JSON.parse(result.stdout)).toEqual({ run_id: "" }); + expect(output).toBe("run-id=\n"); +}); + +it("bounds run lookups to five artifacts even when inventory has more", async () => { + const ids = [25, 24, 23, 22, 21, 20, 19]; + const { result, calls } = await invoke( + ids.map((id) => artifact(id)), + Object.fromEntries( + ids.map((id, index) => [ + id, + run(id, index < 5 ? { conclusion: "failure" } : {}), + ]), + ), + ); + + expect(JSON.parse(result.stdout)).toEqual({ run_id: "" }); + expect(calls.match(/actions\/runs\//gu)).toHaveLength(5); +}); + +it("returns empty history without querying for an invalid repository identity", async () => { + const { result, output, calls } = await invoke( + [], + {}, + { + repository: "owner/repo?per_page=100", + repositoryId: "not-an-id", + }, + ); + expect(JSON.parse(result.stdout)).toEqual({ run_id: "" }); + expect(output).toBe("run-id=\n"); + expect(calls).toBe(""); +}); diff --git a/tests/boundary/filesystem/ciTestReports.test.ts b/tests/boundary/filesystem/ciTestReports.test.ts index 0c9bb4ccf..06212ad07 100644 --- a/tests/boundary/filesystem/ciTestReports.test.ts +++ b/tests/boundary/filesystem/ciTestReports.test.ts @@ -1,5 +1,5 @@ import { execFile } from "node:child_process"; -import { readFile, writeFile } from "node:fs/promises"; +import { mkdir, readFile, rm, writeFile } from "node:fs/promises"; import { join, resolve } from "node:path"; import { fileURLToPath } from "node:url"; import { promisify } from "node:util"; @@ -8,8 +8,11 @@ import { expect, it } from "vitest"; import { parse } from "yaml"; import { z } from "zod"; +import { planTestSelection } from "../../../scripts/ci/test-selection.mjs"; + import { parseTestDurations } from "../../ciTestSequencer.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { workflowGit } from "../../support/workflowGit.js"; const execute = promisify(execFile); const script = new URL("../../../scripts/ci/test-report.mjs", import.meta.url); @@ -53,6 +56,23 @@ it("collects failed-shard diagnostics before enforcing the shard result", async expect(upload).toBeGreaterThan(merge); expect(gate).toBeGreaterThan(upload); expect(steps[merge]?.if).not.toContain("needs.test-shard.result"); + const merges = steps.filter((step) => step.run?.includes("--reporter=json")); + expect(merges).toHaveLength(2); + for (const step of merges) + expect(step.run).toContain( + "--reporter=./scripts/ci/test-timing-reporter.mjs", + ); + expect(steps[upload]?.with?.path).toContain("test-timings.json"); + expect(steps[upload]?.with).toMatchObject({ "include-hidden-files": true }); + expect(steps[upload]?.with?.path).toContain( + ".cache/ci-test-selection/report.json", + ); + const summary = steps.find((step) => + step.run?.includes("scripts/ci/test-report.mjs"), + ); + expect(summary?.env?.CI_TEST_SELECTION).toBe( + "${{ needs.changes.outputs.test_selection }}", + ); const command = steps[gate]?.run; if (command === undefined) throw new Error("Missing shard result gate"); for (const result of ["failure", "cancelled", "skipped", ""]) { @@ -77,11 +97,67 @@ it("keeps diagnosis available after merge failure and publishes only complete su const history = steps.find((step) => step.with?.name === "test-timings"); expect(history?.if).toContain("success()"); expect(history?.if).toContain("needs.test-shard.result == 'success'"); + expect(history?.if).toContain("needs.changes.outputs.tier == 'full'"); expect(history?.with?.path).toBe(".cache/ci-test-history/test-results.json"); - const download = (await workflow()).build!.steps.find( + const buildSteps = (await workflow()).build!.steps; + const lookup = buildSteps.find((step) => + step.run?.includes("test-history.mjs"), + ); + expect(lookup?.env).toMatchObject({ + CI_REPOSITORY: "${{ github.repository }}", + CI_REPOSITORY_ID: "${{ github.repository_id }}", + CI_DEFAULT_BRANCH: "${{ github.event.repository.default_branch }}", + }); + const download = buildSteps.find( (step) => step.name === "Download previous timing report", ); expect(download?.with?.name).toBe(history?.with?.name); + expect(download?.with?.["run-id"]).toBe( + "${{ steps.history.outputs.run-id }}", + ); +}); + +it("rejects successful shard reports with missing or duplicate portable files", async () => { + const root = await createTestTempDirectory("rea-ci-report-inventory-"); + await workflowGit(root, ["init"]); + await mkdir(join(root, "tests")); + for (const name of ["first", "second"]) + await writeFile(join(root, `tests/${name}.test.ts`), "// fixture\n"); + await workflowGit(root, ["add", "tests"]); + const run = async (names: readonly string[]) => { + await writeFile( + join(root, "test-timings.json"), + JSON.stringify({ + version: 1, + success: true, + files: names.map((name) => ({ name, durationMs: 1 })), + }), + ); + return execute(process.execPath, [fileURLToPath(script)], { + cwd: root, + env: { + ...process.env, + SHARD_RESULT: "success", + MERGE_RESULT: "success", + CI_VERIFY_TEST_INVENTORY: "true", + }, + }); + }; + await expect( + run(["tests/first.test.ts", "tests/second.test.ts"]), + ).resolves.toMatchObject({ stderr: "" }); + for (const names of [ + ["tests/first.test.ts"], + ["tests/first.test.ts", "tests/second.test.ts", "tests/unplanned.test.ts"], + ["tests/first.test.ts", "tests/first.test.ts", "tests/second.test.ts"], + ]) { + await expect(run(names)).rejects.toMatchObject({ + stderr: expect.stringContaining("Incomplete portable test inventory"), + }); + await expect( + readFile(join(root, ".cache/ci-test-history/test-results.json")), + ).rejects.toMatchObject({ code: "ENOENT" }); + } }); it("runs the existing onboarding and Python producer guards from their selected CI owners", async () => { @@ -116,20 +192,25 @@ it("exports portable timing history and removes it on a failed, partial, or inva const summary = join(root, "summary.md"); const history = join(root, ".cache/ci-test-history/test-results.json"); const report = { + version: 1, success: true, - numFailedTests: 0, - testResults: [ + files: [ { name: join(root, "tests/slow.test.ts"), - startTime: 10, - endTime: 110, - assertionResults: [{ large: "not timing data" }], + durationMs: 100, + phases: { + execution: 20, + preparation: 20, + environment: 10, + imports: 40, + setup: 10, + }, }, - { name: join(root, "tests/fast.test.ts"), startTime: 20, endTime: 30 }, + { name: join(root, "tests/fast.test.ts"), durationMs: 10 }, ], }; const run = async (value: unknown, shards = "success", merge = "success") => { - await writeFile(join(root, "test-results.json"), JSON.stringify(value)); + await writeFile(join(root, "test-timings.json"), JSON.stringify(value)); await writeFile(summary, ""); return execute(process.execPath, [fileURLToPath(script)], { cwd: root, @@ -143,7 +224,8 @@ it("exports portable timing history and removes it on a failed, partial, or inva }; await run(report); const contents = await readFile(history, "utf8"); - expect(contents).not.toContain("assertionResults"); + expect(contents).not.toContain("phases"); + expect(contents).not.toContain("startTime"); expect(contents).not.toContain(root); expect( parseTestDurations(contents, resolve(root, "different-checkout")), @@ -154,7 +236,7 @@ it("exports portable timing history and removes it on a failed, partial, or inva ]), ); for (const [value, shards, merge] of [ - [{ ...report, success: false, numFailedTests: 1 }, "failure", "failure"], + [{ ...report, success: false }, "failure", "failure"], [report, "cancelled", "success"], [report, "success", "failure"], ] as const) { @@ -167,8 +249,211 @@ it("exports portable timing history and removes it on a failed, partial, or inva await expect( run({ ...report, - testResults: [{ name: "../outside.test.ts", startTime: 1, endTime: 3 }], + files: [{ name: "../outside.test.ts", durationMs: 2 }], + }), + ).rejects.toMatchObject({ code: 1 }); + await expect(readFile(history)).rejects.toMatchObject({ code: "ENOENT" }); + await run(report); + await expect( + run({ + ...report, + files: [{ name: "tests/invalid.test.ts", durationMs: -1 }], }), ).rejects.toMatchObject({ code: 1 }); await expect(readFile(history)).rejects.toMatchObject({ code: "ENOENT" }); }); + +it("recovers complete module costs from real blob reports without executing tests again", async () => { + const root = await createTestTempDirectory("rea-ci-module-timing-"); + const config = join(root, "vitest.config.mjs"); + await writeFile( + config, + 'export default { test: { globals: true, include: ["sample.test.js"], maxWorkers: 1 } };', + ); + await writeFile( + join(root, "sample.test.js"), + [ + 'import { writeFileSync } from "node:fs";', + "beforeAll(() => new Promise(resolve => setTimeout(resolve, 80)));", + "afterAll(() => new Promise(resolve => setTimeout(resolve, 80)));", + 'it("has cheap assertions and expensive outer hooks", () => { writeFileSync("executed", "yes"); expect(1).toBe(1); });', + ].join("\n"), + ); + const vitest = fileURLToPath( + new URL("../../../node_modules/vitest/vitest.mjs", import.meta.url), + ); + const reporter = fileURLToPath( + new URL("../../../scripts/ci/test-timing-reporter.mjs", import.meta.url), + ); + const run = (args: string[]) => + execute(process.execPath, [vitest, ...args, "--config", config], { + cwd: root, + env: process.env, + timeout: 20_000, + }); + await run(["run", "--reporter=blob"]); + expect(await readFile(join(root, "executed"), "utf8")).toBe("yes"); + await rm(join(root, "executed")); + await run([ + "--merge-reports", + `--reporter=${reporter}`, + "--reporter=json", + "--outputFile.json=body.json", + ]); + await expect(readFile(join(root, "executed"))).rejects.toMatchObject({ + code: "ENOENT", + }); + const report = z + .object({ + version: z.literal(1), + success: z.literal(true), + files: z.array( + z.object({ + name: z.string(), + durationMs: z.number(), + phases: z.record(z.string(), z.number()), + }), + ), + }) + .parse(JSON.parse(await readFile(join(root, "test-timings.json"), "utf8"))); + expect( + JSON.parse(await readFile(join(root, "test-timings.json"), "utf8")), + ).toMatchObject({ + reason: "passed", + unhandledErrors: 0, + files: [{ state: "passed" }], + }); + expect(report.files).toHaveLength(1); + const file = report.files[0]!; + const json = z + .object({ + testResults: z.array( + z.object({ startTime: z.number(), endTime: z.number() }), + ), + }) + .parse(JSON.parse(await readFile(join(root, "body.json"), "utf8"))); + const body = json.testResults[0]!; + expect( + file.phases.execution! - (body.endTime - body.startTime), + ).toBeGreaterThan(100); + expect(file.phases.imports).toBeGreaterThan(0); + expect(file.phases.preparation).toBeGreaterThan(0); + expect(file.durationMs).toBeCloseTo( + Object.values(file.phases).reduce((sum, value) => sum + value, 0), + ); + await execute(process.execPath, [fileURLToPath(script)], { + cwd: root, + env: { ...process.env, SHARD_RESULT: "success", MERGE_RESULT: "success" }, + }); + expect( + parseTestDurations( + await readFile( + join(root, ".cache/ci-test-history/test-results.json"), + "utf8", + ), + root, + )?.get("sample.test.js"), + ).toBe(file.durationMs); + + await writeFile( + join(root, "sample.test.js"), + 'it("fails", () => { throw new Error("fixture failure"); });', + ); + await expect(run(["run", "--reporter=blob"])).rejects.toMatchObject({ + code: 1, + }); + await expect( + run(["--merge-reports", `--reporter=${reporter}`]), + ).rejects.toMatchObject({ code: 1 }); + const failed = JSON.parse( + await readFile(join(root, "test-timings.json"), "utf8"), + ) as unknown; + expect(failed).toMatchObject({ + version: 1, + success: false, + reason: "failed", + unhandledErrors: 0, + files: [{ state: "failed" }], + }); + // Even a caller accidentally marking the merge successful cannot publish this report. + await execute(process.execPath, [fileURLToPath(script)], { + cwd: root, + env: { ...process.env, SHARD_RESULT: "success", MERGE_RESULT: "success" }, + }); + await expect( + readFile(join(root, ".cache/ci-test-history/test-results.json")), + ).rejects.toMatchObject({ code: "ENOENT" }); +}); + +it("records failed omissions from full reports without treating partial or unhandled outcomes as parity", async () => { + const root = await createTestTempDirectory("rea-ci-selection-report-"); + await workflowGit(root, ["init"]); + const paths = [ + "src/sqlite/provider.test.ts", + "src/evm/provider.test.ts", + "tests/boundary/mcp/workflow.test.ts", + ]; + for (const path of paths) { + await mkdir(join(root, path, ".."), { recursive: true }); + await writeFile(join(root, path), "// fixture\n"); + } + await workflowGit(root, ["add", "src", "tests"]); + const timing = { + version: 1, + success: false, + reason: "failed", + unhandledErrors: 0, + files: paths.map((name, index) => ({ + name, + durationMs: 10, + state: index === 1 ? "failed" : "passed", + })), + }; + const run = async (report: unknown) => { + await writeFile(join(root, "test-timings.json"), JSON.stringify(report)); + await execute(process.execPath, [fileURLToPath(script)], { + cwd: root, + env: { + ...process.env, + SHARD_RESULT: "failure", + MERGE_RESULT: "failure", + CI_VERIFY_TEST_INVENTORY: "true", + CI_TEST_SELECTION: JSON.stringify( + planTestSelection(["src/sqlite/Provider.ts"], "pull_request", "fast"), + ), + GITHUB_SHA: "merge-sha", + CI_PR_HEAD_SHA: "pr-head-sha", + }, + }); + return JSON.parse( + await readFile( + join(root, ".cache/ci-test-selection/report.json"), + "utf8", + ), + ) as unknown; + }; + expect(await run(timing)).toMatchObject({ + observational: true, + complete: true, + selected: [paths[0], paths[2]], + omitted: [paths[1]], + missedFailures: [paths[1]], + executionSha: "merge-sha", + prHeadSha: "pr-head-sha", + }); + for (const report of [ + { ...timing, files: timing.files.slice(0, 1) }, + { ...timing, unhandledErrors: 1 }, + { ...timing, reason: "interrupted" }, + ]) + expect(await run(report)).toMatchObject({ complete: false }); + await expect(run({ ...timing, files: "invalid" })).rejects.toMatchObject({ + code: 1, + }); + await expect( + readFile(join(root, ".cache/ci-test-selection/report.json")), + ).rejects.toMatchObject({ code: "ENOENT" }); + await expect( + readFile(join(root, ".cache/ci-test-history/test-results.json")), + ).rejects.toMatchObject({ code: "ENOENT" }); +}); diff --git a/tests/boundary/filesystem/ciTestSequencer.test.ts b/tests/boundary/filesystem/ciTestSequencer.test.ts index 6177d9481..fec32826d 100644 --- a/tests/boundary/filesystem/ciTestSequencer.test.ts +++ b/tests/boundary/filesystem/ciTestSequencer.test.ts @@ -17,24 +17,24 @@ describe("CI test history", () => { const root = resolve("/workspace/rea"); const durations = parseTestDurations( JSON.stringify({ + version: 1, success: true, - numFailedTests: 0, - testResults: [ + files: [ { name: join(root, "tests/slow.test.ts"), - startTime: 10, - endTime: 110, + durationMs: 100, }, { name: join(root, "tests/slow.test.ts"), - startTime: 20, - endTime: 80, + durationMs: 60, }, { name: resolve(root, "../outside.test.ts"), - startTime: 10, - endTime: 1000, + durationMs: 990, }, + { name: "tests/negative.test.ts", durationMs: -1 }, + { name: "tests/zero.test.ts", durationMs: 0 }, + { name: "tests/invalid.test.ts", durationMs: "100" }, null, ], }), @@ -43,7 +43,20 @@ describe("CI test history", () => { expect(durations).toEqual(new Map([["tests/slow.test.ts", 100]])); expect( - parseTestDurations('{"success": false, "numFailedTests": 1}', root), + parseTestDurations('{"version": 1, "success": false, "files": []}', root), + ).toBeUndefined(); + // Legacy test-body spans must not be mixed with complete module costs. + expect( + parseTestDurations( + JSON.stringify({ + success: true, + numFailedTests: 0, + testResults: [ + { name: "tests/legacy.test.ts", startTime: 0, endTime: 100 }, + ], + }), + root, + ), ).toBeUndefined(); expect(parseTestDurations("not json", root)).toBeUndefined(); expect(parseTestDurations("null", root)).toBeUndefined(); diff --git a/tests/boundary/filesystem/ciVerifiedPackage.test.ts b/tests/boundary/filesystem/ciVerifiedPackage.test.ts new file mode 100644 index 000000000..ff5045024 --- /dev/null +++ b/tests/boundary/filesystem/ciVerifiedPackage.test.ts @@ -0,0 +1,137 @@ +import { createHash } from "node:crypto"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; + +import { expect, it } from "vitest"; + +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { + executeWorkflowFixture, + workflowGit, +} from "../../support/workflowGit.js"; + +const validator = fileURLToPath( + new URL("../../../scripts/ci/verified-package.mjs", import.meta.url), +); +type Fixture = Awaited>; +type FixtureOptions = { + override?: Record; + manifestContents?: string; + writeManifest?: boolean; +}; + +it("validates a tarball against the exact checkout and current workflow attempt", async () => { + const fixture = await createFixture(); + const result = await validate(fixture); + + expect(result.stdout.trim()).toBe( + join(fixture.artifactDirectory, "package.tgz"), + ); + expect(await readFile(fixture.output, "utf8")).toBe( + `tarball=${join(fixture.artifactDirectory, "package.tgz")}\n`, + ); +}); + +it("rejects a workflow SHA that differs from the checked out HEAD", async () => { + const fixture = await createFixture(); + + await expect(validate(fixture, "c".repeat(40))).rejects.toThrow(); +}); + +it.each([ + ["wrong checkout SHA", { sourceSha: "c".repeat(40) }], + ["wrong workflow run", { runId: 124 }], + ["future workflow attempt", { runAttempt: 3 }], + ["wrong verifier platform", { platform: "darwin" }], + ["wrong verifier architecture", { architecture: "arm64" }], + ["wrong package digest", { sha256: "c".repeat(64) }], +])("rejects %s", async (_label, override) => { + const fixture = await createFixture({ override }); + + await expect(validate(fixture)).rejects.toThrow(); +}); + +it("accepts an earlier successful attempt from the same workflow run", async () => { + const fixture = await createFixture({ override: { runAttempt: 1 } }); + + await expect(validate(fixture)).resolves.toMatchObject({ + stdout: expect.any(String), + }); +}); + +it.each([ + ["malformed manifest", "{"], + ["manifest with missing fields", JSON.stringify({ schemaVersion: 1 })], +])("rejects %s", async (_label, contents) => { + const fixture = await createFixture({ manifestContents: contents }); + + await expect(validate(fixture)).rejects.toThrow(); +}); + +it("rejects an absent manifest", async () => { + const fixture = await createFixture({ writeManifest: false }); + + await expect(validate(fixture)).rejects.toThrow(); +}); + +async function createFixture({ + override = {}, + manifestContents, + writeManifest = true, +}: FixtureOptions = {}) { + const directory = await createTestTempDirectory("rea-verified-package-"); + await workflowGit(directory, ["init", "--initial-branch=main"]); + await writeFile( + join(directory, "package.json"), + '{"name":"fixture","version":"1.2.3"}\n', + ); + await workflowGit(directory, ["add", "package.json"]); + await workflowGit(directory, [ + "commit", + "-m", + "test: verified package fixture", + ]); + const sourceSha = await workflowGit(directory, ["rev-parse", "HEAD"]); + const artifactDirectory = join(directory, "artifact"); + await mkdir(artifactDirectory); + const archive = "verified tarball fixture"; + await writeFile(join(artifactDirectory, "package.tgz"), archive); + const manifest = { + schemaVersion: 1, + sourceSha, + runId: 123, + runAttempt: 2, + platform: "linux", + architecture: "x64", + packageVersion: "1.2.3", + nodeVersion: process.version, + sha256: createHash("sha256").update(archive).digest("hex"), + ...override, + }; + if (writeManifest) + await writeFile( + join(artifactDirectory, "manifest.json"), + manifestContents ?? JSON.stringify(manifest), + ); + const output = join(directory, "github-output"); + await writeFile(output, ""); + return { directory, sourceSha, artifactDirectory, output }; +} + +function validate(fixture: Fixture, sourceSha = fixture.sourceSha) { + return executeWorkflowFixture( + process.execPath, + [validator, "validate", fixture.artifactDirectory], + { + cwd: fixture.directory, + env: { + ...process.env, + GITHUB_SHA: sourceSha, + GITHUB_RUN_ID: "123", + GITHUB_RUN_ATTEMPT: "2", + GITHUB_OUTPUT: fixture.output, + }, + }, + ); +} diff --git a/tests/boundary/filesystem/clientConfigurationTransactions.test.ts b/tests/boundary/filesystem/clientConfigurationTransactions.test.ts index 477085381..507b7137e 100644 --- a/tests/boundary/filesystem/clientConfigurationTransactions.test.ts +++ b/tests/boundary/filesystem/clientConfigurationTransactions.test.ts @@ -1,13 +1,91 @@ import { chmod, readFile, writeFile } from "node:fs/promises"; +import { createHash } from "node:crypto"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; import { parse as parseToml } from "smol-toml"; -import { configureClientConfiguration } from "../../../src/application/SetupClientConfiguration.js"; +import { + configureClientConfiguration, + inspectClientConfiguration, +} from "../../../src/application/SetupClientConfiguration.js"; import { systemUninstallHost } from "../../../src/application/Uninstall.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +it("accepts a literal UTF-8 replacement character without changing its value", async () => { + const root = await createTestTempDirectory("rea-client-config-utf8-"); + const configPath = join(root, "config.json"); + const original = Buffer.concat([ + Buffer.from('{"label":"'), + Buffer.from("\uFFFD"), + Buffer.from('"}'), + ]); + await writeFile(configPath, original); + const result = await configureClientConfiguration( + { name: "cursor", format: "json", configPath }, + {}, + ["rea", "mcp"], + ); + expect(result.status).toBe("configured"); + expect(await readFile(`${configPath}.rea.backup`)).toEqual(original); + expect(JSON.parse(await readFile(configPath, "utf8")).label).toBe("\uFFFD"); +}); + +it("previews and creates a current recovery backup while preserving the first backup", async () => { + const root = await createTestTempDirectory("rea-client-current-recovery-"); + const configPath = join(root, "config.json"); + const firstOriginal = '{"keep":"first"}\n'; + const currentOriginal = '{"keep":"current"}\n'; + const client = { name: "cursor", format: "json", configPath } as const; + await writeFile(configPath, currentOriginal); + await writeFile(`${configPath}.rea.backup`, firstOriginal); + + const recoveryPath = `${configPath}.rea.backup.${createHash("sha256") + .update(currentOriginal, "utf8") + .digest("hex")}.recovery`; + expect(await inspectClientConfiguration(client, {}, ["rea", "mcp"])).toEqual({ + status: "update", + backupPath: recoveryPath, + }); + + expect( + await configureClientConfiguration(client, {}, ["rea", "mcp"]), + ).toEqual({ status: "configured", backupPath: recoveryPath }); + expect(await readFile(`${configPath}.rea.backup`, "utf8")).toBe( + firstOriginal, + ); + expect(await readFile(recoveryPath, "utf8")).toBe(currentOriginal); +}); + +it("blocks configuration when the planned content-addressed backup is occupied", async () => { + const root = await createTestTempDirectory("rea-client-recovery-conflict-"); + const configPath = join(root, "config.json"); + const firstOriginal = '{"keep":"first"}\n'; + const currentOriginal = '{"keep":"current"}\n'; + const client = { name: "cursor", format: "json", configPath } as const; + await writeFile(configPath, currentOriginal); + await writeFile(`${configPath}.rea.backup`, firstOriginal); + const recoveryPath = `${configPath}.rea.backup.${createHash("sha256") + .update(currentOriginal, "utf8") + .digest("hex")}.recovery`; + await writeFile(recoveryPath, "occupied by other data"); + + expect( + await inspectClientConfiguration(client, {}, ["rea", "mcp"]), + ).toMatchObject({ + status: "invalid", + remediation: expect.stringContaining(recoveryPath), + }); + expect( + await configureClientConfiguration(client, {}, ["rea", "mcp"]), + ).toEqual({ status: "failed", reason: "backup" }); + expect(await readFile(configPath, "utf8")).toBe(currentOriginal); + expect(await readFile(`${configPath}.rea.backup`, "utf8")).toBe( + firstOriginal, + ); + expect(await readFile(recoveryPath, "utf8")).toBe("occupied by other data"); +}); + describe("client configuration write failures", () => { it.skipIf(process.platform === "win32" || process.getuid?.() === 0).each([ ["json", configureClientConfiguration], diff --git a/tests/boundary/filesystem/clientLifecycle.test.ts b/tests/boundary/filesystem/clientLifecycle.test.ts index 506a6d6df..63ca0667a 100644 --- a/tests/boundary/filesystem/clientLifecycle.test.ts +++ b/tests/boundary/filesystem/clientLifecycle.test.ts @@ -71,6 +71,7 @@ describe("client configuration filesystem lifecycle", () => { "pi", "hermes", "qoder", + "droid", "zcode", "dsh", "grok_bot", diff --git a/tests/boundary/filesystem/developmentTests.test.ts b/tests/boundary/filesystem/developmentTests.test.ts deleted file mode 100644 index 1b52bb475..000000000 --- a/tests/boundary/filesystem/developmentTests.test.ts +++ /dev/null @@ -1,110 +0,0 @@ -import { - copyFile, - mkdir, - readFile, - symlink, - writeFile, -} from "node:fs/promises"; -import { join } from "node:path"; - -import { expect, it } from "vitest"; - -import { execFileOutput } from "../../../src/process/ExecFileOutput.js"; -import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; - -const run = (root: string, command: string, arguments_: readonly string[]) => - execFileOutput(command, arguments_, { - cwd: root, - stopSignal: "SIGTERM", - timeout: 20_000, - }); - -const commit = (root: string, message: string) => - run(root, "git", [ - "-c", - "user.name=REA fixture", - "-c", - "user.email=fixture@example.invalid", - "-c", - "commit.gpgSign=false", - "commit", - "--quiet", - "-m", - message, - ]); - -const fixture = async () => { - const root = await createTestTempDirectory("rea-development-tests-"); - await mkdir(join(root, "scripts", "lib"), { recursive: true }); - await mkdir(join(root, "src")); - for (const path of [ - "scripts/run-development-tests.mjs", - "scripts/lib/development-tests.mjs", - ]) - await copyFile(join(process.cwd(), path), join(root, path)); - await symlink( - join(process.cwd(), "node_modules"), - join(root, "node_modules"), - process.platform === "win32" ? "junction" : undefined, - ); - await writeFile(join(root, "package.json"), '{"type":"module"}\n'); - await writeFile(join(root, ".gitignore"), "node_modules\nselected.txt\n"); - await writeFile( - join(root, "vitest.config.mjs"), - 'export default { test: { projects: [{ test: { name: "domain", include: ["src/**/*.test.ts"] } }] } };\n', - ); - await writeFile(join(root, "src", "value.ts"), "export const value = 1;\n"); - await writeFile( - join(root, "src", "value.test.ts"), - 'import { expect, it } from "vitest"; import { writeFileSync } from "node:fs"; import { value } from "./value.js"; it("checks the selected source", () => { expect(value).toBe(1); writeFileSync("selected.txt", String(value)); });\n', - ); - await run(root, "git", ["init", "--quiet"]); - await run(root, "git", ["add", "."]); - await commit(root, "fixture"); - return root; -}; - -const testCommand = (root: string, ...arguments_: string[]) => - run(root, process.execPath, [ - "scripts/run-development-tests.mjs", - ...arguments_, - ]); - -it("executes an explicitly selected source test on a clean tree without build artifacts", async () => { - const root = await fixture(); - const { stdout: status } = await run(root, "git", ["status", "--porcelain"]); - expect(status).toBe(""); - await testCommand(root, "local", "src/value.test.ts"); - expect(await readFile(join(root, "selected.txt"), "utf8")).toBe("1"); -}); - -it("uses committed dependency changes since the requested Git base to select source tests", async () => { - const root = await fixture(); - const { stdout: base } = await run(root, "git", ["rev-parse", "HEAD"]); - await writeFile(join(root, "src", "value.ts"), "export const value = 2;\n"); - await run(root, "git", ["add", "src/value.ts"]); - await commit(root, "change dependency"); - await expect( - testCommand(root, "changed", "--base", base.trim()), - ).rejects.toMatchObject({ - code: 1, - stdout: expect.stringContaining("src/value.test.ts"), - stderr: expect.stringContaining("expected 2 to be 1"), - }); -}); - -it("rejects a compiled boundary in the source-only command and a missing exact test", async () => { - const root = await fixture(); - await expect( - testCommand(root, "local", "tests/acceptance/runtime.test.ts"), - ).rejects.toMatchObject({ - code: 1, - stderr: expect.stringContaining("Use test:focused"), - }); - await expect( - testCommand(root, "local", "src/missing.test.ts"), - ).rejects.toMatchObject({ - code: 1, - stderr: expect.stringContaining("ENOENT"), - }); -}); diff --git a/tests/boundary/filesystem/directAnalysisSnapshotReplay.test.ts b/tests/boundary/filesystem/directAnalysisSnapshotReplay.test.ts index 6687b0ef7..7adf193f4 100644 --- a/tests/boundary/filesystem/directAnalysisSnapshotReplay.test.ts +++ b/tests/boundary/filesystem/directAnalysisSnapshotReplay.test.ts @@ -1,5 +1,5 @@ import { parseConfig } from "../../../src/config/parseConfig.js"; -import { writeFile } from "node:fs/promises"; +import { symlink, writeFile } from "node:fs/promises"; import { basename, dirname, join } from "node:path"; import { describe, expect, it, vi } from "vitest"; @@ -14,7 +14,7 @@ import type { import type { DirectAnalysisDependencies } from "../../../src/application/DirectAnalysisDependencies.js"; import { createAnalysisProfile } from "../../../src/domain/analysisProfile.js"; import { createAnalysisExecution } from "../../../src/application/AnalysisProvider.js"; -import { ok } from "../../../src/domain/result.js"; +import { err, ok } from "../../../src/domain/result.js"; import { readAnalysisSnapshot, writeAnalysisSnapshot, @@ -40,6 +40,7 @@ import { } from "../../fixtures/binarySession.js"; import { BinarySession } from "../../../src/application/binary/BinarySession.js"; import { OwnedStagedOutput } from "../../../src/filesystem/OwnedStagedOutput.js"; +import { ProviderAdapterError } from "../../../src/domain/providerAdapterError.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { CAPABILITIES as GHIDRA_CAPABILITIES, @@ -396,7 +397,85 @@ describe("snapshot output cleanup lifecycle", () => { }); }); +const verifyWindowsSnapshotAdmission = async ( + selectedPath: "direct" | "junction", +): Promise => { + const fixture = await createSnapshotReplayFixture( + "rea-direct-snapshot-junction-", + ); + const rawPath = join(dirname(fixture.path), "image.bin"); + await writeFile(rawPath, Buffer.from("a92a60", "hex")); + const linked = join(dirname(fixture.path), "linked"); + const linkedPath = join(linked, basename(rawPath)); + const identity = { ...GHIDRA_PROVIDER_IDENTITY, version: "12.1.4" }; + const profile = createAnalysisProfile(identity, { + fixture: true, + }); + const baseProvider = fixture.createProvider({ + identity, + profile, + supportedOperations: ["search_strings"], + }); + let rejectAdmission = false; + const provider: AnalysisProvider = { + ...baseProvider, + createClient: (target, profile, context) => { + const client = baseProvider.createClient(target, profile, context); + return { + ...client, + execute: (operation, parameters, options) => + rejectAdmission && operation === "search_strings" + ? Promise.resolve( + err(new ProviderAdapterError("ghidra", operation)), + ) + : client.execute(operation, parameters, options), + }; + }, + }; + const dependencies = fixture.dependencies(provider); + const arguments_ = { pattern: "fixture", document: "fixture" }; + const options = { + snapshotPath: fixture.snapshotPath, + formatHint: "raw-binary" as const, + architecture: "6502" as const, + entryPoint: "0x8001", + }; + const first = await runDirectAnalysis( + dependencies, + rawPath, + "search_strings", + arguments_, + options, + ); + expect(first).toMatchObject({ operation: "search_strings" }); + expect(fixture.starts).toHaveLength(1); + if (selectedPath === "junction") + await symlink(dirname(fixture.path), linked, "junction"); + rejectAdmission = true; + const second = await runDirectAnalysis( + dependencies, + selectedPath === "junction" ? linkedPath : rawPath, + "search_strings", + arguments_, + options, + ); + expect(second).toMatchObject({ error: "Analysis failed" }); + expect(fixture.starts).toHaveLength(2); + expect( + fixture.calls.filter((call) => call === "search_strings"), + ).toHaveLength(1); +}; + describe("direct analysis composed snapshot replay", () => { + it.skipIf(process.platform !== "win32")( + "does not replay a CLI snapshot through a Windows junction", + () => verifyWindowsSnapshotAdmission("junction"), + ); + it.skipIf(process.platform !== "win32")( + "does not replay a CLI snapshot before native admission on a canonical path", + () => verifyWindowsSnapshotAdmission("direct"), + ); + it.each([ { tool: "search_strings", @@ -516,7 +595,7 @@ describe("Ghidra composed workflow snapshot replay", () => { { tool: "inspect_native_api", arguments: { procedure: "0x1000" } }, { tool: "trace_feature", arguments: { query: "fixture" } }, ] as const)( - "persists and replays $tool with production Ghidra capability policies", + "persists $tool with production Ghidra capability policies", async ({ tool, arguments: parameters }) => { const fixture = await createSnapshotReplayFixture("rea-ghidra-workflow-"); const profile = createAnalysisProfile( @@ -554,8 +633,14 @@ describe("Ghidra composed workflow snapshot replay", () => { { snapshotPath: fixture.snapshotPath }, ); expect(replay).toEqual(first); - expect(fixture.starts).toHaveLength(1); - expect(fixture.calls).toEqual(initialCalls); + if (process.platform === "win32") { + // The first provider call in a new CLI session admits its source. + expect(fixture.starts).toHaveLength(2); + expect(fixture.calls.length).toBeGreaterThan(initialCalls.length); + } else { + expect(fixture.starts).toHaveLength(1); + expect(fixture.calls).toEqual(initialCalls); + } }, ); }); diff --git a/tests/boundary/filesystem/droidClientConfiguration.test.ts b/tests/boundary/filesystem/droidClientConfiguration.test.ts new file mode 100644 index 000000000..6f9b26423 --- /dev/null +++ b/tests/boundary/filesystem/droidClientConfiguration.test.ts @@ -0,0 +1,167 @@ +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +import { readClientRegistrationStatuses } from "../../../src/application/ClientRegistrationStatus.js"; +import { npxRegistrationCommand } from "../../../src/application/ClientRegistrationIdentity.js"; +import { configureClientConfiguration } from "../../../src/application/SetupClientConfiguration.js"; +import { skillDestinations } from "../../../src/application/SetupSkill.js"; +import { supportedClients } from "../../../src/application/SupportedClients.js"; +import { systemUninstallHost } from "../../../src/application/Uninstall.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { clearClientLocationEnvironment } from "../../fixtures/clientEnvironment.js"; + +/** + * The registration command setup writes on the host running the tests. Windows + * wraps it in `cmd.exe /d /c`, so the expected entry follows the host too. + */ +const command = npxRegistrationCommand(process.platform); + +beforeEach(clearClientLocationEnvironment); +afterEach(() => vi.unstubAllEnvs()); + +const droidPath = ( + home: string, + platform: NodeJS.Platform = "linux", + env: Parameters[2] = {}, +) => supportedClients(home, platform, env).find(({ name }) => name === "droid"); + +describe("Factory Droid configuration paths", () => { + it("resolves the user MCP config file and marker directory", () => { + expect(droidPath("/home/a")).toEqual({ + name: "droid", + displayName: "Factory Droid", + configPath: join("/home/a", ".factory", "mcp.json"), + markerPath: join("/home/a", ".factory"), + format: "json", + }); + expect(droidPath("/Users/a", "darwin")?.configPath).toBe( + join("/Users/a", ".factory", "mcp.json"), + ); + expect(droidPath("C:\\Users\\a", "win32")?.configPath).toBe( + join("C:\\Users\\a", ".factory", "mcp.json"), + ); + }); + + it("resolves the personal skills directory beside the config root", () => { + expect( + skillDestinations("/home/a", ["droid"]).map(({ path }) => path), + ).toEqual([ + join("/home/a", ".factory", "skills", "reverse-engineer-anything"), + ]); + }); +}); + +describe("Factory Droid JSON registration", () => { + const droidClient = async () => { + const home = await createTestTempDirectory("rea-droid-"); + const client = droidPath(home, process.platform); + if (client?.markerPath === undefined) + throw new Error("missing Factory Droid client"); + await mkdir(client.markerPath, { recursive: true }); + return { home, client }; + }; + + it("upserts rea into mcpServers while preserving unrelated servers and permissions", async () => { + const { home, client } = await droidClient(); + await writeFile( + client.configPath, + `{ + "mcpServers": { + "other": { + "type": "stdio", + "command": "other", + "args": [], + "disabled": false + } + }, + "persistentPermissions": { + "servers": { + "other": { "approvedAt": "2026-01-01T00:00:00.000Z" } + } + } +}`, + ); + + expect( + (await configureClientConfiguration(client, {}, command)).status, + ).toBe("configured"); + const parsed = JSON.parse(await readFile(client.configPath, "utf8")); + expect(parsed.mcpServers.other).toEqual({ + type: "stdio", + command: "other", + args: [], + disabled: false, + }); + expect(parsed.persistentPermissions).toEqual({ + servers: { other: { approvedAt: "2026-01-01T00:00:00.000Z" } }, + }); + expect(parsed.mcpServers.rea).toEqual({ + command: command[0], + args: command.slice(1), + }); + + expect(await configureClientConfiguration(client, {}, command)).toEqual({ + status: "unchanged", + }); + expect( + ( + await readClientRegistrationStatuses(home, undefined, { + platform: process.platform, + environment: {}, + }) + ).find(({ client }) => client === "droid"), + ).toMatchObject({ state: "aligned", command }); + }); + + it("creates the mcpServers table in an existing config file without servers", async () => { + const { client } = await droidClient(); + await writeFile( + client.configPath, + JSON.stringify({ persistentPermissions: { servers: {} } }, null, 2), + ); + expect( + (await configureClientConfiguration(client, {}, command)).status, + ).toBe("configured"); + const parsed = JSON.parse(await readFile(client.configPath, "utf8")); + expect(parsed.persistentPermissions).toEqual({ servers: {} }); + expect(parsed.mcpServers.rea).toEqual({ + command: command[0], + args: command.slice(1), + }); + }); + + it("registers into an empty config file", async () => { + const { client } = await droidClient(); + await writeFile(client.configPath, ""); + expect( + (await configureClientConfiguration(client, {}, command)).status, + ).toBe("configured"); + const parsed = JSON.parse(await readFile(client.configPath, "utf8")); + expect(parsed.mcpServers.rea).toEqual({ + command: command[0], + args: command.slice(1), + }); + }); + + it("removes only the rea entry on uninstall", async () => { + const { home, client } = await droidClient(); + await writeFile( + client.configPath, + JSON.stringify( + { mcpServers: { other: { command: "other-server" } } }, + null, + 2, + ), + ); + await configureClientConfiguration(client, {}, command); + + expect((await systemUninstallHost(home).removeClient(client)).status).toBe( + "removed", + ); + const parsed = JSON.parse(await readFile(client.configPath, "utf8")); + expect(parsed.mcpServers.other).toEqual({ command: "other-server" }); + expect(parsed.mcpServers.rea).toBeUndefined(); + }); +}); diff --git a/tests/boundary/filesystem/dshClientConfiguration.test.ts b/tests/boundary/filesystem/dshClientConfiguration.test.ts index 72dbded9e..c0315f67b 100644 --- a/tests/boundary/filesystem/dshClientConfiguration.test.ts +++ b/tests/boundary/filesystem/dshClientConfiguration.test.ts @@ -1,5 +1,5 @@ import { mkdir, readFile, writeFile } from "node:fs/promises"; -import { join } from "node:path"; +import { join, resolve } from "node:path"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { @@ -37,28 +37,76 @@ const dshClient = ( env: Parameters[2], platform: NodeJS.Platform = "linux", ) => - supportedClients(join("/home/a"), platform, env).find( + supportedClients(resolve("/home/a"), platform, env).find( ({ name }) => name === "dsh", ); describe("DeepSeek Harness configuration paths", () => { + it.each([ + [undefined, ".dsh", ".agents", "shared"], + ["", ".dsh", ".agents", "shared"], + [" \t ", ".dsh", ".agents", "shared"], + ["~", "", "", "dsh"], + ["~/selected home", "selected home", "selected home", "dsh"], + ["~\\selected home", "selected home", "selected home", "dsh"], + ] as const)( + "resolves Harness and agents roots for %j", + (override, dshRoot, agentsRoot, client) => { + const home = resolve("/home/a"); + const environment = { DSH_HOME: override, DSH_AGENTS_HOME: override }; + expect( + supportedClients(home, process.platform, environment).find( + ({ name }) => name === "dsh", + ), + ).toMatchObject({ + configPath: join(home, dshRoot, "cordis.patch.yml"), + markerPath: join(home, dshRoot), + }); + expect(clientSkillDirectories(home, ["dsh"], environment)).toEqual([ + { client, directory: join(home, agentsRoot, "skills") }, + ]); + }, + ); + + it.each(["relative home", " literal home ", "~someone"])( + "resolves a literal root %j relative to cwd without trimming it", + (override) => { + const home = resolve("/home/a"); + const root = resolve(override); + const environment = { DSH_HOME: override, DSH_AGENTS_HOME: override }; + expect( + supportedClients(home, process.platform, environment).find( + ({ name }) => name === "dsh", + ), + ).toMatchObject({ + configPath: join(root, "cordis.patch.yml"), + markerPath: root, + }); + expect(clientSkillDirectories(home, ["dsh"], environment)).toEqual([ + { client: "dsh", directory: join(root, "skills") }, + ]); + }, + ); + it("resolves the home Cordis patch under the Harness home", () => { expect(dshClient({})).toEqual({ name: "dsh", displayName: "DeepSeek Harness", - configPath: join("/home/a", ".dsh", "cordis.patch.yml"), - markerPath: join("/home/a", ".dsh"), + configPath: resolve("/home/a", ".dsh", "cordis.patch.yml"), + markerPath: resolve("/home/a", ".dsh"), format: "dsh", }); // The home patch applies to every profile; DSH_HOME relocates it. expect(dshClient({ DSH_HOME: "/custom/harness" })?.configPath).toBe( - join("/custom/harness", "cordis.patch.yml"), + resolve("/custom/harness", "cordis.patch.yml"), ); expect( clientSkillDirectories(join("/home/a"), ["dsh"], { DSH_AGENTS_HOME: "/custom/agents", }), - ).toEqual([{ client: "dsh", directory: join("/custom/agents", "skills") }]); + ).toEqual([ + { client: "dsh", directory: resolve("/custom/agents", "skills") }, + ]); }); }); diff --git a/tests/boundary/filesystem/ghidraJavaProbe.test.ts b/tests/boundary/filesystem/ghidraJavaProbe.test.ts index c0a6974d5..415fbf3dd 100644 --- a/tests/boundary/filesystem/ghidraJavaProbe.test.ts +++ b/tests/boundary/filesystem/ghidraJavaProbe.test.ts @@ -87,3 +87,43 @@ it.each([ ); }, ); + +it("bounds the Java version probe and reports an over-budget probe as a failed Java check", async () => { + const root = await createTestTempDirectory("rea-java-probe-bound-"); + const configuredHome = join(root, "selected-jdk"); + const installDir = join(root, "ghidra"); + await mkdir(join(installDir, "Ghidra"), { recursive: true }); + await writeFile( + join(installDir, "Ghidra", "application.properties"), + `application.version=${SUPPORTED_GHIDRA_VERSION}\napplication.java.min=17\n`, + ); + // A chatty or hostile JVM that overruns the capture budget: spawnSync reports + // ENOBUFS, which the probe already treats as a failed observation. + mockSpawnSync.mockReturnValue({ + stdout: "", + stderr: "java.version = 21.0.11\n", + status: null, + error: Object.assign(new Error("spawnSync /java ENOBUFS"), { + code: "ENOBUFS", + }), + }); + + const inspection = inspectGhidraInstallation({ + environment: {}, + installDir, + javaHome: configuredHome, + }); + + const [, , options] = mockSpawnSync.mock.calls[0] as [ + string, + readonly string[], + { readonly maxBuffer?: number }, + ]; + expect(options.maxBuffer).toBe(1024 * 1024); + expect(inspection.javaVersion).toBeNull(); + expect(inspection.javaHome).toBe(configuredHome); + expect(inspection.checks.find(({ name }) => name === "java")).toMatchObject({ + status: "failed", + code: "runtime_missing", + }); +}); diff --git a/tests/boundary/filesystem/ghidraPspExtension.test.ts b/tests/boundary/filesystem/ghidraPspExtension.test.ts index 88e25939a..65a4d1512 100644 --- a/tests/boundary/filesystem/ghidraPspExtension.test.ts +++ b/tests/boundary/filesystem/ghidraPspExtension.test.ts @@ -52,7 +52,7 @@ const target = { const host: GhidraInstallationHost = { platform: "linux", architecture: "x64", - readText: () => "application.version=12.1.3\n", + readText: () => "application.version=12.1.3\napplication.java.min=21\n", executable: () => true, probeJava: () => ({ version: "21.0.12", @@ -191,12 +191,9 @@ describe.skipIf(process.platform === "win32")( it.each([ ["name=other\nversion=12.1.3\n", "does not match"], ["name=ghidra-allegrex\nversion=12.1.4\n", "does not match"], - [ - "name=ghidra-allegrex\nversion=12.1.3\nversion=12.1.3\n", - "exactly one version", - ], + ["name=ghidra-allegrex\nversion=12.1.\\uZZZZ\n", "Unicode escape"], ])( - "rejects incompatible or ambiguous extension metadata %s", + "rejects mismatched or malformed extension metadata %s", async (properties, reason) => { const { root } = await installation(); await writeFile(join(root, "extension.properties"), properties); @@ -504,3 +501,38 @@ describe.skipIf(process.platform === "win32")( }); }, ); + +describe.skipIf(process.platform === "win32")( + "PSP extension property syntax", + () => { + it.each([ + { + form: "escaped key and value with colon separator", + properties: "n\\u0061me : ghidra-allegrex\nver\\u0073ion=12.1.3\n", + }, + { + form: "continued key and value lines", + properties: "name=ghidra-\\\n allegrex\nversion=12.1.\\\n 3\n", + }, + { + form: "duplicate keys use the last value", + properties: + "name=wrong\nname=ghidra-allegrex\nversion=12.1.4\nversion=12.1.3\n", + }, + ])( + "accepts Java Properties $form and hashes raw bytes", + async ({ properties }) => { + const { root } = await installation(); + const canonical = await inspectGhidraPspExtension(root, "12.1.3"); + const latin1Properties = Buffer.from( + `${properties}author=caf\u00e9\n`, + "latin1", + ); + await writeFile(join(root, "extension.properties"), latin1Properties); + const extension = await inspectGhidraPspExtension(root, "12.1.3"); + expect(extension.id).toBe("ghidra-allegrex"); + expect(extension.sha256).not.toBe(canonical.sha256); + }, + ); + }, +); diff --git a/tests/boundary/filesystem/hermesClientConfiguration.test.ts b/tests/boundary/filesystem/hermesClientConfiguration.test.ts index 43a5f841c..218573d5c 100644 --- a/tests/boundary/filesystem/hermesClientConfiguration.test.ts +++ b/tests/boundary/filesystem/hermesClientConfiguration.test.ts @@ -237,6 +237,38 @@ mcp_servers: expect(await readFile(client.configPath, "utf8")).toBe(original); }); + it("keeps unrelated numbers as written, including Discord IDs", async () => { + const { home, client } = await hermesClient(); + // Hermes reads these exactly: two IDs one apart, an octal mode, a float. + const original = `discord: + free_response_channels: + - 1187654321098765432 + channel_prompts: + 1187654321098765433: Build questions. + 1187654321098765434: Release questions. + mode: 0600 + min_score: 0.0000001 +mcp_servers: + other: + command: other-server +`; + await writeFile(client.configPath, original); + expect( + (await configureClientConfiguration(client, {}, command)).status, + ).toBe("configured"); + const text = await readFile(client.configPath, "utf8"); + expect(text.startsWith(original)).toBe(true); + expect(parseYaml(text, { intAsBigInt: true })).toMatchObject({ + mcp_servers: { + rea: { command: command[0], args: command.slice(1), enabled: true }, + }, + }); + expect((await systemUninstallHost(home).removeClient(client)).status).toBe( + "removed", + ); + expect(await readFile(client.configPath, "utf8")).toBe(original); + }); + it("creates config.yaml when Hermes has no configuration yet", async () => { const { client } = await hermesClient(); expect( diff --git a/tests/boundary/filesystem/installScript.test.ts b/tests/boundary/filesystem/installScript.test.ts index 3fbcf7a3c..0f2e14730 100644 --- a/tests/boundary/filesystem/installScript.test.ts +++ b/tests/boundary/filesystem/installScript.test.ts @@ -11,6 +11,7 @@ import { import { join } from "node:path"; import { promisify } from "node:util"; +import { spawn as spawnPty } from "@lydell/node-pty"; import { describe, expect, it } from "vitest"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; @@ -185,6 +186,69 @@ describe("curl installer scenarios", { timeout: 20_000 }, () => { }); }); +describe("installer TTY fallback", { timeout: 20_000 }, () => { + it("skips automatic setup when stderr is redirected", async () => { + const fixture = await createFixture(); + const script = join(process.cwd(), "install.sh"); + const wrapper = ` + const { spawnSync } = require("node:child_process"); + const { openSync, closeSync } = require("node:fs"); + const stderr = openSync(process.argv[2], "w"); + const child = spawnSync("/bin/bash", [process.argv[1], ...process.argv.slice(3)], { + stdio: ["inherit", "inherit", stderr], + env: process.env, + }); + closeSync(stderr); + process.exit(child.status ?? 1); + `; + let output = ""; + const terminal = spawnPty( + process.execPath, + ["-e", wrapper, script, fixture.installerStderrLog, "--version", "0.3.0"], + { + cwd: process.cwd(), + env: { + ...process.env, + HOME: fixture.home, + TMPDIR: fixture.temporary, + PATH: `${fixture.bin}:/usr/bin:/bin`, + FAKE_NPM_LOG: fixture.npmLog, + FAKE_REA_LOG: fixture.reaLog, + FAKE_REA_SOURCE: join(fixture.bin, "rea-source"), + FAKE_NPM_PREFIX: join(fixture.home, ".npm-global"), + TERM: "xterm-256color", + }, + name: "xterm-256color", + cols: 120, + rows: 40, + }, + ); + let timedOut = false; + const exitCode = await new Promise((resolve, reject) => { + const timeout = setTimeout(() => { + timedOut = true; + terminal.kill(); + }, 10_000); + terminal.onData((data) => { + output += data; + }); + terminal.onExit(({ exitCode: code }) => { + clearTimeout(timeout); + if (timedOut) + reject(new Error(`Installer PTY timed out after output: ${output}`)); + else resolve(code); + }); + }); + + expect(exitCode).toBe(0); + expect(output).toContain( + `Run ${join(fixture.home, ".local/bin/rea")} setup to configure Hopper and agents.`, + ); + expect(await readFile(fixture.reaLog, "utf8")).toBe("--version\n"); + expect(await readFile(fixture.installerStderrLog, "utf8")).toBe(""); + }); +}); + describe("truncated installer download", { timeout: 20_000 }, () => { it("runs nothing from a download truncated after the npm install line", async () => { const fixture = await createFixture(); @@ -433,6 +497,7 @@ interface InstallerFixture { readonly temporary: string; readonly npmLog: string; readonly reaLog: string; + readonly installerStderrLog: string; } const getProcessOutput = ( @@ -456,6 +521,7 @@ const createFixture = async (): Promise => { const temporary = join(root, "tmp"); const npmLog = join(root, "npm.log"); const reaLog = join(root, "rea.log"); + const installerStderrLog = join(root, "installer.stderr.log"); await Promise.all([mkdir(home), mkdir(bin), mkdir(temporary)]); await Promise.all([ executable( @@ -507,7 +573,7 @@ exit 0 symlink(`/usr/bin/${command}`, join(bin, command)), ), ]); - return { home, bin, temporary, npmLog, reaLog }; + return { home, bin, temporary, npmLog, reaLog, installerStderrLog }; }; const executable = async (path: string, source: string): Promise => { diff --git a/tests/boundary/filesystem/interfaceBuilderBundle.test.ts b/tests/boundary/filesystem/interfaceBuilderBundle.test.ts index 4446c80dc..d584aaf3c 100644 --- a/tests/boundary/filesystem/interfaceBuilderBundle.test.ts +++ b/tests/boundary/filesystem/interfaceBuilderBundle.test.ts @@ -1060,6 +1060,142 @@ describe("NIBArchive UIKit event connections", () => { "UIRuntimeEventConnection", ); }); + + it("reports a runtime connection with no resolvable owner as an omission", async () => { + const root = await createTestTempDirectory("rea-ib-ownerless-"); + const bundle = join(root, "Example.app"); + await mkdir(bundle, { recursive: true }); + // The connector is recognized by class, but carries no UISource reference, + // so no connection can be projected from it. + await writeFile( + join(bundle, "Cell.nib"), + encodeNibArchiveFixture({ + classes: [ + "UIProxyObject", + "UIClassSwapper", + "UIButton", + "UIRuntimeEventConnection", + "NSString", + ], + objects: [ + { classIndex: 0, values: { UIProxiedObjectIdentifier: { ref: 5 } } }, + { classIndex: 1, values: { UIClassName: { ref: 6 } } }, + { classIndex: 2, values: {} }, + { + classIndex: 3, + values: { UIDestination: { ref: 1 }, UIEventMask: 64 }, + }, + { classIndex: 2, values: {} }, + { classIndex: 4, values: { "NS.bytes": "IBFilesOwner" } }, + { classIndex: 4, values: { "NS.bytes": "BuildCell" } }, + ], + }), + ); + + const analysis = await analyzeInterfaceBuilderBundle({ + bundlePath: bundle, + targetSha256: "e".repeat(64), + }); + + expect(analysis.graph.coverage).toContainEqual( + expect.objectContaining({ + facet: "connections:Cell.nib", + status: "partial", + reason: "connection_unresolved", + examined: 0, + omitted: 1, + }), + ); + // The keyed-archive producer counts a recognized connector it cannot emit, + // so this producer has to report the same document total for the same + // connector rather than zero. + expect(analysis.documents).toContainEqual( + expect.objectContaining({ + relative_path: "Cell.nib", + connection_count: 1, + }), + ); + expect(analysis.graph.truncated).toBe(true); + expect(analysis.limitations).toContain( + "Some recognized NIBArchive connectors declare no resolvable owner; they carry no connection evidence and are reported as omissions.", + ); + }); +}); + +describe("NIBArchive ownerless connector accounting", () => { + it("keeps a connection dropped by the limit reported as a limit", async () => { + const root = await createTestTempDirectory("rea-ib-ownerless-limit-"); + const bundle = join(root, "Example.app"); + await mkdir(bundle, { recursive: true }); + // Two resolvable connectors and one ownerless connector with the limit + // below the resolvable count, so this facet has both causes at once. Only a + // higher limit recovers the dropped connection, so that is the reason the + // caller needs, exactly as the keyed-archive producer reports it. + await writeFile( + join(bundle, "Cell.nib"), + encodeNibArchiveFixture({ + classes: [ + "UIProxyObject", + "UIClassSwapper", + "UIButton", + "UIRuntimeEventConnection", + "NSString", + ], + objects: [ + { classIndex: 0, values: { UIProxiedObjectIdentifier: { ref: 5 } } }, + { classIndex: 1, values: { UIClassName: { ref: 6 } } }, + { classIndex: 2, values: {} }, + { + classIndex: 3, + values: { + UISource: { ref: 2 }, + UIDestination: { ref: 1 }, + UILabel: { ref: 7 }, + UIEventMask: 64, + }, + }, + { classIndex: 2, values: {} }, + { classIndex: 4, values: { "NS.bytes": "IBFilesOwner" } }, + { classIndex: 4, values: { "NS.bytes": "BuildCell" } }, + { classIndex: 4, values: { "NS.bytes": "buildTapped:" } }, + { + classIndex: 3, + values: { + UISource: { ref: 4 }, + UIDestination: { ref: 1 }, + UILabel: { ref: 7 }, + }, + }, + { + classIndex: 3, + values: { UIDestination: { ref: 1 }, UIEventMask: 64 }, + }, + ], + }), + ); + + const analysis = await analyzeInterfaceBuilderBundle({ + bundlePath: bundle, + targetSha256: "e".repeat(64), + limits: { max_connections: 1 }, + }); + + expect(analysis.graph.coverage).toContainEqual( + expect.objectContaining({ + facet: "connections:Cell.nib", + status: "partial", + reason: "connection_limit_reached", + examined: 1, + omitted: 2, + }), + ); + expect(analysis.documents).toContainEqual( + expect.objectContaining({ + relative_path: "Cell.nib", + connection_count: 3, + }), + ); + }); }); // This exact file is already selected by the unchanged native Apple CI lane. diff --git a/tests/boundary/filesystem/javascript-applications/javascriptArtifactReconstruction.test.ts b/tests/boundary/filesystem/javascript-applications/javascriptArtifactReconstruction.test.ts index 6b7253b6d..416337097 100644 --- a/tests/boundary/filesystem/javascript-applications/javascriptArtifactReconstruction.test.ts +++ b/tests/boundary/filesystem/javascript-applications/javascriptArtifactReconstruction.test.ts @@ -388,7 +388,7 @@ it("produces deterministic ASAR graphs with unpacked native linkage and complete await mkdir(source); await writeJavaScriptArtifactFixture(source); const archive = join(root, "app.asar"); - await createPackageWithOptions(source, archive, { unpack: "**/*.node" }); + await createPackageWithOptions(source, archive, { unpack: "*.node" }); const first = await reconstructJavaScriptArtifact({ input_path: archive, @@ -487,7 +487,7 @@ it("keeps ASAR analysis usable when unpacked native companion bytes are absent", await mkdir(source); await writeJavaScriptArtifactFixture(source); const archive = join(root, "app.asar"); - await createPackageWithOptions(source, archive, { unpack: "**/*.node" }); + await createPackageWithOptions(source, archive, { unpack: "*.node" }); await rm(join(`${archive}.unpacked`, "native", "addon.node")); const snapshot = await scanArtifactInventory(archive); @@ -515,7 +515,7 @@ it("recurses into filesystem-backed ASAR containers without losing container-rel const resources = join(outer, "resources"); await mkdir(resources, { recursive: true }); await createPackageWithOptions(source, join(resources, "app.asar"), { - unpack: "**/*.node", + unpack: "*.node", }); const result = await reconstructJavaScriptArtifact({ diff --git a/tests/boundary/filesystem/javascript-applications/javascriptModuleRelationships.test.ts b/tests/boundary/filesystem/javascript-applications/javascriptModuleRelationships.test.ts index be51b9bb4..cdd95bc97 100644 --- a/tests/boundary/filesystem/javascript-applications/javascriptModuleRelationships.test.ts +++ b/tests/boundary/filesystem/javascript-applications/javascriptModuleRelationships.test.ts @@ -15,6 +15,60 @@ import { } from "../../../support/javascriptApplicationFixture.js"; describe("TypeScript source module loader semantics", () => { + it("keeps whole type-only declarations out of runtime module relationships", async () => { + const root = await createTestTempDirectory("rea-type-module-links-"); + await writeFixtureFiles(root, { + "main.ts": ` + import type { Whole } from "./whole"; + export type { Remote } from "./remote"; + import { type Inline } from "./inline"; + export type LocalType = string; + export interface LocalInterface { value: string } + `, + "whole.ts": "export interface Whole { value: string }", + "remote.ts": "export interface Remote { value: string }", + "inline.ts": "export interface Inline { value: string }", + }); + + const { graph } = await reconstructJavaScriptFixture(root); + const source = findSourceModule(graph, "main.ts"); + const sourceProperties = source?.observations.find( + ({ properties }) => properties.semantic_role === "source-module", + )?.properties; + + expect(sourceProperties).toMatchObject({ + import_relationships: 0, + type_relationships: 5, + export_names: [], + type_export_names: ["LocalInterface", "LocalType", "Remote"], + }); + expect( + graph.edges.filter( + ({ relation, properties }) => + relation === "imports" && + ["./whole", "./remote"].includes(String(properties.specifier)), + ), + ).toEqual([]); + expect( + graph.edges.find( + ({ source_node_id, relation, properties }) => + source_node_id === source?.node_id && + relation === "imports" && + properties.kind === "static-import" && + properties.specifier === "./inline", + ), + ).toBeDefined(); + expect( + source?.observations + .flatMap(({ evidence }) => evidence.limitations) + .join(" "), + ).toContain( + "inline type-only imports or re-exports may preserve a module side-effect dependency", + ); + expect(findExportNode(graph, "main.ts", "LocalType")).toBeUndefined(); + expect(findExportNode(graph, "main.ts", "LocalInterface")).toBeUndefined(); + }); + it("uses emitted TypeScript loader semantics without changing dynamic imports", async () => { const root = await createTestTempDirectory("rea-cts-module-loader-"); await writeFixtureFiles(root, { @@ -98,7 +152,7 @@ describe("TypeScript source module loader semantics", () => { }); }); -describe("CommonJS and ESM module relationships", () => { +describe("JavaScript module relationships", () => { it("composes bindings, re-exports, dynamic imports, and JSON modules", async () => { const root = await moduleFixture(); const first = await reconstructJavaScriptFixture(root); @@ -234,6 +288,122 @@ describe("CommonJS and ESM module relationships", () => { }); }); +describe("Compiler-dependent TypeScript resolution", () => { + it("retains only compiler-independent targets for .ts static imports", async () => { + const root = await createTestTempDirectory("rea-ts-conditional-loader-"); + await writeFixtureFiles(root, { + "package.json": '{"type":"commonjs"}', + "main.ts": `import { value as different } from "different"; +import { value as same } from "same"; +void different; void same;`, + "node_modules/different/package.json": JSON.stringify({ + exports: { + ".": { + import: "./import.mjs", + require: "./require.cjs", + }, + }, + }), + "node_modules/different/import.mjs": "export const value = 'import';", + "node_modules/different/require.cjs": "exports.value = 'require';", + "node_modules/same/package.json": JSON.stringify({ + exports: { ".": { import: "./shared.js", require: "./shared.js" } }, + }), + "node_modules/same/shared.js": "export const value = 'shared';", + }); + + const { graph } = await reconstructJavaScriptFixture(root); + const different = findRelationshipEdge(graph, { + kind: "import", + specifier: "different", + localName: "different", + }); + const same = findRelationshipEdge(graph, { + kind: "import", + specifier: "same", + localName: "same", + resolvedPath: "node_modules/same/shared.js", + }); + const staticFinding = (specifier: string) => + graph.edges.find( + ({ relation, properties }) => + relation === "imports" && + properties.kind === "static-import" && + properties.specifier === specifier, + ); + + expect( + findSourceModule(graph, "main.ts")?.observations[0]?.properties, + ).toMatchObject({ module_format: "unknown" }); + expect(different).toMatchObject({ + properties: { + resolution_mode: "compiler-dependent", + resolution_status: "unavailable", + resolved_path: null, + conditional_resolutions: [ + { + mode: "import", + resolution_status: "resolved", + resolved_path: "node_modules/different/import.mjs", + }, + { + mode: "require", + resolution_status: "resolved", + resolved_path: "node_modules/different/require.cjs", + }, + ], + }, + }); + expect(same).toMatchObject({ + properties: { + resolution_mode: "compiler-dependent", + resolved_path: "node_modules/same/shared.js", + resolution_status: "resolved", + }, + evidence: { + limitations: expect.arrayContaining([ + expect.stringContaining( + "compiler-dependent without retained compiler options", + ), + ]), + }, + }); + expect(staticFinding("different")).toMatchObject({ + properties: { + resolution_mode: "compiler-dependent", + resolution_status: "unavailable", + conditional_resolutions: [ + { + mode: "import", + resolved_path: "node_modules/different/import.mjs", + }, + { + mode: "require", + resolved_path: "node_modules/different/require.cjs", + }, + ], + }, + evidence: { + limitations: expect.arrayContaining([ + expect.stringContaining( + "compiler-dependent without retained compiler options", + ), + ]), + }, + }); + expect(staticFinding("same")).toMatchObject({ + properties: { resolved_path: "node_modules/same/shared.js" }, + evidence: { + limitations: expect.arrayContaining([ + expect.stringContaining( + "compiler-dependent without retained compiler options", + ), + ]), + }, + }); + }); +}); + it("does not assign one module origin to an export with colliding path encodings", async () => { const root = await createTestTempDirectory("rea-ambiguous-module-origin-"); await writeFixtureFiles(root, { diff --git a/tests/boundary/filesystem/javascriptApplicationWorkflows.test.ts b/tests/boundary/filesystem/javascriptApplicationWorkflows.test.ts index 17c3f0048..8393bad54 100644 --- a/tests/boundary/filesystem/javascriptApplicationWorkflows.test.ts +++ b/tests/boundary/filesystem/javascriptApplicationWorkflows.test.ts @@ -21,6 +21,7 @@ import { type ApplicationVersionComparisonResult, } from "../../../src/domain/javascript/javascriptApplicationVersionComparisonSchemas.js"; import { javascriptRuntimeReconciliationResultSchema } from "../../../src/domain/javascript/javascriptRuntimeReconciliationSchemas.js"; +import { electronPageInspectionSchema } from "../../../src/domain/javascript/electronObservation.js"; import { compareJavaScriptApplicationVersions } from "../../../src/domain/javascript/javascriptApplicationVersionComparison.js"; import { createJavaScriptApplicationGraph } from "../../../src/domain/javascript/javascriptApplicationGraph.js"; import { traceApplicationFeature } from "../../../src/domain/javascript/javascriptFeatureTrace.js"; @@ -443,7 +444,98 @@ describe("JavaScript application workflow error diagnostics", () => { }); }); +const RUNTIME_PATH_CASES = [ + { + name: "case-sensitive POSIX input root", + root: null, + path: "/applications/Example.app/Contents/Resources/app/renderer.js", + status: "unknown", + basis: "none", + }, + { + name: "case-sensitive POSIX mapping root", + root: "/runtime/App", + path: "/runtime/app/renderer.js", + status: "unknown", + basis: "none", + }, + { + name: "matching POSIX mapping root", + root: "/runtime/App", + path: "/runtime/App/renderer.js", + status: "matched", + basis: "operator-file-mapping", + }, + { + name: "case-insensitive Windows mapping root", + root: String.raw`C:\runtime\App`, + path: String.raw`c:\runtime\app\renderer.js`, + status: "matched", + basis: "operator-file-mapping", + }, +] as const; + describe("JavaScript runtime reconciliation local paths", () => { + it.each(RUNTIME_PATH_CASES)( + "uses $name on every host", + ({ root, path, status, basis }) => { + const base = JAVASCRIPT_RUNTIME_RECONCILIATION_EXAMPLE; + const original = base.runtime_observations[0]!; + const inspection = electronPageInspectionSchema.parse( + original.normalized_result, + ); + // Keep target and script coordinates consistent so an unrelated target + // cannot activate the static layer after its script mapping was rejected. + const runtime = createEvidence(undefined, original.provider, { + predicateType: original.predicate_type, + operation: original.operation, + parameters: { ...original.parameters, include_script_sources: false }, + result: { + ...inspection, + target: { ...inspection.target, file_path: path }, + scripts: { + ...inspection.scripts, + items: inspection.scripts.items.map((script) => ({ + ...script, + file_path: path, + source: { included: false, reason: "not-requested" }, + })), + }, + }, + confidence: original.confidence, + authority: original.authority, + }); + const reconciled = reconcileJavaScriptRuntimeEvidence({ + static_layers: base.static_layers.map((layer) => ({ + ...layer, + runtime_mappings: root === null ? [] : [{ kind: "file-root", root }], + })), + runtime_observations: [runtime], + }); + if (!reconciled.ok) throw reconciled.error; + const result = javascriptRuntimeReconciliationResultSchema.parse( + reconciled.value.normalized_result, + ); + expect( + result.reconciliations.find( + ({ entity_kind }) => entity_kind === "script", + ), + ).toMatchObject({ status, basis }); + expect( + result.static_load_states.find( + ({ kind }) => kind === "javascript-asset", + ), + ).toMatchObject( + status === "matched" + ? { status: "loaded", reason: "runtime-script-correspondence" } + : { status: "unknown", reason: "layer-outside-runtime-scope" }, + ); + expect(result.static_layers[0]?.runtime_mappings).toEqual( + root === null ? [] : [{ kind: "file-root", root, artifact_prefix: "" }], + ); + }, + ); + it("resolves relative file mapping roots before deriving result Evidence", () => { const absoluteRoot = resolve(process.cwd(), "runtime-cache-fixture"); const relativeRoot = relative(process.cwd(), absoluteRoot); diff --git a/tests/boundary/filesystem/javascriptIntegrityPolicy.test.ts b/tests/boundary/filesystem/javascriptIntegrityPolicy.test.ts index 47b77114a..67b167bdf 100644 --- a/tests/boundary/filesystem/javascriptIntegrityPolicy.test.ts +++ b/tests/boundary/filesystem/javascriptIntegrityPolicy.test.ts @@ -105,6 +105,34 @@ it("keeps a contradicted nested ASAR opaque during JavaScript reconstruction", a expect(result.graph.coverage.omitted_count).toBeNull(); }); +it("keeps an ASAR-named file without an ASAR header opaque during JavaScript reconstruction", async () => { + const root = await createTestTempDirectory("rea-opaque-asar-name-"); + await writeFile(join(root, "main.js"), "export const outer = true;\n"); + // Inventory records this file as ordinary bytes and does not expand it. + await writeFile(join(root, "ordinary.asar"), "ordinary bytes"); + + const snapshot = await scanCanonicalArtifactInventory(root, {}); + const reader = createJavaScriptArtifactReader(root, "directory"); + try { + const read = await readJavaScriptArtifactFiles(reader, snapshot); + expect(read.files.map(({ path }) => path)).toEqual(["main.js"]); + expect(read.containers).toEqual([]); + } finally { + await reader.close(); + } + + const analyzed = await analyzeJavaScriptApplication({ + input_path: root, + format: "directory", + }); + if (!analyzed.ok) throw analyzed.error; + const result = javascriptApplicationAnalysisResultSchema.parse( + analyzed.value.normalized_result, + ); + expect(result.statistics.nested_asar_containers).toBe(0); + expect(result.integrity_contradictions).toEqual([]); +}); + it("retains semantic resource limits alongside recorded integrity contradictions", async () => { const { archive } = await createStrippedAsarAddon([ { path: "app.js", contents: primitiveCandidateExpansionSource() }, diff --git a/tests/boundary/filesystem/rawBinaryTarget.test.ts b/tests/boundary/filesystem/rawBinaryTarget.test.ts index 5fd171d20..2757f0507 100644 --- a/tests/boundary/filesystem/rawBinaryTarget.test.ts +++ b/tests/boundary/filesystem/rawBinaryTarget.test.ts @@ -1,5 +1,11 @@ import { createHash } from "node:crypto"; -import { readFile, writeFile } from "node:fs/promises"; +import { + mkdir, + readFile, + realpath, + symlink, + writeFile, +} from "node:fs/promises"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; @@ -7,6 +13,37 @@ import { parseBinaryTarget } from "../../../src/application/BinaryTargetResolver import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; describe("raw-binary admission", () => { + it("preserves the requested raw path for Windows reparse admission", async () => { + const directory = await createTestTempDirectory("rea-raw-linked-"); + const actual = join(directory, "actual"); + const linked = join(directory, "linked"); + await mkdir(actual); + const bytes = Buffer.from("a90160", "hex"); + await writeFile(join(actual, "fixture.bin"), bytes); + await symlink( + actual, + linked, + process.platform === "win32" ? "junction" : "dir", + ); + const requested = join(linked, "fixture.bin"); + const canonical = await realpath(requested); + const target = await parseBinaryTarget("linked/fixture.bin", { + cwd: directory, + formatHint: "raw-binary", + architecture: "6502", + entryPoint: "0x8000", + }); + expect(target).toMatchObject({ + ok: true, + value: { + path: canonical, + sourcePath: process.platform === "win32" ? requested : canonical, + sha256: createHash("sha256").update(bytes).digest("hex"), + format: "raw-binary", + architecture: "6502", + }, + }); + }); it("requires explicit architecture and entry even with a misleading suffix", async () => { const directory = await createTestTempDirectory("rea-raw-target-"); const path = join(directory, "fixture.js"); diff --git a/tests/boundary/filesystem/referenceSourceImport.test.ts b/tests/boundary/filesystem/referenceSourceImport.test.ts index 75eab7acb..8443dfbdd 100644 --- a/tests/boundary/filesystem/referenceSourceImport.test.ts +++ b/tests/boundary/filesystem/referenceSourceImport.test.ts @@ -378,6 +378,44 @@ describe("reference source path selection", () => { }); describe("reference source rooted module specifiers", () => { + it("keeps dot-prefixed bare specifiers external while resolving explicit relative paths", async () => { + const root = await createTestTempDirectory("rea-reference-dot-prefix-"); + await Promise.all([ + writeFile( + join(root, "main.js"), + 'import ".hidden.js"; import "..pkg.js"; import "./.hidden.js"; require(".hidden.js");\n', + ), + writeFile(join(root, ".hidden.js"), "export {};\n"), + writeFile(join(root, "..pkg.js"), "export {};\n"), + ]); + const result = await importTree(root); + if (!result.ok) throw result.error; + expect(result.value.relationships).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + to: ".hidden.js", + kind: "imports", + resolution: "external", + }), + expect.objectContaining({ + to: "..pkg.js", + kind: "imports", + resolution: "external", + }), + expect.objectContaining({ + to: ".hidden.js", + kind: "imports", + resolution: "internal", + }), + expect.objectContaining({ + to: ".hidden.js", + kind: "requires", + resolution: "external", + }), + ]), + ); + }); + it("does not rebase an absolute import onto a coincidentally matching source member", async () => { const root = await createTestTempDirectory("rea-reference-rooted-"); await mkdir(join(root, "src", "outside"), { recursive: true }); diff --git a/tests/boundary/filesystem/referenceSourceTypeScriptExtensions.test.ts b/tests/boundary/filesystem/referenceSourceTypeScriptExtensions.test.ts index d537a1dca..7b3d8189c 100644 --- a/tests/boundary/filesystem/referenceSourceTypeScriptExtensions.test.ts +++ b/tests/boundary/filesystem/referenceSourceTypeScriptExtensions.test.ts @@ -6,6 +6,31 @@ import { expect, it } from "vitest"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { importReferenceSource } from "../../support/referenceSourceResourceScope.js"; +it("imports valid CommonJS wrapper syntax without inventing a parse failure", async () => { + const root = await createTestTempDirectory("rea-reference-commonjs-mode-"); + await Promise.all([ + writeFile( + join(root, "main.cjs"), + 'void new.target; require("./dep.cjs"); return;\n', + ), + writeFile(join(root, "dep.cjs"), "module.exports = 1;\n"), + ]); + const imported = await importReferenceSource({ + root, + caller: "native-commonjs-source", + policy: { secretPatterns: [] }, + }); + if (!imported.ok) throw imported.error; + expect(imported.value.parse_failures).toEqual([]); + expect(imported.value.relationships).toContainEqual({ + from_path: "main.cjs", + to: "dep.cjs", + kind: "requires", + resolution: "internal", + parse_state: "parsed", + }); +}); + it.each([ ["main.ts", "dep.js", "dep.ts"], ["main.mts", "dep.mjs", "dep.mts"], diff --git a/tests/boundary/filesystem/releaseCheckpoint.test.ts b/tests/boundary/filesystem/releaseCheckpoint.test.ts index e3d425810..d511a6a58 100644 --- a/tests/boundary/filesystem/releaseCheckpoint.test.ts +++ b/tests/boundary/filesystem/releaseCheckpoint.test.ts @@ -649,9 +649,8 @@ it.skipIf(process.platform === "win32").each([ const sourceSha = await candidate(f, valid ? "6.0.0" : "5.1.0"); const workflow = z .object({ - jobs: z.record( - z.string(), - z.object({ + jobs: z.object({ + [job]: z.object({ steps: z.array( z.object({ name: z.string().optional(), @@ -659,7 +658,7 @@ it.skipIf(process.platform === "win32").each([ }), ), }), - ), + }), }) .parse( parse( diff --git a/tests/boundary/filesystem/releaseWorkflow.test.ts b/tests/boundary/filesystem/releaseWorkflow.test.ts index 0bfebd1d2..4d3859433 100644 --- a/tests/boundary/filesystem/releaseWorkflow.test.ts +++ b/tests/boundary/filesystem/releaseWorkflow.test.ts @@ -34,6 +34,12 @@ async function readReleaseWorkflow() { "release-proposal": jobSchema, "release-please": jobSchema, "merged-release": jobSchema, + qualification: z.object({ + if: z.string(), + needs: z.array(z.string()), + uses: z.string(), + with: z.record(z.string(), z.string()), + }), publish: jobSchema, "publish-mcp": jobSchema, }), @@ -271,10 +277,19 @@ it("binds npm and MCP publication to the same immutable release SHA", async () = release_created: "${{ steps.release.outputs.release_created }}", sha: "${{ steps.release.outputs.sha }}", }); - expect(workflow.jobs.publish.if).toContain("inputs.phase == 'publish'"); - expect(workflow.jobs.publish.if).toContain( + expect(workflow.jobs.qualification.if).toContain("inputs.phase == 'publish'"); + expect(workflow.jobs.qualification.if).toContain( "needs.release-please.outputs.release_created", ); + expect(workflow.jobs.qualification.uses).toBe("./.github/workflows/ci.yml"); + expect(workflow.jobs.qualification.with).toEqual({ + source_sha: + "${{ needs.merged-release.outputs.sha || needs.release-please.outputs.sha }}", + }); + expect(workflow.jobs.publish.needs).toContain("qualification"); + expect(workflow.jobs.publish.if).toContain( + "needs.qualification.result == 'success'", + ); expect(workflow.jobs["publish-mcp"].if).toContain( "needs.publish.result == 'success'", ); @@ -309,7 +324,7 @@ it("binds npm and MCP publication to the same immutable release SHA", async () = )?.run; expect(publishCommand).toContain("scripts/release-npm-tag.mjs"); expect(publishCommand).toContain( - 'npm publish --access public --tag "${tag}"', + 'npm publish "${VERIFIED_PACKAGE_TARBALL}" --access public --tag "${tag}"', ); for (const [version, tag] of [ ["6.1.0", "latest"], @@ -644,7 +659,7 @@ it("routes website verification through the central PR policy", async () => { // Publishing is irreversible. Keep the release authority invariant as a static // boundary check; it is not proof that package installation or publishing works. -it("requires package verification before publishing and retains the registry canary", async () => { +it("requires qualified package validation before publishing and retains the registry canary", async () => { const workflow = z .object({ jobs: z.object({ @@ -663,7 +678,7 @@ it("requires package verification before publishing and retains the registry can ); const commands = workflow.jobs.publish.steps.map(({ run }) => run ?? ""); const verify = commands.findIndex((command) => - command.includes("npm run verify:package"), + command.includes("scripts/ci/verified-package.mjs validate"), ); const publish = commands.findIndex((command) => command.includes("npm publish"), diff --git a/tests/boundary/filesystem/safeOutputTree.test.ts b/tests/boundary/filesystem/safeOutputTree.test.ts index f67704456..3e82a62ab 100644 --- a/tests/boundary/filesystem/safeOutputTree.test.ts +++ b/tests/boundary/filesystem/safeOutputTree.test.ts @@ -489,4 +489,28 @@ describe("safe artifact output Windows stream paths", () => { await expect(access(output)).rejects.toMatchObject({ code: "ENOENT" }); expect(await readdir(parent)).toEqual([]); }); + + it.each(["Folder/Icon\r", "Fold\ter/file.txt"])( + "refuses Windows control characters before creating %j", + async (path) => { + const parent = await createTestTempDirectory("rea-safe-output-control-"); + const output = join(parent, "published"); + const tree = await SafeOutputTree.create(output, "win32"); + const bytes = Buffer.from("regular file content"); + await expect( + tree.write(path, Readable.from(bytes), { + sha256: createHash("sha256").update(bytes).digest("hex"), + bytes: bytes.byteLength, + }), + ).rejects.toMatchObject({ + reason: "path", + message: `Windows destination cannot materialize artifact path: ${JSON.stringify(path)}; Windows names cannot contain U+0001-U+001F.`, + }); + expect(await readdir(output)).toEqual([]); + expect(await tree.rollback()).toEqual({ + status: "complete", + residualPaths: [], + }); + }, + ); }); diff --git a/tests/boundary/filesystem/safeOutputTreeCloseFailures.test.ts b/tests/boundary/filesystem/safeOutputTreeCloseFailures.test.ts index ee285dbd5..d835b1e46 100644 --- a/tests/boundary/filesystem/safeOutputTreeCloseFailures.test.ts +++ b/tests/boundary/filesystem/safeOutputTreeCloseFailures.test.ts @@ -10,6 +10,8 @@ const injectedFilesystemFailures = vi.hoisted(() => ({ closeNext: false, blockCloseRetries: false, chmodNext: false, + syncAbortAfter: undefined as AbortController | undefined, + syncCount: 0, rootIdentityPath: undefined as string | undefined, })); @@ -44,6 +46,15 @@ vi.mock("node:fs/promises", async (importOriginal) => { await close(); }; } + if (injectedFilesystemFailures.syncAbortAfter !== undefined) { + const sync = handle.sync.bind(handle); + handle.sync = async () => { + await sync(); + injectedFilesystemFailures.syncCount += 1; + if (injectedFilesystemFailures.syncCount === 2) + injectedFilesystemFailures.syncAbortAfter?.abort(); + }; + } return handle; }, }; @@ -82,6 +93,34 @@ describe("safe artifact output close failures", () => { }, ); + it.skipIf(process.platform === "win32")( + "keeps the tree unpublished when cancellation arrives during commit durability", + async () => { + const parent = await createTestTempDirectory("rea-safe-output-cancel-"); + const output = join(parent, "unpublished"); + const tree = await SafeOutputTree.create(output); + const bytes = Buffer.from("pending output"); + await tree.write("file.txt", Readable.from(bytes), { + sha256: createHash("sha256").update(bytes).digest("hex"), + bytes: bytes.byteLength, + }); + const controller = new AbortController(); + injectedFilesystemFailures.syncCount = 0; + injectedFilesystemFailures.syncAbortAfter = controller; + try { + await expect(tree.commit(controller.signal)).rejects.toMatchObject({ + reason: "cancelled", + }); + expect(tree.published).toBe(false); + } finally { + injectedFilesystemFailures.syncAbortAfter = undefined; + injectedFilesystemFailures.syncCount = 0; + } + expect(await tree.rollback()).toMatchObject({ status: "complete" }); + await expect(access(output)).rejects.toMatchObject({ code: "ENOENT" }); + }, + ); + it.skipIf(process.platform === "win32")( "preserves setup failure when closing the staging handle also fails", async () => { diff --git a/tests/boundary/filesystem/setupClientConfigCleanupOwnership.test.ts b/tests/boundary/filesystem/setupClientConfigCleanupOwnership.test.ts index 3bf33c312..ab2b5e5a7 100644 --- a/tests/boundary/filesystem/setupClientConfigCleanupOwnership.test.ts +++ b/tests/boundary/filesystem/setupClientConfigCleanupOwnership.test.ts @@ -11,6 +11,7 @@ import { } from "../../../src/application/SetupHost.js"; import { RegularFileCleanupFailure } from "../../../src/application/RegularFileRead.js"; import { configureDetectedClients } from "../../../src/application/SetupClients.js"; +import { systemUninstallHost } from "../../../src/application/Uninstall.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; vi.mock("node:fs/promises", async (importOriginal) => { @@ -31,6 +32,61 @@ beforeEach(async () => { afterEach(() => openMock.mockReset()); +const holdReadHandle = (path: string, reason: string) => { + const closeAttempts: object[] = []; + let released = false; + openMock.mockImplementation(async (...args) => { + const handle = await actualFs.open(...args); + if (String(args[0]) !== path) return handle; + const nativeClose = handle.close.bind(handle); + vi.spyOn(handle, "close").mockImplementation(async () => { + closeAttempts.push(handle); + if (!released) throw new Error(reason); + await nativeClose(); + }); + return handle; + }); + return { + closeAttempts, + release: () => { + released = true; + }, + }; +}; + +it("retains invalid UTF-8 diagnostics and the uninstall read owner when close fails", async () => { + const home = await createTestTempDirectory("rea-uninstall-utf8-close-"); + const path = join(home, "client.json"); + const original = Buffer.from([0xff]); + await writeFile(path, original); + const { closeAttempts, release } = holdReadHandle( + path, + "held uninstall handle", + ); + const host = systemUninstallHost(home, undefined, { HOME: home }, "linux"); + try { + expect( + await host.removeClient({ + name: "cursor", + format: "json", + configPath: path, + }), + ).toMatchObject({ + status: "failed", + detail: expect.stringContaining("not valid UTF-8"), + }); + expect(await readFile(path)).toEqual(original); + expect(closeAttempts).toHaveLength(2); + release(); + expect(await host.close?.()).toBeUndefined(); + expect(closeAttempts).toHaveLength(3); + expect(closeAttempts[2]).toBe(closeAttempts[0]); + } finally { + release(); + await host.close?.(); + } +}); + describe("ZCode fallback cleanup ownership", () => { it.each(["preflight", "configure"])( "retains a failed ZCode fallback close during %s until host cleanup", @@ -49,19 +105,10 @@ describe("ZCode fallback cleanup ownership", () => { await mkdir(dirname(fallback), { recursive: true }); const original = '{"mcpServers":{"other":{"command":"other"}}}\n'; await writeFile(fallback, original); - const closeAttempts: object[] = []; - let allowClose = false; - openMock.mockImplementation(async (...args) => { - const handle = await actualFs.open(...args); - if (String(args[0]) !== fallback) return handle; - const nativeClose = handle.close.bind(handle); - vi.spyOn(handle, "close").mockImplementation(async () => { - closeAttempts.push(handle); - if (!allowClose) throw new Error("held fallback handle"); - await nativeClose(); - }); - return handle; - }); + const { closeAttempts, release } = holdReadHandle( + fallback, + "held fallback handle", + ); try { const action = operation === "preflight" @@ -79,12 +126,12 @@ describe("ZCode fallback cleanup ownership", () => { await expect(readFile(client.configPath)).rejects.toMatchObject({ code: "ENOENT", }); - allowClose = true; + release(); expect(await host.close?.()).toBeUndefined(); expect(closeAttempts).toHaveLength(3); expect(closeAttempts[2]).toBe(closeAttempts[0]); } finally { - allowClose = true; + release(); await host.close?.(); } }, @@ -92,83 +139,87 @@ describe("ZCode fallback cleanup ownership", () => { }); describe("setup client configuration cleanup ownership", () => { - it("stops client configuration and retries the same alignment handle from host close", async () => { - const home = await createTestTempDirectory("rea-setup-config-close-"); - const environment = { - HOME: home, - XDG_CONFIG_HOME: join(home, "xdg"), - }; - const host = systemSetupHost( - { platform: "linux", homeDirectory: home } as DoctorHost, - environment, - ); - const client = (await host.supportedClients()).find( - ({ name }) => name === "vscode", - ); - if (client === undefined) throw new Error("VS Code setup client missing"); - await mkdir(dirname(client.configPath), { recursive: true }); - const original = Buffer.from('{"servers": {"other": {}}}\n'); - await writeFile(client.configPath, original); + it.each([ + ["valid UTF-8", Buffer.from('{"servers": {"other": {}}}\n')], + ["malformed UTF-8", Buffer.from([0xff])], + ] as const)( + "stops configuration and retains the %s read owner until host close", + async (encoding, original) => { + const home = await createTestTempDirectory("rea-setup-config-close-"); + const environment = { + HOME: home, + XDG_CONFIG_HOME: join(home, "xdg"), + }; + const host = systemSetupHost( + { platform: "linux", homeDirectory: home } as DoctorHost, + environment, + ); + const client = (await host.supportedClients()).find( + ({ name }) => name === "vscode", + ); + if (client === undefined) throw new Error("VS Code setup client missing"); + await mkdir(dirname(client.configPath), { recursive: true }); + await writeFile(client.configPath, original); - const closeAttempts: object[] = []; - let allowClose = false; - openMock.mockImplementation(async (...args) => { - const handle = await actualFs.open(...args); - if (String(args[0]) !== client.configPath) return handle; - const nativeClose = handle.close.bind(handle); - vi.spyOn(handle, "close").mockImplementation(async () => { - closeAttempts.push(handle); - if (!allowClose) throw new Error("test holds the config handle open"); - await nativeClose(); - }); - return handle; - }); - const configure = vi.spyOn(host, "configureClient"); + const { closeAttempts, release } = holdReadHandle( + client.configPath, + "test holds the config handle open", + ); + const configure = vi.spyOn(host, "configureClient"); - try { - const failure = await filterClientsNeedingConfigure(host, [client], {}, [ - "rea", - "mcp", - ]) - .then((detectedClients) => - configureDetectedClients({ - host, - detectedClients, - providerEnvironment: {}, - command: ["rea", "mcp"], - clients: {}, - appliedActions: [], - }), + try { + const failure = await filterClientsNeedingConfigure( + host, + [client], + {}, + ["rea", "mcp"], ) - .then( - () => undefined, - (cause: unknown) => cause, - ); - expect(failure).toBeInstanceOf(ConfigurationError); - expect(failure).toMatchObject({ - _tag: "ConfigurationError", - cause: expect.any(RegularFileCleanupFailure), - cleanup: { - resources: [client.configPath], - reason: expect.stringContaining("test holds the config handle open"), - }, - }); - expect(configure).not.toHaveBeenCalled(); - expect(closeAttempts).toHaveLength(2); - expect(closeAttempts[0]).toBe(closeAttempts[1]); - expect(await readFile(client.configPath)).toEqual(original); + .then((detectedClients) => + configureDetectedClients({ + host, + detectedClients, + providerEnvironment: {}, + command: ["rea", "mcp"], + clients: {}, + appliedActions: [], + }), + ) + .then( + () => undefined, + (cause: unknown) => cause, + ); + expect(failure).toBeInstanceOf(ConfigurationError); + if (encoding === "malformed UTF-8") + expect(failure).toMatchObject({ + cause: { + outcome: { kind: "failed", cause: expect.any(TypeError) }, + }, + }); + expect(failure).toMatchObject({ + _tag: "ConfigurationError", + cause: expect.any(RegularFileCleanupFailure), + cleanup: { + resources: [client.configPath], + reason: expect.stringContaining( + "test holds the config handle open", + ), + }, + }); + expect(configure).not.toHaveBeenCalled(); + expect(closeAttempts).toHaveLength(2); + expect(closeAttempts[0]).toBe(closeAttempts[1]); + expect(await readFile(client.configPath)).toEqual(original); - allowClose = true; - const close = host.close; - if (close === undefined) throw new Error("Setup host close is missing"); - expect(await close()).toBeUndefined(); - expect(closeAttempts).toHaveLength(3); - expect(closeAttempts[2]).toBe(closeAttempts[0]); - } finally { - allowClose = true; - await host.close?.(); - } - }); + release(); + expect(await host.close?.()).toBeUndefined(); + expect(closeAttempts).toHaveLength(3); + expect(closeAttempts[2]).toBe(closeAttempts[0]); + } finally { + release(); + await host.close?.(); + } + }, + ); it("waits for an admitted configuration read and rejects later admissions", async () => { const home = await createTestTempDirectory("rea-setup-config-inflight-"); diff --git a/tests/boundary/filesystem/sourceToBundlePaths.test.ts b/tests/boundary/filesystem/sourceToBundlePaths.test.ts index 19aa07a1a..8bc9caa27 100644 --- a/tests/boundary/filesystem/sourceToBundlePaths.test.ts +++ b/tests/boundary/filesystem/sourceToBundlePaths.test.ts @@ -6,7 +6,7 @@ import { analyzeJavaScriptApplication } from "../../support/javascriptApplicatio import { parseApplicationGraphEvidence } from "../../../src/application/javascript/JavaScriptApplicationEvidenceGraph.js"; import { findApplicationFeatureSeeds } from "../../../src/domain/javascript/javascriptFeatureSeed.js"; import { compareSourceToBundle } from "../../../src/domain/javascript/sourceToBundleComparison.js"; -import { resolveSourceMapSource } from "../../../src/javascript/sourceMaps/DecodedSourceMap.js"; +import { resolveJavaScriptSourceMapUrl } from "../../../src/domain/javascript/javascriptSourceMapPaths.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { importReferenceSource } from "../../support/referenceSourceResourceScope.js"; @@ -180,7 +180,7 @@ it.each([ decoy: decoyPath = "src/a.js", }) => { if (sourceRoot !== undefined) { - const resolved = resolveSourceMapSource( + const resolved = resolveJavaScriptSourceMapUrl( source, sourceRoot, `https://example.test/${mapPath}`, diff --git a/tests/boundary/filesystem/stableArtifact.test.ts b/tests/boundary/filesystem/stableArtifact.test.ts index 454d9064b..3bd587ad0 100644 --- a/tests/boundary/filesystem/stableArtifact.test.ts +++ b/tests/boundary/filesystem/stableArtifact.test.ts @@ -31,9 +31,12 @@ it("reads exact binary and empty bytes and rejects symlink/size/cancellation bou }); const controller = new AbortController(); controller.abort(); - await expect(readStableArtifact(real, 3, controller.signal)).rejects.toBe( - controller.signal.reason, - ); + await expect( + readStableArtifact(real, 3, controller.signal), + ).rejects.toMatchObject({ + reason: "cancelled", + cause: controller.signal.reason, + }); }); const checkChange = async ( diff --git a/tests/boundary/filesystem/staticCheckReports.test.ts b/tests/boundary/filesystem/staticCheckReports.test.ts index c2708e1db..7d022da72 100644 --- a/tests/boundary/filesystem/staticCheckReports.test.ts +++ b/tests/boundary/filesystem/staticCheckReports.test.ts @@ -1,6 +1,6 @@ import { spawnSync } from "node:child_process"; import { access, mkdir, readFile, writeFile } from "node:fs/promises"; -import { join, sep } from "node:path"; +import { dirname, join, sep } from "node:path"; import { fileURLToPath } from "node:url"; import { expect, it } from "vitest"; import { z } from "zod"; @@ -138,6 +138,36 @@ it("keeps lint errors failing and visible alongside the retained warnings", asyn ); }); +it("excludes translation staging while retaining authored website errors", async () => { + const root = await fixture(); + for (const name of [".gitignore", ".oxlintrc.json"]) + await writeFile(join(root, name), await readFile(join(repository, name))); + const authored = "website/public/assets/authored.js"; + const staged = + "website/.i18n-regression/site/examples/notes-electron/main.js"; + for (const name of [authored, staged]) { + const path = join(root, name); + await mkdir(dirname(path), { recursive: true }); + await writeFile( + path, + 'const source = require("node:fs");\nconsole.log(source);\n', + ); + } + + const result = invoke(root, "lint"); + expect(result.error).toBeUndefined(); + expect(result.status).toBe(1); + expect(result.stdout).toContain("1 errors, 0 warnings"); + expect(result.stderr).toContain(authored); + expect(await lintDiagnostics(root)).toEqual([ + expect.objectContaining({ + severity: "error", + code: "typescript(no-var-requires)", + filename: authored, + }), + ]); +}); + it("keeps malformed source failing with its parser diagnostic", async () => { const root = await fixture(); await writeFile(join(root, "src/malformed.js"), "const value = ;\n"); diff --git a/tests/boundary/filesystem/targetDiagnostics.test.ts b/tests/boundary/filesystem/targetDiagnostics.test.ts index 5d1953760..dacd21687 100644 --- a/tests/boundary/filesystem/targetDiagnostics.test.ts +++ b/tests/boundary/filesystem/targetDiagnostics.test.ts @@ -4,9 +4,35 @@ import { describe, expect, it } from "vitest"; import { systemDoctorHost } from "../../../src/application/Doctor.js"; import { parseBinaryTarget } from "../../../src/application/BinaryTargetResolver.js"; import { projectAnalysisError } from "../../../src/domain/analysisErrorProjection.js"; +import { pe } from "../../../src/domain/binaryTarget.fixture.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; describe("executable format refusal diagnostics", () => { + it.each([ + [0x01a6, "0x01a6"], + [0x0000, "0x0000"], + [0xffff, "0xffff"], + ] as const)( + "retains unsupported PE machine %i in the refusal", + async (machine, expectedMachine) => { + const directory = await createTestTempDirectory("rea-target-pe-machine-"); + const path = join(directory, "unsupported.exe"); + const bytes = pe(0x014c); + bytes.writeUInt16LE(machine, 0x44); + await writeFile(path, bytes); + + const result = await parseBinaryTarget(path); + if (result.ok) throw new Error("unsupported PE target was admitted"); + const reason = `unsupported PE architecture (COFF machine ${expectedMachine})`; + expect(projectAnalysisError(result.error)).toMatchObject({ + code: "target_unavailable", + category: "unsupported_target", + message: `Cannot open artifact: ${reason}.`, + details: { reason, constraint: "file_format_rejected" }, + }); + }, + ); + it.each(["NE", "LE", "LX"] as const)( "reports unsupported %s executables without path advice", async (signature) => { diff --git a/tests/boundary/filesystem/testTaskCache.test.ts b/tests/boundary/filesystem/testTaskCache.test.ts index 7e06ce1a1..5d877e829 100644 --- a/tests/boundary/filesystem/testTaskCache.test.ts +++ b/tests/boundary/filesystem/testTaskCache.test.ts @@ -1,6 +1,6 @@ import { execFile } from "node:child_process"; -import { readFile, writeFile } from "node:fs/promises"; -import { join } from "node:path"; +import { cp, readFile, writeFile } from "node:fs/promises"; +import { basename, join } from "node:path"; import { promisify } from "node:util"; import { expect, it } from "vitest"; @@ -78,7 +78,7 @@ it.skipIf(process.platform === "android")( await workflowGit(root, ["init", "--initial-branch=main"]); await workflowGit(root, ["add", "."]); await workflowGit(root, ["commit", "-m", "test: seed cache fixture"]); - const run = () => + const run = (cwd = root) => execute( process.execPath, [ @@ -87,7 +87,7 @@ it.skipIf(process.platform === "android")( "test:run", "--output-logs=errors-only", ], - { cwd: root, env: { ...process.env, TURBO_DAEMON: "false" } }, + { cwd, env: { ...process.env, TURBO_DAEMON: "false" } }, ); await run(); await writeFile(join(root, "README_zh.md"), "wrong command"); @@ -104,5 +104,32 @@ it.skipIf(process.platform === "android")( "prepared\n", ); } + // Provider jobs receive the producer's cache in a fresh checkout, not its + // outputs. Turbo must restore identical inputs and rebuild changed ones. + const consumer = await createTestTempDirectory("rea-cache-consumer-"); + await cp(root, consumer, { + recursive: true, + filter: (path) => + basename(path) !== ".turbo" && + !/^prepared-\d+\.txt$/u.test(basename(path)), + }); + await cp(join(root, ".turbo/cache"), join(consumer, ".turbo/cache"), { + recursive: true, + }); + const restored = await run(consumer); + expect(restored.stdout).toContain(`${dependencies.length} cached`); + for (const index of dependencies.keys()) + expect( + await readFile(join(consumer, `prepared-${index}.txt`), "utf8"), + ).toBe("prepared\n"); + await writeFile( + join(consumer, "artifact-input.txt"), + "changed producer input", + ); + await run(consumer); + for (const index of dependencies.keys()) + expect( + await readFile(join(consumer, `prepared-${index}.txt`), "utf8"), + ).toBe("prepared\nprepared\n"); }, ); diff --git a/tests/boundary/filesystem/thirdPartyInventory.test.ts b/tests/boundary/filesystem/thirdPartyInventory.test.ts new file mode 100644 index 000000000..7e61f883b --- /dev/null +++ b/tests/boundary/filesystem/thirdPartyInventory.test.ts @@ -0,0 +1,117 @@ +import { createHash } from "node:crypto"; +import { copyFile, mkdir, readFile, writeFile } from "node:fs/promises"; +import { resolve } from "node:path"; + +import { expect } from "vitest"; + +import { processTest as test } from "../../support/process/processFixture.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +test("generates locked inventory and rejects stale notice metadata or preserved texts", async ({ + processes, +}) => { + const root = await createTestTempDirectory("rea-notice-records-"); + await mkdir(resolve(root, "scripts/lib"), { recursive: true }); + await mkdir(resolve(root, "licenses")); + for (const file of [ + "scripts/generate-third-party-inventory.mjs", + "scripts/lib/generated-file.mjs", + ]) + await copyFile(resolve(file), resolve(root, file)); + const script = resolve(root, "scripts/generate-third-party-inventory.mjs"); + const runtime = { + version: "1.0.0", + license: "MIT", + resolved: + "https://registry.npmjs.org/fixture-runtime/-/fixture-runtime-1.0.0.tgz", + integrity: "sha512-fixture-integrity", + optional: true, + }; + await writeFile( + resolve(root, "package-lock.json"), + JSON.stringify({ + packages: { + "": { name: "fixture", version: "1.0.0" }, + "node_modules/fixture-runtime": runtime, + "node_modules/fixture-dev": { ...runtime, dev: true, optional: false }, + }, + }), + ); + const generated = await processes.run(process.execPath, [script]); + expect(generated.exitCode, generated.stderr).toBe(0); + const inventory = await readFile( + resolve(root, "licenses/npm-dependencies.txt"), + "utf8", + ); + expect(inventory).toContain("Locked package entries: 2; runtime: 1."); + expect(inventory).toContain( + "fixture-runtime@1.0.0 | runtime, optional | MIT", + ); + expect(inventory).toContain("fixture-dev@1.0.0 | development | MIT"); + const fingerprint = ( + JSON.parse(generated.stdout) as { runtime_dependency_fingerprint: string } + ).runtime_dependency_fingerprint; + const notice = [ + `Runtime dependency fingerprint (SHA-256): ${fingerprint}`, + "", + "===== fixture-runtime@1.0.0 =====", + "Declared license: MIT", + `Archive: ${runtime.resolved}`, + `Integrity: ${runtime.integrity}`, + "", + "Fixture text; this test does not establish real package attribution.", + "", + ].join("\n"); + const noticePath = resolve(root, "licenses/npm-runtime-notices.txt"); + const referencePath = resolve(root, "licenses/reference.txt"); + await writeFile(noticePath, notice); + await writeFile(referencePath, "preserved fixture text\n"); + await writeFile( + resolve(root, "licenses/upstream-sources.json"), + JSON.stringify([ + { + file: "licenses/reference.txt", + sha256: createHash("sha256") + .update("preserved fixture text\n") + .digest("hex"), + }, + ]), + ); + const check = () => processes.run(process.execPath, [script, "--check"]); + const valid = await check(); + expect(valid.exitCode, valid.stderr).toBe(0); + for (const [changed, diagnostic] of [ + [ + notice.replace( + "===== fixture-runtime@1.0.0 =====", + "===== another-package@1.0.0 =====", + ), + "Missing runtime notice", + ], + [ + notice.replace(runtime.integrity, "sha512-different"), + "Runtime notice metadata differs", + ], + [ + notice.replace(fingerprint, "0".repeat(64)), + "Runtime notice fingerprint differs", + ], + [ + notice + notice.slice(notice.indexOf("===== ")), + "Duplicate runtime notice", + ], + ] as const) { + await writeFile(noticePath, changed); + const rejected = await check(); + expect(rejected.exitCode).toBe(1); + expect(rejected.stderr).toContain(diagnostic); + } + await writeFile(noticePath, notice); + await writeFile(referencePath, "changed fixture text\n"); + const tampered = await check(); + expect(tampered.exitCode).toBe(1); + expect(tampered.stderr).toContain("Preserved notice digest differs"); + expect( + await readFile(resolve(root, "licenses/npm-dependencies.txt"), "utf8"), + ).toBe(inventory); +}); diff --git a/tests/boundary/filesystem/webArtifactReadOwnership.test.ts b/tests/boundary/filesystem/webArtifactReadOwnership.test.ts new file mode 100644 index 000000000..0a8791366 --- /dev/null +++ b/tests/boundary/filesystem/webArtifactReadOwnership.test.ts @@ -0,0 +1,96 @@ +import { mkdir, rm, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect, it, onTestFinished, vi } from "vitest"; + +const injected = vi.hoisted(() => ({ + sourcePath: undefined as string | undefined, + abortController: undefined as AbortController | undefined, + failClose: false, + closeAttempts: 0, +})); + +vi.mock("node:fs/promises", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + open: async (...args: Parameters) => { + const handle = await actual.open(...args); + if (args[0] !== injected.sourcePath) return handle; + const createReadStream = handle.createReadStream.bind(handle); + handle.createReadStream = (options) => { + injected.abortController?.abort(); + return createReadStream(options); + }; + const close = handle.close.bind(handle); + handle.close = async () => { + injected.closeAttempts += 1; + if (injected.failClose) throw new Error("fixture close refusal"); + await close(); + }; + return handle; + }, + }; +}); + +import { WebModuleTraceService } from "../../../src/application/WebModuleTraceService.js"; +import { LocalWebModuleArtifacts } from "../../../src/browser/modules/WebModuleArtifacts.js"; +import { webModuleArtifactsFixture } from "../../fixtures/webModuleTrace.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +it("retains cancelled source read ownership until service close retries it", async () => { + const root = await createTestTempDirectory("rea-web-artifact-read-owner-"); + onTestFinished(async () => { + injected.sourcePath = undefined; + injected.abortController = undefined; + injected.failClose = false; + injected.closeAttempts = 0; + await rm(root, { recursive: true, force: true }); + }); + const source = 'import "./dep.js";'; + const fixture = webModuleArtifactsFixture(source); + const manifestPath = join(root, "manifest.json"); + const sourcePath = join(root, "files", "modules", "main.js"); + await mkdir(join(root, "files", "modules"), { recursive: true }); + await writeFile(sourcePath, source); + await writeFile(manifestPath, JSON.stringify(fixture.manifest)); + + injected.sourcePath = sourcePath; + injected.failClose = true; + const controller = new AbortController(); + injected.abortController = controller; + const resolver = { + resolve: vi.fn(), + close: vi.fn(async () => undefined), + }; + const service = new WebModuleTraceService( + new LocalWebModuleArtifacts(), + resolver, + ); + const input = { manifest_path: manifestPath, script_index: 0 }; + + const cancelled = await service.trace(input, { signal: controller.signal }); + expect(cancelled.ok).toBe(false); + if (cancelled.ok) throw new Error("Expected cancelled source read"); + expect(cancelled.error.cleanupIncomplete).toBe(true); + expect(cancelled.error.cleanupResources).toContain(sourcePath); + expect(resolver.resolve).not.toHaveBeenCalled(); + + injected.abortController = undefined; + const blocked = await service.trace(input); + expect(blocked.ok).toBe(false); + if (blocked.ok) throw new Error("Expected retained cleanup to block load"); + expect(blocked.error.cleanupIncomplete).toBe(true); + expect(injected.closeAttempts).toBe(2); + + await expect(service.close()).rejects.toThrow( + "Web module trace cleanup failed", + ); + expect(resolver.close).toHaveBeenCalledTimes(1); + expect(injected.closeAttempts).toBe(3); + + injected.failClose = false; + const attemptsBeforeRetry = injected.closeAttempts; + await service.close(); + expect(injected.closeAttempts).toBeGreaterThan(attemptsBeforeRetry); + expect(resolver.close).toHaveBeenCalledTimes(2); +}); diff --git a/tests/boundary/go/goBinaryImage.test.ts b/tests/boundary/go/goBinaryImage.test.ts index 24321ecc8..29f0e952c 100644 --- a/tests/boundary/go/goBinaryImage.test.ts +++ b/tests/boundary/go/goBinaryImage.test.ts @@ -1,4 +1,10 @@ import { expect, it } from "vitest"; +import { javaClassFile } from "../../fixtures/javaClassFile.js"; +import { + CPU, + fatImage, + machoImage, +} from "../../../src/artifacts/apple/MachoImage.fixture.js"; import { readGoBinaryImage } from "../../../src/go/GoBinaryImage.js"; import { createGoBinaryFixture, @@ -221,6 +227,35 @@ it("distinguishes unsupported containers from malformed recognized images", () = expect(() => readGoBinaryImage(pe.bytes)).toThrowError(/range|truncated/i); }); +it("reports JVM class files as unsupported images instead of fat Mach-O containers", () => { + expect(() => readGoBinaryImage(javaClassFile())).toThrowError( + /Only ELF, PE, and thin Mach-O native images are supported/, + ); + expect(() => readGoBinaryImage(javaClassFile(4000))).toThrowError( + /Only ELF, PE, and thin Mach-O native images are supported/, + ); + expect(() => readGoBinaryImage(javaClassFile(4000, true))).toThrowError( + /Only ELF, PE, and thin Mach-O native images are supported/, + ); +}); + +it("keeps the thin-slice failure for confirmed universal Mach-O containers", () => { + const slices = [ + { cpu: CPU.x86_64, bytes: machoImage({ cpu: CPU.x86_64 }) }, + { cpu: CPU.arm64, bytes: machoImage({ cpu: CPU.arm64 }) }, + ]; + for (const universal of [ + fatImage(slices), + fatImage(slices, true), + fatImage(slices, false, true), + fatImage(slices, true, true), + ]) { + expect(() => readGoBinaryImage(Buffer.from(universal))).toThrowError( + /fat Mach-O containers require selecting a thin slice/, + ); + } +}); + it("rejects ELF header tables overlapping the native header", () => { const fixture = createGoBinaryFixture({ sectionless: true }); fixture.bytes.writeBigUInt64LE(0n, 32); diff --git a/tests/boundary/javascript-recovery/wakaruPublicationCancellation.test.ts b/tests/boundary/javascript-recovery/wakaruPublicationCancellation.test.ts new file mode 100644 index 000000000..250e75623 --- /dev/null +++ b/tests/boundary/javascript-recovery/wakaruPublicationCancellation.test.ts @@ -0,0 +1,128 @@ +import { createHash } from "node:crypto"; +import { access, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { Readable } from "node:stream"; +import { expect, it, vi } from "vitest"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +const publicationCancellation = vi.hoisted(() => ({ + controller: undefined as AbortController | undefined, + directorySyncs: 0, +})); + +vi.mock("node:fs/promises", async (importOriginal) => { + const filesystem = await importOriginal(); + return { + ...filesystem, + open: async (...args: Parameters) => { + const handle = await filesystem.open(...args); + if (publicationCancellation.controller !== undefined) { + const stats = await handle.stat(); + if (stats.isDirectory()) { + const sync = handle.sync.bind(handle); + handle.sync = async () => { + await sync(); + publicationCancellation.directorySyncs += 1; + if (publicationCancellation.directorySyncs === 2) + publicationCancellation.controller?.abort(); + }; + } + } + return handle; + }, + }; +}); + +vi.mock( + "../../../src/javascript/recovery/WakaruCommand.js", + async (importOriginal) => { + const actual = + await importOriginal< + typeof import("../../../src/javascript/recovery/WakaruCommand.js") + >(); + return { + ...actual, + resolveWakaruCommand: async () => ({ + command: "/unused/wakaru", + limiter: "/unused/prlimit", + sha256: "a".repeat(64), + bytes: 1, + }), + }; + }, +); + +vi.mock( + "../../../src/javascript/recovery/WakaruExecution.js", + async (importOriginal) => { + const actual = + await importOriginal< + typeof import("../../../src/javascript/recovery/WakaruExecution.js") + >(); + const { createAnalysisExecution } = + await import("../../../src/application/AnalysisProvider.js"); + return { + ...actual, + prepareWakaruExecution: async ( + context: Parameters[0], + ) => { + const source = context.sources[0]; + if (source === undefined) throw new Error("Missing recovery source"); + const bytes = Buffer.from("module.exports = 42;"); + await context.tree.write("module.js", Readable.from([bytes]), { + sha256: createHash("sha256").update(bytes).digest("hex"), + bytes: bytes.byteLength, + }); + return createAnalysisExecution( + { status: "complete" }, + { id: "wakaru", name: "Wakaru", version: "1.14.0" }, + { + subject: { + path: source.path, + sha256: source.sha256, + format: "javascript", + }, + }, + ); + }, + }; + }, +); + +import { AnalysisCancelledError } from "../../../src/domain/analysisErrorCore.js"; +import { projectAnalysisError } from "../../../src/domain/analysisErrorProjection.js"; +import { WakaruProvider } from "../../../src/javascript/recovery/WakaruProvider.js"; + +it.skipIf(process.platform === "win32")( + "keeps mid-commit output-tree cancellation typed through the provider", + async () => { + const root = await createTestTempDirectory("rea-wakaru-cancel-"); + const sourcePath = join(root, "source.js"); + const output = join(root, "output"); + await writeFile(sourcePath, "module.exports = 42;"); + const controller = new AbortController(); + publicationCancellation.controller = controller; + publicationCancellation.directorySyncs = 0; + const provider = new WakaruProvider({}); + try { + const result = await provider.recover( + { + path: sourcePath, + output_directory: output, + extraction_mode: "structural", + rewrite_level: "standard", + }, + { signal: controller.signal }, + ); + if (result.ok) throw new Error("Expected publication cancellation"); + expect(result.error).toBeInstanceOf(AnalysisCancelledError); + expect(projectAnalysisError(result.error).code).toBe("cancelled"); + expect(publicationCancellation.directorySyncs).toBe(2); + await expect(access(output)).rejects.toMatchObject({ code: "ENOENT" }); + } finally { + publicationCancellation.controller = undefined; + publicationCancellation.directorySyncs = 0; + await provider.close(); + } + }, +); diff --git a/tests/boundary/javascript/sourceMapRuntimeAvailability.test.ts b/tests/boundary/javascript/sourceMapRuntimeAvailability.test.ts new file mode 100644 index 000000000..753a3bd47 --- /dev/null +++ b/tests/boundary/javascript/sourceMapRuntimeAvailability.test.ts @@ -0,0 +1,35 @@ +import { expect, it } from "vitest"; +import { projectAnalysisError } from "../../../src/domain/analysisErrorProjection.js"; +import { runSourceMapCommand } from "../../../src/javascript/sourceMaps/SourceMapCommand.js"; +import { PrivateRuntimeRootUnavailableError } from "../../../src/process/PrivateRuntimeRootUnavailableError.js"; + +it.each([false, true])( + "preserves private-runtime retryable=%s through the source-map operation", + async (retryable) => { + const reason = "Native runtime self-test failed with a sharing violation"; + let launches = 0; + const result = await runSourceMapCommand("{}", "/selected.map", undefined, { + prepareOwnership: async () => {}, + createRuntime: async () => { + throw new PrivateRuntimeRootUnavailableError(reason, { retryable }); + }, + launcher: async () => { + launches += 1; + throw new Error("A refused runtime must not launch a codec"); + }, + }); + expect(launches).toBe(0); + if (result.ok) throw new Error("Unavailable runtime must refuse decoding"); + expect(projectAnalysisError(result.error)).toMatchObject({ + code: "capability_unavailable", + category: retryable ? "unavailable" : "unsupported_provider", + retryable, + details: { + provider_id: "source-map-decoder", + operation: "trace_web_source_location", + reason, + }, + }); + expect(result.error.cleanupIncomplete).toBe(false); + }, +); diff --git a/tests/boundary/mcp/analysisViewMcp.test.ts b/tests/boundary/mcp/analysisViewMcp.test.ts index f23295829..b95aebe57 100644 --- a/tests/boundary/mcp/analysisViewMcp.test.ts +++ b/tests/boundary/mcp/analysisViewMcp.test.ts @@ -39,6 +39,8 @@ import { } from "../../fixtures/analysisView.js"; import { createTestBinarySession } from "../../fixtures/binarySession.js"; import { BINARY_LAYOUT_TEST_PROVIDER } from "../../fixtures/binaryDiagnostics/layout.js"; +import { wasmArtifactFixture } from "../../fixtures/wasm/artifact.js"; +import { evmInterfaceFixture } from "../../fixtures/evm/interface.js"; const connect = async ( binaryLayout?: BinaryLayoutService, @@ -69,6 +71,12 @@ const connect = async ( it("discovers an oversized native dossier by exact metadata and reads its retained view", async () => { const { client, session } = await connect(); + const advertised = (await client.listTools()).tools.find( + (tool) => tool.name === "inspect_analysis_view", + ); + expect(advertised?.description).toContain( + 'Native function Evidence requires kind native: view: {kind: "native", facet: "procedure"}', + ); const provider = { id: "ghidra", name: "Ghidra", version: "12.1.4" }; const profile = createAnalysisProfile(provider, { language: "fixture" }); const parent = createEvidence( @@ -98,6 +106,24 @@ it("discovers an oversized native dossier by exact metadata and reads its retain const retained = session.recordEvidence(record); if (!retained.ok) throw retained.error; } + const unsupportedSummary = await client.callTool({ + name: "inspect_analysis_view", + arguments: { + source: { kind: "retained-evidence", evidence_id: parent.evidence_id }, + view: { kind: "summary" }, + }, + }); + expect(parseMcpToolError(unsupportedSummary).error).toMatchObject({ + code: "invalid_request", + details: { + issues: [ + { + path: ["view"], + message: "analyze_function Evidence requires a native view.", + }, + ], + }, + }); const broad = await client.callTool({ name: "get_evidence_bundle", arguments: {}, @@ -286,6 +312,65 @@ it("reports a tampered inline Evidence ID at its full MCP source path", async () }); }); +it("rejects self-contained WASM and EVM Evidence views when the subject format differs", async () => { + const { client } = await connect(); + const wasm = wasmArtifactFixture(); + const evm = evmInterfaceFixture(); + const cases = [ + { + evidence: createEvidence( + { + path: wasm.artifact.path, + format: "elf", + sha256: wasm.artifact.sha256, + }, + { id: "fixture-wabt", name: "WABT", version: "1" }, + { + operation: "inspect_wasm_artifact", + parameters: { path: wasm.artifact.path }, + result: wasm, + }, + ), + }, + { + evidence: createEvidence( + { + path: evm.artifact.path, + format: "elf", + sha256: evm.artifact.sha256, + }, + { id: "fixture-evm", name: "EVMole", version: "1" }, + { + operation: "inspect_evm_interface", + parameters: { path: evm.artifact.path, encoding: "hex" }, + result: evm, + }, + ), + }, + ] as const; + + for (const { evidence } of cases) { + const response = await client.callTool({ + name: "inspect_analysis_view", + arguments: { + source: { kind: "inline", evidence }, + view: { kind: "summary" }, + }, + }); + expect(parseMcpToolError(response).error).toMatchObject({ + code: "invalid_request", + details: { + issues: [ + expect.objectContaining({ + path: ["source", "subject"], + reason: "invalid_value", + }), + ], + }, + }); + } +}); + it("admits portable captures before SDK Evidence parsing and preserves capture caveats", async () => { const { client } = await connect(); const dense = analysisViewCaptureEvidence(analysisViewObjectDenseCapture()); diff --git a/tests/boundary/mcp/androidPackageSummaryMcp.test.ts b/tests/boundary/mcp/androidPackageSummaryMcp.test.ts new file mode 100644 index 000000000..960886b02 --- /dev/null +++ b/tests/boundary/mcp/androidPackageSummaryMcp.test.ts @@ -0,0 +1,295 @@ +import { unlink } from "node:fs/promises"; +import { Client, InMemoryTransport } from "@modelcontextprotocol/client"; +import { Ajv2020 } from "ajv/dist/2020.js"; +import { expect, it, onTestFinished, vi } from "vitest"; +import { AndroidAnalysisService } from "../../../src/application/android/AndroidAnalysisService.js"; +import { inspectAndroidPackage } from "../../../src/application/android/AndroidPackageInspection.js"; +import { inspectAnalysisView } from "../../../src/application/analysisView/AnalysisViewService.js"; +import { androidResultSchemas } from "../../../src/domain/android/androidAnalysis.js"; +import { + androidPackageSummaryViewResultSchema, + analysisViewResultSchema, +} from "../../../src/domain/analysisView/analysisView.js"; +import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js"; +import type { JsonValue } from "../../../src/domain/jsonValue.js"; +import { createServer } from "../../../src/server/createServer.js"; +import { createApplicationMcpHarness } from "../../fixtures/applicationMcpHarness.js"; +import { createTestBinarySession } from "../../fixtures/binarySession.js"; +import { parseMcpToolError } from "../../fixtures/mcpToolError.js"; +import { + createJadxProtocolFixture, + verifyJadxFixtureCleanup, +} from "../../fixtures/android/jadx.js"; + +it.skipIf(process.platform === "win32")( + "retains complete APK Evidence and reads exact manifest without its source or another engine launch", + async () => { + const { provider, apk, launches } = + await createJadxProtocolFixture("large-manifest"); + const harness = await createApplicationMcpHarness({ + serverOptions: { androidAnalysis: provider }, + }); + onTestFinished(harness.close); + const { client, session } = harness; + const catalog = (await client.listTools()).tools; + const definition = catalog.find( + (tool) => tool.name === "inspect_android_package", + ); + if (definition?.outputSchema === undefined) + throw new Error("Missing APK schema"); + const outputSchema: Record = definition.outputSchema; + const ajv = new Ajv2020({ strict: false, validateFormats: false }); + expect(ajv.validateSchema(definition.inputSchema)).toBe(true); + expect(ajv.validateSchema(outputSchema)).toBe(true); + expect( + ajv.validate(definition.inputSchema, { path: apk, detail: "summary" }), + ).toBe(true); + expect( + ajv.validate( + catalog.find((tool) => tool.name === "inspect_android_class")! + .inputSchema, + { + path: apk, + class_name: "fixture.Target", + detail: "summary", + }, + ), + ).toBe(false); + const reply = await client.callTool({ + name: "inspect_android_package", + arguments: { path: apk, detail: "summary" }, + }); + expect(reply.isError).not.toBe(true); + expect( + ajv.validate(outputSchema, reply.structuredContent), + JSON.stringify(ajv.errors), + ).toBe(true); + const receipt = parseEvidence(reply.structuredContent); + const view = androidPackageSummaryViewResultSchema.parse( + receipt.normalized_result, + ); + const parent = session.evidenceById(view.parent_evidence_id); + if (parent === undefined) + throw new Error("Complete package Evidence not retained"); + const report = androidResultSchemas.inspect_android_package.parse( + parent.normalized_result, + ); + expect(parent.operation).toBe("inspect_android_package"); + expect(parent.parameters).toEqual({ path: apk }); + expect(view.artifact.sha256).toBe(parent.subject?.digest.sha256); + expect(view.artifact.sha256).not.toBe(report.engine.artifact_sha256); + expect(receipt.evidence_id).not.toBe(parent.evidence_id); + expect(receipt.evidence_links).toEqual([parent.evidence_id]); + const { manifest, ...declarations } = report; + expect(view.summary).toMatchObject(declarations); + expect(view.summary.manifest).toEqual({ + status: manifest.status, + reported_total_bytes: manifest.reported_total_bytes, + text_code_units: manifest.text.length, + text_included: false, + }); + expect(receipt.raw_result).toBeNull(); + expect(parent.raw_result).not.toBeNull(); + for (const detail of [undefined, "complete"]) { + const full = parseEvidence( + ( + await client.callTool({ + name: "inspect_android_package", + arguments: { + path: apk, + ...(detail === undefined ? {} : { detail }), + }, + }) + ).structuredContent, + ); + expect(full.operation).toBe("inspect_android_package"); + expect(full.normalized_result).toEqual(parent.normalized_result); + expect(full.parameters).toEqual(parent.parameters); + expect(full.raw_result).not.toBeNull(); + } + const before = JSON.stringify(parent); + await unlink(apk); + let text = ""; + let offset = 0; + do { + const paged = await client.callTool({ + name: "inspect_analysis_view", + arguments: { + source: { + kind: "retained-evidence", + evidence_id: parent.evidence_id, + }, + view: { kind: "android", facet: "manifest", offset, limit: 700000 }, + }, + }); + expect(paged.isError).not.toBe(true); + const page = analysisViewResultSchema.parse( + parseEvidence(paged.structuredContent).normalized_result, + ); + if (page.kind !== "android") throw new Error("Unexpected manifest page"); + text += page.item.text; + expect(page.item.source_status).toBe(manifest.status); + expect(page.item.reported_total_bytes).toBe( + manifest.reported_total_bytes, + ); + offset = page.coverage.next_offset ?? manifest.text.length; + } while (offset < manifest.text.length); + expect(text).toBe(manifest.text); + const portable = inspectAnalysisView({ + source: { kind: "inline", evidence: parseEvidence(JSON.parse(before)) }, + view: { kind: "summary" }, + }); + if (!portable.ok) throw portable.error; + expect(portable.value.normalized_result).toEqual(view); + expect(JSON.stringify(session.evidenceById(parent.evidence_id))).toBe( + before, + ); + expect( + session + .exportEvidenceBundle() + .records.find((record) => record.evidence_id === parent.evidence_id), + ).toEqual(parent); + expect(launches).toHaveLength(1); + await harness.close(); + await verifyJadxFixtureCleanup(launches); + }, + 30000, +); + +it("preserves partial/unknown declarations and Unicode text while rejecting missing APK identity and malformed source", () => { + const report = { + engine: { + name: "jadx", + version: null, + artifact_sha256: "b".repeat(64), + source_revision: null, + worker_count: 1, + heap_limit_mib: null, + }, + package_name: null, + version_name: null, + version_code: null, + min_sdk: null, + target_sdk: null, + permissions: [], + manifest: { status: "partial", text: "a😀b", reported_total_bytes: null }, + class_count: 0, + resource_count: 0, + signature_verification: "not_performed", + }; + const evidence = (result: JsonValue = report, hasSubject = true) => + createEvidence( + hasSubject + ? { path: "/original.apk", sha256: "a".repeat(64), format: "apk" } + : undefined, + { id: "jadx", name: "JADX", version: "0.7.1" }, + { + operation: "inspect_android_package", + parameters: {}, + result, + limitations: ["Original limit"], + }, + ); + const parent = evidence(); + const source = { kind: "inline", evidence: parent }; + const summary = inspectAnalysisView({ source, view: { kind: "summary" } }); + if (!summary.ok) throw summary.error; + const overview = androidPackageSummaryViewResultSchema.parse( + summary.value.normalized_result, + ); + expect(overview.summary.package_name).toBeNull(); + expect(overview.summary.manifest.status).toBe("partial"); + expect(overview.limitations).toEqual(parent.limitations); + const page = inspectAnalysisView({ + source, + view: { kind: "android", facet: "manifest", offset: 0, limit: 4 }, + }); + if (!page.ok) throw page.error; + const result = analysisViewResultSchema.parse(page.value.normalized_result); + expect(result).toMatchObject({ + kind: "android", + item: { + text: "a😀b", + source_status: "partial", + reported_total_bytes: null, + }, + coverage: { exhausted: true }, + }); + expect(result.unknowns.join(" ")).toContain("partial"); + for (const offset of [2]) + expect( + inspectAnalysisView({ + source, + view: { kind: "android", facet: "manifest", offset, limit: 2 }, + }), + ).toMatchObject({ ok: false, error: { _tag: "AnalysisInputError" } }); + for (const invalid of [ + evidence(report, false), + evidence({ ...report, class_count: -1 }), + ]) + expect( + inspectAnalysisView({ + source: { kind: "inline", evidence: invalid }, + view: { kind: "summary" }, + }), + ).toMatchObject({ ok: false, error: { _tag: "AnalysisInputError" } }); +}); + +it("rejects unavailable retention, invalid selectors and cancellation before provider execution", async () => { + const execute = vi.fn(); + const provider = { + execute, + close: async () => {}, + inspectAvailability: vi.fn(), + }; + const service = new AndroidAnalysisService(provider); + const retention = { recordEvidence: undefined, evidenceById: undefined }; + expect( + await inspectAndroidPackage( + service, + { path: "/missing.apk", detail: "summary" }, + retention, + ), + ).toMatchObject({ + ok: false, + error: { _tag: "AnalysisCapabilityUnavailableError" }, + }); + expect( + await inspectAndroidPackage( + service, + { path: "/missing.apk", detail: "other" }, + retention, + ), + ).toMatchObject({ ok: false, error: { _tag: "AnalysisInputError" } }); + expect( + await inspectAndroidPackage( + service, + { path: "/missing.apk", detail: "summary" }, + retention, + { signal: AbortSignal.abort() }, + ), + ).toMatchObject({ ok: false, error: { _tag: "AnalysisCancelledError" } }); + expect(execute).not.toHaveBeenCalled(); + const session = createTestBinarySession(() => { + throw new Error("Unexpected deep provider"); + }); + const server = createServer( + { kind: "fixed", analysis: session }, + { androidAnalysis: provider }, + ); + const client = new Client({ name: "android-fixed-summary", version: "1" }); + onTestFinished(async () => { + await client.close(); + await server.close(); + await session.close(); + }); + const [a, b] = InMemoryTransport.createLinkedPair(); + await server.connect(b); + await client.connect(a); + const reply = await client.callTool({ + name: "inspect_android_package", + arguments: { path: "/missing.apk", detail: "summary" }, + }); + expect(parseMcpToolError(reply).error.code).toBe("capability_unavailable"); + expect(execute).not.toHaveBeenCalled(); +}); diff --git a/tests/boundary/mcp/applicationEvidenceMcp.test.ts b/tests/boundary/mcp/applicationEvidenceMcp.test.ts index 16dfc142c..ba71b7921 100644 --- a/tests/boundary/mcp/applicationEvidenceMcp.test.ts +++ b/tests/boundary/mcp/applicationEvidenceMcp.test.ts @@ -2,6 +2,7 @@ import { parseMcpToolError } from "../../fixtures/mcpToolError.js"; import type { Client } from "@modelcontextprotocol/client"; import { describe, expect, it } from "vitest"; import { Ajv2020 } from "ajv/dist/2020.js"; +import type { AnySchema } from "ajv"; import type { BinarySession } from "../../../src/application/binary/BinarySession.js"; import { @@ -10,12 +11,143 @@ import { SOURCE_TO_BUNDLE_COMPARISON_EXAMPLE, } from "../../../src/contracts/javascript/javascriptApplicationWorkflowExamples.js"; import { createEvidence } from "../../../src/domain/evidence.js"; +import { javascriptApplicationAnalysisResultSchema } from "../../../src/domain/javascript/javascriptApplicationAnalysis.js"; +import { + createJavaScriptApplicationGraph, + createJavaScriptApplicationNode, +} from "../../../src/domain/javascript/javascriptApplicationGraph.js"; +import { createJavaScriptSemanticGraph } from "../../../src/domain/javascript/javascriptSemanticGraph.js"; +import { + artifactEvidence, + sourceMapReference, +} from "../../../src/domain/javascript/javascriptApplicationGraph.fixture.js"; import { createApplicationMcpHarness, type ApplicationMcpHarness, } from "../../fixtures/applicationMcpHarness.js"; import { createHistoricalSourceGraph } from "../../../src/domain/referenceSourceGraph.js"; +async function runUnicodeSourceMapComparison( + harness: ApplicationMcpHarness, +): Promise { + const baseApplication = SOURCE_TO_BUNDLE_COMPARISON_EXAMPLE.application; + const subject = baseApplication.subject; + if (subject === null) throw new Error("Application Evidence has no subject"); + const baseResult = javascriptApplicationAnalysisResultSchema.parse( + baseApplication.normalized_result, + ); + const baseGraph = baseResult.graph; + const decomposedSource = "cafe\u0301.ts"; + const unicodeSourceNode = createJavaScriptApplicationNode({ + kind: "source-module", + identity: { + strategy: "source-map-original", + source_root: "../src/", + stability: "source-map-exact", + source_map_sha256: "1".repeat(64), + original_source: decomposedSource, + source_sha256: "3".repeat(64), + }, + observations: [ + { + label: "decomposed source-map spelling", + source_map_reference: sourceMapReference( + decomposedSource, + "../src/", + "dist/main.js.map", + ), + properties: { content_available: true, content_sha256: "3".repeat(64) }, + evidence: artifactEvidence("1".repeat(64), "dist/main.js.map"), + }, + ], + }); + const applicationGraph = createJavaScriptApplicationGraph({ + schema: "JavaScriptApplicationGraph", + root_node_ids: [...baseGraph.root_node_ids, unicodeSourceNode.node_id], + nodes: [...baseGraph.nodes, unicodeSourceNode], + edges: baseGraph.edges, + coverage: baseGraph.coverage, + limitations: baseGraph.limitations, + }); + const semantic = baseResult.semantic_graph; + const { graph_id: _semanticGraphId, ...semanticInput } = semantic; + const semanticGraph = createJavaScriptSemanticGraph({ + ...semanticInput, + application_graph_id: applicationGraph.graph_id, + }); + const unicodeApplication = createEvidence( + { + path: baseResult.input_path, + sha256: subject.digest.sha256, + format: subject.format, + }, + baseApplication.provider, + { + predicateType: baseApplication.predicate_type, + operation: baseApplication.operation, + parameters: baseApplication.parameters, + result: { + ...baseResult, + graph: applicationGraph, + semantic_graph: semanticGraph, + }, + confidence: baseApplication.confidence, + authority: baseApplication.authority, + limitations: baseApplication.limitations, + evidenceLinks: baseApplication.evidence_links, + }, + ); + const historicalUnicodePath = "src/caf\u00e9.ts"; + const unicodeReference = createHistoricalSourceGraph({ + ...SOURCE_TO_BUNDLE_COMPARISON_EXAMPLE.reference, + root_sha256: undefined, + entries: SOURCE_TO_BUNDLE_COMPARISON_EXAMPLE.reference.entries.map( + (entry) => + entry.path === "src/main.ts" + ? { ...entry, path: historicalUnicodePath } + : entry, + ), + }); + const reply = await harness.client.callTool({ + name: "compare_source_to_bundle", + arguments: { reference: unicodeReference, application: unicodeApplication }, + }); + expect(reply.isError).not.toBe(true); + expect(reply.structuredContent).toMatchObject({ + normalized_result: { + items: [ + { + source_path: historicalUnicodePath, + status: "unknown", + current_node_ids: [], + candidates: [ + expect.objectContaining({ + confidence: "low", + signals: expect.arrayContaining([ + expect.objectContaining({ + kind: "unicode-equivalent-path", + source_value: historicalUnicodePath, + current_values: ["src/cafe\u0301.ts"], + }), + ]), + }), + ], + }, + ], + }, + }); + const advertised = (await harness.client.listTools()).tools.find( + ({ name }) => name === "compare_source_to_bundle", + ); + if (advertised?.outputSchema === undefined) + throw new Error("Comparison tool output schema is missing"); + const ajv = new Ajv2020({ strict: false, validateFormats: false }); + const outputSchema = advertised.outputSchema as AnySchema; + expect(ajv.validateSchema(outputSchema)).toBe(true); + const validateOutput = ajv.compile(outputSchema); + expect(validateOutput(reply.structuredContent)).toBe(true); +} + async function runInlineWorkflowScenarios( harness: ApplicationMcpHarness, ): Promise { @@ -92,6 +224,8 @@ async function runInlineWorkflowScenarios( }, }); + await runUnicodeSourceMapComparison(harness); + const referenceWithExclusion = createHistoricalSourceGraph({ ...SOURCE_TO_BUNDLE_COMPARISON_EXAMPLE.reference, root_sha256: undefined, diff --git a/tests/boundary/mcp/dartAotSummaryMcp.test.ts b/tests/boundary/mcp/dartAotSummaryMcp.test.ts new file mode 100644 index 000000000..be6a970d3 --- /dev/null +++ b/tests/boundary/mcp/dartAotSummaryMcp.test.ts @@ -0,0 +1,285 @@ +import { unlink } from "node:fs/promises"; +import { Ajv2020 } from "ajv/dist/2020.js"; +import { expect, it, onTestFinished, vi } from "vitest"; +import { FlutterBuildAnalysisService } from "../../../src/application/flutter/FlutterBuildAnalysisService.js"; +import { inspectDartAot } from "../../../src/application/flutter/DartAotInspection.js"; +import { inspectAnalysisView } from "../../../src/application/analysisView/AnalysisViewService.js"; +import { createFlutterBuildAnalysisProvider } from "../../../src/composition/flutter.js"; +import { + analysisViewResultSchema, + dartAotSummaryViewResultSchema, +} from "../../../src/domain/analysisView/analysisView.js"; +import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js"; +import { flutterResultSchemas } from "../../../src/domain/flutter/flutterBuildAnalysis.js"; +import type { JsonValue } from "../../../src/domain/jsonValue.js"; +import { aotFixtureImage } from "../../../src/flutter/FlutterBuild.fixture.js"; +import { createFlutterApk } from "../../fixtures/flutterApk.js"; +import { createApplicationMcpHarness } from "../../fixtures/applicationMcpHarness.js"; + +it("honors explicit ABI and old full requests, validates SDK schemas and reuses complete Evidence after APK removal", async () => { + const path = await createFlutterApk("views.apk", [ + { path: "lib/arm64-v8a/libapp.so", bytes: aotFixtureImage() }, + { path: "lib/x86_64/libapp.so", bytes: aotFixtureImage("b".repeat(32)) }, + ]); + const provider = createFlutterBuildAnalysisProvider(); + const execute = vi.spyOn(provider, "execute"); + const harness = await createApplicationMcpHarness({ + serverOptions: { flutterAnalysis: provider }, + }); + onTestFinished(harness.close); + const { client, session } = harness; + const catalog = (await client.listTools()).tools; + const definition = catalog.find((tool) => tool.name === "inspect_dart_aot"); + if (definition?.outputSchema === undefined) + throw new Error("Missing AOT output schema"); + const outputSchema: Record = definition.outputSchema; + const ajv = new Ajv2020({ strict: false, validateFormats: false }); + expect(ajv.validateSchema(definition.inputSchema)).toBe(true); + expect(ajv.validateSchema(outputSchema)).toBe(true); + expect( + ajv.validate( + catalog.find((tool) => tool.name === "identify_flutter_build")! + .inputSchema, + { path, detail: "summary" }, + ), + ).toBe(false); + const reply = await client.callTool({ + name: "inspect_dart_aot", + arguments: { path, abi: "x86_64", detail: "summary" }, + }); + expect(reply.isError).not.toBe(true); + expect( + ajv.validate(outputSchema, reply.structuredContent), + JSON.stringify(ajv.errors), + ).toBe(true); + const receipt = parseEvidence(reply.structuredContent); + const view = dartAotSummaryViewResultSchema.parse(receipt.normalized_result); + const parent = session.evidenceById(view.parent_evidence_id); + if (parent === undefined) + throw new Error("Complete AOT source was not retained"); + const report = flutterResultSchemas.inspect_dart_aot.parse( + parent.normalized_result, + ); + expect(report.abi).toBe("x86_64"); + expect(report.libapp.snapshot_hash).toBe("b".repeat(32)); + expect(parent.parameters).toEqual({ path, abi: "x86_64" }); + expect(execute).toHaveBeenLastCalledWith( + { operation: "inspect_dart_aot", input: { path, abi: "x86_64" } }, + expect.anything(), + ); + expect(view.summary.libapp).toEqual(report.libapp); + expect(view.summary.target).toEqual(report.target); + expect(view.summary.coverage).toBe(report.coverage); + expect(receipt.evidence_id).not.toBe(parent.evidence_id); + expect(receipt.evidence_links).toEqual([parent.evidence_id]); + for (const detail of [undefined, "complete"]) { + const complete = parseEvidence( + ( + await client.callTool({ + name: "inspect_dart_aot", + arguments: { + path, + abi: "x86_64", + ...(detail === undefined ? {} : { detail }), + }, + }) + ).structuredContent, + ); + expect(complete.operation).toBe("inspect_dart_aot"); + expect(complete.normalized_result).toEqual(parent.normalized_result); + expect(complete.parameters).toEqual(parent.parameters); + } + const preferred = parseEvidence( + (await client.callTool({ name: "inspect_dart_aot", arguments: { path } })) + .structuredContent, + ); + expect( + flutterResultSchemas.inspect_dart_aot.parse(preferred.normalized_result) + .abi, + ).toBe("arm64-v8a"); + const original = JSON.stringify(parent); + await unlink(path); + const calls = execute.mock.calls.length; + for (const collection of [ + "package_uris", + "dart_uris", + "class_like_tokens", + ] as const) { + const strings: string[] = []; + let offset = 0; + do { + const reply = await client.callTool({ + name: "inspect_analysis_view", + arguments: { + source: { + kind: "retained-evidence", + evidence_id: parent.evidence_id, + }, + view: { kind: "dart-aot", collection, offset, limit: 1 }, + }, + }); + expect(reply.isError).not.toBe(true); + const page = analysisViewResultSchema.parse( + parseEvidence(reply.structuredContent).normalized_result, + ); + if (page.kind !== "dart-aot") throw new Error("Wrong pool page kind"); + strings.push(...page.items); + expect(page.libapp).toEqual(report.libapp); + expect(page.source_coverage).toBe(report.coverage); + expect(page.string_pool.coverage).toBe(report.string_pool.coverage); + offset = + page.coverage.next_offset ?? report.string_pool[collection].length; + } while (offset < report.string_pool[collection].length); + expect(strings).toEqual(report.string_pool[collection]); + } + expect(execute).toHaveBeenCalledTimes(calls); + expect(JSON.stringify(session.evidenceById(parent.evidence_id))).toBe( + original, + ); + expect( + session + .exportEvidenceBundle() + .records.find((record) => record.evidence_id === parent.evidence_id), + ).toEqual(parent); + const portable = inspectAnalysisView({ + source: { kind: "inline", evidence: parent }, + view: { kind: "summary" }, + }); + if (!portable.ok) throw portable.error; + expect(portable.value.normalized_result).toEqual(view); +}, 30000); + +it("preserves null snapshot facts and pool lower bounds after exhaustion, rejecting malformed and conflicting identities", () => { + const report = { + target: { path: "/source.apk", bytes: 10, sha256: "a".repeat(64) }, + abi: "arm64-v8a", + libapp: { + coverage: "partial", + bytes: 1, + sha256: "b".repeat(64), + snapshot_hash: null, + snapshot_hash_candidates: [], + snapshot_hash_sources: 0, + sections: [], + }, + string_pool: { + coverage: "partial", + package_uris: ["package:opaque/a.dart"], + package_uri_count_at_least: 1000, + dart_uris: [], + dart_uri_count_at_least: 0, + class_like_tokens: ["incidental"], + class_like_token_count_at_least: 4000, + printable_run_count: 12, + }, + coverage: "partial", + }; + const evidence = ( + result: JsonValue = report, + sha: string | null = "a".repeat(64), + ) => + createEvidence( + sha === null + ? undefined + : { path: "/source.apk", sha256: sha, format: "apk" }, + { id: "flutter", name: "Flutter", version: null }, + { + operation: "inspect_dart_aot", + parameters: {}, + result, + limitations: ["Original uncertainty"], + }, + ); + for (const parent of [evidence(), evidence(report, null)]) { + const summary = inspectAnalysisView({ + source: { kind: "inline", evidence: parent }, + view: { kind: "summary" }, + }); + if (!summary.ok) throw summary.error; + const result = dartAotSummaryViewResultSchema.parse( + summary.value.normalized_result, + ); + expect(result.summary.libapp.snapshot_hash).toBeNull(); + expect(result.summary.string_pool.package_uri_count_at_least).toBe(1000); + expect(result.summary.string_pool.returned_counts.package_uris).toBe(1); + expect(result.limitations).toEqual(parent.limitations); + const page = inspectAnalysisView({ + source: { kind: "inline", evidence: parent }, + view: { + kind: "dart-aot", + collection: "package_uris", + offset: 0, + limit: 10, + }, + }); + if (!page.ok) throw page.error; + expect(page.value.normalized_result).toMatchObject({ + kind: "dart-aot", + items: report.string_pool.package_uris, + source_coverage: "partial", + string_pool: { coverage: "partial", package_uri_count_at_least: 1000 }, + coverage: { total: 1, exhausted: true }, + }); + } + expect( + inspectAnalysisView({ + source: { kind: "inline", evidence: evidence(report, "c".repeat(64)) }, + view: { kind: "summary" }, + }), + ).toMatchObject({ ok: false, error: { _tag: "EvidenceIntegrityError" } }); + expect( + inspectAnalysisView({ + source: { + kind: "inline", + evidence: evidence({ + ...report, + string_pool: { + ...report.string_pool, + package_uri_count_at_least: -1, + }, + }), + }, + view: { kind: "summary" }, + }), + ).toMatchObject({ ok: false, error: { _tag: "AnalysisInputError" } }); + expect( + inspectAnalysisView({ + source: { kind: "inline", evidence: evidence() }, + view: { kind: "android", facet: "manifest", offset: 0, limit: 1 }, + }), + ).toMatchObject({ ok: false, error: { _tag: "AnalysisInputError" } }); +}); + +it("rejects unavailable retention, invalid delivery and cancellation before producer work", async () => { + const provider = createFlutterBuildAnalysisProvider(); + onTestFinished(() => provider.close()); + const execute = vi.spyOn(provider, "execute"); + const service = new FlutterBuildAnalysisService(provider); + const retention = { recordEvidence: undefined, evidenceById: undefined }; + expect( + await inspectDartAot( + service, + { path: "/missing.apk", abi: "x86_64", detail: "summary" }, + retention, + ), + ).toMatchObject({ + ok: false, + error: { _tag: "AnalysisCapabilityUnavailableError" }, + }); + expect( + await inspectDartAot( + service, + { path: "/missing.apk", detail: "other" }, + retention, + ), + ).toMatchObject({ ok: false, error: { _tag: "AnalysisInputError" } }); + expect( + await inspectDartAot( + service, + { path: "/missing.apk", detail: "summary" }, + retention, + { signal: AbortSignal.abort() }, + ), + ).toMatchObject({ ok: false, error: { _tag: "AnalysisCancelledError" } }); + expect(execute).not.toHaveBeenCalled(); +}); diff --git a/tests/boundary/mcp/electronMcp.test.ts b/tests/boundary/mcp/electronMcp.test.ts index fef9839aa..72d9fdaf6 100644 --- a/tests/boundary/mcp/electronMcp.test.ts +++ b/tests/boundary/mcp/electronMcp.test.ts @@ -471,3 +471,49 @@ it("exposes the target-free static JavaScript application workflow", async () => const evidenceFor = (value: unknown) => { return parseEvidence(value); }; + +it("reports every unresolved Electron path without starting a provider", async () => { + const root = await createTestTempDirectory("rea-electron-invalid-paths-"); + const capture = vi.fn(); + const session = createTestBinarySession(() => { + throw new Error("must not start"); + }); + const server = createServer( + { kind: "session", session }, + { + electronActiveObservation: { + capture, + identity: () => ({ id: "test", name: "test", version: "1" }), + }, + }, + ); + const client = new Client({ name: "electron-path-test", version: "1" }); + const [clientTransport, serverTransport] = + InMemoryTransport.createLinkedPair(); + resources.push(client, server, session); + await server.connect(serverTransport); + await client.connect(clientTransport); + const fields = ["executable_path", "application_path", "application_root"]; + const result = await client.callTool({ + name: "capture_electron_scenario", + arguments: { + ...Object.fromEntries(fields.map((field) => [field, join(root, field)])), + actions: [], + }, + }); + expect(result.isError).toBe(true); + expect(parseMcpToolError(result)).toMatchObject({ + error: { + code: "invalid_request", + details: { + issues: fields.map((field) => ({ + path: [field], + reason: "invalid_value", + message: expect.stringContaining("ENOENT"), + })), + }, + }, + }); + expect(capture).not.toHaveBeenCalled(); + await client.ping(); +}); diff --git a/tests/boundary/mcp/enhancedTools.04-uses-complete-inventories-inline.test.ts b/tests/boundary/mcp/enhancedTools.04-uses-complete-inventories-inline.test.ts index 8e3a39969..e452d70c3 100644 --- a/tests/boundary/mcp/enhancedTools.04-uses-complete-inventories-inline.test.ts +++ b/tests/boundary/mcp/enhancedTools.04-uses-complete-inventories-inline.test.ts @@ -144,14 +144,26 @@ describe("enhanced MCP tools", () => { ); expect(result).toEqual({ document: "fixture", + image_base: "0x1000", segments: [ - { name: "__TEXT", start: "0x1000", end: "0x1800", length: 2048 }, - { name: "__DATA", start: "0x1800", end: "0x2000", length: 2048 }, + { + name: "__TEXT", + start: "0x1000", + end: "0x1800", + length: 2048, + image_base: "0x1000", + }, + { + name: "__DATA", + start: "0x1800", + end: "0x2000", + length: 2048, + image_base: "0x1000", + }, ], segment_count: 2, procedure_count: 2, string_count: 700, - image_base: "0x1000", }); }); diff --git a/tests/boundary/mcp/enhancedTools.traceFeatureCoverage.test.ts b/tests/boundary/mcp/enhancedTools.traceFeatureCoverage.test.ts new file mode 100644 index 000000000..8c94d5de0 --- /dev/null +++ b/tests/boundary/mcp/enhancedTools.traceFeatureCoverage.test.ts @@ -0,0 +1,249 @@ +import { afterEach, describe, expect, it } from "vitest"; + +import type { AnalysisOperationPort } from "../../../src/application/AnalysisProvider.js"; +import { createAnalysisExecution } from "../../../src/application/AnalysisProvider.js"; +import { EnhancedTools } from "../../../src/application/EnhancedTools.js"; +import { AnalysisOutputError } from "../../../src/domain/analysisErrorCore.js"; +import { err, ok } from "../../../src/domain/result.js"; + +import { closeEnhancedToolResources, connect } from "./enhancedToolsHarness.js"; + +afterEach(closeEnhancedToolResources); + +const PROVIDER = { + id: "fixture", + name: "Fixture analysis provider", + version: "1", +} as const; + +type Coverage = boolean | undefined; + +interface Behavior { + /** What the provider reports for the references of each match address. */ + readonly coverage?: Readonly>; + readonly limitations?: readonly string[]; + /** Also report the first string match as a procedure match. */ + readonly duplicateProcedure?: boolean; + /** Fail the references read of this address. */ + readonly failXrefsOf?: string; + readonly calls?: string[]; +} + +const containing = (address: string) => ({ + query_address: address, + found: false, + procedure: null, + reason: "not_in_procedure", +}); + +/** + * A provider with two string matches (0x10 and 0x11), each referenced from + * one address (0x20 and 0x21). + */ +const provider = (behavior: Behavior = {}): AnalysisOperationPort => ({ + execute: async (name, arguments_) => { + const address = String(arguments_.address); + behavior.calls?.push(`${name}:${address}`); + switch (name) { + case "search_strings": + return ok( + createAnalysisExecution( + [ + { address: "0x10", value: "hello" }, + { address: "0x11", value: "hello again" }, + ], + PROVIDER, + ), + ); + case "search_procedures": + return ok( + createAnalysisExecution( + behavior.duplicateProcedure === true + ? [{ address: "0x10", value: "hello" }] + : [], + PROVIDER, + ), + ); + case "xrefs": { + if (address === behavior.failXrefsOf) + return err(new AnalysisOutputError("xrefs", "provider failed")); + const reported = behavior.coverage?.[address]; + return ok( + createAnalysisExecution( + [address === "0x10" ? "0x20" : "0x21"], + PROVIDER, + { + limitations: behavior.limitations ?? [], + ...(reported === undefined + ? {} + : { referencesTruncated: reported }), + }, + ), + ); + } + case "resolve_containing_procedure": + return ok(createAnalysisExecution(containing(address), PROVIDER)); + default: + throw new Error(`Unexpected operation: ${name}`); + } + }, +}); + +const expectedTrace = ( + residual: readonly string[], + extraMatches: readonly object[] = [], +) => ({ + query: "hello", + search_mode: "literal", + matches: [ + { type: "string", address: "0x10", value: "hello" }, + { type: "string", address: "0x11", value: "hello again" }, + ...extraMatches, + ], + references: [ + { + target_address: "0x10", + source_address: "0x20", + containing_procedure: containing("0x20"), + }, + { + target_address: "0x11", + source_address: "0x21", + containing_procedure: containing("0x21"), + }, + ...extraMatches.map(() => ({ + target_address: "0x10", + source_address: "0x20", + containing_procedure: containing("0x20"), + })), + ], + truncated: residual.length > 0, + residual_unknowns: residual, +}); + +const unknownFor = (address: string) => + `References to ${address} were truncated by the provider; additional references may exist.`; + +const trace = async (analysis: AnalysisOperationPort) => { + const client = await connect(analysis); + const result = await client.callTool({ + name: "trace_feature", + arguments: { query: "hello" }, + }); + expect(result.isError).not.toBe(true); + return result.structuredContent as { normalized_result: unknown }; +}; + +describe("trace_feature reference coverage", () => { + it("marks the trace truncated for the target whose references the provider capped", async () => { + const output = await trace( + provider({ coverage: { "0x10": true, "0x11": false } }), + ); + + expect( + output.normalized_result, + "the received references are kept and only the capped target is a residual unknown", + ).toEqual(expectedTrace([unknownFor("0x10")])); + }); + + it("does not carry a truncation over to the next target that reports nothing", async () => { + const output = await trace(provider({ coverage: { "0x10": true } })); + + expect(output.normalized_result).toEqual( + expectedTrace([unknownFor("0x10")]), + ); + }); + + it("reports a capped target once when it is matched twice", async () => { + const output = await trace( + provider({ coverage: { "0x10": true }, duplicateProcedure: true }), + ); + + expect(output.normalized_result).toEqual( + expectedTrace( + [unknownFor("0x10")], + [{ type: "procedure", address: "0x10", value: "hello" }], + ), + ); + }); + + it("resolves the containing procedure of a repeated reference source once", async () => { + const calls: string[] = []; + const output = await trace(provider({ duplicateProcedure: true, calls })); + + expect( + calls.filter((call) => call.startsWith("resolve_containing_procedure:")), + "the source 0x20 is referenced by two matches of the same address and resolved once", + ).toEqual([ + "resolve_containing_procedure:0x20", + "resolve_containing_procedure:0x21", + ]); + expect( + output.normalized_result, + "every reference is still reported with its containing procedure", + ).toEqual( + expectedTrace( + [], + [{ type: "procedure", address: "0x10", value: "hello" }], + ), + ); + }); + + it("keeps the trace complete when the provider explicitly reports no cap", async () => { + const output = await trace( + provider({ coverage: { "0x10": false, "0x11": false } }), + ); + + expect(output.normalized_result).toEqual(expectedTrace([])); + }); + + it("does not change the trace of a provider that reports no coverage", async () => { + const output = await trace( + provider({ + limitations: ["Upstream xrefs_to capped results at 1000 references."], + }), + ); + + expect( + output.normalized_result, + "a limitation text alone never truncates, and an absent field keeps the previous result", + ).toEqual(expectedTrace([])); + }); + + it("never exposes the coverage field in the tool output", async () => { + const output = await trace( + provider({ coverage: { "0x10": true, "0x11": false } }), + ); + + expect(JSON.stringify(output)).not.toMatch( + /referencesTruncated|references_truncated/, + ); + }); + + it("fails the trace when a references read fails and reads nothing further", async () => { + const calls: string[] = []; + const result = await new EnhancedTools( + provider({ failXrefsOf: "0x10", calls }), + ).execute("trace_feature", { query: "hello" }); + + expect(result.ok).toBe(false); + expect( + calls.filter((call) => call.startsWith("xrefs:")), + "the next target is not read after the failure", + ).toEqual(["xrefs:0x10"]); + }); + + it("stops without reading references when the trace is cancelled", async () => { + const calls: string[] = []; + const controller = new AbortController(); + controller.abort(); + const result = await new EnhancedTools(provider({ calls })).execute( + "trace_feature", + { query: "hello" }, + controller.signal, + ); + + expect(result.ok).toBe(false); + expect(calls.filter((call) => call.startsWith("xrefs:"))).toEqual([]); + }); +}); diff --git a/tests/boundary/mcp/evidenceBundleInline.test.ts b/tests/boundary/mcp/evidenceBundleInline.test.ts index adb95ef8f..ae07a1be5 100644 --- a/tests/boundary/mcp/evidenceBundleInline.test.ts +++ b/tests/boundary/mcp/evidenceBundleInline.test.ts @@ -1,10 +1,20 @@ +import { readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; + import { ok as resultOk } from "../../../src/domain/result.js"; import { Client, InMemoryTransport } from "@modelcontextprotocol/client"; import { expect, it } from "vitest"; import { createTestBinarySession } from "../../fixtures/binarySession.js"; import { observed } from "../../fixtures/analysisExecution.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { parseMcpToolError } from "../../fixtures/mcpToolError.js"; +import { createAnalysisProfile } from "../../../src/domain/analysisProfile.js"; import { createEvidence } from "../../../src/domain/evidence.js"; +import { ANALYSIS_SNAPSHOT_TARGET } from "../../../src/domain/analysisSnapshot.fixture.js"; +import { jsonValueSchema } from "../../../src/domain/jsonValue.js"; +import { EMPTY_PROCESS_CAPTURE_EXAMPLE } from "../../../src/domain/process/processCaptureExample.js"; +import { PROCESS_PROVIDER } from "../../../src/domain/process/processEvidenceProvider.js"; import { createServer } from "../../../src/server/createServer.js"; it("returns the complete session Evidence bundle in the tool result", async () => { @@ -44,3 +54,251 @@ it("returns the complete session Evidence bundle in the tool result", async () = await Promise.allSettled([client.close(), server.close(), session.close()]); } }); + +it("rejects bundle field stripping at MCP import without changing retained evidence", async () => { + const root = await createTestTempDirectory("rea-evidence-bundle-mcp-"); + const sourceSession = createBundleSourceSession(); + const targetSession = createTestBinarySession(() => ({ + execute: () => Promise.resolve(observed(null)), + close: () => Promise.resolve(resultOk(null)), + })); + const source = await connectEvidenceMcp("bundle-source", sourceSession); + const target = await connectEvidenceMcp("bundle-target", targetSession); + const { client: sourceClient } = source; + const { client: targetClient } = target; + const bundlePath = join(root, "exported.json"); + const targetExportPath = join(root, "round-trip.json"); + + try { + const { sourceJson, records, evidenceIndex } = await exportFixtureBundle( + sourceClient, + bundlePath, + ); + const initial = structured( + await targetClient.callTool({ + name: "get_evidence_bundle", + arguments: {}, + }), + ).result; + for (const [label, path] of unsupportedBundleFields(evidenceIndex)) { + const candidate = structuredClone(sourceJson); + setPath(candidate, path, "unsupported-metadata"); + const malformedPath = join(root, `${label.replaceAll(" ", "-")}.json`); + await writeFile(malformedPath, JSON.stringify(candidate)); + const rejected = await targetClient.callTool({ + name: "import_evidence_bundle", + arguments: { path: malformedPath }, + }); + expect(parseMcpToolError(rejected), label).toMatchObject({ + error: { code: "evidence_integrity_mismatch" }, + }); + expect( + structured( + await targetClient.callTool({ + name: "get_evidence_bundle", + arguments: {}, + }), + ).result, + label, + ).toEqual(initial); + } + + const imported = await targetClient.callTool({ + name: "import_evidence_bundle", + arguments: { path: bundlePath }, + }); + if (imported.isError === true) + throw new Error(JSON.stringify(parseMcpToolError(imported))); + await targetClient.callTool({ + name: "export_evidence_bundle", + arguments: { path: targetExportPath }, + }); + expect(JSON.parse(await readFile(targetExportPath, "utf8"))).toEqual( + sourceJson, + ); + const observedRecord = records[evidenceIndex]!; + expect(observedRecord.parameters).toEqual({ caller: { note: "preserve" } }); + expect(observedRecord.normalized_result).toEqual({ + producer: { extension: "preserve" }, + }); + expect(object(observedRecord.analysis_profile).parameters).toEqual({ + loader: { mode: "full" }, + }); + } finally { + await Promise.allSettled([ + sourceClient.close(), + source.server.close(), + sourceSession.close(), + targetClient.close(), + target.server.close(), + targetSession.close(), + ]); + } +}); + +const createBundleSourceSession = () => { + const provider = { id: "fixture", name: "Fixture", version: "1" }; + const evidence = createEvidence(ANALYSIS_SNAPSHOT_TARGET, provider, { + operation: "observe", + parameters: { caller: { note: "preserve" } }, + result: { producer: { extension: "preserve" } }, + analysisProfile: createAnalysisProfile(provider, { + loader: { mode: "full" }, + }), + environment: { + id: "fixture-linux", + platform: "linux", + architecture: "x86_64", + isolation: "process", + }, + locations: [{ kind: "address", address: "0x1000" }], + }); + const session = createTestBinarySession(() => ({ + execute: () => Promise.resolve(observed(null)), + close: () => Promise.resolve(resultOk(null)), + })); + expect(session.recordEvidence(evidence).ok).toBe(true); + for (const extra of [ + createEvidence(ANALYSIS_SNAPSHOT_TARGET, provider, { + operation: "controlled_replay", + parameters: {}, + result: { completed: true }, + authority: "controlled-replay", + }), + createEvidence(undefined, PROCESS_PROVIDER, { + predicateType: "rea.process-capture", + operation: "capture_process_scenario", + parameters: {}, + result: jsonValueSchema.parse(EMPTY_PROCESS_CAPTURE_EXAMPLE), + confidence: "observed", + authority: "controlled-replay", + }), + ]) + expect(session.recordEvidence(extra).ok).toBe(true); + return session; +}; + +const exportFixtureBundle = async ( + client: Client, + path: string, +): Promise<{ + readonly sourceJson: unknown; + readonly records: Record[]; + readonly evidenceIndex: number; +}> => { + const unknown = await client.callTool({ + name: "record_unknown", + arguments: { + question: "Does this remain unknown?", + severity: "medium", + domain: "bundle-format", + required_authority: null, + required_confidence: "derived", + required_environment: null, + recommended_probes: [{ operation: "observe", rationale: "Inspect it." }], + relationships: [], + }, + }); + expect(unknown.isError).not.toBe(true); + const exported = await client.callTool({ + name: "export_evidence_bundle", + arguments: { path }, + }); + expect(exported.isError).not.toBe(true); + const sourceJson = JSON.parse(await readFile(path, "utf8")) as unknown; + const records = array(object(sourceJson).records).map(object); + const evidenceIndex = records.findIndex( + (record) => record.operation === "observe", + ); + expect(evidenceIndex).toBeGreaterThanOrEqual(0); + return { sourceJson, records, evidenceIndex }; +}; + +const unsupportedBundleFields = ( + evidenceIndex: number, +): readonly [string, readonly (string | number)[]][] => [ + ["bundle envelope", ["schema_version"]], + ["record provider", ["records", evidenceIndex, "provider", "future_field"]], + ["provider manifest", ["providers", 0, "future_field"]], + ["record subject", ["records", evidenceIndex, "subject", "future_field"]], + [ + "subject digest", + ["records", evidenceIndex, "subject", "digest", "future_field"], + ], + ["artifact manifest", ["artifacts", 0, "future_field"]], + [ + "record environment", + ["records", evidenceIndex, "environment", "future_field"], + ], + ["environment manifest", ["environments", 0, "future_field"]], + [ + "source location", + ["records", evidenceIndex, "locations", 0, "future_field"], + ], + ["scenario manifest", ["scenarios", 0, "future_field"]], + ["capture manifest", ["captures", 0, "future_field"]], + [ + "analysis profile", + ["records", evidenceIndex, "analysis_profile", "future_field"], + ], + [ + "profile provider", + ["records", evidenceIndex, "analysis_profile", "provider", "future_field"], + ], + ["unknown revision", ["unknowns", 0, "future_field"]], + ["unknown probe", ["unknowns", 0, "recommended_probes", 0, "future_field"]], +]; + +const connectEvidenceMcp = async ( + name: string, + session: ReturnType, +) => { + const server = createServer({ kind: "session", session }); + const client = new Client({ name, version: "1" }); + const [clientTransport, serverTransport] = + InMemoryTransport.createLinkedPair(); + await server.connect(serverTransport); + await client.connect(clientTransport); + return { client, server }; +}; + +const object = (value: unknown): Record => { + if (typeof value !== "object" || value === null || Array.isArray(value)) + throw new Error("Expected object in exported Evidence bundle"); + return value as Record; +}; + +const array = (value: unknown): unknown[] => { + if (!Array.isArray(value)) + throw new Error("Expected array in Evidence bundle"); + return value; +}; + +const setPath = ( + root: unknown, + path: readonly (string | number)[], + value: unknown, +): void => { + const [last, ...parents] = [...path].reverse(); + if (last === undefined) throw new Error("Expected a nonempty bundle path"); + const target = parents + .reverse() + .reduce( + (current, segment) => + typeof segment === "number" + ? array(current)[segment] + : object(current)[segment], + root, + ); + object(target)[String(last)] = value; +}; + +const structured = (result: Awaited>) => { + if (result.isError === true) throw parseMcpToolError(result); + if ( + typeof result.structuredContent !== "object" || + result.structuredContent === null + ) + throw new Error("Missing structured MCP result"); + return result.structuredContent as Record; +}; diff --git a/tests/boundary/mcp/evidenceImportLifecycle.test.ts b/tests/boundary/mcp/evidenceImportLifecycle.test.ts new file mode 100644 index 000000000..f0371ef89 --- /dev/null +++ b/tests/boundary/mcp/evidenceImportLifecycle.test.ts @@ -0,0 +1,155 @@ +import { Client, InMemoryTransport } from "@modelcontextprotocol/client"; +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect, it, onTestFinished, vi } from "vitest"; + +import { createEvidence } from "../../../src/domain/evidence.js"; +import { createEvidenceBundle } from "../../../src/domain/evidenceBundle.js"; +import { OwnedFileHandle } from "../../../src/filesystem/OwnedFileHandle.js"; +import { createServer } from "../../../src/server/createServer.js"; +import { + createDeferred, + createTestBinarySession, +} from "../../fixtures/binarySession.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +const connectImportSession = async () => { + const root = await createTestTempDirectory("rea-mcp-import-lifecycle-"); + const session = createTestBinarySession(() => { + throw new Error("Evidence import must not start a binary provider"); + }); + const server = createServer({ kind: "session", session }); + const client = new Client({ + name: "evidence-import-lifecycle", + version: "1", + }); + const readCompleted = createDeferred(); + const releaseRead = createDeferred(); + const originalFileClose = OwnedFileHandle.prototype.close; + vi.spyOn(OwnedFileHandle.prototype, "close").mockImplementationOnce( + async function (this: OwnedFileHandle) { + await originalFileClose.call(this); + readCompleted.resolve(); + await releaseRead.promise; + }, + ); + const closeStarted = createDeferred(); + const originalSessionClose = session.close.bind(session); + const events: string[] = []; + vi.spyOn(session, "close").mockImplementation((...args) => { + const closing = originalSessionClose(...args); + closeStarted.resolve(); + return closing.then((result) => { + events.push("close"); + return result; + }); + }); + const originalImport = session.importEvidenceBundle.bind(session); + vi.spyOn(session, "importEvidenceBundle").mockImplementation((bundle) => { + const imported = originalImport(bundle); + events.push("import"); + return imported; + }); + onTestFinished(async () => { + releaseRead.resolve(); + vi.restoreAllMocks(); + await client.close(); + await server.close(); + await session.close(); + }); + const [clientTransport, serverTransport] = + InMemoryTransport.createLinkedPair(); + const laterImportDispatched = createDeferred(); + let importsDispatched = 0; + const send = clientTransport.send.bind(clientTransport); + vi.spyOn(clientTransport, "send").mockImplementation( + async (message, ...args) => { + await send(message, ...args); + if ( + "method" in message && + message.method === "tools/call" && + message.params?.name === "import_evidence_bundle" + ) { + importsDispatched += 1; + if (importsDispatched === 2) laterImportDispatched.resolve(); + } + }, + ); + await server.connect(serverTransport); + await client.connect(clientTransport); + const importBundle = async (operation: string) => { + const record = createEvidence( + undefined, + { id: "fixture", name: "Fixture", version: "1" }, + { operation, parameters: {}, result: { retained: operation } }, + ); + const path = join(root, `${operation}.json`); + await writeFile(path, JSON.stringify(createEvidenceBundle([record]))); + return client.callTool({ + name: "import_evidence_bundle", + arguments: { path }, + }); + }; + return { + client, + session, + events, + importBundle, + readCompleted: readCompleted.promise, + closeStarted: closeStarted.promise, + laterImportDispatched: laterImportDispatched.promise, + releaseRead: () => releaseRead.resolve(), + }; +}; + +it("drains a file import before close_binary clears the session", async () => { + const fixture = await connectImportSession(); + const importing = fixture.importBundle("before_close"); + await fixture.readCompleted; + const closing = fixture.client.callTool({ + name: "close_binary", + arguments: {}, + }); + await fixture.closeStarted; + await fixture.client.ping(); + fixture.releaseRead(); + const [imported, closed] = await Promise.all([importing, closing]); + + expect(imported.isError, JSON.stringify(imported)).not.toBe(true); + expect(imported.structuredContent).toMatchObject({ + result: { imported: 1, unknowns_added: 0, total: 1 }, + }); + expect(closed.isError, JSON.stringify(closed)).not.toBe(true); + expect(fixture.events).toEqual(["import", "close"]); + expect(fixture.session.exportEvidenceBundle().records).toEqual([]); + await fixture.client.ping(); +}); + +it("admits a later import after close_binary without retaining the earlier bundle", async () => { + const fixture = await connectImportSession(); + const earlier = fixture.importBundle("before_close"); + await fixture.readCompleted; + const closing = fixture.client.callTool({ + name: "close_binary", + arguments: {}, + }); + await fixture.closeStarted; + const later = fixture.importBundle("after_close"); + await fixture.laterImportDispatched; + await fixture.client.ping(); + fixture.releaseRead(); + const results = await Promise.all([earlier, closing, later]); + + for (const result of results) + expect(result.isError, JSON.stringify(result)).not.toBe(true); + expect(results[2]?.structuredContent).toMatchObject({ + result: { imported: 1, unknowns_added: 0, total: 1 }, + }); + expect(fixture.events).toEqual(["import", "close", "import"]); + expect( + fixture.session + .exportEvidenceBundle() + .records.map((record) => record.operation), + ).toEqual(["after_close"]); + await fixture.client.ping(); +}); diff --git a/tests/boundary/mcp/evmInterfaceMcp.test.ts b/tests/boundary/mcp/evmInterfaceMcp.test.ts index f3594dd89..823a4d3cf 100644 --- a/tests/boundary/mcp/evmInterfaceMcp.test.ts +++ b/tests/boundary/mcp/evmInterfaceMcp.test.ts @@ -107,6 +107,30 @@ it("advertises exact valid schemas and records inline EVM interface evidence wit ).toBe(true); }); +it("rejects an EVM provider response with a non-file subject before recording Evidence", async () => { + const value = evmInterfaceFixture(); + const execution = evmInterfaceExecution(value); + const malformed = { + ...execution, + subject: { ...execution.subject, format: "elf" }, + } as unknown as typeof execution; + const harness = await createApplicationMcpHarness({ + serverOptions: { + evmInterface: new EvmInterfaceService({ + inspect: () => Promise.resolve(ok(malformed)), + }), + }, + }); + onTestFinished(harness.close); + const before = harness.session.exportEvidenceBundle().records.length; + const response = await harness.client.callTool({ + name: "inspect_evm_interface", + arguments: { path: value.artifact.path, encoding: "hex" }, + }); + expect(parseMcpToolError(response).error.code).toBe("unreadable_output"); + expect(harness.session.exportEvidenceBundle().records).toHaveLength(before); +}); + it.each(["unavailable", "incompatible-output"])( "preserves a healthy WASM peer and explicit EVM failure: %s", async (scenario) => { diff --git a/tests/boundary/mcp/ghidraEvidenceMcp.test.ts b/tests/boundary/mcp/ghidraEvidenceMcp.test.ts index c28fcca39..ed82400aa 100644 --- a/tests/boundary/mcp/ghidraEvidenceMcp.test.ts +++ b/tests/boundary/mcp/ghidraEvidenceMcp.test.ts @@ -46,6 +46,8 @@ it("retains terminal-call qualifications across function views and snapshot repl limitations: [limitation], }, ], + ["procedure_callees", ["0x402000"]], + ["procedure_callers", [identity.address]], ["analyze_function", jsonValueSchema.parse(dossier)], ]); const harness = await connectGhidraMcp( @@ -81,7 +83,7 @@ it("retains terminal-call qualifications across function views and snapshot repl }); expect(evidence.limitations, name).toContain(limitation); expect(evidence.analysis_profile?.parameters).toMatchObject({ - function_boundary_observations: "ghidra-terminal-call-limitations-v1", + function_boundary_observations: "ghidra-terminal-call-limitations-v2", }); const replay = await harness.session.execute(name, evidence.parameters); if (!replay.ok) throw replay.error; diff --git a/tests/boundary/mcp/ghidraMcpHarness.ts b/tests/boundary/mcp/ghidraMcpHarness.ts index 7e2c3ae73..a9b2248dc 100644 --- a/tests/boundary/mcp/ghidraMcpHarness.ts +++ b/tests/boundary/mcp/ghidraMcpHarness.ts @@ -95,7 +95,7 @@ export const sessionEvidence = (session: BinarySession, value: unknown) => { const installationHost = (): GhidraInstallationHost => ({ platform: "linux", architecture: "x64", - readText: () => "application.version=12.1.4\n", + readText: () => "application.version=12.1.4\napplication.java.min=21\n", executable: () => true, probeJava: () => ({ version: "21.0.11", @@ -263,6 +263,8 @@ const resultFor = ( const functionObservationOperations: ReadonlySet = new Set([ "analyze_function", "procedure_assembly", + "procedure_callees", + "procedure_callers", "procedure_info", "procedure_pseudo_code", "read_function_instructions", diff --git a/tests/boundary/mcp/ghidraReopen.test.ts b/tests/boundary/mcp/ghidraReopen.test.ts index bdac57233..6a90f0118 100644 --- a/tests/boundary/mcp/ghidraReopen.test.ts +++ b/tests/boundary/mcp/ghidraReopen.test.ts @@ -99,7 +99,7 @@ class FixtureLauncher implements GhidraLauncher { const installationHost: GhidraInstallationHost = { platform: "linux", architecture: "x64", - readText: () => "application.version=12.1.4\n", + readText: () => "application.version=12.1.4\napplication.java.min=21\n", executable: () => true, probeJava: () => ({ version: "21.0.11", diff --git a/tests/boundary/mcp/nativeAddressContextMcp.test.ts b/tests/boundary/mcp/nativeAddressContextMcp.test.ts new file mode 100644 index 000000000..5b855fc26 --- /dev/null +++ b/tests/boundary/mcp/nativeAddressContextMcp.test.ts @@ -0,0 +1,121 @@ +import { afterEach, expect, it } from "vitest"; +import { Ajv2020 } from "ajv/dist/2020.js"; +import type { AnySchemaObject } from "ajv"; +import { + closeEnhancedToolResources, + connect, + jsonResult, +} from "./enhancedToolsHarness.js"; +import { observed as ok } from "../../fixtures/analysisExecution.js"; +import type { SegmentSummary } from "../../../src/domain/hopperValues.js"; + +afterEach(closeEnhancedToolResources); + +const segment = { + name: ".text", + start: "0x101000", + end: "0x101100", + readable: true, + writable: false, + executable: true, + image_base: "0x100000", + address_space: "ram", +}; +const overviewClient = (segments: SegmentSummary[]) => + connect({ + execute: (name) => { + switch (name) { + case "current_document": + return Promise.resolve(ok("fixture")); + case "list_documents": + return Promise.resolve(ok(["fixture"])); + case "list_segments": + return Promise.resolve(ok(segments)); + case "list_procedures": + return Promise.resolve(ok([{ address: "0x101013", value: "entry" }])); + case "list_strings": + return Promise.resolve(ok([])); + default: + throw new Error(`Unexpected operation: ${name}`); + } + }, + }); + +it("preserves observed coordinates and main's optional-string output contract after SDK conversion", async () => { + const client = await overviewClient([segment]); + const schema = (await client.listTools()).tools.find( + ({ name }) => name === "binary_overview", + )?.outputSchema; + if (schema === undefined) throw new Error("Missing overview output schema"); + const validate = new Ajv2020({ strict: false }).compile( + schema as AnySchemaObject, + ); + const response = await client.callTool({ + name: "binary_overview", + arguments: {}, + }); + expect(validate(response.structuredContent)).toBe(true); + expect(jsonResult(response)).toMatchObject({ + image_base: "0x100000", + segments: [ + { start: "0x101000", image_base: "0x100000", address_space: "ram" }, + ], + }); + const absent = JSON.parse(JSON.stringify(response.structuredContent)); + delete absent.normalized_result.image_base; + expect(validate(absent)).toBe(true); + absent.normalized_result.image_base = null; + expect(validate(absent)).toBe(false); +}); + +it("omits unreported bases without inventing zero and preserves explicit zero", async () => { + const { image_base: _base, ...unreported } = segment; + const client = await overviewClient([unreported]); + expect( + jsonResult( + await client.callTool({ name: "binary_overview", arguments: {} }), + ), + ).not.toHaveProperty("image_base"); + const zero = await overviewClient([{ ...segment, image_base: "0x0" }]); + expect( + jsonResult(await zero.callTool({ name: "binary_overview", arguments: {} })), + ).toMatchObject({ image_base: "0x0" }); +}); + +it("keeps a partially reported base unknown while retaining the reported segment observation", async () => { + const { image_base: _base, ...unreported } = segment; + const client = await overviewClient([ + segment, + { ...unreported, name: ".data" }, + ]); + const result = jsonResult( + await client.callTool({ name: "binary_overview", arguments: {} }), + ); + expect(result).not.toHaveProperty("image_base"); + expect(result).toMatchObject({ + segments: [{ image_base: "0x100000" }, { name: ".data" }], + }); +}); + +it("retains all conflicting observations after repeated early bases without selecting one", async () => { + const { address_space: _space, ...unreported } = segment; + const client = await overviewClient([ + ...Array.from({ length: 40 }, () => unreported), + { + ...segment, + name: ".data", + start: "other%20space:0x201000", + end: "other%20space:0x201100", + image_base: "0x200000", + address_space: "other space", + }, + ]); + const result = jsonResult( + await client.callTool({ name: "binary_overview", arguments: {} }), + ); + expect(result).not.toHaveProperty("image_base"); + expect(result).toHaveProperty("segments.length", 41); + expect(result).toHaveProperty("segments.40.image_base", "0x200000"); + expect(result).toHaveProperty("segments.40.address_space", "other space"); + expect(result).not.toHaveProperty("segments.0.address_space"); +}); diff --git a/tests/boundary/mcp/nativeMachoSummaryMcp.test.ts b/tests/boundary/mcp/nativeMachoSummaryMcp.test.ts new file mode 100644 index 000000000..e492263fa --- /dev/null +++ b/tests/boundary/mcp/nativeMachoSummaryMcp.test.ts @@ -0,0 +1,216 @@ +import { silentLogger } from "../../../src/logger.js"; +import { copyFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { describe, expect } from "vitest"; +import { z } from "zod"; + +import { createBinarySession } from "../../../src/composition/binary.js"; +import { parseConfig } from "../../../src/config/parseConfig.js"; +import { inspectMachoSchema } from "../../../src/domain/native/nativeInspection.js"; +import { createServer } from "../../../src/server/createServer.js"; +import { thinMach } from "../../../src/domain/binaryTarget.fixture.js"; +import { createTestBinarySession } from "../../fixtures/binarySession.js"; +import { observed } from "../../fixtures/analysisExecution.js"; +import { ok } from "../../../src/domain/result.js"; +import { parseMcpToolError } from "../../fixtures/mcpToolError.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { mcpTest } from "../../support/mcp/mcpFixture.js"; + +const evidenceOf = (schema: Schema) => + z.object({ + evidence_id: z.string(), + parameters: z.record(z.string(), z.unknown()), + normalized_result: schema, + }); +const countSchema = z.object({ + returned: z.number(), + total: z.number().nullable(), + exhaustive: z.boolean(), +}); +const summarySchema = z.object({ + kind: z.literal("summary"), + parent_evidence_id: z.string(), + summary: z.object({ + uuid: z.string().nullable(), + collections: z.record(z.string(), countSchema), + provenance_count: z.number(), + }), +}); +const pageSchema = z.object({ + kind: z.literal("macho"), + item: z.array(z.unknown()), + coverage: z.object({ next_offset: z.number().nullable() }), +}); + +describe.skipIf(process.platform !== "darwin")( + "Mach-O summary delivery through MCP", + () => { + mcpTest( + "retains complete inspect_macho Evidence and pages its exact rows", + async ({ mcp, onTestFinished }) => { + const target = join( + await createTestTempDirectory("rea-macho-summary-mcp-"), + "ls", + ); + await copyFile("/bin/ls", target); + const configured = parseConfig({ REA_ANALYSIS_PROVIDER: "auto" }); + if (!configured.ok) throw configured.error; + const session = createBinarySession( + configured.value, + silentLogger, + process.env, + ); + onTestFinished(async () => { + await session.close(); + }); + const client = await mcp.connect( + createServer({ kind: "session", session }), + ); + const opened = await client.callTool({ + name: "open_binary", + arguments: { path: target }, + }); + expect(opened.isError, JSON.stringify(opened.content)).not.toBe(true); + + const complete = await client.callTool({ + name: "inspect_macho", + arguments: {}, + }); + expect(complete.isError, JSON.stringify(complete.content)).not.toBe( + true, + ); + const full = evidenceOf(inspectMachoSchema).parse( + complete.structuredContent, + ); + // Omitted detail keeps the default Evidence parameters unchanged. + expect(full.parameters).toEqual({}); + + const summarized = await client.callTool({ + name: "inspect_macho", + arguments: { detail: "summary" }, + }); + expect(summarized.isError, JSON.stringify(summarized.content)).not.toBe( + true, + ); + const summary = evidenceOf(summarySchema).parse( + summarized.structuredContent, + ).normalized_result; + // Delivery detail never reaches the tools or the source Evidence ID. + expect(summary.parent_evidence_id).toBe(full.evidence_id); + const report = full.normalized_result; + expect(summary.summary.uuid).toBe(report.uuid); + expect(summary.summary.provenance_count).toBe(report.provenance.length); + for (const name of ["imports", "exports", "dependencies"] as const) + expect(summary.summary.collections[name]).toEqual({ + returned: report[name].items.length, + total: report[name].total, + exhaustive: report[name].exhaustive, + }); + expect(JSON.stringify(summarized.structuredContent)).not.toContain( + report.imports.items[0]?.name ?? "_", + ); + + // Every facet must reassemble the unparsed complete rows exactly. + const raw = z + .object({ normalized_result: z.record(z.string(), z.unknown()) }) + .parse(complete.structuredContent).normalized_result; + const facets = [ + "entrypoints", + "architectures", + "build_metadata", + "load_commands", + "dependencies", + "imports", + "exports", + "segments", + "provenance", + ] as const; + for (const facet of facets) { + const rows: unknown[] = []; + let offset: number | null = 0; + while (offset !== null) { + const paged = await client.callTool({ + name: "inspect_analysis_view", + arguments: { + source: { + kind: "retained-evidence", + evidence_id: summary.parent_evidence_id, + }, + view: { kind: "macho", facet, offset, limit: 16 }, + }, + }); + expect(paged.isError, JSON.stringify(paged.content)).not.toBe(true); + const page = evidenceOf(pageSchema).parse( + paged.structuredContent, + ).normalized_result; + rows.push(...page.item); + offset = page.coverage.next_offset; + } + expect(rows, facet).toEqual( + facet === "provenance" + ? raw.provenance + : z.object({ items: z.array(z.unknown()) }).parse(raw[facet]) + .items, + ); + } + + const wrongKind = await client.callTool({ + name: "inspect_analysis_view", + arguments: { + source: { + kind: "retained-evidence", + evidence_id: summary.parent_evidence_id, + }, + view: { kind: "wasm", facet: "exports" }, + }, + }); + expect(wrongKind.isError).toBe(true); + expect(JSON.stringify(wrongKind.content)).toContain( + "Mach-O inspection supports summary and macho views.", + ); + }, + ); + }, +); + +mcpTest( + "refuses summary delivery before analysis when the server retains no Evidence", + async ({ mcp, onTestFinished }) => { + const target = join( + await createTestTempDirectory("rea-summary-preflight-"), + "tool", + ); + await writeFile(target, thinMach(0xfeedfacf, 0x0100000c)); + const executed: string[] = []; + const session = createTestBinarySession(() => ({ + execute: (operation) => { + executed.push(operation); + return Promise.resolve(observed(null)); + }, + close: () => Promise.resolve(ok(null)), + })); + onTestFinished(async () => { + await session.close(); + }); + const opened = await session.open(target); + if (!opened.ok) throw opened.error; + const client = await mcp.connect( + createServer({ kind: "fixed", analysis: session }), + ); + const refused = []; + for (const name of ["inspect_macho", "inspect_artifact"]) + refused.push( + await client.callTool({ name, arguments: { detail: "summary" } }), + ); + // Neither analysis ran only to be discarded by the delivery refusal. + expect(executed.filter((operation) => operation !== "health")).toEqual([]); + for (const result of refused) + expect(parseMcpToolError(result)).toMatchObject({ + error: { + code: "capability_unavailable", + message: expect.stringContaining("does not retain Evidence"), + }, + }); + }, +); diff --git a/tests/boundary/mcp/packetCaptureMcp.test.ts b/tests/boundary/mcp/packetCaptureMcp.test.ts index e11b60d3e..a5acf1925 100644 --- a/tests/boundary/mcp/packetCaptureMcp.test.ts +++ b/tests/boundary/mcp/packetCaptureMcp.test.ts @@ -8,42 +8,53 @@ import { Ajv2020 } from "ajv/dist/2020.js"; import { createApplicationMcpHarness } from "../../fixtures/applicationMcpHarness.js"; import { parseMcpToolError } from "../../fixtures/mcpToolError.js"; -it("advertises valid target-free packet capture schemas and actionable missing-tool diagnostics", async () => { - const harness = await createApplicationMcpHarness({ - serverOptions: { environment: {} }, - }); - onTestFinished(harness.close); - const tool = (await harness.client.listTools()).tools.find( - ({ name }) => name === "inspect_packet_capture", - ); - if (tool?.outputSchema === undefined) throw new Error("Capture tool missing"); - const ajv = new Ajv2020({ strict: false, validateFormats: false }); - const inputSchema: Record = tool.inputSchema; - const outputSchema: Record = tool.outputSchema; - expect(ajv.validateSchema(inputSchema)).toBe(true); - expect(ajv.validateSchema(outputSchema)).toBe(true); - const input = { - path: - process.platform === "win32" ? "C:\\recording.pcap" : "/recording.pcap", - fields: ["ip.addr"], - }; - expect(ajv.validate(inputSchema, input)).toBe(true); - expect(ajv.validate(inputSchema, { ...input, interface: "lo0" })).toBe(false); - const result = await harness.client.callTool({ - name: tool.name, - arguments: input, - }); - expect(parseMcpToolError(result)).toMatchObject({ - error: { - code: "capability_unavailable", - details: { - operation: tool.name, - reason: expect.stringContaining("REA_TSHARK_COMMAND"), +it.each(["inspect_packet_capture", "inspect_packet_fields"])( + "advertises valid target-free schemas and missing-tool diagnostics: %s", + async (name) => { + const harness = await createApplicationMcpHarness({ + serverOptions: { environment: {} }, + }); + onTestFinished(harness.close); + const tool = (await harness.client.listTools()).tools.find( + (tool) => tool.name === name, + ); + if (tool?.outputSchema === undefined) + throw new Error("Capture tool missing"); + const ajv = new Ajv2020({ strict: false, validateFormats: false }); + const inputSchema: Record = tool.inputSchema; + const outputSchema: Record = tool.outputSchema; + expect(ajv.validateSchema(inputSchema)).toBe(true); + expect(ajv.validateSchema(outputSchema)).toBe(true); + const input = + name === "inspect_packet_fields" + ? { query: "usb" } + : { + path: + process.platform === "win32" + ? "C:\\recording.pcap" + : "/recording.pcap", + fields: ["ip.addr"], + }; + expect(ajv.validate(inputSchema, input)).toBe(true); + expect(ajv.validate(inputSchema, { ...input, interface: "lo0" })).toBe( + false, + ); + const result = await harness.client.callTool({ + name: tool.name, + arguments: input, + }); + expect(parseMcpToolError(result)).toMatchObject({ + error: { + code: "capability_unavailable", + details: { + operation: tool.name, + reason: expect.stringContaining("REA_TSHARK_COMMAND"), + }, }, - }, - }); - await harness.client.ping(); -}); + }); + await harness.client.ping(); + }, +); it.each(["cancelled", "cleanup"])( "retains completed Evidence after %s failure", @@ -78,6 +89,9 @@ it.each(["cancelled", "cleanup"])( }, ); const packetCapture = new PacketCaptureService({ + inspectFields: async () => { + throw new Error("Unexpected field discovery call"); + }, inspect: async () => err(failure), close: async () => {}, }); @@ -120,6 +134,9 @@ it.each(["cancelled", "cleanup"])( it("rejects oversized selector expansion through the MCP boundary before invoking the provider", async () => { let calls = 0; const packetCapture = new PacketCaptureService({ + inspectFields: async () => { + throw new Error("Unexpected field discovery call"); + }, inspect: async () => { calls++; return err(new AnalysisCancelledError("inspect_packet_capture")); diff --git a/tests/boundary/mcp/pcapMcp.test.ts b/tests/boundary/mcp/pcapMcp.test.ts new file mode 100644 index 000000000..d3a6d203d --- /dev/null +++ b/tests/boundary/mcp/pcapMcp.test.ts @@ -0,0 +1,118 @@ +import { Client, InMemoryTransport } from "@modelcontextprotocol/client"; +import { expect, it, onTestFinished } from "vitest"; + +import type { PcapAnalysisPort } from "../../../src/application/pcap/PcapAnalysisPort.js"; +import { createAnalysisExecution } from "../../../src/application/AnalysisProvider.js"; +import { assertAdvertisedMcpContracts } from "./mcpContractHarness.js"; +import { parseEvidence } from "../../../src/domain/evidence.js"; +import { ok } from "../../../src/domain/result.js"; +import { createServer } from "../../../src/server/createServer.js"; +import { createTestBinarySession } from "../../fixtures/binarySession.js"; + +const identity = { id: "pcap", name: "Packet capture analysis", version: null }; + +const clientIdentity = { + tshark_command: "/usr/bin/tshark", + tshark_source: "path", + tshark_version: "TShark (Wireshark) 4.6.6", + capinfos_command: "/usr/bin/capinfos", + capinfos_source: "path", + capinfos_version: "Capinfos (Wireshark) 4.6.6", +}; + +it("advertises exact valid pcap schemas and records inline evidence through MCP", async () => { + const provider: PcapAnalysisPort = { + async inspectAvailability() { + return { status: "available", code: null, reason: null, diagnostics: {} }; + }, + async close() {}, + async execute(request) { + if (request.operation === "inspect_pcap_client") + return ok( + createAnalysisExecution({ client: clientIdentity }, identity), + ); + return ok( + createAnalysisExecution( + { + client: clientIdentity, + target: { + path: "/captures/session.pcap", + bytes: 23279, + sha256: "a".repeat(64), + }, + file_type: "Wireshark/tcpdump/... - pcap", + encapsulation: "Linux cooked-mode capture v2", + packet_count: 50, + capture_start: "2026-10-11 14:13:40.608505", + capture_end: "2026-10-11 14:13:40.981978", + capture_duration_seconds: 0.373473, + snapshot_length: 262144, + endpoints: [ + { address: "192.168.1.100", port: null, packet_count: 30 }, + { address: "8.8.8.8", port: 53, packet_count: 5 }, + ], + endpoint_count: 2, + dns_queries: [ + { + query: "example.com", + response_addresses: ["93.184.216.34"], + response_address_count: 1, + }, + ], + dns_query_count: 1, + tls_sni_values: ["example.com"], + tls_sni_count: 1, + protocol_distribution: [ + { protocol: "ip:tcp:tls", packet_count: 25 }, + { protocol: "ip:udp:dns", packet_count: 5 }, + ], + protocol_count: 2, + http_request_uris: ["http://example.com/api"], + http_request_uri_count: 1, + http_uri_redaction_count: 0, + unattributed_dns_addresses: [], + unattributed_dns_address_count: 0, + unattributed_dns_packet_count: 0, + coverage: "complete", + }, + identity, + ), + ); + }, + }; + const session = createTestBinarySession(() => { + throw new Error("deep provider must not start"); + }); + const server = createServer( + { kind: "session", session }, + { pcapAnalysis: provider }, + ); + onTestFinished(() => server.close()); + const [clientTransport, serverTransport] = + InMemoryTransport.createLinkedPair(); + await server.connect(serverTransport); + const client = new Client({ name: "pcap-boundary", version: "0.0.0" }); + onTestFinished(() => client.close()); + await client.connect(clientTransport); + + const { tools } = await client.listTools(); + assertAdvertisedMcpContracts( + ["inspect_pcap_client", "inspect_pcap_capture"], + tools, + ); + + const inspected = await client.callTool({ + name: "inspect_pcap_capture", + arguments: { path: "/captures/session.pcap" }, + }); + expect(inspected.isError).not.toBe(true); + const text = (inspected.content as { readonly text?: string }[]) + .map((part) => part.text ?? "") + .join(""); + const evidence = parseEvidence(JSON.parse(text)); + expect(evidence.normalized_result).toMatchObject({ + packet_count: 50, + tls_sni_values: ["example.com"], + coverage: "complete", + }); +}); diff --git a/tests/boundary/mcp/processComparisonDiagnosticsMcp.test.ts b/tests/boundary/mcp/processComparisonDiagnosticsMcp.test.ts index 69fdc3944..bda4672de 100644 --- a/tests/boundary/mcp/processComparisonDiagnosticsMcp.test.ts +++ b/tests/boundary/mcp/processComparisonDiagnosticsMcp.test.ts @@ -252,6 +252,56 @@ it("reports CLI trace specification issues like the MCP trace_spec input", async }); }); +it("rejects a deep trace cycle as an input error through CLI and MCP", async () => { + const events = Array.from({ length: 6000 }, (_, index) => ({ + id: `e${String(index)}`, + source: "lifecycle", + exact: { marker: index }, + })); + const trace = { + events, + language: { + kind: "partial_order", + happens_before: events.map((event, index) => ({ + before: event.id, + after: events[(index + 1) % events.length]!.id, + })), + not_before: events.map((event, index) => ({ + event: event.id, + anchor: events[(index + 1) % events.length]!.id, + })), + unordered_groups: [{ events: events.map(({ id }) => id) }], + }, + }; + const issue = { + path: ["trace_spec", "language"], + message: "happens-before graph must be acyclic", + }; + const capture = captureEvidence("left"); + const root = await createTestTempDirectory("rea-process-trace-cycle-"); + const capturePath = join(root, "capture.json"); + const specPath = join(root, "spec.json"); + await writeFile(capturePath, JSON.stringify(capture)); + await writeFile(specPath, JSON.stringify(trace)); + expect( + await compareProcessEvidenceFiles(capturePath, capturePath, specPath), + ).toMatchObject({ + code: "invalid_request", + details: { issues: [issue] }, + }); + + const client = await connect(); + const response = await client.callTool({ + name: "compare_process_captures", + arguments: { left: capture, right: capture, trace_spec: trace }, + }); + expect(response.isError).toBe(true); + expect(response.content).toEqual([ + { type: "text", text: expect.stringContaining(issue.message) }, + ]); + await client.ping(); +}); + it("names the side and constraint when Evidence is not a usable capture", async () => { const client = await connect(); const capture = captureEvidence("right"); diff --git a/tests/boundary/mcp/processEvidenceIdentityMcp.test.ts b/tests/boundary/mcp/processEvidenceIdentityMcp.test.ts index fb20dd79b..993f1df69 100644 --- a/tests/boundary/mcp/processEvidenceIdentityMcp.test.ts +++ b/tests/boundary/mcp/processEvidenceIdentityMcp.test.ts @@ -309,7 +309,9 @@ it("separately rejects a tampered digest before admission to any consumer or ses }; const digestMismatch = /semantic identifier does not match/u; expect(() => parseEvidence(tampered)).toThrow(digestMismatch); - expect(() => findChangedBehavior([tampered])).toThrow(digestMismatch); + expect(() => findChangedBehavior({ comparisons: [tampered] })).toThrow( + digestMismatch, + ); expect(() => correlateStaticAndRuntime(correlationInput(tampered))).toThrow( digestMismatch, ); @@ -336,6 +338,19 @@ it("separately rejects a tampered digest before admission to any consumer or ses } }); +it("maps malformed changed-behavior Evidence to an invalid request before closure lookup", async () => { + const { client, comparison } = await connectedComparison(); + const malformed = { ...comparison, evidence_links: undefined } as unknown; + const response = await client.callTool({ + name: "find_changed_behavior", + arguments: { comparisons: [malformed] }, + }); + expect(response.isError).toBe(true); + expect(parseMcpToolError(response)).toMatchObject({ + error: { code: "invalid_request" }, + }); +}); + it("keeps withheld ownership unknown through CLI and MCP comparison", async () => { const { client, right } = await connectedComparison(); const left = captureEvidence("left", { diff --git a/tests/boundary/mcp/processTraceAdaptersParity.test.ts b/tests/boundary/mcp/processTraceAdaptersParity.test.ts index cb7c1439c..d8c9b6173 100644 --- a/tests/boundary/mcp/processTraceAdaptersParity.test.ts +++ b/tests/boundary/mcp/processTraceAdaptersParity.test.ts @@ -9,76 +9,16 @@ import { createTestBinarySession } from "../../fixtures/binarySession.js"; import type { AnalysisClient } from "../../../src/application/AnalysisProvider.js"; import { compareProcessEvidenceFiles } from "../../../src/application/process/ProcessCli.js"; import { PROCESS_PROVIDER } from "../../../src/domain/process/processEvidenceProvider.js"; -import { EMPTY_PROCESS_CAPTURE_EXAMPLE } from "../../../src/domain/process/processCaptureExample.js"; +import { processTraceCaptureFixture } from "../../../src/domain/process/processCapture.fixture.js"; import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js"; import { jsonValueSchema } from "../../../src/domain/jsonValue.js"; -import type { ProcessTraceSpecification } from "../../../src/domain/process/processTraceSpecification.js"; import { createServer } from "../../../src/server/createServer.js"; import { observed as ok } from "../../fixtures/analysisExecution.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; const resources: Array<{ close(): Promise }> = []; const temporaryRoots: string[] = []; -const beforeCheckpoint = { - name: "before", - at_ms: 0, - files: [], - effects: [], - truncated: false, -}; -const afterCheckpoint = { - name: "after_settlement", - at_ms: 50, - files: [], - effects: [], - truncated: false, -}; -const capture = { - ...EMPTY_PROCESS_CAPTURE_EXAMPLE, - event_journal: [ - { capture_order: 0, collection: "filesystem_checkpoints", index: 0 }, - { capture_order: 1, collection: "lifecycle", index: 0 }, - { capture_order: 2, collection: "lifecycle", index: 1 }, - { capture_order: 3, collection: "filesystem_checkpoints", index: 1 }, - ], -} as const; -const traceSpecification: ProcessTraceSpecification = { - events: [ - { - id: "before", - source: "filesystem", - exact: beforeCheckpoint, - cardinality: { kind: "required" }, - }, - { - id: "exit", - source: "lifecycle", - exact: { event: "exit", ...capture.exit }, - cardinality: { kind: "required" }, - }, - { - id: "settlement", - source: "lifecycle", - exact: { event: "settlement", ...capture.settlement }, - cardinality: { kind: "required" }, - }, - { - id: "after", - source: "filesystem", - exact: afterCheckpoint, - cardinality: { kind: "required" }, - }, - ], - language: { - kind: "finite_traces", - variants: [ - { - id: "normal", - trace: ["before", "exit", "settlement", "after"], - }, - ], - }, -}; +const { capture, traceSpecification } = processTraceCaptureFixture(); const captureEvidence = (side: "left" | "right") => createEvidence(undefined, PROCESS_PROVIDER, { diff --git a/tests/boundary/mcp/rawBinaryMcp.test.ts b/tests/boundary/mcp/rawBinaryMcp.test.ts index c1d2029c2..26a0e6501 100644 --- a/tests/boundary/mcp/rawBinaryMcp.test.ts +++ b/tests/boundary/mcp/rawBinaryMcp.test.ts @@ -1,4 +1,4 @@ -import { writeFile } from "node:fs/promises"; +import { mkdir, symlink, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { Client, InMemoryTransport } from "@modelcontextprotocol/client"; @@ -10,7 +10,8 @@ import { createServer } from "../../../src/server/createServer.js"; import { toolContract } from "../../../src/contracts/toolContracts.js"; import type { AnalysisProvider } from "../../../src/application/AnalysisProvider.js"; import { createAnalysisProfile } from "../../../src/domain/analysisProfile.js"; -import { ok } from "../../../src/domain/result.js"; +import { ProviderAdapterError } from "../../../src/domain/providerAdapterError.js"; +import { err, ok } from "../../../src/domain/result.js"; import { observed } from "../../fixtures/analysisExecution.js"; import { createCacheProvider, @@ -18,6 +19,7 @@ import { } from "../../fixtures/binarySession.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { silentLogger } from "../../../src/logger.js"; +import { GHIDRA_PROVIDER_IDENTITY } from "../../../src/ghidra/GhidraProviderCapabilities.js"; import type { CallToolResult } from "@modelcontextprotocol/server"; const connect = async ( @@ -50,29 +52,97 @@ const structured = (result: CallToolResult): Record => { return z.record(z.string(), z.unknown()).parse(result.structuredContent); }; -it("opens a headerless image when architecture and entry are supplied", async () => { - const root = await createTestTempDirectory("rea-raw-mcp-"); - const path = join(root, "ecu.bin"); - await writeFile(path, Buffer.from("1eff2fe1", "hex")); - const { client } = await connect(); - const opened = await client.callTool({ - name: "open_binary", - arguments: { +const withGhidraFixtureIdentity = ( + provider: AnalysisProvider, +): AnalysisProvider => { + const identity = { ...GHIDRA_PROVIDER_IDENTITY, version: "12.1.4" }; + const profile = createAnalysisProfile(identity, { fixture: true }); + return { + ...provider, + identity: () => identity, + capabilities: () => + provider.capabilities().map((capability) => ({ + ...capability, + provider: identity, + })), + resolveAnalysisProfile: () => Promise.resolve(ok({ profile })), + createClient: (target, _profile, context) => { + const client = provider.createClient(target, profile, context); + return { + ...client, + execute: async (operation, parameters, options) => { + const result = await client.execute(operation, parameters, options); + return result.ok + ? ok({ + ...result.value, + provider: identity, + analysisProfile: profile, + }) + : result; + }, + }; + }, + }; +}; + +it("keeps 6502 out of detected lifecycle output contracts", () => { + const schema = toolContract("open_binary").outputSchema; + const result = { + path: "/fixture/image.bin", + kind: "executable", + sha256: "a".repeat(64), + architecture: "6502", + }; + expect( + schema.safeParse({ result: { ...result, format: "pe" } }).success, + ).toBe(false); + expect( + schema.safeParse({ + result: { ...result, format: "raw-binary", entry_point: "0x8001" }, + }).success, + ).toBe(true); + const validateAdvertised = new Ajv2020({ strict: false }).compile( + z.toJSONSchema(schema), + ); + expect(validateAdvertised({ result: { ...result, format: "pe" } })).toBe( + false, + ); + expect( + validateAdvertised({ + result: { ...result, format: "raw-binary", entry_point: "0x8001" }, + }), + ).toBe(true); +}); + +it.each([ + ["arm", "0x08000000", "0x8000000", "1eff2fe1"], + ["6502", "0x008001", "0x8001", "a92a60"], +])( + "opens a headerless %s image when architecture and entry are supplied", + async (architecture, entryPoint, canonical, bytes) => { + const root = await createTestTempDirectory("rea-raw-mcp-"); + const path = join(root, "ecu.bin"); + await writeFile(path, Buffer.from(bytes, "hex")); + const { client } = await connect(); + const opened = await client.callTool({ + name: "open_binary", + arguments: { + path, + format: "raw-binary", + architecture, + entry_point: entryPoint, + }, + }); + expect(opened.isError).not.toBe(true); + expect(structured(opened).result).toMatchObject({ path, format: "raw-binary", - architecture: "arm", - entry_point: "0x08000000", - }, - }); - expect(opened.isError).not.toBe(true); - expect(structured(opened).result).toMatchObject({ - path, - format: "raw-binary", - kind: "executable", - architecture: "arm", - entry_point: "0x8000000", - }); -}); + kind: "executable", + architecture, + entry_point: canonical, + }); + }, +); it("rejects an unaligned AArch64 entry before selecting or starting a provider", async () => { const root = await createTestTempDirectory("rea-raw-mcp-alignment-"); @@ -217,3 +287,118 @@ it("reuses a canonical raw entry and replaces the session when the entry changes expect(created).toEqual(["0x8000000", "0x9000000"]); expect(closed).toEqual(["0x8000000"]); }); + +it.skipIf(process.platform !== "win32")( + "does not replay a saved read after switching to a Windows junction", + async () => { + const root = await createTestTempDirectory("rea-raw-mcp-linked-reopen-"); + const actual = join(root, "actual"); + const linked = join(root, "linked"); + await mkdir(actual); + const path = join(actual, "image.bin"); + await writeFile(path, Buffer.from("a92a60", "hex")); + await symlink(actual, linked, "junction"); + const created: string[] = []; + const executed: string[] = []; + const fixture = createCacheProvider([]); + const { client } = await connect( + withGhidraFixtureIdentity({ + ...fixture, + createClient: (target) => { + created.push(target.sourcePath ?? target.path); + return { + execute: (operation) => { + if (operation === "address_name") + executed.push(target.sourcePath ?? target.path); + if ( + operation === "address_name" && + target.sourcePath === join(linked, "image.bin") + ) + return Promise.resolve( + err(new ProviderAdapterError("fixture", operation)), + ); + return Promise.resolve( + observed(operation === "address_name" ? "entry" : null), + ); + }, + close: () => Promise.resolve(ok(null)), + }; + }, + }), + ); + const linkedPath = join(linked, "image.bin"); + for (const requested of [path, linkedPath]) { + const opened = await client.callTool({ + name: "open_binary", + arguments: { + path: requested, + format: "raw-binary", + architecture: "6502", + entry_point: "0x8001", + }, + }); + expect(opened.isError, JSON.stringify(opened)).not.toBe(true); + const address = await client.callTool({ + name: "address_name", + arguments: { address: "0x8001", document: "image.bin" }, + }); + expect(address.isError === true, JSON.stringify(address)).toBe( + requested === linkedPath, + ); + } + expect(created).toEqual([path, linkedPath]); + expect(executed).toEqual([path, linkedPath]); + }, +); + +it.skipIf(process.platform !== "win32")( + "reuses saved reads after a fixture admits a noncanonical Windows path", + async () => { + const root = await createTestTempDirectory("rea-raw-mcp-admitted-path-"); + const actual = join(root, "actual"); + const linked = join(root, "linked"); + await mkdir(actual); + const path = join(actual, "image.bin"); + await writeFile(path, Buffer.from("a92a60", "hex")); + await symlink(actual, linked, "junction"); + const calls: string[] = []; + const fixture = createCacheProvider([]); + // This fixture accepts the alias to isolate cache behavior after admission. + // The real Ghidra verifier checks that a junction itself is rejected. + const { client, session } = await connect( + withGhidraFixtureIdentity({ + ...fixture, + createClient: () => ({ + execute: (operation) => { + if (operation === "address_name") calls.push(operation); + return Promise.resolve( + observed(operation === "health" ? null : "entry"), + ); + }, + close: () => Promise.resolve(ok(null)), + }), + }), + ); + const linkedPath = join(linked, "image.bin"); + const opened = await client.callTool({ + name: "open_binary", + arguments: { + path: linkedPath, + format: "raw-binary", + architecture: "6502", + entry_point: "0x8001", + }, + }); + expect(opened.isError, JSON.stringify(opened)).not.toBe(true); + expect(session.activeTarget()?.sourcePath).toBe(linkedPath); + expect(session.activeTarget()?.path).toBe(path); + for (const _ of [0, 1]) { + const address = await client.callTool({ + name: "address_name", + arguments: { address: "0x8001", document: "image.bin" }, + }); + expect(address.isError, JSON.stringify(address)).not.toBe(true); + } + expect(calls).toEqual(["address_name"]); + }, +); diff --git a/tests/boundary/mcp/reconstructionCoverageMcp.test.ts b/tests/boundary/mcp/reconstructionCoverageMcp.test.ts index cb3605d14..1e167b8b0 100644 --- a/tests/boundary/mcp/reconstructionCoverageMcp.test.ts +++ b/tests/boundary/mcp/reconstructionCoverageMcp.test.ts @@ -3,7 +3,10 @@ import { ok as resultOk } from "../../../src/domain/result.js"; import { Client, InMemoryTransport } from "@modelcontextprotocol/client"; import { describe, expect, it } from "vitest"; -import { completeReconstructionCoverageData } from "../../../src/domain/reconstructionCoverage.fixture.js"; +import { + completeReconstructionCoverageData, + reconstructionCoverageEvidenceId, +} from "../../../src/domain/reconstructionCoverage.fixture.js"; import { createReconstructionCoverageData } from "../../../src/domain/reconstructionCoverage.js"; import { createTestBinarySession } from "../../fixtures/binarySession.js"; import { createServer } from "../../../src/server/createServer.js"; @@ -36,6 +39,42 @@ describe("reconstruction coverage MCP", () => { boundary_id: boundaryId, summary: { reasons: 0 }, }); + const coverage = currentFixtureCoverage(boundaryId); + const latest = coverage.verifier_results[0]; + if (latest === undefined) + throw new Error("Expected fixture verifier result"); + const ambiguous = await client.callTool({ + name: "evaluate_reconstruction_coverage", + arguments: { + coverage: createReconstructionCoverageData({ + ...coverage, + verifier_results: [ + latest, + { + ...latest, + status: "fail", + evidence_ids: [reconstructionCoverageEvidenceId("1")], + }, + ], + }), + boundary_id: boundaryId, + }, + }); + expect(ambiguous.isError).not.toBe(true); + expect(ambiguous.structuredContent).toMatchObject({ + status: "unknown", + boundary_id: boundaryId, + reasons: [ + expect.objectContaining({ + code: "verifier-result-ambiguous", + subject_id: "claim.cli.help", + }), + ], + evidence_ids: expect.arrayContaining([ + reconstructionCoverageEvidenceId("3"), + reconstructionCoverageEvidenceId("1"), + ]), + }); const unknown = await client.callTool({ name: "evaluate_reconstruction_coverage", arguments: { diff --git a/tests/boundary/mcp/serverLifecycleMcp.test.ts b/tests/boundary/mcp/serverLifecycleMcp.test.ts index c841e08ed..cdec09a3e 100644 --- a/tests/boundary/mcp/serverLifecycleMcp.test.ts +++ b/tests/boundary/mcp/serverLifecycleMcp.test.ts @@ -34,6 +34,9 @@ import { } from "../../support/workspace/workspaceFixture.js"; import { z } from "zod"; import { createDeferred } from "../../fixtures/binarySession.js"; +import { GoBinaryService } from "../../../src/application/go/GoBinaryService.js"; +import { WebNetworkCaptureService } from "../../../src/application/WebNetworkCaptureService.js"; +import { AnalysisUnsupportedTargetError } from "../../../src/domain/analysisErrorCore.js"; const resources: Array<{ close(): Promise }> = []; @@ -334,6 +337,68 @@ it("awaits SQLite cleanup and retries its retained owner after failed server shu expect(attempts).toBe(2); }); +it("closes the bound Go and web-capture services and retries failed shutdown", async () => { + let goCalls = 0; + let goCloseAttempts = 0; + let captureCloseAttempts = 0; + let cleanupAllowed = false; + const goBinary = new GoBinaryService({ + identity: { id: "go-fixture", name: "Go fixture", version: "1" }, + inspect: async ({ path }) => { + goCalls += 1; + return err( + new AnalysisUnsupportedTargetError( + "inspect_go_binary", + path, + "fixture rejection", + ), + ); + }, + close: async () => { + goCloseAttempts += 1; + if (!cleanupAllowed) throw new Error("Go cleanup refused"); + }, + }); + const webNetworkCapture = new WebNetworkCaptureService({ + inspect: async () => { + throw new Error("No capture inspection requested"); + }, + close: async () => { + captureCloseAttempts += 1; + if (!cleanupAllowed) throw new Error("Capture cleanup refused"); + }, + }); + const server = createServer( + { kind: "fixed", analysis: { execute: async () => ok(null) } }, + { goBinary, webNetworkCapture }, + ); + const client = new Client({ name: "bound-service-lifecycle", version: "1" }); + const [clientTransport, serverTransport] = + InMemoryTransport.createLinkedPair(); + resources.push(client, server); + try { + await server.connect(serverTransport); + await client.connect(clientTransport); + const result = await client.callTool({ + name: "inspect_go_binary", + arguments: { path: "/tmp/go-fixture" }, + }); + expect(result.isError).toBe(true); + expect(goCalls).toBe(1); + + await expect(server.close()).rejects.toBeInstanceOf(AggregateError); + expect(goCloseAttempts).toBe(1); + expect(captureCloseAttempts).toBe(1); + + cleanupAllowed = true; + await server.close(); + expect(goCloseAttempts).toBe(2); + expect(captureCloseAttempts).toBe(2); + } finally { + cleanupAllowed = true; + } +}); + afterEach(async () => { await Promise.all( resources.splice(0).map(async (resource) => resource.close()), diff --git a/tests/boundary/mcp/serverUnknownsMcp.test.ts b/tests/boundary/mcp/serverUnknownsMcp.test.ts index 0a27324ec..e39ce2aec 100644 --- a/tests/boundary/mcp/serverUnknownsMcp.test.ts +++ b/tests/boundary/mcp/serverUnknownsMcp.test.ts @@ -281,7 +281,7 @@ it("pages filtered unknown heads through the public MCP contract", async () => { supporting_evidence_ids: [], contradicting_evidence_ids: [], }); - expect(result.ok).toBe(true); + if (!result.ok) throw result.error; } const defaultPage = structured( @@ -372,12 +372,11 @@ it("pages filtered unknown heads through the public MCP contract", async () => { .result as typeof defaultPage; expect(first).toMatchObject({ total: 3, offset: 0, next_offset: 2 }); expect(first.items).toHaveLength(2); - const second = structured( - await client.callTool({ - ...request, - arguments: { ...request.arguments, offset: first.next_offset ?? 0 }, - }), - ).result as typeof defaultPage; + const secondResponse = await client.callTool({ + ...request, + arguments: { ...request.arguments, offset: first.next_offset ?? 0 }, + }); + const second = structured(secondResponse).result as typeof defaultPage; expect(second).toMatchObject({ total: 3, offset: 2, next_offset: null }); expect(second.items).toHaveLength(1); expect( @@ -389,7 +388,7 @@ it("pages filtered unknown heads through the public MCP contract", async () => { ); expect( toolContract("list_unknowns").outputSchema.safeParse( - (await client.callTool(request)).structuredContent, + secondResponse.structuredContent, ).success, ).toBe(true); diff --git a/tests/boundary/mcp/serverWorkflowMcp.test.ts b/tests/boundary/mcp/serverWorkflowMcp.test.ts index c72fca5cb..f25027b2a 100644 --- a/tests/boundary/mcp/serverWorkflowMcp.test.ts +++ b/tests/boundary/mcp/serverWorkflowMcp.test.ts @@ -122,11 +122,18 @@ it("advertises the complete currently available inventory with a session", async const listed = await client.listTools(); const names = listed.tools.map((t) => t.name); + const sessionTool = listed.tools.find( + ({ name }) => name === "binary_session", + ); + if (sessionTool === undefined) + throw new Error("Missing advertised binary_session tool"); + // Validate this reply with its current advertised schema. The SDK otherwise + // builds output validators for the entire catalog on the first cached call. const status = structured( - await client.callTool({ - name: "binary_session", - arguments: {}, - }), + await client.callTool( + { name: "binary_session", arguments: {} }, + { toolDefinition: sessionTool }, + ), ); const inventory = expandToolAvailability( toolContract("binary_session").outputSchema.parse(status).result diff --git a/tests/boundary/mcp/signalRecordingMcp.test.ts b/tests/boundary/mcp/signalRecordingMcp.test.ts index ecabfd8e5..edd74a2bf 100644 --- a/tests/boundary/mcp/signalRecordingMcp.test.ts +++ b/tests/boundary/mcp/signalRecordingMcp.test.ts @@ -8,41 +8,51 @@ import { Ajv2020 } from "ajv/dist/2020.js"; import { createApplicationMcpHarness } from "../../fixtures/applicationMcpHarness.js"; import { parseMcpToolError } from "../../fixtures/mcpToolError.js"; -it("advertises valid target-free signal decoding schemas and actionable missing-tool diagnostics", async () => { - const harness = await createApplicationMcpHarness({ - serverOptions: { environment: {} }, - }); - onTestFinished(harness.close); - const tool = (await harness.client.listTools()).tools.find( - ({ name }) => name === "decode_signal_recording", - ); - if (tool?.outputSchema === undefined) throw new Error("Signal tool missing"); - const ajv = new Ajv2020({ strict: false, validateFormats: false }); - const inputSchema: Record = tool.inputSchema; - const outputSchema: Record = tool.outputSchema; - expect(ajv.validateSchema(inputSchema)).toBe(true); - expect(ajv.validateSchema(outputSchema)).toBe(true); - const input = { - path: process.platform === "win32" ? "C:\\recording.vcd" : "/recording.vcd", - decoders: "uart", - }; - expect(ajv.validate(inputSchema, input)).toBe(true); - expect(ajv.validate(inputSchema, { ...input, driver: "demo" })).toBe(false); - const result = await harness.client.callTool({ - name: tool.name, - arguments: input, - }); - expect(parseMcpToolError(result)).toMatchObject({ - error: { - code: "capability_unavailable", - details: { - operation: tool.name, - reason: expect.stringContaining("REA_SIGROK_COMMAND"), +it.each(["decode_signal_recording", "inspect_signal_decoders"])( + "advertises valid target-free schemas and missing-tool diagnostics: %s", + async (name) => { + const harness = await createApplicationMcpHarness({ + serverOptions: { environment: {} }, + }); + onTestFinished(harness.close); + const tool = (await harness.client.listTools()).tools.find( + (tool) => tool.name === name, + ); + if (tool?.outputSchema === undefined) + throw new Error("Signal tool missing"); + const ajv = new Ajv2020({ strict: false, validateFormats: false }); + const inputSchema: Record = tool.inputSchema; + const outputSchema: Record = tool.outputSchema; + expect(ajv.validateSchema(inputSchema)).toBe(true); + expect(ajv.validateSchema(outputSchema)).toBe(true); + const input = + name === "inspect_signal_decoders" + ? { query: "uart" } + : { + path: + process.platform === "win32" + ? "C:\\recording.vcd" + : "/recording.vcd", + decoders: "uart", + }; + expect(ajv.validate(inputSchema, input)).toBe(true); + expect(ajv.validate(inputSchema, { ...input, driver: "demo" })).toBe(false); + const result = await harness.client.callTool({ + name: tool.name, + arguments: input, + }); + expect(parseMcpToolError(result)).toMatchObject({ + error: { + code: "capability_unavailable", + details: { + operation: tool.name, + reason: expect.stringContaining("REA_SIGROK_COMMAND"), + }, }, - }, - }); - await harness.client.ping(); -}); + }); + await harness.client.ping(); + }, +); it.each(["cancelled", "cleanup"])( "retains completed Evidence after %s failure", @@ -77,6 +87,9 @@ it.each(["cancelled", "cleanup"])( }, ); const signalRecording = new SignalRecordingService({ + inspectDecoders: async () => { + throw new Error("Unexpected discovery call"); + }, decode: async () => err(failure), close: async () => {}, }); diff --git a/tests/boundary/mcp/wasmArtifactMcp.test.ts b/tests/boundary/mcp/wasmArtifactMcp.test.ts index 49c7ad762..003a37cba 100644 --- a/tests/boundary/mcp/wasmArtifactMcp.test.ts +++ b/tests/boundary/mcp/wasmArtifactMcp.test.ts @@ -213,6 +213,31 @@ it("advertises exact valid schemas and records inline WASM interface evidence wi expect(close).toHaveBeenCalled(); }); +it("rejects a WASM provider response with a non-file subject before recording Evidence", async () => { + const value = wasmArtifactFixture(); + const execution = wasmArtifactExecution(value); + const malformed = { + ...execution, + subject: { ...execution.subject, format: "elf" }, + } as unknown as typeof execution; + const harness = await createApplicationMcpHarness({ + serverOptions: { + wasmArtifact: new WasmArtifactService({ + close: () => Promise.resolve(), + inspect: () => Promise.resolve(ok(malformed)), + }), + }, + }); + onTestFinished(harness.close); + const before = harness.session.exportEvidenceBundle().records.length; + const response = await harness.client.callTool({ + name: "inspect_wasm_artifact", + arguments: { path: value.artifact.path }, + }); + expect(parseMcpToolError(response).error.code).toBe("unreadable_output"); + expect(harness.session.exportEvidenceBundle().records).toHaveLength(before); +}); + it("reads an oversized completed WASM result through its reported retained reference", async () => { const text = "(module)\n" + ";;😀\n" + " ".repeat(6 * 1024 * 1024); const value = { diff --git a/tests/boundary/native/PwndbgCoreContext.test.ts b/tests/boundary/native/PwndbgCoreContext.test.ts index e8f007a9b..d38b01ac3 100644 --- a/tests/boundary/native/PwndbgCoreContext.test.ts +++ b/tests/boundary/native/PwndbgCoreContext.test.ts @@ -2,6 +2,7 @@ import { access } from "node:fs/promises"; import { join } from "node:path"; import { expect, it, onTestFinished } from "vitest"; import { AnalysisCancelledError } from "../../../src/domain/analysisErrorCore.js"; +import { ArtifactResourceScope } from "../../../src/artifacts/ArtifactResourceScope.js"; import { createTestWorkspace, removeTestWorkspace, @@ -40,6 +41,7 @@ it.each(["pre-aborted", "aborted-during-probe"] as const)( REA_PWNDBG_VENV_PATH: join(workspace.root, "missing-venv"), }, rootPath: workspace.root, + resources: new ArtifactResourceScope(), diagnostics, options: { signal: controller.signal }, probePath, diff --git a/tests/boundary/native/pwntoolsArtifactCleanup.test.ts b/tests/boundary/native/pwntoolsArtifactCleanup.test.ts new file mode 100644 index 000000000..82760cfd2 --- /dev/null +++ b/tests/boundary/native/pwntoolsArtifactCleanup.test.ts @@ -0,0 +1,147 @@ +import { mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { expect, it, vi } from "vitest"; +import { z } from "zod"; + +const closeState = vi.hoisted(() => ({ + fail: false, + failOnce: false, + opened: 0, + closes: 0, +})); + +vi.mock("node:fs/promises", async (importOriginal) => { + const fs = await importOriginal(); + return { + ...fs, + open: async (...args: Parameters) => { + const handle = await fs.open(...args); + closeState.opened++; + const close = handle.close.bind(handle); + handle.close = async () => { + closeState.closes++; + if (closeState.fail || closeState.failOnce) { + closeState.failOnce = false; + throw new Error("injected stable artifact close failure"); + } + return close(); + }; + return handle; + }, + }; +}); + +import { PwntoolsDecoder } from "../../../src/native/pwntools/PwntoolsDecoder.js"; +import { PWNTOOLS_PROVIDER_IDENTITY } from "../../../src/native/pwntools/PwntoolsRelease.js"; +import { projectAnalysisError } from "../../../src/domain/analysisErrorProjection.js"; + +const makeDecoder = () => + new PwntoolsDecoder( + { + identity: PWNTOOLS_PROVIDER_IDENTITY, + operation: "test_pwntools_artifact_read", + bridge: new URL("file:///unused-bridge.py"), + payloadSchema: z.string(), + project: () => { + throw new Error("input read should fail before decoder launch"); + }, + }, + { REA_PWNTOOLS_PYTHON: process.execPath }, + undefined, + async () => { + throw new Error("runtime should not be created for failed input read"); + }, + ); + +const emulateSupportedHost = async ( + run: () => Promise, +): Promise => { + const platform = Object.getOwnPropertyDescriptor(process, "platform"); + const arch = Object.getOwnPropertyDescriptor(process, "arch"); + Object.defineProperty(process, "platform", { ...platform, value: "linux" }); + Object.defineProperty(process, "arch", { ...arch, value: "x64" }); + try { + await run(); + } finally { + if (platform !== undefined) + Object.defineProperty(process, "platform", platform); + if (arch !== undefined) Object.defineProperty(process, "arch", arch); + } +}; + +it("reports the read failure without stale incomplete-cleanup state when close retry succeeds", async () => + emulateSupportedHost(async () => { + const root = await mkdtemp(join(tmpdir(), "rea-pwntools-close-retry-")); + const path = join(root, "selected.bin"); + await writeFile(path, "selected evidence"); + closeState.fail = false; + closeState.failOnce = true; + closeState.opened = 0; + closeState.closes = 0; + const decoder = makeDecoder(); + try { + const result = await decoder.inspect({ path }); + expect(result.ok).toBe(false); + if (result.ok) throw new Error("Injected close failure was not returned"); + expect(result.error.cleanupIncomplete).toBe(false); + expect(projectAnalysisError(result.error).code).not.toBe( + "cleanup_incomplete", + ); + expect(result.error.cause).toMatchObject({ + cleanup: { + resources: [path], + reason: expect.stringContaining( + "injected stable artifact close failure", + ), + }, + }); + expect(closeState.closes).toBeGreaterThanOrEqual(2); + } finally { + closeState.fail = false; + closeState.failOnce = false; + await decoder.close(); + await rm(root, { recursive: true, force: true }); + } + })); + +it("retains a persistent failed input descriptor and refuses the next acquisition", async () => + emulateSupportedHost(async () => { + const root = await mkdtemp(join(tmpdir(), "rea-pwntools-held-close-")); + const path = join(root, "selected.bin"); + await writeFile(path, "selected evidence"); + closeState.fail = true; + closeState.failOnce = false; + closeState.opened = 0; + closeState.closes = 0; + const decoder = makeDecoder(); + try { + const first = await decoder.inspect({ path }); + expect(first.ok).toBe(false); + if (first.ok) + throw new Error("Persistent close failure was not returned"); + expect(projectAnalysisError(first.error).code).toBe("cleanup_incomplete"); + const openedAfterFailure = closeState.opened; + + const retry = await decoder.inspect({ path }); + expect(retry.ok).toBe(false); + if (retry.ok) throw new Error("Retained descriptor was not retried"); + expect(projectAnalysisError(retry.error).code).toBe("cleanup_incomplete"); + expect(closeState.opened).toBe(openedAfterFailure); + + await expect(decoder.close()).rejects.toMatchObject({ + cleanupIncomplete: true, + }); + closeState.fail = false; + await decoder.close(); + } finally { + closeState.fail = false; + closeState.failOnce = false; + try { + await decoder.close(); + } catch { + // The test already asserts persistent cleanup failure before releasing it. + } + await rm(root, { recursive: true, force: true }); + } + })); diff --git a/tests/boundary/pcap/pcapCommand.test.ts b/tests/boundary/pcap/pcapCommand.test.ts new file mode 100644 index 000000000..cda8dbf77 --- /dev/null +++ b/tests/boundary/pcap/pcapCommand.test.ts @@ -0,0 +1,53 @@ +import { setTimeout as delay } from "node:timers/promises"; +import { expect, it, onTestFinished, vi } from "vitest"; +import { runPcapCommand } from "../../../src/pcap/PcapCommand.js"; +import * as ownership from "../../../src/process/ProcessOwnershipObservation.js"; + +it("does not spend the tool execution deadline preparing native ownership", async () => { + const original = ownership.prepareProcessOwnershipInspection; + let entered!: () => void; + let release!: () => void; + const preparing = new Promise((resolve) => { + entered = resolve; + }); + const gate = new Promise((resolve) => { + release = resolve; + }); + let first = true; + const prepare = vi + .spyOn(ownership, "prepareProcessOwnershipInspection") + .mockImplementation(async (signal) => { + await original(signal); + if (first) { + first = false; + entered(); + await gate; + } + }); + onTestFinished(() => { + prepare.mockRestore(); + release(); + }); + const pending = runPcapCommand({ + operation: "inspect_pcap_client", + command: process.execPath, + arguments: ["--version"], + limits: { timeoutMs: 1000, maxOutputBytes: 1024 }, + environment: {}, + signal: undefined, + retain: (owner) => { + onTestFinished(async () => { + expect((await owner.stop()).status).not.toBe("incomplete"); + }); + }, + }).then( + (value) => ({ ok: true as const, value }), + (error: unknown) => ({ ok: false as const, error }), + ); + await preparing; + await delay(1100); + release(); + const result = await pending; + if (!result.ok) throw result.error; + expect(result.value.stdout.trim()).toBe(process.version); +}); diff --git a/tests/boundary/pcap/pcapProvider.test.ts b/tests/boundary/pcap/pcapProvider.test.ts new file mode 100644 index 000000000..fcb299ade --- /dev/null +++ b/tests/boundary/pcap/pcapProvider.test.ts @@ -0,0 +1,533 @@ +import { createHash } from "node:crypto"; +import { watch } from "node:fs"; +import { chmod, readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect, onTestFinished, test, vi } from "vitest"; + +import { PcapProvider } from "../../../src/pcap/PcapProvider.js"; +import { projectAnalysisError } from "../../../src/domain/analysisErrorProjection.js"; +import { + pcapResultSchemas, + type PcapRequest, +} from "../../../src/domain/pcap/pcapAnalysis.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { holdFirstFileRead } from "../../support/holdFirstFileRead.js"; +import { PrivateRuntimeRoot } from "../../../src/process/PrivateRuntimeRoot.js"; + +const it = test.skipIf(process.platform !== "linux"); + +const CAPINFOS_OUTPUT = [ + "File name: /work/capture.pcap", + "File type: Wireshark/tcpdump/... - pcap", + "File encapsulation: Ethernet", + "Packet size limit: file hdr: 262144 bytes", + "Number of packets: 3", + "Capture duration: 0.123 seconds", +].join("\n"); + +const TSHARK_JSON = JSON.stringify([ + { + _source: { + layers: { + "frame.number": ["1"], + "frame.protocols": ["eth:ethertype:ip:udp:dns"], + "ip.src": ["192.168.1.100"], + "ip.dst": ["8.8.8.8"], + "dns.qry.name": ["api.example.com"], + "dns.flags.response": ["1"], + "dns.a": ["93.184.216.34"], + }, + }, + }, + { + _source: { + layers: { + "frame.number": ["2"], + "frame.protocols": ["eth:ethertype:ip:tcp:tls"], + "ip.src": ["192.168.1.100"], + "ip.dst": ["93.184.216.34"], + "tls.handshake.extensions_server_name": ["api.example.com"], + }, + }, + }, + { + _source: { + layers: { + "frame.number": ["3"], + "frame.protocols": ["eth:ethertype:ip:tcp:http"], + "ip.src": ["192.168.1.100"], + "ip.dst": ["93.184.216.34"], + "http.request.full_uri": ["http://api.example.com/v1/data"], + }, + }, + }, +]); + +const writeStubTools = async ( + root: string, + options: { readonly failCapinfos?: boolean } = {}, +): Promise<{ tshark: string; capinfos: string }> => { + const tshark = join(root, "tshark"); + const capinfos = join(root, "capinfos"); + await writeFile( + tshark, + `#!/bin/sh +if [ "$1" = "--version" ]; then printf 'TShark (Wireshark) 4.6.6-stub\\n'; exit 0; fi +[ "$1" = "-n" ] && [ "$2" = "-r" ] && [ "$4" = "-T" ] && [ "$5" = "json" ] || exit 2 +printf '%s' '${TSHARK_JSON.replace(/'/gu, "'\\''")}' +exit 0 +`, + { mode: 0o755 }, + ); + await writeFile( + capinfos, + `#!/bin/sh +if [ "$1" = "--version" ]; then printf 'Capinfos (Wireshark) 4.6.6-stub\\n'; exit 0; fi +[ "$1" = "-M" ] && [ "$2" = "--" ] && [ "$#" = "3" ] || exit 2 +if [ "${String(options.failCapinfos === true)}" = "true" ]; then + printf 'capinfos: The file is not a capture file' >&2; exit 1 +fi +printf '%s\\n' '${CAPINFOS_OUTPUT.replace(/'/gu, "'\\''")}' +exit 0 +`, + { mode: 0o755 }, + ); + await chmod(tshark, 0o755); + await chmod(capinfos, 0o755); + return { tshark, capinfos }; +}; + +const providerWith = async (options: { failCapinfos?: boolean } = {}) => { + const root = await createTestTempDirectory("rea-pcap-"); + const { tshark, capinfos } = await writeStubTools(root, options); + const provider = new PcapProvider({ + environment: { REA_TSHARK_COMMAND: tshark, REA_CAPINFOS_COMMAND: capinfos }, + }); + onTestFinished(() => provider.close()); + return { + provider, + root, + }; +}; + +const evidenceResult = async ( + provider: PcapProvider, + request: PcapRequest, +): Promise> => { + const result = await provider.execute(request); + if (!result.ok) throw new Error(result.error.message); + return pcapResultSchemas[request.operation].parse(result.value.result); +}; + +const captureFixture = async (root: string): Promise => { + const path = join(root, "capture.pcap"); + await writeFile(path, Buffer.from("pcap-fixture-bytes")); + return path; +}; + +it("inspects both tools without reading a capture", async () => { + const { provider } = await providerWith(); + const result = await evidenceResult(provider, { + operation: "inspect_pcap_client", + input: {}, + }); + expect(result.client).toMatchObject({ + tshark_source: "environment", + tshark_version: "TShark (Wireshark) 4.6.6-stub", + capinfos_source: "environment", + capinfos_version: "Capinfos (Wireshark) 4.6.6-stub", + }); +}); + +it("inspects a capture with endpoints, DNS, TLS SNI, and HTTP", async () => { + const { provider, root } = await providerWith(); + const path = await captureFixture(root); + const result = await evidenceResult(provider, { + operation: "inspect_pcap_capture", + input: { path }, + }); + expect(result.target).toMatchObject({ + path, + bytes: 18, + sha256: createHash("sha256").update("pcap-fixture-bytes").digest("hex"), + }); + expect(result.file_type).toBe("Wireshark/tcpdump/... - pcap"); + expect(result.packet_count).toBe(3); + expect(result.endpoints).toEqual( + expect.arrayContaining([ + { address: "192.168.1.100", port: null, packet_count: 3 }, + { address: "8.8.8.8", port: null, packet_count: 1 }, + ]), + ); + expect(result.dns_queries).toEqual([ + { + query: "api.example.com", + response_addresses: ["93.184.216.34"], + response_address_count: 1, + }, + ]); + expect(result.tls_sni_values).toEqual(["api.example.com"]); + expect(result.http_request_uris).toEqual(["http://api.example.com/v1/data"]); + expect(result.coverage).toBe("complete"); +}); + +it("reports a non-capture file as unsupported", async () => { + const { provider, root } = await providerWith({ failCapinfos: true }); + const failPath = join(root, "fail.pcap"); + await writeFile(failPath, "not a capture file"); + const result = await provider.execute({ + operation: "inspect_pcap_capture", + input: { path: failPath }, + }); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(projectAnalysisError(result.error)).toMatchObject({ + code: "unsupported_target", + }); +}); + +it("reports a missing tshark binary as a capability gap", async () => { + const provider = new PcapProvider({ + environment: { REA_TSHARK_COMMAND: "/nonexistent/tshark" }, + }); + expect(await provider.inspectAvailability()).toMatchObject({ + status: "unavailable", + code: "executable_missing", + }); +}); + +it("rejects a capture that is not a readable file", async () => { + const { provider } = await providerWith(); + const result = await provider.execute({ + operation: "inspect_pcap_capture", + input: { path: "/nonexistent/capture.pcap" }, + }); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(projectAnalysisError(result.error)).toMatchObject({ + code: "invalid_request", + }); +}); + +it("cancels before spawning anything", async () => { + const { provider, root } = await providerWith(); + await writeFile(join(root, "spawned"), ""); + const controller = new AbortController(); + controller.abort(); + const result = await provider.execute( + { + operation: "inspect_pcap_capture", + input: { path: "/some/capture.pcap" }, + }, + { signal: controller.signal }, + ); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(projectAnalysisError(result.error)).toMatchObject({ + code: "cancelled", + }); +}); + +it("reports relative configuration and missing capinfos as capability gaps", async () => { + const root = await createTestTempDirectory("rea-pcap-config-"); + const { tshark } = await writeStubTools(root); + for (const environment of [ + { REA_TSHARK_COMMAND: "relative/tshark" }, + { + REA_TSHARK_COMMAND: tshark, + REA_CAPINFOS_COMMAND: join(root, "missing-capinfos"), + }, + ]) { + const provider = new PcapProvider({ environment }); + onTestFinished(() => provider.close()); + expect((await provider.inspectAvailability()).status).toBe("unavailable"); + const result = await provider.execute({ + operation: "inspect_pcap_client", + input: {}, + }); + if (result.ok) throw new Error("expected unavailable tools"); + const projected = projectAnalysisError(result.error); + expect(projected.code).toBe("capability_unavailable"); + expect(projected.message).toContain( + "Supply Wireshark 4.x tshark and capinfos", + ); + } +}); + +it("names the prlimit selection when the Linux limiter is unusable", async () => { + const root = await createTestTempDirectory("rea-pcap-prlimit-"); + const tools = await writeStubTools(root); + const provider = new PcapProvider({ + environment: { + REA_TSHARK_COMMAND: tools.tshark, + REA_CAPINFOS_COMMAND: tools.capinfos, + REA_TSHARK_PRLIMIT_COMMAND: "relative/prlimit", + }, + }); + onTestFinished(() => provider.close()); + const result = await provider.execute({ + operation: "inspect_pcap_capture", + input: { path: "/not-opened/capture.pcap" }, + }); + if (result.ok) throw new Error("expected an unusable limiter"); + const projected = projectAnalysisError(result.error); + expect(projected.code).toBe("capability_unavailable"); + expect(projected.message).toContain( + "(REA_TSHARK_PRLIMIT_COMMAND must be absolute.) Check that REA_TSHARK_PRLIMIT_COMMAND", + ); +}); + +const nodeTool = async ( + root: string, + name: string, + body: string, +): Promise => { + const path = join(root, name); + await writeFile( + path, + `#!/usr/bin/env -S ${process.execPath} --jitless --max-old-space-size=64\nconst fs = require('node:fs'); const crypto = require('node:crypto');\n${body}\n`, + { mode: 0o755 }, + ); + return path; +}; +const versionProbe = + "if (process.argv[2] === '--version') { console.log((process.argv[1].includes('capinfos') ? 'Capinfos' : 'TShark') + ' (Wireshark) 4.6.6-test'); process.exit(0); }"; + +it("refuses observations whose owned snapshot changed during dissection", async () => { + const root = await createTestTempDirectory("rea-pcap-mutated-"); + const { capinfos } = await writeStubTools(root); + const path = await captureFixture(root); + const before = await readFile(path); + const tshark = await nodeTool( + root, + "tshark.cjs", + `${versionProbe} +const selected = process.argv[process.argv.indexOf('-r') + 1]; +fs.chmodSync(selected, 0o600); fs.writeFileSync(selected, 'changed private snapshot'); +console.log(${JSON.stringify(TSHARK_JSON)});`, + ); + const provider = new PcapProvider({ + environment: { REA_TSHARK_COMMAND: tshark, REA_CAPINFOS_COMMAND: capinfos }, + }); + onTestFinished(() => provider.close()); + const result = await provider.execute({ + operation: "inspect_pcap_capture", + input: { path }, + }); + expect(result.ok).toBe(false); + if (!result.ok) + expect(projectAnalysisError(result.error)).toMatchObject({ + code: "artifact_changed", + }); + expect(await readFile(path)).toEqual(before); +}); + +it("binds both tools to the snapshot when the original capture changes", async () => { + const root = await createTestTempDirectory("rea-pcap-identity-"); + const path = join(root, "caller capture.pcap"); + await writeFile(path, "192.0.2.1"); + const capinfos = await nodeTool( + root, + "capinfos.cjs", + `${versionProbe} +const selected = process.argv.at(-1); const bytes = fs.readFileSync(selected); +console.log('File type: pcap\\nFile encapsulation: Raw IP\\nNumber of packets: 1\\nSHA256: ' + crypto.createHash('sha256').update(bytes).digest('hex')); +fs.writeFileSync(${JSON.stringify(path)}, '192.0.2.2');`, + ); + const tshark = await nodeTool( + root, + "tshark.cjs", + `${versionProbe} +const selected = process.argv[process.argv.indexOf('-r') + 1]; +console.log(JSON.stringify([{_source:{layers:{'frame.number':['1'],'frame.protocols':['ip'],'ip.src':[fs.readFileSync(selected,'utf8')]}}}]));`, + ); + const provider = new PcapProvider({ + environment: { REA_TSHARK_COMMAND: tshark, REA_CAPINFOS_COMMAND: capinfos }, + }); + onTestFinished(() => provider.close()); + const result = await provider.execute({ + operation: "inspect_pcap_capture", + input: { path }, + }); + if (!result.ok) throw result.error; + const value = pcapResultSchemas.inspect_pcap_capture.parse( + result.value.result, + ); + expect(value.target.sha256).toBe( + createHash("sha256").update("192.0.2.1").digest("hex"), + ); + expect(value.endpoints).toEqual([ + { address: "192.0.2.1", port: null, packet_count: 1 }, + ]); + expect(await readFile(path, "utf8")).toBe("192.0.2.2"); +}); + +it.each(["probe", "capinfos", "tshark"] as const)( + "preserves in-flight %s cancellation and cleans up", + async (phase) => { + const root = await createTestTempDirectory("rea-pcap-cancel-"); + const marker = join(root, "ready"); + const slow = await nodeTool( + root, + `slow-${phase}.cjs`, + `${phase !== "probe" ? versionProbe : ""} +fs.writeFileSync(${JSON.stringify(marker)}, 'ready'); setInterval(() => {}, 1000);`, + ); + const { capinfos, tshark } = await writeStubTools(root); + const path = await captureFixture(root); + const provider = new PcapProvider({ + environment: { + REA_TSHARK_COMMAND: phase === "capinfos" ? tshark : slow, + REA_CAPINFOS_COMMAND: phase === "capinfos" ? slow : capinfos, + }, + }); + onTestFinished(() => provider.close()); + const controller = new AbortController(); + let watcher: ReturnType | undefined; + let timer: ReturnType | undefined; + const ready = new Promise((resolve, reject) => { + watcher = watch(root, (_event, name) => { + if (name === "ready") resolve(); + }); + timer = setTimeout(() => { + controller.abort(); + reject(new Error("tool did not start")); + }, 5000); + }); + try { + const pending = provider.execute( + { operation: "inspect_pcap_capture", input: { path } }, + { signal: controller.signal }, + ); + await ready; + controller.abort(new Error("caller cancellation")); + const result = await pending; + if (result.ok) throw new Error("expected cancellation"); + expect(projectAnalysisError(result.error).code).toBe("cancelled"); + await provider.close(); + } finally { + watcher?.close(); + clearTimeout(timer); + } + }, +); + +it.each(["change", "cancel"] as const)( + "preserves %s during snapshot acquisition", + async (action) => { + const { provider, root } = await providerWith(); + const path = await captureFixture(root); + const controller = new AbortController(); + const held = await holdFirstFileRead(path); + onTestFinished(() => { + vi.restoreAllMocks(); + }); + const pending = provider.execute( + { operation: "inspect_pcap_capture", input: { path } }, + { signal: controller.signal }, + ); + try { + await held.firstRead; + if (action === "cancel") controller.abort(); + else await writeFile(path, "changed source bytes"); + } finally { + held.release(); + } + const result = await pending; + if (result.ok) throw new Error("expected interrupted acquisition"); + expect(result.error._tag).toBe( + action === "cancel" + ? "AnalysisCancelledError" + : "AnalysisArtifactChangedError", + ); + }, +); + +it("admits one capture and drains it before close releases resources", async () => { + const { provider, root } = await providerWith(); + const path = await captureFixture(root); + const held = await holdFirstFileRead(path); + onTestFinished(() => { + vi.restoreAllMocks(); + }); + const pending = provider.execute({ + operation: "inspect_pcap_capture", + input: { path }, + }); + let closing: Promise | undefined; + try { + await held.firstRead; + const concurrent = await provider.execute({ + operation: "inspect_pcap_capture", + input: { path }, + }); + expect(concurrent.ok).toBe(false); + if (!concurrent.ok) + expect(projectAnalysisError(concurrent.error)).toMatchObject({ + code: "resource_constraint", + details: { resource: "capacity" }, + }); + let closed = false; + closing = provider.close().then(() => { + closed = true; + }); + await Promise.resolve(); + expect(closed).toBe(false); + } finally { + held.release(); + } + expect((await pending).ok).toBe(true); + await closing; + const afterClose = await provider.execute({ + operation: "inspect_pcap_capture", + input: { path }, + }); + expect(afterClose.ok).toBe(false); +}); + +it.each([false, true])( + "retains failed snapshot cleanup after an analysis failure=%s", + async (failCapinfos) => { + const { provider, root } = await providerWith({ failCapinfos }); + const path = await captureFixture(root); + let retained: PrivateRuntimeRoot | undefined; + const spy = vi + .spyOn(PrivateRuntimeRoot.prototype, "close") + .mockImplementation(function (this: PrivateRuntimeRoot) { + retained = this; + return Promise.reject(new Error("controlled cleanup failure")); + }); + const result = await provider.execute({ + operation: "inspect_pcap_capture", + input: { path }, + }); + try { + expect(result.ok).toBe(false); + if (result.ok) throw new Error("Cleanup failure cannot return success"); + expect(projectAnalysisError(result.error)).toMatchObject({ + code: "cleanup_incomplete", + }); + if (failCapinfos) + expect(projectAnalysisError(result.error)).toMatchObject({ + details: { + diagnostics: { primary_error: { code: "unsupported_target" } }, + }, + }); + else + expect(result.error.partialObservation).toMatchObject({ + operation: "inspect_pcap_capture", + normalized_result: { packet_count: 3 }, + }); + expect(retained).toBeDefined(); + await expect(provider.close()).rejects.toMatchObject({ + cleanupIncomplete: true, + }); + } finally { + spy.mockRestore(); + } + await provider.close(); + await expect( + readFile(join(retained!.path, "capture.pcap")), + ).rejects.toMatchObject({ code: "ENOENT" }); + }, +); diff --git a/tests/boundary/pcap/pcapProviderCrossPlatform.test.ts b/tests/boundary/pcap/pcapProviderCrossPlatform.test.ts new file mode 100644 index 000000000..c19e682fe --- /dev/null +++ b/tests/boundary/pcap/pcapProviderCrossPlatform.test.ts @@ -0,0 +1,153 @@ +/** + * Cross-platform pcap provider tests. + * + * POSIX hosts run the #!/bin/sh fixtures; + * Windows runs the .cmd stubs from tests/fixtures/pcap/. Both validate + * the same provider contract surface: tool probing, capture inspection, + * refusal paths, and cancellation before spawn. + */ +import { createHash } from "node:crypto"; +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect, it, onTestFinished } from "vitest"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +import { PcapProvider } from "../../../src/pcap/PcapProvider.js"; +import { projectAnalysisError } from "../../../src/domain/analysisErrorProjection.js"; +import type { PcapRequest } from "../../../src/domain/pcap/pcapAnalysis.js"; + +const isWindows = process.platform === "win32"; +const hasNativeBound = process.platform === "linux" || isWindows; +const stubDir = join(import.meta.dirname, "../../fixtures/pcap"); +const tsharkStub = isWindows + ? join(stubDir, "tshark-stub.cmd") + : join(stubDir, "tshark-stub.sh"); +const capinfosStub = isWindows + ? join(stubDir, "capinfos-stub.cmd") + : join(stubDir, "capinfos-stub.sh"); + +const providerOver = (): PcapProvider => { + const provider = new PcapProvider({ + environment: { + REA_TSHARK_COMMAND: tsharkStub, + REA_CAPINFOS_COMMAND: capinfosStub, + }, + }); + onTestFinished(() => provider.close()); + return provider; +}; + +const evidenceResult = async ( + provider: PcapProvider, + request: PcapRequest, +): Promise> => { + const result = await provider.execute(request); + if (!result.ok) throw new Error(result.error.message); + return result.value.result as Record; +}; + +it("inspects both tools through cross-platform stubs", async () => { + const result = await evidenceResult(providerOver(), { + operation: "inspect_pcap_client", + input: {}, + }); + expect(result.client).toMatchObject({ + tshark_source: "environment", + tshark_version: expect.stringContaining("4.6.6") as unknown, + capinfos_source: "environment", + capinfos_version: expect.stringContaining("4.6.6") as unknown, + }); +}); + +it.runIf(hasNativeBound)( + "inspects a capture with metadata, endpoints, DNS, and protocol facts", + async () => { + const root = await createTestTempDirectory("rea-pcap-x-"); + const capturePath = join(root, "capture.pcap"); + const captureBytes = "cross-platform-pcap-fixture"; + await writeFile(capturePath, captureBytes); + + const result = await evidenceResult(providerOver(), { + operation: "inspect_pcap_capture", + input: { path: capturePath }, + }); + + expect(result.target).toMatchObject({ + path: capturePath, + bytes: captureBytes.length, + sha256: createHash("sha256").update(captureBytes).digest("hex"), + }); + expect(result.file_type).toContain("pcap"); + expect(result.packet_count).toBe(3); + expect((result.endpoints as unknown[]).length).toBeGreaterThan(0); + expect((result.protocol_distribution as unknown[]).length).toBeGreaterThan( + 0, + ); + expect(result.coverage).toBe("complete"); + }, +); + +// Hosts without a native bound refuse before touching the path; that order +// is pinned by the unsupported-boundary test below. +it.runIf(hasNativeBound)( + "rejects a path that is not a readable file", + async () => { + const result = await providerOver().execute({ + operation: "inspect_pcap_capture", + input: { path: "/nonexistent/definitely-missing.pcap" }, + }); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(projectAnalysisError(result.error)).toMatchObject({ + code: "invalid_request", + }); + }, +); + +it("reports a missing binary as a capability gap", async () => { + const provider = new PcapProvider({ + environment: { REA_TSHARK_COMMAND: "/nonexistent/tshark" }, + }); + expect(await provider.inspectAvailability()).toMatchObject({ + status: "unavailable", + code: "executable_missing", + }); +}); + +it("cancels before spawning anything", async () => { + const controller = new AbortController(); + controller.abort(); + const result = await providerOver().execute( + { operation: "inspect_pcap_capture", input: { path: "/any/capture.pcap" } }, + { signal: controller.signal }, + ); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(projectAnalysisError(result.error)).toMatchObject({ + code: "cancelled", + }); +}); + +it.runIf(!hasNativeBound)( + "reports an unsupported native memory boundary before reading the capture", + async () => { + const provider = providerOver(); + expect(await provider.inspectAvailability()).toMatchObject({ + status: "unavailable", + code: "unsupported_host", + }); + const result = await provider.execute({ + operation: "inspect_pcap_capture", + input: { path: "/not-opened/capture.pcap" }, + }); + expect(result.ok).toBe(false); + if (result.ok) return; + const projected = projectAnalysisError(result.error); + expect(projected.code).toBe("capability_unavailable"); + // The refusal concerns this host, so it must not suggest another target. + expect(projected.message).toContain( + "Linux with util-linux prlimit, or Windows with owned Job Objects", + ); + expect(projected.remediation.action).toBe(projected.message); + }, +); diff --git a/tests/boundary/pcap/pcapRetainedView.test.ts b/tests/boundary/pcap/pcapRetainedView.test.ts new file mode 100644 index 000000000..a3f890a37 --- /dev/null +++ b/tests/boundary/pcap/pcapRetainedView.test.ts @@ -0,0 +1,196 @@ +import { createHash } from "node:crypto"; +import { readFile, rm, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { execFile } from "node:child_process"; +import { promisify } from "node:util"; +import { expect, it } from "vitest"; +import { parseEvidence } from "../../../src/domain/evidence.js"; +import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { parseMcpToolError } from "../../fixtures/mcpToolError.js"; + +const quote = (value: string) => `'${value.replaceAll("'", "'\\''")}'`; + +const createPcapToolFixture = async (root: string) => { + const path = join(root, "caller capture.pcap"); + const bytes = Buffer.from("recording fixture bytes"); + const sha = createHash("sha256").update(bytes).digest("hex"); + await writeFile(path, bytes); + const tshark = join(root, "tshark"); + const capinfos = join(root, "capinfos"); + const marker = join(root, "reads"); + const frames = Array.from({ length: 132 }, (_, index) => ({ + _source: { + layers: { + "frame.number": [String(index + 1)], + "frame.protocols": ["ip:udp:dns"], + "ip.src": ["192.0.2.1"], + "ip.dst": ["192.0.2.2"], + "dns.qry.name": ["many.example"], + "dns.flags.response": ["True"], + "dns.a": [`198.51.100.${index}`], + "tls.handshake.extensions_server_name": [`host${index}.example`], + "http.request.full_uri": [ + "http://user:password@example.invalid/p?q=keep", + ], + }, + }, + })); + await writeFile( + tshark, + `#!/bin/sh\nif [ "$1" = "--version" ]; then printf 'TShark (Wireshark) 4.6.6-emulated\\n'; exit 0; fi\nprintf 'read\\n' >> ${quote(marker)}\nprintf '%s' ${quote(JSON.stringify(frames))}\nprintf 'retained warning' >&2\n`, + { mode: 0o755 }, + ); + await writeFile( + capinfos, + `#!/bin/sh\nif [ "$1" = "--version" ]; then printf 'Capinfos (Wireshark) 4.6.6-emulated\\n'; exit 0; fi\nprintf '%s\\n' ${quote(`File type: pcap\nFile encapsulation: Ethernet\nNumber of packets: 132\nSHA256: ${sha}`)}\n`, + { mode: 0o755 }, + ); + const environment = { + REA_TSHARK_COMMAND: tshark, + REA_CAPINFOS_COMMAND: capinfos, + }; + return { path, marker, environment }; +}; + +// Native tools are emulated here; this proves the production CLI/MCP boundary, +// retention and paging, not Wireshark's real dissector semantics. +it.runIf(process.platform === "linux")( + "retains every fact and pages late DNS/SNI rows after source deletion without another tool launch", + async () => { + const root = await createTestTempDirectory("rea-pcap-view-"); + const { path, marker, environment } = await createPcapToolFixture(root); + const complete = await promisify(execFile)( + process.execPath, + [ + join(import.meta.dirname, "../../../scripts/rea.mjs"), + "inspect-pcap-capture", + path, + "--json", + ], + { env: { ...process.env, ...environment } }, + ); + const parent = parseEvidence(JSON.parse(complete.stdout)); + expect(parent.normalized_result).toMatchObject({ + tls_sni_count: 132, + tls_sni_values: expect.arrayContaining(["host131.example"]), + dns_queries: [{ query: "many.example", response_address_count: 132 }], + }); + expect(JSON.stringify(parent)).not.toContain("user:password"); + const { call } = await connectLocalToolsMcp(environment); + const response = await call("inspect_pcap_capture", { + path, + detail: "summary", + }); + expect(response.isError, JSON.stringify(response)).not.toBe(true); + const summary = parseEvidence(response.structuredContent).normalized_result; + expect(summary).toMatchObject({ + parent_evidence_id: parent.evidence_id, + summary: { + tls_sni_count: 132, + dns_query_count: 1, + retained_counts: { tls_sni_values: 132 }, + diagnostics_code_units: { tshark: 16, capinfos: 0 }, + }, + }); + expect(JSON.stringify(summary)).not.toContain("host131.example"); + await rm(path); + const source = { + kind: "retained-evidence", + evidence_id: parent.evidence_id, + }; + const before = await readFile(marker, "utf8"); + const cases = [ + [ + { kind: "pcap", facet: "endpoints", limit: 1 }, + { items: [{ address: "192.0.2.1", packet_count: 132 }] }, + ], + [ + { kind: "pcap", facet: "dns-queries", limit: 1 }, + { + items: [ + { query: "many.example", retained_response_address_count: 132 }, + ], + }, + ], + [ + { + kind: "pcap", + facet: "dns-addresses", + query: "many.example", + offset: 130, + limit: 2, + }, + { items: ["198.51.100.130", "198.51.100.131"] }, + ], + [ + { kind: "pcap", facet: "tls-sni", offset: 130, limit: 2 }, + { items: ["host130.example", "host131.example"] }, + ], + [ + { kind: "pcap", facet: "protocols" }, + { items: [{ protocol: "ip:udp:dns", packet_count: 132 }] }, + ], + [ + { kind: "pcap", facet: "http-uris" }, + { items: ["http://example.invalid/p?q=keep"] }, + ], + [{ kind: "pcap", facet: "unattributed-addresses" }, { items: [] }], + [ + { + kind: "pcap", + facet: "diagnostics", + stream: "tshark", + offset: 9, + limit: 7, + }, + { text: "warning", status: "retained" }, + ], + ] as const; + for (const [view, expected] of cases) { + const response = await call("inspect_analysis_view", { source, view }); + expect(response.isError, JSON.stringify(response)).not.toBe(true); + expect( + parseEvidence(response.structuredContent).normalized_result, + ).toMatchObject({ page: expected, source_coverage: "complete" }); + } + const workflow = join(root, "portable.json"); + const view = { + kind: "pcap", + facet: "dns-addresses", + query: "many.example", + offset: 130, + limit: 2, + }; + await writeFile( + workflow, + JSON.stringify({ source: { kind: "inline", evidence: parent }, view }), + ); + const portable = await promisify(execFile)(process.execPath, [ + join(import.meta.dirname, "../../../scripts/rea.mjs"), + "inspect-analysis-view", + workflow, + "--json", + ]); + const retained = parseEvidence( + (await call("inspect_analysis_view", { source, view })).structuredContent, + ); + expect( + parseEvidence(JSON.parse(portable.stdout)).normalized_result, + ).toEqual(retained.normalized_result); + expect(await readFile(marker, "utf8")).toBe(before); + expect( + parseMcpToolError( + await call("inspect_analysis_view", { + source, + view: { + kind: "pcap", + facet: "dns-addresses", + query: "missing.example", + }, + }), + ), + ).toMatchObject({ error: { code: "invalid_request" } }); + }, + 60_000, +); diff --git a/tests/boundary/process/darwinProcessRunTokenReader.test.ts b/tests/boundary/process/darwinProcessRunTokenReader.test.ts index bfdb047f2..9f377440c 100644 --- a/tests/boundary/process/darwinProcessRunTokenReader.test.ts +++ b/tests/boundary/process/darwinProcessRunTokenReader.test.ts @@ -26,6 +26,7 @@ import { import { createDarwinProcessRunTokenReader, DarwinProcessOwnershipInspectionError, + SWIFTC_INTERRUPT, } from "../../../src/process/DarwinProcessRunTokenReader.js"; import { execFileOutput } from "../../../src/process/ExecFileOutput.js"; import { @@ -1110,16 +1111,8 @@ it.skipIf(!onDarwin)( try { await execFileOutput( "/usr/bin/xcrun", - [ - "swiftc", - "-module-cache-path", - join(root, "modules"), - core, - fixture, - "-o", - executable, - ], - { timeout: 60_000 }, + ["swiftc", core, fixture, "-o", executable], + { timeout: 120_000, stopSignal: SWIFTC_INTERRUPT }, ); const { stdout } = await execFileAsync(executable, [], { encoding: "utf8", @@ -1179,6 +1172,7 @@ it.skipIf(!onDarwin)( await rm(root, { recursive: true, force: true }); } }, + 150_000, ); it.skipIf(!onDarwin)( @@ -1203,16 +1197,8 @@ it.skipIf(!onDarwin)( try { await execFileOutput( "/usr/bin/xcrun", - [ - "swiftc", - "-module-cache-path", - join(root, "modules"), - core, - fixture, - "-o", - executable, - ], - { timeout: 60_000 }, + ["swiftc", core, fixture, "-o", executable], + { timeout: 120_000, stopSignal: SWIFTC_INTERRUPT }, ); const { stdout } = await execFileAsync(executable, [], { encoding: "utf8", @@ -1226,6 +1212,7 @@ it.skipIf(!onDarwin)( await rm(root, { recursive: true, force: true }); } }, + 150_000, ); it.skipIf(!onDarwin)( diff --git a/tests/boundary/process/javascriptSourceModes.test.ts b/tests/boundary/process/javascriptSourceModes.test.ts new file mode 100644 index 000000000..64c15a2d8 --- /dev/null +++ b/tests/boundary/process/javascriptSourceModes.test.ts @@ -0,0 +1,39 @@ +import { expect } from "vitest"; + +import { parseJavaScriptSource } from "../../../src/domain/javascript/javascriptSourceParser.js"; +import { processTest } from "../../support/process/processFixture.js"; + +for (const source of [ + "return;", + "void new.target;", + "var await = 1; return;", + "with ({ value: 1 }) { void value; } void new.target;", +]) { + processTest( + `accepts Node CommonJS syntax while retaining module errors: ${source}`, + async ({ processes }) => { + const commonjs = await processes.run( + process.execPath, + ["--check", "--input-type=commonjs"], + { input: source, timeoutMs: 5_000 }, + ); + expect(commonjs.exitCode).toBe(0); + const module = await processes.run( + process.execPath, + ["--check", "--input-type=module"], + { input: source, timeoutMs: 5_000 }, + ); + expect(module.exitCode).not.toBe(0); + + const parsed = parseJavaScriptSource(source, "main.cjs"); + expect(parsed).not.toBeNull(); + expect(parsed?.errors).toEqual([]); + expect( + parseJavaScriptSource(source, "main.mjs")?.errors.length, + ).toBeGreaterThan(0); + expect( + parseJavaScriptSource(source, "main.mts")?.errors.length, + ).toBeGreaterThan(0); + }, + ); +} diff --git a/tests/boundary/process/ownedCommand.test.ts b/tests/boundary/process/ownedCommand.test.ts index 34205af3d..ca0cb2db6 100644 --- a/tests/boundary/process/ownedCommand.test.ts +++ b/tests/boundary/process/ownedCommand.test.ts @@ -257,9 +257,11 @@ it.skipIf(process.platform === "win32").each(["cancelled", "timeout"] as const)( failures: [expect.objectContaining({ reason: "run-token-mismatch" })], }); expect(failure.cleanupResult).toEqual(cleanup); - expect(failure.resources).toEqual( - expect.arrayContaining([request.runId]), - ); + expect(failure.resources).toEqual([ + request.runId, + `pid:${launched.ownership.leaderPid}`, + `process-group:${launched.ownership.processGroupId}`, + ]); // An unresolved member does not grant authority to signal the shared group. const acquiredPid = launched.ownership.leaderPid; expect(() => process.kill(acquiredPid, 0)).not.toThrow(); diff --git a/tests/boundary/process/processCaptureLifecycle.test.ts b/tests/boundary/process/processCaptureLifecycle.test.ts index dcba0510f..01d8eba13 100644 --- a/tests/boundary/process/processCaptureLifecycle.test.ts +++ b/tests/boundary/process/processCaptureLifecycle.test.ts @@ -4,7 +4,7 @@ import { access, readFile, rm, symlink, writeFile } from "node:fs/promises"; import { dirname, join } from "node:path"; import { fileURLToPath } from "node:url"; import { promisify } from "node:util"; -import { expect } from "vitest"; +import { expect, vi } from "vitest"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { itWithCaptureCapability } from "./processCaptureCapability.js"; @@ -123,6 +123,93 @@ const terminateFixtureProcess = async (pid: number): Promise => { await waitForProcessExit(pid); }; +itWithCaptureCapability( + "keeps the process deadline bounded when the wall clock moves backward", + async () => { + const wallStart = Date.now(); + let wallReads = 0; + const now = vi.spyOn(Date, "now").mockImplementation(() => { + wallReads += 1; + return wallReads === 1 ? wallStart : wallStart - 60_000 + wallReads; + }); + const controller = new AbortController(); + const fallback = setTimeout(() => controller.abort(), 1_500); + try { + const result = await captureProcessScenario( + parseProcessScenario({ + executable: process.execPath, + arguments: [processFixture, "hang"], + working_directory: dirname(processFixture), + timeout_ms: 100, + idle_timeout_ms: 5_000, + }), + controller.signal, + ); + expect(result.ok).toBe(true); + if (!result.ok) throw result.error; + expect(result.value.exit.reason).toBe("timeout"); + } finally { + clearTimeout(fallback); + now.mockRestore(); + } + }, + 10_000, +); + +itWithCaptureCapability( + "preserves reversed wall-clock observations and monotonic frames across capture", + async () => { + vi.useFakeTimers({ toFake: ["Date"] }); + vi.setSystemTime(new Date("2026-01-02T00:00:00.000Z")); + try { + const result = await captureProcessScenario( + parseProcessScenario({ + executable: process.execPath, + arguments: [processFixture, "partial"], + working_directory: dirname(processFixture), + settle_ms: 0, + timeout_ms: 5_000, + idle_timeout_ms: 5_000, + }), + undefined, + process.platform, + process.env, + undefined, + undefined, + { + async report(update) { + if (update.phase === "running") + vi.setSystemTime(new Date("2026-01-01T00:00:00.000Z")); + }, + }, + ); + expect(result.ok).toBe(true); + if (!result.ok) throw result.error; + expect(result.value.frames.length).toBeGreaterThan(0); + expect(Date.parse(result.value.manifest.completed_at)).toBeLessThan( + Date.parse(result.value.manifest.started_at), + ); + expect(result.value.limitations).toContain( + "Wall-clock completion time precedes start time; elapsed capture timing is recorded separately and remains monotonic.", + ); + expect( + result.value.frames.map(({ at_ms }) => at_ms), + "capture-relative frame times stay ordered and nonnegative", + ).toEqual( + result.value.frames + .map(({ at_ms }) => at_ms) + .sort((left, right) => left - right), + ); + expect(result.value.frames.every(({ at_ms }) => at_ms >= 0)).toBe(true); + expect(result.value.exit.reason).toBe("exited"); + expect(result.value.settlement.elapsed_ms).toBeGreaterThan(0); + } finally { + vi.useRealTimers(); + } + }, + 10_000, +); + itWithCaptureCapability( "cleans detached run-token children omitted by sampling and preserves an unowned neighbor", async () => { diff --git a/tests/boundary/process/processCaptureNormalization.test.ts b/tests/boundary/process/processCaptureNormalization.test.ts index 965dc2af5..159f84c93 100644 --- a/tests/boundary/process/processCaptureNormalization.test.ts +++ b/tests/boundary/process/processCaptureNormalization.test.ts @@ -1,4 +1,4 @@ -import { readFile } from "node:fs/promises"; +import { mkdir, readFile, symlink } from "node:fs/promises"; import { join } from "node:path"; import { fileURLToPath } from "node:url"; import { expect, it } from "vitest"; @@ -28,6 +28,35 @@ const identitySchema = z.object({ const itWithLinuxCaptureCapability = process.platform === "linux" ? itWithCaptureCapability : it.skip; +itWithCaptureCapability( + "captures a POSIX PATH executable with a literal backslash in its basename", + async () => { + const root = await createTestTempDirectory("rea-process-backslash-name-"); + const bin = join(root, "bin"); + await mkdir(bin); + const command = "selected\\node"; + const selected = join(bin, command); + await symlink(process.execPath, selected); + const scenario = parseProcessScenario({ + executable: command, + working_directory: root, + environment: { PATH: bin }, + arguments: ["-e", 'process.stdout.write("selected executable")'], + settle_ms: 100, + }); + const result = await captureProcessScenario(scenario); + if (!result.ok) throw result.error; + expect(result.value.frames.map(({ data }) => data).join("")).toBe( + "selected executable", + ); + expect(result.value.exit).toMatchObject({ code: 0, reason: "exited" }); + expect(result.value.cleanup).toMatchObject({ + owned_process_group: "verified", + temporary_root: "removed", + }); + }, +); + itWithCaptureCapability( "preserves numeric reports and original normalized PTY text in Evidence", async () => { @@ -202,3 +231,61 @@ itWithLinuxCaptureCapability.each([false, true])( }, 20_000, ); + +itWithCaptureCapability( + "preserves selected nested path identities in actual PTY Evidence", + async () => { + const root = await createTestTempDirectory("rea-process-nested-paths-"); + const executable = join(root, "selected-node"); + const observed = join(root, "data"); + await symlink(process.execPath, executable); + await mkdir(observed); + const original = `${executable}\n${observed}/result\n${root}/log`; + const scenario = parseProcessScenario({ + executable, + working_directory: root, + filesystem_observation_paths: [observed], + arguments: ["-e", `process.stdout.write(${JSON.stringify(original)})`], + normalization: { paths: true, pids: false, ports: false, patterns: [] }, + settle_ms: 100, + }); + const result = await captureProcessScenario(scenario); + if (!result.ok) throw result.error; + const capture = parseProcessCapture( + createProcessCaptureEvidence(scenario, result.value).normalized_result, + ); + expect(capture.frames.map(({ data }) => data).join("")).toBe( + "\r\n/result\r\n/log", + ); + expect( + capture.frames.map(({ raw_data, data }) => raw_data ?? data).join(""), + ).toBe(original.replaceAll("\n", "\r\n")); + expect(capture.manifest.comparison_contract).toHaveProperty( + "path_normalization_version", + "longest-original-paths-v1", + ); + const legacyContract = { ...capture.manifest.comparison_contract }; + delete legacyContract.path_normalization_version; + const legacy = parseProcessCapture({ + ...capture, + manifest: { + ...capture.manifest, + comparison_contract: legacyContract, + comparison_contract_sha256: digestProcessCommitment(legacyContract), + }, + }); + expect(() => compareProcessCaptures(legacy, capture)).toThrow( + expect.objectContaining({ + issues: [ + expect.objectContaining({ expected: ["path_normalization_version"] }), + ], + }), + ); + expect(capture.exit).toMatchObject({ code: 0, reason: "exited" }); + expect(capture.cleanup).toMatchObject({ + owned_process_group: "verified", + temporary_root: "removed", + }); + }, + 20_000, +); diff --git a/tests/boundary/process/sourceMapUncHosts.test.ts b/tests/boundary/process/sourceMapUncHosts.test.ts index 293a3f71a..99953509f 100644 --- a/tests/boundary/process/sourceMapUncHosts.test.ts +++ b/tests/boundary/process/sourceMapUncHosts.test.ts @@ -73,9 +73,9 @@ processTest.for(cases)( [ "--input-type=module", "--eval", - `import { resolveJavaScriptSourceMapReference } from "./dist/domain/javascript/javascriptSourceMapPaths.js"; + `import { resolveJavaScriptSourceMapReferences } from "./dist/domain/javascript/javascriptSourceMapPaths.js"; const [source, root] = JSON.parse(process.argv[1]); - console.log(JSON.stringify(resolveJavaScriptSourceMapReference(source, root, "maps/main.js.map")));`, + console.log(JSON.stringify(resolveJavaScriptSourceMapReferences([source], root, "maps/main.js.map")[0]));`, JSON.stringify([scenario.source, sourceRoot]), ], { timeoutMs: 10_000 }, diff --git a/tests/boundary/providers/artifactReadCleanupRetry.test.ts b/tests/boundary/providers/artifactReadCleanupRetry.test.ts new file mode 100644 index 000000000..ebaba673a --- /dev/null +++ b/tests/boundary/providers/artifactReadCleanupRetry.test.ts @@ -0,0 +1,100 @@ +import type { FileHandle } from "node:fs/promises"; +import { mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { expect, it, vi } from "vitest"; + +const cleanupState = vi.hoisted(() => ({ + failFirstClose: false, + abortController: undefined as AbortController | undefined, + firstHandle: undefined as FileHandle | undefined, + firstHandleCloseAttempts: 0, +})); + +vi.mock("node:fs/promises", async (importOriginal) => { + const fs = await importOriginal(); + return { + ...fs, + open: async (...args: Parameters) => { + const handle = await fs.open(...args); + const close = handle.close.bind(handle); + if (cleanupState.abortController !== undefined) + setImmediate(() => { + cleanupState.abortController?.abort(); + cleanupState.abortController = undefined; + }); + handle.close = async () => { + if (handle === cleanupState.firstHandle) + cleanupState.firstHandleCloseAttempts += 1; + else if (cleanupState.firstHandle === undefined) + cleanupState.firstHandle = handle; + if ( + cleanupState.failFirstClose && + handle === cleanupState.firstHandle + ) { + cleanupState.failFirstClose = false; + throw new Error("injected descriptor close failure"); + } + return close(); + }; + return handle; + }, + }; +}); + +import { projectAnalysisError } from "../../../src/domain/analysisErrorProjection.js"; +import { GoBinaryProvider } from "../../../src/go/GoBinaryProvider.js"; +import { PeResourcesProvider } from "../../../src/native/pe/PeResourcesProvider.js"; +import { createGoBinaryFixture } from "../../fixtures/go/image.js"; +import { peResourceFixture } from "../../../src/native/pe/PeResources.fixture.js"; + +it.each(["go", "pe"] as const)( + "%s provider retains a failed descriptor close and retries that owner before the next read", + async (kind) => { + const root = await mkdtemp(join(tmpdir(), "rea-read-cleanup-")); + const path = join(root, kind === "go" ? "program.elf" : "program.exe"); + const bytes = + kind === "go" ? createGoBinaryFixture().bytes : peResourceFixture().bytes; + await writeFile(path, bytes); + const provider = + kind === "go" ? new GoBinaryProvider() : new PeResourcesProvider(); + cleanupState.failFirstClose = true; + const controller = new AbortController(); + cleanupState.abortController = controller; + cleanupState.firstHandle = undefined; + cleanupState.firstHandleCloseAttempts = 0; + try { + const input = { + path, + max_file_bytes: 8 * 1024 * 1024, + max_entries: 4096, + }; + const first = await provider.inspect(input, { + signal: controller.signal, + }); + expect(first.ok).toBe(false); + if (first.ok) throw new Error("Injected close failure was not returned"); + expect(projectAnalysisError(first.error)).toMatchObject({ + code: "cleanup_incomplete", + details: { diagnostics: { primary_error: { code: "cancelled" } } }, + }); + expect(first.error.cause).toBeInstanceOf(AggregateError); + if (!(first.error.cause instanceof AggregateError)) + throw first.error.cause; + expect(first.error.cause.errors[1]).toMatchObject({ + cleanupResources: [path], + }); + + const retry = await provider.inspect(input); + expect(retry.ok).toBe(true); + expect(cleanupState.firstHandleCloseAttempts).toBe(2); + } finally { + await provider.close(); + cleanupState.failFirstClose = false; + cleanupState.abortController = undefined; + cleanupState.firstHandle = undefined; + cleanupState.firstHandleCloseAttempts = 0; + await rm(root, { recursive: true, force: true }); + } + }, +); diff --git a/tests/boundary/providers/ghidra/ghidraJavaLaunch.test.ts b/tests/boundary/providers/ghidra/ghidraJavaLaunch.test.ts index dc4be5ef2..56fec2151 100644 --- a/tests/boundary/providers/ghidra/ghidraJavaLaunch.test.ts +++ b/tests/boundary/providers/ghidra/ghidraJavaLaunch.test.ts @@ -286,7 +286,8 @@ describe.skipIf(process.platform === "win32")( { platform: process.platform, architecture: process.arch, - readText: () => "application.version=12.1.4\n", + readText: () => + "application.version=12.1.4\napplication.java.min=21\n", executable: () => true, probeJava: () => ({ version: "21.0.12", diff --git a/tests/boundary/providers/ghidra/ghidraLauncher.test.ts b/tests/boundary/providers/ghidra/ghidraLauncher.test.ts index 270bf02b3..ea8fb964c 100644 --- a/tests/boundary/providers/ghidra/ghidraLauncher.test.ts +++ b/tests/boundary/providers/ghidra/ghidraLauncher.test.ts @@ -1,4 +1,11 @@ -import { access, chmod, readFile, stat } from "node:fs/promises"; +import { + access, + chmod, + mkdir, + readFile, + stat, + writeFile, +} from "node:fs/promises"; import { join, posix, win32 } from "node:path"; import { fileURLToPath } from "node:url"; @@ -480,6 +487,64 @@ describe("Ghidra Windows launcher", () => { ).toThrow(/metacharacters/u); }); + it.skipIf(process.platform !== "win32").each([ + { + properties: + "application.version=12.1.4\napplication.release.name=PUBLIC\n", + release: "ghidra_12.1.4_PUBLIC", + }, + { properties: undefined, release: "renamed-install" }, + { properties: "application.version=12.1.4\n", release: "renamed-install" }, + ])( + "prepares the release directory through the Windows runtime ($release)", + async ({ properties, release }) => { + const parent = await createTestTempDirectory("rea-launch-config-"); + const installRoot = join(parent, "renamed-install"); + if (properties !== undefined) { + await mkdir(join(installRoot, "Ghidra"), { recursive: true }); + await writeFile( + join(installRoot, "Ghidra", "application.properties"), + properties, + ); + } + const runtime = await createGhidraTestRuntime(parent); + runtimes.push(runtime); + const spawnProcess = vi.fn(async () => { + throw new Error("fixture stops before process creation"); + }); + const launcher = new GhidraHeadlessLauncher({ + environment: process.env, + platform: "win32", + analyzeHeadlessPath: join( + installRoot, + "support", + "analyzeHeadless.bat", + ), + javaHome: "C:\\Java\\jdk-21", + bridgeScriptPath: join(parent, "ReaGhidraBridge.java"), + spawnProcess, + }); + const launched = await launcher.launch({ + runtimeRoot: runtime.path, + transport: "unix-socket", + endpointPath: join(runtime.path, "bridge.sock"), + token: "fixture-token", + runId: "d6fcbb66-e829-4ff6-a535-0035aec63139", + targetPath: join(parent, "fixture.exe"), + targetSha256: "b".repeat(64), + providerVersion: "12.1.4", + profileDigest: "a".repeat(64), + }); + expect(launched.ok).toBe(false); + expect(spawnProcess).toHaveBeenCalledOnce(); + expect( + ( + await stat(join(runtime.path, "config", "ghidra", release)) + ).isDirectory(), + ).toBe(true); + }, + ); + it.skipIf(process.platform !== "win32")( "keeps authority in a private descriptor and isolates Ghidra state through the batch launcher", async () => { diff --git a/tests/boundary/providers/hopper/bridgeLauncher.test.ts b/tests/boundary/providers/hopper/bridgeLauncher.test.ts index 722a25bed..542858a0d 100644 --- a/tests/boundary/providers/hopper/bridgeLauncher.test.ts +++ b/tests/boundary/providers/hopper/bridgeLauncher.test.ts @@ -84,6 +84,7 @@ describe("Hopper bridge launcher selection", () => { "private", "/usr/bin/python3", "/rea/hopper-demo-x11.py", + "--owner-stdin", "--strategy", "user-mount-namespace", "--mount-private-x11", @@ -117,6 +118,7 @@ describe("Hopper bridge launcher selection", () => { command: "/usr/bin/python3", args: [ "/rea/hopper-demo-x11.py", + "--owner-stdin", "--strategy", "direct", "--hopper", @@ -202,3 +204,59 @@ it("launches Hopper helpers with the caller-selected environment without restori await launched.value.releaseLease?.(); } }); + +it.skipIf(process.platform !== "linux")( + "gives the demo helper a live owner stream through Node's actual stdio transport", + async () => { + const directory = await createTestTempDirectory("rea-hopper-owner-stdio-"); + const launcher = new HopperApplicationLauncher( + { + ...options(process.execPath), + launchMode: "verified_linux_demo", + demoHelperPath: fileURLToPath( + new URL("../../../fixtures/hopperLifetimeLaunch.py", import.meta.url), + ), + }, + { + selectPrivateDisplay: async () => ({ + ok: true, + strategy: "direct", + diagnostic: { + component: "hopper_private_display", + operation: "probe", + status: "ready", + failure_code: null, + reason: "ready", + socket_directory: "/tmp/.X11-unix", + socket_directory_mode: "1777", + mount_read_only: false, + effective_socket_directory_mode: "1777", + effective_mount_read_only: false, + wsl: false, + strategy: "direct", + fallback_reason: null, + xvfb_stderr_bytes: 0, + }, + }), + }, + ); + const launched = await launcher.launch({ + directory, + socketPath: join(directory, "bridge.sock"), + token: "fixture-token", + runId: randomUUID(), + }); + if (!launched.ok) throw launched.error; + const supervisor = new ProviderProcessSupervisor(launched.value); + try { + expect(await supervisor.waitForOutputClose(5_000)).toBe(true); + expect(supervisor.snapshot().exitCode).toBe(0); + expect(JSON.parse(supervisor.snapshot().stdout.text)).toEqual({ + owner_alive: true, + }); + } finally { + await supervisor.stop(); + await launched.value.releaseLease?.(); + } + }, +); diff --git a/tests/boundary/providers/hopper/linuxPrivateDisplay.test.ts b/tests/boundary/providers/hopper/linuxPrivateDisplay.test.ts index f2c99980f..c5fd45c96 100644 --- a/tests/boundary/providers/hopper/linuxPrivateDisplay.test.ts +++ b/tests/boundary/providers/hopper/linuxPrivateDisplay.test.ts @@ -31,6 +31,9 @@ const dialogDetectionFixturePath = fileURLToPath( const cleanupTimeoutFixturePath = fileURLToPath( new URL("../../../fixtures/hopperCleanupTimeout.py", import.meta.url), ); +const ownerLifetimeFixturePath = fileURLToPath( + new URL("../../../fixtures/hopperOwnerLifetime.py", import.meta.url), +); type DiagnosticOverrides = Partial< Omit @@ -107,6 +110,23 @@ describe("Hopper demo dialog detection", () => { }); describe("Hopper process cleanup", () => { + it.skipIf(process.platform !== "linux").each(["startup", "ready"])( + "reaps only its own children when the owner pipe closes during %s", + (phase) => { + const result = spawnSync( + "/usr/bin/python3", + [ownerLifetimeFixturePath, helperPath, phase], + { encoding: "utf8", timeout: 10_000 }, + ); + expect(result.status, result.stderr).toBe(0); + expect(JSON.parse(result.stdout)).toEqual({ + phase, + children_cleaned: true, + sentinel_alive: true, + }); + }, + ); + it.skipIf(process.platform !== "linux")( "kills a child that ignores SIGTERM after the bounded timeout", () => { diff --git a/tests/boundary/providers/ida/containment.test.ts b/tests/boundary/providers/ida/containment.test.ts new file mode 100644 index 000000000..75eb78f42 --- /dev/null +++ b/tests/boundary/providers/ida/containment.test.ts @@ -0,0 +1,144 @@ +import { rm } from "node:fs/promises"; +import { afterEach, describe, expect, it } from "vitest"; +import { inspectAddressContext } from "../../../../src/application/AnalysisContextQueries.js"; +import { EnhancedTools } from "../../../../src/application/EnhancedTools.js"; +import { IdaSessionClient } from "../../../../src/ida/IdaSessionClient.js"; +import { createIdaTarget, RecordingIdaMcp } from "../../../fixtures/idaMcp.js"; + +const roots: string[] = []; +afterEach(async () => { + for (const root of roots.splice(0)) + await rm(root, { recursive: true, force: true }); +}); + +const session = async (cappedReferences = false) => { + const { root, target } = await createIdaTarget("arm64"); + roots.push(root); + const producer = new RecordingIdaMcp(target, "headless"); + if (cappedReferences) { + const inner = producer.overrideCall; + producer.overrideCall = (name, args) => + name === "xrefs_to" + ? [ + { + xrefs: [ + { + addr: "0x1004", + type: "code", + fn: { addr: "0x1000", name: "main", size: "0x20" }, + }, + ], + more: true, + }, + ] + : inner?.(name, args); + } + const client = new IdaSessionClient( + { + command: "fixture", + args: [], + env: {}, + mode: "headless", + timeoutMs: 1000, + workspaceRoot: root, + }, + target, + producer, + ); + return { client, target }; +}; + +const MAIN = { + found: true, + procedure: { address: "0x1000", name: "main" }, +}; + +describe("trace_feature on headless IDA", () => { + it("attributes a reference to its containing procedure", async () => { + const { client } = await session(); + try { + const result = await new EnhancedTools(client).execute("trace_feature", { + query: "a.b", + }); + + if (!result.ok) throw result.error; + expect(result.value).toMatchObject({ + matches: [{ type: "string", address: "0x3000" }], + references: [ + { + target_address: "0x3000", + source_address: "0x1004", + containing_procedure: MAIN, + }, + ], + truncated: false, + residual_unknowns: [], + }); + } finally { + expect((await client.close()).ok).toBe(true); + } + }); + + it("keeps an upstream reference cap visible once containment makes the trace run", async () => { + const { client } = await session(true); + try { + const result = await new EnhancedTools(client).execute("trace_feature", { + query: "a.b", + }); + + if (!result.ok) throw result.error; + expect(result.value).toMatchObject({ + references: [{ source_address: "0x1004", containing_procedure: MAIN }], + truncated: true, + residual_unknowns: [ + "References to 0x3000 were truncated by the provider; additional references may exist.", + ], + }); + } finally { + expect((await client.close()).ok).toBe(true); + } + }); +}); + +describe("inspect_address_context on headless IDA", () => { + it("resolves the admitted program as the document and reports the containing procedure", async () => { + const { client, target } = await session(); + try { + const result = await inspectAddressContext(client, { + address: "0x1004", + }); + + if (!result.ok) throw result.error; + expect(result.value).toMatchObject({ + address: "0x1004", + document: target.path, + procedure: { state: "available", value: MAIN }, + name: { state: "unavailable" }, + comment: { state: "unavailable" }, + inline_comment: { state: "unavailable" }, + bookmarks: { state: "unavailable" }, + }); + } finally { + expect((await client.close()).ok).toBe(true); + } + }); + + it("answers the same way for an explicit admitted document", async () => { + const { client, target } = await session(); + try { + const implicit = await inspectAddressContext(client, { + address: "0x1004", + }); + const explicit = await inspectAddressContext(client, { + address: "0x1004", + document: target.path, + }); + + if (!implicit.ok) throw implicit.error; + if (!explicit.ok) throw explicit.error; + expect(explicit.value).toEqual(implicit.value); + } finally { + expect((await client.close()).ok).toBe(true); + } + }); +}); diff --git a/tests/boundary/providers/ida/retainedConnectionRetry.test.ts b/tests/boundary/providers/ida/retainedConnectionRetry.test.ts new file mode 100644 index 000000000..4f9092842 --- /dev/null +++ b/tests/boundary/providers/ida/retainedConnectionRetry.test.ts @@ -0,0 +1,114 @@ +import { createHash } from "node:crypto"; +import { mkdir, readdir, writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { describe, expect, it, onTestFinished } from "vitest"; + +import { createTestTempDirectory } from "../../../fixtures/temporaryDirectory.js"; + +import { elf } from "../../../../src/domain/binaryTarget.fixture.js"; +import type { BinaryTarget } from "../../../../src/domain/binaryTargetTypes.js"; +import { createIdaMcpConnection } from "../../../../src/ida/IdaMcpConnection.js"; +import { IdaSessionClient } from "../../../../src/ida/IdaSessionClient.js"; + +/** + * A headless IDA supervisor over real stdio that answers the session + * lifecycle, but rejects the first inventory after a close so cleanup fails + * once and must be retried through the retained connection. + */ +const FLAKY_INVENTORY_SUPERVISOR = `import { createInterface } from "node:readline"; +const tools = ["idb_open", "idb_list", "idb_close", "server_health", "list_funcs", + "lookup_funcs", "find_regex", "decompile", "disasm", "callees", "xrefs_to"]; +let session; +let inputPath; +let closed = false; +let inventoryFailed = false; +const send = (message) => + process.stdout.write(JSON.stringify({ jsonrpc: "2.0", ...message }) + "\\n"); +const result = (value) => ({ + content: [{ type: "text", text: JSON.stringify(value) }], + structuredContent: value, +}); +const call = ({ name, arguments: args }) => { + if (name === "idb_open") { + session = args.preferred_session_id; + inputPath = args.input_path; + return result({ success: true, session: { session_id: session, input_path: inputPath } }); + } + if (name === "server_health") + return result({ status: "ok", input_path: inputPath, idb_path: inputPath + ".i64", + module: "target.elf", imagebase: "0x1000", auto_analysis_ready: true }); + if (name === "idb_close") { + closed = true; + return result({ success: true, session_id: session, saved: null, owned: true, backend: "worker" }); + } + if (name === "idb_list") { + if (closed && !inventoryFailed) { + inventoryFailed = true; + return { isError: true, content: [{ type: "text", text: "inventory unavailable" }] }; + } + return result({ sessions: closed ? [] : [{ session_id: session, input_path: inputPath, + owned: true, backend: "worker", is_active: true }] }); + } + return { isError: true, content: [{ type: "text", text: "unsupported " + name }] }; +}; +for await (const line of createInterface({ input: process.stdin })) { + const request = JSON.parse(line); + if (request.method === "initialize") + send({ id: request.id, result: { + protocolVersion: request.params.protocolVersion, + capabilities: { tools: {} }, + serverInfo: { name: "flaky-inventory-ida-fixture", version: "1" }, + } }); + else if (request.method === "tools/list") + send({ id: request.id, result: { + tools: tools.map((name) => ({ name, inputSchema: { type: "object" } })), + } }); + else if (request.method === "tools/call") + send({ id: request.id, result: call(request.params) }); +} +`; + +describe("retained headless IDA connection in a long-lived host", () => { + it("retries cleanup through the detached stdio connection", async () => { + const root = await createTestTempDirectory("rea-ida-retry-"); + const supervisor = join(root, "supervisor.mjs"); + await writeFile(supervisor, FLAKY_INVENTORY_SUPERVISOR); + const workspaceRoot = join(root, "workspaces"); + await mkdir(workspaceRoot); + const bytes = elf(2, 1, 62); + const targetPath = join(root, "target.elf"); + await writeFile(targetPath, bytes); + const target: BinaryTarget = { + path: targetPath, + sha256: createHash("sha256").update(bytes).digest("hex"), + kind: "executable", + format: "elf", + architecture: "x86_64", + availableArchitectures: ["x86_64"], + }; + const config = { + command: process.execPath, + args: [supervisor], + env: {}, + mode: "headless" as const, + timeoutMs: 5000, + workspaceRoot, + }; + const connection = createIdaMcpConnection(config, process.env); + // Close the supervisor even when an assertion fails before cleanup does. + onTestFinished(() => connection.close().catch(() => undefined)); + const client = new IdaSessionClient(config, target, connection); + + expect((await client.execute("health", {})).ok).toBe(true); + const failed = await client.close(); + expect(failed.ok).toBe(false); + if (!failed.ok) expect(failed.error.cleanupIncomplete).toBe(true); + expect(await readdir(workspaceRoot)).toHaveLength(1); + + // The connection was detached when cleanup failed; it must still serve + // the retry that confirms release and removes the private workspace. + expect((await client.close()).ok).toBe(true); + expect(await readdir(workspaceRoot)).toEqual([]); + }); +}); diff --git a/tests/boundary/providers/ida/verifierFeatureTrace.test.ts b/tests/boundary/providers/ida/verifierFeatureTrace.test.ts new file mode 100644 index 000000000..77f654ffc --- /dev/null +++ b/tests/boundary/providers/ida/verifierFeatureTrace.test.ts @@ -0,0 +1,212 @@ +import { describe, expect, it } from "vitest"; +import { + assertFeatureTrace, + findMappedNonFunction, +} from "../../../../scripts/lib/ida-feature-trace.mjs"; + +const procedure = { address: "0x1000", name: "main" }; + +const containing = (found: boolean) => + found + ? { + query_address: "0x1004", + found: true, + procedure: { address: "0x1000", name: "main" }, + } + : { + query_address: "0x3010", + found: false, + procedure: null, + reason: "not_in_procedure", + }; + +const trace = ( + matches: readonly { type: string; address: string }[], + references: readonly { + target_address: string; + containing_procedure: unknown; + }[], +) => ({ + query: "main", + search_mode: "literal", + matches, + references, + truncated: false, + residual_unknowns: [], +}); + +const SELECTED = { type: "procedure", address: "0x1000" }; +const SUBSTRING = { type: "procedure", address: "0x2000" }; +const STRING = { type: "string", address: "0x3000" }; + +describe("the real IDA lane's feature trace assertions", () => { + it("accepts references to every matched address, whoever contains the source", () => { + expect(() => + assertFeatureTrace( + trace( + [SELECTED, SUBSTRING, STRING], + [ + { + target_address: "0x1000", + containing_procedure: containing(true), + }, + // A substring match such as main_helper, referenced from code. + { + target_address: "0x2000", + containing_procedure: containing(true), + }, + // A string containing the query, referenced from data outside functions. + { + target_address: "0x3000", + containing_procedure: containing(false), + }, + ], + ), + procedure, + ), + ).not.toThrow(); + }); + + it("accepts a reference to the selected procedure from data outside functions", () => { + expect(() => + assertFeatureTrace( + trace( + [SELECTED], + [ + { + target_address: "0x1000", + containing_procedure: containing(false), + }, + ], + ), + procedure, + ), + ).not.toThrow(); + }); + + it("rejects a reference whose target is not one of the matches", () => { + expect(() => + assertFeatureTrace( + trace( + [SELECTED], + [ + { + target_address: "0x9999", + containing_procedure: containing(true), + }, + ], + ), + procedure, + ), + ).toThrow(); + }); + + it("rejects a trace without the selected procedure's match", () => { + expect(() => assertFeatureTrace(trace([STRING], []), procedure)).toThrow(); + }); + + it("rejects a containment answer that is neither found nor a known reason", () => { + expect(() => + assertFeatureTrace( + trace( + [SELECTED], + [ + { + target_address: "0x1000", + containing_procedure: { + query_address: "0x1004", + found: false, + procedure: null, + reason: "unknown", + }, + }, + ], + ), + procedure, + ), + ).toThrow(); + }); + + it("rejects a trace that does not say whether it was truncated", () => { + expect(() => + assertFeatureTrace( + { ...trace([SELECTED], []), truncated: undefined }, + procedure, + ), + ).toThrow(); + }); +}); + +describe("the real IDA lane's mapped non-function probe", () => { + const inFunction = { + query_address: "0x1100", + found: true, + procedure: { address: "0x1000", name: "main" }, + }; + const data = { + query_address: "0x3000", + found: false, + procedure: null, + reason: "not_in_procedure", + }; + const resolver = + (answers: Readonly>, asked: string[] = []) => + async (address: string) => { + asked.push(address); + return answers[address]; + }; + + it("skips a string inside a function chunk and uses the next one", async () => { + const asked: string[] = []; + + await expect( + findMappedNonFunction( + [{ address: "0x1100" }, { address: "0x3000" }], + resolver({ "0x1100": inFunction, "0x3000": data }, asked), + ), + ).resolves.toBe("0x3000"); + expect(asked, "strings are tried in order until one qualifies").toEqual([ + "0x1100", + "0x3000", + ]); + }); + + it("stops at the first string that qualifies", async () => { + const asked: string[] = []; + + await expect( + findMappedNonFunction( + [{ address: "0x3000" }, { address: "0x3100" }], + resolver({ "0x3000": data, "0x3100": data }, asked), + ), + ).resolves.toBe("0x3000"); + expect(asked).toEqual(["0x3000"]); + }); + + it("reports no qualifying string when every string is inside a function", async () => { + await expect( + findMappedNonFunction( + [{ address: "0x1100" }, { address: "0x1200" }], + resolver({ "0x1100": inFunction, "0x1200": inFunction }), + ), + ).resolves.toBeUndefined(); + }); + + it("reports no qualifying string when the target has no strings", async () => { + await expect( + findMappedNonFunction([], resolver({})), + ).resolves.toBeUndefined(); + }); + + it("rejects a listed string that resolves as unmapped, since IDA cannot list an unmapped address", async () => { + await expect( + findMappedNonFunction( + [{ address: "0x4000" }, { address: "0x3000" }], + resolver({ + "0x4000": { ...data, reason: "outside_segments" }, + "0x3000": data, + }), + ), + ).rejects.toThrow(); + }); +}); diff --git a/tests/boundary/providers/managed/managedMemberComparisonFromMetadata.test.ts b/tests/boundary/providers/managed/managedMemberComparisonFromMetadata.test.ts index 7f4408811..56014fbd4 100644 --- a/tests/boundary/providers/managed/managedMemberComparisonFromMetadata.test.ts +++ b/tests/boundary/providers/managed/managedMemberComparisonFromMetadata.test.ts @@ -116,6 +116,66 @@ describe("managed member comparison", () => { }); }); +describe("managed member comparison floating-point constants", () => { + it.each([ + { + opcode: "ldc.r4", + positive: "22000000002a", + negative: "22000000802a", + type: 0x0c, + }, + { + opcode: "ldc.r8", + positive: "2300000000000000002a", + negative: "2300000000000000802a", + type: 0x0d, + }, + ])( + "detects a change from positive to negative zero in $opcode", + ({ opcode, positive, negative, type }) => { + const observe = (hex: string, path: string) => { + const il = Buffer.from(hex, "hex"); + return inspect( + buildManagedPeFixture({ + methodSignature: Buffer.from([0x00, 0x00, type]), + ilBody: Buffer.concat([Buffer.from([(il.length << 2) | 2]), il]), + }), + path, + ); + }; + const left = observe(positive, "/tmp/positive-zero.dll"); + const right = observe(negative, "/tmp/negative-zero.dll"); + const comparison = compareManagedMembers(left, right); + + expect(comparison.methods[0]).toMatchObject({ + status: "changed", + match: { status: "matched", basis: "exact-signature" }, + dimensions: ["cil"], + }); + for (const [observation, operand] of [ + [left, "0"], + [right, "-0"], + ] as const) { + expect(observation.result.methods[0]?.body).toMatchObject({ + status: "present", + anchors: [ + { il_offset: 0, opcode, operand_kind: "constant", operand }, + ], + }); + } + expect(left.result.methods[0]?.body.normalized_il_sha256).not.toBe( + right.result.methods[0]?.body.normalized_il_sha256, + ); + const repeated = observe(negative, "/tmp/negative-zero-copy.dll"); + expect(compareManagedMembers(right, repeated).methods[0]).toMatchObject({ + status: "unchanged", + match: { status: "matched", basis: "exact-il-signature" }, + dimensions: [], + }); + }, + ); +}); + describe("managed member comparison uncertainty", () => { it("does not infer a structural method match from undecoded signature fields", () => { const left = inspect( diff --git a/tests/boundary/providers/native/artifactProvider.extractionCompleteness.test.ts b/tests/boundary/providers/native/artifactProvider.extractionCompleteness.test.ts index d42762601..53ffcd7d3 100644 --- a/tests/boundary/providers/native/artifactProvider.extractionCompleteness.test.ts +++ b/tests/boundary/providers/native/artifactProvider.extractionCompleteness.test.ts @@ -3,7 +3,7 @@ import { access, mkdir, readFile, rm, writeFile } from "node:fs/promises"; import { join } from "node:path"; import { createPackage, createPackageWithOptions } from "@electron/asar"; -import { expect, it } from "vitest"; +import { expect, it, vi } from "vitest"; import { ArtifactProvider } from "../../../../src/artifacts/ArtifactProvider.js"; import { @@ -13,6 +13,7 @@ import { import { artifactExtractionResultSchema } from "../../../../src/domain/artifactGraph.js"; import { projectAnalysisError } from "../../../../src/domain/analysisErrorProjection.js"; import { createTestTempDirectory } from "../../../fixtures/temporaryDirectory.js"; +import { DirectoryArtifactReader } from "../../../../src/artifacts/DirectoryArtifactReader.js"; it.each(["file", "directory"] as const)( "rolls back when an inventoried %s disappears before extraction", @@ -61,6 +62,42 @@ it("still refuses a regular file added after inventory and rolls back", async () await expect(access(output)).rejects.toThrow(); }); +it("rolls back when cancellation arrives while closing the materialization reader", async () => { + const root = await createTestTempDirectory("rea-extract-cancel-close-"); + const source = join(root, "source"); + await mkdir(source); + await writeFile(join(source, "payload.txt"), "complete payload\n"); + const snapshot = await scanArtifactInventory(source); + const output = join(root, "output"); + const controller = new AbortController(); + const close = DirectoryArtifactReader.prototype.close; + const closeSpy = vi + .spyOn(DirectoryArtifactReader.prototype, "close") + .mockImplementation(async function (this: DirectoryArtifactReader) { + controller.abort(); + await close.call(this); + }); + try { + await expect( + materializeArtifactInventory( + { + inputPath: source, + inputFormat: "asar", + outputRoot: output, + environment: process.env, + integrityPolicy: "fail", + }, + source, + snapshot, + controller.signal, + ), + ).rejects.toMatchObject({ reason: "cancelled" }); + await expect(access(output)).rejects.toThrow(); + } finally { + closeSpy.mockRestore(); + } +}); + it("preserves every equal-content occurrence and copies a nested ASAR as its containing file", async () => { const root = await createTestTempDirectory("rea-extract-nested-"); const source = join(root, "source"); @@ -98,7 +135,7 @@ it("ignores unavailable members of a nested ASAR but refuses them as active entr const source = join(root, "bundle"); await mkdir(source); const archive = join(source, "app.asar"); - await createPackageWithOptions(contents, archive, { unpack: "**/*.node" }); + await createPackageWithOptions(contents, archive, { unpack: "*.node" }); await rm(join(`${archive}.unpacked`, "native", "addon.node")); const nestedOutput = join(root, "nested-output"); diff --git a/tests/boundary/providers/native/artifactProvider.extractionFormat.test.ts b/tests/boundary/providers/native/artifactProvider.extractionFormat.test.ts index 81b3cce52..e4accd098 100644 --- a/tests/boundary/providers/native/artifactProvider.extractionFormat.test.ts +++ b/tests/boundary/providers/native/artifactProvider.extractionFormat.test.ts @@ -3,10 +3,12 @@ import { join } from "node:path"; import { describe, expect, it } from "vitest"; +import { writeOrderedZip } from "../../../fixtures/artifactEntryOrder.js"; import { createTestTempDirectory } from "../../../fixtures/temporaryDirectory.js"; import { ArtifactProvider } from "../../../../src/artifacts/ArtifactProvider.js"; import { artifactExtractionExecutionSchema } from "../../../../src/contracts/artifactToolContracts.js"; +import { artifactExtractionResultSchema } from "../../../../src/domain/artifactGraph.js"; import type { BinaryTarget } from "../../../../src/domain/binaryTargetTypes.js"; describe("artifact extraction format support", () => { @@ -42,6 +44,33 @@ describe("artifact extraction format support", () => { }); await expect(access(output)).rejects.toThrow(); }); + + it.each(["zip", "hap", "hsp", "app-pack"] as const)( + "extracts the ZIP-backed %s package family that inventory already supports", + async (format) => { + const root = await createTestTempDirectory("rea-extract-zip-family-"); + const archive = join( + root, + `Payload.${format === "app-pack" ? "app" : format}`, + ); + await writeOrderedZip(archive, ["payload.txt"]); + const result = await new ArtifactProvider(process.env) + .createClient(archiveTarget(archive, format)) + .execute( + "extract_artifact", + artifactExtractionExecutionSchema.parse({ + output_root: join(root, "output"), + }), + ); + if (!result.ok) throw result.error; + const extracted = artifactExtractionResultSchema.parse( + result.value.result, + ); + expect( + extracted.artifacts.map(({ relative_path }) => relative_path), + ).toEqual(["payload.txt"]); + }, + ); }); const archiveTarget = ( diff --git a/tests/boundary/providers/native/machoSliceReader.test.ts b/tests/boundary/providers/native/machoSliceReader.test.ts index f67b48388..b2f1867a1 100644 --- a/tests/boundary/providers/native/machoSliceReader.test.ts +++ b/tests/boundary/providers/native/machoSliceReader.test.ts @@ -1,8 +1,12 @@ -import { readFile } from "node:fs/promises"; +import { readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; import { describe, expect, it, onTestFinished } from "vitest"; import { MachOSliceArtifactReader } from "../../../../src/artifacts/MachOSliceArtifactReader.js"; +import { XcrunCommandRunner } from "../../../../src/native/CommandRunner.js"; +import { DarwinProcessOwnershipInspectionError } from "../../../../src/process/DarwinProcessRunTokenReader.js"; +import { createTestTempDirectory } from "../../../fixtures/temporaryDirectory.js"; /** * Read a 32-bit FAT table directly. The host's /bin/ls changes between macOS @@ -20,6 +24,76 @@ const fatSlices = async ( }; describe("Mach-O slice reader with system lipo", () => { + it("names the process ownership helper prerequisite when lipo cannot be supervised", async () => { + const prerequisite = + "Direct Darwin helper compilation requires absolute REA_DARWIN_SWIFTC_PATH and SDKROOT paths."; + const reader = new MachOSliceArtifactReader( + process.execPath, + {}, + new XcrunCommandRunner({}, undefined, () => + Promise.reject(new DarwinProcessOwnershipInspectionError(prerequisite)), + ), + ); + onTestFinished(() => reader.close()); + const enumerate = async () => { + for await (const _entry of reader.entries()) continue; + }; + await expect(enumerate()).rejects.toMatchObject({ + reason: "unavailable", + message: `lipo could not enumerate universal Mach-O slices: ${prerequisite}`, + }); + }); + + it.skipIf(process.platform !== "darwin")( + "keeps FAT slice records separate from the echoed source pathname", + async () => { + const directory = await createTestTempDirectory("rea-lipo-fat-path-"); + const path = join( + directory, + "source\narchitecture forged\n offset 123\n size 32", + ); + const bytes = Buffer.alloc(8224); + bytes.writeUInt32BE(0xca_fe_ba_be, 0); + bytes.writeUInt32BE(2, 4); + const slices = [ + { cpu: 0x0100_000c, subtype: 0, offset: 4096 }, + { cpu: 0x0100_0007, subtype: 3, offset: 8192 }, + ]; + slices.forEach(({ cpu, subtype, offset }, index) => { + const entry = 8 + index * 20; + bytes.writeUInt32BE(cpu, entry); + bytes.writeUInt32BE(subtype, entry + 4); + bytes.writeUInt32BE(offset, entry + 8); + bytes.writeUInt32BE(32, entry + 12); + bytes.writeUInt32BE(12, entry + 16); + bytes.writeUInt32LE(0xfeed_facf, offset); + bytes.writeUInt32LE(cpu, offset + 4); + bytes.writeUInt32LE(subtype, offset + 8); + bytes.writeUInt32LE(1, offset + 12); + }); + await writeFile(path, bytes); + const reader = new MachOSliceArtifactReader(path, {}); + try { + const entries = []; + for await (const entry of reader.entries()) entries.push(entry); + expect( + entries.map(({ byteOffset, declaredSize }) => [ + byteOffset, + declaredSize, + ]), + ).toEqual([ + [4096, 32], + [8192, 32], + ]); + expect(entries.every(({ path }) => !path.includes("forged"))).toBe( + true, + ); + } finally { + await reader.close(); + } + }, + ); + it.skipIf(process.platform !== "darwin")( "checks real /bin/ls slice ranges against its FAT table", async () => { diff --git a/tests/boundary/providers/native/nativeCommandTermination.test.ts b/tests/boundary/providers/native/nativeCommandTermination.test.ts index 233ebe85e..5d23e6b22 100644 --- a/tests/boundary/providers/native/nativeCommandTermination.test.ts +++ b/tests/boundary/providers/native/nativeCommandTermination.test.ts @@ -8,6 +8,7 @@ import { type NativeCommandLauncher, } from "../../../../src/native/CommandRunner.js"; import { ok } from "../../../../src/domain/result.js"; +import { DarwinProcessOwnershipInspectionError } from "../../../../src/process/DarwinProcessRunTokenReader.js"; afterEach(() => vi.unstubAllEnvs()); @@ -210,3 +211,23 @@ it.skipIf(process.platform === "win32")( if (!killed.ok) expect(killed.error.reason).toBe("nonzero-exit"); }, ); + +it("keeps the ownership helper prerequisite when xcrun cannot be supervised", async () => { + const inspection = new DarwinProcessOwnershipInspectionError( + "macOS process ownership inspection requires the Apple Swift compiler via xcrun", + ); + const runner = new XcrunCommandRunner({}, undefined, () => + Promise.reject(inspection), + ); + + const result = await runner.run("file", [], {}); + + expect(result.ok).toBe(false); + if (result.ok) return; + expect(result.error).toMatchObject({ + tool: "file", + reason: "io", + detail: inspection.message, + cause: { tool: "xcrun", detail: inspection.message, cause: inspection }, + }); +}); diff --git a/tests/boundary/providers/native/nativeDmgArtifactReader.test.ts b/tests/boundary/providers/native/nativeDmgArtifactReader.test.ts index 9e03727c1..5d6af987d 100644 --- a/tests/boundary/providers/native/nativeDmgArtifactReader.test.ts +++ b/tests/boundary/providers/native/nativeDmgArtifactReader.test.ts @@ -1200,6 +1200,138 @@ describe("native DMG command diagnostics", () => { }); }); +describe("native DMG verification resource availability", () => { + const transientFailure = { + stdout: "", + stderr: + 'hdiutil: verify: unable to recognize "/tmp/image.dmg" as a disk image. (Resource temporarily unavailable)\nhdiutil: verify failed - Resource temporarily unavailable\n', + exitCode: 1, + }; + + it("retries the explicit transient refusal before attachment and retains both attempts", async () => { + const { host, state } = createOmittedInventoryHost(); + state.inventoryMode = "listed"; + state.detachAttempts = 4; + const delays = vi.fn(async () => {}); + const timeouts: number[] = []; + let attempts = 0; + const reader = await createAttachedReader( + "/tmp/image.dmg", + process.env, + undefined, + { + ...host, + delay: delays, + async run(arguments_, signal, options) { + if (arguments_[0] === "verify") { + attempts += 1; + if (options === undefined) + throw new Error("missing verify deadline"); + timeouts.push(options.timeoutMs); + if (attempts === 1) return transientFailure; + } + return host.run(arguments_, signal, options); + }, + }, + ); + onTestFinished(() => reader.close()); + expect(attempts).toBe(2); + expect(delays.mock.calls).toEqual([[250]]); + expect(timeouts[0]).toBeLessThanOrEqual(120_000); + expect(timeouts[1]).toBeLessThanOrEqual(timeouts[0]!); + expect(reader.provenance().slice(0, 2)).toMatchObject([ + { + arguments: ["verify", "/tmp/image.dmg"], + exit_code: 1, + effects: ["read"], + }, + { + arguments: ["verify", "/tmp/image.dmg"], + exit_code: 0, + effects: ["read"], + }, + ]); + await reader.close(); + expect(state.attached).toBe(false); + }); + + it("bounds repeated transient refusals and never attaches an unverified image", async () => { + const run = vi.fn(async () => transientFailure); + const delay = vi.fn(async () => {}); + const reader = new NativeDmgArtifactReader("/tmp/image.dmg", process.env, { + run, + delay, + }); + await expect(entryPaths(reader)).rejects.toMatchObject({ + reason: "unavailable", + message: expect.stringContaining("Resource temporarily unavailable"), + }); + await reader.close(); + expect(run.mock.calls).toHaveLength(3); + expect(delay.mock.calls).toEqual([[250], [500]]); + expect(reader.provenance().map(({ exit_code }) => exit_code)).toEqual([ + 1, 1, 1, + ]); + }); + + it("honors cancellation during the wait without starting another command", async () => { + const controller = new AbortController(); + const run = vi.fn(async () => transientFailure); + await expect( + createAttachedReader("/tmp/image.dmg", process.env, controller.signal, { + run, + async delay() { + controller.abort(new Error("cancel verification")); + }, + }), + ).rejects.toMatchObject({ reason: "cancelled" }); + expect(run).toHaveBeenCalledTimes(1); + }); + + it("does not start another verification after the shared deadline", async () => { + const clock = vi.spyOn(performance, "now").mockReturnValue(0); + const run = vi.fn(async () => transientFailure); + try { + await expect( + createAttachedReader("/tmp/image.dmg", process.env, undefined, { + run, + async delay() { + clock.mockReturnValue(120_001); + }, + }), + ).rejects.toMatchObject({ + reason: "unavailable", + message: expect.stringContaining("Resource temporarily unavailable"), + }); + expect(run).toHaveBeenCalledTimes(1); + } finally { + clock.mockRestore(); + } + }); + + it.each([ + ["image data corrupted", "integrity"], + ["image not recognized", "format"], + ["Permission denied", "io"], + ["an unknown failure", "unavailable"], + ])("does not retry %s", async (diagnostic, reason) => { + const run = vi.fn(async () => ({ + stdout: "", + stderr: `hdiutil: verify failed - ${diagnostic}\n`, + exitCode: 1, + })); + const delay = vi.fn(async () => {}); + await expect( + createAttachedReader("/tmp/image.dmg", process.env, undefined, { + run, + delay, + }), + ).rejects.toMatchObject({ reason }); + expect(run).toHaveBeenCalledTimes(1); + expect(delay).not.toHaveBeenCalled(); + }); +}); + describe("native DMG real verification diagnostics", () => { it.skipIf(process.platform !== "darwin")( "classifies a real hdiutil verify refusal as a format failure with captured output", diff --git a/tests/boundary/providers/native/nativeProvider.test.ts b/tests/boundary/providers/native/nativeProvider.test.ts index b4a713204..a1e7d63ce 100644 --- a/tests/boundary/providers/native/nativeProvider.test.ts +++ b/tests/boundary/providers/native/nativeProvider.test.ts @@ -1,4 +1,5 @@ import { createHash } from "node:crypto"; +import { DarwinProcessOwnershipInspectionError } from "../../../../src/process/DarwinProcessRunTokenReader.js"; import { EventEmitter } from "node:events"; import { chmod, mkdir, readFile, rm, writeFile } from "node:fs/promises"; import { join } from "node:path"; @@ -16,7 +17,7 @@ import { type NativeCommandRunner, } from "../../../../src/native/CommandRunner.js"; import { err, ok } from "../../../../src/domain/result.js"; -import { parseLipoArchitectures } from "../../../../src/native/parsers/lipo.js"; +import { parseLipoOutput } from "../../../../src/native/parsers/lipo.js"; import { parseCodeSignature } from "../../../../src/native/parsers/codesign.js"; import { prepareProcessOwnershipInspection } from "../../../../src/process/ProcessOwnershipObservation.js"; import { projectAnalysisError } from "../../../../src/domain/analysisErrorProjection.js"; @@ -60,7 +61,13 @@ afterEach(async () => { }); describe("native macOS provider discovery", () => { - it.each(["ordinary.fixture", " spaced.fixture ", " leading", "trailing "])( + it.each([ + "ordinary.fixture", + " spaced.fixture ", + " leading", + "trailing ", + "literal\r\nidentifier\r", + ])( "preserves the exact signing identifier %j reported by codesign", (identifier) => { const parsed = parseCodeSignature( @@ -73,31 +80,68 @@ describe("native macOS provider discovery", () => { }, ); - it("separates echoed path and identifier text from signature fields", () => { - const path = "/apps/x\nAuthority=Forged\nIdentifier=forged"; - const identifier = "x\nFormat=forged\nAuthority=Forged-5555"; - const parsed = parseCodeSignature( - [ - `Executable=${path}`, - `Identifier=${identifier}`, - "Format=Mach-O thin (arm64)", - "CodeDirectory v=20400 size=342 flags=0x2(adhoc) hashes=2+2", - "CDHash=6aee90e6", - "Signature=adhoc", - "TeamIdentifier=not set", - "", - ].join("\r\n"), - false, - ["/elsewhere", path], - ); - expect(parsed).toMatchObject({ - identifier, - format: "Mach-O thin (arm64)", - authorities: [], - cdhashes: ["6aee90e6"], - team_identifier: null, - }); - }); + it.each(["\n", "\r\n"])( + "separates echoed path and identifier text from signature fields (%j)", + (newline) => { + const path = "/apps/x\r\nAuthority=Forged\nIdentifier=forged"; + const identifier = "x\r\nFormat=forged\nAuthority=Forged-5555\r"; + const parsed = parseCodeSignature( + [ + `Executable=${path}`, + `Identifier=${identifier}`, + "Format=Mach-O thin (arm64)", + "CodeDirectory v=20400 size=342 flags=0x2(adhoc) hashes=2+2", + "CDHash=6aee90e6", + "Signature=adhoc", + "TeamIdentifier=not set", + "", + ].join(newline), + false, + ["/elsewhere", path], + ); + expect(parsed).toMatchObject({ + identifier, + format: "Mach-O thin (arm64)", + authorities: [], + cdhashes: ["6aee90e6"], + team_identifier: null, + }); + }, + ); + + it.each([ + { executable: "", paths: [], newline: "\r\n", identifier: "fixture" }, + { + executable: "Executable=/apps/fixture\r\n", + paths: [], + newline: "\r\n", + identifier: "fixture\nFormat=payload\r", + }, + { + executable: "Executable=/apps/fixture\n", + paths: ["/elsewhere"], + newline: "\n", + identifier: "fixture\r\nFormat=payload\r", + }, + { + executable: "Executable=/apps/fixture\r\n", + paths: ["/elsewhere"], + newline: "\r\n", + identifier: "fixture\r", + }, + ])( + "frames the identifier without a matching executable path: %j", + ({ executable, paths, newline, identifier }) => { + const parsed = parseCodeSignature( + `${executable}Identifier=${identifier}${newline}Format=Mach-O thin (arm64)${newline}TeamIdentifier=not set${newline}`, + false, + paths, + ); + expect(parsed.identifier).toBe(identifier); + expect(parsed.format).toBe("Mach-O thin (arm64)"); + expect(parsed.team_identifier).toBeNull(); + }, + ); it.each(["timeout", "output-limit"] as const)( "preserves %s resolution failures with their captured output and cleanup status", @@ -280,6 +324,30 @@ describe("native command failure capture and limits", () => { stderr_bytes: expect.any(Number), }, }); + expect(projectAnalysisError(result.error).details).not.toHaveProperty( + "diagnostics.detail", + ); + }); + + it("projects the ownership helper prerequisite when a command cannot be supervised", async () => { + const inspection = new DarwinProcessOwnershipInspectionError( + "Direct Darwin helper compilation requires absolute REA_DARWIN_SWIFTC_PATH and SDKROOT paths.", + ); + const runner = new XcrunCommandRunner( + {}, + () => + Promise.resolve(ok({ path: process.execPath, sha256: "a".repeat(64) })), + () => Promise.reject(inspection), + ); + const result = await new NativeMacOSProvider({}, runner, "darwin") + .createClient(machoTarget("/private/fixture")) + .execute("demangle_swift", { symbols: ["fixture-symbol"] }); + + expect(result.ok).toBe(false); + if (result.ok) return; + expect(projectAnalysisError(result.error).details).toMatchObject({ + diagnostics: { reason: "io", detail: inspection.message }, + }); }); it("retains output written before cancellation and reaps the stalled command", async () => { @@ -892,15 +960,17 @@ describe("native macOS provider failures and parsing", () => { }); it("parses thin and universal lipo fixtures", async () => { - expect( - parseLipoArchitectures(await fixture("lipo-thin.txt")), - ).toMatchObject([{ name: "arm64" }]); - expect(parseLipoArchitectures(await fixture("lipo-fat.txt"))).toMatchObject( - [ + expect(parseLipoOutput(await fixture("lipo-thin.txt"))).toMatchObject({ + format: "thin", + architectures: [{ name: "arm64" }], + }); + expect(parseLipoOutput(await fixture("lipo-fat.txt"))).toMatchObject({ + format: "fat", + architectures: [ { name: "x86_64", file_offset: 16384, size: 4096 }, { name: "arm64", file_offset: 32768, size: 8192 }, ], - ); + }); }); it("rejects trailing text in lipo numeric fields instead of truncating", () => { @@ -912,7 +982,7 @@ describe("native macOS provider failures and parsing", () => { " size 0x10", " align 2^14 (16384)", ].join("\n"); - expect(parseLipoArchitectures(output)).toMatchObject([ + expect(parseLipoOutput(output).architectures).toMatchObject([ { name: "arm64", file_offset: null, size: null, alignment: 16384 }, ]); }); @@ -925,10 +995,11 @@ describe("native macOS provider failures and parsing", () => { " align 2^14 (16384)", ].join("\n"); expect( - parseLipoArchitectures(output.replace("(16384)", "(8192)"))[0]?.alignment, + parseLipoOutput(output.replace("(16384)", "(8192)")).architectures[0] + ?.alignment, ).toBeNull(); expect( - parseLipoArchitectures(output.replace("2^14 (16384)", "2^14"))[0] + parseLipoOutput(output.replace("2^14 (16384)", "2^14")).architectures[0] ?.alignment, ).toBe(16384); }); @@ -953,6 +1024,38 @@ describe("native macOS provider failures and parsing", () => { }); }); +describe("native signature inspection target scope", () => { + // This provider's inspect_signature path is scoped to Mach-O targets; on + // another container a codesign verdict cannot see a format-level signature + // such as a xar pkg signature. + it.each([ + { + path: "/fixture.pkg", + sha256: "0".repeat(64), + kind: "archive" as const, + format: "pkg" as const, + }, + { + ...machoTarget("/fixture.elf"), + format: "elf" as const, + }, + ])( + "refuses inspect_signature on a $format target without running codesign", + async (target) => { + const runner = new CountingRunner(); + const result = await new NativeMacOSProvider({}, runner, "darwin") + .createClient(target) + .execute("inspect_signature", {}); + if (result.ok) throw new Error("Expected a non-Mach-O refusal"); + expect(result.error._tag).toBe("AnalysisCapabilityUnavailableError"); + expect(projectAnalysisError(result.error)).toMatchObject({ + code: "capability_unavailable", + }); + expect(runner.calls).toBe(0); + }, + ); +}); + class FailingRunner implements NativeCommandRunner { constructor(private readonly reason: NativeCommandFailure["reason"]) {} diff --git a/tests/boundary/providers/native/nativeSelectedArchitecture.test.ts b/tests/boundary/providers/native/nativeSelectedArchitecture.test.ts index 006b13003..182c33612 100644 --- a/tests/boundary/providers/native/nativeSelectedArchitecture.test.ts +++ b/tests/boundary/providers/native/nativeSelectedArchitecture.test.ts @@ -1,7 +1,11 @@ import { expect, it } from "vitest"; +import { CPU } from "../../../../src/artifacts/apple/MachoImage.fixture.js"; import { inspectMachoSchema } from "../../../../src/domain/native/nativeInspection.js"; import { err } from "../../../../src/domain/result.js"; -import { NativeCommandFailure } from "../../../../src/native/CommandRunner.js"; +import { + NativeCommandFailure, + type NativeCommandRunner, +} from "../../../../src/native/CommandRunner.js"; import { NativeMacOSProvider } from "../../../../src/native/NativeMacOSProvider.js"; import { NativeFixtureRunner, @@ -9,6 +13,35 @@ import { nativeMachoTarget, } from "../../../fixtures/nativeCommands.js"; +/** Replay fixtures, failing any slice command that does not name the expected slice. */ +const sliceSelectingRunner = ( + fixtures: NativeFixtureRunner, + selected: string, + dyldArchitecture: string, + calls: { + readonly tool: string; + readonly arguments_: readonly string[]; + }[] = [], +): NativeCommandRunner => ({ + run(tool, arguments_) { + calls.push({ tool, arguments_ }); + const expected = tool === "dyld_info" ? dyldArchitecture : selected; + if ( + !["file", "lipo"].includes(tool) && + !arguments_.some( + (argument, index) => + argument === expected || + argument === `--arch=${expected}` || + (argument === "-arch" && arguments_[index + 1] === expected), + ) + ) + return Promise.resolve( + err(new NativeCommandFailure(tool, "nonzero-exit", 1)), + ); + return fixtures.run(tool, arguments_); + }, +}); + it("does not attribute the first universal slice's UUID and segments to the selected target", async () => { const singleSlice = await nativeFixture("otool-load.txt"); const selected = new NativeFixtureRunner(); @@ -58,18 +91,39 @@ it.each([ selected: "arm64e", dyldArchitecture: "arm64e.v1", }, -])( + { + architectures: ["arm64v8", "arm64e.v1"], + selected: "arm64e", + dyldArchitecture: "arm64e.v1", + }, + { + architectures: ["arm64e.v1", "arm64v8"], + selected: "arm64e", + dyldArchitecture: "arm64e.v1", + }, +] as const)( "selects the $dyldArchitecture slice of a universal arm64e target", async ({ architectures, selected, dyldArchitecture }) => { const calls: { readonly tool: string; readonly arguments_: readonly string[]; }[] = []; + const cpus = { + x86_64: CPU.x86_64, + arm64e: CPU.arm64e, + arm64v8: { type: CPU.arm64.type, subtype: 1 }, + "arm64e.x1": { type: CPU.arm64.type, subtype: 0x8000000c }, + "arm64e.v1": { type: CPU.arm64.type, subtype: 0x81000002 }, + }; const lipo = architectures - .map( - (architecture, index) => - `architecture ${architecture}\n cputype 16777228\n cpusubtype 2\n offset ${index * 16384}\n size 8192\n align 2^14 (16384)`, - ) + .map((architecture, index) => { + const cpu = cpus[architecture]; + const subtype = + architecture === "arm64e.v1" + ? "CPU_SUBTYPE_ARM64E\n capabilities PTR_AUTH_VERSION USERSPACE 1" + : String(cpu.subtype); + return `architecture ${architecture}\n cputype ${cpu.type}\n cpusubtype ${subtype}\n offset ${index * 16384}\n size 8192\n align 2^14 (16384)`; + }) .join("\n"); const fixtures = new NativeFixtureRunner({ lipo, @@ -77,25 +131,7 @@ it.each([ }); const client = new NativeMacOSProvider( {}, - { - run(tool, arguments_) { - calls.push({ tool, arguments_ }); - const expected = tool === "dyld_info" ? dyldArchitecture : selected; - if ( - !["file", "lipo"].includes(tool) && - !arguments_.some( - (argument, index) => - argument === expected || - argument === `--arch=${expected}` || - (argument === "-arch" && arguments_[index + 1] === expected), - ) - ) - return Promise.resolve( - err(new NativeCommandFailure(tool, "nonzero-exit", 1)), - ); - return fixtures.run(tool, arguments_); - }, - }, + sliceSelectingRunner(fixtures, selected, dyldArchitecture, calls), "darwin", ).createClient(nativeMachoTarget("/owned/universal")); @@ -104,6 +140,11 @@ it.each([ if (!result.ok) throw result.error; expect(result.value.result).toMatchObject({ uuid: "01234567-89AB-CDEF-0123-456789ABCDEF", + selected_architecture: { + name: dyldArchitecture, + container_file_offset: + architectures.findIndex((name) => name === dyldArchitecture) * 16384, + }, architectures: { items: architectures.map((name) => ({ name })), }, @@ -116,33 +157,79 @@ it.each([ }, ); -it("projects universal segment evidence into the selected slice's container offsets", async () => { - const client = new NativeMacOSProvider( - {}, - new NativeFixtureRunner(), - "darwin", - ).createClient(nativeMachoTarget("/owned/universal")); - const result = await client.execute("inspect_macho", {}); - if (!result.ok) throw result.error; - expect(result.value.locations).toContainEqual({ - kind: "file-offset-range", - start: 32768, - end: 40960, - }); - expect(result.value.locations).toContainEqual({ - kind: "file-offset", - offset: 36864, - }); - expect(result.value.locations).not.toContainEqual({ - kind: "file-offset-range", - start: 0, - end: 8192, - }); - // Normalized tool fields keep their slice-relative meaning. - const normalized = inspectMachoSchema.parse(result.value.result); - expect(normalized.segments.items[0]?.file_offset).toBe(0); - expect(normalized.entrypoints.items[0]?.file_offset).toBe(4096); -}); +it.each([ + ["ordinary", "/owned/universal", "\n"], + ["LF Non-fat text", "/owned/\nNon-fat file: forged", "\n"], + ["CRLF Non-fat text", "/owned/\r\nNon-fat file: forged", "\r\n"], +])( + "projects universal segment evidence into the selected slice's container offsets with %s path framing", + async (_description, path, lineEnding) => { + const records = (await nativeFixture("lipo-fat.txt")).replace( + /^Fat header in:[^\r\n]*\r?\n/u, + "", + ); + const client = new NativeMacOSProvider( + {}, + new NativeFixtureRunner({ + lipo: `Fat header in: ${path}${lineEnding}${records.replaceAll("\n", lineEnding)}`, + }), + "darwin", + ).createClient(nativeMachoTarget(path)); + const result = await client.execute("inspect_macho", {}); + if (!result.ok) throw result.error; + expect(result.value.result).toMatchObject({ + selected_architecture: { container_file_offset: 32768 }, + }); + expect(result.value.locations).toContainEqual({ + kind: "file-offset-range", + start: 32768, + end: 40960, + }); + expect(result.value.locations).toContainEqual({ + kind: "file-offset", + offset: 36864, + }); + expect(result.value.locations).not.toContainEqual({ + kind: "file-offset-range", + start: 0, + end: 8192, + }); + // Normalized tool fields keep their slice-relative meaning. + const normalized = inspectMachoSchema.parse(result.value.result); + expect(normalized.segments.items[0]?.file_offset).toBe(0); + expect(normalized.entrypoints.items[0]?.file_offset).toBe(4096); + }, +); + +it.each([ + { capture: "calculator.txt", selected: "arm64e" }, + { capture: "shortcut-droplet.txt", selected: "arm64" }, +])( + "selects one slice when every universal slice has the target's CPU type ($capture)", + async ({ capture, selected }) => { + const base = nativeMachoTarget("/owned/universal"); + if (base.kind !== "executable") + throw new Error("Expected executable fixture"); + // Admission names arm64, arm64e and arm64e.x1 slices by one CPU family. + const target = { ...base, availableArchitectures: ["arm64"] as const }; + const fixtures = new NativeFixtureRunner({ + lipo: await nativeFixture(`lipo-same-cpu/${capture}`), + }); + const result = await new NativeMacOSProvider( + {}, + sliceSelectingRunner(fixtures, selected, selected), + "darwin", + ) + .createClient(target) + .execute("inspect_macho", {}); + + if (!result.ok) throw result.error; + expect(result.value.result).toMatchObject({ + selected_architecture: { name: selected, container_file_offset: 16384 }, + architectures: { total: 2 }, + }); + }, +); it.each([ { diff --git a/tests/boundary/providers/native/nativeTextRobustness.test.ts b/tests/boundary/providers/native/nativeTextRobustness.test.ts index 256aa0a98..ce82bd520 100644 --- a/tests/boundary/providers/native/nativeTextRobustness.test.ts +++ b/tests/boundary/providers/native/nativeTextRobustness.test.ts @@ -2,7 +2,7 @@ import { expect, it } from "vitest"; import { parseOtoolLoadCommands } from "../../../../src/native/parsers/otool.js"; import { parsePlistJson } from "../../../../src/native/parsers/plist.js"; -import { parseLipoArchitectures } from "../../../../src/native/parsers/lipo.js"; +import { parseLipoOutput } from "../../../../src/native/parsers/lipo.js"; import { nativeFixture } from "../../../fixtures/nativeCommands.js"; const toCrlf = (value: string): string => value.replaceAll("\n", "\r\n"); @@ -15,7 +15,10 @@ it.each([ "retains the reported thin architecture when lipo echoes filename %j", (filename) => { const output = `input file /owned/${filename} is not a fat file\nNon-fat file: /owned/${filename} is architecture: arm64\n`; - expect(parseLipoArchitectures(output)).toMatchObject([{ name: "arm64" }]); + expect(parseLipoOutput(output)).toMatchObject({ + format: "thin", + architectures: [{ name: "arm64" }], + }); }, ); @@ -38,10 +41,13 @@ it("does not mistake fat slice output for a thin Non-fat file record", () => { " align 2^12 (4096)", "", ].join("\n"); - expect(parseLipoArchitectures(output)).toMatchObject([ - { name: "arm64", file_offset: 16384 }, - { name: "x86_64", file_offset: 16793600 }, - ]); + expect(parseLipoOutput(output)).toMatchObject({ + format: "fat", + architectures: [ + { name: "arm64", file_offset: 16384 }, + { name: "x86_64", file_offset: 16793600 }, + ], + }); }); it("keeps section blocks separate under CRLF captures", async () => { diff --git a/tests/boundary/setup/macosHopperPublisherVerification.test.ts b/tests/boundary/setup/macosHopperPublisherVerification.test.ts new file mode 100644 index 000000000..29b696e85 --- /dev/null +++ b/tests/boundary/setup/macosHopperPublisherVerification.test.ts @@ -0,0 +1,108 @@ +import { execFile } from "node:child_process"; +import { appendFile, copyFile, mkdir, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { promisify } from "node:util"; + +import { describe, expect, it } from "vitest"; + +import { + HOPPER_PUBLISHER, + macHopperInstallDisclosure, + systemMacHopperInstallHost, +} from "../../../src/application/MacHopper.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +const run = promisify(execFile); +const itDarwin = it.skipIf(process.platform !== "darwin"); +const executableSegments = ["Contents", "MacOS", "hopper"]; + +const infoPlist = (identifier: string): string => + ` + + + + CFBundleIdentifier + ${identifier} + CFBundleExecutable + hopper + CFBundlePackageType + APPL + + +`; + +/** Build a minimal `.app` whose launcher is a real signed Mach-O copy. */ +const buildBundle = async ( + root: string, + identifier: string, + options: { readonly sign?: boolean } = {}, +): Promise => { + const app = join(root, "Hopper Disassembler.app"); + const executable = join(app, ...executableSegments); + await mkdir(join(app, "Contents", "MacOS"), { recursive: true }); + await copyFile("/usr/bin/true", executable); + await writeFile(join(app, "Contents", "Info.plist"), infoPlist(identifier)); + if (options.sign !== false) + await run("/usr/bin/codesign", [ + "--sign", + "-", + "--force", + "--deep", + "--identifier", + identifier, + app, + ]); + return app; +}; + +const host = () => + systemMacHopperInstallHost("/home/rea-test", { ...process.env }); + +describe("macOS Hopper publisher verification", () => { + itDarwin( + "rejects an ad-hoc signed bundle carrying the pinned bundle identifier", + async () => { + const root = await createTestTempDirectory("rea-hopper-publisher-"); + const app = await buildBundle(root, HOPPER_PUBLISHER.bundleIdentifier); + // The pre-pinned checks accept this bundle: `codesign --verify` succeeds + // for ad-hoc signatures and the identifier clause alone also matches. + await run("/usr/bin/codesign", ["--verify", "--deep", "--strict", app]); + await expect(host().validBundle(app)).resolves.toBe( + "unverified_publisher", + ); + }, + ); + + itDarwin("rejects an unsigned bundle before any copy", async () => { + const root = await createTestTempDirectory("rea-hopper-unsigned-"); + const app = await buildBundle(root, HOPPER_PUBLISHER.bundleIdentifier, { + sign: false, + }); + await expect(host().validBundle(app)).resolves.toBe("invalid_signature"); + }); + + itDarwin("rejects a bundle modified after it was signed", async () => { + const root = await createTestTempDirectory("rea-hopper-tampered-"); + const app = await buildBundle(root, HOPPER_PUBLISHER.bundleIdentifier); + await appendFile(join(app, ...executableSegments), "appended\n"); + await expect(host().validBundle(app)).resolves.toBe("invalid_signature"); + }); + + it("discloses the exact code requirement the installer enforces", () => { + expect(macHopperInstallDisclosure.commands).toContain( + `codesign --verify --strict '-R= identifier "${HOPPER_PUBLISHER.bundleIdentifier}" and anchor apple generic and certificate leaf[subject.OU] = "${HOPPER_PUBLISHER.teamIdentifier}"' `, + ); + }); + + // Opt-in positive control: point this at a mounted official Hopper bundle, + // for example `hdiutil attach -readonly -nobrowse Hopper-6.6.0-demo.dmg`. + // It proves the pinned requirement is satisfiable by the real publisher, so + // a quoting or argument mistake cannot hide behind the rejection arms. + const realApp = process.env.REA_HOPPER_MACOS_APP; + it.skipIf(realApp === undefined)( + "accepts the officially signed Hopper application", + async () => { + await expect(host().validBundle(realApp ?? "")).resolves.toBe("verified"); + }, + ); +}); diff --git a/tests/boundary/signals/decoderDiscovery.test.ts b/tests/boundary/signals/decoderDiscovery.test.ts new file mode 100644 index 000000000..6b69d679f --- /dev/null +++ b/tests/boundary/signals/decoderDiscovery.test.ts @@ -0,0 +1,48 @@ +import { expect, it } from "vitest"; +import { inspectSignalDecodersInputSchema } from "../../../src/domain/signals/signalDecoders.js"; +import { parseDecoderDiscovery } from "../../../src/signals/parseDecoderDiscovery.js"; + +it("rejects malformed catalogs and the wrong decoder identity", () => { + for (const text of [ + "", + "Supported protocol decoders:\n", + "Supported protocol decoders:\n uart UART\n uart Duplicate", + "Supported protocol decoders:\n malformed", + ]) { + expect(() => + parseDecoderDiscovery(text, inspectSignalDecodersInputSchema.parse({})), + ).toThrow(); + } + expect(() => + parseDecoderDiscovery( + "ID: spi\nOptions:", + inspectSignalDecodersInputSchema.parse({ decoder: "uart" }), + ), + ).toThrow(); +}); + +it("keeps help line ordinals, blank lines and literal special characters through selection", () => { + const text = "ID: uart\r\nOptions:\r\n\r\n- mode: A[0] (default 'x')\r\n"; + const result = parseDecoderDiscovery( + text, + inspectSignalDecodersInputSchema.parse({ + decoder: "uart", + query: "[0]", + offset: 0, + limit: 1, + }), + ); + expect(result.help_lines).toEqual([ + { line_number: 4, text: "- mode: A[0] (default 'x')" }, + ]); + expect(result.output_window.examined).toBe(4); +}); + +it("bounds line expansion before constructing help records", () => { + expect(() => + parseDecoderDiscovery( + "ID: uart\n" + "\n".repeat(65536), + inspectSignalDecodersInputSchema.parse({ decoder: "uart" }), + ), + ).toThrow("representation budget"); +}); diff --git a/tests/boundary/signals/recordingProvider.test.ts b/tests/boundary/signals/recordingProvider.test.ts index 9ddf2c21a..27bc6ff09 100644 --- a/tests/boundary/signals/recordingProvider.test.ts +++ b/tests/boundary/signals/recordingProvider.test.ts @@ -1,3 +1,4 @@ +import { signalDecodersSchema } from "../../../src/domain/signals/signalDecoders.js"; import { OwnedCommandFailure } from "../../../src/process/OwnedCommand.js"; import { it, expect, onTestFinished } from "vitest"; import { writeFile, readdir, chmod, symlink } from "node:fs/promises"; @@ -23,13 +24,17 @@ const fixture = async (mode = "ok", failCleanup = false) => { await writeFile( script, `import {readFileSync, writeFileSync, chmodSync, truncateSync} from 'node:fs'; -if (process.argv.includes('--version')) { console.log('sigrok-cli 0.7.2\\nlibsigrokdecode 0.5.3'); console.error('version warning'); } +if (process.argv.includes('--version')) { console.log(${JSON.stringify(mode)} === 'bad-version' ? 'sigrok-cli 0.7.1' : 'sigrok-cli 0.7.2\\nlibsigrokdecode 0.5.3'); console.error('version warning'); } +else if (${JSON.stringify(mode)} === 'bad-version') console.log('unexpected version fallback'); else if (${JSON.stringify(mode)} === 'nonzero') { console.error('invalid decoder'); process.exitCode = 1; } else if (${JSON.stringify(mode)} === 'malformed-copy') { chmodSync('recording', 0o600); writeFileSync('recording', 'changed'); } else if (${JSON.stringify(mode)} === 'oversized-copy') { chmodSync('recording', 0o600); truncateSync('recording', 32 * 1024 * 1024 + 1); } +else if (${JSON.stringify(mode)} === 'line-budget') process.stdout.write('ID: uart' + '\\n'.repeat(65536)); else if (${JSON.stringify(mode)} === 'overflow') process.stdout.write('x'.repeat(5 * 1024 * 1024)); else if (${JSON.stringify(mode)} === 'wait') { console.log('ready'); setInterval(() => {}, 1000); } else if (${JSON.stringify(mode)} === 'many') process.stdout.write(Array.from({length:90},(_,i)=>'line '+(i+1)+'\\r\\n').join('')); +else if (process.argv.includes('--list-supported')) { console.log('Supported hardware drivers:\\n ignored Device\\nSupported protocol decoders:\\n uart Universal Asynchronous Receiver/Transmitter\\n i2c Inter-Integrated Circuit\\n'); console.error('catalog warning'); } +else if (process.argv.includes('--show')) console.log('ID: uart\\nOptions:\\n- baudrate: Baud rate (default 115200)\\n- parity: Parity'); else { if(readFileSync('recording', 'utf8') !== 'recording bytes') throw Error('wrong copy'); console.log('10-20 uart-1: 42'); console.error('producer warning'); } `, ); @@ -285,6 +290,9 @@ it("retains a completed observation when cancellation arrives at the service bou const execution = await f.service.provider.decode(input); if (!execution.ok) throw execution.error; const service = new SignalRecordingService({ + inspectDecoders: async () => { + throw new Error("Unexpected discovery call"); + }, decode: async () => { f.controller.abort(); return execution; @@ -419,3 +427,182 @@ it("preserves termination signal and effective file ceiling as a resource failur capturedOutput: { signal: "SIGXFSZ", stderr: "before file limit\n" }, }); }); + +it("discovers decoders without reading a recording and retains installation provenance", async () => { + const f = await fixture(); + for (const [input, expected] of [ + [ + { query: "UART" }, + { + decoders: [ + { id: "uart", name: "Universal Asynchronous Receiver/Transmitter" }, + ], + output_window: { + examined: 2, + matching: 1, + returned: 1, + has_more: false, + }, + }, + ], + [ + { limit: 1 }, + { + decoders: [ + { id: "uart", name: "Universal Asynchronous Receiver/Transmitter" }, + ], + output_window: { matching: 2, returned: 1, has_more: true }, + }, + ], + [ + { offset: 1, limit: null }, + { + decoders: [{ id: "i2c", name: "Inter-Integrated Circuit" }], + output_window: { returned: 1, has_more: false }, + }, + ], + [ + { decoder: "uart", query: "baud" }, + { + help_lines: [ + { line_number: 3, text: "- baudrate: Baud rate (default 115200)" }, + ], + output_window: { unit: "help-lines", examined: 4, matching: 1 }, + }, + ], + [ + { query: "no-such-decoder" }, + { + decoders: [], + output_window: { examined: 2, matching: 0, returned: 0 }, + }, + ], + ] as const) { + const result = await f.service.inspectDecoders(input); + if (!result.ok) throw result.error; + const report = signalDecodersSchema.parse(result.value.normalized_result); + expect(report).toMatchObject(expected); + expect(result.value.operation).toBe("inspect_signal_decoders"); + expect(report.tool.version_diagnostics).toBe("version warning\n"); + expect(report.tool.arguments).toContain("--dont-scan"); + expect( + report.tool.arguments.some((arg) => arg.startsWith("--input-file")), + ).toBe(false); + } + expect( + (await readdir(f.directory)).filter((name) => name.startsWith("decoder-")), + ).toEqual([]); +}); + +it("attributes discovery admission failures to discovery while sharing the decoding slot", async () => { + const f = await fixture("gated"); + const active = f.service.decode({ path: f.path, decoders: "uart" }); + await f.acquired; + const busy = await f.service.inspectDecoders({}); + f.release(); + if (!(await active).ok) throw new Error("Decode failed"); + expect(busy).toMatchObject({ + ok: false, + error: { operation: "inspect_signal_decoders", resource: "capacity" }, + }); + await f.service.close(); + expect(await f.service.inspectDecoders({})).toMatchObject({ + ok: false, + error: { + operation: "inspect_signal_decoders", + _tag: "AnalysisCancelledError", + }, + }); +}); + +it.each([ + "bad-version", + "nonzero", + "overflow", + "line-budget", + "wait", + "oversized-tool", +])("preserves discovery failure diagnostics: %s", async (mode) => { + const f = await fixture(mode); + const result = await f.service.inspectDecoders( + {}, + { signal: f.controller.signal }, + ); + expect(result.ok).toBe(false); + if (result.ok) throw new Error("Expected failure"); + expect(result.error).toMatchObject({ + operation: "inspect_signal_decoders", + }); + if (mode === "line-budget") { + expect(result.error).toMatchObject({ + _tag: "AnalysisResourceConstraintError", + }); + expect(result.error.capturedOutput?.stdout).toBe( + "ID: uart" + "\n".repeat(65536), + ); + } + if (mode === "bad-version") + expect(result.error.capturedOutput).toMatchObject({ + stdout: "sigrok-cli 0.7.1\n", + stderr: "version warning\n", + }); +}); + +it("retains completed discovery when cleanup fails, then retries before accepting work", async () => { + const f = await fixture("ok", true); + const first = await f.service.inspectDecoders({ query: "uart" }); + expect(first).toMatchObject({ + ok: false, + error: { + operation: "inspect_signal_decoders", + partialObservation: { operation: "inspect_signal_decoders" }, + }, + }); + await f.service.inspectDecoders({}); + await f.service.decode({ path: f.path, decoders: "uart" }); + expect(f.calls()).toBe(2); + for (let attempt = 0; attempt < 2; attempt++) + await expect(f.service.close()).rejects.toMatchObject({ + operation: "inspect_signal_decoders", + cleanupIncomplete: true, + }); + f.allowCleanup(); + await f.service.close(); +}); + +it("rejects ambiguous decoder selectors before launch and retains malformed help output", async () => { + const f = await fixture(); + for (const decoder of ["uart:baudrate=1", "uart,spi", "uart\0"]) { + expect(await f.service.inspectDecoders({ decoder })).toMatchObject({ + ok: false, + error: { _tag: "AnalysisInputError" }, + }); + } + expect(f.calls()).toBe(0); + expect(await f.service.inspectDecoders({ decoder: "spi" })).toMatchObject({ + ok: false, + error: { + _tag: "AnalysisOutputError", + capturedOutput: { stdout: expect.stringContaining("ID: uart") }, + }, + }); +}); + +it("retains completed discovery on cancellation during identity verification", async () => { + const f = await fixture("post-run-abort"); + expect( + await f.service.inspectDecoders( + { query: "uart" }, + { signal: f.controller.signal }, + ), + ).toMatchObject({ + ok: false, + error: { + _tag: "AnalysisCancelledError", + partialObservation: { + operation: "inspect_signal_decoders", + normalized_result: { decoders: [{ id: "uart" }] }, + }, + }, + }); +}); diff --git a/tests/boundary/sqlite/sqliteDatabaseProvider.test.ts b/tests/boundary/sqlite/sqliteDatabaseProvider.test.ts index 24dadc128..b5adf8297 100644 --- a/tests/boundary/sqlite/sqliteDatabaseProvider.test.ts +++ b/tests/boundary/sqlite/sqliteDatabaseProvider.test.ts @@ -5,6 +5,7 @@ import { DatabaseSync } from "node:sqlite"; import { promisify } from "node:util"; import { fileURLToPath } from "node:url"; import { expect, it as nativeTest, onTestFinished } from "vitest"; +import { projectAnalysisError } from "../../../src/domain/analysisErrorProjection.js"; import { SqliteDatabaseProvider } from "../../../src/sqlite/SqliteDatabaseProvider.js"; import { spawnOwnedProviderProcess } from "../../../src/process/ProviderProcess.js"; import { PrivateRuntimeRoot } from "../../../src/process/PrivateRuntimeRoot.js"; @@ -356,6 +357,49 @@ it("does not execute views when selecting rows", async () => { }); }); +it("attributes table and column selection failures to their own arguments", async () => { + const { database } = await createFixture(); + const provider = createProvider(); + onTestFinished(() => provider.close()); + const projection = async (selection: { + table: string; + columns?: string[]; + }) => { + const result = await provider.inspect({ + path: database.path, + row_limit: 1, + ...selection, + }); + if (result.ok) throw new Error("Expected a rejected selection"); + return projectAnalysisError(result.error); + }; + expect( + await projection({ table: "records", columns: ["nope"] }), + ).toMatchObject({ + details: { + issues: [ + { + path: ["columns"], + reason: "invalid_format", + message: + "Selected column does not exist with that exact name in table records: nope", + }, + ], + }, + }); + expect(await projection({ table: "absent" })).toMatchObject({ + details: { issues: [{ path: ["table"] }] }, + }); + const selected = await provider.inspect({ + path: database.path, + table: "records", + columns: ["id"], + row_limit: 1, + }); + if (!selected.ok) throw selected.error; + expect(selected.value.result).toMatchObject({ rows: { columns: ["id"] } }); +}); + it("rejects malformed databases without altering the selected file", async () => { const workspace = await createTestWorkspace("rea-sqlite-invalid-"); onTestFinished(() => removeTestWorkspace(workspace.root)); diff --git a/tests/ciTestSequencer.ts b/tests/ciTestSequencer.ts index 5e8c4fe5f..996e8ce7a 100644 --- a/tests/ciTestSequencer.ts +++ b/tests/ciTestSequencer.ts @@ -5,14 +5,13 @@ import { BaseSequencer, type TestSpecification } from "vitest/node"; import { z } from "zod"; const reportSchema = z.object({ + version: z.literal(1), success: z.literal(true), - numFailedTests: z.literal(0), - testResults: z.array(z.unknown()), + files: z.array(z.unknown()), }); const fileSchema = z.object({ name: z.string(), - startTime: z.number(), - endTime: z.number(), + durationMs: z.number().finite().positive(), }); export function parseTestDurations( @@ -30,15 +29,12 @@ export function parseTestDurations( if (!report.success) return undefined; const durations = new Map(); - for (const value of report.data.testResults) { + for (const value of report.data.files) { const result = fileSchema.safeParse(value); if (!result.success) continue; - const { name, startTime, endTime } = result.data; - const duration = endTime - startTime; + const { name, durationMs } = result.data; const path = relative(root, resolve(root, name)); if ( - !Number.isFinite(duration) || - duration <= 0 || path === "" || path === ".." || path.startsWith(".." + sep) || @@ -46,7 +42,7 @@ export function parseTestDurations( ) continue; const key = path.split(sep).join("/"); - durations.set(key, Math.max(durations.get(key) ?? 0, duration)); + durations.set(key, Math.max(durations.get(key) ?? 0, durationMs)); } return durations.size ? durations : undefined; } diff --git a/tests/composition/analysis-sessions/binarySession.test.ts b/tests/composition/analysis-sessions/binarySession.test.ts index 813b90004..092f869e2 100644 --- a/tests/composition/analysis-sessions/binarySession.test.ts +++ b/tests/composition/analysis-sessions/binarySession.test.ts @@ -1278,3 +1278,102 @@ describe("selector origin forwarding", () => { ).toEqual(["provider", "provider", undefined]); }); }); + +describe("reference truncation forwarding", () => { + const referenceProvider = ( + referencesTruncated: boolean | undefined, + calls: string[], + ): AnalysisProvider => { + const identity = { + id: "fixture", + name: "Fixture analysis provider", + version: "1", + } as const; + return { + identity: () => identity, + resolveAnalysisProfile: () => + Promise.resolve( + resultOk({ + profile: createAnalysisProfile(identity, { fixture: true }), + }), + ), + capabilities: () => [ + { + provider: identity, + operation: "xrefs", + available: true, + reason: null, + effects: { + mutatesArtifact: false, + launchesProcess: false, + mayShowUi: false, + mayAccessNetwork: false, + mayWriteFilesystem: false, + changesPermissions: false, + requiresRoot: false, + }, + limitations: [], + }, + ], + createClient: () => ({ + execute: (operation) => { + calls.push(operation); + return Promise.resolve( + resultOk( + createAnalysisExecution( + ["0x2000"], + identity, + referencesTruncated === undefined + ? {} + : { referencesTruncated }, + ), + ), + ); + }, + close: () => Promise.resolve(resultOk(null)), + }), + }; + }; + const parameters = { address: "0x1000", document: "fixture" }; + + it("forwards a truncation on both execution paths and never replays it from the snapshot", async () => { + const [first] = await createBinarySessionTargets(); + const calls: string[] = []; + const session = createTestBinarySession(referenceProvider(true, calls)); + expect((await session.open(first)).ok).toBe(true); + calls.length = 0; + + const direct = await session.execute("xrefs", parameters); + const admitted = await session.withAdmittedAnalysis( + "trace_feature", + undefined, + (analysis) => analysis.execute("xrefs", parameters), + ); + + expect(direct.ok && direct.value.referencesTruncated).toBe(true); + if (!admitted.ok) throw admitted.error; + expect(admitted.value.ok && admitted.value.value.referencesTruncated).toBe( + true, + ); + expect( + calls, + "a truncated read is not recorded, so every call reaches the provider", + ).toEqual(["xrefs", "xrefs"]); + }); + + it("keeps an explicit complete report out of the way of snapshot replay", async () => { + const [first] = await createBinarySessionTargets(); + const calls: string[] = []; + const session = createTestBinarySession(referenceProvider(false, calls)); + expect((await session.open(first)).ok).toBe(true); + calls.length = 0; + + const live = await session.execute("xrefs", parameters); + await session.execute("xrefs", parameters); + + expect(live.ok && live.value.referencesTruncated).toBe(false); + expect(calls, "the complete read is recorded and replayed").toEqual([ + "xrefs", + ]); + }); +}); diff --git a/tests/conformance/c/fixture.c b/tests/conformance/c/fixture.c index 6708ac9dd..e1dff9a9c 100644 --- a/tests/conformance/c/fixture.c +++ b/tests/conformance/c/fixture.c @@ -5,13 +5,24 @@ volatile int rea_c_global = 7; #define REA_LONG_CHUNK "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" #define REA_LONG_256 REA_LONG_CHUNK REA_LONG_CHUNK REA_LONG_CHUNK REA_LONG_CHUNK #define REA_LONG_1024 REA_LONG_256 REA_LONG_256 REA_LONG_256 REA_LONG_256 +/* Keep the full-size literal even when a loader does not type it as a string. */ +__attribute__((used)) const char rea_long_literal[] = + "REA_LONG_LITERAL_" REA_LONG_1024 REA_LONG_1024 REA_LONG_1024 REA_LONG_1024 "needle"; +/* Linux Hopper 6.4.2 recognizes this extent and splits it into native objects. */ +__attribute__((used)) const char rea_fragmented_literal[] = + "REA_FRAGMENTED_LITERAL_" REA_LONG_256 REA_LONG_256 "fragment_tail"; +__attribute__((used)) const char rea_utf8_literal[] = "REA_UTF8_é_😀"; +__attribute__((used)) const char rea_ellipsis_literal[] = "REA_LITERAL_…"; +__attribute__((used)) const char rea_latin1_literal[] = "REA_LATIN1_\xff"; + __attribute__((noinline, used)) void rea_long_literal_procedure(void) { - puts("REA_LONG_LITERAL_" REA_LONG_1024 REA_LONG_1024 REA_LONG_1024 REA_LONG_1024 "needle"); - puts("REA_UTF8_é_😀"); + puts(rea_long_literal); + puts(rea_fragmented_literal); + puts(rea_utf8_literal); puts("REA_ESCAPED_\"\\line\nend\t\r"); puts("REA_LITERAL_BACKSLASH_\\n"); - puts("REA_LITERAL_…"); - puts("REA_LATIN1_\xff"); + puts(rea_ellipsis_literal); + puts(rea_latin1_literal); } __attribute__((noinline, used)) int rea_leaf(int value) { diff --git a/tests/conformance/c/pie-address-context.c b/tests/conformance/c/pie-address-context.c new file mode 100644 index 000000000..970d2fc6e --- /dev/null +++ b/tests/conformance/c/pie-address-context.c @@ -0,0 +1,7 @@ +__attribute__((noinline)) int rea_pie_add(int value) { + return value + 7; +} + +int rea_pie_entry(void) { + return rea_pie_add(35); +} diff --git a/tests/conformance/ghidra/6502.cfg b/tests/conformance/ghidra/6502.cfg new file mode 100644 index 000000000..cb85a4e76 --- /dev/null +++ b/tests/conformance/ghidra/6502.cfg @@ -0,0 +1,9 @@ +FEATURES { + STARTADDRESS: default = $8001; +} +MEMORY { + IMAGE: start = %S, size = $10000 - %S, file = %O; +} +SEGMENTS { + CODE: load = IMAGE, type = ro; +} diff --git a/tests/conformance/ghidra/6502.s b/tests/conformance/ghidra/6502.s new file mode 100644 index 000000000..201f8bcd2 --- /dev/null +++ b/tests/conformance/ghidra/6502.s @@ -0,0 +1,19 @@ +; Source-owned NMOS 6502 fixture: direct call, conditional branch, and arithmetic. +.setcpu "6502" +.segment "CODE" +.export entry, add_three + +entry: + lda #7 + jsr add_three + cmp #10 + bne mismatch + lda #42 + rts +mismatch: + lda #0 + rts +add_three: + clc + adc #3 + rts diff --git a/tests/conformance/ghidra/ReaFileOffsetSourceProbe.java b/tests/conformance/ghidra/ReaFileOffsetSourceProbe.java new file mode 100644 index 000000000..57da9d28e --- /dev/null +++ b/tests/conformance/ghidra/ReaFileOffsetSourceProbe.java @@ -0,0 +1,118 @@ +//@category REA Verification + +import java.io.ByteArrayInputStream; +import java.io.PrintWriter; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.util.Locale; + +import com.google.gson.JsonNull; +import com.google.gson.JsonObject; +import ghidra.app.script.GhidraScriptUtil; +import ghidra.app.util.headless.HeadlessScript; +import ghidra.program.model.address.AddressSet; +import ghidra.program.model.listing.Function; +import ghidra.program.model.symbol.SourceType; +import ghidra.app.decompiler.DecompInterface; +import ghidra.app.decompiler.DecompileResults; +import ghidra.app.decompiler.DecompileProcess.DisposeState; +import ghidra.util.task.TaskMonitor; + +/** Real source-identity cases unavailable through ordinary single-file CLI imports. */ +public final class ReaFileOffsetSourceProbe extends HeadlessScript { + @Override + public void run() throws Exception { + var file = GhidraScriptUtil.findScriptByName("ReaGhidraBridge.java"); + var bridge = GhidraScriptUtil.getProvider(file).getScriptInstance(file, new PrintWriter(System.out, true)); + bridge.set(state, monitor, new PrintWriter(System.out, true)); + var space = bridge.getClass().getDeclaredField("sessionDefaultAddressSpace"); + space.setAccessible(true); + space.set(null, currentProgram.getAddressFactory().getDefaultAddressSpace()); + var identity = bridge.getClass().getDeclaredField("admittedTargetSha256"); + identity.setAccessible(true); + identity.set(bridge, currentProgram.getExecutableSHA256().toLowerCase(Locale.ROOT)); + var resolve = bridge.getClass().getDeclaredMethod("resolveDecompileProcedure", String.class, JsonObject.class); + resolve.setAccessible(true); + var reverse = bridge.getClass().getDeclaredMethod("addressToFileOffset", JsonObject.class); + reverse.setAccessible(true); + + var memory = currentProgram.getMemory(); + var original = memory.getAllFileBytes().get(0); + byte[] bytes = new byte[Math.toIntExact(original.getSize())]; + require(original.getOriginalBytes(0, bytes) == bytes.length, "complete fixture source"); + byte[] auxiliary = new byte[0x1000]; + auxiliary[0x910] = (byte)0xc3; + var foreign = memory.createFileBytes("auxiliary.bin", 0, auxiliary.length, + new ByteArrayInputStream(auxiliary), monitor); + var address = toAddr(0x140100000L); + memory.createInitializedBlock("auxiliary", address, foreign, 0, auxiliary.length, false); + var entry = address.add(0x910); + currentProgram.getFunctionManager().createFunction("foreign_only", entry, + new AddressSet(entry), SourceType.USER_DEFINED); + rejected(resolve, bridge, "no loaded source mapping", "file-offset:0x910", new JsonObject()); + rejected(reverse, bridge, "0 authoritative target-file mappings", reverseInput("0x140100910")); + + // A source object's declared original-file start does not make its local + // offsets a complete, digest-matched target representation. + var partial = memory.createFileBytes("partial.bin", 0x200, 0x200, + new ByteArrayInputStream(bytes, 0x200, 0x200), monitor); + memory.createInitializedBlock("partial", toAddr(0x140300000L), partial, 0, 0x200, false); + rejected(reverse, bridge, "0 authoritative target-file mappings", reverseInput("0x140300010")); + var selected = (Function)resolve.invoke(bridge, "file-offset:0x210", new JsonObject()); + require(selected.getEntryPoint().equals(toAddr(0x140001010L)), "original target selected despite other sources"); + var mapped = (JsonObject)reverse.invoke(bridge, reverseInput("0x140001010")); + require(mapped.get("file_offset").getAsLong() == 0x210, "original target round trip"); + + // Inject backend failures after real source/function resolution. This + // verifies context propagation, not a naturally failing decompiler. + var pseudocode = bridge.getClass().getDeclaredMethod("procedurePseudocode", JsonObject.class); + pseudocode.setAccessible(true); + var decompiler = bridge.getClass().getDeclaredField("decompiler"); + decompiler.setAccessible(true); + JsonObject request = new JsonObject(); + request.add("document", JsonNull.INSTANCE); + request.addProperty("procedure", "file-offset:0x210"); + for (DisposeState disposition : new DisposeState[] { DisposeState.NOT_DISPOSED, DisposeState.DISPOSED_ON_CANCEL }) { + decompiler.set(bridge, new DecompInterface() { + @Override + public DecompileResults decompileFunction(Function function, int timeout, TaskMonitor taskMonitor) { + return new DecompileResults(function, currentProgram.getLanguage(), + currentProgram.getCompilerSpec(), null, "Injected backend failure", null, disposition); + } + }); + Throwable failure = rejected(pseudocode, bridge, + "Selector file-offset:0x210 mapped to analysis address 0x140001010 and procedure 0x140001010", request); + var code = failure.getClass().getDeclaredField("code"); + code.setAccessible(true); + require(code.get(failure).equals(disposition == DisposeState.DISPOSED_ON_CANCEL + ? "decompile_cancelled" : "decompile_failed"), "backend failure code retained"); + } + + var duplicate = memory.createFileBytes("duplicate.bin", 0, bytes.length, + new ByteArrayInputStream(bytes), monitor); + memory.createInitializedBlock("duplicate", toAddr(0x140201000L), duplicate, 0x200, 0x200, false); + rejected(resolve, bridge, "multiple loaded mappings", "file-offset:0x210", new JsonObject()); + println("REA_FILE_OFFSET_SOURCE_OK foreign/partial rejected; original round trip; duplicate ambiguity; injected backend failure context retained"); + } + + private JsonObject reverseInput(String address) { + JsonObject params = new JsonObject(); + params.add("document", JsonNull.INSTANCE); + params.addProperty("address", address); + return params; + } + + private Throwable rejected(Method method, Object bridge, String reason, Object... arguments) throws Exception { + try { + method.invoke(bridge, arguments); + } catch (InvocationTargetException rejected) { + require(rejected.getCause().getMessage().contains(reason), "expected mapping failure: " + reason); + return rejected.getCause(); + } + throw new AssertionError("Expected mapping failure: " + reason); + } + + private void require(boolean condition, String message) { + if (!condition) throw new AssertionError(message); + } +} diff --git a/tests/conformance/ida/providerContracts.test.ts b/tests/conformance/ida/providerContracts.test.ts index 39c524637..4e4b04c3b 100644 --- a/tests/conformance/ida/providerContracts.test.ts +++ b/tests/conformance/ida/providerContracts.test.ts @@ -29,15 +29,25 @@ const parametersFor = ( operation: string, ): Readonly> => { if (operation.startsWith("search_")) return { pattern: "a.b" }; - if (operation === "xrefs") return { address: "0x1000" }; + if (operation === "xrefs" || operation === "resolve_containing_procedure") + return { address: "0x1000" }; if (operation.startsWith("list_")) return {}; return { procedure: "main" }; }; +/** Operations only the headless profile declares. */ +const HEADLESS_ONLY: ReadonlySet = new Set([ + "resolve_containing_procedure", + "list_documents", +]); + describe.each(["attached", "headless"] as const)( "IDA %s caller-visible contracts", (mode) => { - it.each(IDA_OPERATIONS)( + const operations = IDA_OPERATIONS.filter( + (operation) => mode === "headless" || !HEADLESS_ONLY.has(operation), + ); + it.each(operations)( "declares %s UI effects matching the upstream tools it reaches", async (operation) => { const { root, target } = await createIdaTarget(); @@ -69,7 +79,7 @@ describe.each(["attached", "headless"] as const)( }, ); - it.each(IDA_OPERATIONS)( + it.each(operations)( "serializes %s against its named advertised output contract", async (operation) => { const { root, target } = await createIdaTarget(); diff --git a/tests/conformance/providers/ghidra/realGhidraWindowsWorkflow.test.ts b/tests/conformance/providers/ghidra/realGhidraWindowsWorkflow.test.ts index 39eb8ff61..0e936b062 100644 --- a/tests/conformance/providers/ghidra/realGhidraWindowsWorkflow.test.ts +++ b/tests/conformance/providers/ghidra/realGhidraWindowsWorkflow.test.ts @@ -26,14 +26,18 @@ describe("real Windows Ghidra workflow trust boundary", () => { environment: "real-ghidra-windows", "runs-on": expect.arrayContaining(["self-hosted", "Windows", "x64"]), steps: expect.arrayContaining([ - expect.objectContaining({ uses: "actions/setup-node@v4" }), + expect.objectContaining({ + uses: expect.stringMatching(/^actions\/setup-node@[0-9a-f]{40}$/), + }), expect.objectContaining({ run: "npm ci" }), expect.objectContaining({ env: { REA_ANALYSIS_PROVIDER: "ghidra" }, run: "npm run verify:ghidra:windows | Tee-Object -FilePath windows-ghidra-proof.log", }), expect.objectContaining({ - uses: "actions/upload-artifact@v4", + uses: expect.stringMatching( + /^actions\/upload-artifact@[0-9a-f]{40}$/, + ), with: expect.objectContaining({ path: expect.stringContaining("windows-ghidra-proof.log"), }), diff --git a/tests/conformance/providers/hopper/realHopperSearch.test.ts b/tests/conformance/providers/hopper/realHopperSearch.test.ts new file mode 100644 index 000000000..10aedb433 --- /dev/null +++ b/tests/conformance/providers/hopper/realHopperSearch.test.ts @@ -0,0 +1,97 @@ +import { expect, it } from "vitest"; + +import { verifyHopperStringObjects } from "../../../../scripts/lib/real-hopper-search.mjs"; + +const value = "REA_UTF8_é_😀"; +const address = "0x402020"; +const bytes = Buffer.from(`${value}\0`); +const item = { + address, + value, + string: { + encoding: "utf-8", + encoding_status: "inferred" as const, + termination: "present_or_not_required" as const, + byte_length: bytes.length, + }, +}; +const expectation = { + value, + encoding: "utf-8", + symbol: "rea_utf8_literal", + recognitionOptional: true, +}; + +it("keeps source identity and unknown decoding when the same text exists elsewhere", async () => { + const report = await verifyHopperStringObjects( + fixtureCall(), + [expectation], + [{ ...item, address: "0x403000" }], + ); + expect(report.recognition).toEqual([ + { value, address, state: "not_observed", decodedObject: "unverified" }, + ]); +}); + +it("reuses source bytes and inventory while checking real address filters and searches", async () => { + const calls: string[] = []; + const report = await verifyHopperStringObjects( + fixtureCall(calls), + [expectation], + [item], + ); + expect(report.recognition).toEqual([ + { value, address, state: "observed", decodedObject: "verified" }, + ]); + expect(calls).toEqual([ + "read_bytes:rea_utf8_literal", + `list_strings:${address}`, + `search_strings:${value}`, + "search_strings:\\xC3\\xA9", + ]); +}); + +it.each([ + ["changed source bytes", { bytes_hex: "00" }, item], + ["incomplete source read", { complete: false }, item], + ["wrong decoded value", {}, { ...item, value: "wrong" }], + [ + "wrong decoded extent", + {}, + { ...item, string: { ...item.string, byte_length: 1 } }, + ], +])( + "rejects %s even when native recognition is optional", + async (_label, override, decoded) => { + await expect( + verifyHopperStringObjects( + fixtureCall([], override), + [expectation], + [decoded], + ), + ).rejects.toThrow(); + }, +); + +function fixtureCall( + calls: string[] = [], + override: Record = {}, +) { + return async (name: string, arguments_: Record = {}) => { + calls.push( + `${name}:${String(arguments_.address ?? arguments_.pattern ?? "")}`, + ); + if (name === "read_bytes") + return { + address, + bytes_hex: bytes.toString("hex"), + complete: true, + ...override, + }; + if (name === "list_strings" && arguments_.address === address) + return [item]; + if (name === "search_strings") + return arguments_.pattern === value ? [item] : []; + throw new Error(`Unmodeled verifier call: ${name}`); + }; +} diff --git a/tests/conformance/readmeConsistency.test.ts b/tests/conformance/readmeConsistency.test.ts index 79a1fc518..90f8e5886 100644 --- a/tests/conformance/readmeConsistency.test.ts +++ b/tests/conformance/readmeConsistency.test.ts @@ -35,6 +35,20 @@ const shellExamples = (content: string): string[] => ); describe("onboarding documentation product facts", () => { + it.each([ + ["docs/native-investigation.md", "raw-binary.md#nmos-6502"], + ["docs/ghidra-riscv.md", "raw-binary.md"], + ])( + "routes raw admission guidance in %s to the owning guide", + async (path, target) => { + const content = await readFile(resolve(path), "utf8"); + const links = [...content.matchAll(/\]\(([^)\s]+)\)/gu)].map( + (match) => match[1], + ); + expect(links, path).toContain(target); + }, + ); + it("keeps the installation MCP example aligned with the package registration command", async () => { const content = await readFile(resolve("docs/installation.md"), "utf8"); expect(jsonExamples(content)).toContainEqual( diff --git a/tests/conformance/swift/string-literals.swift b/tests/conformance/swift/string-literals.swift new file mode 100644 index 000000000..345c4cf74 --- /dev/null +++ b/tests/conformance/swift/string-literals.swift @@ -0,0 +1,23 @@ +@inline(never) +func consume(_ value: String) { print(value) } +@inline(never) +func smallLiteral() { consume("SleepDisabled") } +@inline(never) +func largeLiteral() { consume("The power setting could not be changed.") } +@inline(never) +func unicodeLiterals() { + consume("café") + consume("Résumé of the naïve décor, written twice") +} +@inline(never) +func emptyAndControlLiterals() { + consume("") + consume("tab\there") +} +@inline(never) +func returnedLiteral() -> String { "Holding awake" } +smallLiteral() +largeLiteral() +unicodeLiterals() +emptyAndControlLiterals() +print(returnedLiteral(), CommandLine.arguments.count) diff --git a/tests/evaluation/ciTestSelection.test.ts b/tests/evaluation/ciTestSelection.test.ts new file mode 100644 index 000000000..1dff7f9e3 --- /dev/null +++ b/tests/evaluation/ciTestSelection.test.ts @@ -0,0 +1,240 @@ +import { expect, it } from "vitest"; + +import { + compareTestSelection, + planTestSelection, +} from "../../scripts/ci/test-selection.mjs"; + +const sqliteSource = "src/sqlite/SqliteDatabaseProvider.test.ts"; +const sqliteBoundary = "tests/boundary/sqlite/sqliteDatabaseProvider.test.ts"; +const adbSource = "src/adb/AdbProvider.test.ts"; +const adbBoundary = + "tests/boundary/providers/adb/adbBugreportPublication.test.ts"; +const cliBoundary = "tests/boundary/cli/apktoolCli.test.ts"; +const mcpBoundary = "tests/boundary/mcp/apktoolMcp.test.ts"; +const processBoundary = + "tests/boundary/process/processCaptureLifecycle.test.ts"; +const acceptance = "tests/acceptance/tools/localToolContracts.test.ts"; +const composition = "tests/composition/directAnalysisCleanup.test.ts"; +const evaluation = "tests/evaluation/testDiscovery.test.ts"; + +const passed = (name: string) => ({ name, durationMs: 10, state: "passed" }); + +it("unions affected leaf owners, keeps changed tests and retains unowned integration tests", () => { + const plan = planTestSelection( + ["src/sqlite/SqliteDatabaseProvider.ts", adbBoundary], + "pull_request", + "fast", + ); + const inventory = [ + sqliteSource, + sqliteBoundary, + adbSource, + adbBoundary, + cliBoundary, + mcpBoundary, + processBoundary, + acceptance, + composition, + evaluation, + ]; + const report = compareTestSelection(plan, inventory, inventory.map(passed)); + + expect(plan).toMatchObject({ + mode: "candidate", + owners: ["adb", "sqlite"], + }); + expect(report.selected).toEqual( + expect.arrayContaining([ + sqliteSource, + sqliteBoundary, + adbSource, + adbBoundary, + cliBoundary, + mcpBoundary, + processBoundary, + acceptance, + composition, + evaluation, + ]), + ); + expect(report.omitted).toEqual([]); +}); + +it("omits only unrelated owned tests while retaining dynamic boundary suites", () => { + const plan = planTestSelection( + ["src/sqlite/SqliteDatabaseProvider.ts"], + "pull_request", + "fast", + ); + const inventory = [ + sqliteSource, + sqliteBoundary, + adbSource, + adbBoundary, + cliBoundary, + mcpBoundary, + processBoundary, + acceptance, + composition, + evaluation, + ]; + const report = compareTestSelection(plan, inventory, inventory.map(passed)); + + expect(report.selected).toEqual( + expect.arrayContaining([ + sqliteSource, + sqliteBoundary, + cliBoundary, + mcpBoundary, + processBoundary, + acceptance, + composition, + evaluation, + ]), + ); + expect(report.omitted).toEqual([adbSource, adbBoundary]); + expect(report.complete).toBe(true); +}); + +it.each([ + ["unowned shared source", "src/application/Workflow.ts"], + ["fixture input", "tests/conformance/c/fixture.c"], + ["configuration", "vitest.config.ts"], + ["documentation", "README.md"], + ["CLI implementation", "src/cli/apktoolCommands.ts"], + ["MCP implementation", "src/server/registerSqliteTools.ts"], + ["process ownership", "src/process/ProcessOwnership.ts"], + ["acceptance test", acceptance], + ["composition test", composition], +])("uses full fallback for %s changes", (_description, path) => { + expect(planTestSelection([path], "pull_request", "fast")).toMatchObject({ + mode: "full", + owners: [], + changedTests: [], + }); +}); + +it("uses full fallback outside ordinary fast pull requests or without diff paths", () => { + expect( + planTestSelection(["src/sqlite/SqliteDatabaseProvider.ts"], "push", "fast") + .mode, + ).toBe("full"); + expect( + planTestSelection( + ["src/sqlite/SqliteDatabaseProvider.ts"], + "pull_request", + "full", + ).mode, + ).toBe("full"); + expect(planTestSelection([], "pull_request", "fast").mode).toBe("full"); +}); + +it("keeps new changed tests and falls back when a changed test was removed", () => { + const addedTest = "src/sqlite/NewDatabaseRule.test.ts"; + const candidate = planTestSelection([addedTest], "pull_request", "fast"); + expect(candidate.changedTests).toEqual([addedTest]); + const inventory = [addedTest, adbSource]; + expect( + compareTestSelection(candidate, inventory, inventory.map(passed)).selected, + ).toContain(addedTest); + + const removedTest = "src/sqlite/RemovedDatabaseRule.test.ts"; + const report = compareTestSelection( + planTestSelection([removedTest], "pull_request", "fast"), + [sqliteSource, adbSource], + [passed(sqliteSource), passed(adbSource)], + ); + expect(report.plan.mode).toBe("full"); + expect(report.selected).toEqual([adbSource, sqliteSource]); + expect(report.omitted).toEqual([]); +}); + +it("marks incomplete inventories and nonterminal outcomes as incomplete", () => { + const plan = planTestSelection( + ["src/sqlite/SqliteDatabaseProvider.ts"], + "pull_request", + "fast", + ); + const inventory = [sqliteSource, adbSource]; + const completeResults = inventory.map(passed); + + expect( + compareTestSelection(plan, inventory, [passed(sqliteSource)]).complete, + ).toBe(false); + expect( + compareTestSelection(plan, inventory, [ + ...completeResults, + passed("tests/evaluation/unexpected.test.ts"), + ]), + ).toMatchObject({ + complete: false, + unexpected: ["tests/evaluation/unexpected.test.ts"], + }); + expect( + compareTestSelection(plan, inventory, [ + ...completeResults, + passed(adbSource), + ]), + ).toMatchObject({ complete: false, duplicateFiles: 1 }); + for (const state of ["pending", "skipped"]) + expect( + compareTestSelection(plan, inventory, [ + passed(sqliteSource), + { name: adbSource, durationMs: 0, state }, + ]), + ).toMatchObject({ complete: false, unobserved: [adbSource] }); +}); + +it("reports a failure from an omitted test, including collection or module failures", () => { + const plan = planTestSelection( + ["src/sqlite/SqliteDatabaseProvider.ts"], + "pull_request", + "fast", + ); + const report = compareTestSelection( + plan, + [sqliteSource, adbSource], + [passed(sqliteSource), { name: adbSource, durationMs: 1, state: "failed" }], + ); + + expect(report.omitted).toContain(adbSource); + expect(report.missedFailures).toEqual([adbSource]); +}); + +it("rejects plans that name unowned owners or malformed changed tests", () => { + const plan = planTestSelection( + ["src/sqlite/SqliteDatabaseProvider.ts"], + "pull_request", + "fast", + ); + expect(() => + compareTestSelection( + { ...plan, owners: ["unknown"] }, + [sqliteSource], + [passed(sqliteSource)], + ), + ).toThrow("Invalid observational test selection plan"); + expect(() => + compareTestSelection( + { ...plan, changedTests: ["src/sqlite/not-a-test.ts"] }, + [sqliteSource], + [passed(sqliteSource)], + ), + ).toThrow("Invalid observational test selection plan"); +}); + +it("uses full fallback when a changed owner has no executable tests", () => { + const report = compareTestSelection( + planTestSelection( + ["src/sqlite/SqliteDatabaseProvider.ts"], + "pull_request", + "fast", + ), + [adbSource], + [passed(adbSource)], + ); + + expect(report.plan.mode).toBe("full"); + expect(report.omitted).toEqual([]); +}); diff --git a/tests/evaluation/model-evals/codexAgentEval.test.ts b/tests/evaluation/model-evals/codexAgentEval.test.ts index d96265202..6b46d9b1d 100644 --- a/tests/evaluation/model-evals/codexAgentEval.test.ts +++ b/tests/evaluation/model-evals/codexAgentEval.test.ts @@ -134,6 +134,114 @@ describe("Codex agent release evaluation", () => { }); }); +describe("Codex completed tool call parsing", () => { + it("does not let an unfinished direct event reserve a completed call identity", () => { + const evidence = observation(); + const metrics = evaluateCodexEvents( + [ + { + type: "mcp_tool_call", + id: "same-call", + status: "in_progress", + server: "rea", + tool: evidence.operation, + arguments: {}, + }, + { + type: "item.completed", + item: { + id: "same-call", + type: "mcp_tool_call", + status: "completed", + server: "rea", + tool: evidence.operation, + arguments: {}, + result: { structuredContent: evidence }, + }, + }, + { + type: "item.completed", + item: { + type: "agent_message", + text: `Observed Evidence ${evidence.evidence_id} supports the artifact claim; runtime behavior remains unknown without runtime authority.`, + }, + }, + ], + evidence.operation, + { requireEvidence: true }, + ); + + expect(metrics).toMatchObject({ + naturalUse: true, + correctFirstTool: true, + reaCalls: [{ id: "same-call" }], + evidenceIds: [evidence.evidence_id], + finalCitesEvidence: true, + answerHeuristicsMet: true, + }); + }); + + it.each([ + { server: "rea", server_name: "other" }, + { tool: "inspect_managed_artifact", name: "other_tool" }, + { server: 42 }, + { arguments: { value: 1 }, input: { value: 2 } }, + ])( + "rejects completed calls with invalid or conflicting fields: %j", + (aliases) => { + const evidence = observation(); + const metrics = evaluateCodexEvents( + [ + { + type: "item.completed", + item: { + type: "mcp_tool_call", + status: "completed", + server: "rea", + tool: evidence.operation, + arguments: {}, + result: { structuredContent: evidence }, + ...aliases, + }, + }, + ], + evidence.operation, + { requiredToolSubsequence: [evidence.operation] }, + ); + + expect(metrics).toMatchObject({ + naturalUse: false, + correctFirstTool: false, + requiredToolSubsequenceMet: false, + evidenceIds: [], + }); + }, + ); + + it("preserves legacy completed calls with no server field", () => { + const metrics = evaluateCodexEvents( + [ + { + type: "item.completed", + item: { + type: "mcp_tool_call", + status: "completed", + tool: "inspect_managed_artifact", + arguments: {}, + }, + }, + ], + "inspect_managed_artifact", + ); + + expect(metrics).toMatchObject({ + naturalUse: true, + correctFirstTool: true, + reaCalls: [{ server: null }], + }); + }); +}); + describe("Codex Evidence delivery", () => { it("rejects diagnostic IDs and failed results as delivered Evidence", () => { const evidenceId = `ev_${"a".repeat(64)}`; @@ -524,28 +632,32 @@ describe("Codex agent completion evaluation", () => { }); }); - it("recognizes Codex MCP failed status when the error field is null", () => { - const metrics = evaluateCodexEvents( - [ - { - type: "item.completed", - item: { - id: "item-failed", - type: "mcp_tool_call", - server: "rea", - tool: "analyze_javascript_application", - arguments: { input_path: "/tmp/app.asar" }, - result: { structured_content: { error: { code: "denied" } } }, - error: null, - status: "failed", - }, - }, - ], - "analyze_javascript_application", - ); + it.each(["item.completed", "mcp_tool_call"] as const)( + "counts a failed completed invocation from %s without delivering Evidence", + (type) => { + const item = { + id: "item-failed", + type: "mcp_tool_call", + server: "rea", + tool: "analyze_javascript_application", + arguments: { input_path: "/tmp/app.asar" }, + result: { structured_content: { error: { code: "denied" } } }, + error: null, + status: "failed", + }; + const metrics = evaluateCodexEvents( + [type === "item.completed" ? { type, item } : item], + "analyze_javascript_application", + ); - expect(metrics.reaCalls[0]?.error).toBe(true); - }); + expect(metrics).toMatchObject({ + naturalUse: true, + correctFirstTool: true, + evidenceIds: [], + reaCalls: [{ error: true }], + }); + }, + ); }); describe("Codex agent workflow evaluation", () => { diff --git a/tests/evaluation/package/nativeAotBuild.test.ts b/tests/evaluation/package/nativeAotBuild.test.ts new file mode 100644 index 000000000..6e71db42a --- /dev/null +++ b/tests/evaluation/package/nativeAotBuild.test.ts @@ -0,0 +1,90 @@ +import { join } from "node:path"; + +import { expect, it, vi } from "vitest"; + +import { buildNativeAotFixture } from "../../../scripts/lib/nativeaot-build.mjs"; + +const options = { + dotnet: "dotnet", + project: "fixture.csproj", + runtime: "win-x64", + output: "fixture-output", +}; + +it("restores the exact publish target before compiling without network restore", async () => { + const run = vi.fn<(command: string, args: string[]) => void>(); + await buildNativeAotFixture(run, options); + expect(run).toHaveBeenCalledTimes(2); + const restore = run.mock.calls[0]?.[1] ?? []; + const publish = run.mock.calls[1]?.[1] ?? []; + expect(restore[0]).toBe("restore"); + expect(publish[0]).toBe("publish"); + expect(publish).toContain("--no-restore"); + for (const argument of [ + options.project, + options.runtime, + "-p:Configuration=Release", + `-p:BaseIntermediateOutputPath=${join(options.output, "obj")}/`, + ]) { + expect(restore).toContain(argument); + expect(publish).toContain(argument); + } +}); + +it("retries without logging credential-bearing restore diagnostics", async () => { + const logged = vi.spyOn(console, "error").mockImplementation(() => {}); + const timeout = Object.assign( + new Error( + "dotnet restore ETIMEDOUT https://fixture-user:fixture-password@feed.invalid/v3/index.json", + ), + { + code: "ETIMEDOUT", + }, + ); + const run = vi + .fn<(command: string, args: string[]) => void>() + .mockImplementationOnce(() => { + throw timeout; + }); + try { + await buildNativeAotFixture(run, options); + expect( + logged.mock.calls.flat().every((value) => typeof value === "string"), + ).toBe(true); + expect(logged.mock.calls.flat().join("\n")).not.toContain( + "fixture-password", + ); + } finally { + logged.mockRestore(); + } + expect(run.mock.calls.map((call) => call[1][0])).toEqual([ + "restore", + "restore", + "publish", + ]); +}); + +it("stops after two restore failures and preserves the final diagnostic", async () => { + const failure = new Error("NuGet source unavailable"); + const run = vi.fn<(command: string, args: string[]) => void>(() => { + throw failure; + }); + await expect(buildNativeAotFixture(run, options)).rejects.toMatchObject({ + message: "NativeAOT dependency restore failed after two attempts", + cause: failure, + }); + expect(run).toHaveBeenCalledTimes(2); + expect(run.mock.calls.every((call) => call[1][0] === "restore")).toBe(true); +}); + +it("does not retry compilation failures", async () => { + const failure = new Error("Native compiler rejected the fixture"); + const run = vi + .fn<(command: string, args: string[]) => void>() + .mockImplementationOnce(() => {}) + .mockImplementationOnce(() => { + throw failure; + }); + await expect(buildNativeAotFixture(run, options)).rejects.toBe(failure); + expect(run).toHaveBeenCalledTimes(2); +}); diff --git a/tests/evaluation/package/nativeAotDiagnostics.test.ts b/tests/evaluation/package/nativeAotDiagnostics.test.ts new file mode 100644 index 000000000..f5b42a5ce --- /dev/null +++ b/tests/evaluation/package/nativeAotDiagnostics.test.ts @@ -0,0 +1,150 @@ +import { execFile } from "node:child_process"; +import { readdir, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { promisify } from "node:util"; + +import { expect, it } from "vitest"; + +import { + nativeAotCommandError, + supportsNativeAotFixtureHost, +} from "../../../scripts/lib/nativeaot-build.mjs"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +const generator = fileURLToPath( + new URL("../../../scripts/generate-nativeaot-fixtures.mjs", import.meta.url), +); + +const supportedHost = supportsNativeAotFixtureHost( + process.platform, + process.arch, +); + +it.skipIf(!supportedHost).each(["restore", "publish"])( + "redacts authenticated feed URLs from the uncaught %s diagnostic", + async (operation) => { + const directory = await createTestTempDirectory( + "rea-nativeaot-diagnostic-", + ); + const hook = join(directory, "child-output.cjs"); + await writeFile( + hook, + ` +const child = require("node:child_process"); +child.spawnSync = (_command, args) => { + if (args[0] === "--version") return { status: 0, stdout: "8.0.416\\n", stderr: "" }; + if (args[0] === ${JSON.stringify(operation)}) return { + status: 1, + error: new Error("failed HTTPS://error-user:error-secret@feed.invalid/v3/index.json"), + stdout: "NU1301 https://stdout-user:stdout-secret@feed.invalid/v3/index.json\\n", + stderr: "source https://stderr-user:stderr-secret%40suffix@feed.invalid/v3/index.json\\nkeep https://feed.invalid/path/user@name?label=token-name\\n" + }; + return { status: 0, stdout: "", stderr: "" }; +}; +require("node:module").syncBuiltinESMExports(); +`, + "utf8", + ); + const failure = await promisify(execFile)( + process.execPath, + ["--require", hook, generator], + { + env: { + ...process.env, + REA_NATIVEAOT_FIXTURE_ROOT: join(directory, "output"), + }, + timeout: 15_000, + }, + ).then( + () => { + throw new Error("Expected fixture generation to fail"); + }, + (error: { stderr: string; code: number }) => error, + ); + expect(failure.code).toBe(1); + for (const secret of [ + "error-user", + "error-secret", + "stdout-user", + "stdout-secret", + "stderr-user", + "stderr-secret", + ]) + expect(failure.stderr).not.toContain(secret); + expect(failure.stderr).toContain("NU1301"); + expect(failure.stderr).toContain("feed.invalid/v3/index.json"); + expect(failure.stderr).toContain( + "https://feed.invalid/path/user@name?label=token-name", + ); + if (operation === "restore") { + expect(failure.stderr).toContain("attempt 2/2"); + expect(failure.stderr).toContain("failed after two attempts"); + expect(failure.stderr).toContain("[cause]"); + } + }, +); + +it.skipIf(supportedHost)( + "refuses unsupported fixture hosts before tooling or output creation", + async () => { + const directory = await createTestTempDirectory("rea-nativeaot-host-"); + const hook = join(directory, "unexpected-child.cjs"); + await writeFile( + hook, + `const child = require("node:child_process"); +child.spawnSync = () => { throw new Error("Unexpected fixture tooling invocation"); }; +require("node:module").syncBuiltinESMExports(); +`, + "utf8", + ); + const failure = await promisify(execFile)( + process.execPath, + ["--require", hook, generator], + { + env: { + ...process.env, + REA_NATIVEAOT_FIXTURE_ROOT: join(directory, "output"), + }, + timeout: 15_000, + }, + ).then( + () => { + throw new Error("Expected unsupported fixture host refusal"); + }, + (error: { stderr: string; code: number }) => error, + ); + expect(failure.code).toBe(1); + expect(failure.stderr).toContain("requires Linux or Windows x64"); + expect(failure.stderr).not.toContain( + "Unexpected fixture tooling invocation", + ); + expect(await readdir(directory)).toEqual(["unexpected-child.cjs"]); + }, +); + +it("preserves diagnostics without transport authentication exactly", () => { + const error = nativeAotCommandError( + "C:/token projects/dotnet.exe", + "restore", + { + status: 1, + stdout: "NU1301 https://feed.invalid/user@name?label=password\n", + stderr: "project C:/token projects/fixture.csproj unavailable", + }, + ); + expect(error.message).toBe( + "NativeAOT fixture lane requires/action failed C:/token projects/dotnet.exe restore: exit 1\nNU1301 https://feed.invalid/user@name?label=password\nproject C:/token projects/fixture.csproj unavailable", + ); + expect(error.cause).toBeUndefined(); +}); + +it("treats HTTP backslashes as path separators rather than authentication", () => { + const output = String.raw`https://feed.invalid\path\user@name`; + expect(new URL(output).username).toBe(""); + const error = nativeAotCommandError("dotnet", "restore", { + status: 1, + stdout: output, + }); + expect(error.message).toContain(output); +}); diff --git a/tests/fixtures/evm/interface.ts b/tests/fixtures/evm/interface.ts index 23bfcfa02..f0f1960ba 100644 --- a/tests/fixtures/evm/interface.ts +++ b/tests/fixtures/evm/interface.ts @@ -1,5 +1,6 @@ import { createHash } from "node:crypto"; import { createAnalysisExecution } from "../../../src/application/AnalysisProvider.js"; +import type { EvmInterfaceExecution } from "../../../src/application/evm/EvmInterfacePort.js"; import type { EvmInterface } from "../../../src/domain/evm/evmInterface.js"; /** Source-owned transport seam, independent of real engine verification. */ @@ -27,8 +28,10 @@ export const evmInterfaceFixture = ( }); /** Bind a fixture to its explicit producing seam and carrier identity. */ -export const evmInterfaceExecution = (value = evmInterfaceFixture()) => - createAnalysisExecution( +export const evmInterfaceExecution = ( + value = evmInterfaceFixture(), +): EvmInterfaceExecution => { + return createAnalysisExecution( value, { id: "fixture-evm", name: "Source-owned EVM seam", version: "1" }, { @@ -42,3 +45,4 @@ export const evmInterfaceExecution = (value = evmInterfaceFixture()) => locations: [{ kind: "artifact-path", path: value.artifact.path }], }, ); +}; diff --git a/tests/fixtures/hopperLifetimeLaunch.py b/tests/fixtures/hopperLifetimeLaunch.py new file mode 100644 index 000000000..68d65c9ff --- /dev/null +++ b/tests/fixtures/hopperLifetimeLaunch.py @@ -0,0 +1,16 @@ +#!/usr/bin/env python3 +"""Inspect the real Node launcher stdio with the production lifetime parser.""" + +import importlib.util +import json +from pathlib import Path +import sys + +spec = importlib.util.spec_from_file_location( + "hopper_demo_x11", Path(__file__).resolve().parents[2] / "scripts/hopper-demo-x11.py" +) +assert spec is not None and spec.loader is not None +helper = importlib.util.module_from_spec(spec) +spec.loader.exec_module(helper) +assert "--owner-stdin" in sys.argv +print(json.dumps({"owner_alive": helper.OwnerLifetime(True).alive()})) diff --git a/tests/fixtures/hopperOwnerLifetime.py b/tests/fixtures/hopperOwnerLifetime.py new file mode 100644 index 000000000..741bffbd9 --- /dev/null +++ b/tests/fixtures/hopperOwnerLifetime.py @@ -0,0 +1,110 @@ +#!/usr/bin/env python3 +"""Exercise the production runner with real children and a simulated X11 surface.""" + +import argparse +import importlib.util +import json +from pathlib import Path +import signal +import subprocess +import sys +import tempfile +import time + + +spec = importlib.util.spec_from_file_location("hopper_demo_x11", sys.argv[1]) +assert spec is not None and spec.loader is not None +helper = importlib.util.module_from_spec(spec) +spec.loader.exec_module(helper) + + +def child_command(receipt): + return [ + sys.executable, + "-c", + "import os,pathlib,time;" + f"receipt=pathlib.Path({str(receipt)!r});" + "pending=receipt.with_suffix('.pending');" + "pending.write_text(str(os.getpid()));pending.replace(receipt);" + "time.sleep(100)", + ] + + +if len(sys.argv) > 3: + phase, directory = sys.argv[2], Path(sys.argv[3]) + + def terminate_worker(*_): + raise SystemExit(143) + + # Failed assertions in the parent must still unwind run_launch's finally. + signal.signal(signal.SIGTERM, terminate_worker) + + class X11: + def XOpenDisplay(self, *_): + return 1 + + def XDefaultRootWindow(self, *_): + return 1 + + def XDisplayWidth(self, *_): + return 1280 + + def XDisplayHeight(self, *_): + return 1024 + + def XCloseDisplay(self, *_): + return 1 + + def start_xvfb(*_): + child = subprocess.Popen( + child_command(directory / "xvfb.pid"), + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + ) + return child, ":123", directory / "Xauthority", helper.CapturedStderr(child.stderr) + + helper.digest = lambda _: next(iter(helper.SUPPORTED_HOPPER_SHA256)) + helper.start_xvfb = start_xvfb + helper.ctypes.CDLL = lambda _: X11() + helper.configure_x11 = lambda *_: None + helper.windows = lambda *_: [] + socket = directory / "bridge.sock" + if phase == "ready": + socket.touch() + raise SystemExit(helper.run_launch(argparse.Namespace( + owner_stdin=True, + hopper=sys.executable, + socket=str(socket), + command=child_command(directory / "hopper.pid"), + ))) + + +with tempfile.TemporaryDirectory(prefix="rea-owner-lifetime-") as root: + directory = Path(root) + sentinel = subprocess.Popen([sys.executable, "-c", "import time;time.sleep(100)"]) + worker = subprocess.Popen( + [sys.executable, __file__, sys.argv[1], sys.argv[2], root], + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + try: + deadline = time.monotonic() + 5 + receipts = [directory / "hopper.pid", directory / "xvfb.pid"] + while not all(receipt.exists() for receipt in receipts): + assert worker.poll() is None, worker.communicate() + assert time.monotonic() < deadline, "owned children did not start" + time.sleep(0.01) + pids = [int(receipt.read_text()) for receipt in receipts] + assert worker.poll() is None, "live owner did not retain the helper" + assert worker.stdin is not None + worker.stdin.close() + assert worker.wait(timeout=5) == 0, worker.stderr.read() + for pid in pids: + assert not Path(f"/proc/{pid}").exists(), f"owned child {pid} survived EOF" + assert sentinel.poll() is None, "unrelated process was stopped" + print(json.dumps({"phase": sys.argv[2], "children_cleaned": True, "sentinel_alive": True})) + finally: + helper.stop_process(worker) + helper.stop_process(sentinel) diff --git a/tests/fixtures/idaMcp.ts b/tests/fixtures/idaMcp.ts index 04bdb5f16..6be35a8bf 100644 --- a/tests/fixtures/idaMcp.ts +++ b/tests/fixtures/idaMcp.ts @@ -57,6 +57,7 @@ const modernNames = [ "callees", "xrefs_to", "callgraph", + "survey_binary", ]; /** Stateful producer fixture at the real connection seam; no module mocking. */ @@ -67,6 +68,7 @@ export class RecordingIdaMcp implements IdaMcpConnection { }[] = []; connects = 0; closes = 0; + detaches = 0; inputSha256: string; inputPath: string; imageBase = "0x1000"; @@ -103,6 +105,9 @@ export class RecordingIdaMcp implements IdaMcpConnection { async close() { this.closes += 1; } + detach() { + this.detaches += 1; + } async call( name: string, args: Readonly>, diff --git a/tests/fixtures/javaClassFile.ts b/tests/fixtures/javaClassFile.ts index 4f7727193..30f5ffc37 100644 --- a/tests/fixtures/javaClassFile.ts +++ b/tests/fixtures/javaClassFile.ts @@ -1,5 +1,5 @@ /** Build a valid minimal JVM class file, optionally with an unused UTF-8 constant. */ -export const javaClassFile = (extraUtf8Bytes = 0): Buffer => { +export const javaClassFile = (extraUtf8Bytes = 0, fatCpuLookalike = false): Buffer => { const utf8 = (value: string): Buffer => { const bytes = Buffer.from(value, "utf8"); const entry = Buffer.alloc(3 + bytes.length); @@ -13,6 +13,7 @@ export const javaClassFile = (extraUtf8Bytes = 0): Buffer => { utf8("Hello"), Buffer.from([7, 0, 4]), utf8("java/lang/Object"), + ...(fatCpuLookalike ? [utf8("x"), Buffer.from([3, 0, 0, 0, 7])] : []), ...(extraUtf8Bytes === 0 ? [] : [utf8("x".repeat(extraUtf8Bytes))]), ]; const headerAndPool = Buffer.alloc(10); diff --git a/tests/fixtures/localToolsMcp.ts b/tests/fixtures/localToolsMcp.ts index 1e1e91909..bf9c7617a 100644 --- a/tests/fixtures/localToolsMcp.ts +++ b/tests/fixtures/localToolsMcp.ts @@ -34,7 +34,9 @@ const compileValidator = (schema: unknown): ValidateFunction => { }; /** Start the built production server without configuring deep binary providers. */ -export async function connectLocalToolsMcp() { +export async function connectLocalToolsMcp( + environment: Readonly> = {}, +) { const transport = new StdioClientTransport({ command: process.execPath, args: [ @@ -45,6 +47,7 @@ export async function connectLocalToolsMcp() { PATH: process.env.PATH ?? "/usr/bin:/bin", REA_LOG_LEVEL: "silent", HOPPER_LAUNCHER_PATH: "/rea-unconfigured-deep-provider/hopper", + ...environment, }, stderr: "pipe", }); diff --git a/tests/fixtures/native-macos/lipo-same-cpu/README.md b/tests/fixtures/native-macos/lipo-same-cpu/README.md new file mode 100644 index 000000000..d376e3b5d --- /dev/null +++ b/tests/fixtures/native-macos/lipo-same-cpu/README.md @@ -0,0 +1,19 @@ +# Universal Mach-O captures whose slices share one CPU type + +Captured on macOS 27.0.1 (26A434), arm64, from two system executables whose +FAT slices all use `CPU_TYPE_ARM64`: + +```sh +lipo -detailed_info /System/Applications/Calculator.app/Contents/MacOS/Calculator > calculator.txt +lipo -detailed_info /System/Library/CoreServices/ShortcutDroplet.app/Contents/MacOS/ShortcutDroplet > shortcut-droplet.txt +``` + +Calculator holds `arm64e` (subtype `0x80000002`) and `arm64e.x1` (subtype +`0x8000000c`); ShortcutDroplet holds `arm64` (subtype `0`) and `arm64e.x1`. +Target admission names both files by the single CPU family `arm64`, while lipo +lists two slices. + +The executable path on the `Fat header in:` line was replaced with `fixture`, +and trailing whitespace (lipo prints `capabilities ` for a slice without +capabilities) and the final blank line were removed. Tool fields are otherwise +unchanged. diff --git a/tests/fixtures/native-macos/lipo-same-cpu/calculator.txt b/tests/fixtures/native-macos/lipo-same-cpu/calculator.txt new file mode 100644 index 000000000..8e90b4902 --- /dev/null +++ b/tests/fixtures/native-macos/lipo-same-cpu/calculator.txt @@ -0,0 +1,17 @@ +Fat header in: fixture +fat_magic 0xcafebabe +nfat_arch 2 +architecture arm64e + cputype CPU_TYPE_ARM64 + cpusubtype CPU_SUBTYPE_ARM64E + capabilities PTR_AUTH_VERSION USERSPACE 0 + offset 16384 + size 1727552 + align 2^14 (16384) +architecture arm64e.x1 + cputype CPU_TYPE_ARM64 + cpusubtype unknown arm64 cpusubtype + capabilities PTR_AUTH_VERSION USERSPACE 0 + offset 1753088 + size 1711088 + align 2^14 (16384) diff --git a/tests/fixtures/native-macos/lipo-same-cpu/shortcut-droplet.txt b/tests/fixtures/native-macos/lipo-same-cpu/shortcut-droplet.txt new file mode 100644 index 000000000..252ff7234 --- /dev/null +++ b/tests/fixtures/native-macos/lipo-same-cpu/shortcut-droplet.txt @@ -0,0 +1,17 @@ +Fat header in: fixture +fat_magic 0xcafebabe +nfat_arch 2 +architecture arm64 + cputype CPU_TYPE_ARM64 + cpusubtype CPU_SUBTYPE_ARM64_ALL + capabilities + offset 16384 + size 52000 + align 2^14 (16384) +architecture arm64e.x1 + cputype CPU_TYPE_ARM64 + cpusubtype unknown arm64 cpusubtype + capabilities PTR_AUTH_VERSION USERSPACE 0 + offset 81920 + size 52000 + align 2^14 (16384) diff --git a/tests/fixtures/pcap/capinfos-stub.cmd b/tests/fixtures/pcap/capinfos-stub.cmd new file mode 100644 index 000000000..6f783bcf5 --- /dev/null +++ b/tests/fixtures/pcap/capinfos-stub.cmd @@ -0,0 +1,19 @@ +@echo off +rem capinfos stub for Windows: emulates capinfos --version and metadata output. + +if "%~1"=="--version" ( + echo Capinfos ^(Wireshark^) 4.6.6-windows-stub + exit /b 0 +) + +if not "%~1"=="-M" exit /b 2 +if not "%~2"=="--" exit /b 2 +if "%~3"=="" exit /b 2 + +echo File name: capture.pcap +echo File type: Wireshark/tcpdump/... - pcap +echo File encapsulation: Ethernet +echo Packet size limit: file hdr: 262144 bytes +echo Number of packets: 3 +echo Capture duration: 0.123 seconds +exit /b 0 diff --git a/tests/fixtures/pcap/capinfos-stub.sh b/tests/fixtures/pcap/capinfos-stub.sh new file mode 100755 index 000000000..47d61b1b1 --- /dev/null +++ b/tests/fixtures/pcap/capinfos-stub.sh @@ -0,0 +1,11 @@ +#!/bin/sh +# POSIX capinfos stub (mirrors capinfos-stub.cmd for Windows parity). +if [ "$1" = "--version" ]; then printf 'Capinfos (Wireshark) 4.6.6-posix-stub\n'; exit 0; fi +[ "$1" = "-M" ] && [ "$2" = "--" ] && [ "$#" = "3" ] || exit 2 +printf '%s\n' 'File name: capture.pcap' +printf '%s\n' 'File type: Wireshark/tcpdump/... - pcap' +printf '%s\n' 'File encapsulation: Ethernet' +printf '%s\n' 'Packet size limit: file hdr: 262144 bytes' +printf '%s\n' 'Number of packets: 3' +printf '%s\n' 'Capture duration: 0.123 seconds' +exit 0 diff --git a/tests/fixtures/pcap/tshark-stub.cmd b/tests/fixtures/pcap/tshark-stub.cmd new file mode 100644 index 000000000..d157ed721 --- /dev/null +++ b/tests/fixtures/pcap/tshark-stub.cmd @@ -0,0 +1,17 @@ +@echo off +rem Pcap tool stub for Windows: emulates tshark --version and field output. +rem %1 is the first argument; the capture path arrives as %2 for -r/-c. + +if "%~1"=="--version" ( + echo TShark ^(Wireshark^) 4.6.6-windows-stub + exit /b 0 +) + +if not "%~1"=="-n" exit /b 2 +if not "%~2"=="-r" exit /b 2 +if not "%~4"=="-T" exit /b 2 +if not "%~5"=="json" exit /b 2 + +rem Emit the same JSON as the POSIX stub +echo [{"_source":{"layers":{"frame.number":["1"],"frame.protocols":["eth:ethertype:ip:udp:dns"],"ip.src":["192.168.1.100"],"ip.dst":["8.8.8.8"],"dns.qry.name":["api.example.com"],"dns.flags.response":["1"],"dns.a":["93.184.216.34"]}}},{"_source":{"layers":{"frame.number":["2"],"frame.protocols":["eth:ethertype:ip:tcp:tls"],"ip.src":["192.168.1.100"],"ip.dst":["93.184.216.34"],"tls.handshake.extensions_server_name":["api.example.com"]}}},{"_source":{"layers":{"frame.number":["3"],"frame.protocols":["eth:ethertype:ip:tcp:http"],"ip.src":["192.168.1.100"],"ip.dst":["93.184.216.34"],"http.request.full_uri":["http://api.example.com/v1/data"]}}}] +exit /b 0 diff --git a/tests/fixtures/pcap/tshark-stub.sh b/tests/fixtures/pcap/tshark-stub.sh new file mode 100755 index 000000000..8633dd7c0 --- /dev/null +++ b/tests/fixtures/pcap/tshark-stub.sh @@ -0,0 +1,6 @@ +#!/bin/sh +# POSIX pcap tool stub (mirrors tshark-stub.cmd for Windows parity). +if [ "$1" = "--version" ]; then printf 'TShark (Wireshark) 4.6.6-posix-stub\n'; exit 0; fi +[ "$1" = "-n" ] && [ "$2" = "-r" ] && [ "$4" = "-T" ] && [ "$5" = "json" ] || exit 2 +printf '%s' '[{"_source":{"layers":{"frame.number":["1"],"frame.protocols":["eth:ethertype:ip:udp:dns"],"ip.src":["192.168.1.100"],"ip.dst":["8.8.8.8"],"dns.qry.name":["api.example.com"],"dns.flags.response":["1"],"dns.a":["93.184.216.34"]}}},{"_source":{"layers":{"frame.number":["2"],"frame.protocols":["eth:ethertype:ip:tcp:tls"],"ip.src":["192.168.1.100"],"ip.dst":["93.184.216.34"],"tls.handshake.extensions_server_name":["api.example.com"]}}},{"_source":{"layers":{"frame.number":["3"],"frame.protocols":["eth:ethertype:ip:tcp:http"],"ip.src":["192.168.1.100"],"ip.dst":["93.184.216.34"],"http.request.full_uri":["http://api.example.com/v1/data"]}}}]' +exit 0 diff --git a/tests/fixtures/strippedAsarAddon.ts b/tests/fixtures/strippedAsarAddon.ts index 5ff76356c..3422ac2ca 100644 --- a/tests/fixtures/strippedAsarAddon.ts +++ b/tests/fixtures/strippedAsarAddon.ts @@ -29,7 +29,7 @@ export const createStrippedAsarAddon = async ( await mkdir(dirname(path), { recursive: true }); await writeFile(path, file.contents); } - await createPackageWithOptions(source, archive, { unpack: "**/*.node" }); + await createPackageWithOptions(source, archive, { unpack: "*.node" }); await writeFile( join(`${archive}.unpacked`, "node_modules", "btime", "binding.node"), stripped, diff --git a/tests/fixtures/wasm/artifact.ts b/tests/fixtures/wasm/artifact.ts index d4cc803b4..3b615ca06 100644 --- a/tests/fixtures/wasm/artifact.ts +++ b/tests/fixtures/wasm/artifact.ts @@ -1,5 +1,6 @@ import { createHash } from "node:crypto"; import { createAnalysisExecution } from "../../../src/application/AnalysisProvider.js"; +import type { WasmArtifactExecution } from "../../../src/application/wasm/WasmArtifactPort.js"; import type { WasmArtifact } from "../../../src/domain/wasm/wasmArtifact.js"; /** Source-owned transport seam; upstream binaries are verified in a separate lane. */ export const wasmArtifactFixture = ( @@ -42,8 +43,10 @@ export const wasmArtifactFixture = ( network_fetch: "not-performed" as const, limitations: ["Source-owned transport seam."], }); -export const wasmArtifactExecution = (value = wasmArtifactFixture()) => - createAnalysisExecution( +export const wasmArtifactExecution = ( + value = wasmArtifactFixture(), +): WasmArtifactExecution => { + return createAnalysisExecution( value, { id: "fixture-wabt", name: "Source-owned WABT seam", version: "1" }, { @@ -55,3 +58,4 @@ export const wasmArtifactExecution = (value = wasmArtifactFixture()) => limitations: value.limitations, }, ); +}; diff --git a/tests/fixtures/windows/processBoundary.cc b/tests/fixtures/windows/processBoundary.cc index 18dcb2dbd..05ee218e8 100644 --- a/tests/fixtures/windows/processBoundary.cc +++ b/tests/fixtures/windows/processBoundary.cc @@ -4,6 +4,7 @@ #include #include #include +#include #include #include @@ -33,6 +34,41 @@ int wmain(int count, wchar_t** args) { value(L"REA_\u00df") == expectedSharp && value(L"REA_SS") == L"double" ? "true" : "false"); return 0; } + if ((count == 2 || count == 3) && std::wstring(args[1]) == L"affinity") { + // Without a process id this observes itself, which is what an owned child + // reports. With one it observes that process, so a caller's own affinity is + // read from the process that spawns the child rather than from a separate + // probe Windows may place in another processor group. + HANDLE opened = nullptr; + if (count == 3) { + opened = OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, FALSE, + static_cast(std::wcstoul(args[2], nullptr, 10))); + if (!opened) return 9; + } + const HANDLE target = opened ? opened : GetCurrentProcess(); + DWORD_PTR affinity = 0, systemAffinity = 0; + USHORT groupCount = 1; + USHORT groups[1]{}; + const bool read = GetProcessAffinityMask(target, &affinity, &systemAffinity) != 0; + const bool single = read && GetProcessGroupAffinity(target, &groupCount, groups) && groupCount == 1; + if (opened) CloseHandle(opened); + if (!read) return 8; + // A mask can set bits above JavaScript's safe-integer range, so masks cross + // as fixed-width hexadecimal strings and are compared without rounding. The + // group's own active set distinguishes an unconstrained process from one + // deliberately pinned to fewer processors. + auto mask = [](DWORD_PTR value, char* text) { + std::snprintf(text, 19, "0x%08lX%08lX", static_cast(value >> 32), + static_cast(value & 0xFFFFFFFFu)); + return text; + }; + char affinityText[19]{}, systemText[19]{}; + std::printf("{\"affinityMask\":\"%s\",\"systemMask\":\"%s\"," + "\"singleProcessorGroup\":%s,\"processorGroup\":%u}\n", + mask(affinity, affinityText), mask(systemAffinity, systemText), + single ? "true" : "false", single ? static_cast(groups[0]) : 0u); + return 0; + } if (count == 4 && std::wstring(args[1]) == L"reparse-directory") { // Only the conformance lane's disposable, nonempty source directory is used. const auto substitute = L"\\??\\" + std::wstring(args[3]); diff --git a/tests/process-global/cleanLinuxHopperRuntime.test.ts b/tests/process-global/cleanLinuxHopperRuntime.test.ts new file mode 100644 index 000000000..f6b9fc677 --- /dev/null +++ b/tests/process-global/cleanLinuxHopperRuntime.test.ts @@ -0,0 +1,116 @@ +import { execFile } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import { chmod, copyFile, readFile, symlink, unlink } from "node:fs/promises"; +import { join } from "node:path"; +import { promisify } from "node:util"; + +import { expect, it } from "vitest"; + +import { assertCleanLinuxHopperRuntime } from "../../scripts/lib/clean-linux-hopper-runtime.mjs"; +import { completeVerifierRun } from "../../scripts/lib/verifier-run.mjs"; +import { cleanupOwnedProcessGroup } from "../../src/process/ProcessOwnership.js"; +import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js"; + +const execFileAsync = promisify(execFile); + +it + .runIf(process.platform === "linux") + .each(["demo helper", "alternate launcher"])( + "rejects a reparented %s without signaling it", + async (mode) => { + const runId = randomUUID(); + const childCode = + "process.title = 'Xvfb'; setInterval(() => {}, 1000); process.send('ready');"; + const directory = await createTestTempDirectory("rea-clean-runtime-test-"); + const launcher = join(directory, "Hopper"); + const alternate = join(directory, "Hopper-alternate"); + const alternateMode = mode === "alternate launcher"; + if (alternateMode) { + await copyFile("/usr/bin/sleep", launcher); + await chmod(launcher, 0o700); + await symlink(launcher, alternate); + } + // A rejection must come from the fixture we are about to launch. An + // existing Xvfb would otherwise mask a broken executable-identity check. + await assertCleanLinuxHopperRuntime(alternateMode ? [launcher] : []); + const command = alternateMode ? alternate : process.execPath; + const arguments_ = alternateMode ? ["30"] : ["-e", childCode]; + const parentCode = ` + const { spawn } = require('node:child_process'); + const child = spawn(${JSON.stringify(command)}, ${JSON.stringify(arguments_)}, { + detached: true, stdio: ${JSON.stringify(alternateMode ? "ignore" : ["ignore", "ignore", "ignore", "ipc"])} + }); + child.once('error', () => process.exit(1)); + child.once(${JSON.stringify(alternateMode ? "spawn" : "message")}, () => { + process.stdout.write(JSON.stringify({ pid: child.pid, parentPid: process.pid })); + if (child.connected) child.disconnect(); + child.unref(); + }); + `; + const parent = execFileAsync(process.execPath, ["-e", parentCode], { + env: { ...process.env, REA_PROCESS_RUN_ID: runId }, + timeout: 10_000, + }); + const parentPid = parent.child.pid; + if (parentPid === undefined) + throw new Error("Fixture parent did not spawn"); + let orphanPid: number | undefined; + try { + const { stdout } = await parent; + const { pid } = JSON.parse(stdout) as { pid: number }; + orphanPid = pid; + const { stdout: coordinates } = await execFileAsync("ps", [ + "-p", + String(pid), + "-o", + "ppid=,pgid=", + ]); + const [currentParent, group] = coordinates + .trim() + .split(/\s+/u) + .map(Number); + expect(currentParent).not.toBe(parentPid); + expect(group).toBe(pid); + + const lineage = await completeVerifierRun({ + run_id: runId, + verifier_pid: process.pid, + parent_pid: process.ppid, + }); + expect(lineage.process_lineage.status).toBe("verified"); + if (lineage.process_lineage.status === "verified") + expect( + lineage.process_lineage.descendants.some((p) => p.pid === pid), + ).toBe(false); + + if (alternateMode) { + const commandLine = await readFile( + `/proc/${String(pid)}/cmdline`, + "utf8", + ); + expect(commandLine.split("\0")[0]).toBe(alternate); + await unlink(alternate); + } + await expect( + assertCleanLinuxHopperRuntime(alternateMode ? [launcher] : []), + ).rejects.toThrow("Clean Linux runner retained a Hopper demo process"); + // Observation must not terminate the fixture, even though it resembles Xvfb. + const { stdout: alive } = await execFileAsync("ps", [ + "-p", + String(pid), + "-o", + "stat=", + ]); + expect(alive.trim()).not.toMatch(/^Z/u); + } finally { + if (orphanPid !== undefined) { + const cleanup = await cleanupOwnedProcessGroup({ + runId, + leaderPid: orphanPid, + processGroupId: orphanPid, + }); + expect(cleanup.cleaned, JSON.stringify(cleanup)).toBe(true); + } + } + }, +); diff --git a/tests/process-global/pwntoolsStageRetirement.test.ts b/tests/process-global/pwntoolsStageRetirement.test.ts index bcdaf858f..7cdf0b48a 100644 --- a/tests/process-global/pwntoolsStageRetirement.test.ts +++ b/tests/process-global/pwntoolsStageRetirement.test.ts @@ -18,7 +18,10 @@ import { PWNTOOLS_MEMORY_FAILURE_EXIT, PWNTOOLS_PROVIDER_IDENTITY, } from "../../src/native/pwntools/PwntoolsRelease.js"; -import { cleanupOwnedProcessGroup } from "../../src/process/ProcessOwnership.js"; +import { + cleanupOwnedProcessGroup, + type OwnedProcessGroup, +} from "../../src/process/ProcessOwnership.js"; import { prepareProcessOwnershipInspection } from "../../src/process/ProcessOwnershipObservation.js"; import { spawnOwnedProviderProcess } from "../../src/process/ProviderProcess.js"; import { PrivateRuntimeRoot } from "../../src/process/PrivateRuntimeRoot.js"; @@ -232,12 +235,13 @@ it.runIf(supportedHost).each([ const { path, root, environment, basic } = await fixture(); let launches = 0; let cleanupAllowed = false; + let retainedOwnership: OwnedProcessGroup | undefined; const read = vi.spyOn(artifacts, "readStableArtifact"); const provider = new PwntoolsRecordedCrashProvider( environment, async (spawn) => { if (stage === "debugger" && ++launches === 1) return basic(spawn); - return changingWorker( + const launched = await changingWorker( spawn, join(spawn.cwd ?? "", "resource.failure"), { @@ -247,6 +251,8 @@ it.runIf(supportedHost).each([ }, () => cleanupAllowed, ); + retainedOwnership = launched.ownership; + return launched; }, async () => root, ); @@ -261,6 +267,14 @@ it.runIf(supportedHost).each([ if (result.ok) throw new Error("Expected unresolved process ownership"); const projected = projectAnalysisError(result.error); expect(projected.code).toBe("cleanup_incomplete"); + if (retainedOwnership === undefined) + throw new Error("Expected retained process ownership"); + expect(projected.details?.resources).toEqual([ + retainedOwnership.runId, + root.path, + `pid:${retainedOwnership.leaderPid}`, + `process-group:${retainedOwnership.processGroupId}`, + ]); if (stage === "debugger") expect(projected).toMatchObject({ details: { diff --git a/tests/process-global/regularFileNoFollowFallback.test.ts b/tests/process-global/regularFileNoFollowFallback.test.ts new file mode 100644 index 000000000..44ce66c7c --- /dev/null +++ b/tests/process-global/regularFileNoFollowFallback.test.ts @@ -0,0 +1,55 @@ +import { readFile, symlink } from "node:fs/promises"; +import { join } from "node:path"; + +import { expect, it, vi } from "vitest"; + +vi.mock("node:fs", async (importOriginal) => { + const original = await importOriginal(); + return { + ...original, + constants: { ...original.constants, O_NOFOLLOW: 0 }, + }; +}); + +import { openRegularFile } from "../../src/filesystem/RegularFile.js"; +import { ZipArtifactReader } from "../../src/artifacts/ZipArtifactReader.js"; +import { writeOrderedZip } from "../fixtures/artifactEntryOrder.js"; +import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js"; + +it("rejects a symlink when O_NOFOLLOW is unavailable and still follows when requested", async () => { + const root = await createTestTempDirectory("rea-no-nofollow-"); + const target = join(root, "target.zip"); + const link = join(root, "selected.zip"); + await writeOrderedZip(target, ["member.bin"]); + await symlink(target, link); + + const reader = new ZipArtifactReader(target, "zip"); + try { + const entries = []; + for await (const entry of reader.entries()) entries.push(entry); + expect(entries.map(({ path }) => path)).toEqual(["member.bin"]); + } finally { + await reader.close(); + } + + const linkedReader = new ZipArtifactReader(link, "zip"); + try { + await expect(async () => { + for await (const _entry of linkedReader.entries()) { + // Consume through the public reader boundary. + } + }).rejects.toMatchObject({ + name: "ArtifactReaderFailure", + reason: "format", + }); + } finally { + await linkedReader.close(); + } + + const followed = await openRegularFile(link, { symlinks: "follow" }); + try { + await expect(readFile(followed)).resolves.toEqual(await readFile(target)); + } finally { + await followed.close(); + } +}); diff --git a/tests/process-global/setupConfigurationRollback.test.ts b/tests/process-global/setupConfigurationRollback.test.ts new file mode 100644 index 000000000..cf6504c32 --- /dev/null +++ b/tests/process-global/setupConfigurationRollback.test.ts @@ -0,0 +1,325 @@ +import { open, readFile } from "node:fs/promises"; +import { createHash } from "node:crypto"; +import { join } from "node:path"; + +import { afterEach, expect, it, onTestFinished, vi } from "vitest"; + +const state = vi.hoisted(() => ({ + readbackFailed: false, + rejectRestore: false, + rejectRemove: false, + removeCopiedRecovery: false, + initialCleanupFailure: undefined as unknown, + readbackCleanupFailure: undefined as unknown, +})); + +afterEach(() => { + state.readbackFailed = false; + state.rejectRestore = false; + state.rejectRemove = false; + state.removeCopiedRecovery = false; + state.initialCleanupFailure = undefined; + state.readbackCleanupFailure = undefined; +}); + +vi.mock("node:fs/promises", async (importOriginal) => { + const original = await importOriginal(); + return { + ...original, + rm: ( + ...arguments_: Parameters + ): ReturnType => { + if (state.rejectRemove && state.readbackFailed) + return Promise.reject(new Error("injected restore remove failure")); + return original.rm(...arguments_); + }, + writeFile: ( + ...arguments_: Parameters + ): ReturnType => { + if (state.rejectRestore && state.readbackFailed) + return Promise.reject(new Error("injected restore write failure")); + return original.writeFile(...arguments_); + }, + copyFile: async ( + ...arguments_: Parameters + ): Promise => { + await original.copyFile(...arguments_); + const destination = arguments_[1]; + if ( + state.removeCopiedRecovery && + typeof destination === "string" && + destination.endsWith(".recovery") + ) + await original.rm(destination); + }, + }; +}); + +vi.mock("write-file-atomic", async (importOriginal) => { + const original = await importOriginal<{ + default: typeof import("write-file-atomic"); + }>(); + return { + ...original, + default: async (...arguments_: Parameters) => { + if (state.initialCleanupFailure !== undefined) + throw new Error("unexpected config write before initial read"); + const result = await original.default(...arguments_); + state.readbackFailed = true; + return result; + }, + }; +}); + +vi.mock("../../src/application/RegularFileRead.js", async (importOriginal) => { + const original = + await importOriginal< + typeof import("../../src/application/RegularFileRead.js") + >(); + return { + ...original, + readRegularFileText: ( + ...arguments_: Parameters + ) => + state.initialCleanupFailure !== undefined + ? Promise.reject(state.initialCleanupFailure) + : state.readbackFailed && state.readbackCleanupFailure !== undefined + ? Promise.reject(state.readbackCleanupFailure) + : state.readbackFailed + ? Promise.reject(new Error("injected readback failure")) + : original.readRegularFileText(...arguments_), + }; +}); + +import { configureClientConfiguration } from "../../src/application/SetupClientConfiguration.js"; +import { RegularFileCleanupFailure } from "../../src/application/RegularFileRead.js"; +import { OwnedFileHandle } from "../../src/filesystem/OwnedFileHandle.js"; +import type { DoctorHost } from "../../src/application/Doctor.js"; +import { systemSetupHost } from "../../src/application/SetupHost.js"; +import { ConfigurationError } from "../../src/domain/configurationErrors.js"; +import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js"; + +const ownReadHandle = async (path: string) => { + const owner = new OwnedFileHandle(await open(path, "r")); + onTestFinished(() => owner.close()); + return owner; +}; + +it("reports a failed config rollback and retains its backup path", async () => { + state.readbackFailed = false; + state.rejectRestore = false; + state.rejectRemove = false; + state.initialCleanupFailure = undefined; + state.readbackCleanupFailure = undefined; + const root = await createTestTempDirectory("rea-config-rollback-"); + const path = join(root, "client.json"); + const original = '{"keep":true}\n'; + await import("node:fs/promises").then(({ writeFile }) => + writeFile(path, original), + ); + state.rejectRestore = true; + + const result = await configureClientConfiguration( + { name: "cursor", configPath: path, format: "json" }, + {}, + ["rea", "mcp"], + ); + + expect(result).toEqual({ + status: "failed", + reason: "rollback", + backupPath: `${path}.rea.backup`, + failure: { + kind: "restore", + path, + detail: "injected restore write failure", + }, + }); + expect(await readFile(`${path}.rea.backup`, "utf8")).toBe(original); + expect(JSON.parse(await readFile(path, "utf8"))).toMatchObject({ + mcpServers: { rea: { command: "rea", args: ["mcp"] } }, + }); +}); + +it("preserves the first backup and reports a verified backup of the current original", async () => { + state.readbackFailed = false; + state.rejectRestore = false; + state.rejectRemove = false; + state.initialCleanupFailure = undefined; + state.readbackCleanupFailure = undefined; + const root = await createTestTempDirectory("rea-config-stale-backup-"); + const path = join(root, "client.json"); + const firstOriginal = '{"keep":"first"}\n'; + const currentOriginal = '{"keep":"current"}\n'; + await import("node:fs/promises").then(async ({ writeFile }) => { + await writeFile(path, currentOriginal); + await writeFile(`${path}.rea.backup`, firstOriginal); + }); + state.rejectRestore = true; + + const result = await configureClientConfiguration( + { name: "cursor", configPath: path, format: "json" }, + {}, + ["rea", "mcp"], + ); + state.rejectRestore = false; + + const recoveryPath = `${path}.rea.backup.${createHash("sha256") + .update(currentOriginal, "utf8") + .digest("hex")}.recovery`; + expect(result).toMatchObject({ + status: "failed", + reason: "rollback", + backupPath: recoveryPath, + failure: { + kind: "restore", + path, + detail: "injected restore write failure", + }, + }); + expect(await readFile(`${path}.rea.backup`, "utf8")).toBe(firstOriginal); + expect(await readFile(recoveryPath, "utf8")).toBe(currentOriginal); +}); + +it("does not write config when a recovery copy disappears before verification", async () => { + state.readbackFailed = false; + state.rejectRestore = false; + state.rejectRemove = false; + state.removeCopiedRecovery = false; + state.initialCleanupFailure = undefined; + state.readbackCleanupFailure = undefined; + const root = await createTestTempDirectory("rea-config-vanished-recovery-"); + const path = join(root, "client.json"); + const firstOriginal = '{"keep":"first"}\n'; + const currentOriginal = '{"keep":"current"}\n'; + await import("node:fs/promises").then(async ({ writeFile }) => { + await writeFile(path, currentOriginal); + await writeFile(`${path}.rea.backup`, firstOriginal); + }); + state.removeCopiedRecovery = true; + + const result = await configureClientConfiguration( + { name: "cursor", configPath: path, format: "json" }, + {}, + ["rea", "mcp"], + ); + state.removeCopiedRecovery = false; + + expect(result).toEqual({ status: "failed", reason: "backup" }); + expect(await readFile(path, "utf8")).toBe(currentOriginal); + expect(await readFile(`${path}.rea.backup`, "utf8")).toBe(firstOriginal); +}); + +it("does not roll back or remove config when its initial read cleanup fails", async () => { + state.readbackFailed = false; + state.rejectRestore = false; + state.rejectRemove = false; + state.readbackCleanupFailure = undefined; + const root = await createTestTempDirectory("rea-config-initial-cleanup-"); + const path = join(root, "client.json"); + const original = '{"keep":true}\n'; + await import("node:fs/promises").then(({ writeFile }) => + writeFile(path, original), + ); + const failure = new RegularFileCleanupFailure( + path, + await ownReadHandle(path), + { kind: "completed", value: original }, + new Error("injected initial close failure"), + ); + state.initialCleanupFailure = failure; + + await expect( + configureClientConfiguration( + { name: "cursor", configPath: path, format: "json" }, + {}, + ["rea", "mcp"], + ), + ).rejects.toBe(failure); + expect(await readFile(path, "utf8")).toBe(original); +}); + +it("reports first-run rollback removal failure without a backup", async () => { + state.readbackFailed = false; + state.rejectRestore = false; + state.rejectRemove = false; + state.initialCleanupFailure = undefined; + state.readbackCleanupFailure = undefined; + const root = await createTestTempDirectory("rea-config-first-run-rollback-"); + const path = join(root, "client.json"); + state.rejectRemove = true; + + const result = await configureClientConfiguration( + { name: "cursor", configPath: path, format: "json" }, + {}, + ["rea", "mcp"], + ); + state.rejectRemove = false; + + expect(result).toMatchObject({ + status: "failed", + reason: "rollback", + failure: { + kind: "restore", + path, + detail: "injected restore remove failure", + }, + }); + expect(result).not.toHaveProperty("backupPath"); + expect(JSON.parse(await readFile(path, "utf8"))).toMatchObject({ + mcpServers: { rea: { command: "rea", args: ["mcp"] } }, + }); +}); + +it("retains readback cleanup ownership when rollback also fails", async () => { + state.readbackFailed = false; + state.rejectRestore = false; + state.rejectRemove = false; + state.initialCleanupFailure = undefined; + const root = await createTestTempDirectory("rea-config-readback-cleanup-"); + const path = join(root, "client.json"); + const original = '{"keep":true}\n'; + await import("node:fs/promises").then(({ writeFile }) => + writeFile(path, original), + ); + const owner = await ownReadHandle(path); + const cleanupFailure = new RegularFileCleanupFailure( + path, + owner, + { kind: "completed", value: "" }, + new Error("injected readback close failure"), + ); + state.readbackCleanupFailure = cleanupFailure; + state.rejectRestore = true; + + const host = systemSetupHost( + { platform: "linux", homeDirectory: root } as DoctorHost, + { HOME: root }, + ); + let caught: unknown; + try { + await host.configureClient( + { name: "cursor", configPath: path, format: "json" }, + {}, + ["rea", "mcp"], + ); + } catch (cause: unknown) { + caught = cause; + } finally { + await host.close?.(); + } + + if (!(caught instanceof ConfigurationError)) throw caught; + expect(caught.message).toContain("injected restore write failure"); + expect(caught.message).toContain(`${path}.rea.backup`); + const cleanupOwnerError = caught.cause; + if (!(cleanupOwnerError instanceof RegularFileCleanupFailure)) + throw new Error("Configuration failure lost its read owner"); + expect(cleanupOwnerError).toMatchObject({ + path, + outcome: { kind: "completed", value: "" }, + cleanupCause: expect.any(AggregateError), + }); + expect(cleanupOwnerError.owner).toBe(owner); + expect(await readFile(`${path}.rea.backup`, "utf8")).toBe(original); +}); diff --git a/tests/process-global/updateMaintenance.test.ts b/tests/process-global/updateMaintenance.test.ts index a60a2e213..ae057bab2 100644 --- a/tests/process-global/updateMaintenance.test.ts +++ b/tests/process-global/updateMaintenance.test.ts @@ -126,6 +126,41 @@ describe("OMP integration maintenance", () => { }); }); +describe("maintenance activation and layered configuration", () => { + it("leaves a natively disabled ZCode registration outside maintenance", async () => { + const configPath = await writeClient( + "zcode", + JSON.stringify({ + mcp: { + servers: { + rea: { command: entryPoint, args: ["mcp"], enable: false }, + }, + }, + }), + ); + const original = await readFile(configPath, "utf8"); + await expect( + existingMaintenanceScope(home, entryPoint, process.env), + ).resolves.toMatchObject({ clients: [] }); + expect(await readFile(configPath, "utf8")).toBe(original); + }); + + it("selects an enabled registration inherited through layered OpenCode files", async () => { + const directory = join(home, ".config", "opencode"); + await mkdir(directory, { recursive: true }); + await writeFile( + join(directory, "opencode.json"), + JSON.stringify({ + mcp: { rea: { type: "local", command: [entryPoint, "mcp"] } }, + }), + ); + await writeClient("opencode", JSON.stringify({ theme: "user-selected" })); + await expect( + existingMaintenanceScope(home, entryPoint, process.env), + ).resolves.toMatchObject({ clients: ["opencode"] }); + }); +}); + describe("existing REA integration maintenance", () => { it("selects owned registrations while preserving unconfigured, foreign, and disabled choices", async () => { await writeClient("codex", staleCodex); diff --git a/tests/process-global/windowsRuntimeAdmission.test.ts b/tests/process-global/windowsRuntimeAdmission.test.ts new file mode 100644 index 000000000..a9f328a18 --- /dev/null +++ b/tests/process-global/windowsRuntimeAdmission.test.ts @@ -0,0 +1,99 @@ +import { beforeEach, expect, it, vi } from "vitest"; +import { WindowsPrivateRuntime } from "../../src/windows/WindowsPrivateRuntime.js"; +import { PrivateRuntimeRootUnavailableError } from "../../src/process/PrivateRuntimeRootUnavailableError.js"; + +const { nativeCall } = vi.hoisted(() => ({ nativeCall: vi.fn() })); +vi.mock("../../src/windows/WindowsNativeLoader.js", async (original) => ({ + ...(await original< + typeof import("../../src/windows/WindowsNativeLoader.js") + >()), + requireWindowsNativeAuthority: () => ({ call: nativeCall }), +})); +beforeEach(() => { + nativeCall.mockReset(); +}); + +it.each([ + { + label: "sharing conflict before allocation", + win32Code: 32, + runtimeAllocation: "not-started", + retryable: true, + }, + { + label: "lock conflict before allocation", + win32Code: 33, + runtimeAllocation: "not-started", + retryable: true, + }, + { label: "unknown allocation", win32Code: 32, retryable: false }, + { + label: "possible allocation", + win32Code: 32, + runtimeAllocation: "started", + retryable: false, + }, + { + label: "permission failure", + win32Code: 5, + runtimeAllocation: "not-started", + retryable: false, + }, + { + label: "string code", + win32Code: "32", + runtimeAllocation: "not-started", + retryable: false, + }, + { + label: "inconsistent error code", + win32Code: 32, + code: "EACCES", + runtimeAllocation: "not-started", + retryable: false, + }, +])( + "retains $label without guessing runtime ownership", + ({ retryable, label: _label, ...facts }) => { + const cause = Object.assign(new Error("Selected native diagnostic"), { + code: "ERR_REA_WINDOWS_NATIVE", + ...facts, + }); + nativeCall.mockImplementation(() => { + throw cause; + }); + let observed: unknown; + try { + WindowsPrivateRuntime.create("D:/selected parent", "selected-prefix-"); + } catch (failure: unknown) { + observed = failure; + } + expect(nativeCall).toHaveBeenCalledTimes(1); + expect(nativeCall).toHaveBeenCalledWith("runtime_create", [ + "D:/selected parent", + "selected-prefix-", + ]); + if (retryable) { + expect(observed).toBeInstanceOf(PrivateRuntimeRootUnavailableError); + expect(observed).toMatchObject({ + message: cause.message, + cause, + retryable: true, + }); + } else expect(observed).toBe(cause); + }, +); + +it("does not infer a transient status from a human-readable diagnostic", () => { + const cause = new Error("Open Windows path component failed (Win32 32)"); + nativeCall.mockImplementation(() => { + throw cause; + }); + let observed: unknown; + try { + WindowsPrivateRuntime.create("D:/selected parent", "selected-prefix-"); + } catch (failure: unknown) { + observed = failure; + } + expect(observed).toBe(cause); +}); diff --git a/turbo.json b/turbo.json index e8bfbf06e..758f31e32 100644 --- a/turbo.json +++ b/turbo.json @@ -151,6 +151,7 @@ "scripts/fixtures/apple/*.mjs", "src/generated*.ts", "src/contracts/**/*.ts", + "licenses/**", "docs/**", "!docs/public/product-catalog.json", "!docs/verification/managed-conformance-*.json", diff --git a/vitest.config.ts b/vitest.config.ts index 74dbf5305..bfef8e139 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -51,7 +51,7 @@ const TEST_PROJECTS = [ name: "adapters", include: [ "src/*.test.ts", - "src/{apktool,jeb,adb,artifacts,browser,dotnet,evm,firmware,flutter,ghidra,go,hopper,ida,inspector,javascript,native,process,reference,server,wasm,windows}/**/*.test.ts", + "src/{apktool,jeb,adb,artifacts,browser,dotnet,evm,firmware,flutter,pcap,ghidra,go,hopper,ida,inspector,javascript,native,process,reference,server,wasm,windows}/**/*.test.ts", ], pool: "forks" as const, maxWorkers: MAX_TEST_WORKERS, diff --git a/website/README.md b/website/README.md index b7a159570..f09c7f603 100644 --- a/website/README.md +++ b/website/README.md @@ -131,6 +131,7 @@ describe its recorded investigation; update them only when repeating that work. - `public/showcase/index.html`: the case-study index. - `public/blog/index.html`: articles about reconstruction methods, ports, mods and reverse engineering. - `public/blog/touhou-reconstruction/index.html`: a personal essay about agent autonomy, oracle quality, knowledge carried between TH08 and TH095, PC-98 TH04, and the path from exact reconstruction to readable source and modern ports. +- `public/blog/reverse-engineering-models/index.html`: an essay on general models, compiler specialization and learning useful reconstruction experiments, with historical evidence from TH08, TH095, TH04 and TH09. Model attribution and verification scope are in [evidence/reverse-engineering-models.md](evidence/reverse-engineering-models.md). - `public/showcase/aegis/index.html`: Aegis's Android login-code calculation, with an adjustable clock and reference checks. - `public/showcase/dx-ball/index.html`: sound-pan investigation and project status. - `public/showcase/notion/index.html`: Notion's Electron clipboard bridge and rich clipboard format. diff --git a/website/evidence/reverse-engineering-models.md b/website/evidence/reverse-engineering-models.md new file mode 100644 index 000000000..069d6eea3 --- /dev/null +++ b/website/evidence/reverse-engineering-models.md @@ -0,0 +1,166 @@ +# General models and specialist models: article evidence + +Supporting record for `public/blog/reverse-engineering-models/index.html`. +The article is published from the English source and translated through the +website's gettext catalogs. + +## Research scope + +Repository history, maintained source and recorded comparison notes were reviewed +on 13 October 2026. This research did not run game targets, compile reconstruction +candidates or repeat historical runtime checks. Public commit availability was +checked through GitHub CLI. + +The evidence describes specific historical functions and their recorded checks. +Source presence, compiler exactness, complete-product builds and runtime agreement +remain distinct. The proposed specialist/RL comparisons are future experiments; +no trained specialist was evaluated for this article. + +## Model attribution + +- The project operator identifies `gpt-dots` as **GPT-6-Astra**. +- `gpt-web` is a changing label for the model used at the time. The operator + describes then-latest models, usually xhigh, commonly GPT-5.6-Sol earlier and + GPT-6-Sol recently. Do not assign all such commits a single model or infer + individual identities from a release date. +- Explicit `gpt-6-sol` and `gpt-6.1-sol` subjects establish the recorded names. + Per-run reasoning effort and full budgets are not uniformly recorded. +- `gpt-6-luna-max` records GPT-6-Luna and max effort. It does not establish + parameter count or specialist training. + +Later sessions inherited earlier source, diagnostics and failed experiments. +Historical before/after closures support the essay's questions; they are not +controlled estimates of model-size effects. + +## TH09 Type18/24 attack update + +Earlier source-context review: +[GPT-6-Sol, 26 September](https://github.com/N0zoM1z0/th09/commit/de63c0f9121963f3a6b1ab5471c4878625fd3425). +Closing reconstruction: +[GPT-6.1-Sol, 10 October](https://github.com/N0zoM1z0/th09/commit/52929ed7c299f1a5f29cc13964e22510145fd616). + +The earlier candidate was 1,435 bytes against a 1,436-byte target. Matching the +overall frame size concealed different homes for returned vectors and real +collision storage. Declaration and helper-visibility controls did not close it. + +The closing session introduces an ordinary XYZ destination assignment, consumes +real returned vectors directly and groups six actual publication/rotation +operations into one comma-sequenced full expression. Keeping the real collision +point at function scope preserves separate storage. The seventh warm candidate +matches, followed by a maintained-path canonical cold comparison of all 1,436 +bytes and 44 encoded fields. Original class/source spelling and later runtime +gates remain separate. + +The article's four-row table groups selected lessons from seven candidates. It +does not enumerate every build or reproduce a complete source listing. + +## TH09 Front callback + +Earlier diagnostic checkpoint: +[GPT-6.1-Sol, 2 October](https://github.com/N0zoM1z0/th09/commit/f8d12d72aaf02212ee1b585a8f3f86d7ed000269). +Full graph closure: +[GPT-6-Astra (`gpt-dots`), 6 October](https://github.com/N0zoM1z0/th09/commit/da33a646221198ceb7f3df2b96e2ca0e9b2f705f). +Accepted replay: +[6 October follow-up](https://github.com/N0zoM1z0/th09/commit/5f825b28a862f8e1a6d4aea2f78a42f04c09a1c0). + +The earlier maintained owner was 1,510 bytes versus the 1,491-byte target. Its +prefix and suffix matched only as independently positioned diagnostic regions; +they did not establish a complete-function match. + +A common outer continuation places completion and frame dispatch in complementary +arms with one final frame increment/return. Within this graph, the side/stage +decision recovers target block sharing. The record establishes the complete +1,491-byte owner, 107 independently bound fields, 384 instructions and 42 calls. +The follow-up records accepted forced-recompile replay for `da33a646`. +Bounded trace models are static checks, not game-runtime validation. + +## TH095 identifier-bucket reuse + +[Supervisor::LoadDat, 2 September](https://github.com/N0zoM1z0/th095/commit/26bb64bdb093a13f73abd0ee8ae7ce199ee4658f). + +Under observed VC7.1 build 3077, declaration order alone did not recover the real +local homes. Earlier background work established useful identifier buckets. +Backing the real file-size scalar with `averagedPanLocal12` places it at +`EBP-0x04` and the genuine 128-byte buffer at `EBP-0x88`. All 151 bytes and 15 +relocations close. + +This is compiler-context evidence. It does not establish a universal C++ rule +or uniquely recover the original author's variable name. The `gpt-web` label +does not identify this run's model individually. + +## TH09 Luna enemy-spawn closure + +[GPT-6-Luna-max, 27 September](https://github.com/N0zoM1z0/th09/commit/1eb843383941d00065622a9e353146c4bb96979e). + +An ordinary loop with `continue`/`break`, branch-local RunEcl failure handling +and a shared success tail closes the 373-byte owner. TH08 source supplies the +source-family hypothesis; TH09 establishes the bounds, fields, ABI and callees. +Two independent cold builds and canonical replay are recorded. + +The earlier stable 361-byte candidate is `gpt-web`, with an individually +unresolved model identity. This case demonstrates a useful discovery across +model tiers, not a measured specialist or smaller-parameter advantage. + +## TH04 compiler diversity + +[Player MAIN_0 cohort, 15 September](https://github.com/N0zoM1z0/th04/commit/6dce0a79d5b931f0e2a08475b90e0c33d2321b98). + +Natural Turbo C++ 4.0J source produces a 629-byte update/miss contribution and a +334-byte renderer. Expression lifetimes, branch-local stores, declaration order +and a shared firing tail matter. Correct physical linker order avoids an extra +alignment byte. The recorded focused and aggregate replays cover 129 and 208 +owners respectively. These are historical owned-extent checks; this checkpoint +does not establish whole-game execution. + +TH08 uses VC7, while TH095/TH09 use VC7.1. Avoid treating every game's source +patterns as output from the same compiler. Shared game authorship supplies a +prior; target-local evidence validates an actual transfer. + +## TH08 numerical investigation + +[Polar motion and x87 behavior, 13 October](https://github.com/N0zoM1z0/th08/commit/af9cb1d3e839b15a78b47ac1701506083cc82847). + +A collision divergence at stage 2 frame 6,825 leads back to a one-ULP +displacement difference at frame 5,573. Historical CRT wrappers retain a wider +x87 return despite storing a rounded float. A distinct i386 case involves x87 +precision-control state. The record includes field probes and replay checks for +specified scenarios. Several preceding investigations use the same model; this +is breadth-of-investigation evidence, not a model-switch comparison. + +## TH08 floating-literal oracle repair + +[Repository-wide audit, 2 September](https://github.com/N0zoM1z0/th08/commit/a393f400786e4cc6cbb1000af0dad3276da12c78). + +Literal payload checks previously depended on optional explicit `data_hex` +attestation. The fix decodes and target-checks every VC7 `__real@...` relocation. +The follow-up audits 1,548 relocations and corrects 12 stale rows in five already +accepted units. One source change corrects a bullet-spread increment from +`1.0f` to `32.0f`. + +Affected-unit replay and a complete 1,106-unit cold replay are recorded after +repair. This defect is separate from the later one-ULP numerical investigation. +The RL consequence in the article is an inference: a reward based on the earlier +check could reinforce wrong source. The record does not describe an actual RL +exploit or trained model doing so. + +## TH09 target-sized replay-save candidate + +[Replay-save closure, 2 October](https://github.com/N0zoM1z0/th09/commit/ab72fd77728293fc0beacfc656ca701b0e266016). + +The earlier candidate already had the target's 1,825-byte length but 467 complete +comparison differences. Shared index identity, loop structure and the real +cancel/overwrite continuation close the complete owner. The example supports +using size as a diagnostic rather than final acceptance. + +## Related research + +The article links these primary sources: + +- [LLM4Decompile, EMNLP 2024](https://aclanthology.org/2024.emnlp-main.203/). +- [AutoDecompiler, June 2026 preprint](https://arxiv.org/html/2606.16162v1). +- [When LLM Decompilers Recompile More and Preserve Less, September 2026 preprint](https://arxiv.org/html/2609.05370v1). + +These works concern specialized decompilation, feedback-based refinement and +functional checking. Their published setups do not establish a winner for the +article's proposed historical-compiler reconstruction experiment. Preserve that +objective distinction when expanding the discussion. diff --git a/website/locales/ar/messages.po b/website/locales/ar/messages.po index 8a7ea9d40..daaa1d33a 100644 --- a/website/locales/ar/messages.po +++ b/website/locales/ar/messages.po @@ -32,18 +32,18 @@ msgid "Thoughts on reverse engineering." msgstr "أفكار حول الهندسة العكسية." #. {ph0}:
-#: website/public/blog/index.html:200 +#: website/public/blog/index.html:214 #, python-brace-format msgctxt "page:blog/index.html" msgid "Game reconstruction{ph0}Essay" msgstr "لعبة إعادة الإعمار{ph0}مقال" -#: website/public/blog/index.html:202 +#: website/public/blog/index.html:216 msgctxt "page:blog/index.html" msgid "The industrial revolution in reverse engineering" msgstr "الثورة الصناعية في الهندسة العكسية" -#: website/public/blog/index.html:203 +#: website/public/blog/index.html:217 msgctxt "page:blog/index.html" msgid "" "What manual RE and reconstructing TH08, TH095 and TH04 taught me about " @@ -52,25 +52,26 @@ msgstr "" "ما دليل إعادة تعمير TH08, TH095 و TH04 علمني عن وكلاء والتحقق المعرفة أن " "المركبات عبر المشاريع." -#: website/public/blog/index.html:208 +#: website/public/blog/index.html:207 website/public/blog/index.html:222 msgctxt "page:blog/index.html" msgid "Read the article" msgstr "اقرأ المقال" -#: website/public/blog/index.html:211 +#: website/public/blog/index.html:225 msgctxt "page:blog/index.html" msgid "By N0zoM1z0 · 10 October 2026" msgstr "بواسطة N0zoM1z0 * 10 أكتوبر 2026" #. {ph0}: -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 +#: website/public/blog/reverse-engineering-models/index.html:883 #: website/public/blog/touhou-reconstruction/index.html:1080 #: website/public/examples/dino-lab/index.html:475 #: website/public/faq/index.html:573 #: website/public/first-investigation/index.html:523 #: website/public/get-started/index.html:882 -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 #: website/public/guides/index.html:282 #: website/public/guides/javascript/index.html:503 #: website/public/guides/native/index.html:509 website/public/index.html:1191 @@ -100,6 +101,8 @@ msgstr "" "العفو الدولية." #: website/public/blog/index.html:75 website/public/blog/index.html:88 +#: website/public/blog/reverse-engineering-models/index.html:75 +#: website/public/blog/reverse-engineering-models/index.html:88 #: website/public/blog/touhou-reconstruction/index.html:75 #: website/public/blog/touhou-reconstruction/index.html:88 #: website/public/examples/dino-lab/index.html:75 @@ -134,6 +137,7 @@ msgid "" msgstr "REA - اكتشف كيف يعمل البرنامج. الهندسة العكسية مع مساعدك البرمجي." #: website/public/blog/index.html:108 +#: website/public/blog/reverse-engineering-models/index.html:108 #: website/public/blog/touhou-reconstruction/index.html:108 #: website/public/examples/dino-lab/index.html:109 #: website/public/faq/index.html:108 @@ -154,6 +158,7 @@ msgid "REA home" msgstr "الصفحة الرئيسية لـ REA" #: website/public/blog/index.html:112 +#: website/public/blog/reverse-engineering-models/index.html:112 #: website/public/blog/touhou-reconstruction/index.html:112 #: website/public/examples/dino-lab/index.html:113 #: website/public/faq/index.html:112 @@ -174,6 +179,7 @@ msgid "Main navigation" msgstr "التنقل الرئيسي" #: website/public/blog/index.html:130 +#: website/public/blog/reverse-engineering-models/index.html:130 #: website/public/blog/touhou-reconstruction/index.html:130 #: website/public/examples/dino-lab/index.html:131 #: website/public/faq/index.html:130 @@ -194,6 +200,7 @@ msgid "REA on GitHub" msgstr "REA على GitHub" #: website/public/blog/index.html:150 +#: website/public/blog/reverse-engineering-models/index.html:150 #: website/public/blog/touhou-reconstruction/index.html:150 #: website/public/examples/dino-lab/index.html:151 #: website/public/faq/index.html:150 @@ -214,6 +221,7 @@ msgid "Join REA on Discord" msgstr "انضم إلى REA على Discord" #: website/public/blog/index.html:169 +#: website/public/blog/reverse-engineering-models/index.html:169 #: website/public/blog/touhou-reconstruction/index.html:169 #: website/public/examples/dino-lab/index.html:170 #: website/public/faq/index.html:169 @@ -233,7 +241,7 @@ msgctxt "shared" msgid "Follow REA on X" msgstr "تابع REA على X" -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 msgctxt "page:blog/index.html" msgid "Back to top" msgstr "العودة للأعلى" @@ -4660,7 +4668,7 @@ msgctxt "page:guides/browser/index.html" msgid "Copy read the loaded scripts" msgstr "نسخ قراءة البرامج النصية المحملة" -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 msgctxt "page:guides/browser/index.html" msgid "Back to top" msgstr "العودة للأعلى" @@ -11384,6 +11392,7 @@ msgid "Same spacing: 256 ÷ 16 = 16 angle units = 22.5°" msgstr "نفس التباعد: 256 1 16 = 16 وحدة زاوية = 22.5°" #: website/public/blog/index.html:114 +#: website/public/blog/reverse-engineering-models/index.html:114 #: website/public/blog/touhou-reconstruction/index.html:114 #: website/public/examples/dino-lab/index.html:115 #: website/public/faq/index.html:114 @@ -11404,6 +11413,7 @@ msgid "Showcases" msgstr "أمثلة عملية" #: website/public/blog/index.html:115 +#: website/public/blog/reverse-engineering-models/index.html:115 #: website/public/blog/touhou-reconstruction/index.html:115 #: website/public/examples/dino-lab/index.html:116 #: website/public/faq/index.html:115 @@ -11424,6 +11434,7 @@ msgid "Blog" msgstr "المدونة" #: website/public/blog/index.html:113 +#: website/public/blog/reverse-engineering-models/index.html:113 #: website/public/blog/touhou-reconstruction/index.html:113 #: website/public/examples/dino-lab/index.html:114 #: website/public/faq/index.html:113 @@ -11444,6 +11455,7 @@ msgid "Get started" msgstr "ابدأ هنا" #: website/public/blog/index.html:116 +#: website/public/blog/reverse-engineering-models/index.html:116 #: website/public/blog/touhou-reconstruction/index.html:116 #: website/public/examples/dino-lab/index.html:117 #: website/public/faq/index.html:116 @@ -11463,13 +11475,14 @@ msgctxt "shared" msgid "Guides" msgstr "الأدلة" -#: website/public/blog/index.html:222 +#: website/public/blog/index.html:236 +#: website/public/blog/reverse-engineering-models/index.html:875 #: website/public/blog/touhou-reconstruction/index.html:1072 #: website/public/examples/dino-lab/index.html:467 #: website/public/faq/index.html:565 #: website/public/first-investigation/index.html:515 #: website/public/get-started/index.html:874 -#: website/public/guides/browser/index.html:490 +#: website/public/guides/browser/index.html:499 #: website/public/guides/index.html:274 #: website/public/guides/javascript/index.html:495 #: website/public/guides/native/index.html:501 website/public/index.html:1183 @@ -11483,13 +11496,14 @@ msgctxt "shared" msgid "Source" msgstr "الكود المصدري" -#: website/public/blog/index.html:223 +#: website/public/blog/index.html:237 +#: website/public/blog/reverse-engineering-models/index.html:876 #: website/public/blog/touhou-reconstruction/index.html:1073 #: website/public/examples/dino-lab/index.html:468 #: website/public/faq/index.html:566 #: website/public/first-investigation/index.html:516 #: website/public/get-started/index.html:875 -#: website/public/guides/browser/index.html:491 +#: website/public/guides/browser/index.html:500 #: website/public/guides/index.html:275 #: website/public/guides/javascript/index.html:496 #: website/public/guides/native/index.html:502 website/public/index.html:1184 @@ -11503,13 +11517,14 @@ msgctxt "shared" msgid "MIT license" msgstr "رخصة MIT" -#: website/public/blog/index.html:224 +#: website/public/blog/index.html:238 +#: website/public/blog/reverse-engineering-models/index.html:877 #: website/public/blog/touhou-reconstruction/index.html:1074 #: website/public/examples/dino-lab/index.html:469 #: website/public/faq/index.html:567 #: website/public/first-investigation/index.html:517 #: website/public/get-started/index.html:876 -#: website/public/guides/browser/index.html:492 +#: website/public/guides/browser/index.html:501 #: website/public/guides/index.html:276 #: website/public/guides/javascript/index.html:497 #: website/public/guides/native/index.html:503 website/public/index.html:1185 @@ -11523,13 +11538,14 @@ msgctxt "shared" msgid "Join the community" msgstr "انضم إلى المجتمع" -#: website/public/blog/index.html:225 +#: website/public/blog/index.html:239 +#: website/public/blog/reverse-engineering-models/index.html:878 #: website/public/blog/touhou-reconstruction/index.html:1075 #: website/public/examples/dino-lab/index.html:470 #: website/public/faq/index.html:568 #: website/public/first-investigation/index.html:518 #: website/public/get-started/index.html:877 -#: website/public/guides/browser/index.html:493 +#: website/public/guides/browser/index.html:502 #: website/public/guides/index.html:277 #: website/public/guides/javascript/index.html:498 #: website/public/guides/native/index.html:504 website/public/index.html:1186 @@ -12461,6 +12477,1335 @@ msgctxt "runtime" msgid "Switch to dark mode" msgstr "التبديل إلى الوضع الداكن" +#. {ph0}: +#. {ph1}: +#: website/public/guides/browser/index.html:480 +#, python-brace-format +msgctxt "page:guides/browser/index.html" +msgid "" +"For HTTP request and response bodies, use {ph0}Fiddler Everywhere: SAZ " +"captures, MCP and public skills{ph1}. Export SAZ or HAR 1.2 for offline REA " +"capture Evidence, or connect Everywhere's MCP server to inspect captured " +"traffic with your agent." +msgstr "" +"لفحص أجسام طلبات HTTP واستجاباتها، استخدم {ph0}Fiddler Everywhere: تسجيلات " +"SAZ وMCP والمهارات العامة{ph1}. صدّر بتنسيق SAZ ?? HAR 1.2 للحصول على أدلة " +"تسجيل في REA دون اتصال، أو اربط خادم MCP في Everywhere لفحص حركة المرور " +"المسجلة مع وكيلك." + +#. {ph0}:
+#: website/public/blog/index.html:200 +#, python-brace-format +msgctxt "page:blog/index.html" +msgid "Model specialization{ph0}Essay" +msgstr "نماذج متخصصة{ph0}مقال" + +#: website/public/blog/index.html:202 +msgctxt "page:blog/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "هل تحتاج الهندسة العكسية إلى نموذج أكبر؟" + +#: website/public/blog/index.html:203 +msgctxt "page:blog/index.html" +msgid "" +"What Touhou reconstruction suggests about general models, specialist models " +"and learning from compiler feedback." +msgstr "" +"أفكار من إعادة بناء Touhou حول النماذج العامة والمتخصصة والتعلم من نتائج " +"المُصرّف." + +#: website/public/blog/index.html:210 +msgctxt "page:blog/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "بواسطة N0zoM1z0 · 13 أكتوبر 2026" + +#: website/public/blog/reverse-engineering-models/index.html:103 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Skip to content" +msgstr "تخطي إلى المحتوى" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:194 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Home{ph1} / {ph2}Blog{ph3} / Models and RE" +msgstr "" +"{ph0}الصفحة الرئيسية{ph1} / {ph2}المدونة{ph3} / النماذج والهندسة العكسية" + +#: website/public/blog/reverse-engineering-models/index.html:197 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Essay" +msgstr "مقال" + +# Reviewed for technical meaning and readability; preserve the distinction +# between historical evidence and proposed experiments. +#: website/public/blog/reverse-engineering-models/index.html:198 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "هل تحتاج الهندسة العكسية إلى نموذج أكبر؟" + +#: website/public/blog/reverse-engineering-models/index.html:199 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "بواسطة N0zoM1z0 · 13 أكتوبر 2026" + +#: website/public/blog/reverse-engineering-models/index.html:201 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Takeaway" +msgstr "أهم الأفكار" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:203 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Model capability changes the hypotheses an agent can find.{ph1} Later " +"sessions have solved problems that earlier work left unresolved." +msgstr "" +"{ph0}تؤثر قدرة النموذج على الفرضيات التي يمكن أن يقترحها وكيل الذكاء " +"الاصطناعي.{ph1} قامت الجلسات اللاحقة بحل المشكلات التي تركها العمل السابق " +"دون حل." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:211 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A familiar compiler gives specialization something concrete to " +"learn.{ph1} Some exact matches depend on narrow, reusable source patterns." +msgstr "" +"{ph0}يقدم المترجم المألوف أنماطًا محددة لنموذج متخصص للتعلم.{ph1} تعتمد بعض " +"المطابقات الدقيقة لرمز الآلة على أنماط محددة من كود المصدر قابلة لإعادة " +"الاستخدام." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:218 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Give repository knowledge a fair comparison.{ph1} A smaller model with " +"good retrieval may already know enough to make progress." +msgstr "" +"{ph0}قم بتضمين معرفة المستودع في مقارنة عادلة.{ph1} قد يحتوي النموذج الأصغر " +"الذي يتمتع باسترجاع جيد للمعرفة على معلومات كافية لإحراز تقدم." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:223 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Train the next useful experiment, and test its oracle.{ph1} The value " +"of feedback depends on what the checker actually verifies." +msgstr "" +"{ph0}قم بتدريب النموذج على اختيار التجربة المفيدة التالية، واختبار أداة " +"التحقق الخاص به.{ph1} تكون التعليقات مفيدة فقط إلى الحد الذي يتحقق فيه " +"المدقق من الأشياء الصحيحة." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:230 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Measure accepted reconstruction against time and cost.{ph1} That is the" +" result I want from the whole agent." +msgstr "" +"{ph0}قم بتقييم نتائج إعادة بناء الشيفرة التي تم التحقق منها جنبًا إلى جنب مع" +" الوقت والتكلفة.{ph1} هذا ما أريد قياسه لسير عمل وكيل الذكاء الاصطناعي " +"الكامل." + +#: website/public/blog/reverse-engineering-models/index.html:241 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "In this article" +msgstr "في هذه المقالة" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:243 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What better models changed{ph1}" +msgstr "{ph0}ما الذي غيّرته النماذج الأقوى؟{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:244 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Why specialize?{ph1}" +msgstr "{ph0}لماذا التخصص؟{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:245 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Where experience lives{ph1}" +msgstr "{ph0}أين ينبغي تخزين هذه التجربة؟{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:246 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Learning the next experiment{ph1}" +msgstr "{ph0}تعلم التجربة المفيدة التالية{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:247 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}RE and reinforcement learning{ph1}" +msgstr "{ph0}الهندسة العكسية والتعلم المعزز{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:248 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}When the oracle is wrong{ph1}" +msgstr "{ph0}عندما يكون أداة التحقق خاطئًا{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:249 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}A comparison worth running{ph1}" +msgstr "{ph0}مقارنة تستحق التشغيل{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:250 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What kind of agent?{ph1}" +msgstr "{ph0}ما هو نوع وكيل الذكاء الاصطناعي الذي يشير إليه هذا؟{ph1}" + +#: website/public/blog/reverse-engineering-models/index.html:258 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What better models changed" +msgstr "ما الذي غيّرته النماذج الأقوى؟" + +#: website/public/blog/reverse-engineering-models/index.html:259 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Since August 2026, I have been exploring how much reverse engineering an " +"agent can carry out with access to a program and a way to check its work. " +"Most of my experience comes from reconstructing games, especially Touhou. As" +" I have changed models, the repositories have kept a record of what each " +"session managed to recover." +msgstr "" +"منذ أغسطس 2026، كنت أستكشف مقدار الهندسة العكسية التي يمكن للوكيل تنفيذها من" +" خلال الوصول إلى البرنامج وطريقة التحقق من عمله. معظم خبرتي تأتي من إعادة " +"بناء الألعاب، وخاصة Touhou. وبما أنني قمت بتغيير النماذج، فقد احتفظت " +"المستودعات بسجل لما تمكنت كل جلسة من استعادته." + +#: website/public/blog/reverse-engineering-models/index.html:267 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of that work aims at exact reconstruction: we compile the maintained " +"source under a pinned historical toolchain and check whether it reproduces " +"the target function. When a later model succeeds where earlier work stalled," +" the source changes show what made that progress possible." +msgstr "" +"يهدف كثير من هذا العمل إلى إعادة بناء دقيقة: نترجم الشيفرة المصدرية التي " +"نحتفظ بها باستخدام أدوات من تلك الحقبة بإصدارات ثابتة، ثم نتحقق من أنها تعيد" +" إنتاج الدالة المستهدفة. عندما يحل نموذج أحدث مشكلة تعثر عندها العمل السابق،" +" تكشف تغييرات الشيفرة ما الذي أتاح هذا التقدم." + +#: website/public/blog/reverse-engineering-models/index.html:274 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The temporary values had to stay alive together" +msgstr "تحتاج المتغيرات المؤقتة إلى عمر متداخل" + +#: website/public/blog/reverse-engineering-models/index.html:275 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of a TH09 attack function's logic had already been reconstructed, but " +"the compiler still stored temporary results differently from the original. " +"The remaining problem was how long those results needed to stay alive and " +"when their storage could be reused." +msgstr "" +"تمت إعادة بناء الكثير من منطق دالة الهجوم TH09 بالفعل، لكن المترجم ما زال " +"يخزن النتائج المؤقتة بشكل مختلف عن الأصل. وكانت المشكلة المتبقية هي المدة " +"التي تحتاجها تلك النتائج للبقاء على قيد الحياة، ومتى يمكن إعادة استخدام " +"مخزونها." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:282 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A {ph0}GPT-6-Sol checkpoint{ph1} recorded this mismatch after testing " +"several ordinary source arrangements. When {ph2}GPT-6.1-Sol returned to the " +"problem{ph3}, it found a useful hypothesis about how long those temporary " +"values needed to stay alive." +msgstr "" +"أبلغ {ph0}GPT-6-Sol سجل الجلسة{ph1} عن عدم التطابق هذا بعد اختبار العديد من " +"ترتيبات كود المصدر العادية. عندما فحص {ph2}GPT-6.1-Sol المشكلة مرة " +"أخرى{ph3}، اقترح فرضية مفيدة حول عمر هذه المتغيرات المؤقتة." + +#: website/public/blog/reverse-engineering-models/index.html:296 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The function produced temporary results and copied them into animation " +"positions. In C++, a temporary result stays alive until the expression using" +" it finishes. Grouping the real operations into one expression kept those " +"results alive together, so the compiler could no longer reuse the same " +"storage for all of them." +msgstr "" +"أنتجت الوظيفة نتائج مؤقتة ونسختها في مواضع الرسوم المتحركة. في لغة C++، تظل " +"النتيجة المؤقتة حية حتى انتهاء التعبير الذي يستخدمها. يؤدي تجميع العمليات " +"الحقيقية في تعبير واحد إلى إبقاء هذه النتائج حية معًا، لذلك لم يعد بإمكان " +"المترجم إعادة استخدام نفس وحدة التخزين لجميعها." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:303 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"One result still shared storage with a value used later for collision " +"checks. Giving that value its own lifetime resolved the remaining overlap. " +"After several targeted experiments, {ph0}a clean rebuild confirmed that the " +"full function matched{ph1}." +msgstr "" +"لا تزال إحدى النتائج مشتركة في مساحة التخزين مع القيمة المستخدمة لاحقًا " +"للتحقق من التضارب. إن إعطاء هذه القيمة مدى الحياة الخاص بها أدى إلى حل " +"التداخل المتبقي. وبعد عدة تجارب مستهدفة، {ph0}أكدت إعادة الترجمة النظيفة من " +"المصدر أن الوظيفة الكاملة مطابقة{ph1}." + +#: website/public/blog/reverse-engineering-models/index.html:312 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What interests me is how the discovery and the tests worked together. The " +"lifetime hypothesis gave the agent a new direction; compiler feedback helped" +" it refine that hypothesis into a source arrangement that matched." +msgstr "" +"أنا مهتم بكيفية دعم الاكتشاف والاختبارات لبعضهما البعض. أعطت الفرضية " +"المتعلقة بعمر الكائنات المؤقتة لعامل الذكاء الاصطناعي اتجاهًا جديدًا. ساعدت " +"تعليقات المترجم في تحسين هذه الفرضية إلى ترتيب كود المصدر المطابق." + +#: website/public/blog/reverse-engineering-models/index.html:318 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A different control-flow graph" +msgstr "رسم بياني مختلف للتحكم في التدفق" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:319 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A second TH09 function shows why that new direction can matter. Here, the " +"obstacle was how several branches shared their final work. " +"{ph0}GPT-6.1-Sol's earlier work{ph1} got parts of the function to match, but" +" the branches still joined up differently from the original." +msgstr "" +"توضح وظيفة TH09 الثانية سبب أهمية هذا الاتجاه الجديد. وهنا، كانت العقبة هي " +"كيفية مشاركة العديد من الفروع لعملها النهائي. يحتوي العمل السابق لـ " +"{ph0}GPT-6.1-Sol{ph1} على أجزاء من الدالة متطابقة، ولكن لا تزال الفروع متصلة" +" بشكل مختلف عن الأصل." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:330 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}GPT-6-Astra changed the surrounding control-flow graph{ph1}. The main " +"paths now led to a shared final update and return. In that source structure," +" the compiler combined the branches in the same way as the original. The " +"complete function matched, and a {ph2}separate rebuild check confirmed " +"it{ph3}." +msgstr "" +"{ph0}غيّر GPT-6-Astra الرسم البياني المحيط لتدفق التحكم{ph1}. أصبحت المسارات" +" الرئيسية تؤدي إلى تحديث أخير وعودة مشتركين. بهذه البنية، دمج المترجم " +"البرمجي الفروع كما في الأصل. تطابقت الدالة كاملة، وأكد ذلك {ph2}فحص مستقل " +"لإعادة البناء{ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:342 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Changing the representation opened a more productive search space.{ph1}" +" Both cases give me a reason to take general model capability seriously: an " +"agent can keep trying local variations for a long time before finding a " +"better way to express the surrounding work." +msgstr "" +"{ph0}أدى تغيير التمثيل إلى فتح مساحة بحث أكثر إنتاجية.{ph1} تمنحني كلتا " +"الحالتين سببًا لأخذ إمكانات النموذج العامة على محمل الجد: يمكن للوكيل " +"الاستمرار في تجربة الاختلافات المحلية لفترة طويلة قبل العثور على طريقة أفضل " +"للتعبير عن العمل المحيط." + +#: website/public/blog/reverse-engineering-models/index.html:351 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"At the same time, the later sessions started with earlier source and " +"recorded failures. That accumulated work was part of their advantage. The " +"history lets us inspect the discoveries and their checks; a controlled " +"comparison would help separate the effect of the model from the experience " +"it inherited." +msgstr "" +"في الوقت نفسه، بدأت الجلسات اللاحقة بمصادر سابقة وإخفاقات مسجلة. وكان هذا " +"العمل المتراكم جزءًا من مصلحتهم. يتيح لنا التاريخ فحص الاكتشافات وفحصها؛ ومن" +" شأن المقارنة الخاضعة للرقابة أن تساعد في فصل تأثير النموذج عن الخبرة التي " +"ورثها." + +#: website/public/blog/reverse-engineering-models/index.html:358 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is where the question becomes interesting to me. Some of the progress " +"came from finding a new representation, while much of the remaining work " +"involved testing source forms against a familiar compiler. How much needs " +"broad reasoning ability, and how much could a model learn through repeated " +"experience with that compiler and code?" +msgstr "" +"وهذا ما يجعل السؤال مثيرًا للاهتمام بالنسبة لي. جاء بعض التقدم من إيجاد " +"تمثيل جديد. تضمن الكثير من العمل المتبقي اختبار ترتيبات مختلفة لشفرة المصدر " +"مقابل مترجم مألوف. كم من هذا يتطلب قدرة تفكير واسعة النطاق؟ ما مقدار ما يمكن" +" أن يتعلمه النموذج من خلال العمل المتكرر مع هذا المترجم والكود؟" + +#: website/public/blog/reverse-engineering-models/index.html:368 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why might a specialist do well?" +msgstr "لماذا قد يكون نموذج الذكاء الاصطناعي المتخصص ناجحًا؟" + +#: website/public/blog/reverse-engineering-models/index.html:369 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Touhou gives us a concrete reason to expect that experience to help. Much of" +" the game code comes from the same author, ZUN, so recurring habits are a " +"reasonable starting assumption. The repositories back that up with useful " +"examples: related loops and helper arrangements have supplied hypotheses " +"across games." +msgstr "" +"Touhou يعطينا سببًا ملموسًا لتوقع أن تساعدنا هذه التجربة. الكثير من رموز " +"اللعبة تأتي من نفس المؤلف، ZUN، لذا فإن العادات المتكررة هي افتراض بداية " +"معقول. تدعم المستودعات ذلك بأمثلة مفيدة: الحلقات ذات الصلة والترتيبات " +"المساعدة قدمت فرضيات عبر الألعاب." + +#: website/public/blog/reverse-engineering-models/index.html:376 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those source patterns become more specific when we include the compiler. " +"Under a particular version and set of options, small changes in source can " +"produce repeatable differences in machine code. Some of those differences " +"are surprisingly obscure." +msgstr "" +"تصبح أنماط المصدر هذه أكثر تحديدًا عندما نقوم بتضمين المترجم. ضمن إصدار معين" +" ومجموعة من الخيارات، يمكن أن تؤدي التغييرات الصغيرة في المصدر إلى اختلافات " +"متكررة في كود الجهاز. بعض هذه الاختلافات غامضة بشكل مدهش." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: Supervisor::LoadDat +#. {ph4}: +#: website/public/blog/reverse-engineering-models/index.html:382 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In TH095, for example, {ph0}even the name used for a local variable could " +"change where VC7.1 stored it{ph1}. Moving the declaration alone did not " +"solve the layout, but an earlier background function had revealed a naming " +"pattern that affected the compiler's choice. That gave us a more promising " +"hypothesis for {ph2}the reconstruction of {ph3}{ph4}." +msgstr "" +"في TH095، على سبيل المثال، {ph0}حتى اسم المتغير المحلي يمكن أن يؤثر على مكان" +" تخزينه VC7.1{ph1}. لم يؤدي نقل إعلانه إلى حل تخطيط التخزين. ومع ذلك، كشفت " +"وظيفة خلفية سابقة عن نمط تسمية أثر على اختيار المترجم. وقد اقترح ذلك فرضية " +"واعدة أكثر بشأن {ph2}إعادة بناء {ph3}{ph4}." + +#: website/public/blog/reverse-engineering-models/index.html:396 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reusing that pattern for the file-size variable produced the required layout" +" without adding storage. The full function then matched." +msgstr "" +"أدت إعادة استخدام هذا النمط لمتغير حجم الملف إلى إنتاج التخطيط المطلوب دون " +"إضافة مساحة تخزين. ثم تمت مطابقة الوظيفة الكاملة." + +#: website/public/blog/reverse-engineering-models/index.html:401 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is the sort of narrow detail a general model may not know when it " +"arrives. A specialist trained on this compiler could encounter many such " +"cases and learn which source variations are worth testing first." +msgstr "" +"هذه تفاصيل محددة جدًا قد لا يعرفها نموذج الذكاء الاصطناعي للأغراض العامة " +"عندما يواجه المشكلة لأول مرة. يمكن للنموذج المتخصص لهذا المترجم أن يرى " +"العديد من هذه الحالات ويتعرف على اختلافات التعليمات البرمجية المصدر التي " +"تستحق الاختبار أولاً." + +#: website/public/blog/reverse-engineering-models/index.html:407 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The opportunity is especially clear in final matching problems where we " +"already understand the behavior. We are then asking which ordinary source " +"form this compiler turns into the observed instructions. Specialization " +"could help the model propose those forms more efficiently, while the target " +"still determines which hypotheses survive." +msgstr "" +"تكون الفرصة واضحة بشكل خاص في مشاكل المطابقة النهائية حيث نفهم السلوك " +"بالفعل. ثم نسأل عن ترتيب التعليمات البرمجية المصدر العادي الذي يحوله هذا " +"المترجم إلى التعليمات المرصودة. يمكن أن يساعد التخصص النموذج على اقتراح تلك " +"الأشكال بشكل أكثر كفاءة، في حين أن الهدف لا يزال يحدد الفرضيات التي ستبقى." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:415 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The useful patterns would have to retain their context, though. TH08 uses an" +" earlier Visual C++ version than TH095 and TH09. TH04 uses a different DOS-" +"era compiler, and its {ph0}player reconstruction{ph1} needed different " +"hypotheses about source patterns and how the compiled code was combined into" +" an executable. Shared authorship gives us a prior, but each target tells us" +" whether it applies. Distinguishing those contexts is part of what a " +"specialist would need to learn." +msgstr "" +"يجب أن تحتفظ الأنماط بسياقها. يستخدم TH08 إصدارًا من Visual C++ أقدم مما " +"يستخدمه TH095 وTH09. أما TH04 فيستخدم مُترجمًا برمجيًا آخر من حقبة DOS، وقد " +"احتاجت {ph0}إعادة بناء شيفرة اللاعب{ph1} إلى فرضيات مختلفة عن أنماط الشيفرة " +"وكيفية ربطها في ملف تنفيذي. وحدة المؤلف تعطينا توقعًا أوليًا، لكن كل هدف " +"يبين إن كان ينطبق. وعلى النموذج المتخصص تعلم هذه الفروق." + +#: website/public/blog/reverse-engineering-models/index.html:431 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Where should that experience live?" +msgstr "أين ينبغي تخزين هذه التجربة؟" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:432 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before deciding to teach those patterns through model training, I would look" +" at the knowledge we already keep in the repository. In the {ph0}previous " +"essay{ph1}, I wrote about it becoming the project's working memory. That " +"memory includes compiler lessons, along with the source and failed " +"experiments that made them useful." +msgstr "" +"قبل أن أقرر تعليم هذه الأنماط للنموذج، سأراجع المعرفة التي نحفظها بالفعل في " +"المستودع. في {ph0}المقال السابق{ph1} كتبت عن المستودع وهو يصبح الذاكرة " +"العاملة للمشروع. تتضمن هذه الذاكرة ما تعلمناه عن سلوك المترجم البرمجي، مع " +"الشيفرة والتجارب الفاشلة التي جعلت تلك المعرفة مفيدة." + +#: website/public/blog/reverse-engineering-models/index.html:441 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"By retrieving that record, an agent can start with a known result and " +"inspect how we reached it. It can also avoid alternatives that have already " +"failed in the same context. The starting point improves without changing the" +" model's weights." +msgstr "" +"من خلال استرداد هذا السجل، يمكن للوكيل البدء بنتيجة معروفة وفحص كيفية وصولنا" +" إليها. ويمكنه أيضًا تجنب البدائل التي فشلت بالفعل في نفس السياق. تتحسن نقطة" +" البداية دون تغيير معلمات النموذج المدربة." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:447 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"When reconstructing TH09's enemy-spawn function, a {ph0}GPT-6-Luna-max " +"session{ph1} drew a hypothesis from TH08's corresponding source family. That" +" earlier source suggested how to structure the loop and handle failures " +"within it. TH09's own evidence supplied the game-specific details, and clean" +" rebuilds confirmed the complete match." +msgstr "" +"أثناء إعادة بناء وظيفة TH09 لإنشاء الأعداء، أخذت جلسة {ph0}GPT-6-Luna-" +"max{ph1} فرضية من كود مصدر TH08 ذي الصلة. اقترح هذا الكود السابق كيفية تنظيم" +" الحلقة والتعامل مع حالات الفشل بداخلها. قدمت أدلة TH09 الخاصة التفاصيل " +"الخاصة باللعبة. أكدت عمليات إعادة البناء من المصدر أن الوظيفة الكاملة " +"متطابقة." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:458 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}\"The model did not know that compiler quirk\" and \"the model could " +"not use it\" are different problems.{ph1} Retrieval may solve the first, but" +" turning that fact into a useful candidate still takes work. And if the " +"candidate fails, the model must decide what the result means for its next " +"attempt. Knowing the relevant pattern is only the beginning of that process." +msgstr "" +"{ph0}\"النموذج لم يكن يعرف هذه الخاصية المحددة للمترجم البرمجي\" و\"النموذج " +"لم يستطع استخدامها\" مشكلتان مختلفتان.{ph1} قد يحل استرجاع المعرفة المشكلة " +"الأولى، لكن تحويلها إلى شيفرة مرشحة مفيدة يحتاج إلى عمل. وإذا فشلت الشيفرة " +"المرشحة، فعلى النموذج أن يقرر ما تعنيه النتيجة لمحاولته التالية. معرفة النمط" +" المناسب هي بداية العملية فقط." + +#: website/public/blog/reverse-engineering-models/index.html:468 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For that reason, I would include a smaller general model with good retrieval" +" in any comparison. Giving it the same library of experience would help us " +"identify what training adds: better candidate generation, better decisions " +"after feedback, or both." +msgstr "" +"لهذا السبب، سأقوم بتضمين نموذج ذكاء اصطناعي أصغر حجمًا للأغراض العامة مع " +"استرجاع جيد في أي مقارنة. إن منحها نفس مكتبة الخبرة من شأنه أن يساعدنا في " +"تحديد ما يضيفه التدريب: جيل أفضل للمرشحين، أو قرارات أفضل بعد التعليقات، أو " +"كليهما." + +#: website/public/blog/reverse-engineering-models/index.html:476 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Learning the next useful experiment" +msgstr "تعلم التجربة المفيدة التالية" + +#: website/public/blog/reverse-engineering-models/index.html:477 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those decisions after feedback are what I would most like to study. In " +"agentic reconstruction, the model proposes an explanation and tests it " +"against the program. Even when the candidate fails, the result can narrow " +"the remaining possibilities and suggest a more useful next experiment." +msgstr "" +"تلك القرارات بعد ردود الفعل هي أكثر ما أرغب في دراسته. في إعادة البناء " +"الوكيل، يقترح النموذج تفسيرًا ويختبره مقابل البرنامج. وحتى عندما يفشل الكود " +"المصدري المرشح، فإن النتيجة يمكن أن تضيق الاحتمالات المتبقية وتقترح تجربة " +"تالية أكثر فائدة." + +#: website/public/blog/reverse-engineering-models/index.html:484 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Consider the temporary-results example again. Its final source shows the " +"arrangement that worked, but the route to that source contains more " +"information. Each of the following experiments helped identify what still " +"needed to change." +msgstr "" +"خذ بعين الاعتبار مثال النتائج المؤقتة مرة أخرى. يُظهر مصدره النهائي الترتيب " +"الذي نجح، لكن المسار إلى هذا المصدر يحتوي على مزيد من المعلومات. ساعدت كل " +"تجربة من التجارب التالية في تحديد ما لا يزال بحاجة إلى التغيير." + +#: website/public/blog/reverse-engineering-models/index.html:497 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Experiment" +msgstr "تجربة" + +#: website/public/blog/reverse-engineering-models/index.html:498 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What its result taught us" +msgstr "ماذا علمتنا نتيجته" + +#: website/public/blog/reverse-engineering-models/index.html:503 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Try a different way to copy the results" +msgstr "جرب طريقة مختلفة لنسخ النتائج" + +#: website/public/blog/reverse-engineering-models/index.html:504 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The copy code changes, but the function still differs." +msgstr "يتغير رمز النسخ، لكن الوظيفة لا تزال مختلفة." + +#: website/public/blog/reverse-engineering-models/index.html:509 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Use the returned values directly" +msgstr "استخدم القيم التي تم إرجاعها مباشرة" + +#: website/public/blog/reverse-engineering-models/index.html:510 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Copying follows the original pattern, but temporary storage still differs." +msgstr "يتبع النسخ النمط الأصلي، لكن التخزين المؤقت لا يزال مختلفًا." + +#: website/public/blog/reverse-engineering-models/index.html:516 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the temporary results alive together" +msgstr "إبقاء كافة المتغيرات المؤقتة صالحة في نفس الوقت" + +#: website/public/blog/reverse-engineering-models/index.html:517 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"They get separate storage, but one still overlaps a value used later for " +"collision checks." +msgstr "" +"يحصلون على مساحة تخزين منفصلة، ولكن لا يزال هناك تداخل مع قيمة يتم استخدامها" +" لاحقًا للتحقق من التصادم." + +#: website/public/blog/reverse-engineering-models/index.html:523 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the collision-check value separate" +msgstr "احتفظ بقيمة التحقق من التصادم منفصلة" + +#: website/public/blog/reverse-engineering-models/index.html:524 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A clean rebuild confirms the complete match." +msgstr "تؤكد إعادة الترجمة النظيفة من المصدر التطابق الكامل." + +#: website/public/blog/reverse-engineering-models/index.html:529 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Selected experiments from the attack-function reconstruction above, grouped " +"by what they taught us." +msgstr "" +"تجارب مختارة من إعادة بناء وظيفة الهجوم أعلاه، مجمعة حسب ما علمتنا إياه." + +#: website/public/blog/reverse-engineering-models/index.html:533 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A training record that keeps this sequence can teach more than the final " +"source alone. It connects each proposed change to an observed result and the" +" next decision. Keeping ineffective variations matters too: they tell a " +"later agent which hypotheses have already been tested in that context." +msgstr "" +"يمكن لسجل التدريب الذي يحتفظ بهذا التسلسل أن يعلم أكثر من المصدر النهائي " +"وحده. فهو يربط كل تغيير مقترح بالنتيجة الملحوظة والقرار التالي. إن الاحتفاظ " +"بالتغيرات غير الفعالة أمر مهم أيضًا: فهي تخبر وكيلًا لاحقًا بالفرضيات التي " +"تم اختبارها بالفعل في هذا السياق." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:540 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}I would want the model to learn how to use feedback.{ph1} A neutral " +"result is a reason to reconsider the variation; a changed mismatch may point" +" to another cause. When local edits stop being informative, the agent should" +" be able to revisit the source representation or inspect the evidence again." +msgstr "" +"{ph0}أريد أن يتعلم النموذج كيفية استخدام التعليقات.{ph1} النتيجة المحايدة هي" +" سبب لإعادة النظر في الشكل؛ قد يشير عدم التطابق المتغير إلى سبب آخر. عندما " +"تتوقف التعديلات المحلية عن تقديم المعلومات، يجب أن يكون الوكيل قادرًا على " +"إعادة النظر في تمثيل المصدر أو فحص الأدلة مرة أخرى." + +#: website/public/blog/reverse-engineering-models/index.html:549 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Learning to make those choices requires autonomy. Sometimes the useful next " +"step is inspecting another caller rather than compiling another edit. A " +"rigid sequence of required rewrites would remove that choice, along with " +"part of the behavior we want the model to learn." +msgstr "" +"تعلم هذه الاختيارات يتطلب الاستقلالية. أحيانًا تكون الخطوة الأنسب فحص دالة " +"أخرى تستدعي هذه الدالة بدل ترجمة تعديل جديد. فرض سلسلة جامدة من إعادة " +"الكتابة يلغي هذا الاختيار ومعه جزءًا من السلوك الذي نريد أن يتعلمه النموذج." + +#: website/public/blog/reverse-engineering-models/index.html:558 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why RE is interesting for reinforcement learning" +msgstr "لماذا تعتبر الهندسة العكسية مثيرة للاهتمام للتعلم المعزز" + +#: website/public/blog/reverse-engineering-models/index.html:559 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Bounded reconstruction gives us a practical environment for studying these " +"choices. Once the target and compiler are pinned, many source experiments " +"produce quick, repeatable feedback. The agent can compile a candidate, " +"inspect the mismatch and use that observation to choose its next step." +msgstr "" +"توفر إعادة البناء ذات النطاق المحدد بيئة عملية لدراسة هذه القرارات. بعد " +"تثبيت إصدارات الهدف والمترجم البرمجي، تعطي تجارب كثيرة على الشيفرة ملاحظات " +"سريعة وقابلة للتكرار. يستطيع الوكيل ترجمة شيفرة مرشحة وفحص الفروق ثم اختيار " +"الخطوة التالية." + +#: website/public/blog/reverse-engineering-models/index.html:566 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reinforcement learning could train that decision-making process through the " +"feedback it receives. A candidate accepted by a defined check gives us a " +"concrete outcome, while the sequence of attempts lets us observe how much " +"work it took to reach it." +msgstr "" +"يمكن للتعلم المعزز تدريب عملية صنع القرار من خلال ردود الفعل التي تتلقاها. " +"يمنحنا المرشح الذي يتم قبوله عن طريق فحص محدد نتيجة ملموسة، في حين يتيح لنا " +"تسلسل المحاولات ملاحظة مقدار العمل الذي استغرقه الوصول إلى هذه النتيجة." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:572 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"There is already research supporting this direction. {ph0}LLM4Decompile{ph1}" +" studies models specialized for decompilation. {ph2}AutoDecompiler{ph3} " +"trains multi-turn refinement using compiler and execution feedback. Their " +"studies focus on functional recovery. For this work, I would use them as a " +"starting point for the related task of finding source that matches a " +"historical compiler's output." +msgstr "" +"هناك أبحاث تدعم هذا الاتجاه بالفعل. يدرس {ph0}LLM4Decompile{ph1} نماذج " +"متخصصة في استعادة الشيفرة المصدرية من البرامج المترجمة. ويستخدم " +"{ph2}AutoDecompiler{ph3} نتائج الترجمة والتنفيذ لتدريب التحسين عبر جولات " +"متعددة. تركز هذه الدراسات على استعادة سلوك البرنامج. سأستخدمها نقطة بداية " +"لمهمة قريبة: إيجاد شيفرة تنتج مخرجات مطابقة لمترجم برمجي تاريخي." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:584 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To reach such a well-defined task, though, we may first need to establish " +"what a function owns or devise a new runtime measurement. In TH08, for " +"example, a {ph0}collision difference led back to how the original program " +"handled floating-point calculations{ph1}. Before the agent could test a " +"useful correction, it had to work out what to measure." +msgstr "" +"للوصول إلى مثل هذه المهمة المحددة جيدًا، قد نحتاج أولاً إلى تحديد الموارد " +"التي تمتلكها الوظيفة أو ابتكار طريقة جديدة لملاحظة سلوك البرنامج أثناء " +"تشغيله. في TH08، على سبيل المثال، {ph0}أدى اختلاف التصادم إلى كيفية تعامل " +"البرنامج الأصلي مع حسابات الفاصلة العائمة{ph1}. قبل أن يتمكن الوكيل من " +"اختبار تصحيح مفيد، كان عليه أن يحدد ما يجب قياسه." + +#: website/public/blog/reverse-engineering-models/index.html:595 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This suggests another role for a broadly capable model: helping make the " +"problem checkable in the first place. Once that check exists, we can " +"investigate how far a learned search policy can get with it." +msgstr "" +"ويشير هذا إلى دور آخر لنموذج قادر على نطاق واسع: المساعدة في جعل المشكلة " +"قابلة للتحقق في المقام الأول. بمجرد وجود هذا الفحص، يمكننا التحقيق في المدى " +"الذي يمكن أن تصل إليه سياسة البحث المستفادة." + +#: website/public/blog/reverse-engineering-models/index.html:603 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "When the oracle is wrong" +msgstr "عندما يكون أداة التحقق خاطئًا" + +#: website/public/blog/reverse-engineering-models/index.html:604 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Once we train against that check, its coverage becomes part of what the " +"model learns. I call the reference or checker an oracle. Even if it gives a " +"repeatable answer, we still need to understand what that answer covers." +msgstr "" +"عندما نستخدم فحصًا في التدريب، يصبح نطاقه جزءًا من بيئة التعلم. أسمي هذا " +"المرجع أو أداة التحقق oracle. حتى الإجابة القابلة للتكرار يجب فهمها في حدود " +"ما تتحقق منه فعلًا." + +#: website/public/blog/reverse-engineering-models/index.html:610 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"TH08 showed how much can hide outside that coverage. Some reconstructed " +"functions passed their exact comparisons with wrong floating constants. The " +"comparator adjusted relocated addresses before checking instructions, but it" +" checked the values behind those addresses only when they had been " +"explicitly recorded for verification. Other literal references could pass " +"without their values being examined." +msgstr "" +"أظهر TH08 مقدار ما يمكن إخفاءه خارج تلك التغطية. اجتازت بعض الوظائف المعاد " +"بناؤها مقارناتها الدقيقة مع ثوابت عائمة خاطئة. قامت أداة المقارنة بتعديل " +"العناوين المنقولة قبل التحقق من التعليمات، ولكنها لم تتحقق من القيم الموجودة" +" خلف تلك العناوين إلا عندما تم تسجيلها بشكل صريح للتحقق منها. يمكن أن تمر " +"المراجع الحرفية الأخرى دون فحص قيمها." + +#. {ph0}: +#. {ph1}: +#. {ph2}: 1.0f +#. {ph3}: 32.0f +#: website/public/blog/reverse-engineering-models/index.html:619 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To close that gap, a {ph0}wider audit{ph1} found incorrect constants in " +"functions that had already been accepted. In one calculation, the " +"reconstructed source added {ph2} where the original added {ph3}." +msgstr "" +"لسد هذه الفجوة، كشف {ph0}تدقيق أوسع{ph1} ثوابت خاطئة في دوال كانت قد اجتازت " +"التحقق. في إحدى العمليات الحسابية، أضافت الشيفرة المعاد بناؤها {ph2}، بينما " +"أضاف الأصل {ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:629 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A learner rewarded by the old check could learn to produce the wrong " +"reconstruction.{ph1} We had not trained an RL model against this oracle; the" +" observed failure was in the check itself. But using that check as a reward " +"would have made the same blind spot part of the learning environment." +msgstr "" +"{ph0}يمكن للنموذج الذي تمت مكافأته بفحص التحقق القديم أن يتعلم إنتاج إعادة " +"بناء غير صحيحة.{ph1} لم نقوم بتدريب نموذج RL باستخدام هذا المدقق. الفشل الذي" +" لاحظناه كان في الفحص نفسه. ولكن إذا استخدمناها كمكافأة، فإن نفس النقطة " +"العمياء ستصبح جزءًا من بيئة التعلم." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:639 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A functional test can have a similar blind spot. {ph0}Differential-fuzzing " +"research on LLM decompilers{ph1} has found behavior differences beyond the " +"supplied tests. Passing those tests therefore needs to be read in terms of " +"the cases they exercise, just as an instruction match needs to be read in " +"terms of the bytes and data it checks." +msgstr "" +"يمكن أن يحتوي الاختبار الوظيفي على نقطة عمياء مماثلة. {ph0}وجدت الأبحاث التي" +" تستخدم التشويش التفاضلي في برامج فك التشفير LLM{ph1} وجود اختلافات سلوكية " +"خارج الاختبارات المتوفرة. لذلك نحن بحاجة إلى فهم اختبار النجاح من حيث " +"الحالات التي يمارسها. وبالمثل، يجب فهم مطابقة التعليمات من حيث البايتات " +"والبيانات التي تتحقق منها." + +#: website/public/blog/reverse-engineering-models/index.html:649 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before using either kind of check as a reward, I would test it with " +"deliberate failures. I would also keep final acceptance outside the " +"candidate's editable source, so the model has to satisfy the check through " +"its reconstruction. Otherwise, something like a byte array copied from the " +"target could satisfy a naive score without recovering useful source." +msgstr "" +"قبل استخدام أي نوع من اختبارات التحقق كمكافأة، سأختبره بنتائج غير صحيحة " +"عمدًا. أود أيضًا أن أبقي القبول النهائي منفصلاً عن كود المصدر الذي يمكن " +"لعامل الذكاء الاصطناعي تحريره. يجب أن يجتاز النموذج التحقق من خلال إعادة " +"بناء البرنامج بشكل صحيح. وبخلاف ذلك، فإن مجرد نسخ مصفوفة بايت من الهدف يمكن " +"أن يخدع نظام تسجيل ساذج دون استعادة كود المصدر المفيد." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:657 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Improving the learner and improving its oracle belong to the same " +"research problem.{ph1} As the learner gets better at optimizing the feedback" +" we give it, understanding that feedback becomes more important too." +msgstr "" +"{ph0}إن تحسين المتعلم وتحسين أداة التحقق من الصحة ينتميان إلى نفس مشكلة " +"البحث.{ph1} ومع تحسن المتعلم في تحسين التعليقات التي نقدمها له، يصبح فهم هذه" +" التعليقات أكثر أهمية أيضًا." + +#: website/public/blog/reverse-engineering-models/index.html:667 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A comparison worth running" +msgstr "مقارنة تستحق التشغيل" + +#: website/public/blog/reverse-engineering-models/index.html:668 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"With those checks in place, I would start the comparison on a fixed VC7.1 " +"task family. First, I would establish baselines for a strong general model " +"and a smaller one. Adding repository knowledge to the smaller model would " +"show what retrieval changes. From there, supervised specialization and an " +"RL-trained version would let us examine what training adds." +msgstr "" +"مع إجراء عمليات التحقق هذه، سأبدأ المقارنة على مجموعة مهام VC7.1 ثابتة. " +"أولاً، أود أن أضع خطوط أساس لنموذج قوي للذكاء الاصطناعي للأغراض العامة " +"ونموذج أصغر. إن إضافة معرفة المستودع إلى النموذج الأصغر سيُظهر تغييرات " +"الاسترجاع. ومن هنا، فإن التخصص الخاضع للإشراف والنسخة المدربة RL ستسمح لنا " +"بفحص ما يضيفه التدريب." + +#: website/public/blog/reverse-engineering-models/index.html:676 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To make those comparisons informative, each model would receive the same " +"target evidence and tools. For the training-specific comparisons, I would " +"keep the base model and available knowledge fixed as well. Repeated trials " +"would help distinguish a reliable search strategy from a lucky candidate." +msgstr "" +"ولجعل هذه المقارنات مفيدة، سيحصل كل نموذج على نفس الأدلة والأدوات المستهدفة." +" بالنسبة للمقارنات الخاصة بالتدريب، سأبقي النموذج الأساسي والمعرفة المتاحة " +"ثابتة أيضًا. من شأن التجارب المتكررة أن تساعد في التمييز بين استراتيجية " +"البحث الموثوقة والحل المرشح الذي ينجح بالصدفة." + +#: website/public/blog/reverse-engineering-models/index.html:683 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would then separate two starting conditions: reconstructing a function " +"from its instructions, and closing the last few differences in maintained " +"source. A specialist might be excellent at the second task while a general " +"model does much of the work needed to reach it. Measuring them separately " +"would reveal that division of work." +msgstr "" +"سأفصل بعد ذلك شرطين للبدء: إعادة بناء دالة من تعليماتها، وإغلاق الاختلافات " +"القليلة الأخيرة في المصدر المحفوظ. قد يكون نموذج الذكاء الاصطناعي المتخصص " +"ممتازًا في المهمة الثانية، بينما يقوم نموذج الذكاء الاصطناعي للأغراض العامة " +"بالكثير من العمل اللازم للوصول إليه. وقياسها بشكل منفصل من شأنه أن يكشف عن " +"تقسيم العمل هذا." + +#: website/public/blog/reverse-engineering-models/index.html:691 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both tasks need genuinely held-out examples. Because related Touhou games " +"share engine code, holding out a game's name would still allow nearly " +"identical functions to appear in training. I would instead hold out source " +"families and keep their accepted answers out of both training and retrieval." +msgstr "" +"تحتاج كلتا المهمتين إلى أمثلة اختبارية مستثناة من التدريب والاسترجاع. ألعاب " +"Touhou ذات الصلة تشترك في رمز المحرك. قد يؤدي استبعاد لعبة واحدة من التدريب " +"إلى ترك وظائف متطابقة تقريبًا في مجموعة التدريب. بدلاً من ذلك، سأستبعد " +"مجموعات كاملة من أكواد المصدر ذات الصلة وأحتفظ بإجاباتها التي تم التحقق منها" +" خارج كل من بيانات التدريب ومكتبة الاسترجاع." + +#: website/public/blog/reverse-engineering-models/index.html:698 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"On those examples, the outcome would be complete, independently validated " +"matches. Giving each model the same candidate or compile budget would help " +"compare search quality. Measuring elapsed time and total cost, including " +"compilation, would tell us whether that quality produces an efficient " +"workflow. Training costs would need their own accounting too." +msgstr "" +"بالنسبة لهذه الأمثلة، ستكون النتيجة تطابقات كاملة تم التحقق منها بشكل مستقل." +" من شأن الميزانيات المتساوية لمقترحات المرشحين أو محاولات التجميع أن تساعد " +"في مقارنة جودة البحث. إن قياس إجمالي الوقت والتكلفة، بما في ذلك التجميع، " +"سيُظهر ما إذا كانت هذه الجودة تنتج سير عمل فعالاً. وينبغي قياس تكاليف " +"التدريب بشكل منفصل." + +#: website/public/blog/reverse-engineering-models/index.html:706 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both views matter because a cheaper proposal can become an expensive " +"workflow if it causes more uninformative compiles. Conversely, a slower " +"model may pay for itself by finding a useful experiment sooner. The " +"comparison should capture those costs." +msgstr "" +"كلا القياسين مهمان. يمكن أن يؤدي الاقتراح الرخيص إلى سير عمل مكلف إذا تسبب " +"في المزيد من التجميعات التي لا تقدم معلومات مفيدة. وعلى العكس من ذلك، قد " +"يكون النموذج الأبطأ أكثر اقتصادا إذا وجد تجربة مفيدة في وقت أقرب. وينبغي أن " +"تشمل المقارنة هذه التكاليف." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:712 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Counting accepted results also keeps intermediate progress in perspective. " +"One TH09 {ph0}replay-save candidate{ph1} looked promising because the " +"compiled function had the expected size, but a full comparison still found " +"substantial differences. Changes in how the compiler arranges instructions " +"can also make a candidate look closer without resolving the underlying " +"problem. Such diagnostics can guide the next experiment, but I would " +"validate any reward for \"getting closer\" against the full acceptance " +"check." +msgstr "" +"كما أن حساب النتائج المقبولة يحافظ على التقدم المتوسط في المنظور الصحيح. بدا" +" أحد مرشحات TH09 {ph0}replay-save{ph1} واعدًا لأن الدالة المترجمة لها الحجم " +"المتوقع، ولكن المقارنة الكاملة لا تزال تجد اختلافات جوهرية. إن التغييرات في " +"كيفية قيام المترجم بترتيب التعليمات يمكن أن تجعل المرشح يبدو أقرب دون حل " +"المشكلة الأساسية. يمكن لمثل هذه التشخيصات أن توجه التجربة التالية، لكنني " +"سأؤكد صحة أي مكافأة مقابل \"الاقتراب\" مقابل فحص القبول الكامل." + +#: website/public/blog/reverse-engineering-models/index.html:727 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would be interested in the pattern across these results. If a specialist " +"does well within a familiar compiler family, does that advantage survive a " +"toolchain change? And once both models have good retrieval, how much does " +"training still add? The answers would help us decide where each kind of " +"model is most useful." +msgstr "" +"سأكون مهتمًا بالنمط عبر هذه النتائج. إذا كان نموذج الذكاء الاصطناعي المتخصص " +"يعمل بشكل جيد ضمن عائلة المترجمات المألوفة، فهل ستصمد هذه الميزة بعد تغيير " +"سلسلة الأدوات؟ وبمجرد أن يتمتع كلا نموذجي الذكاء الاصطناعي بقدرة جيدة على " +"الاسترجاع، ما مقدار التدريب الذي سيضيفه؟ ستساعدنا الإجابات في تحديد المكان " +"الذي يكون فيه كل نوع من النماذج أكثر فائدة." + +#: website/public/blog/reverse-engineering-models/index.html:736 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What kind of agent does this suggest?" +msgstr "ما هو نوع وكيل الذكاء الاصطناعي الذي يشير إليه هذا؟" + +#: website/public/blog/reverse-engineering-models/index.html:737 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"If the models turn out to help at different stages, a useful arrangement " +"might be a broad agent with access to a specialist. Once a task becomes a " +"familiar compiler search, it could ask the specialist for candidates. If the" +" results stop making sense, the broader agent could return to the target " +"evidence or investigate the measurement itself." +msgstr "" +"إذا تبين أن النماذج تساعد في مراحل مختلفة، فقد يكون الترتيب المفيد هو وكيل " +"واسع النطاق لديه إمكانية الوصول إلى نموذج متخصص للذكاء الاصطناعي. بمجرد أن " +"تصبح المهمة عبارة عن بحث مألوف للمترجم، يمكن أن تطلب نموذج الذكاء الاصطناعي " +"المتخصص للمرشحين. إذا لم تعد النتائج منطقية، فيمكن للعامل الأوسع العودة إلى " +"الدليل المستهدف أو التحقيق في القياس نفسه." + +#: website/public/blog/reverse-engineering-models/index.html:745 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In that arrangement, specialization becomes a tool the agent can choose as " +"the investigation develops. The agent keeps the freedom to change direction " +"that made the earlier breakthroughs possible." +msgstr "" +"وفي هذا الترتيب، يصبح التخصص أداة يمكن للعميل اختيارها مع تطور التحقيق. " +"يحتفظ الوكيل بحرية تغيير الاتجاه الذي جعل الاختراقات السابقة ممكنة." + +#: website/public/blog/reverse-engineering-models/index.html:750 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This follows naturally from the industrial shift I described in the previous" +" essay. Being able to run many more experiments changes how we organize the " +"work. It also gives us a reason to ask which decisions need broad judgment " +"and which improve through repeated experience with a particular craft." +msgstr "" +"ويأتي هذا بشكل طبيعي من التحول الصناعي الذي وصفته في المقالة السابقة. إن " +"القدرة على إجراء المزيد من التجارب تغير الطريقة التي ننظم بها العمل. كما أنه" +" يمنحنا سببًا للتساؤل عن القرارات التي تحتاج إلى حكم واسع النطاق وتلك التي " +"تتحسن من خلال الخبرة المتكررة في حرفة معينة." + +#: website/public/blog/reverse-engineering-models/index.html:757 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For now, the strongest general model has earned its place in my workflow by " +"opening problems I had left unresolved. Those successes are what make me " +"want to examine the work more closely: how much depended on the model's " +"breadth, and how much could be handled by a model trained in this particular" +" kind of reconstruction?" +msgstr "" +"حتى الآن، استحق أقوى نموذج عام مكانه في سير عملي لأنه حل مشكلات كنت قد " +"تركتها دون حل. وهذه النجاحات تدفعني إلى دراسة العملية: كم منها اعتمد على " +"اتساع قدراته، وكم يمكن أن يتولاه نموذج مدرب على هذا النوع من إعادة البناء؟" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:765 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}The repositories give us a way to test the question.{ph1} I would like " +"to use that record to build the comparison and see what we learn." +msgstr "" +"{ph0}توفر لنا المستودعات طريقة لاختبار السؤال.{ph1} أود استخدام هذا السجل " +"لإجراء المقارنة ومعرفة ما نتعلمه." + +#: website/public/blog/reverse-engineering-models/index.html:775 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Evidence and model attribution" +msgstr "الأدلة وإسناد النموذج" + +#: website/public/blog/reverse-engineering-models/index.html:783 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The examples describe historical function-level comparisons recorded in the " +"linked commits. Whole-game runtime checks are separate milestones. Source " +"and matching records were reviewed on 13 October 2026." +msgstr "" +"تصف الأمثلة المقارنات التاريخية على مستوى الوظيفة المسجلة في الالتزامات " +"المرتبطة. تعتبر عمليات التحقق من وقت تشغيل اللعبة بأكملها بمثابة مراحل " +"منفصلة. تمت مراجعة سجلات المصدر والمطابقة في 13 أكتوبر 2026." + +#. {ph0}: gpt-dots +#. {ph1}: gpt-web +#: website/public/blog/reverse-engineering-models/index.html:789 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The TH09 label {ph0} identifies GPT-6-Astra. {ph1} changed with the model " +"used at the time, so that label alone does not identify an individual run. " +"Explicit model names in the other cited commits establish their recorded " +"attribution; the histories do not contain uniform per-run cost or effort " +"measurements." +msgstr "" +"التسمية TH09 {ph0} تعني GPT-6-Astra. تغير النموذج المرتبط بـ {ph1} بمرور " +"الوقت، لذا لا يمكن لهذه التسمية وحدها تحديد النموذج المستخدم في عملية تشغيل " +"معينة. توفر أسماء النماذج الصريحة في الكوميتات المذكورة الأخرى الإسناد " +"المسجل. لا تتضمن هذه السجلات قياسات متسقة للتكلفة أو الجهد لكل عملية تشغيل." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#: website/public/blog/reverse-engineering-models/index.html:798 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 temporary lifetimes:{ph1} the {ph2}earlier context review{ph3} and" +" {ph4}complete cold reconstruction{ph5}." +msgstr "" +"{ph0}TH09 عمر الكائن المؤقت:{ph1} تم التحقق من {ph2}مراجعة السياق " +"السابقة{ph3} و{ph4}إعادة البناء الكاملة من خلال إعادة تجميع نظيفة من " +"المصدر{ph5}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:810 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 Front callback:{ph1} {ph2}non-exact diagnostic regions{ph3}, " +"{ph4}Astra's new graph{ph5} and its {ph6}accepted recompile replay{ph7}." +msgstr "" +"{ph0}TH09 Front رد الاتصال:{ph1} {ph2}مناطق التشخيص التي لم تتطابق مع{ph3}، " +"والرسم البياني الجديد لتدفق التحكم في {ph4}Astra{ph5} و{ph6}فحص إعادة " +"الترجمة المقبول{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:826 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Compiler knowledge:{ph1} {ph2}TH095 identifier-bucket reuse{ph3}, " +"{ph4}TH04 player source and linker order{ph5} and {ph6}Luna's TH09 enemy-" +"spawn closure{ph7}." +msgstr "" +"{ph0}معرفة المترجم:{ph1} {ph2}TH095 إعادة استخدام سلوك مجموعة المعرفات{ph3}،" +" {ph4}TH04 شيفرة اللاعب وترتيب الرابط{ph5} و{ph6}Luna إعادة البناء الكاملة " +"لوظيفة إنشاء العدو لـ TH09{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:842 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Checks and diagnostics:{ph1} {ph2}TH08 floating-literal audit{ph3}, " +"{ph4}TH08 numerical investigation{ph5} and {ph6}TH09 replay-save " +"reconstruction{ph7}." +msgstr "" +"{ph0}عمليات الفحص والتشخيص:{ph1} {ph2}TH08 التدقيق الثابت للفاصلة " +"العائمة{ph3}، و{ph4}TH08 التحقيق الرقمي{ph5} و{ph6}TH09 إعادة البناء مع " +"توفير إعادة التشغيل{ph7}." + +#: website/public/blog/reverse-engineering-models/index.html:859 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The proposed specialist and RL comparisons are future experiments. The " +"project histories motivate them; no trained specialist was evaluated for " +"this essay." +msgstr "" +"المقارنات المقترحة للنماذج المتخصصة والتدريب RL هي تجارب مستقبلية. تاريخ " +"المشروع يحفزهم. لم يتم تقييم أي نموذج متخصص مدرب لهذا المقال." + +#: website/public/blog/reverse-engineering-models/index.html:6 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model? | REA" +msgstr "هل تحتاج الهندسة العكسية إلى نموذج أكبر؟ | REA" + +#: website/public/blog/reverse-engineering-models/index.html:7 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What Touhou reconstruction taught me about general models, specialist models" +" and learning to propose better hypotheses from compiler feedback." +msgstr "" +"ما تعلمته من إعادة بناء Touhou عن النماذج العامة والمتخصصة، وعن تعلم اقتراح " +"فرضيات أفضل من ملاحظات المترجم البرمجي." + +#: website/public/blog/reverse-engineering-models/index.html:240 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "On this page" +msgstr "في هذه الصفحة" + +#: website/public/blog/reverse-engineering-models/index.html:253 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Models and reverse engineering essay" +msgstr "النماذج والهندسة العكسية مقال" + +#: website/public/blog/reverse-engineering-models/index.html:491 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Selected reconstruction experiments" +msgstr "تجارب إعادة بناء الشيفرة المختارة" + +#: website/public/blog/reverse-engineering-models/index.html:883 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Back to top" +msgstr "العودة للأعلى" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:776 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Detailed evidence notes{ph1} include the recorded comparisons and " +"experiment context." +msgstr "" +"تتضمن {ph0}ملاحظات الأدلة التفصيلية{ph1} المقارنات المسجلة وسياق التجارب." + +#~ msgctxt "page:blog/index.html" +#~ msgid "By N0zoM1z0 · 13 October 2026 · Draft for review" +#~ msgstr "بقلم N0zoM1z0 · 13 أكتوبر 2026 · مسودة للمراجعة" + #~ msgctxt "page:index.html" #~ msgid "Setup guide" #~ msgstr "دليل الإعداد" diff --git a/website/locales/de/messages.po b/website/locales/de/messages.po index 3b4690d00..2b23bc4a3 100644 --- a/website/locales/de/messages.po +++ b/website/locales/de/messages.po @@ -32,18 +32,18 @@ msgid "Thoughts on reverse engineering." msgstr "Gedanken zum Reverse Engineering." #. {ph0}:
-#: website/public/blog/index.html:200 +#: website/public/blog/index.html:214 #, python-brace-format msgctxt "page:blog/index.html" msgid "Game reconstruction{ph0}Essay" msgstr "Spielrekonstruktion{ph0}Essay" -#: website/public/blog/index.html:202 +#: website/public/blog/index.html:216 msgctxt "page:blog/index.html" msgid "The industrial revolution in reverse engineering" msgstr "Die industrielle Revolution im Reverse Engineering" -#: website/public/blog/index.html:203 +#: website/public/blog/index.html:217 msgctxt "page:blog/index.html" msgid "" "What manual RE and reconstructing TH08, TH095 and TH04 taught me about " @@ -53,25 +53,26 @@ msgstr "" "gelehrt, über Agenten, Überprüfung und Wissen, dass Verbindungen über " "Projekte hinweg." -#: website/public/blog/index.html:208 +#: website/public/blog/index.html:207 website/public/blog/index.html:222 msgctxt "page:blog/index.html" msgid "Read the article" msgstr "Lesen Sie den Artikel" -#: website/public/blog/index.html:211 +#: website/public/blog/index.html:225 msgctxt "page:blog/index.html" msgid "By N0zoM1z0 · 10 October 2026" msgstr "Von N0zoM1z0 · 10. Oktober 2026" #. {ph0}: -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 +#: website/public/blog/reverse-engineering-models/index.html:883 #: website/public/blog/touhou-reconstruction/index.html:1080 #: website/public/examples/dino-lab/index.html:475 #: website/public/faq/index.html:573 #: website/public/first-investigation/index.html:523 #: website/public/get-started/index.html:882 -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 #: website/public/guides/index.html:282 #: website/public/guides/javascript/index.html:503 #: website/public/guides/native/index.html:509 website/public/index.html:1191 @@ -101,6 +102,8 @@ msgstr "" "REA und KI-Agenten." #: website/public/blog/index.html:75 website/public/blog/index.html:88 +#: website/public/blog/reverse-engineering-models/index.html:75 +#: website/public/blog/reverse-engineering-models/index.html:88 #: website/public/blog/touhou-reconstruction/index.html:75 #: website/public/blog/touhou-reconstruction/index.html:88 #: website/public/examples/dino-lab/index.html:75 @@ -137,6 +140,7 @@ msgstr "" "Ihrem Codierungsagenten." #: website/public/blog/index.html:108 +#: website/public/blog/reverse-engineering-models/index.html:108 #: website/public/blog/touhou-reconstruction/index.html:108 #: website/public/examples/dino-lab/index.html:109 #: website/public/faq/index.html:108 @@ -157,6 +161,7 @@ msgid "REA home" msgstr "REA-Startseite" #: website/public/blog/index.html:112 +#: website/public/blog/reverse-engineering-models/index.html:112 #: website/public/blog/touhou-reconstruction/index.html:112 #: website/public/examples/dino-lab/index.html:113 #: website/public/faq/index.html:112 @@ -177,6 +182,7 @@ msgid "Main navigation" msgstr "Hauptnavigation" #: website/public/blog/index.html:130 +#: website/public/blog/reverse-engineering-models/index.html:130 #: website/public/blog/touhou-reconstruction/index.html:130 #: website/public/examples/dino-lab/index.html:131 #: website/public/faq/index.html:130 @@ -197,6 +203,7 @@ msgid "REA on GitHub" msgstr "REA auf GitHub" #: website/public/blog/index.html:150 +#: website/public/blog/reverse-engineering-models/index.html:150 #: website/public/blog/touhou-reconstruction/index.html:150 #: website/public/examples/dino-lab/index.html:151 #: website/public/faq/index.html:150 @@ -217,6 +224,7 @@ msgid "Join REA on Discord" msgstr "REA auf Discord beitreten" #: website/public/blog/index.html:169 +#: website/public/blog/reverse-engineering-models/index.html:169 #: website/public/blog/touhou-reconstruction/index.html:169 #: website/public/examples/dino-lab/index.html:170 #: website/public/faq/index.html:169 @@ -236,7 +244,7 @@ msgctxt "shared" msgid "Follow REA on X" msgstr "REA auf X folgen" -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 msgctxt "page:blog/index.html" msgid "Back to top" msgstr "Nach oben" @@ -4854,7 +4862,7 @@ msgctxt "page:guides/browser/index.html" msgid "Copy read the loaded scripts" msgstr "Kopieren Lesen Sie die geladenen Skripte" -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 msgctxt "page:guides/browser/index.html" msgid "Back to top" msgstr "Nach oben" @@ -11845,6 +11853,7 @@ msgid "Same spacing: 256 ÷ 16 = 16 angle units = 22.5°" msgstr "Gleicher Abstand: 256 ÷ 16 = 16 Winkeleinheiten = 22,5°" #: website/public/blog/index.html:114 +#: website/public/blog/reverse-engineering-models/index.html:114 #: website/public/blog/touhou-reconstruction/index.html:114 #: website/public/examples/dino-lab/index.html:115 #: website/public/faq/index.html:114 @@ -11865,6 +11874,7 @@ msgid "Showcases" msgstr "Fallstudien" #: website/public/blog/index.html:115 +#: website/public/blog/reverse-engineering-models/index.html:115 #: website/public/blog/touhou-reconstruction/index.html:115 #: website/public/examples/dino-lab/index.html:116 #: website/public/faq/index.html:115 @@ -11885,6 +11895,7 @@ msgid "Blog" msgstr "Blog" #: website/public/blog/index.html:113 +#: website/public/blog/reverse-engineering-models/index.html:113 #: website/public/blog/touhou-reconstruction/index.html:113 #: website/public/examples/dino-lab/index.html:114 #: website/public/faq/index.html:113 @@ -11905,6 +11916,7 @@ msgid "Get started" msgstr "Erste Schritte" #: website/public/blog/index.html:116 +#: website/public/blog/reverse-engineering-models/index.html:116 #: website/public/blog/touhou-reconstruction/index.html:116 #: website/public/examples/dino-lab/index.html:117 #: website/public/faq/index.html:116 @@ -11924,13 +11936,14 @@ msgctxt "shared" msgid "Guides" msgstr "Anleitungen" -#: website/public/blog/index.html:222 +#: website/public/blog/index.html:236 +#: website/public/blog/reverse-engineering-models/index.html:875 #: website/public/blog/touhou-reconstruction/index.html:1072 #: website/public/examples/dino-lab/index.html:467 #: website/public/faq/index.html:565 #: website/public/first-investigation/index.html:515 #: website/public/get-started/index.html:874 -#: website/public/guides/browser/index.html:490 +#: website/public/guides/browser/index.html:499 #: website/public/guides/index.html:274 #: website/public/guides/javascript/index.html:495 #: website/public/guides/native/index.html:501 website/public/index.html:1183 @@ -11944,13 +11957,14 @@ msgctxt "shared" msgid "Source" msgstr "Quellcode" -#: website/public/blog/index.html:223 +#: website/public/blog/index.html:237 +#: website/public/blog/reverse-engineering-models/index.html:876 #: website/public/blog/touhou-reconstruction/index.html:1073 #: website/public/examples/dino-lab/index.html:468 #: website/public/faq/index.html:566 #: website/public/first-investigation/index.html:516 #: website/public/get-started/index.html:875 -#: website/public/guides/browser/index.html:491 +#: website/public/guides/browser/index.html:500 #: website/public/guides/index.html:275 #: website/public/guides/javascript/index.html:496 #: website/public/guides/native/index.html:502 website/public/index.html:1184 @@ -11964,13 +11978,14 @@ msgctxt "shared" msgid "MIT license" msgstr "MIT-Lizenz" -#: website/public/blog/index.html:224 +#: website/public/blog/index.html:238 +#: website/public/blog/reverse-engineering-models/index.html:877 #: website/public/blog/touhou-reconstruction/index.html:1074 #: website/public/examples/dino-lab/index.html:469 #: website/public/faq/index.html:567 #: website/public/first-investigation/index.html:517 #: website/public/get-started/index.html:876 -#: website/public/guides/browser/index.html:492 +#: website/public/guides/browser/index.html:501 #: website/public/guides/index.html:276 #: website/public/guides/javascript/index.html:497 #: website/public/guides/native/index.html:503 website/public/index.html:1185 @@ -11984,13 +11999,14 @@ msgctxt "shared" msgid "Join the community" msgstr "Zur Community" -#: website/public/blog/index.html:225 +#: website/public/blog/index.html:239 +#: website/public/blog/reverse-engineering-models/index.html:878 #: website/public/blog/touhou-reconstruction/index.html:1075 #: website/public/examples/dino-lab/index.html:470 #: website/public/faq/index.html:568 #: website/public/first-investigation/index.html:518 #: website/public/get-started/index.html:877 -#: website/public/guides/browser/index.html:493 +#: website/public/guides/browser/index.html:502 #: website/public/guides/index.html:277 #: website/public/guides/javascript/index.html:498 #: website/public/guides/native/index.html:504 website/public/index.html:1186 @@ -12939,6 +12955,1430 @@ msgctxt "runtime" msgid "Switch to dark mode" msgstr "Zum dunklen Modus wechseln" +#. {ph0}: +#. {ph1}: +#: website/public/guides/browser/index.html:480 +#, python-brace-format +msgctxt "page:guides/browser/index.html" +msgid "" +"For HTTP request and response bodies, use {ph0}Fiddler Everywhere: SAZ " +"captures, MCP and public skills{ph1}. Export SAZ or HAR 1.2 for offline REA " +"capture Evidence, or connect Everywhere's MCP server to inspect captured " +"traffic with your agent." +msgstr "" +"Für HTTP-Anfrage- und Antwortinhalte verwenden Sie {ph0}Fiddler Everywhere: " +"SAZ-Aufzeichnungen, MCP und öffentliche Skills{ph1}. Exportieren Sie SAZ " +"oder HAR 1.2 für die Offline-Auswertung der Aufzeichnung als REA Evidence, " +"oder verbinden Sie den MCP-Server von Everywhere, um aufgezeichneten " +"Datenverkehr mit Ihrem Agenten zu untersuchen." + +#. {ph0}:
+#: website/public/blog/index.html:200 +#, python-brace-format +msgctxt "page:blog/index.html" +msgid "Model specialization{ph0}Essay" +msgstr "Spezialisierte Modelle{ph0}Essay" + +#: website/public/blog/index.html:202 +msgctxt "page:blog/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "Braucht Reverse Engineering ein größeres Modell?" + +#: website/public/blog/index.html:203 +msgctxt "page:blog/index.html" +msgid "" +"What Touhou reconstruction suggests about general models, specialist models " +"and learning from compiler feedback." +msgstr "" +"Was die Rekonstruktion von Touhou über allgemeine und spezialisierte Modelle" +" sowie das Lernen aus Compiler-Rückmeldungen zeigt." + +#: website/public/blog/index.html:210 +msgctxt "page:blog/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Von N0zoM1z0 · 13. Oktober 2026" + +#: website/public/blog/reverse-engineering-models/index.html:103 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Skip to content" +msgstr "Zum Inhalt springen" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:194 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Home{ph1} / {ph2}Blog{ph3} / Models and RE" +msgstr "{ph0}Home{ph1} / {ph2}Blog{ph3} / Modelle und Reverse Engineering" + +#: website/public/blog/reverse-engineering-models/index.html:197 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Essay" +msgstr "Aufsatz" + +# Reviewed for technical meaning and readability; preserve the distinction +# between historical evidence and proposed experiments. +#: website/public/blog/reverse-engineering-models/index.html:198 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "Braucht Reverse Engineering ein größeres Modell?" + +#: website/public/blog/reverse-engineering-models/index.html:199 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Von N0zoM1z0 · 13. Oktober 2026" + +#: website/public/blog/reverse-engineering-models/index.html:201 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Takeaway" +msgstr "Das Wichtigste" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:203 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Model capability changes the hypotheses an agent can find.{ph1} Later " +"sessions have solved problems that earlier work left unresolved." +msgstr "" +"{ph0}Die Modellfähigkeit beeinflusst, welche Hypothesen ein KI-Agent " +"vorschlagen kann.{ph1} Spätere Sitzungen lösten Probleme, die in früheren " +"Arbeiten ungelöst geblieben waren." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:211 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A familiar compiler gives specialization something concrete to " +"learn.{ph1} Some exact matches depend on narrow, reusable source patterns." +msgstr "" +"{ph0}Ein bekannter Compiler bietet konkrete Muster für ein spezielles Modell" +" zum Lernen.{ph1} Einige genaue Maschinencode-Übereinstimmungen hängen von " +"bestimmten, wiederverwendbaren Quellcodemustern ab." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:218 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Give repository knowledge a fair comparison.{ph1} A smaller model with " +"good retrieval may already know enough to make progress." +msgstr "" +"{ph0}Beziehen Sie das Repository-Wissen in einen fairen Vergleich ein.{ph1} " +"Ein kleineres Modell mit gutem Wissensabruf verfügt möglicherweise bereits " +"über genügend Informationen, um Fortschritte zu erzielen." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:223 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Train the next useful experiment, and test its oracle.{ph1} The value " +"of feedback depends on what the checker actually verifies." +msgstr "" +"{ph0}Trainieren Sie das Modell, um das nächste nützliche Experiment " +"auszuwählen, und testen Sie seinen Prüfer.{ph1} Feedback ist nur insoweit " +"nützlich, als der Prüfer die richtigen Dinge überprüft." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:230 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Measure accepted reconstruction against time and cost.{ph1} That is the" +" result I want from the whole agent." +msgstr "" +"{ph0}Bewerten Sie verifizierte Rekonstruktionsergebnisse neben Zeit und " +"Kosten.{ph1} Das ist es, was ich für den gesamten KI-Agent-Workflow messen " +"möchte." + +#: website/public/blog/reverse-engineering-models/index.html:241 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "In this article" +msgstr "In diesem Artikel" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:243 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What better models changed{ph1}" +msgstr "{ph0}Was leistungsfähigere Modelle verändert haben{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:244 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Why specialize?{ph1}" +msgstr "{ph0}Warum spezialisieren?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:245 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Where experience lives{ph1}" +msgstr "{ph0}Wo sollte diese Erfahrung gespeichert werden?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:246 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Learning the next experiment{ph1}" +msgstr "{ph0}Lernen Sie das nächste nützliche Experiment{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:247 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}RE and reinforcement learning{ph1}" +msgstr "{ph0}Reverse Engineering und verstärkendes Lernen{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:248 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}When the oracle is wrong{ph1}" +msgstr "{ph0}Wenn der Prüfer falsch ist{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:249 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}A comparison worth running{ph1}" +msgstr "{ph0}Ein Vergleich, der sich lohnt{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:250 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What kind of agent?{ph1}" +msgstr "{ph0}Was für einen KI-Agenten lässt dies vermuten?{ph1}" + +#: website/public/blog/reverse-engineering-models/index.html:258 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What better models changed" +msgstr "Was leistungsfähigere Modelle verändert haben" + +#: website/public/blog/reverse-engineering-models/index.html:259 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Since August 2026, I have been exploring how much reverse engineering an " +"agent can carry out with access to a program and a way to check its work. " +"Most of my experience comes from reconstructing games, especially Touhou. As" +" I have changed models, the repositories have kept a record of what each " +"session managed to recover." +msgstr "" +"Seit August 2026 erforsche ich, wie viel Reverse Engineering ein Agent mit " +"Zugriff auf ein Programm und einer Möglichkeit zur Überprüfung seiner Arbeit" +" durchführen kann. Der größte Teil meiner Erfahrung stammt aus der " +"Rekonstruktion von Spielen, insbesondere Touhou. Da ich die Modelle geändert" +" habe, haben die Repositorys aufgezeichnet, was jede Sitzung " +"wiederherstellen konnte." + +#: website/public/blog/reverse-engineering-models/index.html:267 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of that work aims at exact reconstruction: we compile the maintained " +"source under a pinned historical toolchain and check whether it reproduces " +"the target function. When a later model succeeds where earlier work stalled," +" the source changes show what made that progress possible." +msgstr "" +"Ein Großteil dieser Arbeit zielt auf eine exakte Rekonstruktion ab: Wir " +"kompilieren die gepflegte Quelle unter einer historischen Toolchain mit " +"festen Versionen und prüfen, ob sie die Zielfunktion reproduziert. Wenn ein " +"späteres Modell erfolgreich ist, wo frühere Arbeiten ins Stocken geraten " +"sind, zeigen die Quelländerungen, was diesen Fortschritt ermöglicht hat." + +#: website/public/blog/reverse-engineering-models/index.html:274 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The temporary values had to stay alive together" +msgstr "Die temporären Variablen benötigten überlappende Lebensdauern" + +#: website/public/blog/reverse-engineering-models/index.html:275 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of a TH09 attack function's logic had already been reconstructed, but " +"the compiler still stored temporary results differently from the original. " +"The remaining problem was how long those results needed to stay alive and " +"when their storage could be reused." +msgstr "" +"Ein Großteil der Logik einer TH09-Angriffsfunktion war bereits " +"rekonstruiert, aber der Compiler speicherte temporäre Ergebnisse immer noch " +"anders als das Original. Das verbleibende Problem bestand darin, wie lange " +"diese Ergebnisse am Leben bleiben mussten und wann ihr Speicher " +"wiederverwendet werden konnte." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:282 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A {ph0}GPT-6-Sol checkpoint{ph1} recorded this mismatch after testing " +"several ordinary source arrangements. When {ph2}GPT-6.1-Sol returned to the " +"problem{ph3}, it found a useful hypothesis about how long those temporary " +"values needed to stay alive." +msgstr "" +"Ein {ph0}GPT-6-Sol-Sitzungsdatensatz{ph1} hat diese Nichtübereinstimmung " +"gemeldet, nachdem mehrere gewöhnliche Quellcode-Anordnungen getestet wurden." +" Als {ph2}GPT-6.1-Sol das Problem erneut untersuchte{ph3}, schlug es eine " +"nützliche Hypothese über die Lebensdauer dieser temporären Variablen vor." + +#: website/public/blog/reverse-engineering-models/index.html:296 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The function produced temporary results and copied them into animation " +"positions. In C++, a temporary result stays alive until the expression using" +" it finishes. Grouping the real operations into one expression kept those " +"results alive together, so the compiler could no longer reuse the same " +"storage for all of them." +msgstr "" +"Die Funktion erzeugte temporäre Ergebnisse und kopierte sie in " +"Animationspositionen. In C++ bleibt ein temporäres Ergebnis bestehen, bis " +"der Ausdruck, der es verwendet, beendet ist. Durch die Gruppierung der " +"realen Operationen in einem Ausdruck blieben diese Ergebnisse zusammen " +"erhalten, sodass der Compiler nicht mehr denselben Speicher für alle " +"wiederverwenden konnte." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:303 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"One result still shared storage with a value used later for collision " +"checks. Giving that value its own lifetime resolved the remaining overlap. " +"After several targeted experiments, {ph0}a clean rebuild confirmed that the " +"full function matched{ph1}." +msgstr "" +"Ein Ergebnis wird weiterhin gemeinsam mit einem Wert gespeichert, der später" +" für Kollisionsprüfungen verwendet wird. Indem man diesem Wert seine eigene " +"Lebensdauer zuordnete, wurde die verbleibende Überlappung behoben. Nach " +"mehreren gezielten Experimenten bestätigte {ph0}eine saubere Neukompilierung" +" aus dem Quellcode, dass die vollständige Funktion übereinstimmte{ph1}." + +#: website/public/blog/reverse-engineering-models/index.html:312 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What interests me is how the discovery and the tests worked together. The " +"lifetime hypothesis gave the agent a new direction; compiler feedback helped" +" it refine that hypothesis into a source arrangement that matched." +msgstr "" +"Mich interessiert, wie sich Entdeckung und Tests gegenseitig unterstützten. " +"Die Hypothese über die temporäre Lebensdauer von Objekten gab dem KI-Agenten" +" eine neue Richtung. Das Feedback des Compilers half dabei, diese Hypothese " +"in eine passende Quellcode-Anordnung zu verfeinern." + +#: website/public/blog/reverse-engineering-models/index.html:318 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A different control-flow graph" +msgstr "Ein anderes Kontrollflussdiagramm" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:319 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A second TH09 function shows why that new direction can matter. Here, the " +"obstacle was how several branches shared their final work. " +"{ph0}GPT-6.1-Sol's earlier work{ph1} got parts of the function to match, but" +" the branches still joined up differently from the original." +msgstr "" +"Eine zweite TH09-Funktion zeigt, warum eine neue Richtung wichtig sein kann." +" Hier ging es darum, wie mehrere Verzweigungen dieselben abschließenden " +"Operationen nutzen. {ph0}GPT-6.1-Sols frühere Arbeit{ph1} brachte Teile der " +"Funktion zur Übereinstimmung, doch die Verzweigungen liefen noch anders " +"zusammen als im Original." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:330 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}GPT-6-Astra changed the surrounding control-flow graph{ph1}. The main " +"paths now led to a shared final update and return. In that source structure," +" the compiler combined the branches in the same way as the original. The " +"complete function matched, and a {ph2}separate rebuild check confirmed " +"it{ph3}." +msgstr "" +"{ph0}GPT-6-Astra hat das umgebende Kontrollflussdiagramm{ph1} geändert. Die " +"Hauptwege führten nun zu einer gemeinsamen finalen Aktualisierung und " +"Rückkehr. In dieser Quellstruktur kombinierte der Compiler die Zweige auf " +"die gleiche Weise wie im Original. Die vollständige Funktion stimmte überein" +" und eine {ph2}separate Neuerstellungsprüfung bestätigte dies{ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:342 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Changing the representation opened a more productive search space.{ph1}" +" Both cases give me a reason to take general model capability seriously: an " +"agent can keep trying local variations for a long time before finding a " +"better way to express the surrounding work." +msgstr "" +"{ph0}Durch die Änderung der Darstellung wurde ein produktiverer Suchraum " +"eröffnet.{ph1} Beide Fälle geben mir einen Grund, die allgemeine " +"Modellfähigkeit ernst zu nehmen: Ein Agent kann lange Zeit lokale " +"Variationen ausprobieren, bevor er eine bessere Möglichkeit findet, die " +"umgebende Arbeit auszudrücken." + +#: website/public/blog/reverse-engineering-models/index.html:351 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"At the same time, the later sessions started with earlier source and " +"recorded failures. That accumulated work was part of their advantage. The " +"history lets us inspect the discoveries and their checks; a controlled " +"comparison would help separate the effect of the model from the experience " +"it inherited." +msgstr "" +"Die späteren Sitzungen übernahmen allerdings auch früheren Quellcode und " +"aufgezeichnete Fehlversuche. Diese Vorarbeit war Teil ihres Vorteils. Die " +"Historie macht die Entdeckungen und ihre Prüfungen nachvollziehbar. Ein " +"kontrollierter Vergleich würde helfen, den Beitrag des Modells vom Beitrag " +"der übernommenen Erfahrung zu trennen." + +#: website/public/blog/reverse-engineering-models/index.html:358 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is where the question becomes interesting to me. Some of the progress " +"came from finding a new representation, while much of the remaining work " +"involved testing source forms against a familiar compiler. How much needs " +"broad reasoning ability, and how much could a model learn through repeated " +"experience with that compiler and code?" +msgstr "" +"Das macht die Frage für mich interessant. Einige Fortschritte wurden durch " +"die Suche nach einer neuen Vertretung erzielt. Ein Großteil der " +"verbleibenden Arbeit bestand darin, verschiedene Quellcode-Anordnungen mit " +"einem vertrauten Compiler zu testen. Wie viel davon erfordert ein breites " +"Denkvermögen? Wie viel könnte ein Modell lernen, wenn es wiederholt mit " +"diesem Compiler und Code arbeitet?" + +#: website/public/blog/reverse-engineering-models/index.html:368 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why might a specialist do well?" +msgstr "Warum könnte ein spezialisiertes KI-Modell gut funktionieren?" + +#: website/public/blog/reverse-engineering-models/index.html:369 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Touhou gives us a concrete reason to expect that experience to help. Much of" +" the game code comes from the same author, ZUN, so recurring habits are a " +"reasonable starting assumption. The repositories back that up with useful " +"examples: related loops and helper arrangements have supplied hypotheses " +"across games." +msgstr "" +"Touhou gibt uns einen konkreten Grund zu der Annahme, dass diese Erfahrung " +"hilfreich sein wird. Ein Großteil des Spielcodes stammt vom selben Autor, " +"ZUN, daher sind wiederkehrende Gewohnheiten eine vernünftige " +"Ausgangsannahme. Die Repositories untermauern dies mit nützlichen " +"Beispielen: Verwandte Schleifen und Hilfsanordnungen haben spielübergreifend" +" Hypothesen geliefert." + +#: website/public/blog/reverse-engineering-models/index.html:376 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those source patterns become more specific when we include the compiler. " +"Under a particular version and set of options, small changes in source can " +"produce repeatable differences in machine code. Some of those differences " +"are surprisingly obscure." +msgstr "" +"Diese Quellmuster werden spezifischer, wenn wir den Compiler einbeziehen. " +"Unter einer bestimmten Version und einem bestimmten Optionssatz können " +"kleine Änderungen in der Quelle zu wiederholbaren Unterschieden im " +"Maschinencode führen. Einige dieser Unterschiede sind überraschend unklar." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: Supervisor::LoadDat +#. {ph4}: +#: website/public/blog/reverse-engineering-models/index.html:382 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In TH095, for example, {ph0}even the name used for a local variable could " +"change where VC7.1 stored it{ph1}. Moving the declaration alone did not " +"solve the layout, but an earlier background function had revealed a naming " +"pattern that affected the compiler's choice. That gave us a more promising " +"hypothesis for {ph2}the reconstruction of {ph3}{ph4}." +msgstr "" +"In TH095 beispielsweise könnte {ph0}sogar der Name einer lokalen Variablen " +"Einfluss darauf haben, wo VC7.1 sie gespeichert hat{ph1}. Das Verschieben " +"der Deklaration hat das Speicherlayout nicht gelöst. Allerdings hatte eine " +"frühere Hintergrundfunktion ein Namensmuster offenbart, das die Wahl des " +"Compilers beeinflusste. Das legte eine vielversprechendere Hypothese für die" +" {ph2}Rekonstruktion von {ph3}{ph4} nahe." + +#: website/public/blog/reverse-engineering-models/index.html:396 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reusing that pattern for the file-size variable produced the required layout" +" without adding storage. The full function then matched." +msgstr "" +"Durch die Wiederverwendung dieses Musters für die Dateigrößenvariable wurde " +"das erforderliche Layout erstellt, ohne dass Speicherplatz hinzugefügt " +"werden musste. Die gesamte Funktion stimmte dann überein." + +#: website/public/blog/reverse-engineering-models/index.html:401 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is the sort of narrow detail a general model may not know when it " +"arrives. A specialist trained on this compiler could encounter many such " +"cases and learn which source variations are worth testing first." +msgstr "" +"Dies ist ein sehr spezifisches Detail, das einem allgemeines KI-Modell " +"möglicherweise nicht bekannt ist, wenn es zum ersten Mal auf das Problem " +"stößt. Ein auf diesen Compiler spezialisiertes Modell könnte viele solcher " +"Fälle erkennen und herausfinden, welche Quellcodevarianten es wert sind, " +"zuerst getestet zu werden." + +#: website/public/blog/reverse-engineering-models/index.html:407 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The opportunity is especially clear in final matching problems where we " +"already understand the behavior. We are then asking which ordinary source " +"form this compiler turns into the observed instructions. Specialization " +"could help the model propose those forms more efficiently, while the target " +"still determines which hypotheses survive." +msgstr "" +"Die Chance wird besonders deutlich bei finalen Matching-Problemen, bei denen" +" wir das Verhalten bereits verstehen. Wir fragen uns dann, welche " +"gewöhnliche Quellcodeanordnung dieser Compiler in die beobachteten " +"Anweisungen umwandelt. Durch die Spezialisierung könnte das Modell diese " +"Formen effizienter vorschlagen, während das Ziel weiterhin bestimmt, welche " +"Hypothesen bestehen bleiben." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:415 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The useful patterns would have to retain their context, though. TH08 uses an" +" earlier Visual C++ version than TH095 and TH09. TH04 uses a different DOS-" +"era compiler, and its {ph0}player reconstruction{ph1} needed different " +"hypotheses about source patterns and how the compiled code was combined into" +" an executable. Shared authorship gives us a prior, but each target tells us" +" whether it applies. Distinguishing those contexts is part of what a " +"specialist would need to learn." +msgstr "" +"Nützliche Muster müssen ihren Kontext behalten. TH08 verwendet eine frühere " +"Visual C++-Version als TH095 und TH09. TH04 verwendet einen anderen Compiler" +" aus der DOS-Ära. Für die {ph0}Rekonstruktion des Spielercodes{ph1} waren " +"unterschiedliche Hypothesen über Quellcodemuster und die Art und Weise " +"erforderlich, wie kompilierter Code mit einer ausführbaren Datei verknüpft " +"wurde. Wenn wir denselben Autor haben, haben wir zunächst eine Erwartung, " +"aber jedes Ziel sagt uns, ob es zutrifft. Ein spezielles Modell müsste diese" +" Kontexte unterscheiden." + +#: website/public/blog/reverse-engineering-models/index.html:431 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Where should that experience live?" +msgstr "Wo sollte diese Erfahrung gespeichert werden?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:432 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before deciding to teach those patterns through model training, I would look" +" at the knowledge we already keep in the repository. In the {ph0}previous " +"essay{ph1}, I wrote about it becoming the project's working memory. That " +"memory includes compiler lessons, along with the source and failed " +"experiments that made them useful." +msgstr "" +"Bevor ich mich entscheide, diese Muster durch Modelltraining zu lehren, " +"würde ich mir das Wissen ansehen, das wir bereits im Repository speichern. " +"Im {ph0}vorherigen Aufsatz{ph1} habe ich darüber geschrieben, dass es zum " +"Arbeitsgedächtnis des Projekts wird. Dieser Speicher umfasst Compiler-" +"Lektionen sowie die Quelle und fehlgeschlagene Experimente, die sie nützlich" +" gemacht haben." + +#: website/public/blog/reverse-engineering-models/index.html:441 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"By retrieving that record, an agent can start with a known result and " +"inspect how we reached it. It can also avoid alternatives that have already " +"failed in the same context. The starting point improves without changing the" +" model's weights." +msgstr "" +"Durch das Abrufen dieses Datensatzes kann ein Agent mit einem bekannten " +"Ergebnis beginnen und prüfen, wie wir zu diesem Ergebnis gelangt sind. Es " +"können auch Alternativen vermieden werden, die im gleichen Kontext bereits " +"gescheitert sind. Der Ausgangspunkt verbessert sich, ohne die trainierten " +"Parameter des Modells zu ändern." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:447 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"When reconstructing TH09's enemy-spawn function, a {ph0}GPT-6-Luna-max " +"session{ph1} drew a hypothesis from TH08's corresponding source family. That" +" earlier source suggested how to structure the loop and handle failures " +"within it. TH09's own evidence supplied the game-specific details, and clean" +" rebuilds confirmed the complete match." +msgstr "" +"Bei der Rekonstruktion der Funktion von TH09 zum Erstellen von Feinden hat " +"eine {ph0}GPT-6-Luna-max-Sitzung{ph1} eine Hypothese aus dem zugehörigen " +"TH08-Quellcode übernommen. Dieser frühere Code schlug vor, wie die Schleife " +"organisiert und Fehler darin behandelt werden sollten. Die eigenen Beweise " +"von TH09 lieferten die spielspezifischen Details. Neuaufbauten aus der " +"Quelle bestätigten, dass die vollständige Funktion übereinstimmte." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:458 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}\"The model did not know that compiler quirk\" and \"the model could " +"not use it\" are different problems.{ph1} Retrieval may solve the first, but" +" turning that fact into a useful candidate still takes work. And if the " +"candidate fails, the model must decide what the result means for its next " +"attempt. Knowing the relevant pattern is only the beginning of that process." +msgstr "" +"{ph0}„Das Modell kannte diese Compiler-Eigenart nicht“ und „Das Modell " +"konnte sie nicht verwenden“ sind unterschiedliche Probleme.{ph1} Das erste " +"Problem kann durch das Abrufen gelöst werden, aber die Umwandlung dieser " +"Tatsache in einen nützlichen Kandidaten erfordert noch Arbeit. Und wenn der " +"Quellcode des Kandidaten fehlschlägt, muss das Modell entscheiden, was das " +"Ergebnis für seinen nächsten Versuch bedeutet. Die Kenntnis des relevanten " +"Musters ist nur der Anfang dieses Prozesses." + +#: website/public/blog/reverse-engineering-models/index.html:468 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For that reason, I would include a smaller general model with good retrieval" +" in any comparison. Giving it the same library of experience would help us " +"identify what training adds: better candidate generation, better decisions " +"after feedback, or both." +msgstr "" +"Aus diesem Grund würde ich in jeden Vergleich ein kleineres allgemeines KI-" +"Modell mit gutem Abruf einbeziehen. Wenn wir ihm die gleiche " +"Erfahrungsbibliothek zur Verfügung stellen, können wir herausfinden, was die" +" Schulung bringt: bessere Kandidatengenerierung, bessere Entscheidungen nach" +" Feedback oder beides." + +#: website/public/blog/reverse-engineering-models/index.html:476 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Learning the next useful experiment" +msgstr "Lernen Sie das nächste nützliche Experiment" + +#: website/public/blog/reverse-engineering-models/index.html:477 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those decisions after feedback are what I would most like to study. In " +"agentic reconstruction, the model proposes an explanation and tests it " +"against the program. Even when the candidate fails, the result can narrow " +"the remaining possibilities and suggest a more useful next experiment." +msgstr "" +"Diese Entscheidungen nach Feedback sind das, was ich am liebsten studieren " +"würde. Bei der Agentenrekonstruktion schlägt das Modell eine Erklärung vor " +"und testet sie anhand des Programms. Selbst wenn der Quellcode des " +"Kandidaten fehlschlägt, kann das Ergebnis die verbleibenden Möglichkeiten " +"einschränken und ein nützlicheres nächstes Experiment vorschlagen." + +#: website/public/blog/reverse-engineering-models/index.html:484 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Consider the temporary-results example again. Its final source shows the " +"arrangement that worked, but the route to that source contains more " +"information. Each of the following experiments helped identify what still " +"needed to change." +msgstr "" +"Das Beispiel der temporären Ergebnisse zeigt auch das. Der endgültige " +"Quellcode enthält die Anordnung, die funktioniert hat. Der Weg dorthin " +"enthält jedoch zusätzliche Informationen: Jedes der folgenden Experimente " +"zeigte, was noch geändert werden musste." + +#: website/public/blog/reverse-engineering-models/index.html:497 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Experiment" +msgstr "Experimentieren" + +#: website/public/blog/reverse-engineering-models/index.html:498 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What its result taught us" +msgstr "Was uns das Ergebnis lehrte" + +#: website/public/blog/reverse-engineering-models/index.html:503 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Try a different way to copy the results" +msgstr "" +"Versuchen Sie es mit einer anderen Möglichkeit, die Ergebnisse zu kopieren" + +#: website/public/blog/reverse-engineering-models/index.html:504 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The copy code changes, but the function still differs." +msgstr "" +"Der Kopiercode ändert sich, aber die Funktion unterscheidet sich immer noch." + +#: website/public/blog/reverse-engineering-models/index.html:509 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Use the returned values directly" +msgstr "Verwenden Sie die zurückgegebenen Werte direkt" + +#: website/public/blog/reverse-engineering-models/index.html:510 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Copying follows the original pattern, but temporary storage still differs." +msgstr "" +"Das Kopieren erfolgt nach dem ursprünglichen Muster, die temporäre " +"Speicherung unterscheidet sich jedoch noch." + +#: website/public/blog/reverse-engineering-models/index.html:516 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the temporary results alive together" +msgstr "Halten Sie alle temporären Variablen gleichzeitig gültig" + +#: website/public/blog/reverse-engineering-models/index.html:517 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"They get separate storage, but one still overlaps a value used later for " +"collision checks." +msgstr "" +"Sie erhalten einen separaten Speicher, aber einer überlappt immer noch einen" +" Wert, der später für Kollisionsprüfungen verwendet wird." + +#: website/public/blog/reverse-engineering-models/index.html:523 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the collision-check value separate" +msgstr "Halten Sie den Wert für die Kollisionsprüfung getrennt" + +#: website/public/blog/reverse-engineering-models/index.html:524 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A clean rebuild confirms the complete match." +msgstr "" +"Eine saubere Neukompilierung aus dem Quellcode bestätigt die vollständige " +"Übereinstimmung." + +#: website/public/blog/reverse-engineering-models/index.html:529 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Selected experiments from the attack-function reconstruction above, grouped " +"by what they taught us." +msgstr "" +"Ausgewählte Experimente aus der Angriffsfunktionsrekonstruktion oben, " +"gruppiert nach dem, was sie uns gelehrt haben." + +#: website/public/blog/reverse-engineering-models/index.html:533 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A training record that keeps this sequence can teach more than the final " +"source alone. It connects each proposed change to an observed result and the" +" next decision. Keeping ineffective variations matters too: they tell a " +"later agent which hypotheses have already been tested in that context." +msgstr "" +"Eine Trainingsaufzeichnung, die diese Reihenfolge beibehält, kann mehr " +"lehren als die endgültige Quelle allein. Es verbindet jede vorgeschlagene " +"Änderung mit einem beobachteten Ergebnis und der nächsten Entscheidung. Es " +"ist auch wichtig, ineffektive Variationen beizubehalten: Sie teilen einem " +"späteren Agenten mit, welche Hypothesen in diesem Zusammenhang bereits " +"getestet wurden." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:540 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}I would want the model to learn how to use feedback.{ph1} A neutral " +"result is a reason to reconsider the variation; a changed mismatch may point" +" to another cause. When local edits stop being informative, the agent should" +" be able to revisit the source representation or inspect the evidence again." +msgstr "" +"{ph0}Ich möchte, dass das Modell lernt, Feedback zu nutzen.{ph1} Ein " +"neutrales Ergebnis ist ein Grund, die Variation noch einmal zu überdenken. " +"Eine veränderte Nichtübereinstimmung kann auf eine andere Ursache hinweisen." +" Wenn lokale Änderungen nicht mehr informativ sind, sollte der Agent in der " +"Lage sein, die Quellendarstellung noch einmal aufzurufen oder die Beweise " +"erneut zu prüfen." + +#: website/public/blog/reverse-engineering-models/index.html:549 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Learning to make those choices requires autonomy. Sometimes the useful next " +"step is inspecting another caller rather than compiling another edit. A " +"rigid sequence of required rewrites would remove that choice, along with " +"part of the behavior we want the model to learn." +msgstr "" +"Zu lernen, diese Entscheidungen zu treffen, erfordert Autonomie. Manchmal " +"besteht der nützliche nächste Schritt darin, einen anderen Aufrufer zu " +"überprüfen, anstatt eine weitere Bearbeitung zu kompilieren. Eine starre " +"Abfolge erforderlicher Umschreibungen würde diese Wahlmöglichkeit zusammen " +"mit einem Teil des Verhaltens, das das Modell lernen soll, aufheben." + +#: website/public/blog/reverse-engineering-models/index.html:558 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why RE is interesting for reinforcement learning" +msgstr "Warum Reverse Engineering für Reinforcement Learning interessant ist" + +#: website/public/blog/reverse-engineering-models/index.html:559 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Bounded reconstruction gives us a practical environment for studying these " +"choices. Once the target and compiler are pinned, many source experiments " +"produce quick, repeatable feedback. The agent can compile a candidate, " +"inspect the mismatch and use that observation to choose its next step." +msgstr "" +"Rekonstruktion mit klar abgegrenztem Umfang bietet eine praktische Umgebung," +" um diese Entscheidungen zu untersuchen. Bei festgelegtem Ziel und Compiler " +"liefern viele Quellcodeexperimente schnelles, wiederholbares Feedback. Der " +"Agent kann einen Kandidaten kompilieren, die Abweichungen prüfen und daraus " +"den nächsten Schritt wählen." + +#: website/public/blog/reverse-engineering-models/index.html:566 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reinforcement learning could train that decision-making process through the " +"feedback it receives. A candidate accepted by a defined check gives us a " +"concrete outcome, while the sequence of attempts lets us observe how much " +"work it took to reach it." +msgstr "" +"Reinforcement Learning könnte diesen Entscheidungsprozess durch das " +"erhaltene Feedback trainieren. Ein durch eine definierte Prüfung " +"angenommener Kandidat gibt uns ein konkretes Ergebnis, während wir anhand " +"der Abfolge der Versuche sehen können, wie viel Arbeit es gekostet hat, " +"dieses zu erreichen." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:572 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"There is already research supporting this direction. {ph0}LLM4Decompile{ph1}" +" studies models specialized for decompilation. {ph2}AutoDecompiler{ph3} " +"trains multi-turn refinement using compiler and execution feedback. Their " +"studies focus on functional recovery. For this work, I would use them as a " +"starting point for the related task of finding source that matches a " +"historical compiler's output." +msgstr "" +"Es gibt bereits Forschungsergebnisse, die diese Richtung unterstützen. " +"{ph0}LLM4Decompile{ph1} untersucht Modelle, die auf die Dekompilierung " +"spezialisiert sind. {ph2}AutoDecompiler{ph3} trainiert die Multi-Turn-" +"Verfeinerung mithilfe von Compiler- und Ausführungsfeedback. Der Schwerpunkt" +" ihrer Studien liegt auf der Wiederherstellung des Programmverhaltens. Für " +"diese Arbeit würde ich sie als Ausgangspunkt für die damit verbundene " +"Aufgabe verwenden, eine Quelle zu finden, die mit der Ausgabe eines " +"historischen Compilers übereinstimmt." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:584 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To reach such a well-defined task, though, we may first need to establish " +"what a function owns or devise a new runtime measurement. In TH08, for " +"example, a {ph0}collision difference led back to how the original program " +"handled floating-point calculations{ph1}. Before the agent could test a " +"useful correction, it had to work out what to measure." +msgstr "" +"Um eine so klar definierte Aufgabe zu erfüllen, müssen wir jedoch " +"möglicherweise zunächst feststellen, welche Ressourcen eine Funktion " +"besitzt, oder eine neue Methode zur Beobachtung des Programms während der " +"Ausführung entwickeln. In TH08 beispielsweise führte ein " +"{ph0}Kollisionsunterschied dazu, wie das ursprüngliche Programm " +"Gleitkommaberechnungen verarbeitete{ph1}. Bevor der Agent eine nützliche " +"Korrektur testen konnte, musste er herausfinden, was gemessen werden sollte." + +#: website/public/blog/reverse-engineering-models/index.html:595 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This suggests another role for a broadly capable model: helping make the " +"problem checkable in the first place. Once that check exists, we can " +"investigate how far a learned search policy can get with it." +msgstr "" +"Dies deutet auf eine weitere Rolle eines allgemein leistungsfähigen Modells " +"hin: Es hilft dabei, das Problem überhaupt überprüfbar zu machen. Sobald " +"diese Prüfung vorliegt, können wir untersuchen, wie weit eine erlernte " +"Suchrichtlinie damit kommen kann." + +#: website/public/blog/reverse-engineering-models/index.html:603 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "When the oracle is wrong" +msgstr "Wenn der Prüfer falsch ist" + +#: website/public/blog/reverse-engineering-models/index.html:604 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Once we train against that check, its coverage becomes part of what the " +"model learns. I call the reference or checker an oracle. Even if it gives a " +"repeatable answer, we still need to understand what that answer covers." +msgstr "" +"Wenn wir eine Prüfung für das Training verwenden, wird auch ihr " +"Prüfungsumfang Teil der Lernumgebung. Ich nenne eine solche Referenz oder " +"einen solchen Prüfer ein Oracle. Selbst eine wiederholbare Antwort müssen " +"wir danach beurteilen, was sie tatsächlich überprüft." + +#: website/public/blog/reverse-engineering-models/index.html:610 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"TH08 showed how much can hide outside that coverage. Some reconstructed " +"functions passed their exact comparisons with wrong floating constants. The " +"comparator adjusted relocated addresses before checking instructions, but it" +" checked the values behind those addresses only when they had been " +"explicitly recorded for verification. Other literal references could pass " +"without their values being examined." +msgstr "" +"TH08 hat gezeigt, wie viel sich außerhalb dieser Abdeckung verbergen kann. " +"Einige rekonstruierte Funktionen haben ihre exakten Vergleiche mit falschen " +"Gleitkommakonstanten bestanden. Der Komparator passte die verschobenen " +"Adressen an, bevor er die Anweisungen überprüfte, überprüfte die Werte " +"hinter diesen Adressen jedoch nur, wenn sie explizit zur Überprüfung " +"aufgezeichnet wurden. Andere wörtliche Verweise könnten ohne Prüfung ihrer " +"Werte bestehen bleiben." + +#. {ph0}: +#. {ph1}: +#. {ph2}: 1.0f +#. {ph3}: 32.0f +#: website/public/blog/reverse-engineering-models/index.html:619 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To close that gap, a {ph0}wider audit{ph1} found incorrect constants in " +"functions that had already been accepted. In one calculation, the " +"reconstructed source added {ph2} where the original added {ph3}." +msgstr "" +"Um diese Lücke zu schließen, wurden bei einer {ph0}umfassenderen " +"Prüfung{ph1} falsche Konstanten in Funktionen gefunden, die bereits die " +"Überprüfung bestanden hatten. In einer Berechnung fügte der rekonstruierte " +"Quellcode {ph2} hinzu, während das Original {ph3} hinzufügte." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:629 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A learner rewarded by the old check could learn to produce the wrong " +"reconstruction.{ph1} We had not trained an RL model against this oracle; the" +" observed failure was in the check itself. But using that check as a reward " +"would have made the same blind spot part of the learning environment." +msgstr "" +"{ph0}Ein Modell, das durch die alte Verifizierungsprüfung belohnt wurde, " +"könnte lernen, eine falsche Rekonstruktion zu erzeugen.{ph1} Wir haben mit " +"diesem Prüfer kein RL-Modell trainiert. Der von uns beobachtete Fehler lag " +"in der Prüfung selbst. Aber wenn wir es als Belohnung nutzen würden, würde " +"derselbe blinde Fleck Teil der Lernumgebung werden." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:639 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A functional test can have a similar blind spot. {ph0}Differential-fuzzing " +"research on LLM decompilers{ph1} has found behavior differences beyond the " +"supplied tests. Passing those tests therefore needs to be read in terms of " +"the cases they exercise, just as an instruction match needs to be read in " +"terms of the bytes and data it checks." +msgstr "" +"Ein Funktionstest kann einen ähnlichen blinden Fleck haben. " +"{ph0}Untersuchungen mit differenziellem Fuzzing auf LLM-Decompilern{ph1} " +"ergaben Verhaltensunterschiede außerhalb der bereitgestellten Tests. Wir " +"müssen daher einen bestandenen Test im Hinblick auf die Fälle verstehen, in " +"denen er durchgeführt wird. Ebenso muss eine Befehlsübereinstimmung im " +"Hinblick auf die Bytes und Daten verstanden werden, die sie überprüft." + +#: website/public/blog/reverse-engineering-models/index.html:649 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before using either kind of check as a reward, I would test it with " +"deliberate failures. I would also keep final acceptance outside the " +"candidate's editable source, so the model has to satisfy the check through " +"its reconstruction. Otherwise, something like a byte array copied from the " +"target could satisfy a naive score without recovering useful source." +msgstr "" +"Bevor ich eine Art Verifizierungsprüfung als Belohnung verwende, würde ich " +"sie mit absichtlich falschen Ergebnissen testen. Ich würde die Endabnahme " +"auch getrennt vom Quellcode durchführen, den der KI-Agent bearbeiten kann. " +"Das Modell muss die Verifizierung bestehen, indem es das Programm korrekt " +"rekonstruiert. Andernfalls könnte das einfache Kopieren eines Byte-Arrays " +"vom Ziel ein naives Bewertungssystem täuschen, ohne dass nützlicher " +"Quellcode wiederhergestellt wird." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:657 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Improving the learner and improving its oracle belong to the same " +"research problem.{ph1} As the learner gets better at optimizing the feedback" +" we give it, understanding that feedback becomes more important too." +msgstr "" +"{ph0}Die Verbesserung des Lernenden und die Verbesserung seines Prüfers " +"gehören zum selben Forschungsproblem.{ph1} Je besser der Lernende darin " +"wird, das Feedback, das wir ihm geben, zu optimieren, desto wichtiger wird " +"auch das Verständnis dieses Feedbacks." + +#: website/public/blog/reverse-engineering-models/index.html:667 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A comparison worth running" +msgstr "Ein Vergleich, der sich lohnt" + +#: website/public/blog/reverse-engineering-models/index.html:668 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"With those checks in place, I would start the comparison on a fixed VC7.1 " +"task family. First, I would establish baselines for a strong general model " +"and a smaller one. Adding repository knowledge to the smaller model would " +"show what retrieval changes. From there, supervised specialization and an " +"RL-trained version would let us examine what training adds." +msgstr "" +"Wenn diese Prüfungen vorhanden sind, würde ich den Vergleich für eine feste " +"VC7.1-Aufgabenfamilie starten. Zunächst würde ich Grundlinien für ein " +"starkes allgemeines KI-Modell und ein kleineres Modell festlegen. Das " +"Hinzufügen von Repository-Wissen zum kleineren Modell würde zeigen, welche " +"Änderungen sich beim Abruf ergeben. Von dort aus würden wir durch eine " +"überwachte Spezialisierung und eine RL-trainierte Version untersuchen " +"können, was Training hinzufügt." + +#: website/public/blog/reverse-engineering-models/index.html:676 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To make those comparisons informative, each model would receive the same " +"target evidence and tools. For the training-specific comparisons, I would " +"keep the base model and available knowledge fixed as well. Repeated trials " +"would help distinguish a reliable search strategy from a lucky candidate." +msgstr "" +"Um diese Vergleiche aussagekräftig zu machen, würde jedes Modell die " +"gleichen Zieldaten und Tools erhalten. Für die ausbildungsspezifischen " +"Vergleiche würde ich das Basismodell und das verfügbare Wissen ebenfalls " +"unverändert lassen. Wiederholte Versuche würden dabei helfen, eine " +"zuverlässige Suchstrategie von einer Kandidatenlösung zu unterscheiden, die " +"zufällig erfolgreich ist." + +#: website/public/blog/reverse-engineering-models/index.html:683 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would then separate two starting conditions: reconstructing a function " +"from its instructions, and closing the last few differences in maintained " +"source. A specialist might be excellent at the second task while a general " +"model does much of the work needed to reach it. Measuring them separately " +"would reveal that division of work." +msgstr "" +"Ich würde dann zwei Startbedingungen trennen: die Rekonstruktion einer " +"Funktion aus ihren Anweisungen und die Beseitigung der letzten paar " +"Unterschiede in der gepflegten Quelle. Bei der zweiten Aufgabe könnte ein " +"spezialisiertes KI-Modell hervorragend geeignet sein, während ein " +"allgemeines KI-Modell einen Großteil der dafür erforderlichen Arbeit " +"übernimmt. Eine getrennte Messung würde diese Arbeitsteilung offenbaren." + +#: website/public/blog/reverse-engineering-models/index.html:691 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both tasks need genuinely held-out examples. Because related Touhou games " +"share engine code, holding out a game's name would still allow nearly " +"identical functions to appear in training. I would instead hold out source " +"families and keep their accepted answers out of both training and retrieval." +msgstr "" +"Für beide Aufgaben sind Testbeispiele erforderlich, die vom Training und " +"Abruf ausgeschlossen sind. Verwandte Touhou-Spiele teilen den Engine-Code. " +"Der Ausschluss eines Spiels vom Training könnte dennoch dazu führen, dass " +"nahezu identische Funktionen im Trainingsset verbleiben. Stattdessen würde " +"ich ganze Gruppen verwandten Quellcodes ausschließen und ihre verifizierten " +"Antworten sowohl aus den Trainingsdaten als auch aus der Abrufbibliothek " +"heraushalten." + +#: website/public/blog/reverse-engineering-models/index.html:698 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"On those examples, the outcome would be complete, independently validated " +"matches. Giving each model the same candidate or compile budget would help " +"compare search quality. Measuring elapsed time and total cost, including " +"compilation, would tell us whether that quality produces an efficient " +"workflow. Training costs would need their own accounting too." +msgstr "" +"Das Ergebnis wären vollständige, unabhängig verifizierte Übereinstimmungen. " +"Gleiche Budgets für Kandidaten oder Kompilierungsversuche helfen, die " +"Suchqualität zu vergleichen. Gesamtzeit und Gesamtkosten einschließlich " +"Kompilierung zeigen, ob diese Qualität einen effizienten Arbeitsablauf " +"ermöglicht. Trainingskosten sollten separat erfasst werden." + +#: website/public/blog/reverse-engineering-models/index.html:706 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both views matter because a cheaper proposal can become an expensive " +"workflow if it causes more uninformative compiles. Conversely, a slower " +"model may pay for itself by finding a useful experiment sooner. The " +"comparison should capture those costs." +msgstr "" +"Beide Messungen sind wichtig. Ein billiger Vorschlag kann zu einem teuren " +"Arbeitsablauf führen, wenn er mehr Kompilierungen verursacht, die keine " +"nützlichen Informationen liefern. Umgekehrt kann ein langsameres Modell " +"wirtschaftlicher sein, wenn es früher ein nützliches Experiment findet. Der " +"Vergleich sollte diese Kosten einbeziehen." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:712 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Counting accepted results also keeps intermediate progress in perspective. " +"One TH09 {ph0}replay-save candidate{ph1} looked promising because the " +"compiled function had the expected size, but a full comparison still found " +"substantial differences. Changes in how the compiler arranges instructions " +"can also make a candidate look closer without resolving the underlying " +"problem. Such diagnostics can guide the next experiment, but I would " +"validate any reward for \"getting closer\" against the full acceptance " +"check." +msgstr "" +"Akzeptierte Ergebnisse zu zählen hilft auch, Zwischenfortschritte " +"einzuordnen. Ein {ph0}Replay-Save-Kandidat{ph1} aus TH09 wirkte " +"vielversprechend, weil die kompilierte Funktion die erwartete Größe hatte. " +"Der vollständige Vergleich fand aber noch erhebliche Unterschiede. Eine " +"andere Anordnung der Befehle kann einen Kandidaten näher am Ziel erscheinen " +"lassen, ohne das eigentliche Problem zu lösen. Solche Diagnosen können das " +"nächste Experiment leiten; eine Belohnung für mehr Nähe würde ich jedoch " +"anhand der vollständigen Abnahmeprüfung validieren." + +#: website/public/blog/reverse-engineering-models/index.html:727 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would be interested in the pattern across these results. If a specialist " +"does well within a familiar compiler family, does that advantage survive a " +"toolchain change? And once both models have good retrieval, how much does " +"training still add? The answers would help us decide where each kind of " +"model is most useful." +msgstr "" +"Mich würde das Muster dieser Ergebnisse interessieren. Wenn ein " +"spezialisiertes KI-Modell in einer bekannten Compilerfamilie gut " +"funktioniert, überlebt dieser Vorteil dann auch eine Änderung der Toolchain?" +" Und wenn beide KI-Modelle eine gute Retrievalfähigkeit haben, wie viel " +"bringt das Training dann noch? Die Antworten würden uns helfen zu " +"entscheiden, wo welche Art von Modell am nützlichsten ist." + +#: website/public/blog/reverse-engineering-models/index.html:736 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What kind of agent does this suggest?" +msgstr "Was für einen KI-Agenten lässt dies vermuten?" + +#: website/public/blog/reverse-engineering-models/index.html:737 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"If the models turn out to help at different stages, a useful arrangement " +"might be a broad agent with access to a specialist. Once a task becomes a " +"familiar compiler search, it could ask the specialist for candidates. If the" +" results stop making sense, the broader agent could return to the target " +"evidence or investigate the measurement itself." +msgstr "" +"Wenn sich herausstellt, dass die Modelle in verschiedenen Phasen hilfreich " +"sind, könnte ein umfassender Agent mit Zugriff auf ein spezielles KI-Modell " +"eine nützliche Lösung sein. Sobald eine Aufgabe zu einer vertrauten " +"Compiler-Suche wird, könnte sie das spezialisierte KI-Modell nach Kandidaten" +" fragen. Wenn die Ergebnisse keinen Sinn mehr ergeben, könnte der breitere " +"Agent zum Zielbeweis zurückkehren oder die Messung selbst untersuchen." + +#: website/public/blog/reverse-engineering-models/index.html:745 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In that arrangement, specialization becomes a tool the agent can choose as " +"the investigation develops. The agent keeps the freedom to change direction " +"that made the earlier breakthroughs possible." +msgstr "" +"In dieser Vereinbarung wird die Spezialisierung zu einem Instrument, das der" +" Agent im Verlauf der Untersuchung wählen kann. Der Agent behält die " +"Freiheit, die Richtung zu ändern, die die früheren Durchbrüche ermöglicht " +"hat." + +#: website/public/blog/reverse-engineering-models/index.html:750 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This follows naturally from the industrial shift I described in the previous" +" essay. Being able to run many more experiments changes how we organize the " +"work. It also gives us a reason to ask which decisions need broad judgment " +"and which improve through repeated experience with a particular craft." +msgstr "" +"Dies ergibt sich natürlich aus dem industriellen Wandel, den ich im " +"vorherigen Aufsatz beschrieben habe. Die Möglichkeit, viel mehr Experimente " +"durchführen zu können, verändert die Art und Weise, wie wir die Arbeit " +"organisieren. Es gibt uns auch einen Grund zu fragen, welche Entscheidungen " +"ein breites Urteilsvermögen erfordern und welche sich durch wiederholte " +"Erfahrung mit einem bestimmten Handwerk verbessern." + +#: website/public/blog/reverse-engineering-models/index.html:757 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For now, the strongest general model has earned its place in my workflow by " +"opening problems I had left unresolved. Those successes are what make me " +"want to examine the work more closely: how much depended on the model's " +"breadth, and how much could be handled by a model trained in this particular" +" kind of reconstruction?" +msgstr "" +"Vorerst hat sich das stärkste allgemeine KI-Modell seinen Platz in meinem " +"Arbeitsablauf verdient, indem es Probleme löste, die ich ungelöst gelassen " +"hatte. Gerade diese Erfolge machen mich neugierig: Wie viel hing von seiner " +"breiten Fähigkeit ab? Wie viel könnte ein Modell übernehmen, das auf diese " +"Art der Rekonstruktion spezialisiert ist?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:765 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}The repositories give us a way to test the question.{ph1} I would like " +"to use that record to build the comparison and see what we learn." +msgstr "" +"{ph0}Die Repositorys bieten uns eine Möglichkeit, die Frage zu testen.{ph1} " +"Ich möchte diesen Datensatz verwenden, um den Vergleich zu erstellen und zu " +"sehen, was wir lernen." + +#: website/public/blog/reverse-engineering-models/index.html:775 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Evidence and model attribution" +msgstr "Beweise und Modellzuschreibung" + +#: website/public/blog/reverse-engineering-models/index.html:783 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The examples describe historical function-level comparisons recorded in the " +"linked commits. Whole-game runtime checks are separate milestones. Source " +"and matching records were reviewed on 13 October 2026." +msgstr "" +"Die Beispiele beschreiben historische Vergleiche auf Funktionsebene, die in " +"den verknüpften Commits aufgezeichnet wurden. Laufzeitprüfungen für das " +"gesamte Spiel sind separate Meilensteine. Quelle und übereinstimmende " +"Aufzeichnungen wurden am 13. Oktober 2026 überprüft." + +#. {ph0}: gpt-dots +#. {ph1}: gpt-web +#: website/public/blog/reverse-engineering-models/index.html:789 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The TH09 label {ph0} identifies GPT-6-Astra. {ph1} changed with the model " +"used at the time, so that label alone does not identify an individual run. " +"Explicit model names in the other cited commits establish their recorded " +"attribution; the histories do not contain uniform per-run cost or effort " +"measurements." +msgstr "" +"Die Bezeichnung TH09 {ph0} bedeutet GPT-6-Astra. Das mit {ph1} verknüpfte " +"Modell hat sich im Laufe der Zeit geändert, sodass die Bezeichnung allein " +"das in einem bestimmten Lauf verwendete Modell nicht identifizieren kann. " +"Explizite Modellnamen in den anderen zitierten Commits liefern die " +"aufgezeichnete Zuschreibung. Diese Historien enthalten keine konsistenten " +"Messungen der Kosten oder des Aufwands für jeden Lauf." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#: website/public/blog/reverse-engineering-models/index.html:798 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 temporary lifetimes:{ph1} the {ph2}earlier context review{ph3} and" +" {ph4}complete cold reconstruction{ph5}." +msgstr "" +"{ph0}TH09 temporäre Objektlebensdauern:{ph1} die {ph2}frühere " +"Kontextüberprüfung{ph3} und {ph4}vollständige Rekonstruktion, verifiziert " +"durch eine saubere Neukompilierung aus der Quelle{ph5}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:810 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 Front callback:{ph1} {ph2}non-exact diagnostic regions{ph3}, " +"{ph4}Astra's new graph{ph5} and its {ph6}accepted recompile replay{ph7}." +msgstr "" +"{ph0}TH09 Front-Callback:{ph1} {ph2}Diagnosebereiche, die nicht " +"übereinstimmten{ph3}, {ph4}Astras neues Kontrollflussdiagramm{ph5} und seine" +" {ph6}akzeptierte Neukompilierungsprüfung{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:826 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Compiler knowledge:{ph1} {ph2}TH095 identifier-bucket reuse{ph3}, " +"{ph4}TH04 player source and linker order{ph5} and {ph6}Luna's TH09 enemy-" +"spawn closure{ph7}." +msgstr "" +"{ph0}Compiler-Kenntnisse:{ph1} {ph2}TH095 Wiederverwendung des Identifier-" +"Bucket-Verhaltens{ph3}, {ph4}TH04 Player-Quellcode und Linker-" +"Reihenfolge{ph5} und {ph6}Lunas vollständige Rekonstruktion der " +"Feinderschaffungsfunktion von TH09{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:842 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Checks and diagnostics:{ph1} {ph2}TH08 floating-literal audit{ph3}, " +"{ph4}TH08 numerical investigation{ph5} and {ph6}TH09 replay-save " +"reconstruction{ph7}." +msgstr "" +"{ph0}Prüfungen und Diagnosen:{ph1} {ph2}TH08-Prüfung der " +"Gleitkommakonstanten{ph3}, {ph4}TH08-Untersuchung numerischer " +"Unterschiede{ph5} und {ph6}TH09-Rekonstruktion der Replay-Speicherung{ph7}." + +#: website/public/blog/reverse-engineering-models/index.html:859 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The proposed specialist and RL comparisons are future experiments. The " +"project histories motivate them; no trained specialist was evaluated for " +"this essay." +msgstr "" +"Die vorgeschlagenen Vergleiche spezialisierter Modelle und RL-Training sind " +"zukünftige Experimente. Die Projektgeschichten motivieren sie. Für diesen " +"Aufsatz wurde kein trainiertes Spezialmodell evaluiert." + +#: website/public/blog/reverse-engineering-models/index.html:6 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model? | REA" +msgstr "Braucht Reverse Engineering ein größeres Modell? | REA" + +#: website/public/blog/reverse-engineering-models/index.html:7 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What Touhou reconstruction taught me about general models, specialist models" +" and learning to propose better hypotheses from compiler feedback." +msgstr "" +"Was ich durch die Rekonstruktion von Touhou über allgemeine Modelle, " +"Spezialmodelle und das Lernen, bessere Hypothesen anhand des Compiler-" +"Feedbacks aufzustellen, gelernt habe." + +#: website/public/blog/reverse-engineering-models/index.html:240 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "On this page" +msgstr "Auf dieser Seite" + +#: website/public/blog/reverse-engineering-models/index.html:253 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Models and reverse engineering essay" +msgstr "Essay über Modelle und Reverse Engineering" + +#: website/public/blog/reverse-engineering-models/index.html:491 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Selected reconstruction experiments" +msgstr "Ausgewählte Rekonstruktionsexperimente" + +#: website/public/blog/reverse-engineering-models/index.html:883 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Back to top" +msgstr "Nach oben" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:776 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Detailed evidence notes{ph1} include the recorded comparisons and " +"experiment context." +msgstr "" +"Die {ph0}ausführlichen Evidenznotizen{ph1} enthalten die aufgezeichneten " +"Vergleiche und den Kontext der Experimente." + +#~ msgctxt "page:blog/index.html" +#~ msgid "By N0zoM1z0 · 13 October 2026 · Draft for review" +#~ msgstr "Von N0zoM1z0 · 13. Oktober 2026 · Entwurf zur Durchsicht" + #~ msgctxt "page:index.html" #~ msgid "Setup guide" #~ msgstr "Einrichtungsanleitung" diff --git a/website/locales/es/messages.po b/website/locales/es/messages.po index a9d6e8fb6..bc68e0b74 100644 --- a/website/locales/es/messages.po +++ b/website/locales/es/messages.po @@ -32,18 +32,18 @@ msgid "Thoughts on reverse engineering." msgstr "Reflexiones sobre ingeniería inversa." #. {ph0}:
-#: website/public/blog/index.html:200 +#: website/public/blog/index.html:214 #, python-brace-format msgctxt "page:blog/index.html" msgid "Game reconstruction{ph0}Essay" msgstr "Reconstrucción del juego{ph0}Ensayo" -#: website/public/blog/index.html:202 +#: website/public/blog/index.html:216 msgctxt "page:blog/index.html" msgid "The industrial revolution in reverse engineering" msgstr "La revolución industrial en ingeniería inversa" -#: website/public/blog/index.html:203 +#: website/public/blog/index.html:217 msgctxt "page:blog/index.html" msgid "" "What manual RE and reconstructing TH08, TH095 and TH04 taught me about " @@ -53,25 +53,26 @@ msgstr "" "enseñó sobre los agentes, la verificación y el conocimiento que componen los" " proyectos." -#: website/public/blog/index.html:208 +#: website/public/blog/index.html:207 website/public/blog/index.html:222 msgctxt "page:blog/index.html" msgid "Read the article" msgstr "Leer el artículo" -#: website/public/blog/index.html:211 +#: website/public/blog/index.html:225 msgctxt "page:blog/index.html" msgid "By N0zoM1z0 · 10 October 2026" msgstr "Por N0zoM1z0 * 10 octubre 2026" #. {ph0}: -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 +#: website/public/blog/reverse-engineering-models/index.html:883 #: website/public/blog/touhou-reconstruction/index.html:1080 #: website/public/examples/dino-lab/index.html:475 #: website/public/faq/index.html:573 #: website/public/first-investigation/index.html:523 #: website/public/get-started/index.html:882 -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 #: website/public/guides/index.html:282 #: website/public/guides/javascript/index.html:503 #: website/public/guides/native/index.html:509 website/public/index.html:1191 @@ -101,6 +102,8 @@ msgstr "" "con REA y agentes de IA." #: website/public/blog/index.html:75 website/public/blog/index.html:88 +#: website/public/blog/reverse-engineering-models/index.html:75 +#: website/public/blog/reverse-engineering-models/index.html:88 #: website/public/blog/touhou-reconstruction/index.html:75 #: website/public/blog/touhou-reconstruction/index.html:88 #: website/public/examples/dino-lab/index.html:75 @@ -137,6 +140,7 @@ msgstr "" "de codificación." #: website/public/blog/index.html:108 +#: website/public/blog/reverse-engineering-models/index.html:108 #: website/public/blog/touhou-reconstruction/index.html:108 #: website/public/examples/dino-lab/index.html:109 #: website/public/faq/index.html:108 @@ -157,6 +161,7 @@ msgid "REA home" msgstr "Inicio de REA" #: website/public/blog/index.html:112 +#: website/public/blog/reverse-engineering-models/index.html:112 #: website/public/blog/touhou-reconstruction/index.html:112 #: website/public/examples/dino-lab/index.html:113 #: website/public/faq/index.html:112 @@ -177,6 +182,7 @@ msgid "Main navigation" msgstr "Navegación principal" #: website/public/blog/index.html:130 +#: website/public/blog/reverse-engineering-models/index.html:130 #: website/public/blog/touhou-reconstruction/index.html:130 #: website/public/examples/dino-lab/index.html:131 #: website/public/faq/index.html:130 @@ -197,6 +203,7 @@ msgid "REA on GitHub" msgstr "REA en GitHub" #: website/public/blog/index.html:150 +#: website/public/blog/reverse-engineering-models/index.html:150 #: website/public/blog/touhou-reconstruction/index.html:150 #: website/public/examples/dino-lab/index.html:151 #: website/public/faq/index.html:150 @@ -217,6 +224,7 @@ msgid "Join REA on Discord" msgstr "Únete a REA en Discord" #: website/public/blog/index.html:169 +#: website/public/blog/reverse-engineering-models/index.html:169 #: website/public/blog/touhou-reconstruction/index.html:169 #: website/public/examples/dino-lab/index.html:170 #: website/public/faq/index.html:169 @@ -236,7 +244,7 @@ msgctxt "shared" msgid "Follow REA on X" msgstr "Sigue a REA en X" -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 msgctxt "page:blog/index.html" msgid "Back to top" msgstr "Volver arriba" @@ -4827,7 +4835,7 @@ msgctxt "page:guides/browser/index.html" msgid "Copy read the loaded scripts" msgstr "Copiar leer los scripts cargados" -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 msgctxt "page:guides/browser/index.html" msgid "Back to top" msgstr "Volver arriba" @@ -11764,6 +11772,7 @@ msgid "Same spacing: 256 ÷ 16 = 16 angle units = 22.5°" msgstr "Mismo espaciado: 256 ÷ 16 = 16 unidades de ángulo = 22,5°" #: website/public/blog/index.html:114 +#: website/public/blog/reverse-engineering-models/index.html:114 #: website/public/blog/touhou-reconstruction/index.html:114 #: website/public/examples/dino-lab/index.html:115 #: website/public/faq/index.html:114 @@ -11784,6 +11793,7 @@ msgid "Showcases" msgstr "Casos prácticos" #: website/public/blog/index.html:115 +#: website/public/blog/reverse-engineering-models/index.html:115 #: website/public/blog/touhou-reconstruction/index.html:115 #: website/public/examples/dino-lab/index.html:116 #: website/public/faq/index.html:115 @@ -11804,6 +11814,7 @@ msgid "Blog" msgstr "Blog" #: website/public/blog/index.html:113 +#: website/public/blog/reverse-engineering-models/index.html:113 #: website/public/blog/touhou-reconstruction/index.html:113 #: website/public/examples/dino-lab/index.html:114 #: website/public/faq/index.html:113 @@ -11824,6 +11835,7 @@ msgid "Get started" msgstr "Primeros pasos" #: website/public/blog/index.html:116 +#: website/public/blog/reverse-engineering-models/index.html:116 #: website/public/blog/touhou-reconstruction/index.html:116 #: website/public/examples/dino-lab/index.html:117 #: website/public/faq/index.html:116 @@ -11843,13 +11855,14 @@ msgctxt "shared" msgid "Guides" msgstr "Guías" -#: website/public/blog/index.html:222 +#: website/public/blog/index.html:236 +#: website/public/blog/reverse-engineering-models/index.html:875 #: website/public/blog/touhou-reconstruction/index.html:1072 #: website/public/examples/dino-lab/index.html:467 #: website/public/faq/index.html:565 #: website/public/first-investigation/index.html:515 #: website/public/get-started/index.html:874 -#: website/public/guides/browser/index.html:490 +#: website/public/guides/browser/index.html:499 #: website/public/guides/index.html:274 #: website/public/guides/javascript/index.html:495 #: website/public/guides/native/index.html:501 website/public/index.html:1183 @@ -11863,13 +11876,14 @@ msgctxt "shared" msgid "Source" msgstr "Código fuente" -#: website/public/blog/index.html:223 +#: website/public/blog/index.html:237 +#: website/public/blog/reverse-engineering-models/index.html:876 #: website/public/blog/touhou-reconstruction/index.html:1073 #: website/public/examples/dino-lab/index.html:468 #: website/public/faq/index.html:566 #: website/public/first-investigation/index.html:516 #: website/public/get-started/index.html:875 -#: website/public/guides/browser/index.html:491 +#: website/public/guides/browser/index.html:500 #: website/public/guides/index.html:275 #: website/public/guides/javascript/index.html:496 #: website/public/guides/native/index.html:502 website/public/index.html:1184 @@ -11883,13 +11897,14 @@ msgctxt "shared" msgid "MIT license" msgstr "Licencia MIT" -#: website/public/blog/index.html:224 +#: website/public/blog/index.html:238 +#: website/public/blog/reverse-engineering-models/index.html:877 #: website/public/blog/touhou-reconstruction/index.html:1074 #: website/public/examples/dino-lab/index.html:469 #: website/public/faq/index.html:567 #: website/public/first-investigation/index.html:517 #: website/public/get-started/index.html:876 -#: website/public/guides/browser/index.html:492 +#: website/public/guides/browser/index.html:501 #: website/public/guides/index.html:276 #: website/public/guides/javascript/index.html:497 #: website/public/guides/native/index.html:503 website/public/index.html:1185 @@ -11903,13 +11918,14 @@ msgctxt "shared" msgid "Join the community" msgstr "Únete a la comunidad" -#: website/public/blog/index.html:225 +#: website/public/blog/index.html:239 +#: website/public/blog/reverse-engineering-models/index.html:878 #: website/public/blog/touhou-reconstruction/index.html:1075 #: website/public/examples/dino-lab/index.html:470 #: website/public/faq/index.html:568 #: website/public/first-investigation/index.html:518 #: website/public/get-started/index.html:877 -#: website/public/guides/browser/index.html:493 +#: website/public/guides/browser/index.html:502 #: website/public/guides/index.html:277 #: website/public/guides/javascript/index.html:498 #: website/public/guides/native/index.html:504 website/public/index.html:1186 @@ -12853,6 +12869,1418 @@ msgctxt "runtime" msgid "Switch to dark mode" msgstr "Cambiar al modo oscuro" +#. {ph0}: +#. {ph1}: +#: website/public/guides/browser/index.html:480 +#, python-brace-format +msgctxt "page:guides/browser/index.html" +msgid "" +"For HTTP request and response bodies, use {ph0}Fiddler Everywhere: SAZ " +"captures, MCP and public skills{ph1}. Export SAZ or HAR 1.2 for offline REA " +"capture Evidence, or connect Everywhere's MCP server to inspect captured " +"traffic with your agent." +msgstr "" +"Para examinar los cuerpos de las solicitudes y respuestas HTTP, use " +"{ph0}Fiddler Everywhere: capturas SAZ, MCP y habilidades públicas{ph1}. " +"Exporte SAZ o HAR 1.2 para obtener Evidence de capturas en REA sin conexión," +" o conecte el servidor MCP de Everywhere para inspeccionar el tráfico " +"capturado con su agente." + +#. {ph0}:
+#: website/public/blog/index.html:200 +#, python-brace-format +msgctxt "page:blog/index.html" +msgid "Model specialization{ph0}Essay" +msgstr "Modelos especializados{ph0}Ensayo" + +#: website/public/blog/index.html:202 +msgctxt "page:blog/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "¿Necesita la ingeniería inversa un modelo más grande?" + +#: website/public/blog/index.html:203 +msgctxt "page:blog/index.html" +msgid "" +"What Touhou reconstruction suggests about general models, specialist models " +"and learning from compiler feedback." +msgstr "" +"Lo que la reconstrucción de Touhou sugiere sobre los modelos generales, los " +"especializados y el aprendizaje a partir de los resultados del compilador." + +#: website/public/blog/index.html:210 +msgctxt "page:blog/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Por N0zoM1z0 · 13 de octubre de 2026" + +#: website/public/blog/reverse-engineering-models/index.html:103 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Skip to content" +msgstr "Saltar al contenido" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:194 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Home{ph1} / {ph2}Blog{ph3} / Models and RE" +msgstr "{ph0}Inicio{ph1} / {ph2}Blog{ph3} / Modelos e ingeniería inversa" + +#: website/public/blog/reverse-engineering-models/index.html:197 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Essay" +msgstr "ensayo" + +# Reviewed for technical meaning and readability; preserve the distinction +# between historical evidence and proposed experiments. +#: website/public/blog/reverse-engineering-models/index.html:198 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "¿La ingeniería inversa necesita un modelo más grande?" + +#: website/public/blog/reverse-engineering-models/index.html:199 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Por N0zoM1z0 · 13 de octubre de 2026" + +#: website/public/blog/reverse-engineering-models/index.html:201 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Takeaway" +msgstr "Ideas clave" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:203 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Model capability changes the hypotheses an agent can find.{ph1} Later " +"sessions have solved problems that earlier work left unresolved." +msgstr "" +"{ph0}La capacidad del modelo afecta qué hipótesis puede proponer un agente " +"de IA.{ph1} Las sesiones posteriores resolvieron problemas que el trabajo " +"anterior había dejado sin resolver." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:211 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A familiar compiler gives specialization something concrete to " +"learn.{ph1} Some exact matches depend on narrow, reusable source patterns." +msgstr "" +"{ph0}Un compilador familiar ofrece patrones concretos para que los aprenda " +"un modelo especializado.{ph1} Algunas coincidencias exactas de código de " +"máquina dependen de patrones de código fuente específicos y reutilizables." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:218 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Give repository knowledge a fair comparison.{ph1} A smaller model with " +"good retrieval may already know enough to make progress." +msgstr "" +"{ph0}Incluya el conocimiento del repositorio en una comparación justa.{ph1} " +"Es posible que un modelo más pequeño con una buena recuperación de " +"conocimiento ya tenga suficiente información para avanzar." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:223 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Train the next useful experiment, and test its oracle.{ph1} The value " +"of feedback depends on what the checker actually verifies." +msgstr "" +"{ph0}Entrene el modelo para elegir el siguiente experimento útil y pruebe su" +" verificador.{ph1} La retroalimentación es útil solo en la medida en que el " +"verificador verifica lo correcto." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:230 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Measure accepted reconstruction against time and cost.{ph1} That is the" +" result I want from the whole agent." +msgstr "" +"{ph0}Evaluar los resultados de reconstrucción verificados junto con el " +"tiempo y el costo.{ph1} Eso es lo que quiero medir para todo el flujo de " +"trabajo del agente de IA." + +#: website/public/blog/reverse-engineering-models/index.html:241 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "In this article" +msgstr "En este artículo" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:243 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What better models changed{ph1}" +msgstr "{ph0}Qué cambiaron los modelos más capaces{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:244 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Why specialize?{ph1}" +msgstr "{ph0}¿Por qué especializarse?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:245 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Where experience lives{ph1}" +msgstr "{ph0}¿Dónde debería almacenarse esta experiencia?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:246 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Learning the next experiment{ph1}" +msgstr "{ph0}Aprendiendo el próximo experimento útil{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:247 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}RE and reinforcement learning{ph1}" +msgstr "{ph0}Ingeniería inversa y aprendizaje por refuerzo{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:248 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}When the oracle is wrong{ph1}" +msgstr "{ph0}Cuando el verificador está equivocado{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:249 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}A comparison worth running{ph1}" +msgstr "{ph0}Una comparación que vale la pena realizar{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:250 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What kind of agent?{ph1}" +msgstr "{ph0}¿Qué tipo de agente de IA sugiere esto?{ph1}" + +#: website/public/blog/reverse-engineering-models/index.html:258 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What better models changed" +msgstr "Qué cambiaron los modelos más capaces" + +#: website/public/blog/reverse-engineering-models/index.html:259 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Since August 2026, I have been exploring how much reverse engineering an " +"agent can carry out with access to a program and a way to check its work. " +"Most of my experience comes from reconstructing games, especially Touhou. As" +" I have changed models, the repositories have kept a record of what each " +"session managed to recover." +msgstr "" +"Desde agosto de 2026, he estado explorando cuánta ingeniería inversa puede " +"realizar un agente con acceso a un programa y una forma de comprobar su " +"funcionamiento. La mayor parte de mi experiencia proviene de la " +"reconstrucción de juegos, especialmente Touhou. A medida que he ido " +"cambiando de modelo, los repositorios han ido manteniendo un registro de lo " +"que cada sesión logró recuperar." + +#: website/public/blog/reverse-engineering-models/index.html:267 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of that work aims at exact reconstruction: we compile the maintained " +"source under a pinned historical toolchain and check whether it reproduces " +"the target function. When a later model succeeds where earlier work stalled," +" the source changes show what made that progress possible." +msgstr "" +"Gran parte de ese trabajo apunta a una reconstrucción exacta: compilamos la " +"fuente mantenida bajo una cadena de herramientas histórica con versiones " +"fijas y comprobamos si reproduce la función de destino. Cuando un modelo " +"posterior tiene éxito donde el trabajo anterior se estancó, los cambios en " +"las fuentes muestran qué hizo posible ese progreso." + +#: website/public/blog/reverse-engineering-models/index.html:274 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The temporary values had to stay alive together" +msgstr "Los valores temporales debían seguir siendo válidos al mismo tiempo" + +#: website/public/blog/reverse-engineering-models/index.html:275 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of a TH09 attack function's logic had already been reconstructed, but " +"the compiler still stored temporary results differently from the original. " +"The remaining problem was how long those results needed to stay alive and " +"when their storage could be reused." +msgstr "" +"Gran parte de la lógica de la función de ataque TH09 ya había sido " +"reconstruida, pero el compilador aún almacenaba resultados temporales de " +"manera diferente al original. El problema restante era cuánto tiempo " +"necesitaban esos resultados para permanecer vivos y cuándo se podía " +"reutilizar su almacenamiento." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:282 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A {ph0}GPT-6-Sol checkpoint{ph1} recorded this mismatch after testing " +"several ordinary source arrangements. When {ph2}GPT-6.1-Sol returned to the " +"problem{ph3}, it found a useful hypothesis about how long those temporary " +"values needed to stay alive." +msgstr "" +"Un registro de sesión {ph0}GPT-6-Sol{ph1} informó esta discrepancia después " +"de probar varias disposiciones de código fuente ordinarias. Cuando " +"{ph2}GPT-6.1-Sol examinó el problema nuevamente{ph3}, propuso una hipótesis " +"útil sobre la vida útil de estas variables temporales." + +#: website/public/blog/reverse-engineering-models/index.html:296 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The function produced temporary results and copied them into animation " +"positions. In C++, a temporary result stays alive until the expression using" +" it finishes. Grouping the real operations into one expression kept those " +"results alive together, so the compiler could no longer reuse the same " +"storage for all of them." +msgstr "" +"La función produjo resultados temporales y los copió en posiciones de " +"animación. En C++, un resultado temporal permanece vivo hasta que finaliza " +"la expresión que lo utiliza. Agrupar las operaciones reales en una expresión" +" mantuvo esos resultados vivos juntos, por lo que el compilador ya no pudo " +"reutilizar el mismo almacenamiento para todos ellos." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:303 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"One result still shared storage with a value used later for collision " +"checks. Giving that value its own lifetime resolved the remaining overlap. " +"After several targeted experiments, {ph0}a clean rebuild confirmed that the " +"full function matched{ph1}." +msgstr "" +"Un resultado aún compartía almacenamiento con un valor utilizado más " +"adelante para comprobaciones de colisiones. Darle a ese valor su propia vida" +" útil resolvió la superposición restante. Después de varios experimentos " +"específicos, {ph0}una recompilación limpia del código fuente confirmó que la" +" función completa coincidía{ph1}." + +#: website/public/blog/reverse-engineering-models/index.html:312 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What interests me is how the discovery and the tests worked together. The " +"lifetime hypothesis gave the agent a new direction; compiler feedback helped" +" it refine that hypothesis into a source arrangement that matched." +msgstr "" +"Me interesa saber cómo el descubrimiento y las pruebas se apoyaron " +"mutuamente. La hipótesis sobre la vida temporal de los objetos dio al agente" +" de IA una nueva dirección. Los comentarios del compilador le ayudaron a " +"refinar esa hipótesis hasta convertirla en una disposición de código fuente " +"que coincidiera." + +#: website/public/blog/reverse-engineering-models/index.html:318 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A different control-flow graph" +msgstr "Un gráfico de flujo de control diferente" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:319 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A second TH09 function shows why that new direction can matter. Here, the " +"obstacle was how several branches shared their final work. " +"{ph0}GPT-6.1-Sol's earlier work{ph1} got parts of the function to match, but" +" the branches still joined up differently from the original." +msgstr "" +"Una segunda función TH09 muestra por qué esa nueva dirección puede ser " +"importante. Aquí el obstáculo fue cómo varias ramas compartieron su trabajo " +"final. El trabajo anterior de {ph0}GPT-6.1-Sol{ph1} consiguió que partes de " +"la función coincidieran, pero las ramas aún se unían de manera diferente al " +"original." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:330 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}GPT-6-Astra changed the surrounding control-flow graph{ph1}. The main " +"paths now led to a shared final update and return. In that source structure," +" the compiler combined the branches in the same way as the original. The " +"complete function matched, and a {ph2}separate rebuild check confirmed " +"it{ph3}." +msgstr "" +"{ph0}GPT-6-Astra cambió el gráfico de flujo de control circundante{ph1}. Los" +" caminos principales conducían ahora a una actualización final compartida y " +"al regreso. En esa estructura fuente, el compilador combinó las ramas de la " +"misma manera que el original. La función completa coincidió y una " +"{ph2}verificación de reconstrucción separada lo confirmó{ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:342 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Changing the representation opened a more productive search space.{ph1}" +" Both cases give me a reason to take general model capability seriously: an " +"agent can keep trying local variations for a long time before finding a " +"better way to express the surrounding work." +msgstr "" +"{ph0}Cambiar la representación abrió un espacio de búsqueda más " +"productivo.{ph1} Ambos casos me dan una razón para tomar en serio la " +"capacidad del modelo general: un agente puede seguir probando variaciones " +"locales durante mucho tiempo antes de encontrar una mejor manera de expresar" +" el trabajo circundante." + +#: website/public/blog/reverse-engineering-models/index.html:351 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"At the same time, the later sessions started with earlier source and " +"recorded failures. That accumulated work was part of their advantage. The " +"history lets us inspect the discoveries and their checks; a controlled " +"comparison would help separate the effect of the model from the experience " +"it inherited." +msgstr "" +"Las sesiones posteriores también heredaron el código anterior y los intentos" +" fallidos documentados. Esa experiencia acumulada formaba parte de su " +"ventaja. El historial permite examinar los hallazgos y sus comprobaciones; " +"una comparación controlada ayudaría a separar el efecto del modelo del de la" +" experiencia que recibió." + +#: website/public/blog/reverse-engineering-models/index.html:358 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is where the question becomes interesting to me. Some of the progress " +"came from finding a new representation, while much of the remaining work " +"involved testing source forms against a familiar compiler. How much needs " +"broad reasoning ability, and how much could a model learn through repeated " +"experience with that compiler and code?" +msgstr "" +"Esto es lo que hace que la pregunta me resulte interesante. Algunos avances " +"se produjeron al encontrar una nueva representación. Gran parte del trabajo " +"restante implicó probar diferentes disposiciones de código fuente con un " +"compilador familiar. ¿Cuánto de esto requiere una amplia capacidad de " +"razonamiento? ¿Cuánto podría aprender un modelo trabajando repetidamente con" +" ese compilador y código?" + +#: website/public/blog/reverse-engineering-models/index.html:368 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why might a specialist do well?" +msgstr "¿Por qué podría funcionar bien un modelo de IA especializado?" + +#: website/public/blog/reverse-engineering-models/index.html:369 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Touhou gives us a concrete reason to expect that experience to help. Much of" +" the game code comes from the same author, ZUN, so recurring habits are a " +"reasonable starting assumption. The repositories back that up with useful " +"examples: related loops and helper arrangements have supplied hypotheses " +"across games." +msgstr "" +"Touhou nos da una razón concreta para esperar que esa experiencia ayude. " +"Gran parte del código del juego proviene del mismo autor, ZUN, por lo que " +"los hábitos recurrentes son una suposición inicial razonable. Los " +"repositorios respaldan esto con ejemplos útiles: bucles relacionados y " +"arreglos de ayuda han proporcionado hipótesis en todos los juegos." + +#: website/public/blog/reverse-engineering-models/index.html:376 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those source patterns become more specific when we include the compiler. " +"Under a particular version and set of options, small changes in source can " +"produce repeatable differences in machine code. Some of those differences " +"are surprisingly obscure." +msgstr "" +"Esos patrones fuente se vuelven más específicos cuando incluimos el " +"compilador. Bajo una versión particular y un conjunto de opciones, pequeños " +"cambios en la fuente pueden producir diferencias repetibles en el código de " +"máquina. Algunas de esas diferencias son sorprendentemente oscuras." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: Supervisor::LoadDat +#. {ph4}: +#: website/public/blog/reverse-engineering-models/index.html:382 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In TH095, for example, {ph0}even the name used for a local variable could " +"change where VC7.1 stored it{ph1}. Moving the declaration alone did not " +"solve the layout, but an earlier background function had revealed a naming " +"pattern that affected the compiler's choice. That gave us a more promising " +"hypothesis for {ph2}the reconstruction of {ph3}{ph4}." +msgstr "" +"En TH095, por ejemplo, {ph0}incluso el nombre de una variable local podría " +"afectar dónde la almacenó VC7.1{ph1}. Mover su declaración no resolvió el " +"diseño del almacenamiento. Sin embargo, una función en segundo plano " +"anterior había revelado un patrón de nombres que afectaba la elección del " +"compilador. Eso sugirió una hipótesis más prometedora para {ph2}reconstruir " +"{ph3}{ph4}." + +#: website/public/blog/reverse-engineering-models/index.html:396 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reusing that pattern for the file-size variable produced the required layout" +" without adding storage. The full function then matched." +msgstr "" +"La reutilización de ese patrón para la variable de tamaño de archivo produjo" +" el diseño requerido sin agregar almacenamiento. Entonces coincidió la " +"función completa." + +#: website/public/blog/reverse-engineering-models/index.html:401 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is the sort of narrow detail a general model may not know when it " +"arrives. A specialist trained on this compiler could encounter many such " +"cases and learn which source variations are worth testing first." +msgstr "" +"Este es un detalle muy específico que un modelo de IA de propósito general " +"puede no conocer cuando encuentra el problema por primera vez. Un modelo " +"especializado para este compilador podría ver muchos de estos casos y " +"aprender qué variaciones del código fuente vale la pena probar primero." + +#: website/public/blog/reverse-engineering-models/index.html:407 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The opportunity is especially clear in final matching problems where we " +"already understand the behavior. We are then asking which ordinary source " +"form this compiler turns into the observed instructions. Specialization " +"could help the model propose those forms more efficiently, while the target " +"still determines which hypotheses survive." +msgstr "" +"La oportunidad es especialmente clara en los problemas finales de " +"emparejamiento en los que ya entendemos el comportamiento. Entonces nos " +"preguntamos qué disposición ordinaria del código fuente convierte este " +"compilador en las instrucciones observadas. La especialización podría ayudar" +" al modelo a proponer esas formas de manera más eficiente, mientras que el " +"objetivo aún determina qué hipótesis sobreviven." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:415 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The useful patterns would have to retain their context, though. TH08 uses an" +" earlier Visual C++ version than TH095 and TH09. TH04 uses a different DOS-" +"era compiler, and its {ph0}player reconstruction{ph1} needed different " +"hypotheses about source patterns and how the compiled code was combined into" +" an executable. Shared authorship gives us a prior, but each target tells us" +" whether it applies. Distinguishing those contexts is part of what a " +"specialist would need to learn." +msgstr "" +"Los patrones útiles deben conservar su contexto. TH08 utiliza una versión " +"Visual C++ anterior a TH095 y TH09. TH04 utiliza un compilador diferente de " +"la era DOS. Su {ph0}reconstrucción del código del jugador{ph1} necesitaba " +"diferentes hipótesis sobre los patrones del código fuente y cómo el código " +"compilado se vinculaba a un ejecutable. Tener el mismo autor nos da una " +"expectativa inicial, pero cada objetivo nos dice si aplica. Un modelo " +"especializado necesitaría distinguir estos contextos." + +#: website/public/blog/reverse-engineering-models/index.html:431 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Where should that experience live?" +msgstr "¿Dónde debería almacenarse esta experiencia?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:432 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before deciding to teach those patterns through model training, I would look" +" at the knowledge we already keep in the repository. In the {ph0}previous " +"essay{ph1}, I wrote about it becoming the project's working memory. That " +"memory includes compiler lessons, along with the source and failed " +"experiments that made them useful." +msgstr "" +"Antes de decidir enseñar esos patrones mediante el entrenamiento de modelos," +" miraría el conocimiento que ya guardamos en el repositorio. En el " +"{ph0}ensayo anterior{ph1}, escribí sobre cómo se convertiría en la memoria " +"de trabajo del proyecto. Esa memoria incluye lecciones del compilador, junto" +" con la fuente y los experimentos fallidos que los hicieron útiles." + +#: website/public/blog/reverse-engineering-models/index.html:441 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"By retrieving that record, an agent can start with a known result and " +"inspect how we reached it. It can also avoid alternatives that have already " +"failed in the same context. The starting point improves without changing the" +" model's weights." +msgstr "" +"Al recuperar ese registro, un agente puede comenzar con un resultado " +"conocido e inspeccionar cómo llegamos a él. También puede evitar " +"alternativas que ya han fracasado en el mismo contexto. El punto de partida " +"mejora sin cambiar los parámetros entrenados del modelo." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:447 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"When reconstructing TH09's enemy-spawn function, a {ph0}GPT-6-Luna-max " +"session{ph1} drew a hypothesis from TH08's corresponding source family. That" +" earlier source suggested how to structure the loop and handle failures " +"within it. TH09's own evidence supplied the game-specific details, and clean" +" rebuilds confirmed the complete match." +msgstr "" +"Mientras reconstruía la función de TH09 para crear enemigos, una sesión " +"{ph0}GPT-6-Luna-max{ph1} tomó una hipótesis del código fuente relacionado de" +" TH08. Ese código anterior sugirió cómo organizar el bucle y manejar las " +"fallas dentro de él. La propia evidencia de TH09 proporcionó los detalles " +"específicos del juego. Las reconstrucciones de la fuente confirmaron que la " +"función completa coincidía." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:458 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}\"The model did not know that compiler quirk\" and \"the model could " +"not use it\" are different problems.{ph1} Retrieval may solve the first, but" +" turning that fact into a useful candidate still takes work. And if the " +"candidate fails, the model must decide what the result means for its next " +"attempt. Knowing the relevant pattern is only the beginning of that process." +msgstr "" +"{ph0}\"El modelo no sabía esa peculiaridad del compilador\" y \"el modelo no" +" podía usarlo\" son problemas diferentes.{ph1} La recuperación puede " +"resolver el primero, pero convertir ese hecho en un candidato útil aún " +"requiere trabajo. Y si el código fuente candidato falla, el modelo debe " +"decidir qué significa el resultado para su próximo intento. Conocer el " +"patrón relevante es sólo el comienzo de ese proceso." + +#: website/public/blog/reverse-engineering-models/index.html:468 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For that reason, I would include a smaller general model with good retrieval" +" in any comparison. Giving it the same library of experience would help us " +"identify what training adds: better candidate generation, better decisions " +"after feedback, or both." +msgstr "" +"Por esa razón, incluiría en cualquier comparación un modelo de IA de " +"propósito general más pequeño con buena recuperación. Darle la misma " +"biblioteca de experiencia nos ayudaría a identificar qué aporta la " +"capacitación: mejor generación de candidatos, mejores decisiones después de " +"la retroalimentación, o ambas." + +#: website/public/blog/reverse-engineering-models/index.html:476 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Learning the next useful experiment" +msgstr "Aprendiendo el próximo experimento útil" + +#: website/public/blog/reverse-engineering-models/index.html:477 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those decisions after feedback are what I would most like to study. In " +"agentic reconstruction, the model proposes an explanation and tests it " +"against the program. Even when the candidate fails, the result can narrow " +"the remaining possibilities and suggest a more useful next experiment." +msgstr "" +"Esas decisiones después del feedback son las que más me gustaría estudiar. " +"En la reconstrucción agente, el modelo propone una explicación y la compara " +"con el programa. Incluso cuando el código fuente candidato falla, el " +"resultado puede reducir las posibilidades restantes y sugerir un próximo " +"experimento más útil." + +#: website/public/blog/reverse-engineering-models/index.html:484 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Consider the temporary-results example again. Its final source shows the " +"arrangement that worked, but the route to that source contains more " +"information. Each of the following experiments helped identify what still " +"needed to change." +msgstr "" +"Considere nuevamente el ejemplo de los resultados temporales. Su fuente " +"final muestra el arreglo que funcionó, pero la ruta a esa fuente contiene " +"más información. Cada uno de los siguientes experimentos ayudó a identificar" +" lo que aún era necesario cambiar." + +#: website/public/blog/reverse-engineering-models/index.html:497 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Experiment" +msgstr "experimento" + +#: website/public/blog/reverse-engineering-models/index.html:498 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What its result taught us" +msgstr "Lo que nos enseñó su resultado" + +#: website/public/blog/reverse-engineering-models/index.html:503 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Try a different way to copy the results" +msgstr "Pruebe una forma diferente de copiar los resultados." + +#: website/public/blog/reverse-engineering-models/index.html:504 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The copy code changes, but the function still differs." +msgstr "El código de copia cambia, pero la función aún difiere." + +#: website/public/blog/reverse-engineering-models/index.html:509 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Use the returned values directly" +msgstr "Utilice los valores devueltos directamente" + +#: website/public/blog/reverse-engineering-models/index.html:510 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Copying follows the original pattern, but temporary storage still differs." +msgstr "" +"La copia sigue el patrón original, pero el almacenamiento temporal aún " +"difiere." + +#: website/public/blog/reverse-engineering-models/index.html:516 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the temporary results alive together" +msgstr "Mantener todas las variables temporales válidas al mismo tiempo." + +#: website/public/blog/reverse-engineering-models/index.html:517 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"They get separate storage, but one still overlaps a value used later for " +"collision checks." +msgstr "" +"Se almacenan por separado, pero uno aún se superpone a un valor utilizado " +"más adelante para las comprobaciones de colisiones." + +#: website/public/blog/reverse-engineering-models/index.html:523 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the collision-check value separate" +msgstr "Mantenga el valor de verificación de colisiones separado" + +#: website/public/blog/reverse-engineering-models/index.html:524 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A clean rebuild confirms the complete match." +msgstr "" +"Una recompilación limpia del código fuente confirma la coincidencia " +"completa." + +#: website/public/blog/reverse-engineering-models/index.html:529 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Selected experiments from the attack-function reconstruction above, grouped " +"by what they taught us." +msgstr "" +"Experimentos seleccionados de la reconstrucción de la función de ataque " +"anterior, agrupados según lo que nos enseñaron." + +#: website/public/blog/reverse-engineering-models/index.html:533 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A training record that keeps this sequence can teach more than the final " +"source alone. It connects each proposed change to an observed result and the" +" next decision. Keeping ineffective variations matters too: they tell a " +"later agent which hypotheses have already been tested in that context." +msgstr "" +"Un registro de capacitación que mantenga esta secuencia puede enseñar más " +"que la fuente final por sí sola. Conecta cada cambio propuesto con un " +"resultado observado y la siguiente decisión. Mantener variaciones ineficaces" +" también es importante: le dicen a un agente posterior qué hipótesis ya han " +"sido probadas en ese contexto." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:540 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}I would want the model to learn how to use feedback.{ph1} A neutral " +"result is a reason to reconsider the variation; a changed mismatch may point" +" to another cause. When local edits stop being informative, the agent should" +" be able to revisit the source representation or inspect the evidence again." +msgstr "" +"{ph0}Me gustaría que el modelo aprendiera a utilizar la " +"retroalimentación.{ph1} Un resultado neutral es una razón para reconsiderar " +"la variación; una discrepancia modificada puede indicar otra causa. Cuando " +"las ediciones locales dejan de ser informativas, el agente debería poder " +"revisar la representación de la fuente o inspeccionar la evidencia " +"nuevamente." + +#: website/public/blog/reverse-engineering-models/index.html:549 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Learning to make those choices requires autonomy. Sometimes the useful next " +"step is inspecting another caller rather than compiling another edit. A " +"rigid sequence of required rewrites would remove that choice, along with " +"part of the behavior we want the model to learn." +msgstr "" +"Aprender a tomar esas decisiones requiere autonomía. A veces, el siguiente " +"paso útil es inspeccionar a otra persona que llama en lugar de compilar otra" +" edición. Una secuencia rígida de reescrituras requeridas eliminaría esa " +"elección, junto con parte del comportamiento que queremos que aprenda el " +"modelo." + +#: website/public/blog/reverse-engineering-models/index.html:558 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why RE is interesting for reinforcement learning" +msgstr "" +"Por qué la ingeniería inversa es interesante para el aprendizaje por " +"refuerzo" + +#: website/public/blog/reverse-engineering-models/index.html:559 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Bounded reconstruction gives us a practical environment for studying these " +"choices. Once the target and compiler are pinned, many source experiments " +"produce quick, repeatable feedback. The agent can compile a candidate, " +"inspect the mismatch and use that observation to choose its next step." +msgstr "" +"La reconstrucción acotada nos brinda un entorno práctico para estudiar estas" +" opciones. Una vez que se corrigen las versiones de destino y del " +"compilador, muchos experimentos de origen producen comentarios rápidos y " +"repetibles. El agente puede compilar un candidato, inspeccionar la " +"discrepancia y utilizar esa observación para elegir el siguiente paso." + +#: website/public/blog/reverse-engineering-models/index.html:566 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reinforcement learning could train that decision-making process through the " +"feedback it receives. A candidate accepted by a defined check gives us a " +"concrete outcome, while the sequence of attempts lets us observe how much " +"work it took to reach it." +msgstr "" +"El aprendizaje por refuerzo podría entrenar ese proceso de toma de " +"decisiones a través de la retroalimentación que recibe. Un candidato " +"aceptado mediante un control definido nos da un resultado concreto, mientras" +" que la secuencia de intentos nos permite observar cuánto trabajo ha sido " +"necesario para alcanzarlo." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:572 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"There is already research supporting this direction. {ph0}LLM4Decompile{ph1}" +" studies models specialized for decompilation. {ph2}AutoDecompiler{ph3} " +"trains multi-turn refinement using compiler and execution feedback. Their " +"studies focus on functional recovery. For this work, I would use them as a " +"starting point for the related task of finding source that matches a " +"historical compiler's output." +msgstr "" +"Ya existen investigaciones que respaldan esta dirección. " +"{ph0}LLM4Decompile{ph1} estudia modelos especializados para la " +"descompilación. {ph2}AutoDecompiler{ph3} entrena el refinamiento de varios " +"turnos utilizando el compilador y los comentarios de ejecución. Sus estudios" +" se centran en la recuperación funcional. Para este trabajo, los usaría como" +" punto de partida para la tarea relacionada de encontrar una fuente que " +"coincida con la salida de un compilador histórico." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:584 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To reach such a well-defined task, though, we may first need to establish " +"what a function owns or devise a new runtime measurement. In TH08, for " +"example, a {ph0}collision difference led back to how the original program " +"handled floating-point calculations{ph1}. Before the agent could test a " +"useful correction, it had to work out what to measure." +msgstr "" +"Sin embargo, para alcanzar una tarea tan bien definida, es posible que " +"primero necesitemos establecer qué recursos posee una función o diseñar una " +"nueva forma de observar el comportamiento del programa durante su ejecución." +" En TH08, por ejemplo, una {ph0}diferencia de colisión llevó a cómo el " +"programa original manejaba los cálculos de punto flotante{ph1}. Antes de que" +" el agente pudiera probar una corrección útil, tenía que decidir qué medir." + +#: website/public/blog/reverse-engineering-models/index.html:595 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This suggests another role for a broadly capable model: helping make the " +"problem checkable in the first place. Once that check exists, we can " +"investigate how far a learned search policy can get with it." +msgstr "" +"Esto sugiere otra función para un modelo con amplias capacidades: ayudar a " +"que el problema sea verificable en primer lugar. Una vez que exista esa " +"verificación, podemos investigar hasta dónde puede llegar una política de " +"búsqueda aprendida." + +#: website/public/blog/reverse-engineering-models/index.html:603 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "When the oracle is wrong" +msgstr "Cuando el verificador está equivocado" + +#: website/public/blog/reverse-engineering-models/index.html:604 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Once we train against that check, its coverage becomes part of what the " +"model learns. I call the reference or checker an oracle. Even if it gives a " +"repeatable answer, we still need to understand what that answer covers." +msgstr "" +"Al entrenar con una comprobación, su cobertura pasa a formar parte del " +"entorno de aprendizaje. Llamo oracle a esa referencia o verificador. Aunque " +"la respuesta sea repetible, debemos entender qué verifica realmente." + +#: website/public/blog/reverse-engineering-models/index.html:610 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"TH08 showed how much can hide outside that coverage. Some reconstructed " +"functions passed their exact comparisons with wrong floating constants. The " +"comparator adjusted relocated addresses before checking instructions, but it" +" checked the values behind those addresses only when they had been " +"explicitly recorded for verification. Other literal references could pass " +"without their values being examined." +msgstr "" +"TH08 mostró cuánto se puede ocultar fuera de esa cobertura. Algunas " +"funciones reconstruidas pasaron sus comparaciones exactas con constantes " +"flotantes incorrectas. El comparador ajustó las direcciones reubicadas antes" +" de verificar las instrucciones, pero verificó los valores detrás de esas " +"direcciones solo cuando se habían registrado explícitamente para su " +"verificación. Otras referencias literales podrían pasar sin que se examinen " +"sus valores." + +#. {ph0}: +#. {ph1}: +#. {ph2}: 1.0f +#. {ph3}: 32.0f +#: website/public/blog/reverse-engineering-models/index.html:619 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To close that gap, a {ph0}wider audit{ph1} found incorrect constants in " +"functions that had already been accepted. In one calculation, the " +"reconstructed source added {ph2} where the original added {ph3}." +msgstr "" +"Para abordar esta brecha, una {ph0}auditoría más amplia{ph1} encontró " +"constantes incorrectas en funciones que ya habían pasado la verificación. En" +" un cálculo, el código fuente reconstruido agregó {ph2}, mientras que el " +"original agregó {ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:629 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A learner rewarded by the old check could learn to produce the wrong " +"reconstruction.{ph1} We had not trained an RL model against this oracle; the" +" observed failure was in the check itself. But using that check as a reward " +"would have made the same blind spot part of the learning environment." +msgstr "" +"{ph0}Un modelo recompensado por la comprobación anterior podría aprender a " +"producir una reconstrucción incorrecta.{ph1} No entrenamos un modelo RL " +"usando ese verificador. El fallo que observamos estaba en la propia " +"comprobación. Pero si lo usáramos como recompensa, el mismo punto ciego " +"pasaría a formar parte del entorno de aprendizaje." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:639 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A functional test can have a similar blind spot. {ph0}Differential-fuzzing " +"research on LLM decompilers{ph1} has found behavior differences beyond the " +"supplied tests. Passing those tests therefore needs to be read in terms of " +"the cases they exercise, just as an instruction match needs to be read in " +"terms of the bytes and data it checks." +msgstr "" +"Una prueba funcional puede tener un punto ciego similar. {ph0}La " +"investigación que utilizó fuzzing diferencial en descompiladores LLM{ph1} " +"encontró diferencias de comportamiento fuera de las pruebas proporcionadas. " +"Por lo tanto, debemos entender una prueba aprobatoria en términos de los " +"casos en los que se aplica. De manera similar, una coincidencia de " +"instrucciones debe entenderse en términos de los bytes y datos que verifica." + +#: website/public/blog/reverse-engineering-models/index.html:649 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before using either kind of check as a reward, I would test it with " +"deliberate failures. I would also keep final acceptance outside the " +"candidate's editable source, so the model has to satisfy the check through " +"its reconstruction. Otherwise, something like a byte array copied from the " +"target could satisfy a naive score without recovering useful source." +msgstr "" +"Antes de utilizar cualquier tipo de control de verificación como recompensa," +" lo probaría con resultados deliberadamente incorrectos. También mantendría " +"la aceptación final separada del código fuente que el agente de IA puede " +"editar. El modelo debe pasar la verificación reconstruyendo el programa " +"correctamente. De lo contrario, simplemente copiar una matriz de bytes del " +"destino podría engañar a un sistema de puntuación ingenuo sin recuperar el " +"código fuente útil." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:657 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Improving the learner and improving its oracle belong to the same " +"research problem.{ph1} As the learner gets better at optimizing the feedback" +" we give it, understanding that feedback becomes more important too." +msgstr "" +"{ph0}Mejorar al alumno y mejorar su verificador pertenecen al mismo problema" +" de investigación.{ph1} A medida que el alumno mejora en la optimización de " +"los comentarios que le damos, comprender esos comentarios también se vuelve " +"más importante." + +#: website/public/blog/reverse-engineering-models/index.html:667 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A comparison worth running" +msgstr "Una comparación que vale la pena realizar" + +#: website/public/blog/reverse-engineering-models/index.html:668 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"With those checks in place, I would start the comparison on a fixed VC7.1 " +"task family. First, I would establish baselines for a strong general model " +"and a smaller one. Adding repository knowledge to the smaller model would " +"show what retrieval changes. From there, supervised specialization and an " +"RL-trained version would let us examine what training adds." +msgstr "" +"Con esas comprobaciones implementadas, comenzaría la comparación en una " +"familia de tareas fija VC7.1. Primero, establecería líneas de base para un " +"modelo de IA de propósito general sólido y uno más pequeño. Agregar " +"conocimiento del repositorio al modelo más pequeño mostraría qué cambios en " +"la recuperación. A partir de ahí, la especialización supervisada y una " +"versión capacitada con RL nos permitirían examinar qué agrega la " +"capacitación." + +#: website/public/blog/reverse-engineering-models/index.html:676 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To make those comparisons informative, each model would receive the same " +"target evidence and tools. For the training-specific comparisons, I would " +"keep the base model and available knowledge fixed as well. Repeated trials " +"would help distinguish a reliable search strategy from a lucky candidate." +msgstr "" +"Para que esas comparaciones sean informativas, cada modelo recibiría la " +"misma evidencia y herramientas objetivo. Para las comparaciones específicas " +"de la capacitación, también mantendría fijo el modelo base y el conocimiento" +" disponible. Las pruebas repetidas ayudarían a distinguir una estrategia de " +"búsqueda confiable de una solución candidata que tiene éxito por casualidad." + +#: website/public/blog/reverse-engineering-models/index.html:683 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would then separate two starting conditions: reconstructing a function " +"from its instructions, and closing the last few differences in maintained " +"source. A specialist might be excellent at the second task while a general " +"model does much of the work needed to reach it. Measuring them separately " +"would reveal that division of work." +msgstr "" +"Luego separaría dos condiciones iniciales: reconstruir una función a partir " +"de sus instrucciones y cerrar las últimas diferencias en la fuente " +"mantenida. Un modelo de IA especializado podría ser excelente en la segunda " +"tarea, mientras que un modelo de IA de propósito general hace gran parte del" +" trabajo necesario para alcanzarla. Medirlos por separado revelaría esa " +"división del trabajo." + +#: website/public/blog/reverse-engineering-models/index.html:691 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both tasks need genuinely held-out examples. Because related Touhou games " +"share engine code, holding out a game's name would still allow nearly " +"identical functions to appear in training. I would instead hold out source " +"families and keep their accepted answers out of both training and retrieval." +msgstr "" +"Ambas tareas necesitan ejemplos de prueba excluidos del entrenamiento y la " +"recuperación. Los juegos Touhou relacionados comparten el código del motor. " +"Excluir un juego del entrenamiento aún podría dejar funciones casi idénticas" +" en el conjunto de entrenamiento. En cambio, excluiría grupos enteros de " +"código fuente relacionado y mantendría sus respuestas verificadas fuera de " +"los datos de entrenamiento y de la biblioteca de recuperación." + +#: website/public/blog/reverse-engineering-models/index.html:698 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"On those examples, the outcome would be complete, independently validated " +"matches. Giving each model the same candidate or compile budget would help " +"compare search quality. Measuring elapsed time and total cost, including " +"compilation, would tell us whether that quality produces an efficient " +"workflow. Training costs would need their own accounting too." +msgstr "" +"Para estos ejemplos, el resultado serían coincidencias completas y " +"verificadas de forma independiente. La igualdad de presupuestos para las " +"propuestas de los candidatos o los intentos de compilación ayudaría a " +"comparar la calidad de la búsqueda. Medir el tiempo y el costo total, " +"incluida la compilación, mostraría si esa calidad produce un flujo de " +"trabajo eficiente. Los costos de capacitación deben medirse por separado." + +#: website/public/blog/reverse-engineering-models/index.html:706 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both views matter because a cheaper proposal can become an expensive " +"workflow if it causes more uninformative compiles. Conversely, a slower " +"model may pay for itself by finding a useful experiment sooner. The " +"comparison should capture those costs." +msgstr "" +"Ambas medidas importan. Una propuesta barata puede generar un flujo de " +"trabajo costoso si genera más compilaciones que no proporcionan información " +"útil. Por el contrario, un modelo más lento puede resultar más económico si " +"encuentra antes un experimento útil. La comparación debe incluir estos " +"costos." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:712 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Counting accepted results also keeps intermediate progress in perspective. " +"One TH09 {ph0}replay-save candidate{ph1} looked promising because the " +"compiled function had the expected size, but a full comparison still found " +"substantial differences. Changes in how the compiler arranges instructions " +"can also make a candidate look closer without resolving the underlying " +"problem. Such diagnostics can guide the next experiment, but I would " +"validate any reward for \"getting closer\" against the full acceptance " +"check." +msgstr "" +"Contar los resultados aceptados también ayuda a interpretar los avances " +"intermedios. Un {ph0}candidato para guardar repeticiones{ph1} de TH09 " +"parecía prometedor porque la función compilada tenía el tamaño esperado, " +"pero la comparación completa aún encontró diferencias importantes. La forma " +"en que el compilador ordena las instrucciones puede hacer que un candidato " +"parezca más cercano sin resolver el problema de fondo. Estos diagnósticos " +"orientan el siguiente experimento; antes de premiar una mayor cercanía, la " +"contrastaría con la comprobación completa de aceptación." + +#: website/public/blog/reverse-engineering-models/index.html:727 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would be interested in the pattern across these results. If a specialist " +"does well within a familiar compiler family, does that advantage survive a " +"toolchain change? And once both models have good retrieval, how much does " +"training still add? The answers would help us decide where each kind of " +"model is most useful." +msgstr "" +"Me interesaría conocer el patrón de estos resultados. Si un modelo de IA " +"especializado funciona bien dentro de una familia de compiladores familiar, " +"¿esa ventaja sobrevive a un cambio en la cadena de herramientas? Y una vez " +"que ambos modelos de IA tengan una buena recuperación, ¿cuánto aporta aún el" +" entrenamiento? Las respuestas nos ayudarían a decidir dónde es más útil " +"cada tipo de modelo." + +#: website/public/blog/reverse-engineering-models/index.html:736 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What kind of agent does this suggest?" +msgstr "¿Qué tipo de agente de IA sugiere esto?" + +#: website/public/blog/reverse-engineering-models/index.html:737 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"If the models turn out to help at different stages, a useful arrangement " +"might be a broad agent with access to a specialist. Once a task becomes a " +"familiar compiler search, it could ask the specialist for candidates. If the" +" results stop making sense, the broader agent could return to the target " +"evidence or investigate the measurement itself." +msgstr "" +"Si los modelos resultan útiles en diferentes etapas, un acuerdo útil podría " +"ser un agente amplio con acceso a un modelo de IA especializado. Una vez que" +" una tarea se convierte en una búsqueda familiar del compilador, podría " +"solicitar candidatos al modelo de IA especializado. Si los resultados dejan " +"de tener sentido, el agente más amplio podría volver a la evidencia objetivo" +" o investigar la medición en sí." + +#: website/public/blog/reverse-engineering-models/index.html:745 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In that arrangement, specialization becomes a tool the agent can choose as " +"the investigation develops. The agent keeps the freedom to change direction " +"that made the earlier breakthroughs possible." +msgstr "" +"En ese arreglo, la especialización se convierte en una herramienta que el " +"agente puede elegir a medida que se desarrolla la investigación. El agente " +"conserva la libertad de cambiar la dirección que hizo posibles los avances " +"anteriores." + +#: website/public/blog/reverse-engineering-models/index.html:750 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This follows naturally from the industrial shift I described in the previous" +" essay. Being able to run many more experiments changes how we organize the " +"work. It also gives us a reason to ask which decisions need broad judgment " +"and which improve through repeated experience with a particular craft." +msgstr "" +"Esto se deriva naturalmente del cambio industrial que describí en el ensayo " +"anterior. Poder realizar muchos más experimentos cambia la forma en que " +"organizamos el trabajo. También nos da una razón para preguntar qué " +"decisiones requieren un juicio amplio y cuáles mejoran a través de " +"experiencias repetidas con un oficio en particular." + +#: website/public/blog/reverse-engineering-models/index.html:757 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For now, the strongest general model has earned its place in my workflow by " +"opening problems I had left unresolved. Those successes are what make me " +"want to examine the work more closely: how much depended on the model's " +"breadth, and how much could be handled by a model trained in this particular" +" kind of reconstruction?" +msgstr "" +"Por ahora, el modelo general más potente se ha ganado su lugar en mi flujo " +"de trabajo al resolver problemas que había dejado pendientes. Esos éxitos me" +" llevan a examinar el proceso: ¿cuánto dependía de la amplitud del modelo y " +"cuánto podría asumir uno entrenado específicamente para esta clase de " +"reconstrucción?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:765 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}The repositories give us a way to test the question.{ph1} I would like " +"to use that record to build the comparison and see what we learn." +msgstr "" +"{ph0}Los repositorios nos brindan una forma de probar la pregunta.{ph1} Me " +"gustaría usar ese registro para realizar la comparación y ver qué " +"aprendemos." + +#: website/public/blog/reverse-engineering-models/index.html:775 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Evidence and model attribution" +msgstr "Atribución de evidencia y modelo" + +#: website/public/blog/reverse-engineering-models/index.html:783 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The examples describe historical function-level comparisons recorded in the " +"linked commits. Whole-game runtime checks are separate milestones. Source " +"and matching records were reviewed on 13 October 2026." +msgstr "" +"Los ejemplos describen comparaciones históricas a nivel de función " +"registradas en las confirmaciones vinculadas. Las comprobaciones del tiempo " +"de ejecución del juego completo son hitos separados. Los registros fuente y " +"coincidentes se revisaron el 13 de octubre de 2026." + +#. {ph0}: gpt-dots +#. {ph1}: gpt-web +#: website/public/blog/reverse-engineering-models/index.html:789 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The TH09 label {ph0} identifies GPT-6-Astra. {ph1} changed with the model " +"used at the time, so that label alone does not identify an individual run. " +"Explicit model names in the other cited commits establish their recorded " +"attribution; the histories do not contain uniform per-run cost or effort " +"measurements." +msgstr "" +"La etiqueta TH09 {ph0} significa GPT-6-Astra. El modelo asociado con {ph1} " +"cambió con el tiempo, por lo que esa etiqueta por sí sola no puede " +"identificar el modelo utilizado en una ejecución específica. Los nombres de " +"modelos explícitos en las otras confirmaciones citadas proporcionan la " +"atribución registrada. Estos historiales no incluyen mediciones consistentes" +" de costo o esfuerzo para cada ejecución." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#: website/public/blog/reverse-engineering-models/index.html:798 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 temporary lifetimes:{ph1} the {ph2}earlier context review{ph3} and" +" {ph4}complete cold reconstruction{ph5}." +msgstr "" +"{ph0}TH09 duración de los objetos temporales:{ph1} la {ph2}revisión de " +"contexto anterior{ph3} y la {ph4}reconstrucción completa verificada mediante" +" una recompilación limpia del código fuente{ph5}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:810 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 Front callback:{ph1} {ph2}non-exact diagnostic regions{ph3}, " +"{ph4}Astra's new graph{ph5} and its {ph6}accepted recompile replay{ph7}." +msgstr "" +"{ph0}TH09 Front devolución de llamada:{ph1} {ph2}regiones de diagnóstico que" +" no coincidieron{ph3}, {ph4}Astra nuevo gráfico de flujo de control{ph5} y " +"su {ph6}verificación de recompilación aceptada{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:826 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Compiler knowledge:{ph1} {ph2}TH095 identifier-bucket reuse{ph3}, " +"{ph4}TH04 player source and linker order{ph5} and {ph6}Luna's TH09 enemy-" +"spawn closure{ph7}." +msgstr "" +"{ph0}Conocimiento del compilador:{ph1} {ph2}TH095 reutilización del " +"comportamiento del identificador-depósito{ph3}, {ph4}TH04 código fuente del " +"jugador y orden del enlazador{ph5} y {ph6}Luna reconstrucción completa de la" +" función de creación de enemigos de TH09{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:842 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Checks and diagnostics:{ph1} {ph2}TH08 floating-literal audit{ph3}, " +"{ph4}TH08 numerical investigation{ph5} and {ph6}TH09 replay-save " +"reconstruction{ph7}." +msgstr "" +"{ph0}Comprobaciones y diagnósticos:{ph1} {ph2}TH08 auditoría constante de " +"punto flotante{ph3}, {ph4}TH08 investigación numérica{ph5} y {ph6}TH09 " +"reconstrucción para guardar reproducciones{ph7}." + +#: website/public/blog/reverse-engineering-models/index.html:859 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The proposed specialist and RL comparisons are future experiments. The " +"project histories motivate them; no trained specialist was evaluated for " +"this essay." +msgstr "" +"Las comparaciones propuestas de modelos especializados y entrenamiento RL " +"son experimentos futuros. Las historias de los proyectos los motivan. Para " +"este ensayo no se evaluó ningún modelo especializado entrenado." + +#: website/public/blog/reverse-engineering-models/index.html:6 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model? | REA" +msgstr "¿La ingeniería inversa necesita un modelo más grande? | REA" + +#: website/public/blog/reverse-engineering-models/index.html:7 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What Touhou reconstruction taught me about general models, specialist models" +" and learning to propose better hypotheses from compiler feedback." +msgstr "" +"Lo que me enseñó la reconstrucción Touhou sobre modelos generales, modelos " +"especializados y cómo aprender a proponer mejores hipótesis a partir de los " +"comentarios del compilador." + +#: website/public/blog/reverse-engineering-models/index.html:240 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "On this page" +msgstr "En esta página" + +#: website/public/blog/reverse-engineering-models/index.html:253 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Models and reverse engineering essay" +msgstr "Modelos y ensayo de ingeniería inversa." + +#: website/public/blog/reverse-engineering-models/index.html:491 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Selected reconstruction experiments" +msgstr "Experimentos de reconstrucción seleccionados." + +#: website/public/blog/reverse-engineering-models/index.html:883 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Back to top" +msgstr "Volver arriba" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:776 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Detailed evidence notes{ph1} include the recorded comparisons and " +"experiment context." +msgstr "" +"Las {ph0}notas detalladas de evidencia{ph1} incluyen las comparaciones " +"registradas y el contexto de los experimentos." + +#~ msgctxt "page:blog/index.html" +#~ msgid "By N0zoM1z0 · 13 October 2026 · Draft for review" +#~ msgstr "Por N0zoM1z0 · 13 de octubre de 2026 · Borrador para revisión" + #~ msgctxt "page:index.html" #~ msgid "Setup guide" #~ msgstr "Guía de instalación" diff --git a/website/locales/fa/messages.po b/website/locales/fa/messages.po index 65fd093ac..02b489a3c 100644 --- a/website/locales/fa/messages.po +++ b/website/locales/fa/messages.po @@ -32,18 +32,18 @@ msgid "Thoughts on reverse engineering." msgstr "افکار در مهندسی معکوس." #. {ph0}:
-#: website/public/blog/index.html:200 +#: website/public/blog/index.html:214 #, python-brace-format msgctxt "page:blog/index.html" msgid "Game reconstruction{ph0}Essay" msgstr "بازسازی بازی{ph0}مقاله" -#: website/public/blog/index.html:202 +#: website/public/blog/index.html:216 msgctxt "page:blog/index.html" msgid "The industrial revolution in reverse engineering" msgstr "انقلاب صنعتی در مهندسی معکوس" -#: website/public/blog/index.html:203 +#: website/public/blog/index.html:217 msgctxt "page:blog/index.html" msgid "" "What manual RE and reconstructing TH08, TH095 and TH04 taught me about " @@ -52,25 +52,26 @@ msgstr "" "چه دستی دوباره و بازسازی TH08, TH095 و TH04 به من آموخت در مورد عوامل, تأیید" " و دانش است که ترکیبات در سراسر پروژه." -#: website/public/blog/index.html:208 +#: website/public/blog/index.html:207 website/public/blog/index.html:222 msgctxt "page:blog/index.html" msgid "Read the article" msgstr "مقاله را بخوانید" -#: website/public/blog/index.html:211 +#: website/public/blog/index.html:225 msgctxt "page:blog/index.html" msgid "By N0zoM1z0 · 10 October 2026" msgstr "توسط N0zoM1z0 * 10 اکتبر 2026" #. {ph0}: -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 +#: website/public/blog/reverse-engineering-models/index.html:883 #: website/public/blog/touhou-reconstruction/index.html:1080 #: website/public/examples/dino-lab/index.html:475 #: website/public/faq/index.html:573 #: website/public/first-investigation/index.html:523 #: website/public/get-started/index.html:882 -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 #: website/public/guides/index.html:282 #: website/public/guides/javascript/index.html:503 #: website/public/guides/native/index.html:509 website/public/index.html:1191 @@ -99,6 +100,8 @@ msgstr "" "افکار در مهندسی معکوس ، با استفاده از تجربه عملی با REA و عوامل هوش مصنوعی." #: website/public/blog/index.html:75 website/public/blog/index.html:88 +#: website/public/blog/reverse-engineering-models/index.html:75 +#: website/public/blog/reverse-engineering-models/index.html:88 #: website/public/blog/touhou-reconstruction/index.html:75 #: website/public/blog/touhou-reconstruction/index.html:88 #: website/public/examples/dino-lab/index.html:75 @@ -135,6 +138,7 @@ msgstr "" "شما." #: website/public/blog/index.html:108 +#: website/public/blog/reverse-engineering-models/index.html:108 #: website/public/blog/touhou-reconstruction/index.html:108 #: website/public/examples/dino-lab/index.html:109 #: website/public/faq/index.html:108 @@ -155,6 +159,7 @@ msgid "REA home" msgstr "صفحهٔ اصلی REA" #: website/public/blog/index.html:112 +#: website/public/blog/reverse-engineering-models/index.html:112 #: website/public/blog/touhou-reconstruction/index.html:112 #: website/public/examples/dino-lab/index.html:113 #: website/public/faq/index.html:112 @@ -175,6 +180,7 @@ msgid "Main navigation" msgstr "ناوبری اصلی" #: website/public/blog/index.html:130 +#: website/public/blog/reverse-engineering-models/index.html:130 #: website/public/blog/touhou-reconstruction/index.html:130 #: website/public/examples/dino-lab/index.html:131 #: website/public/faq/index.html:130 @@ -195,6 +201,7 @@ msgid "REA on GitHub" msgstr "REA در GitHub" #: website/public/blog/index.html:150 +#: website/public/blog/reverse-engineering-models/index.html:150 #: website/public/blog/touhou-reconstruction/index.html:150 #: website/public/examples/dino-lab/index.html:151 #: website/public/faq/index.html:150 @@ -215,6 +222,7 @@ msgid "Join REA on Discord" msgstr "به REA در Discord بپیوندید" #: website/public/blog/index.html:169 +#: website/public/blog/reverse-engineering-models/index.html:169 #: website/public/blog/touhou-reconstruction/index.html:169 #: website/public/examples/dino-lab/index.html:170 #: website/public/faq/index.html:169 @@ -234,7 +242,7 @@ msgctxt "shared" msgid "Follow REA on X" msgstr "REA را در X دنبال کنید" -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 msgctxt "page:blog/index.html" msgid "Back to top" msgstr "بازگشت به بالا" @@ -4694,7 +4702,7 @@ msgctxt "page:guides/browser/index.html" msgid "Copy read the loaded scripts" msgstr "کپی خواندن اسکریپت های لود شده" -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 msgctxt "page:guides/browser/index.html" msgid "Back to top" msgstr "بازگشت به بالا" @@ -11479,6 +11487,7 @@ msgid "Same spacing: 256 ÷ 16 = 16 angle units = 22.5°" msgstr "فاصله یکسان: 256 × 16 = 16 واحد زاویه = 22.5 درجه" #: website/public/blog/index.html:114 +#: website/public/blog/reverse-engineering-models/index.html:114 #: website/public/blog/touhou-reconstruction/index.html:114 #: website/public/examples/dino-lab/index.html:115 #: website/public/faq/index.html:114 @@ -11499,6 +11508,7 @@ msgid "Showcases" msgstr "نمونه‌ها" #: website/public/blog/index.html:115 +#: website/public/blog/reverse-engineering-models/index.html:115 #: website/public/blog/touhou-reconstruction/index.html:115 #: website/public/examples/dino-lab/index.html:116 #: website/public/faq/index.html:115 @@ -11519,6 +11529,7 @@ msgid "Blog" msgstr "وبلاگ" #: website/public/blog/index.html:113 +#: website/public/blog/reverse-engineering-models/index.html:113 #: website/public/blog/touhou-reconstruction/index.html:113 #: website/public/examples/dino-lab/index.html:114 #: website/public/faq/index.html:113 @@ -11539,6 +11550,7 @@ msgid "Get started" msgstr "شروع کنید" #: website/public/blog/index.html:116 +#: website/public/blog/reverse-engineering-models/index.html:116 #: website/public/blog/touhou-reconstruction/index.html:116 #: website/public/examples/dino-lab/index.html:117 #: website/public/faq/index.html:116 @@ -11558,13 +11570,14 @@ msgctxt "shared" msgid "Guides" msgstr "راهنماها" -#: website/public/blog/index.html:222 +#: website/public/blog/index.html:236 +#: website/public/blog/reverse-engineering-models/index.html:875 #: website/public/blog/touhou-reconstruction/index.html:1072 #: website/public/examples/dino-lab/index.html:467 #: website/public/faq/index.html:565 #: website/public/first-investigation/index.html:515 #: website/public/get-started/index.html:874 -#: website/public/guides/browser/index.html:490 +#: website/public/guides/browser/index.html:499 #: website/public/guides/index.html:274 #: website/public/guides/javascript/index.html:495 #: website/public/guides/native/index.html:501 website/public/index.html:1183 @@ -11578,13 +11591,14 @@ msgctxt "shared" msgid "Source" msgstr "کد منبع" -#: website/public/blog/index.html:223 +#: website/public/blog/index.html:237 +#: website/public/blog/reverse-engineering-models/index.html:876 #: website/public/blog/touhou-reconstruction/index.html:1073 #: website/public/examples/dino-lab/index.html:468 #: website/public/faq/index.html:566 #: website/public/first-investigation/index.html:516 #: website/public/get-started/index.html:875 -#: website/public/guides/browser/index.html:491 +#: website/public/guides/browser/index.html:500 #: website/public/guides/index.html:275 #: website/public/guides/javascript/index.html:496 #: website/public/guides/native/index.html:502 website/public/index.html:1184 @@ -11598,13 +11612,14 @@ msgctxt "shared" msgid "MIT license" msgstr "مجوز MIT" -#: website/public/blog/index.html:224 +#: website/public/blog/index.html:238 +#: website/public/blog/reverse-engineering-models/index.html:877 #: website/public/blog/touhou-reconstruction/index.html:1074 #: website/public/examples/dino-lab/index.html:469 #: website/public/faq/index.html:567 #: website/public/first-investigation/index.html:517 #: website/public/get-started/index.html:876 -#: website/public/guides/browser/index.html:492 +#: website/public/guides/browser/index.html:501 #: website/public/guides/index.html:276 #: website/public/guides/javascript/index.html:497 #: website/public/guides/native/index.html:503 website/public/index.html:1185 @@ -11618,13 +11633,14 @@ msgctxt "shared" msgid "Join the community" msgstr "به جامعه بپیوندید" -#: website/public/blog/index.html:225 +#: website/public/blog/index.html:239 +#: website/public/blog/reverse-engineering-models/index.html:878 #: website/public/blog/touhou-reconstruction/index.html:1075 #: website/public/examples/dino-lab/index.html:470 #: website/public/faq/index.html:568 #: website/public/first-investigation/index.html:518 #: website/public/get-started/index.html:877 -#: website/public/guides/browser/index.html:493 +#: website/public/guides/browser/index.html:502 #: website/public/guides/index.html:277 #: website/public/guides/javascript/index.html:498 #: website/public/guides/native/index.html:504 website/public/index.html:1186 @@ -12563,6 +12579,1349 @@ msgctxt "runtime" msgid "Switch to dark mode" msgstr "تغییر به حالت تیره" +#. {ph0}: +#. {ph1}: +#: website/public/guides/browser/index.html:480 +#, python-brace-format +msgctxt "page:guides/browser/index.html" +msgid "" +"For HTTP request and response bodies, use {ph0}Fiddler Everywhere: SAZ " +"captures, MCP and public skills{ph1}. Export SAZ or HAR 1.2 for offline REA " +"capture Evidence, or connect Everywhere's MCP server to inspect captured " +"traffic with your agent." +msgstr "" +"برای بررسی بدنهٔ درخواست‌ها و پاسخ‌های HTTP، از {ph0}Fiddler Everywhere: " +"ضبط‌های SAZ، سرور MCP و مهارت‌های عمومی{ph1} استفاده کنید. برای Evidence ضبط" +" در REA به‌صورت آفلاین، خروجی SAZ ?? HAR 1.2 بگیرید، یا سرور MCP نرم‌افزار " +"Everywhere را متصل کنید تا ترافیک ضبط‌شده را با عامل خود بررسی کنید." + +#. {ph0}:
+#: website/public/blog/index.html:200 +#, python-brace-format +msgctxt "page:blog/index.html" +msgid "Model specialization{ph0}Essay" +msgstr "مدل‌های تخصصی{ph0}یادداشت" + +#: website/public/blog/index.html:202 +msgctxt "page:blog/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "آیا مهندسی معکوس به مدل بزرگ‌تری نیاز دارد؟" + +#: website/public/blog/index.html:203 +msgctxt "page:blog/index.html" +msgid "" +"What Touhou reconstruction suggests about general models, specialist models " +"and learning from compiler feedback." +msgstr "" +"تأملاتی بر پایهٔ بازسازی Touhou دربارهٔ مدل‌های عمومی، مدل‌های تخصصی و " +"یادگیری از بازخورد کامپایلر." + +#: website/public/blog/index.html:210 +msgctxt "page:blog/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "توسط N0zoM1z0 · 13 اکتبر 2026" + +#: website/public/blog/reverse-engineering-models/index.html:103 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Skip to content" +msgstr "پرش به محتوا" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:194 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Home{ph1} / {ph2}Blog{ph3} / Models and RE" +msgstr "{ph0}صفحه اصلی{ph1} / {ph2}وبلاگ{ph3} / مدل‌ها و مهندسی معکوس" + +#: website/public/blog/reverse-engineering-models/index.html:197 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Essay" +msgstr "انشا" + +# Reviewed for technical meaning and readability; preserve the distinction +# between historical evidence and proposed experiments. +#: website/public/blog/reverse-engineering-models/index.html:198 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "آیا مهندسی معکوس به مدل بزرگتری نیاز دارد؟" + +#: website/public/blog/reverse-engineering-models/index.html:199 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "توسط N0zoM1z0 · 13 اکتبر 2026" + +#: website/public/blog/reverse-engineering-models/index.html:201 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Takeaway" +msgstr "نکات اصلی" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:203 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Model capability changes the hypotheses an agent can find.{ph1} Later " +"sessions have solved problems that earlier work left unresolved." +msgstr "" +"{ph0}قابلیت مدل بر فرضیه‌هایی که یک عامل هوش مصنوعی می‌تواند ارائه دهد تأثیر" +" می‌گذارد.{ph1} جلسات بعدی مشکلاتی را که کار قبلی حل نشده رها کرده بود حل " +"کرد." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:211 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A familiar compiler gives specialization something concrete to " +"learn.{ph1} Some exact matches depend on narrow, reusable source patterns." +msgstr "" +"{ph0}یک کامپایلر آشنا الگوهای مشخصی را برای یادگیری یک مدل تخصصی ارائه " +"می‌کند.{ph1} برخی از تطابق دقیق کد ماشین به الگوهای کد منبع خاص و قابل " +"استفاده مجدد بستگی دارد." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:218 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Give repository knowledge a fair comparison.{ph1} A smaller model with " +"good retrieval may already know enough to make progress." +msgstr "" +"{ph0}دانش مخزن را در یک مقایسه منصفانه لحاظ کنید.{ph1} یک مدل کوچکتر با " +"بازیابی دانش خوب ممکن است اطلاعات کافی برای پیشرفت داشته باشد." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:223 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Train the next useful experiment, and test its oracle.{ph1} The value " +"of feedback depends on what the checker actually verifies." +msgstr "" +"{ph0}مدل را آموزش دهید تا آزمایش مفید بعدی را انتخاب کند و بررسی‌کننده تأیید" +" آن را آزمایش کنید.{ph1} بازخورد فقط در حدی مفید است که بررسی‌کننده موارد " +"درست را تأیید کند." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:230 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Measure accepted reconstruction against time and cost.{ph1} That is the" +" result I want from the whole agent." +msgstr "" +"{ph0}نتایج بازسازی تایید شده را در کنار زمان و هزینه ارزیابی کنید.{ph1} این " +"چیزی است که من می‌خواهم برای گردش کار کامل عامل هوش مصنوعی اندازه‌گیری کنم." + +#: website/public/blog/reverse-engineering-models/index.html:241 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "In this article" +msgstr "در این مقاله" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:243 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What better models changed{ph1}" +msgstr "{ph0}مدل‌های قوی‌تر چه چیزی را تغییر دادند؟{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:244 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Why specialize?{ph1}" +msgstr "{ph0}چرا تخصص داشته باشید؟{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:245 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Where experience lives{ph1}" +msgstr "{ph0}این تجربه در کجا باید ذخیره شود؟{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:246 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Learning the next experiment{ph1}" +msgstr "{ph0}یادگیری آزمایش مفید بعدی{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:247 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}RE and reinforcement learning{ph1}" +msgstr "{ph0}مهندسی معکوس و یادگیری تقویتی{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:248 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}When the oracle is wrong{ph1}" +msgstr "{ph0}هنگامی که بررسی کننده تأیید اشتباه است{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:249 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}A comparison worth running{ph1}" +msgstr "{ph0}مقایسه ای که ارزش اجرا دارد{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:250 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What kind of agent?{ph1}" +msgstr "{ph0}این چه نوع عامل هوش مصنوعی را پیشنهاد می کند؟{ph1}" + +#: website/public/blog/reverse-engineering-models/index.html:258 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What better models changed" +msgstr "مدل‌های قوی‌تر چه چیزی را تغییر دادند؟" + +#: website/public/blog/reverse-engineering-models/index.html:259 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Since August 2026, I have been exploring how much reverse engineering an " +"agent can carry out with access to a program and a way to check its work. " +"Most of my experience comes from reconstructing games, especially Touhou. As" +" I have changed models, the repositories have kept a record of what each " +"session managed to recover." +msgstr "" +"از آگوست 2026، من در حال بررسی میزان مهندسی معکوس یک عامل با دسترسی به یک " +"برنامه و راهی برای بررسی کار آن هستم. بیشتر تجربه من از بازسازی بازی ها به " +"خصوص Touhou ناشی می شود. همانطور که من مدل‌ها را تغییر داده‌ام، مخازن رکوردی" +" از آنچه هر جلسه توانسته است بازیابی کند را نگه داشته است." + +#: website/public/blog/reverse-engineering-models/index.html:267 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of that work aims at exact reconstruction: we compile the maintained " +"source under a pinned historical toolchain and check whether it reproduces " +"the target function. When a later model succeeds where earlier work stalled," +" the source changes show what made that progress possible." +msgstr "" +"هدف بیشتر این کار بازسازی دقیق است: ما منبع نگهداری شده را تحت یک زنجیره " +"ابزار تاریخی با نسخه های ثابت کامپایل می کنیم و بررسی می کنیم که آیا تابع " +"هدف را بازتولید می کند یا خیر. هنگامی که مدل بعدی در جایی که کار قبلی متوقف " +"شده بود موفق می شود، تغییرات منبع نشان می دهد که چه چیزی این پیشرفت را ممکن " +"کرده است." + +#: website/public/blog/reverse-engineering-models/index.html:274 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The temporary values had to stay alive together" +msgstr "متغیرهای موقت به طول عمر همپوشانی نیاز داشتند" + +#: website/public/blog/reverse-engineering-models/index.html:275 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of a TH09 attack function's logic had already been reconstructed, but " +"the compiler still stored temporary results differently from the original. " +"The remaining problem was how long those results needed to stay alive and " +"when their storage could be reused." +msgstr "" +"بیشتر منطق یک تابع حمله TH09 قبلاً بازسازی شده بود، اما کامپایلر همچنان " +"نتایج موقت را متفاوت از نسخه اصلی ذخیره می کرد. مشکل باقی‌مانده این بود که " +"این نتایج چقدر طول می‌کشد تا زنده بمانند و چه زمانی می‌توان از ذخیره‌سازی " +"آنها استفاده مجدد کرد." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:282 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A {ph0}GPT-6-Sol checkpoint{ph1} recorded this mismatch after testing " +"several ordinary source arrangements. When {ph2}GPT-6.1-Sol returned to the " +"problem{ph3}, it found a useful hypothesis about how long those temporary " +"values needed to stay alive." +msgstr "" +"یک {ph0}GPT-6-Sol رکورد جلسه{ph1} این عدم تطابق را پس از آزمایش چندین ترتیب " +"کد منبع معمولی گزارش کرد. وقتی {ph2}GPT-6.1-Sol دوباره مشکل را بررسی " +"کرد{ph3}، یک فرضیه مفید در مورد طول عمر این متغیرهای موقت ارائه کرد." + +#: website/public/blog/reverse-engineering-models/index.html:296 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The function produced temporary results and copied them into animation " +"positions. In C++, a temporary result stays alive until the expression using" +" it finishes. Grouping the real operations into one expression kept those " +"results alive together, so the compiler could no longer reuse the same " +"storage for all of them." +msgstr "" +"این تابع نتایج موقتی تولید کرد و آنها را در موقعیت های انیمیشن کپی کرد. در " +"C++، یک نتیجه موقت زنده می ماند تا زمانی که عبارت استفاده شده از آن تمام " +"شود. گروه‌بندی عملیات واقعی در یک عبارت، آن نتایج را با هم زنده نگه می‌دارد،" +" بنابراین کامپایلر دیگر نمی‌تواند از فضای ذخیره‌سازی یکسان برای همه آنها " +"استفاده مجدد کند." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:303 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"One result still shared storage with a value used later for collision " +"checks. Giving that value its own lifetime resolved the remaining overlap. " +"After several targeted experiments, {ph0}a clean rebuild confirmed that the " +"full function matched{ph1}." +msgstr "" +"یک نتیجه همچنان فضای ذخیره‌سازی را با مقداری به اشتراک گذاشته است که بعداً " +"برای بررسی‌های برخورد استفاده می‌شود. دادن این ارزش به طول عمر خودش، " +"همپوشانی باقیمانده را حل کرد. پس از چندین آزمایش هدفمند، {ph0}کامپایل مجدد " +"تمیز از منبع تأیید کرد که تابع کامل مطابقت دارد{ph1}." + +#: website/public/blog/reverse-engineering-models/index.html:312 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What interests me is how the discovery and the tests worked together. The " +"lifetime hypothesis gave the agent a new direction; compiler feedback helped" +" it refine that hypothesis into a source arrangement that matched." +msgstr "" +"من علاقه مندم که چگونه این کشف و آزمایش ها از یکدیگر حمایت کردند. فرضیه در " +"مورد طول عمر اشیاء موقت به عامل هوش مصنوعی مسیر جدیدی داد. بازخورد کامپایلر " +"به آن کمک کرد تا این فرضیه را به ترتیب کد منبع که مطابقت دارد، اصلاح کند." + +#: website/public/blog/reverse-engineering-models/index.html:318 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A different control-flow graph" +msgstr "یک نمودار کنترل جریان متفاوت" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:319 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A second TH09 function shows why that new direction can matter. Here, the " +"obstacle was how several branches shared their final work. " +"{ph0}GPT-6.1-Sol's earlier work{ph1} got parts of the function to match, but" +" the branches still joined up differently from the original." +msgstr "" +"تابع دوم TH09 نشان می دهد که چرا آن جهت جدید می تواند مهم باشد. در اینجا، " +"مانع این بود که چگونه چندین شعبه کار نهایی خود را به اشتراک گذاشتند. کار " +"قبلی {ph0}GPT-6.1-Sol{ph1} بخش‌هایی از تابع را مطابقت داد، اما شاخه‌ها " +"همچنان متفاوت از نسخه اصلی به یکدیگر متصل شدند." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:330 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}GPT-6-Astra changed the surrounding control-flow graph{ph1}. The main " +"paths now led to a shared final update and return. In that source structure," +" the compiler combined the branches in the same way as the original. The " +"complete function matched, and a {ph2}separate rebuild check confirmed " +"it{ph3}." +msgstr "" +"{ph0}GPT-6-Astra نمودار کنترل جریان اطراف را تغییر داد{ph1}. مسیرهای اصلی " +"اکنون به یک به‌روزرسانی نهایی مشترک و بازگشت منجر می‌شوند. در آن ساختار " +"منبع، کامپایلر شاخه ها را به همان روش اصلی ترکیب می کند. تابع کامل مطابقت " +"داشت و {ph2}بررسی بازسازی جداگانه آن را تأیید کرد{ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:342 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Changing the representation opened a more productive search space.{ph1}" +" Both cases give me a reason to take general model capability seriously: an " +"agent can keep trying local variations for a long time before finding a " +"better way to express the surrounding work." +msgstr "" +"{ph0}تغییر نمایش، فضای جستجوی سازنده‌تری را باز کرد.{ph1} هر دو مورد دلیلی " +"برای جدی گرفتن قابلیت مدل کلی به من می‌دهد: یک نماینده می‌تواند قبل از یافتن" +" راهی بهتر برای بیان کار اطراف، برای مدت طولانی تغییرات محلی را امتحان کند." + +#: website/public/blog/reverse-engineering-models/index.html:351 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"At the same time, the later sessions started with earlier source and " +"recorded failures. That accumulated work was part of their advantage. The " +"history lets us inspect the discoveries and their checks; a controlled " +"comparison would help separate the effect of the model from the experience " +"it inherited." +msgstr "" +"در همان زمان، جلسات بعدی با منبع قبلی شروع شد و خرابی ها ثبت شد. این کار " +"انباشته بخشی از مزیت آنها بود. تاریخ به ما اجازه می دهد تا کشفیات و بررسی " +"های آنها را بررسی کنیم. یک مقایسه کنترل شده به تفکیک اثر مدل از تجربه ای که " +"به ارث برده است کمک می کند." + +#: website/public/blog/reverse-engineering-models/index.html:358 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is where the question becomes interesting to me. Some of the progress " +"came from finding a new representation, while much of the remaining work " +"involved testing source forms against a familiar compiler. How much needs " +"broad reasoning ability, and how much could a model learn through repeated " +"experience with that compiler and code?" +msgstr "" +"این چیزی است که سوال را برای من جالب می کند. برخی پیشرفت ها از یافتن " +"نمایندگی جدید حاصل شد. بسیاری از کارهای باقی مانده شامل آزمایش ترتیبات مختلف" +" کد منبع در برابر یک کامپایلر آشنا بود. چه مقدار از این به توانایی استدلال " +"گسترده نیاز دارد؟ یک مدل با کار مکرر با آن کامپایلر و کد چقدر می تواند یاد " +"بگیرد؟" + +#: website/public/blog/reverse-engineering-models/index.html:368 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why might a specialist do well?" +msgstr "چرا یک مدل تخصصی هوش مصنوعی ممکن است خوب عمل کند؟" + +#: website/public/blog/reverse-engineering-models/index.html:369 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Touhou gives us a concrete reason to expect that experience to help. Much of" +" the game code comes from the same author, ZUN, so recurring habits are a " +"reasonable starting assumption. The repositories back that up with useful " +"examples: related loops and helper arrangements have supplied hypotheses " +"across games." +msgstr "" +"Touhou دلیل مشخصی به ما می دهد که انتظار داشته باشیم آن تجربه کمک کند. بیشتر" +" کد بازی از همان نویسنده، ZUN می آید، بنابراین عادت های تکرار شونده یک فرض " +"شروع منطقی هستند. مخازن با مثال‌های مفید این موضوع را تأیید می‌کنند: " +"حلقه‌های مرتبط و ترتیبات کمکی فرضیه‌هایی را در بین بازی‌ها ارائه کرده‌اند." + +#: website/public/blog/reverse-engineering-models/index.html:376 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those source patterns become more specific when we include the compiler. " +"Under a particular version and set of options, small changes in source can " +"produce repeatable differences in machine code. Some of those differences " +"are surprisingly obscure." +msgstr "" +"وقتی کامپایلر را وارد کنیم، این الگوهای منبع خاص تر می شوند. تحت یک نسخه خاص" +" و مجموعه ای از گزینه ها، تغییرات کوچک در منبع می تواند تفاوت های قابل تکرار" +" در کد ماشین ایجاد کند. برخی از این تفاوت ها به طرز شگفت آوری مبهم هستند." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: Supervisor::LoadDat +#. {ph4}: +#: website/public/blog/reverse-engineering-models/index.html:382 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In TH095, for example, {ph0}even the name used for a local variable could " +"change where VC7.1 stored it{ph1}. Moving the declaration alone did not " +"solve the layout, but an earlier background function had revealed a naming " +"pattern that affected the compiler's choice. That gave us a more promising " +"hypothesis for {ph2}the reconstruction of {ph3}{ph4}." +msgstr "" +"به عنوان مثال، در TH095، {ph0}حتی نام یک متغیر محلی می‌تواند بر جایی که " +"VC7.1 آن را ذخیره کرده است، تأثیر بگذارد{ph1}. انتقال اعلان آن طرح ذخیره " +"سازی را حل نکرد. با این حال، یک تابع پس زمینه قبلی یک الگوی نامگذاری را نشان" +" داده بود که بر انتخاب کامپایلر تأثیر می گذاشت. این فرضیه امیدوارکننده تری " +"را برای {ph2}بازسازی {ph3}{ph4} پیشنهاد کرد." + +#: website/public/blog/reverse-engineering-models/index.html:396 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reusing that pattern for the file-size variable produced the required layout" +" without adding storage. The full function then matched." +msgstr "" +"استفاده مجدد از آن الگو برای متغیر اندازه فایل، طرح مورد نیاز را بدون افزودن" +" فضای ذخیره سازی ایجاد کرد. سپس تابع کامل مطابقت داشت." + +#: website/public/blog/reverse-engineering-models/index.html:401 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is the sort of narrow detail a general model may not know when it " +"arrives. A specialist trained on this compiler could encounter many such " +"cases and learn which source variations are worth testing first." +msgstr "" +"این یک جزئیات بسیار خاص است که یک مدل هوش مصنوعی همه منظوره ممکن است زمانی " +"که برای اولین بار با مشکل مواجه می شود از آن مطلع نباشد. یک مدل تخصصی برای " +"این کامپایلر می‌تواند موارد بسیاری از این قبیل را ببیند و یاد بگیرد که کدام " +"تغییرات کد منبع ارزش آزمایش را دارند." + +#: website/public/blog/reverse-engineering-models/index.html:407 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The opportunity is especially clear in final matching problems where we " +"already understand the behavior. We are then asking which ordinary source " +"form this compiler turns into the observed instructions. Specialization " +"could help the model propose those forms more efficiently, while the target " +"still determines which hypotheses survive." +msgstr "" +"این فرصت به ویژه در مسائل تطبیق نهایی که ما قبلاً رفتار را درک کرده ایم، " +"واضح است. سپس می‌پرسیم این کامپایلر کدام ترتیب کد منبع معمولی را به " +"دستورالعمل‌های مشاهده شده تبدیل می‌کند. تخصص می تواند به مدل کمک کند تا آن " +"اشکال را به طور کارآمدتر پیشنهاد کند، در حالی که هدف هنوز تعیین می کند که " +"کدام فرضیه ها باقی می مانند." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:415 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The useful patterns would have to retain their context, though. TH08 uses an" +" earlier Visual C++ version than TH095 and TH09. TH04 uses a different DOS-" +"era compiler, and its {ph0}player reconstruction{ph1} needed different " +"hypotheses about source patterns and how the compiled code was combined into" +" an executable. Shared authorship gives us a prior, but each target tells us" +" whether it applies. Distinguishing those contexts is part of what a " +"specialist would need to learn." +msgstr "" +"الگوهای مفید باید زمینه خود را حفظ کنند. TH08 از نسخه Visual C++ قبلی نسبت " +"به TH095 و TH09 استفاده می کند. TH04 از یک کامپایلر متفاوت از دوران DOS " +"استفاده می کند. {ph0}بازسازی کد بازیکن{ph1} آن به فرضیه‌های مختلفی در مورد " +"الگوهای کد منبع و نحوه پیوند کد کامپایل‌شده به یک فایل اجرایی نیاز داشت. " +"داشتن یک نویسنده یک انتظار اولیه را به ما می‌دهد، اما هر هدف به ما می‌گوید " +"که آیا کاربرد دارد یا خیر. یک مدل تخصصی باید این زمینه ها را متمایز کند." + +#: website/public/blog/reverse-engineering-models/index.html:431 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Where should that experience live?" +msgstr "این تجربه در کجا باید ذخیره شود؟" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:432 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before deciding to teach those patterns through model training, I would look" +" at the knowledge we already keep in the repository. In the {ph0}previous " +"essay{ph1}, I wrote about it becoming the project's working memory. That " +"memory includes compiler lessons, along with the source and failed " +"experiments that made them useful." +msgstr "" +"قبل از تصمیم به آموزش آن الگوها از طریق آموزش مدل، به دانشی که قبلاً در مخزن" +" نگهداری می‌کنیم نگاهی می‌اندازم. در {ph0}مقاله قبلی{ph1}، درباره تبدیل شدن " +"آن به حافظه کاری پروژه نوشتم. این حافظه شامل درس های کامپایلر، همراه با منبع" +" و آزمایش های ناموفق است که آنها را مفید ساخته است." + +#: website/public/blog/reverse-engineering-models/index.html:441 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"By retrieving that record, an agent can start with a known result and " +"inspect how we reached it. It can also avoid alternatives that have already " +"failed in the same context. The starting point improves without changing the" +" model's weights." +msgstr "" +"با بازیابی آن رکورد، یک نماینده می تواند با یک نتیجه شناخته شده شروع کند و " +"نحوه رسیدن ما به آن را بررسی کند. همچنین می تواند از جایگزین هایی که قبلاً " +"در همان زمینه شکست خورده اند اجتناب کند. نقطه شروع بدون تغییر پارامترهای " +"آموزش دیده مدل بهبود می یابد." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:447 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"When reconstructing TH09's enemy-spawn function, a {ph0}GPT-6-Luna-max " +"session{ph1} drew a hypothesis from TH08's corresponding source family. That" +" earlier source suggested how to structure the loop and handle failures " +"within it. TH09's own evidence supplied the game-specific details, and clean" +" rebuilds confirmed the complete match." +msgstr "" +"هنگام بازسازی عملکرد TH09 برای ایجاد دشمن، یک جلسه {ph0}GPT-6-Luna-max{ph1} " +"یک فرضیه را از کد منبع TH08 مرتبط گرفت. آن کد قبلی نحوه سازماندهی حلقه و " +"مدیریت خرابی های داخل آن را پیشنهاد می کرد. شواهد خود TH09 جزئیات مربوط به " +"بازی را ارائه کرد. بازسازی از منبع تأیید کرد که عملکرد کامل مطابقت دارد." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:458 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}\"The model did not know that compiler quirk\" and \"the model could " +"not use it\" are different problems.{ph1} Retrieval may solve the first, but" +" turning that fact into a useful candidate still takes work. And if the " +"candidate fails, the model must decide what the result means for its next " +"attempt. Knowing the relevant pattern is only the beginning of that process." +msgstr "" +"{ph0}\"مدل نمی‌دانست که ویژگی کامپایلر\" و \"مدل نمی‌تواند از آن استفاده " +"کند\" مشکلات متفاوتی هستند.{ph1} بازیابی ممکن است مشکل اول را حل کند، اما " +"تبدیل این واقعیت به یک نامزد مفید هنوز کار می‌برد. و اگر کد منبع کاندید شکست" +" بخورد، مدل باید تصمیم بگیرد که نتیجه برای تلاش بعدی چه معنایی دارد. دانستن " +"الگوی مربوطه تنها آغاز آن فرآیند است." + +#: website/public/blog/reverse-engineering-models/index.html:468 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For that reason, I would include a smaller general model with good retrieval" +" in any comparison. Giving it the same library of experience would help us " +"identify what training adds: better candidate generation, better decisions " +"after feedback, or both." +msgstr "" +"به همین دلیل، من یک مدل هوش مصنوعی همه منظوره کوچکتر را با بازیابی خوب در هر" +" مقایسه قرار می دهم. ارائه همان کتابخانه تجربی به ما کمک می کند تا تشخیص " +"دهیم که آموزش چه چیزی را اضافه می کند: تولید نامزد بهتر، تصمیم گیری بهتر پس " +"از بازخورد، یا هر دو." + +#: website/public/blog/reverse-engineering-models/index.html:476 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Learning the next useful experiment" +msgstr "یادگیری آزمایش مفید بعدی" + +#: website/public/blog/reverse-engineering-models/index.html:477 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those decisions after feedback are what I would most like to study. In " +"agentic reconstruction, the model proposes an explanation and tests it " +"against the program. Even when the candidate fails, the result can narrow " +"the remaining possibilities and suggest a more useful next experiment." +msgstr "" +"آن تصمیمات پس از بازخورد همان چیزی است که من بیشتر دوست دارم مطالعه کنم. در " +"بازسازی عاملی، مدل توضیحی را پیشنهاد می کند و آن را در برابر برنامه آزمایش " +"می کند. حتی زمانی که کد منبع نامزد ناموفق باشد، نتیجه می تواند احتمالات " +"باقیمانده را محدود کند و آزمایش بعدی مفیدتری را پیشنهاد کند." + +#: website/public/blog/reverse-engineering-models/index.html:484 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Consider the temporary-results example again. Its final source shows the " +"arrangement that worked, but the route to that source contains more " +"information. Each of the following experiments helped identify what still " +"needed to change." +msgstr "" +"مثال موقت نتایج را دوباره در نظر بگیرید. منبع نهایی آن ترتیبی را نشان می دهد" +" که کارآمد است، اما مسیر رسیدن به آن منبع حاوی اطلاعات بیشتری است. هر یک از " +"آزمایش‌های زیر به شناسایی آنچه که هنوز باید تغییر کند کمک کرد." + +#: website/public/blog/reverse-engineering-models/index.html:497 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Experiment" +msgstr "آزمایش کنید" + +#: website/public/blog/reverse-engineering-models/index.html:498 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What its result taught us" +msgstr "آنچه نتیجه آن به ما آموخت" + +#: website/public/blog/reverse-engineering-models/index.html:503 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Try a different way to copy the results" +msgstr "روش دیگری را برای کپی کردن نتایج امتحان کنید" + +#: website/public/blog/reverse-engineering-models/index.html:504 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The copy code changes, but the function still differs." +msgstr "کد کپی تغییر می کند، اما عملکرد همچنان متفاوت است." + +#: website/public/blog/reverse-engineering-models/index.html:509 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Use the returned values directly" +msgstr "از مقادیر برگشتی مستقیما استفاده کنید" + +#: website/public/blog/reverse-engineering-models/index.html:510 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Copying follows the original pattern, but temporary storage still differs." +msgstr "" +"کپی کردن از الگوی اصلی پیروی می کند، اما ذخیره سازی موقت همچنان متفاوت است." + +#: website/public/blog/reverse-engineering-models/index.html:516 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the temporary results alive together" +msgstr "همه متغیرهای موقت را همزمان معتبر نگه دارید" + +#: website/public/blog/reverse-engineering-models/index.html:517 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"They get separate storage, but one still overlaps a value used later for " +"collision checks." +msgstr "" +"آنها فضای ذخیره جداگانه ای دریافت می کنند، اما هنوز هم مقداری که بعداً برای " +"بررسی برخورد استفاده می شود، همپوشانی دارد." + +#: website/public/blog/reverse-engineering-models/index.html:523 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the collision-check value separate" +msgstr "مقدار collision-check را جدا نگه دارید" + +#: website/public/blog/reverse-engineering-models/index.html:524 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A clean rebuild confirms the complete match." +msgstr "یک کامپایل مجدد تمیز از منبع، مطابقت کامل را تأیید می کند." + +#: website/public/blog/reverse-engineering-models/index.html:529 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Selected experiments from the attack-function reconstruction above, grouped " +"by what they taught us." +msgstr "" +"آزمایش‌های منتخب از بازسازی عملکرد حمله در بالا، گروه‌بندی شده بر اساس آنچه " +"به ما آموختند." + +#: website/public/blog/reverse-engineering-models/index.html:533 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A training record that keeps this sequence can teach more than the final " +"source alone. It connects each proposed change to an observed result and the" +" next decision. Keeping ineffective variations matters too: they tell a " +"later agent which hypotheses have already been tested in that context." +msgstr "" +"یک رکورد آموزشی که این توالی را حفظ می کند می تواند بیشتر از منبع نهایی به " +"تنهایی آموزش دهد. هر تغییر پیشنهادی را به یک نتیجه مشاهده شده و تصمیم بعدی " +"مرتبط می کند. حفظ تغییرات بی اثر نیز مهم است: آنها به عامل بعدی می گویند که " +"کدام فرضیه قبلاً در آن زمینه آزمایش شده است." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:540 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}I would want the model to learn how to use feedback.{ph1} A neutral " +"result is a reason to reconsider the variation; a changed mismatch may point" +" to another cause. When local edits stop being informative, the agent should" +" be able to revisit the source representation or inspect the evidence again." +msgstr "" +"{ph0}من می‌خواهم مدل یاد بگیرد که چگونه از بازخورد استفاده کند.{ph1} یک " +"نتیجه خنثی دلیلی برای تجدید نظر در تغییر است. تغییر عدم تطابق ممکن است به " +"دلیل دیگری اشاره کند. وقتی ویرایش‌های محلی آموزنده نیستند، عامل باید بتواند " +"بازنمایی منبع را بازبینی کند یا شواهد را دوباره بررسی کند." + +#: website/public/blog/reverse-engineering-models/index.html:549 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Learning to make those choices requires autonomy. Sometimes the useful next " +"step is inspecting another caller rather than compiling another edit. A " +"rigid sequence of required rewrites would remove that choice, along with " +"part of the behavior we want the model to learn." +msgstr "" +"یادگیری انجام آن انتخاب ها مستلزم استقلال است. گاهی اوقات گام بعدی مفید، " +"بررسی تماس گیرنده دیگر به جای کامپایل ویرایش دیگری است. یک توالی سفت و سخت " +"از بازنویسی های مورد نیاز این انتخاب را حذف می کند، همراه با بخشی از رفتاری " +"که می خواهیم مدل یاد بگیرد." + +#: website/public/blog/reverse-engineering-models/index.html:558 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why RE is interesting for reinforcement learning" +msgstr "چرا مهندسی معکوس برای یادگیری تقویتی جالب است" + +#: website/public/blog/reverse-engineering-models/index.html:559 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Bounded reconstruction gives us a practical environment for studying these " +"choices. Once the target and compiler are pinned, many source experiments " +"produce quick, repeatable feedback. The agent can compile a candidate, " +"inspect the mismatch and use that observation to choose its next step." +msgstr "" +"بازسازی با دامنه مشخص محیطی عملی برای مطالعه این تصمیم‌ها فراهم می‌کند. با " +"ثابت نگه داشتن نسخه هدف و کامپایلر، بسیاری از آزمایش‌های کد منبع بازخورد " +"سریع و تکرارپذیر می‌دهند. عامل می‌تواند کد پیشنهادی را کامپایل کند، تفاوت‌ها" +" را بررسی کند و گام بعدی را انتخاب کند." + +#: website/public/blog/reverse-engineering-models/index.html:566 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reinforcement learning could train that decision-making process through the " +"feedback it receives. A candidate accepted by a defined check gives us a " +"concrete outcome, while the sequence of attempts lets us observe how much " +"work it took to reach it." +msgstr "" +"یادگیری تقویتی می تواند این فرآیند تصمیم گیری را از طریق بازخوردی که دریافت " +"می کند آموزش دهد. یک نامزد پذیرفته شده توسط یک چک تعریف شده یک نتیجه مشخص به" +" ما می دهد، در حالی که توالی تلاش ها به ما امکان می دهد مشاهده کنیم که چقدر " +"کار برای رسیدن به آن لازم است." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:572 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"There is already research supporting this direction. {ph0}LLM4Decompile{ph1}" +" studies models specialized for decompilation. {ph2}AutoDecompiler{ph3} " +"trains multi-turn refinement using compiler and execution feedback. Their " +"studies focus on functional recovery. For this work, I would use them as a " +"starting point for the related task of finding source that matches a " +"historical compiler's output." +msgstr "" +"در حال حاضر تحقیقاتی وجود دارد که از این جهت حمایت می کند. " +"{ph0}LLM4Decompile{ph1} مدل های تخصصی برای دیکامپایل را مطالعه می کند. " +"{ph2}AutoDecompiler{ph3} با استفاده از بازخورد کامپایلر و اجرا، اصلاح چند " +"نوبتی را آموزش می‌دهد. مطالعات آنها بر بازیابی عملکردی متمرکز است. برای این " +"کار، من از آنها به عنوان نقطه شروع برای کار مرتبط یافتن منبعی که با خروجی یک" +" کامپایلر تاریخی مطابقت دارد استفاده می کنم." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:584 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To reach such a well-defined task, though, we may first need to establish " +"what a function owns or devise a new runtime measurement. In TH08, for " +"example, a {ph0}collision difference led back to how the original program " +"handled floating-point calculations{ph1}. Before the agent could test a " +"useful correction, it had to work out what to measure." +msgstr "" +"با این حال، برای رسیدن به چنین وظیفه ای به خوبی تعریف شده، ممکن است ابتدا " +"نیاز داشته باشیم که یک تابع دارای چه منابعی است یا یک روش جدید برای مشاهده " +"رفتار برنامه هنگام اجرا را طراحی کنیم. به عنوان مثال، در TH08، یک {ph0}تفاوت" +" برخورد منجر به نحوه مدیریت برنامه اصلی با محاسبات ممیز شناور شد{ph1}. قبل " +"از اینکه عامل بتواند یک تصحیح مفید را آزمایش کند، باید تعیین می کرد که چه " +"چیزی را باید اندازه گیری کند." + +#: website/public/blog/reverse-engineering-models/index.html:595 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This suggests another role for a broadly capable model: helping make the " +"problem checkable in the first place. Once that check exists, we can " +"investigate how far a learned search policy can get with it." +msgstr "" +"این یک نقش دیگر را برای یک مدل به طور گسترده نشان می دهد: کمک به قابل بررسی " +"کردن مشکل در وهله اول. هنگامی که آن چک وجود داشت، می‌توانیم بررسی کنیم که یک" +" خط‌مشی جستجوی آموخته شده تا چه حد می‌تواند با آن برسد." + +#: website/public/blog/reverse-engineering-models/index.html:603 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "When the oracle is wrong" +msgstr "هنگامی که بررسی کننده تأیید اشتباه است" + +#: website/public/blog/reverse-engineering-models/index.html:604 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Once we train against that check, its coverage becomes part of what the " +"model learns. I call the reference or checker an oracle. Even if it gives a " +"repeatable answer, we still need to understand what that answer covers." +msgstr "" +"وقتی یک بررسی را برای آموزش به کار می‌بریم، دامنه آن هم بخشی از محیط یادگیری" +" می‌شود. من چنین مرجع یا ابزار بررسی را اوراکل می‌نامم. حتی پاسخ تکرارپذیر " +"را باید در محدوده چیزی فهمید که واقعاً بررسی می‌کند." + +#: website/public/blog/reverse-engineering-models/index.html:610 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"TH08 showed how much can hide outside that coverage. Some reconstructed " +"functions passed their exact comparisons with wrong floating constants. The " +"comparator adjusted relocated addresses before checking instructions, but it" +" checked the values behind those addresses only when they had been " +"explicitly recorded for verification. Other literal references could pass " +"without their values being examined." +msgstr "" +"TH08 نشان داد که مشکلات می‌توانند بیرون از دامنه بررسی پنهان بمانند. بعضی " +"توابع بازسازی‌شده با وجود ثابت‌های ممیز شناور اشتباه، مقایسه دقیق را قبول " +"شدند. مقایسه‌کننده پیش از بررسی دستورها آدرس‌های جابه‌جا‌شده را تنظیم " +"می‌کرد، اما مقدار اشاره‌شده را فقط وقتی بررسی می‌کرد که برای اعتبارسنجی " +"صریحاً ثبت شده بود. ارجاع‌های دیگر به ثابت‌ها می‌توانستند بدون بررسی مقدار " +"قبول شوند." + +#. {ph0}: +#. {ph1}: +#. {ph2}: 1.0f +#. {ph3}: 32.0f +#: website/public/blog/reverse-engineering-models/index.html:619 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To close that gap, a {ph0}wider audit{ph1} found incorrect constants in " +"functions that had already been accepted. In one calculation, the " +"reconstructed source added {ph2} where the original added {ph3}." +msgstr "" +"برای رفع این شکاف، یک {ph0}ممیزی گسترده‌تر{ph1} ثابت‌های نادرستی را در " +"توابعی که قبلاً تأیید شده بودند پیدا کرد. در یک محاسبه، کد منبع بازسازی شده " +"{ph2} را اضافه کرد، در حالی که کد اصلی {ph3} را اضافه کرد." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:629 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A learner rewarded by the old check could learn to produce the wrong " +"reconstruction.{ph1} We had not trained an RL model against this oracle; the" +" observed failure was in the check itself. But using that check as a reward " +"would have made the same blind spot part of the learning environment." +msgstr "" +"{ph0}مدلی که توسط چک راستی‌آزمایی قدیمی پاداش می‌گیرد، می‌تواند یاد بگیرد که" +" یک بازسازی نادرست ایجاد کند.{ph1} ما یک مدل RL را با استفاده از آن " +"بررسی‌کننده آموزش ندادیم. نقصی که مشاهده کردیم در خود چک بود. اما اگر از آن " +"به عنوان پاداش استفاده می کردیم، همان نقطه کور بخشی از محیط آموزشی می شد." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:639 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A functional test can have a similar blind spot. {ph0}Differential-fuzzing " +"research on LLM decompilers{ph1} has found behavior differences beyond the " +"supplied tests. Passing those tests therefore needs to be read in terms of " +"the cases they exercise, just as an instruction match needs to be read in " +"terms of the bytes and data it checks." +msgstr "" +"یک تست عملکردی می تواند نقطه کور مشابهی داشته باشد. {ph0}تحقیق با استفاده از" +" fuzzing دیفرانسیل در دیکامپایلرهای LLM{ph1} تفاوت‌های رفتاری را خارج از " +"آزمون‌های ارائه شده یافت. بنابراین ما باید یک آزمون قبولی را بر حسب مواردی " +"که انجام می دهد درک کنیم. به طور مشابه، یک تطابق دستورالعمل باید از نظر بایت" +" ها و داده هایی که تأیید می کند درک شود." + +#: website/public/blog/reverse-engineering-models/index.html:649 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before using either kind of check as a reward, I would test it with " +"deliberate failures. I would also keep final acceptance outside the " +"candidate's editable source, so the model has to satisfy the check through " +"its reconstruction. Otherwise, something like a byte array copied from the " +"target could satisfy a naive score without recovering useful source." +msgstr "" +"قبل از استفاده از هر نوع بررسی تأیید به عنوان پاداش، آن را با نتایج عمداً " +"نادرست آزمایش می‌کنم. من همچنین پذیرش نهایی را از کد منبعی که عامل هوش " +"مصنوعی می تواند ویرایش کند جدا نگه می دارم. مدل باید با بازسازی صحیح برنامه،" +" تأیید صحت داشته باشد. در غیر این صورت، کپی کردن یک آرایه بایتی از هدف می " +"تواند یک سیستم امتیازدهی ساده لوح را بدون بازیابی کد منبع مفید فریب دهد." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:657 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Improving the learner and improving its oracle belong to the same " +"research problem.{ph1} As the learner gets better at optimizing the feedback" +" we give it, understanding that feedback becomes more important too." +msgstr "" +"{ph0}بهبود یادگیرنده و بهبود بررسی‌کننده تأیید آن مربوط به یک مسئله تحقیقاتی" +" است.{ph1} همانطور که یادگیرنده در بهینه‌سازی بازخوردی که ما به آن می‌دهیم " +"بهتر می‌شود، درک آن بازخورد نیز مهم‌تر می‌شود." + +#: website/public/blog/reverse-engineering-models/index.html:667 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A comparison worth running" +msgstr "مقایسه ای که ارزش اجرا دارد" + +#: website/public/blog/reverse-engineering-models/index.html:668 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"With those checks in place, I would start the comparison on a fixed VC7.1 " +"task family. First, I would establish baselines for a strong general model " +"and a smaller one. Adding repository knowledge to the smaller model would " +"show what retrieval changes. From there, supervised specialization and an " +"RL-trained version would let us examine what training adds." +msgstr "" +"با انجام این بررسی‌ها، مقایسه را روی یک خانواده کار ثابت VC7.1 شروع می‌کنم. " +"اول، من خطوط پایه را برای یک مدل هوش مصنوعی همه منظوره قوی و یک مدل کوچکتر " +"ایجاد می کنم. افزودن دانش مخزن به مدل کوچکتر، تغییرات بازیابی را نشان می " +"دهد. از آنجا، تخصص تحت نظارت و یک نسخه آموزش داده شده RL به ما اجازه می دهد " +"بررسی کنیم که آموزش چه چیزی را اضافه می کند." + +#: website/public/blog/reverse-engineering-models/index.html:676 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To make those comparisons informative, each model would receive the same " +"target evidence and tools. For the training-specific comparisons, I would " +"keep the base model and available knowledge fixed as well. Repeated trials " +"would help distinguish a reliable search strategy from a lucky candidate." +msgstr "" +"برای اینکه این مقایسه ها آموزنده باشد، هر مدل شواهد و ابزارهای هدف یکسانی را" +" دریافت می کند. برای مقایسه‌های آموزشی خاص، مدل پایه و دانش موجود را نیز " +"ثابت نگه می‌دارم. آزمایش‌های مکرر به تمایز یک استراتژی جستجوی قابل اعتماد از" +" راه‌حل نامزدی که به طور تصادفی موفق می‌شود کمک می‌کند." + +#: website/public/blog/reverse-engineering-models/index.html:683 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would then separate two starting conditions: reconstructing a function " +"from its instructions, and closing the last few differences in maintained " +"source. A specialist might be excellent at the second task while a general " +"model does much of the work needed to reach it. Measuring them separately " +"would reveal that division of work." +msgstr "" +"سپس دو شرط شروع را جدا می‌کنم: بازسازی یک تابع از دستورالعمل‌های آن، و بستن " +"چند تفاوت آخر در منبع حفظ شده. یک مدل هوش مصنوعی تخصصی ممکن است در کار دوم " +"عالی باشد در حالی که یک مدل هوش مصنوعی همه منظوره بسیاری از کارهای مورد نیاز" +" برای رسیدن به آن را انجام می دهد. اندازه گیری آنها به طور جداگانه این تقسیم" +" کار را آشکار می کند." + +#: website/public/blog/reverse-engineering-models/index.html:691 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both tasks need genuinely held-out examples. Because related Touhou games " +"share engine code, holding out a game's name would still allow nearly " +"identical functions to appear in training. I would instead hold out source " +"families and keep their accepted answers out of both training and retrieval." +msgstr "" +"هر دو کار به نمونه‌های آزمایشی نیاز دارند که از آموزش و بازیابی حذف شوند. کد" +" موتور اشتراک گذاری بازی های مرتبط Touhou. حذف یک بازی از تمرین همچنان می " +"تواند عملکردهای تقریباً یکسانی را در مجموعه تمرینی باقی بگذارد. در عوض، من " +"کل گروه‌های کد منبع مرتبط را حذف می‌کنم و پاسخ‌های تأیید شده‌شان را هم از " +"داده‌های آموزشی و هم از کتابخانه بازیابی خارج می‌کنم." + +#: website/public/blog/reverse-engineering-models/index.html:698 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"On those examples, the outcome would be complete, independently validated " +"matches. Giving each model the same candidate or compile budget would help " +"compare search quality. Measuring elapsed time and total cost, including " +"compilation, would tell us whether that quality produces an efficient " +"workflow. Training costs would need their own accounting too." +msgstr "" +"برای این مثال‌ها، نتیجه، مطابقت‌های کامل و تأیید شده مستقل خواهد بود. " +"بودجه‌های برابر برای پیشنهادهای نامزد یا تلاش‌های کامپایل به مقایسه کیفیت " +"جستجو کمک می‌کند. اندازه‌گیری زمان و هزینه کل، از جمله کامپایل، نشان می‌دهد " +"که آیا این کیفیت گردش کار کارآمدی ایجاد می‌کند یا خیر. هزینه های آموزشی باید" +" جداگانه اندازه گیری شود." + +#: website/public/blog/reverse-engineering-models/index.html:706 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both views matter because a cheaper proposal can become an expensive " +"workflow if it causes more uninformative compiles. Conversely, a slower " +"model may pay for itself by finding a useful experiment sooner. The " +"comparison should capture those costs." +msgstr "" +"هر دو اندازه گیری مهم است. یک پروپوزال ارزان می‌تواند منجر به گردش کار " +"گران‌قیمتی شود اگر باعث کامپایل‌های بیشتری شود که اطلاعات مفیدی ارائه " +"نمی‌کند. برعکس، یک مدل کندتر ممکن است اقتصادی تر باشد اگر آزمایش مفیدی را " +"زودتر پیدا کند. مقایسه باید شامل این هزینه ها باشد." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:712 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Counting accepted results also keeps intermediate progress in perspective. " +"One TH09 {ph0}replay-save candidate{ph1} looked promising because the " +"compiled function had the expected size, but a full comparison still found " +"substantial differences. Changes in how the compiler arranges instructions " +"can also make a candidate look closer without resolving the underlying " +"problem. Such diagnostics can guide the next experiment, but I would " +"validate any reward for \"getting closer\" against the full acceptance " +"check." +msgstr "" +"شمارش نتایج پذیرفته شده همچنین پیشرفت متوسط را در چشم انداز نگه می دارد. یک " +"TH09 {ph0}replay-save نامزد{ph1} امیدوارکننده به نظر می‌رسید، زیرا تابع " +"کامپایل‌شده اندازه مورد انتظار را داشت، اما یک مقایسه کامل همچنان تفاوت‌های " +"اساسی را پیدا کرد. تغییرات در نحوه چیدمان دستورالعمل ها توسط کامپایلر می " +"تواند بدون حل مشکل اساسی، نامزد را نزدیک تر کند. چنین تشخیص‌هایی می‌توانند " +"آزمایش بعدی را راهنمایی کنند، اما من هر پاداشی را برای «نزدیک‌تر شدن» در " +"برابر بررسی پذیرش کامل تأیید می‌کنم." + +#: website/public/blog/reverse-engineering-models/index.html:727 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would be interested in the pattern across these results. If a specialist " +"does well within a familiar compiler family, does that advantage survive a " +"toolchain change? And once both models have good retrieval, how much does " +"training still add? The answers would help us decide where each kind of " +"model is most useful." +msgstr "" +"من به الگوی این نتایج علاقه مند هستم. اگر یک مدل تخصصی هوش مصنوعی در یک " +"خانواده کامپایلر آشنا به خوبی عمل کند، آیا این مزیت از تغییر زنجیره ابزار " +"باقی می ماند؟ و هنگامی که هر دو مدل هوش مصنوعی بازیابی خوبی داشته باشند، " +"آموزش هنوز چقدر اضافه می کند؟ پاسخ‌ها به ما کمک می‌کنند تصمیم بگیریم که هر " +"نوع مدل در کجا مفیدتر است." + +#: website/public/blog/reverse-engineering-models/index.html:736 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What kind of agent does this suggest?" +msgstr "این چه نوع عامل هوش مصنوعی را پیشنهاد می کند؟" + +#: website/public/blog/reverse-engineering-models/index.html:737 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"If the models turn out to help at different stages, a useful arrangement " +"might be a broad agent with access to a specialist. Once a task becomes a " +"familiar compiler search, it could ask the specialist for candidates. If the" +" results stop making sense, the broader agent could return to the target " +"evidence or investigate the measurement itself." +msgstr "" +"اگر مدل ها در مراحل مختلف کمک کننده باشند، یک ترتیب مفید ممکن است عاملی " +"گسترده با دسترسی به یک مدل تخصصی هوش مصنوعی باشد. هنگامی که یک کار به یک " +"جستجوی کامپایلر آشنا تبدیل می شود، می تواند مدل تخصصی هوش مصنوعی را برای " +"نامزدها بپرسد. اگر نتایج منطقی نباشند، عامل گسترده‌تر می‌تواند به شواهد هدف " +"بازگردد یا خود اندازه‌گیری را بررسی کند." + +#: website/public/blog/reverse-engineering-models/index.html:745 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In that arrangement, specialization becomes a tool the agent can choose as " +"the investigation develops. The agent keeps the freedom to change direction " +"that made the earlier breakthroughs possible." +msgstr "" +"در آن ترتیب، تخصص به ابزاری تبدیل می‌شود که عامل می‌تواند با توسعه تحقیقات " +"انتخاب کند. عامل آزادی تغییر جهتی را حفظ می کند که پیشرفت های قبلی را ممکن " +"کرده است." + +#: website/public/blog/reverse-engineering-models/index.html:750 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This follows naturally from the industrial shift I described in the previous" +" essay. Being able to run many more experiments changes how we organize the " +"work. It also gives us a reason to ask which decisions need broad judgment " +"and which improve through repeated experience with a particular craft." +msgstr "" +"این به طور طبیعی از تغییر صنعتی که در مقاله قبلی توضیح دادم ناشی می شود. " +"توانایی انجام آزمایش‌های بسیار بیشتر، نحوه سازماندهی کار را تغییر می‌دهد. " +"همچنین دلیلی به ما می دهد تا بپرسیم کدام تصمیمات نیاز به قضاوت گسترده دارند " +"و کدام یک از طریق تجربه مکرر با یک هنر خاص بهبود می یابند." + +#: website/public/blog/reverse-engineering-models/index.html:757 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For now, the strongest general model has earned its place in my workflow by " +"opening problems I had left unresolved. Those successes are what make me " +"want to examine the work more closely: how much depended on the model's " +"breadth, and how much could be handled by a model trained in this particular" +" kind of reconstruction?" +msgstr "" +"تا اینجا قوی‌ترین مدل عمومی جای خود را در گردش کار من پیدا کرده، چون مسائلی " +"را حل کرده که قبلاً حل نشده مانده بودند. همین موفقیت‌ها مرا به بررسی دقیق‌تر" +" فرایند ترغیب می‌کنند: چه مقدار به گستردگی توانایی مدل وابسته بود، و چه " +"مقدار را مدلی آموزش‌دیده برای همین نوع بازسازی می‌تواند انجام دهد؟" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:765 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}The repositories give us a way to test the question.{ph1} I would like " +"to use that record to build the comparison and see what we learn." +msgstr "" +"{ph0}مخزن‌ها راهی برای آزمایش سؤال به ما می‌دهند.{ph1} من می‌خواهم از آن " +"رکورد برای ساخت مقایسه و دیدن آنچه یاد می‌گیریم استفاده کنم." + +#: website/public/blog/reverse-engineering-models/index.html:775 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Evidence and model attribution" +msgstr "مدارک و اسناد مدل" + +#: website/public/blog/reverse-engineering-models/index.html:783 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The examples describe historical function-level comparisons recorded in the " +"linked commits. Whole-game runtime checks are separate milestones. Source " +"and matching records were reviewed on 13 October 2026." +msgstr "" +"مثال‌ها مقایسه‌های سطح عملکرد تاریخی ثبت شده در commit‌های مرتبط را توصیف " +"می‌کنند. بررسی های زمان اجرای کل بازی نقاط عطف جداگانه ای هستند. منبع و " +"سوابق مطابق در 13 اکتبر 2026 بررسی شدند." + +#. {ph0}: gpt-dots +#. {ph1}: gpt-web +#: website/public/blog/reverse-engineering-models/index.html:789 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The TH09 label {ph0} identifies GPT-6-Astra. {ph1} changed with the model " +"used at the time, so that label alone does not identify an individual run. " +"Explicit model names in the other cited commits establish their recorded " +"attribution; the histories do not contain uniform per-run cost or effort " +"measurements." +msgstr "" +"برچسب TH09 {ph0} به معنای GPT-6-Astra است. مدل مرتبط با {ph1} در طول زمان " +"تغییر کرد، به طوری که برچسب به تنهایی نمی تواند مدل مورد استفاده در یک اجرای" +" خاص را شناسایی کند. نام مدل صریح در سایر کامیت‌های ذکرشده، انتساب ثبت شده " +"را ارائه می کند. این تاریخچه ها شامل اندازه گیری ثابت هزینه یا تلاش برای هر " +"اجرا نمی شود." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#: website/public/blog/reverse-engineering-models/index.html:798 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 temporary lifetimes:{ph1} the {ph2}earlier context review{ph3} and" +" {ph4}complete cold reconstruction{ph5}." +msgstr "" +"{ph0}TH09 طول عمر اشیای موقت:{ph1} {ph2}بررسی زمینه اولیه{ph3} و " +"{ph4}بازسازی کامل تأیید شده توسط یک کامپایل مجدد تمیز از منبع{ph5}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:810 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 Front callback:{ph1} {ph2}non-exact diagnostic regions{ph3}, " +"{ph4}Astra's new graph{ph5} and its {ph6}accepted recompile replay{ph7}." +msgstr "" +"{ph0}TH09 Front پاسخ به تماس:{ph1} {ph2}مناطق تشخیصی که مطابقت نداشتند{ph3}،" +" نمودار جریان کنترل جدید {ph4}Astra{ph5} و {ph6}بررسی مجدد پذیرفته شده " +"آن{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:826 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Compiler knowledge:{ph1} {ph2}TH095 identifier-bucket reuse{ph3}, " +"{ph4}TH04 player source and linker order{ph5} and {ph6}Luna's TH09 enemy-" +"spawn closure{ph7}." +msgstr "" +"{ph0}دانش کامپایلر:{ph1} {ph2}استفاده دوباره از رفتار دسته‌بندی شناسه‌ها در " +"TH095{ph3}، {ph4}کد منبع بازیکن TH04 و ترتیب لینک{ph5} و {ph6}بازسازی کامل " +"تابع ایجاد دشمن TH09 توسط Luna{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:842 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Checks and diagnostics:{ph1} {ph2}TH08 floating-literal audit{ph3}, " +"{ph4}TH08 numerical investigation{ph5} and {ph6}TH09 replay-save " +"reconstruction{ph7}." +msgstr "" +"{ph0}بررسی و تشخیص:{ph1} {ph2}TH08 ممیزی ثابت ممیز شناور{ph3}، {ph4}TH08 " +"بررسی عددی{ph5} و {ph6}TH09 بازسازی صرفه جویی در پخش{ph7}." + +#: website/public/blog/reverse-engineering-models/index.html:859 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The proposed specialist and RL comparisons are future experiments. The " +"project histories motivate them; no trained specialist was evaluated for " +"this essay." +msgstr "" +"مقایسه‌های پیشنهادی مدل‌های تخصصی و آموزش RL آزمایش‌های آینده هستند. تاریخچه" +" پروژه به آنها انگیزه می دهد. هیچ مدل تخصصی آموزش دیده ای برای این مقاله " +"ارزیابی نشد." + +#: website/public/blog/reverse-engineering-models/index.html:6 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model? | REA" +msgstr "آیا مهندسی معکوس به مدل بزرگتری نیاز دارد؟ | REA" + +#: website/public/blog/reverse-engineering-models/index.html:7 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What Touhou reconstruction taught me about general models, specialist models" +" and learning to propose better hypotheses from compiler feedback." +msgstr "" +"آنچه که بازسازی Touhou در مورد مدل های عمومی، مدل های تخصصی و یادگیری ارائه " +"فرضیه های بهتر از بازخورد کامپایلر به من آموخت." + +#: website/public/blog/reverse-engineering-models/index.html:240 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "On this page" +msgstr "در این صفحه" + +#: website/public/blog/reverse-engineering-models/index.html:253 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Models and reverse engineering essay" +msgstr "مدل ها و مقاله مهندسی معکوس" + +#: website/public/blog/reverse-engineering-models/index.html:491 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Selected reconstruction experiments" +msgstr "آزمایش های بازسازی منتخب" + +#: website/public/blog/reverse-engineering-models/index.html:883 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Back to top" +msgstr "بازگشت به بالا" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:776 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Detailed evidence notes{ph1} include the recorded comparisons and " +"experiment context." +msgstr "" +"{ph0}یادداشت‌های تفصیلی شواهد{ph1} شامل مقایسه‌های ثبت‌شده و زمینه آزمایش‌ها" +" هستند." + +#~ msgctxt "page:blog/index.html" +#~ msgid "By N0zoM1z0 · 13 October 2026 · Draft for review" +#~ msgstr "نویسنده: N0zoM1z0 · ۱۳ اکتبر ۲۰۲۶ · پیش‌نویس برای بازبینی" + #~ msgctxt "page:index.html" #~ msgid "Setup guide" #~ msgstr "راهنمای نصب" diff --git a/website/locales/fr/messages.po b/website/locales/fr/messages.po index 899844e40..1664f4bd5 100644 --- a/website/locales/fr/messages.po +++ b/website/locales/fr/messages.po @@ -32,18 +32,18 @@ msgid "Thoughts on reverse engineering." msgstr "Réflexions sur l'ingénierie inverse." #. {ph0}:
-#: website/public/blog/index.html:200 +#: website/public/blog/index.html:214 #, python-brace-format msgctxt "page:blog/index.html" msgid "Game reconstruction{ph0}Essay" msgstr "Reconstruction du jeu{ph0}Essai" -#: website/public/blog/index.html:202 +#: website/public/blog/index.html:216 msgctxt "page:blog/index.html" msgid "The industrial revolution in reverse engineering" msgstr "La révolution industrielle dans l'ingénierie inverse" -#: website/public/blog/index.html:203 +#: website/public/blog/index.html:217 msgctxt "page:blog/index.html" msgid "" "What manual RE and reconstructing TH08, TH095 and TH04 taught me about " @@ -53,25 +53,26 @@ msgstr "" "TH04 m'ont appris sur les agents, la vérification et les connaissances qui " "se combinent dans les projets." -#: website/public/blog/index.html:208 +#: website/public/blog/index.html:207 website/public/blog/index.html:222 msgctxt "page:blog/index.html" msgid "Read the article" msgstr "Lire l'article" -#: website/public/blog/index.html:211 +#: website/public/blog/index.html:225 msgctxt "page:blog/index.html" msgid "By N0zoM1z0 · 10 October 2026" msgstr "Par N0zoM1z0 · 10 octobre 2026" #. {ph0}: -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 +#: website/public/blog/reverse-engineering-models/index.html:883 #: website/public/blog/touhou-reconstruction/index.html:1080 #: website/public/examples/dino-lab/index.html:475 #: website/public/faq/index.html:573 #: website/public/first-investigation/index.html:523 #: website/public/get-started/index.html:882 -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 #: website/public/guides/index.html:282 #: website/public/guides/javascript/index.html:503 #: website/public/guides/native/index.html:509 website/public/index.html:1191 @@ -101,6 +102,8 @@ msgstr "" " avec REA et les agents d'IA." #: website/public/blog/index.html:75 website/public/blog/index.html:88 +#: website/public/blog/reverse-engineering-models/index.html:75 +#: website/public/blog/reverse-engineering-models/index.html:88 #: website/public/blog/touhou-reconstruction/index.html:75 #: website/public/blog/touhou-reconstruction/index.html:88 #: website/public/examples/dino-lab/index.html:75 @@ -137,6 +140,7 @@ msgstr "" "agent de codage." #: website/public/blog/index.html:108 +#: website/public/blog/reverse-engineering-models/index.html:108 #: website/public/blog/touhou-reconstruction/index.html:108 #: website/public/examples/dino-lab/index.html:109 #: website/public/faq/index.html:108 @@ -157,6 +161,7 @@ msgid "REA home" msgstr "Accueil REA" #: website/public/blog/index.html:112 +#: website/public/blog/reverse-engineering-models/index.html:112 #: website/public/blog/touhou-reconstruction/index.html:112 #: website/public/examples/dino-lab/index.html:113 #: website/public/faq/index.html:112 @@ -177,6 +182,7 @@ msgid "Main navigation" msgstr "Navigation principale" #: website/public/blog/index.html:130 +#: website/public/blog/reverse-engineering-models/index.html:130 #: website/public/blog/touhou-reconstruction/index.html:130 #: website/public/examples/dino-lab/index.html:131 #: website/public/faq/index.html:130 @@ -197,6 +203,7 @@ msgid "REA on GitHub" msgstr "REA sur GitHub" #: website/public/blog/index.html:150 +#: website/public/blog/reverse-engineering-models/index.html:150 #: website/public/blog/touhou-reconstruction/index.html:150 #: website/public/examples/dino-lab/index.html:151 #: website/public/faq/index.html:150 @@ -217,6 +224,7 @@ msgid "Join REA on Discord" msgstr "Rejoindre REA sur Discord" #: website/public/blog/index.html:169 +#: website/public/blog/reverse-engineering-models/index.html:169 #: website/public/blog/touhou-reconstruction/index.html:169 #: website/public/examples/dino-lab/index.html:170 #: website/public/faq/index.html:169 @@ -236,7 +244,7 @@ msgctxt "shared" msgid "Follow REA on X" msgstr "Suivre REA sur X" -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 msgctxt "page:blog/index.html" msgid "Back to top" msgstr "Haut de page" @@ -4834,7 +4842,7 @@ msgctxt "page:guides/browser/index.html" msgid "Copy read the loaded scripts" msgstr "Copier lire les scripts chargés" -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 msgctxt "page:guides/browser/index.html" msgid "Back to top" msgstr "Haut de page" @@ -11775,6 +11783,7 @@ msgid "Same spacing: 256 ÷ 16 = 16 angle units = 22.5°" msgstr "Même espacement: 256 ÷ 16 = 16 unités d'angle = 22,5°" #: website/public/blog/index.html:114 +#: website/public/blog/reverse-engineering-models/index.html:114 #: website/public/blog/touhou-reconstruction/index.html:114 #: website/public/examples/dino-lab/index.html:115 #: website/public/faq/index.html:114 @@ -11795,6 +11804,7 @@ msgid "Showcases" msgstr "Études de cas" #: website/public/blog/index.html:115 +#: website/public/blog/reverse-engineering-models/index.html:115 #: website/public/blog/touhou-reconstruction/index.html:115 #: website/public/examples/dino-lab/index.html:116 #: website/public/faq/index.html:115 @@ -11815,6 +11825,7 @@ msgid "Blog" msgstr "Blog" #: website/public/blog/index.html:113 +#: website/public/blog/reverse-engineering-models/index.html:113 #: website/public/blog/touhou-reconstruction/index.html:113 #: website/public/examples/dino-lab/index.html:114 #: website/public/faq/index.html:113 @@ -11835,6 +11846,7 @@ msgid "Get started" msgstr "Démarrer" #: website/public/blog/index.html:116 +#: website/public/blog/reverse-engineering-models/index.html:116 #: website/public/blog/touhou-reconstruction/index.html:116 #: website/public/examples/dino-lab/index.html:117 #: website/public/faq/index.html:116 @@ -11854,13 +11866,14 @@ msgctxt "shared" msgid "Guides" msgstr "Guides" -#: website/public/blog/index.html:222 +#: website/public/blog/index.html:236 +#: website/public/blog/reverse-engineering-models/index.html:875 #: website/public/blog/touhou-reconstruction/index.html:1072 #: website/public/examples/dino-lab/index.html:467 #: website/public/faq/index.html:565 #: website/public/first-investigation/index.html:515 #: website/public/get-started/index.html:874 -#: website/public/guides/browser/index.html:490 +#: website/public/guides/browser/index.html:499 #: website/public/guides/index.html:274 #: website/public/guides/javascript/index.html:495 #: website/public/guides/native/index.html:501 website/public/index.html:1183 @@ -11874,13 +11887,14 @@ msgctxt "shared" msgid "Source" msgstr "Code source" -#: website/public/blog/index.html:223 +#: website/public/blog/index.html:237 +#: website/public/blog/reverse-engineering-models/index.html:876 #: website/public/blog/touhou-reconstruction/index.html:1073 #: website/public/examples/dino-lab/index.html:468 #: website/public/faq/index.html:566 #: website/public/first-investigation/index.html:516 #: website/public/get-started/index.html:875 -#: website/public/guides/browser/index.html:491 +#: website/public/guides/browser/index.html:500 #: website/public/guides/index.html:275 #: website/public/guides/javascript/index.html:496 #: website/public/guides/native/index.html:502 website/public/index.html:1184 @@ -11894,13 +11908,14 @@ msgctxt "shared" msgid "MIT license" msgstr "Licence MIT" -#: website/public/blog/index.html:224 +#: website/public/blog/index.html:238 +#: website/public/blog/reverse-engineering-models/index.html:877 #: website/public/blog/touhou-reconstruction/index.html:1074 #: website/public/examples/dino-lab/index.html:469 #: website/public/faq/index.html:567 #: website/public/first-investigation/index.html:517 #: website/public/get-started/index.html:876 -#: website/public/guides/browser/index.html:492 +#: website/public/guides/browser/index.html:501 #: website/public/guides/index.html:276 #: website/public/guides/javascript/index.html:497 #: website/public/guides/native/index.html:503 website/public/index.html:1185 @@ -11914,13 +11929,14 @@ msgctxt "shared" msgid "Join the community" msgstr "Rejoindre la communauté" -#: website/public/blog/index.html:225 +#: website/public/blog/index.html:239 +#: website/public/blog/reverse-engineering-models/index.html:878 #: website/public/blog/touhou-reconstruction/index.html:1075 #: website/public/examples/dino-lab/index.html:470 #: website/public/faq/index.html:568 #: website/public/first-investigation/index.html:518 #: website/public/get-started/index.html:877 -#: website/public/guides/browser/index.html:493 +#: website/public/guides/browser/index.html:502 #: website/public/guides/index.html:277 #: website/public/guides/javascript/index.html:498 #: website/public/guides/native/index.html:504 website/public/index.html:1186 @@ -12867,6 +12883,1435 @@ msgctxt "runtime" msgid "Switch to dark mode" msgstr "Passer au mode sombre" +#. {ph0}: +#. {ph1}: +#: website/public/guides/browser/index.html:480 +#, python-brace-format +msgctxt "page:guides/browser/index.html" +msgid "" +"For HTTP request and response bodies, use {ph0}Fiddler Everywhere: SAZ " +"captures, MCP and public skills{ph1}. Export SAZ or HAR 1.2 for offline REA " +"capture Evidence, or connect Everywhere's MCP server to inspect captured " +"traffic with your agent." +msgstr "" +"Pour examiner les corps des requêtes et réponses HTTP, utilisez {ph0}Fiddler" +" Everywhere : captures SAZ, MCP et compétences publiques{ph1}. Exportez en " +"SAZ ou HAR 1.2 pour obtenir des Evidence de capture dans REA hors ligne, ou " +"connectez le serveur MCP d’Everywhere pour inspecter le trafic capturé avec " +"votre agent." + +#. {ph0}:
+#: website/public/blog/index.html:200 +#, python-brace-format +msgctxt "page:blog/index.html" +msgid "Model specialization{ph0}Essay" +msgstr "Modèles spécialisés{ph0}Essai" + +#: website/public/blog/index.html:202 +msgctxt "page:blog/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "Le reverse engineering a-t-il besoin d’un modèle plus grand ?" + +#: website/public/blog/index.html:203 +msgctxt "page:blog/index.html" +msgid "" +"What Touhou reconstruction suggests about general models, specialist models " +"and learning from compiler feedback." +msgstr "" +"Ce que la reconstruction de Touhou suggère sur les modèles généralistes, les" +" modèles spécialisés et l’apprentissage à partir des retours du compilateur." + +#: website/public/blog/index.html:210 +msgctxt "page:blog/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Par N0zoM1z0 · 13 octobre 2026" + +#: website/public/blog/reverse-engineering-models/index.html:103 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Skip to content" +msgstr "Aller au contenu" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:194 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Home{ph1} / {ph2}Blog{ph3} / Models and RE" +msgstr "{ph0}Accueil{ph1} / {ph2}Blog{ph3} / Modèles et rétro-ingénierie" + +#: website/public/blog/reverse-engineering-models/index.html:197 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Essay" +msgstr "Essai" + +# Reviewed for technical meaning and readability; preserve the distinction +# between historical evidence and proposed experiments. +#: website/public/blog/reverse-engineering-models/index.html:198 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "L'ingénierie inverse nécessite-t-elle un modèle plus grand ?" + +#: website/public/blog/reverse-engineering-models/index.html:199 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Par N0zoM1z0 · 13 octobre 2026" + +#: website/public/blog/reverse-engineering-models/index.html:201 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Takeaway" +msgstr "À retenir" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:203 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Model capability changes the hypotheses an agent can find.{ph1} Later " +"sessions have solved problems that earlier work left unresolved." +msgstr "" +"{ph0}La capacité du modèle affecte les hypothèses qu'un agent d'IA peut " +"proposer.{ph1} Les sessions ultérieures ont résolu des problèmes que les " +"travaux antérieurs n'avaient pas résolus." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:211 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A familiar compiler gives specialization something concrete to " +"learn.{ph1} Some exact matches depend on narrow, reusable source patterns." +msgstr "" +"{ph0}Un compilateur familier propose des modèles concrets à apprendre pour " +"un modèle spécialisé.{ph1} Certaines correspondances exactes de code machine" +" dépendent de modèles de code source spécifiques et réutilisables." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:218 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Give repository knowledge a fair comparison.{ph1} A smaller model with " +"good retrieval may already know enough to make progress." +msgstr "" +"{ph0}Incluez les connaissances du référentiel dans une comparaison " +"équitable.{ph1} Un modèle plus petit avec une bonne récupération des " +"connaissances peut déjà contenir suffisamment d'informations pour " +"progresser." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:223 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Train the next useful experiment, and test its oracle.{ph1} The value " +"of feedback depends on what the checker actually verifies." +msgstr "" +"{ph0}Entraînez le modèle à choisir la prochaine expérience utile et testez " +"son vérificateur.{ph1} Les retours ne sont utiles que dans la mesure où le " +"vérificateur vérifie les bonnes choses." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:230 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Measure accepted reconstruction against time and cost.{ph1} That is the" +" result I want from the whole agent." +msgstr "" +"{ph0}Évaluez les résultats de reconstruction vérifiés en fonction du temps " +"et du coût.{ph1} C'est ce que je souhaite mesurer pour le flux de travail " +"complet de l'agent IA." + +#: website/public/blog/reverse-engineering-models/index.html:241 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "In this article" +msgstr "Dans cet article" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:243 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What better models changed{ph1}" +msgstr "{ph0}Ce que les modèles plus performants ont changé{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:244 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Why specialize?{ph1}" +msgstr "{ph0}Pourquoi se spécialiser ?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:245 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Where experience lives{ph1}" +msgstr "{ph0}Où cette expérience doit-elle être stockée ?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:246 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Learning the next experiment{ph1}" +msgstr "{ph0}Apprendre la prochaine expérience utile{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:247 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}RE and reinforcement learning{ph1}" +msgstr "{ph0}Ingénierie inverse et apprentissage par renforcement{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:248 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}When the oracle is wrong{ph1}" +msgstr "{ph0}Lorsque le vérificateur est erroné{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:249 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}A comparison worth running{ph1}" +msgstr "{ph0}Une comparaison qui vaut la peine d'être menée{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:250 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What kind of agent?{ph1}" +msgstr "{ph0}Quel type d’agent IA cela suggère-t-il ?{ph1}" + +#: website/public/blog/reverse-engineering-models/index.html:258 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What better models changed" +msgstr "Ce que les modèles plus performants ont changé" + +#: website/public/blog/reverse-engineering-models/index.html:259 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Since August 2026, I have been exploring how much reverse engineering an " +"agent can carry out with access to a program and a way to check its work. " +"Most of my experience comes from reconstructing games, especially Touhou. As" +" I have changed models, the repositories have kept a record of what each " +"session managed to recover." +msgstr "" +"Depuis août 2026, j'explore combien de rétro-ingénierie un agent peut " +"effectuer avec accès à un programme et un moyen de vérifier son travail. La " +"majeure partie de mon expérience vient de la reconstruction de jeux, en " +"particulier Touhou. Au fur et à mesure que j'ai changé de modèle, les " +"référentiels ont conservé une trace de ce que chaque session a réussi à " +"récupérer." + +#: website/public/blog/reverse-engineering-models/index.html:267 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of that work aims at exact reconstruction: we compile the maintained " +"source under a pinned historical toolchain and check whether it reproduces " +"the target function. When a later model succeeds where earlier work stalled," +" the source changes show what made that progress possible." +msgstr "" +"Une grande partie du travail vise une reconstruction exacte : nous compilons" +" le code maintenu avec une chaîne d'outils de l'époque dont les versions " +"sont fixées, puis vérifions si elle reproduit la fonction cible. Quand un " +"modèle ultérieur débloque le travail, les changements du code permettent de " +"comprendre ce qui a rendu ce progrès possible." + +#: website/public/blog/reverse-engineering-models/index.html:274 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The temporary values had to stay alive together" +msgstr "" +"Les variables temporaires nécessitaient des durées de vie qui se " +"chevauchaient" + +#: website/public/blog/reverse-engineering-models/index.html:275 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of a TH09 attack function's logic had already been reconstructed, but " +"the compiler still stored temporary results differently from the original. " +"The remaining problem was how long those results needed to stay alive and " +"when their storage could be reused." +msgstr "" +"Une grande partie de la logique d'une fonction d'attaque TH09 avait déjà été" +" reconstruite, mais le compilateur stockait toujours les résultats " +"temporaires différemment de l'original. Le problème restant était de savoir " +"combien de temps ces résultats devaient rester en vie et quand leur stockage" +" pouvait être réutilisé." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:282 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A {ph0}GPT-6-Sol checkpoint{ph1} recorded this mismatch after testing " +"several ordinary source arrangements. When {ph2}GPT-6.1-Sol returned to the " +"problem{ph3}, it found a useful hypothesis about how long those temporary " +"values needed to stay alive." +msgstr "" +"Un {ph0}GPT-6-Sol enregistrement de session{ph1} a signalé cette incohérence" +" après avoir testé plusieurs arrangements de code source ordinaires. Lorsque" +" {ph2}GPT-6.1-Sol a réexaminé le problème{ph3}, il a proposé une hypothèse " +"utile sur la durée de vie de ces variables temporaires." + +#: website/public/blog/reverse-engineering-models/index.html:296 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The function produced temporary results and copied them into animation " +"positions. In C++, a temporary result stays alive until the expression using" +" it finishes. Grouping the real operations into one expression kept those " +"results alive together, so the compiler could no longer reuse the same " +"storage for all of them." +msgstr "" +"La fonction produisait des résultats temporaires et les copiait dans des " +"positions d'animation. En C++, un résultat temporaire reste actif jusqu'à la" +" fin de l'expression qui l'utilise. Le regroupement des opérations réelles " +"en une seule expression permettait de conserver ces résultats ensemble, de " +"sorte que le compilateur ne pouvait plus réutiliser le même stockage pour " +"chacun d'entre eux." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:303 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"One result still shared storage with a value used later for collision " +"checks. Giving that value its own lifetime resolved the remaining overlap. " +"After several targeted experiments, {ph0}a clean rebuild confirmed that the " +"full function matched{ph1}." +msgstr "" +"Un résultat partageait encore son espace de stockage avec une valeur " +"utilisée plus tard pour les collisions. Donner à cette valeur une durée de " +"vie distincte a éliminé le dernier chevauchement. Après plusieurs " +"expériences ciblées, {ph0}une recompilation propre a confirmé que la " +"fonction entière correspondait à l'original{ph1}." + +#: website/public/blog/reverse-engineering-models/index.html:312 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What interests me is how the discovery and the tests worked together. The " +"lifetime hypothesis gave the agent a new direction; compiler feedback helped" +" it refine that hypothesis into a source arrangement that matched." +msgstr "" +"Je suis intéressé par la façon dont la découverte et les tests se sont " +"soutenus mutuellement. L’hypothèse sur la durée de vie des objets " +"temporaires a donné une nouvelle direction à l’agent IA. Les retours du " +"compilateur l'ont aidé à affiner cette hypothèse en un arrangement de code " +"source correspondant." + +#: website/public/blog/reverse-engineering-models/index.html:318 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A different control-flow graph" +msgstr "Un graphique de flux de contrôle différent" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:319 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A second TH09 function shows why that new direction can matter. Here, the " +"obstacle was how several branches shared their final work. " +"{ph0}GPT-6.1-Sol's earlier work{ph1} got parts of the function to match, but" +" the branches still joined up differently from the original." +msgstr "" +"Une deuxième fonction TH09 montre pourquoi cette nouvelle direction peut " +"être importante. Ici, l'obstacle était la façon dont plusieurs branches " +"partageaient leur travail final. Les travaux antérieurs de " +"{ph0}GPT-6.1-Sol{ph1} ont permis de faire correspondre certaines parties de " +"la fonction, mais les branches se sont toujours jointes différemment de " +"l'original." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:330 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}GPT-6-Astra changed the surrounding control-flow graph{ph1}. The main " +"paths now led to a shared final update and return. In that source structure," +" the compiler combined the branches in the same way as the original. The " +"complete function matched, and a {ph2}separate rebuild check confirmed " +"it{ph3}." +msgstr "" +"{ph0}GPT-6-Astra a modifié le graphique de flux de contrôle " +"environnant{ph1}. Les chemins principaux menaient désormais à une mise à " +"jour finale et à un retour partagés. Dans cette structure source, le " +"compilateur combinait les branches de la même manière que l'original. La " +"fonction complète correspondait et une {ph2}vérification de reconstruction " +"distincte l'a confirmée{ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:342 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Changing the representation opened a more productive search space.{ph1}" +" Both cases give me a reason to take general model capability seriously: an " +"agent can keep trying local variations for a long time before finding a " +"better way to express the surrounding work." +msgstr "" +"{ph0}Changer la représentation a ouvert un espace de recherche plus " +"productif.{ph1} Les deux cas me donnent une raison de prendre au sérieux la " +"capacité générale du modèle : un agent peut continuer à essayer des " +"variantes locales pendant longtemps avant de trouver une meilleure façon " +"d'exprimer le travail environnant." + +#: website/public/blog/reverse-engineering-models/index.html:351 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"At the same time, the later sessions started with earlier source and " +"recorded failures. That accumulated work was part of their advantage. The " +"history lets us inspect the discoveries and their checks; a controlled " +"comparison would help separate the effect of the model from the experience " +"it inherited." +msgstr "" +"Les sessions ultérieures ont aussi hérité du code précédent et des échecs " +"documentés. Ce travail accumulé faisait partie de leur avantage. " +"L'historique permet d'examiner les découvertes et leurs vérifications ; une " +"comparaison contrôlée aiderait à séparer l'effet du modèle de celui de " +"l'expérience héritée." + +#: website/public/blog/reverse-engineering-models/index.html:358 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is where the question becomes interesting to me. Some of the progress " +"came from finding a new representation, while much of the remaining work " +"involved testing source forms against a familiar compiler. How much needs " +"broad reasoning ability, and how much could a model learn through repeated " +"experience with that compiler and code?" +msgstr "" +"C'est ce qui rend la question intéressante pour moi. Certains progrès sont " +"venus de la recherche d'une nouvelle représentation. Une grande partie du " +"travail restant consistait à tester différentes dispositions de code source " +"par rapport à un compilateur familier. Dans quelle mesure cela nécessite-t-" +"il une grande capacité de raisonnement ? Combien un modèle pourrait-il " +"apprendre en travaillant de manière répétée avec ce compilateur et ce code ?" + +#: website/public/blog/reverse-engineering-models/index.html:368 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why might a specialist do well?" +msgstr "" +"Pourquoi un modèle d’IA spécialisé pourrait-il fonctionner correctement ?" + +#: website/public/blog/reverse-engineering-models/index.html:369 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Touhou gives us a concrete reason to expect that experience to help. Much of" +" the game code comes from the same author, ZUN, so recurring habits are a " +"reasonable starting assumption. The repositories back that up with useful " +"examples: related loops and helper arrangements have supplied hypotheses " +"across games." +msgstr "" +"Touhou nous donne une raison concrète d'attendre que cette expérience nous " +"aide. Une grande partie du code du jeu provient du même auteur, ZUN, les " +"habitudes récurrentes constituent donc une hypothèse de départ raisonnable. " +"Les référentiels confirment cela avec des exemples utiles : les boucles " +"associées et les arrangements d'assistance ont fourni des hypothèses à " +"travers les jeux." + +#: website/public/blog/reverse-engineering-models/index.html:376 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those source patterns become more specific when we include the compiler. " +"Under a particular version and set of options, small changes in source can " +"produce repeatable differences in machine code. Some of those differences " +"are surprisingly obscure." +msgstr "" +"Ces modèles de source deviennent plus spécifiques lorsque nous incluons le " +"compilateur. Sous une version et un ensemble d'options particuliers, de " +"petites modifications dans la source peuvent produire des différences " +"reproductibles dans le code machine. Certaines de ces différences sont " +"étonnamment obscures." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: Supervisor::LoadDat +#. {ph4}: +#: website/public/blog/reverse-engineering-models/index.html:382 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In TH095, for example, {ph0}even the name used for a local variable could " +"change where VC7.1 stored it{ph1}. Moving the declaration alone did not " +"solve the layout, but an earlier background function had revealed a naming " +"pattern that affected the compiler's choice. That gave us a more promising " +"hypothesis for {ph2}the reconstruction of {ph3}{ph4}." +msgstr "" +"Dans TH095, par exemple, {ph0}même le nom d'une variable locale peut " +"affecter l'endroit où VC7.1 la stocke{ph1}. Le déplacement de sa déclaration" +" n'a pas résolu la configuration du stockage. Cependant, une fonction " +"d'arrière-plan antérieure avait révélé un modèle de dénomination qui " +"affectait le choix du compilateur. Cela suggère une hypothèse plus " +"prometteuse pour {ph2}reconstruire {ph3}{ph4}." + +#: website/public/blog/reverse-engineering-models/index.html:396 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reusing that pattern for the file-size variable produced the required layout" +" without adding storage. The full function then matched." +msgstr "" +"La réutilisation de ce modèle pour la variable de taille de fichier a " +"produit la mise en page requise sans ajouter de stockage. La fonction " +"complète correspondait alors." + +#: website/public/blog/reverse-engineering-models/index.html:401 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is the sort of narrow detail a general model may not know when it " +"arrives. A specialist trained on this compiler could encounter many such " +"cases and learn which source variations are worth testing first." +msgstr "" +"Il s’agit d’un détail très spécifique qu’un modèle d’IA à usage général peut" +" ne pas connaître lorsqu’il rencontre le problème pour la première fois. Un " +"modèle spécialisé pour ce compilateur pourrait détecter de nombreux cas de " +"ce type et découvrir quelles variantes du code source méritent d'être " +"testées en premier." + +#: website/public/blog/reverse-engineering-models/index.html:407 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The opportunity is especially clear in final matching problems where we " +"already understand the behavior. We are then asking which ordinary source " +"form this compiler turns into the observed instructions. Specialization " +"could help the model propose those forms more efficiently, while the target " +"still determines which hypotheses survive." +msgstr "" +"L’opportunité est particulièrement évidente dans les problèmes d’appariement" +" final dont nous comprenons déjà le comportement. Nous nous demandons alors " +"quel arrangement de code source ordinaire ce compilateur transforme en " +"instructions observées. La spécialisation pourrait aider le modèle à " +"proposer ces formes plus efficacement, tandis que la cible détermine " +"toujours quelles hypothèses survivent." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:415 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The useful patterns would have to retain their context, though. TH08 uses an" +" earlier Visual C++ version than TH095 and TH09. TH04 uses a different DOS-" +"era compiler, and its {ph0}player reconstruction{ph1} needed different " +"hypotheses about source patterns and how the compiled code was combined into" +" an executable. Shared authorship gives us a prior, but each target tells us" +" whether it applies. Distinguishing those contexts is part of what a " +"specialist would need to learn." +msgstr "" +"Les modèles utiles doivent conserver leur contexte. TH08 utilise une version" +" Visual C++ antérieure à TH095 et TH09. TH04 utilise un compilateur " +"différent de l'ère DOS. Sa {ph0}reconstruction du code du joueur{ph1} " +"nécessitait différentes hypothèses sur les modèles de code source et sur la " +"manière dont le code compilé était lié à un exécutable. Avoir le même auteur" +" nous donne une attente initiale, mais chaque cible nous dit si elle " +"s'applique. Un modèle spécialisé devrait distinguer ces contextes." + +#: website/public/blog/reverse-engineering-models/index.html:431 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Where should that experience live?" +msgstr "Où cette expérience doit-elle être stockée ?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:432 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before deciding to teach those patterns through model training, I would look" +" at the knowledge we already keep in the repository. In the {ph0}previous " +"essay{ph1}, I wrote about it becoming the project's working memory. That " +"memory includes compiler lessons, along with the source and failed " +"experiments that made them useful." +msgstr "" +"Avant de décider d'enseigner ces modèles via la formation de modèles, " +"j'examinerais les connaissances que nous conservons déjà dans le " +"référentiel. Dans l'{ph0}essai précédent{ph1}, j'ai écrit qu'il deviendrait " +"la mémoire de travail du projet. Cette mémoire comprend les leçons du " +"compilateur, ainsi que la source et les expériences ratées qui les ont " +"rendues utiles." + +#: website/public/blog/reverse-engineering-models/index.html:441 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"By retrieving that record, an agent can start with a known result and " +"inspect how we reached it. It can also avoid alternatives that have already " +"failed in the same context. The starting point improves without changing the" +" model's weights." +msgstr "" +"En récupérant cet enregistrement, un agent peut partir d’un résultat connu " +"et inspecter comment nous l’avons atteint. Cela peut également éviter des " +"alternatives qui ont déjà échoué dans le même contexte. Le point de départ " +"s'améliore sans modifier les paramètres entraînés du modèle." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:447 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"When reconstructing TH09's enemy-spawn function, a {ph0}GPT-6-Luna-max " +"session{ph1} drew a hypothesis from TH08's corresponding source family. That" +" earlier source suggested how to structure the loop and handle failures " +"within it. TH09's own evidence supplied the game-specific details, and clean" +" rebuilds confirmed the complete match." +msgstr "" +"Lors de la reconstruction de la fonction de TH09 pour créer des ennemis, une" +" {ph0}session GPT-6-Luna-max{ph1} a pris une hypothèse à partir du code " +"source TH08 associé. Ce code précédent suggérait comment organiser la boucle" +" et gérer les échecs à l'intérieur. Le propre témoignage de TH09 a fourni " +"des détails spécifiques au jeu. Les reconstructions à partir des sources ont" +" confirmé que la fonction complète correspondait." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:458 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}\"The model did not know that compiler quirk\" and \"the model could " +"not use it\" are different problems.{ph1} Retrieval may solve the first, but" +" turning that fact into a useful candidate still takes work. And if the " +"candidate fails, the model must decide what the result means for its next " +"attempt. Knowing the relevant pattern is only the beginning of that process." +msgstr "" +"{ph0}\"Le modèle ne connaissait pas cette bizarrerie du compilateur\" et " +"\"le modèle n'a pas pu l'utiliser\" sont des problèmes différents.{ph1} La " +"récupération peut résoudre le premier, mais transformer ce fait en un " +"candidat utile demande encore du travail. Et si le code source candidat " +"échoue, le modèle doit décider de la signification du résultat pour sa " +"prochaine tentative. Connaître le modèle pertinent n’est que le début de ce " +"processus." + +#: website/public/blog/reverse-engineering-models/index.html:468 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For that reason, I would include a smaller general model with good retrieval" +" in any comparison. Giving it the same library of experience would help us " +"identify what training adds: better candidate generation, better decisions " +"after feedback, or both." +msgstr "" +"Pour cette raison, j’inclurais un modèle d’IA à usage général plus petit " +"avec une bonne récupération dans toute comparaison. Lui donner la même " +"bibliothèque d'expériences nous aiderait à identifier ce que la formation " +"ajoute : une meilleure génération de candidats, de meilleures décisions " +"après feedback, ou les deux." + +#: website/public/blog/reverse-engineering-models/index.html:476 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Learning the next useful experiment" +msgstr "Apprendre la prochaine expérience utile" + +#: website/public/blog/reverse-engineering-models/index.html:477 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those decisions after feedback are what I would most like to study. In " +"agentic reconstruction, the model proposes an explanation and tests it " +"against the program. Even when the candidate fails, the result can narrow " +"the remaining possibilities and suggest a more useful next experiment." +msgstr "" +"Ces décisions après retour d’information sont ce que j’aimerais le plus " +"étudier. En reconstruction agentique, le modèle propose une explication et " +"la teste par rapport au programme. Même lorsque le code source candidat " +"échoue, le résultat peut réduire les possibilités restantes et suggérer une " +"prochaine expérience plus utile." + +#: website/public/blog/reverse-engineering-models/index.html:484 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Consider the temporary-results example again. Its final source shows the " +"arrangement that worked, but the route to that source contains more " +"information. Each of the following experiments helped identify what still " +"needed to change." +msgstr "" +"Prenons à nouveau l'exemple des résultats temporaires. Sa source finale " +"montre l'arrangement qui a fonctionné, mais l'itinéraire vers cette source " +"contient plus d'informations. Chacune des expériences suivantes a permis " +"d’identifier ce qui devait encore changer." + +#: website/public/blog/reverse-engineering-models/index.html:497 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Experiment" +msgstr "Expérimenter" + +#: website/public/blog/reverse-engineering-models/index.html:498 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What its result taught us" +msgstr "Ce que son résultat nous a appris" + +#: website/public/blog/reverse-engineering-models/index.html:503 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Try a different way to copy the results" +msgstr "Essayez une autre façon de copier les résultats" + +#: website/public/blog/reverse-engineering-models/index.html:504 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The copy code changes, but the function still differs." +msgstr "Le code de copie change, mais la fonction diffère toujours." + +#: website/public/blog/reverse-engineering-models/index.html:509 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Use the returned values directly" +msgstr "Utiliser directement les valeurs renvoyées" + +#: website/public/blog/reverse-engineering-models/index.html:510 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Copying follows the original pattern, but temporary storage still differs." +msgstr "" +"La copie suit le modèle d'origine, mais le stockage temporaire diffère " +"toujours." + +#: website/public/blog/reverse-engineering-models/index.html:516 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the temporary results alive together" +msgstr "Gardez toutes les variables temporaires valides en même temps" + +#: website/public/blog/reverse-engineering-models/index.html:517 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"They get separate storage, but one still overlaps a value used later for " +"collision checks." +msgstr "" +"Ils bénéficient d'un stockage séparé, mais l'un d'entre eux chevauche " +"toujours une valeur utilisée ultérieurement pour les contrôles de collision." + +#: website/public/blog/reverse-engineering-models/index.html:523 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the collision-check value separate" +msgstr "Gardez la valeur de contrôle de collision séparée" + +#: website/public/blog/reverse-engineering-models/index.html:524 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A clean rebuild confirms the complete match." +msgstr "" +"Une recompilation propre à partir de la source confirme la correspondance " +"complète." + +#: website/public/blog/reverse-engineering-models/index.html:529 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Selected experiments from the attack-function reconstruction above, grouped " +"by what they taught us." +msgstr "" +"Expériences sélectionnées de la reconstruction de la fonction d'attaque ci-" +"dessus, regroupées selon ce qu'elles nous ont appris." + +#: website/public/blog/reverse-engineering-models/index.html:533 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A training record that keeps this sequence can teach more than the final " +"source alone. It connects each proposed change to an observed result and the" +" next decision. Keeping ineffective variations matters too: they tell a " +"later agent which hypotheses have already been tested in that context." +msgstr "" +"Un dossier de formation qui conserve cette séquence peut enseigner plus que " +"la seule source finale. Il relie chaque changement proposé à un résultat " +"observé et à la décision suivante. Conserver des variantes inefficaces est " +"également important : elles indiquent à un agent ultérieur quelles " +"hypothèses ont déjà été testées dans ce contexte." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:540 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}I would want the model to learn how to use feedback.{ph1} A neutral " +"result is a reason to reconsider the variation; a changed mismatch may point" +" to another cause. When local edits stop being informative, the agent should" +" be able to revisit the source representation or inspect the evidence again." +msgstr "" +"{ph0}Je voudrais que le modèle apprenne à utiliser les retours.{ph1} Un " +"résultat neutre est une raison pour reconsidérer la variation ; une " +"inadéquation modifiée peut indiquer une autre cause. Lorsque les " +"modifications locales cessent d’être informatives, l’agent doit être en " +"mesure de revoir la représentation source ou d’inspecter à nouveau les " +"preuves." + +#: website/public/blog/reverse-engineering-models/index.html:549 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Learning to make those choices requires autonomy. Sometimes the useful next " +"step is inspecting another caller rather than compiling another edit. A " +"rigid sequence of required rewrites would remove that choice, along with " +"part of the behavior we want the model to learn." +msgstr "" +"Apprendre à faire ces choix nécessite de l’autonomie. Parfois, l'étape " +"suivante utile consiste à inspecter un autre appelant plutôt que de compiler" +" une autre modification. Une séquence rigide de réécritures requises " +"supprimerait ce choix, ainsi qu'une partie du comportement que nous " +"souhaitons que le modèle apprenne." + +#: website/public/blog/reverse-engineering-models/index.html:558 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why RE is interesting for reinforcement learning" +msgstr "" +"Pourquoi l'ingénierie inverse est intéressante pour l'apprentissage par " +"renforcement" + +#: website/public/blog/reverse-engineering-models/index.html:559 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Bounded reconstruction gives us a practical environment for studying these " +"choices. Once the target and compiler are pinned, many source experiments " +"produce quick, repeatable feedback. The agent can compile a candidate, " +"inspect the mismatch and use that observation to choose its next step." +msgstr "" +"La reconstruction limitée nous offre un environnement pratique pour étudier " +"ces choix. Une fois les versions de la cible et du compilateur fixées, de " +"nombreuses expériences sources produisent des retours rapides et " +"reproductibles. L'agent peut compiler un candidat, inspecter l'inadéquation " +"et utiliser cette observation pour choisir la prochaine étape." + +#: website/public/blog/reverse-engineering-models/index.html:566 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reinforcement learning could train that decision-making process through the " +"feedback it receives. A candidate accepted by a defined check gives us a " +"concrete outcome, while the sequence of attempts lets us observe how much " +"work it took to reach it." +msgstr "" +"L'apprentissage par renforcement pourrait entraîner ce processus de prise de" +" décision grâce aux retours qu'il reçoit. Un candidat accepté par un " +"contrôle défini nous donne un résultat concret, tandis que la séquence des " +"tentatives nous permet d'observer combien de travail il a fallu pour y " +"parvenir." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:572 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"There is already research supporting this direction. {ph0}LLM4Decompile{ph1}" +" studies models specialized for decompilation. {ph2}AutoDecompiler{ph3} " +"trains multi-turn refinement using compiler and execution feedback. Their " +"studies focus on functional recovery. For this work, I would use them as a " +"starting point for the related task of finding source that matches a " +"historical compiler's output." +msgstr "" +"Il existe déjà des recherches soutenant cette direction. " +"{ph0}LLM4Decompile{ph1} étudie les modèles spécialisés pour la " +"décompilation. {ph2}AutoDecompiler{ph3} entraîne le raffinement multi-tours " +"à l'aide des retours du compilateur et de l'exécution. Leurs études portent " +"sur la restauration du comportement. Pour ce travail, je les utiliserais " +"comme point de départ pour la tâche connexe consistant à trouver du code " +"source qui correspond à la sortie d'un compilateur historique." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:584 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To reach such a well-defined task, though, we may first need to establish " +"what a function owns or devise a new runtime measurement. In TH08, for " +"example, a {ph0}collision difference led back to how the original program " +"handled floating-point calculations{ph1}. Before the agent could test a " +"useful correction, it had to work out what to measure." +msgstr "" +"Cependant, pour atteindre une tâche aussi bien définie, nous devrons peut-" +"être d’abord établir quelles ressources possède une fonction ou concevoir " +"une nouvelle manière d’observer le comportement du programme pendant son " +"exécution. Dans TH08, par exemple, une {ph0}différence de collision a " +"conduit à la manière dont le programme d'origine gérait les calculs à " +"virgule flottante{ph1}. Avant que l’agent puisse tester une correction " +"utile, il devait déterminer quoi mesurer." + +#: website/public/blog/reverse-engineering-models/index.html:595 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This suggests another role for a broadly capable model: helping make the " +"problem checkable in the first place. Once that check exists, we can " +"investigate how far a learned search policy can get with it." +msgstr "" +"Cela suggère un autre rôle pour un modèle largement performant : contribuer " +"à rendre le problème vérifiable en premier lieu. Une fois que cette " +"vérification existe, nous pouvons étudier jusqu’où une politique de " +"recherche apprise peut aller avec elle." + +#: website/public/blog/reverse-engineering-models/index.html:603 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "When the oracle is wrong" +msgstr "Lorsque le vérificateur est erroné" + +#: website/public/blog/reverse-engineering-models/index.html:604 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Once we train against that check, its coverage becomes part of what the " +"model learns. I call the reference or checker an oracle. Even if it gives a " +"repeatable answer, we still need to understand what that answer covers." +msgstr "" +"Dès qu'une vérification sert à l'entraînement, sa couverture fait partie de " +"l'environnement d'apprentissage. J'appelle oracle cette référence ou ce " +"vérificateur. Même une réponse reproductible doit être comprise à partir de " +"ce qu'elle vérifie réellement." + +#: website/public/blog/reverse-engineering-models/index.html:610 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"TH08 showed how much can hide outside that coverage. Some reconstructed " +"functions passed their exact comparisons with wrong floating constants. The " +"comparator adjusted relocated addresses before checking instructions, but it" +" checked the values behind those addresses only when they had been " +"explicitly recorded for verification. Other literal references could pass " +"without their values being examined." +msgstr "" +"TH08 a montré tout ce qui peut se cacher en dehors de cette couverture. " +"Certaines fonctions reconstruites ont réussi leurs comparaisons exactes avec" +" de mauvaises constantes flottantes. Le comparateur a ajusté les adresses " +"déplacées avant de vérifier les instructions, mais il a vérifié les valeurs " +"derrière ces adresses uniquement lorsqu'elles avaient été explicitement " +"enregistrées pour vérification. D'autres références littérales pourraient " +"passer sans que leurs valeurs soient examinées." + +#. {ph0}: +#. {ph1}: +#. {ph2}: 1.0f +#. {ph3}: 32.0f +#: website/public/blog/reverse-engineering-models/index.html:619 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To close that gap, a {ph0}wider audit{ph1} found incorrect constants in " +"functions that had already been accepted. In one calculation, the " +"reconstructed source added {ph2} where the original added {ph3}." +msgstr "" +"Pour combler cette lacune, un {ph0}audit plus large{ph1} a détecté des " +"constantes incorrectes dans les fonctions qui avaient déjà réussi la " +"vérification. Dans un calcul, le code source reconstruit a ajouté {ph2}, " +"tandis que l'original a ajouté {ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:629 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A learner rewarded by the old check could learn to produce the wrong " +"reconstruction.{ph1} We had not trained an RL model against this oracle; the" +" observed failure was in the check itself. But using that check as a reward " +"would have made the same blind spot part of the learning environment." +msgstr "" +"{ph0}Un modèle récompensé par l'ancien contrôle de vérification pourrait " +"apprendre à produire une reconstruction incorrecte.{ph1} Nous n'avons pas " +"entraîné de modèle RL à l'aide de ce vérificateur. L’échec que nous avons " +"constaté concernait le contrôle lui-même. Mais si nous l’utilisions comme " +"récompense, le même angle mort deviendrait partie intégrante de " +"l’environnement d’apprentissage." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:639 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A functional test can have a similar blind spot. {ph0}Differential-fuzzing " +"research on LLM decompilers{ph1} has found behavior differences beyond the " +"supplied tests. Passing those tests therefore needs to be read in terms of " +"the cases they exercise, just as an instruction match needs to be read in " +"terms of the bytes and data it checks." +msgstr "" +"Un test fonctionnel peut avoir un angle mort similaire. {ph0}Des recherches " +"utilisant le fuzzing différentiel sur les décompilateurs LLM{ph1} ont révélé" +" des différences de comportement en dehors des tests fournis. Il faut donc " +"comprendre une épreuve de réussite en fonction des cas qu'elle exerce. De " +"même, une correspondance d’instructions doit être comprise en termes " +"d’octets et de données qu’elle vérifie." + +#: website/public/blog/reverse-engineering-models/index.html:649 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before using either kind of check as a reward, I would test it with " +"deliberate failures. I would also keep final acceptance outside the " +"candidate's editable source, so the model has to satisfy the check through " +"its reconstruction. Otherwise, something like a byte array copied from the " +"target could satisfy a naive score without recovering useful source." +msgstr "" +"Avant d’utiliser l’un ou l’autre type de contrôle de vérification comme " +"récompense, je le testerais avec des résultats délibérément incorrects. Je " +"garderais également l'acceptation finale séparée du code source que l'agent " +"IA peut modifier. Le modèle doit réussir la vérification en reconstruisant " +"correctement le programme. Sinon, la simple copie d'un tableau d'octets à " +"partir de la cible pourrait tromper un système de notation naïf sans " +"récupérer le code source utile." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:657 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Improving the learner and improving its oracle belong to the same " +"research problem.{ph1} As the learner gets better at optimizing the feedback" +" we give it, understanding that feedback becomes more important too." +msgstr "" +"{ph0}Améliorer le modèle et améliorer son vérificateur relèvent du même " +"problème de recherche.{ph1} Plus le modèle sait optimiser le retour que nous" +" lui donnons, plus il devient important de comprendre ce retour." + +#: website/public/blog/reverse-engineering-models/index.html:667 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A comparison worth running" +msgstr "Une comparaison qui vaut la peine d'être menée" + +#: website/public/blog/reverse-engineering-models/index.html:668 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"With those checks in place, I would start the comparison on a fixed VC7.1 " +"task family. First, I would establish baselines for a strong general model " +"and a smaller one. Adding repository knowledge to the smaller model would " +"show what retrieval changes. From there, supervised specialization and an " +"RL-trained version would let us examine what training adds." +msgstr "" +"Une fois ces contrôles en place, je commencerais la comparaison sur une " +"famille de tâches VC7.1 fixe. Premièrement, j’établirais les bases d’un " +"modèle d’IA à usage général solide et d’un modèle plus petit. L'ajout de " +"connaissances sur le référentiel au modèle plus petit montrerait les " +"changements apportés à la récupération. À partir de là, une spécialisation " +"supervisée et une version formée par RL nous permettraient d'examiner ce que" +" la formation ajoute." + +#: website/public/blog/reverse-engineering-models/index.html:676 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To make those comparisons informative, each model would receive the same " +"target evidence and tools. For the training-specific comparisons, I would " +"keep the base model and available knowledge fixed as well. Repeated trials " +"would help distinguish a reliable search strategy from a lucky candidate." +msgstr "" +"Pour rendre ces comparaisons informatives, chaque modèle recevrait les mêmes" +" preuves et outils cibles. Pour les comparaisons spécifiques à la formation," +" je conserverais également le modèle de base et les connaissances " +"disponibles fixes. Des essais répétés permettraient de distinguer une " +"stratégie de recherche fiable d'une solution candidate qui réussit par " +"hasard." + +#: website/public/blog/reverse-engineering-models/index.html:683 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would then separate two starting conditions: reconstructing a function " +"from its instructions, and closing the last few differences in maintained " +"source. A specialist might be excellent at the second task while a general " +"model does much of the work needed to reach it. Measuring them separately " +"would reveal that division of work." +msgstr "" +"Je séparerais alors deux conditions de départ : reconstruire une fonction à " +"partir de ses instructions, et fermer les dernières différences dans le " +"source maintenu. Un modèle d’IA spécialisé peut être excellent pour la " +"deuxième tâche, tandis qu’un modèle d’IA à usage général effectue une grande" +" partie du travail nécessaire pour y parvenir. Les mesurer séparément " +"révélerait cette division du travail." + +#: website/public/blog/reverse-engineering-models/index.html:691 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both tasks need genuinely held-out examples. Because related Touhou games " +"share engine code, holding out a game's name would still allow nearly " +"identical functions to appear in training. I would instead hold out source " +"families and keep their accepted answers out of both training and retrieval." +msgstr "" +"Les deux tâches nécessitent des exemples de test exclus de la formation et " +"de la récupération. Les jeux Touhou associés partagent le code moteur. " +"L'exclusion d'un jeu de l'entraînement pourrait néanmoins laisser des " +"fonctions presque identiques dans l'ensemble d'entraînement. Au lieu de " +"cela, j'exclurais des groupes entiers de code source associé et garderais " +"leurs réponses vérifiées hors des données de formation et de la bibliothèque" +" de récupération." + +#: website/public/blog/reverse-engineering-models/index.html:698 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"On those examples, the outcome would be complete, independently validated " +"matches. Giving each model the same candidate or compile budget would help " +"compare search quality. Measuring elapsed time and total cost, including " +"compilation, would tell us whether that quality produces an efficient " +"workflow. Training costs would need their own accounting too." +msgstr "" +"Pour ces exemples, le résultat serait des correspondances complètes et " +"vérifiées de manière indépendante. Des budgets égaux pour les propositions " +"de candidats ou les tentatives de compilation aideraient à comparer la " +"qualité de la recherche. Mesurer le temps et le coût totaux, y compris la " +"compilation, montrerait si cette qualité produit un flux de travail " +"efficace. Les coûts de formation doivent être mesurés séparément." + +#: website/public/blog/reverse-engineering-models/index.html:706 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both views matter because a cheaper proposal can become an expensive " +"workflow if it causes more uninformative compiles. Conversely, a slower " +"model may pay for itself by finding a useful experiment sooner. The " +"comparison should capture those costs." +msgstr "" +"Les deux mesures comptent. Une proposition bon marché peut conduire à un " +"flux de travail coûteux si elle entraîne davantage de compilations ne " +"fournissant aucune information utile. À l’inverse, un modèle plus lent peut " +"être plus économique s’il trouve plus tôt une expérience utile. La " +"comparaison doit inclure ces coûts." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:712 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Counting accepted results also keeps intermediate progress in perspective. " +"One TH09 {ph0}replay-save candidate{ph1} looked promising because the " +"compiled function had the expected size, but a full comparison still found " +"substantial differences. Changes in how the compiler arranges instructions " +"can also make a candidate look closer without resolving the underlying " +"problem. Such diagnostics can guide the next experiment, but I would " +"validate any reward for \"getting closer\" against the full acceptance " +"check." +msgstr "" +"Compter les résultats acceptés aide aussi à interpréter les progrès " +"intermédiaires. Un {ph0}candidat pour la sauvegarde des replays{ph1} de TH09" +" semblait prometteur : la fonction compilée avait la taille attendue. " +"Pourtant, la comparaison complète montrait encore des différences " +"importantes. Un autre ordre des instructions peut donner l’impression que le" +" candidat se rapproche sans résoudre le problème de fond. Ces diagnostics " +"guident l’expérience suivante, mais je vérifierais toute récompense pour ce " +"rapprochement avec le contrôle complet d’acceptation." + +#: website/public/blog/reverse-engineering-models/index.html:727 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would be interested in the pattern across these results. If a specialist " +"does well within a familiar compiler family, does that advantage survive a " +"toolchain change? And once both models have good retrieval, how much does " +"training still add? The answers would help us decide where each kind of " +"model is most useful." +msgstr "" +"Je serais intéressé par la tendance de ces résultats. Si un modèle d’IA " +"spécialisé fonctionne bien au sein d’une famille de compilateurs familière, " +"cet avantage survit-il à un changement de chaîne d’outils ? Et une fois que " +"les deux modèles d’IA ont une bonne récupération, combien la formation " +"ajoute-t-elle encore ? Les réponses nous aideraient à décider où chaque type" +" de modèle est le plus utile." + +#: website/public/blog/reverse-engineering-models/index.html:736 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What kind of agent does this suggest?" +msgstr "Quel type d’agent IA cela suggère-t-il ?" + +#: website/public/blog/reverse-engineering-models/index.html:737 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"If the models turn out to help at different stages, a useful arrangement " +"might be a broad agent with access to a specialist. Once a task becomes a " +"familiar compiler search, it could ask the specialist for candidates. If the" +" results stop making sense, the broader agent could return to the target " +"evidence or investigate the measurement itself." +msgstr "" +"Si les modèles s’avèrent utiles à différentes étapes, un arrangement utile " +"pourrait être un agent large ayant accès à un modèle d’IA spécialisé. Une " +"fois qu’une tâche devient une recherche familière du compilateur, elle peut " +"demander des candidats au modèle d’IA spécialisé. Si les résultats n’ont " +"plus de sens, l’agent au sens large pourrait revenir aux preuves cibles ou " +"enquêter sur la mesure elle-même." + +#: website/public/blog/reverse-engineering-models/index.html:745 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In that arrangement, specialization becomes a tool the agent can choose as " +"the investigation develops. The agent keeps the freedom to change direction " +"that made the earlier breakthroughs possible." +msgstr "" +"Dans cet arrangement, la spécialisation devient un outil que l’agent peut " +"choisir au fur et à mesure que l’enquête se développe. L’agent conserve la " +"liberté de changer de direction qui a rendu possible les percées " +"antérieures." + +#: website/public/blog/reverse-engineering-models/index.html:750 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This follows naturally from the industrial shift I described in the previous" +" essay. Being able to run many more experiments changes how we organize the " +"work. It also gives us a reason to ask which decisions need broad judgment " +"and which improve through repeated experience with a particular craft." +msgstr "" +"Cela découle naturellement du virage industriel que j’ai décrit dans l’essai" +" précédent. Être capable de mener beaucoup plus d’expériences change la " +"façon dont nous organisons le travail. Cela nous donne également une raison " +"de nous demander quelles décisions nécessitent un jugement large et " +"lesquelles s'améliorent grâce à une expérience répétée avec un métier " +"particulier." + +#: website/public/blog/reverse-engineering-models/index.html:757 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For now, the strongest general model has earned its place in my workflow by " +"opening problems I had left unresolved. Those successes are what make me " +"want to examine the work more closely: how much depended on the model's " +"breadth, and how much could be handled by a model trained in this particular" +" kind of reconstruction?" +msgstr "" +"Pour l'instant, le modèle général le plus puissant a gagné sa place dans mon" +" travail en résolvant des problèmes que j'avais laissés en suspens. Ces " +"réussites me donnent envie d'étudier le processus : quelle part dépend de " +"ses capacités générales, et quelle part pourrait être confiée à un modèle " +"formé pour ce type de reconstruction ?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:765 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}The repositories give us a way to test the question.{ph1} I would like " +"to use that record to build the comparison and see what we learn." +msgstr "" +"{ph0}Les référentiels nous permettent de tester la question.{ph1} J'aimerais" +" utiliser cet enregistrement pour établir la comparaison et voir ce que nous" +" apprenons." + +#: website/public/blog/reverse-engineering-models/index.html:775 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Evidence and model attribution" +msgstr "Attribution de preuves et de modèles" + +#: website/public/blog/reverse-engineering-models/index.html:783 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The examples describe historical function-level comparisons recorded in the " +"linked commits. Whole-game runtime checks are separate milestones. Source " +"and matching records were reviewed on 13 October 2026." +msgstr "" +"Les exemples décrivent des comparaisons historiques au niveau des fonctions " +"enregistrées dans les validations liées. Les vérifications d'exécution du " +"jeu complet sont des étapes distinctes. Les enregistrements sources et " +"correspondants ont été examinés le 13 octobre 2026." + +#. {ph0}: gpt-dots +#. {ph1}: gpt-web +#: website/public/blog/reverse-engineering-models/index.html:789 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The TH09 label {ph0} identifies GPT-6-Astra. {ph1} changed with the model " +"used at the time, so that label alone does not identify an individual run. " +"Explicit model names in the other cited commits establish their recorded " +"attribution; the histories do not contain uniform per-run cost or effort " +"measurements." +msgstr "" +"L'étiquette TH09 {ph0} signifie GPT-6-Astra. Le modèle associé à {ph1} a " +"changé au fil du temps, de sorte que cette étiquette à elle seule ne peut " +"pas identifier le modèle utilisé dans une exécution spécifique. Les noms de " +"modèles explicites dans les autres commits cités fournissent l'attribution " +"enregistrée. Ces historiques n'incluent pas de mesures cohérentes du coût ou" +" de l'effort pour chaque exécution." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#: website/public/blog/reverse-engineering-models/index.html:798 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 temporary lifetimes:{ph1} the {ph2}earlier context review{ph3} and" +" {ph4}complete cold reconstruction{ph5}." +msgstr "" +"{ph0}TH09 durées de vie des objets temporaires :{ph1} la {ph2}examen du " +"contexte précédent{ph3} et la {ph4}reconstruction complète vérifiée par une " +"recompilation propre à partir de la source{ph5}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:810 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 Front callback:{ph1} {ph2}non-exact diagnostic regions{ph3}, " +"{ph4}Astra's new graph{ph5} and its {ph6}accepted recompile replay{ph7}." +msgstr "" +"{ph0}Callback Front de TH09 :{ph1} {ph2}régions de diagnostic non " +"concordantes{ph3}, {ph4}nouveau graphe de contrôle d'Astra{ph5} et " +"{ph6}vérification de recompilation acceptée{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:826 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Compiler knowledge:{ph1} {ph2}TH095 identifier-bucket reuse{ph3}, " +"{ph4}TH04 player source and linker order{ph5} and {ph6}Luna's TH09 enemy-" +"spawn closure{ph7}." +msgstr "" +"{ph0}Connaissance du compilateur :{ph1} {ph2}TH095 réutilisation du " +"comportement du compartiment d'identifiant{ph3}, {ph4}TH04 code source du " +"joueur et ordre de l'éditeur de liens{ph5} et {ph6}Luna reconstruction " +"complète de la fonction de création d'ennemis de TH09{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:842 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Checks and diagnostics:{ph1} {ph2}TH08 floating-literal audit{ph3}, " +"{ph4}TH08 numerical investigation{ph5} and {ph6}TH09 replay-save " +"reconstruction{ph7}." +msgstr "" +"{ph0}Contrôles et diagnostics :{ph1} {ph2}TH08 audit de constantes à virgule" +" flottante{ph3}, {ph4}TH08 investigation numérique{ph5} et {ph6}TH09 " +"reconstruction permettant de sauvegarder les relectures{ph7}." + +#: website/public/blog/reverse-engineering-models/index.html:859 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The proposed specialist and RL comparisons are future experiments. The " +"project histories motivate them; no trained specialist was evaluated for " +"this essay." +msgstr "" +"Les comparaisons proposées de modèles spécialisés et de formation RL sont " +"des expériences futures. Les histoires de projets les motivent. Aucun modèle" +" spécialisé formé n'a été évalué pour cet essai." + +#: website/public/blog/reverse-engineering-models/index.html:6 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model? | REA" +msgstr "L'ingénierie inverse nécessite-t-elle un modèle plus grand ? | REA" + +#: website/public/blog/reverse-engineering-models/index.html:7 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What Touhou reconstruction taught me about general models, specialist models" +" and learning to propose better hypotheses from compiler feedback." +msgstr "" +"Ce que la reconstruction Touhou m'a appris sur les modèles généraux, les " +"modèles spécialisés et l'apprentissage pour proposer de meilleures " +"hypothèses à partir des retours du compilateur." + +#: website/public/blog/reverse-engineering-models/index.html:240 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "On this page" +msgstr "Sur cette page" + +#: website/public/blog/reverse-engineering-models/index.html:253 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Models and reverse engineering essay" +msgstr "Modèles et essai d'ingénierie inverse" + +#: website/public/blog/reverse-engineering-models/index.html:491 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Selected reconstruction experiments" +msgstr "Expériences de reconstruction sélectionnées" + +#: website/public/blog/reverse-engineering-models/index.html:883 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Back to top" +msgstr "Haut de page" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:776 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Detailed evidence notes{ph1} include the recorded comparisons and " +"experiment context." +msgstr "" +"Les {ph0}notes détaillées sur les preuves{ph1} présentent les comparaisons " +"enregistrées et le contexte des expériences." + +#~ msgctxt "page:blog/index.html" +#~ msgid "By N0zoM1z0 · 13 October 2026 · Draft for review" +#~ msgstr "Par N0zoM1z0 · 13 octobre 2026 · Brouillon à relire" + #~ msgctxt "page:index.html" #~ msgid "Setup guide" #~ msgstr "Guide d’installation" diff --git a/website/locales/id/messages.po b/website/locales/id/messages.po index 39e1ae619..92d94c93a 100644 --- a/website/locales/id/messages.po +++ b/website/locales/id/messages.po @@ -32,18 +32,18 @@ msgid "Thoughts on reverse engineering." msgstr "Pemikiran tentang rekayasa terbalik." #. {ph0}:
-#: website/public/blog/index.html:200 +#: website/public/blog/index.html:214 #, python-brace-format msgctxt "page:blog/index.html" msgid "Game reconstruction{ph0}Essay" msgstr "Rekonstruksi permainan{ph0}Esai" -#: website/public/blog/index.html:202 +#: website/public/blog/index.html:216 msgctxt "page:blog/index.html" msgid "The industrial revolution in reverse engineering" msgstr "Revolusi industri dalam rekayasa terbalik" -#: website/public/blog/index.html:203 +#: website/public/blog/index.html:217 msgctxt "page:blog/index.html" msgid "" "What manual RE and reconstructing TH08, TH095 and TH04 taught me about " @@ -53,25 +53,26 @@ msgstr "" " tentang agen, verifikasi, dan pengetahuan yang digabungkan di seluruh " "proyek." -#: website/public/blog/index.html:208 +#: website/public/blog/index.html:207 website/public/blog/index.html:222 msgctxt "page:blog/index.html" msgid "Read the article" msgstr "Baca artikelnya" -#: website/public/blog/index.html:211 +#: website/public/blog/index.html:225 msgctxt "page:blog/index.html" msgid "By N0zoM1z0 · 10 October 2026" msgstr "Oleh N0zoM1z0 · 10 Oktober 2026" #. {ph0}: -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 +#: website/public/blog/reverse-engineering-models/index.html:883 #: website/public/blog/touhou-reconstruction/index.html:1080 #: website/public/examples/dino-lab/index.html:475 #: website/public/faq/index.html:573 #: website/public/first-investigation/index.html:523 #: website/public/get-started/index.html:882 -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 #: website/public/guides/index.html:282 #: website/public/guides/javascript/index.html:503 #: website/public/guides/native/index.html:509 website/public/index.html:1191 @@ -101,6 +102,8 @@ msgstr "" " REA dan AI." #: website/public/blog/index.html:75 website/public/blog/index.html:88 +#: website/public/blog/reverse-engineering-models/index.html:75 +#: website/public/blog/reverse-engineering-models/index.html:88 #: website/public/blog/touhou-reconstruction/index.html:75 #: website/public/blog/touhou-reconstruction/index.html:88 #: website/public/examples/dino-lab/index.html:75 @@ -137,6 +140,7 @@ msgstr "" "pengkodean Anda." #: website/public/blog/index.html:108 +#: website/public/blog/reverse-engineering-models/index.html:108 #: website/public/blog/touhou-reconstruction/index.html:108 #: website/public/examples/dino-lab/index.html:109 #: website/public/faq/index.html:108 @@ -157,6 +161,7 @@ msgid "REA home" msgstr "Beranda REA" #: website/public/blog/index.html:112 +#: website/public/blog/reverse-engineering-models/index.html:112 #: website/public/blog/touhou-reconstruction/index.html:112 #: website/public/examples/dino-lab/index.html:113 #: website/public/faq/index.html:112 @@ -177,6 +182,7 @@ msgid "Main navigation" msgstr "Navigasi utama" #: website/public/blog/index.html:130 +#: website/public/blog/reverse-engineering-models/index.html:130 #: website/public/blog/touhou-reconstruction/index.html:130 #: website/public/examples/dino-lab/index.html:131 #: website/public/faq/index.html:130 @@ -197,6 +203,7 @@ msgid "REA on GitHub" msgstr "REA di GitHub" #: website/public/blog/index.html:150 +#: website/public/blog/reverse-engineering-models/index.html:150 #: website/public/blog/touhou-reconstruction/index.html:150 #: website/public/examples/dino-lab/index.html:151 #: website/public/faq/index.html:150 @@ -217,6 +224,7 @@ msgid "Join REA on Discord" msgstr "Gabung REA di Discord" #: website/public/blog/index.html:169 +#: website/public/blog/reverse-engineering-models/index.html:169 #: website/public/blog/touhou-reconstruction/index.html:169 #: website/public/examples/dino-lab/index.html:170 #: website/public/faq/index.html:169 @@ -236,7 +244,7 @@ msgctxt "shared" msgid "Follow REA on X" msgstr "Ikuti REA di X" -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 msgctxt "page:blog/index.html" msgid "Back to top" msgstr "Kembali ke atas" @@ -4785,7 +4793,7 @@ msgctxt "page:guides/browser/index.html" msgid "Copy read the loaded scripts" msgstr "Salin baca skrip yang dimuat" -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 msgctxt "page:guides/browser/index.html" msgid "Back to top" msgstr "Kembali ke atas" @@ -11654,6 +11662,7 @@ msgid "Same spacing: 256 ÷ 16 = 16 angle units = 22.5°" msgstr "Jarak yang sama: 256 ÷ 16 = 16 satuan sudut = 22,5°" #: website/public/blog/index.html:114 +#: website/public/blog/reverse-engineering-models/index.html:114 #: website/public/blog/touhou-reconstruction/index.html:114 #: website/public/examples/dino-lab/index.html:115 #: website/public/faq/index.html:114 @@ -11674,6 +11683,7 @@ msgid "Showcases" msgstr "Studi kasus" #: website/public/blog/index.html:115 +#: website/public/blog/reverse-engineering-models/index.html:115 #: website/public/blog/touhou-reconstruction/index.html:115 #: website/public/examples/dino-lab/index.html:116 #: website/public/faq/index.html:115 @@ -11694,6 +11704,7 @@ msgid "Blog" msgstr "Blog" #: website/public/blog/index.html:113 +#: website/public/blog/reverse-engineering-models/index.html:113 #: website/public/blog/touhou-reconstruction/index.html:113 #: website/public/examples/dino-lab/index.html:114 #: website/public/faq/index.html:113 @@ -11714,6 +11725,7 @@ msgid "Get started" msgstr "Mulai" #: website/public/blog/index.html:116 +#: website/public/blog/reverse-engineering-models/index.html:116 #: website/public/blog/touhou-reconstruction/index.html:116 #: website/public/examples/dino-lab/index.html:117 #: website/public/faq/index.html:116 @@ -11733,13 +11745,14 @@ msgctxt "shared" msgid "Guides" msgstr "Panduan" -#: website/public/blog/index.html:222 +#: website/public/blog/index.html:236 +#: website/public/blog/reverse-engineering-models/index.html:875 #: website/public/blog/touhou-reconstruction/index.html:1072 #: website/public/examples/dino-lab/index.html:467 #: website/public/faq/index.html:565 #: website/public/first-investigation/index.html:515 #: website/public/get-started/index.html:874 -#: website/public/guides/browser/index.html:490 +#: website/public/guides/browser/index.html:499 #: website/public/guides/index.html:274 #: website/public/guides/javascript/index.html:495 #: website/public/guides/native/index.html:501 website/public/index.html:1183 @@ -11753,13 +11766,14 @@ msgctxt "shared" msgid "Source" msgstr "Kode sumber" -#: website/public/blog/index.html:223 +#: website/public/blog/index.html:237 +#: website/public/blog/reverse-engineering-models/index.html:876 #: website/public/blog/touhou-reconstruction/index.html:1073 #: website/public/examples/dino-lab/index.html:468 #: website/public/faq/index.html:566 #: website/public/first-investigation/index.html:516 #: website/public/get-started/index.html:875 -#: website/public/guides/browser/index.html:491 +#: website/public/guides/browser/index.html:500 #: website/public/guides/index.html:275 #: website/public/guides/javascript/index.html:496 #: website/public/guides/native/index.html:502 website/public/index.html:1184 @@ -11773,13 +11787,14 @@ msgctxt "shared" msgid "MIT license" msgstr "Lisensi MIT" -#: website/public/blog/index.html:224 +#: website/public/blog/index.html:238 +#: website/public/blog/reverse-engineering-models/index.html:877 #: website/public/blog/touhou-reconstruction/index.html:1074 #: website/public/examples/dino-lab/index.html:469 #: website/public/faq/index.html:567 #: website/public/first-investigation/index.html:517 #: website/public/get-started/index.html:876 -#: website/public/guides/browser/index.html:492 +#: website/public/guides/browser/index.html:501 #: website/public/guides/index.html:276 #: website/public/guides/javascript/index.html:497 #: website/public/guides/native/index.html:503 website/public/index.html:1185 @@ -11793,13 +11808,14 @@ msgctxt "shared" msgid "Join the community" msgstr "Gabung komunitas" -#: website/public/blog/index.html:225 +#: website/public/blog/index.html:239 +#: website/public/blog/reverse-engineering-models/index.html:878 #: website/public/blog/touhou-reconstruction/index.html:1075 #: website/public/examples/dino-lab/index.html:470 #: website/public/faq/index.html:568 #: website/public/first-investigation/index.html:518 #: website/public/get-started/index.html:877 -#: website/public/guides/browser/index.html:493 +#: website/public/guides/browser/index.html:502 #: website/public/guides/index.html:277 #: website/public/guides/javascript/index.html:498 #: website/public/guides/native/index.html:504 website/public/index.html:1186 @@ -12740,6 +12756,1409 @@ msgctxt "runtime" msgid "Switch to dark mode" msgstr "Beralih ke mode gelap" +#. {ph0}: +#. {ph1}: +#: website/public/guides/browser/index.html:480 +#, python-brace-format +msgctxt "page:guides/browser/index.html" +msgid "" +"For HTTP request and response bodies, use {ph0}Fiddler Everywhere: SAZ " +"captures, MCP and public skills{ph1}. Export SAZ or HAR 1.2 for offline REA " +"capture Evidence, or connect Everywhere's MCP server to inspect captured " +"traffic with your agent." +msgstr "" +"Untuk memeriksa isi permintaan dan respons HTTP, gunakan {ph0}Fiddler " +"Everywhere: rekaman SAZ, MCP, dan skill publik{ph1}. Ekspor SAZ atau HAR 1.2" +" untuk menghasilkan Evidence rekaman di REA secara luring, atau hubungkan " +"server MCP Everywhere untuk memeriksa lalu lintas yang direkam bersama agen " +"Anda." + +#. {ph0}:
+#: website/public/blog/index.html:200 +#, python-brace-format +msgctxt "page:blog/index.html" +msgid "Model specialization{ph0}Essay" +msgstr "Model khusus{ph0}Esai" + +#: website/public/blog/index.html:202 +msgctxt "page:blog/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "Apakah rekayasa balik membutuhkan model yang lebih besar?" + +#: website/public/blog/index.html:203 +msgctxt "page:blog/index.html" +msgid "" +"What Touhou reconstruction suggests about general models, specialist models " +"and learning from compiler feedback." +msgstr "" +"Pelajaran dari rekonstruksi Touhou tentang model umum, model khusus, dan " +"pembelajaran dari umpan balik kompiler." + +#: website/public/blog/index.html:210 +msgctxt "page:blog/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Oleh N0zoM1z0 · 13 Oktober 2026" + +#: website/public/blog/reverse-engineering-models/index.html:103 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Skip to content" +msgstr "Lewati ke konten" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:194 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Home{ph1} / {ph2}Blog{ph3} / Models and RE" +msgstr "{ph0}Beranda{ph1} / {ph2}Blog{ph3} / Model dan rekayasa balik" + +#: website/public/blog/reverse-engineering-models/index.html:197 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Essay" +msgstr "Esai" + +# Reviewed for technical meaning and readability; preserve the distinction +# between historical evidence and proposed experiments. +#: website/public/blog/reverse-engineering-models/index.html:198 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "Apakah rekayasa balik memerlukan model yang lebih besar?" + +#: website/public/blog/reverse-engineering-models/index.html:199 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Oleh N0zoM1z0 · 13 Oktober 2026" + +#: website/public/blog/reverse-engineering-models/index.html:201 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Takeaway" +msgstr "Poin utama" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:203 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Model capability changes the hypotheses an agent can find.{ph1} Later " +"sessions have solved problems that earlier work left unresolved." +msgstr "" +"{ph0}Kemampuan model memengaruhi hipotesis mana yang dapat diajukan oleh " +"agen AI.{ph1} Sesi berikutnya memecahkan masalah yang belum terselesaikan " +"oleh pekerjaan sebelumnya." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:211 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A familiar compiler gives specialization something concrete to " +"learn.{ph1} Some exact matches depend on narrow, reusable source patterns." +msgstr "" +"{ph0}Kompiler yang familiar menawarkan pola konkret untuk dipelajari model " +"khusus.{ph1} Beberapa kecocokan kode mesin yang tepat bergantung pada pola " +"kode sumber spesifik yang dapat digunakan kembali." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:218 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Give repository knowledge a fair comparison.{ph1} A smaller model with " +"good retrieval may already know enough to make progress." +msgstr "" +"{ph0}Sertakan pengetahuan repositori dalam perbandingan yang adil.{ph1} " +"Model yang lebih kecil dengan pengambilan pengetahuan yang baik mungkin " +"sudah memiliki informasi yang cukup untuk membuat kemajuan." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:223 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Train the next useful experiment, and test its oracle.{ph1} The value " +"of feedback depends on what the checker actually verifies." +msgstr "" +"{ph0}Latih model untuk memilih eksperimen berguna berikutnya, dan uji " +"pemeriksanya.{ph1} Masukan hanya berguna jika pemeriksa memverifikasi hal " +"yang benar." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:230 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Measure accepted reconstruction against time and cost.{ph1} That is the" +" result I want from the whole agent." +msgstr "" +"{ph0}Evaluasi hasil rekonstruksi terverifikasi beserta waktu dan biaya.{ph1}" +" Itulah yang ingin saya ukur untuk alur kerja agen AI yang lengkap." + +#: website/public/blog/reverse-engineering-models/index.html:241 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "In this article" +msgstr "Dalam artikel ini" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:243 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What better models changed{ph1}" +msgstr "{ph0}Apa yang berubah dengan model yang lebih kuat?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:244 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Why specialize?{ph1}" +msgstr "{ph0}Mengapa mengambil spesialisasi?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:245 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Where experience lives{ph1}" +msgstr "{ph0}Di mana pengalaman ini harus disimpan?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:246 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Learning the next experiment{ph1}" +msgstr "{ph0}Pelajari eksperimen bermanfaat berikutnya{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:247 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}RE and reinforcement learning{ph1}" +msgstr "{ph0}Pembelajaran rekayasa balik dan penguatan{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:248 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}When the oracle is wrong{ph1}" +msgstr "{ph0}Ketika pemeriksa salah{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:249 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}A comparison worth running{ph1}" +msgstr "{ph0}Sebuah perbandingan yang layak untuk dijalankan{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:250 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What kind of agent?{ph1}" +msgstr "{ph0}Agen AI macam apa yang disarankan di sini?{ph1}" + +#: website/public/blog/reverse-engineering-models/index.html:258 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What better models changed" +msgstr "Apa yang berubah dengan model yang lebih kuat?" + +#: website/public/blog/reverse-engineering-models/index.html:259 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Since August 2026, I have been exploring how much reverse engineering an " +"agent can carry out with access to a program and a way to check its work. " +"Most of my experience comes from reconstructing games, especially Touhou. As" +" I have changed models, the repositories have kept a record of what each " +"session managed to recover." +msgstr "" +"Sejak Agustus 2026, saya telah mengeksplorasi seberapa banyak rekayasa balik" +" yang dapat dilakukan agen dengan akses ke suatu program dan cara untuk " +"memeriksa kerjanya. Sebagian besar pengalaman saya berasal dari " +"merekonstruksi game, terutama Touhou. Karena saya telah mengubah model, " +"repositori menyimpan catatan tentang apa yang berhasil dipulihkan oleh " +"setiap sesi." + +#: website/public/blog/reverse-engineering-models/index.html:267 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of that work aims at exact reconstruction: we compile the maintained " +"source under a pinned historical toolchain and check whether it reproduces " +"the target function. When a later model succeeds where earlier work stalled," +" the source changes show what made that progress possible." +msgstr "" +"Sebagian besar pekerjaan tersebut bertujuan untuk rekonstruksi yang tepat: " +"kami mengkompilasi sumber yang dikelola di bawah rantai alat historis dengan" +" versi tetap dan memeriksa apakah sumber tersebut mereproduksi fungsi " +"target. Ketika model selanjutnya berhasil ketika pekerjaan sebelumnya " +"terhenti, perubahan sumber menunjukkan apa yang memungkinkan kemajuan " +"tersebut." + +#: website/public/blog/reverse-engineering-models/index.html:274 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The temporary values had to stay alive together" +msgstr "Variabel sementara membutuhkan masa hidup yang tumpang tindih" + +#: website/public/blog/reverse-engineering-models/index.html:275 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of a TH09 attack function's logic had already been reconstructed, but " +"the compiler still stored temporary results differently from the original. " +"The remaining problem was how long those results needed to stay alive and " +"when their storage could be reused." +msgstr "" +"Sebagian besar logika fungsi serangan TH09 telah direkonstruksi, namun " +"kompiler masih menyimpan hasil sementara yang berbeda dari aslinya. Masalah " +"yang tersisa adalah berapa lama hasil tersebut perlu bertahan dan kapan " +"penyimpanannya dapat digunakan kembali." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:282 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A {ph0}GPT-6-Sol checkpoint{ph1} recorded this mismatch after testing " +"several ordinary source arrangements. When {ph2}GPT-6.1-Sol returned to the " +"problem{ph3}, it found a useful hypothesis about how long those temporary " +"values needed to stay alive." +msgstr "" +"Catatan sesi {ph0}GPT-6-Sol{ph1} melaporkan ketidakcocokan ini setelah " +"menguji beberapa pengaturan kode sumber biasa. Ketika {ph2}GPT-6.1-Sol " +"memeriksa masalahnya lagi{ph3}, ia mengajukan hipotesis yang berguna tentang" +" masa hidup variabel-variabel sementara ini." + +#: website/public/blog/reverse-engineering-models/index.html:296 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The function produced temporary results and copied them into animation " +"positions. In C++, a temporary result stays alive until the expression using" +" it finishes. Grouping the real operations into one expression kept those " +"results alive together, so the compiler could no longer reuse the same " +"storage for all of them." +msgstr "" +"Fungsi tersebut memberikan hasil sementara dan menyalinnya ke posisi " +"animasi. Di C++, hasil sementara tetap aktif hingga ekspresi yang " +"menggunakannya selesai. Mengelompokkan operasi nyata ke dalam satu ekspresi " +"membuat hasil tersebut tetap hidup, sehingga kompiler tidak dapat lagi " +"menggunakan penyimpanan yang sama untuk semuanya." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:303 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"One result still shared storage with a value used later for collision " +"checks. Giving that value its own lifetime resolved the remaining overlap. " +"After several targeted experiments, {ph0}a clean rebuild confirmed that the " +"full function matched{ph1}." +msgstr "" +"Salah satu hasil masih berbagi penyimpanan dengan nilai yang nantinya " +"digunakan untuk pemeriksaan tabrakan. Memberikan nilai tersebut seumur " +"hidupnya akan menyelesaikan sisa tumpang tindih. Setelah beberapa eksperimen" +" yang ditargetkan, {ph0}kompilasi ulang yang bersih dari sumber " +"mengonfirmasi bahwa seluruh fungsi cocok{ph1}." + +#: website/public/blog/reverse-engineering-models/index.html:312 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What interests me is how the discovery and the tests worked together. The " +"lifetime hypothesis gave the agent a new direction; compiler feedback helped" +" it refine that hypothesis into a source arrangement that matched." +msgstr "" +"Saya tertarik pada bagaimana penemuan dan pengujian saling mendukung. " +"Hipotesis tentang masa hidup objek yang bersifat sementara memberikan arah " +"baru bagi agen AI. Umpan balik dari kompiler membantunya menyempurnakan " +"hipotesis tersebut menjadi susunan kode sumber yang cocok." + +#: website/public/blog/reverse-engineering-models/index.html:318 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A different control-flow graph" +msgstr "Grafik aliran kontrol yang berbeda" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:319 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A second TH09 function shows why that new direction can matter. Here, the " +"obstacle was how several branches shared their final work. " +"{ph0}GPT-6.1-Sol's earlier work{ph1} got parts of the function to match, but" +" the branches still joined up differently from the original." +msgstr "" +"Fungsi TH09 kedua menunjukkan mengapa arah baru itu penting. Di sini " +"kendalanya adalah bagaimana beberapa cabang membagikan karya akhirnya. " +"{ph0}Pekerjaan sebelumnya GPT-6.1-Sol{ph1} memiliki bagian-bagian fungsi " +"yang cocok, namun cabang-cabangnya masih digabungkan secara berbeda dari " +"aslinya." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:330 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}GPT-6-Astra changed the surrounding control-flow graph{ph1}. The main " +"paths now led to a shared final update and return. In that source structure," +" the compiler combined the branches in the same way as the original. The " +"complete function matched, and a {ph2}separate rebuild check confirmed " +"it{ph3}." +msgstr "" +"{ph0}GPT-6-Astra mengubah grafik aliran kontrol di sekitarnya{ph1}. Jalur " +"utama sekarang mengarah ke pembaruan akhir bersama dan kembalinya. Dalam " +"struktur sumber tersebut, kompiler menggabungkan cabang-cabang dengan cara " +"yang sama seperti aslinya. Fungsi keseluruhannya cocok, dan {ph2}pemeriksaan" +" pembangunan kembali secara terpisah mengonfirmasi hal tersebut{ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:342 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Changing the representation opened a more productive search space.{ph1}" +" Both cases give me a reason to take general model capability seriously: an " +"agent can keep trying local variations for a long time before finding a " +"better way to express the surrounding work." +msgstr "" +"{ph0}Mengubah representasi membuka ruang pencarian yang lebih " +"produktif.{ph1} Kedua kasus tersebut memberi saya alasan untuk menganggap " +"serius kemampuan model umum: agen dapat terus mencoba variasi lokal untuk " +"waktu yang lama sebelum menemukan cara yang lebih baik untuk mengekspresikan" +" pekerjaan di sekitarnya." + +#: website/public/blog/reverse-engineering-models/index.html:351 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"At the same time, the later sessions started with earlier source and " +"recorded failures. That accumulated work was part of their advantage. The " +"history lets us inspect the discoveries and their checks; a controlled " +"comparison would help separate the effect of the model from the experience " +"it inherited." +msgstr "" +"Pada saat yang sama, sesi selanjutnya dimulai dengan sumber sebelumnya dan " +"mencatat kegagalan. Akumulasi pekerjaan itu adalah bagian dari keuntungan " +"mereka. Sejarah memungkinkan kita memeriksa penemuan-penemuan dan " +"pemeriksaannya; perbandingan terkontrol akan membantu memisahkan pengaruh " +"model dari pengalaman yang diwarisinya." + +#: website/public/blog/reverse-engineering-models/index.html:358 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is where the question becomes interesting to me. Some of the progress " +"came from finding a new representation, while much of the remaining work " +"involved testing source forms against a familiar compiler. How much needs " +"broad reasoning ability, and how much could a model learn through repeated " +"experience with that compiler and code?" +msgstr "" +"Inilah yang membuat pertanyaan ini menarik bagi saya. Beberapa kemajuan " +"datang dari penemuan representasi baru. Sebagian besar pekerjaan yang " +"tersisa melibatkan pengujian pengaturan kode sumber yang berbeda terhadap " +"kompiler yang sudah dikenal. Seberapa banyak hal ini memerlukan kemampuan " +"penalaran yang luas? Berapa banyak yang dapat dipelajari suatu model dengan " +"bekerja berulang kali menggunakan compiler dan kode tersebut?" + +#: website/public/blog/reverse-engineering-models/index.html:368 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why might a specialist do well?" +msgstr "Mengapa model AI khusus dapat berfungsi dengan baik?" + +#: website/public/blog/reverse-engineering-models/index.html:369 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Touhou gives us a concrete reason to expect that experience to help. Much of" +" the game code comes from the same author, ZUN, so recurring habits are a " +"reasonable starting assumption. The repositories back that up with useful " +"examples: related loops and helper arrangements have supplied hypotheses " +"across games." +msgstr "" +"Touhou memberi kita alasan konkret untuk mengharapkan pengalaman itu " +"membantu. Sebagian besar kode permainan berasal dari penulis yang sama, ZUN," +" jadi kebiasaan berulang adalah asumsi awal yang masuk akal. Repositori " +"mendukung hal tersebut dengan contoh-contoh yang berguna: loop terkait dan " +"pengaturan pembantu telah memberikan hipotesis di seluruh game." + +#: website/public/blog/reverse-engineering-models/index.html:376 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those source patterns become more specific when we include the compiler. " +"Under a particular version and set of options, small changes in source can " +"produce repeatable differences in machine code. Some of those differences " +"are surprisingly obscure." +msgstr "" +"Pola sumber tersebut menjadi lebih spesifik ketika kita menyertakan " +"kompiler. Dalam versi dan rangkaian opsi tertentu, perubahan kecil pada " +"sumber dapat menghasilkan perbedaan berulang dalam kode mesin. Beberapa dari" +" perbedaan tersebut ternyata tidak jelas." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: Supervisor::LoadDat +#. {ph4}: +#: website/public/blog/reverse-engineering-models/index.html:382 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In TH095, for example, {ph0}even the name used for a local variable could " +"change where VC7.1 stored it{ph1}. Moving the declaration alone did not " +"solve the layout, but an earlier background function had revealed a naming " +"pattern that affected the compiler's choice. That gave us a more promising " +"hypothesis for {ph2}the reconstruction of {ph3}{ph4}." +msgstr "" +"Di TH095, misalnya, {ph0}bahkan nama variabel lokal dapat memengaruhi tempat" +" VC7.1 menyimpannya{ph1}. Memindahkan deklarasinya tidak menyelesaikan tata " +"letak penyimpanan. Namun, fungsi latar belakang sebelumnya telah " +"mengungkapkan pola penamaan yang mempengaruhi pilihan kompiler. Hal ini " +"menunjukkan hipotesis yang lebih menjanjikan untuk {ph2}merekonstruksi " +"{ph3}{ph4}." + +#: website/public/blog/reverse-engineering-models/index.html:396 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reusing that pattern for the file-size variable produced the required layout" +" without adding storage. The full function then matched." +msgstr "" +"Menggunakan kembali pola tersebut untuk variabel ukuran file menghasilkan " +"tata letak yang diperlukan tanpa menambahkan penyimpanan. Fungsi penuh " +"kemudian dicocokkan." + +#: website/public/blog/reverse-engineering-models/index.html:401 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is the sort of narrow detail a general model may not know when it " +"arrives. A specialist trained on this compiler could encounter many such " +"cases and learn which source variations are worth testing first." +msgstr "" +"Ini adalah detail yang sangat spesifik yang mungkin tidak diketahui oleh " +"model AI tujuan umum saat pertama kali mengalami masalah. Model yang " +"dikhususkan untuk kompiler ini dapat melihat banyak kasus seperti itu dan " +"mempelajari variasi kode sumber mana yang layak untuk diuji terlebih dahulu." + +#: website/public/blog/reverse-engineering-models/index.html:407 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The opportunity is especially clear in final matching problems where we " +"already understand the behavior. We are then asking which ordinary source " +"form this compiler turns into the observed instructions. Specialization " +"could help the model propose those forms more efficiently, while the target " +"still determines which hypotheses survive." +msgstr "" +"Peluangnya sangat jelas dalam masalah pencocokan akhir dimana kita sudah " +"memahami perilakunya. Kami kemudian menanyakan pengaturan kode sumber biasa " +"mana yang diubah oleh kompiler ini menjadi instruksi yang diamati. " +"Spesialisasi dapat membantu model mengusulkan bentuk-bentuk tersebut dengan " +"lebih efisien, sementara target tetap menentukan hipotesis mana yang " +"bertahan." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:415 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The useful patterns would have to retain their context, though. TH08 uses an" +" earlier Visual C++ version than TH095 and TH09. TH04 uses a different DOS-" +"era compiler, and its {ph0}player reconstruction{ph1} needed different " +"hypotheses about source patterns and how the compiled code was combined into" +" an executable. Shared authorship gives us a prior, but each target tells us" +" whether it applies. Distinguishing those contexts is part of what a " +"specialist would need to learn." +msgstr "" +"Pola yang berguna harus mempertahankan konteksnya. TH08 menggunakan versi " +"Visual C++ yang lebih lama dari TH095 dan TH09. TH04 menggunakan compiler " +"yang berbeda dari era DOS. {ph0}rekonstruksi kode pemain{ph1} memerlukan " +"hipotesis berbeda tentang pola kode sumber dan bagaimana kode yang " +"dikompilasi dihubungkan ke dalam file yang dapat dieksekusi. Memiliki " +"penulis yang sama memberi kita ekspektasi awal, namun setiap target memberi " +"tahu kita apakah target tersebut berlaku. Model khusus perlu membedakan " +"konteks-konteks ini." + +#: website/public/blog/reverse-engineering-models/index.html:431 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Where should that experience live?" +msgstr "Di mana pengalaman ini harus disimpan?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:432 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before deciding to teach those patterns through model training, I would look" +" at the knowledge we already keep in the repository. In the {ph0}previous " +"essay{ph1}, I wrote about it becoming the project's working memory. That " +"memory includes compiler lessons, along with the source and failed " +"experiments that made them useful." +msgstr "" +"Sebelum memutuskan untuk mengajarkan pola-pola tersebut melalui pelatihan " +"model, saya akan melihat pengetahuan yang sudah kita simpan di gudang. Dalam" +" {ph0}esai sebelumnya{ph1}, saya menulis tentang hal ini yang menjadi memori" +" kerja proyek. Memori tersebut mencakup pelajaran kompiler, beserta " +"sumbernya dan eksperimen gagal yang menjadikannya berguna." + +#: website/public/blog/reverse-engineering-models/index.html:441 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"By retrieving that record, an agent can start with a known result and " +"inspect how we reached it. It can also avoid alternatives that have already " +"failed in the same context. The starting point improves without changing the" +" model's weights." +msgstr "" +"Dengan mengambil catatan tersebut, agen dapat memulai dengan hasil yang " +"diketahui dan memeriksa bagaimana kami mencapainya. Hal ini juga dapat " +"menghindari alternatif-alternatif yang telah gagal dalam konteks yang sama. " +"Titik awal ditingkatkan tanpa mengubah parameter model yang dilatih." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:447 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"When reconstructing TH09's enemy-spawn function, a {ph0}GPT-6-Luna-max " +"session{ph1} drew a hypothesis from TH08's corresponding source family. That" +" earlier source suggested how to structure the loop and handle failures " +"within it. TH09's own evidence supplied the game-specific details, and clean" +" rebuilds confirmed the complete match." +msgstr "" +"Saat merekonstruksi fungsi TH09 untuk menciptakan musuh, {ph0}sesi " +"GPT-6-Luna-max{ph1} mengambil hipotesis dari kode sumber TH08 terkait. Kode " +"sebelumnya menyarankan cara mengatur loop dan menangani kegagalan di " +"dalamnya. Bukti TH09 sendiri memberikan detail khusus game tersebut. " +"Pembangunan kembali dari sumber mengonfirmasi bahwa fungsi lengkapnya cocok." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:458 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}\"The model did not know that compiler quirk\" and \"the model could " +"not use it\" are different problems.{ph1} Retrieval may solve the first, but" +" turning that fact into a useful candidate still takes work. And if the " +"candidate fails, the model must decide what the result means for its next " +"attempt. Knowing the relevant pattern is only the beginning of that process." +msgstr "" +"{ph0}\"Model tidak mengetahui kekhasan kompiler\" dan \"model tidak dapat " +"menggunakannya\" adalah masalah yang berbeda.{ph1} Pengambilan mungkin " +"memecahkan masalah pertama, namun mengubah fakta tersebut menjadi kandidat " +"yang berguna masih membutuhkan usaha. Dan jika kandidat kode sumber gagal, " +"model harus memutuskan apa arti hasil tersebut untuk percobaan berikutnya. " +"Mengetahui pola yang relevan hanyalah awal dari proses itu." + +#: website/public/blog/reverse-engineering-models/index.html:468 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For that reason, I would include a smaller general model with good retrieval" +" in any comparison. Giving it the same library of experience would help us " +"identify what training adds: better candidate generation, better decisions " +"after feedback, or both." +msgstr "" +"Oleh karena itu, saya akan menyertakan model AI serba guna yang lebih kecil " +"dengan pengambilan yang baik dalam perbandingan apa pun. Memberikan " +"perpustakaan pengalaman yang sama akan membantu kami mengidentifikasi apa " +"saja yang ditambahkan oleh pelatihan: perolehan kandidat yang lebih baik, " +"keputusan yang lebih baik setelah adanya masukan, atau keduanya." + +#: website/public/blog/reverse-engineering-models/index.html:476 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Learning the next useful experiment" +msgstr "Pelajari eksperimen bermanfaat berikutnya" + +#: website/public/blog/reverse-engineering-models/index.html:477 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those decisions after feedback are what I would most like to study. In " +"agentic reconstruction, the model proposes an explanation and tests it " +"against the program. Even when the candidate fails, the result can narrow " +"the remaining possibilities and suggest a more useful next experiment." +msgstr "" +"Keputusan-keputusan setelah mendapat masukan adalah hal yang paling ingin " +"saya pelajari. Dalam rekonstruksi agenik, model mengajukan penjelasan dan " +"mengujinya terhadap program. Bahkan ketika kode sumber kandidat gagal, " +"hasilnya dapat mempersempit kemungkinan yang tersisa dan menyarankan " +"percobaan berikutnya yang lebih berguna." + +#: website/public/blog/reverse-engineering-models/index.html:484 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Consider the temporary-results example again. Its final source shows the " +"arrangement that worked, but the route to that source contains more " +"information. Each of the following experiments helped identify what still " +"needed to change." +msgstr "" +"Perhatikan kembali contoh hasil sementara. Sumber akhirnya menunjukkan " +"pengaturan yang berhasil, namun rute ke sumber tersebut berisi lebih banyak " +"informasi. Setiap eksperimen berikut membantu mengidentifikasi apa yang " +"masih perlu diubah." + +#: website/public/blog/reverse-engineering-models/index.html:497 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Experiment" +msgstr "Eksperimen" + +#: website/public/blog/reverse-engineering-models/index.html:498 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What its result taught us" +msgstr "Apa yang diajarkan hasilnya kepada kita" + +#: website/public/blog/reverse-engineering-models/index.html:503 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Try a different way to copy the results" +msgstr "Coba cara lain untuk menyalin hasilnya" + +#: website/public/blog/reverse-engineering-models/index.html:504 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The copy code changes, but the function still differs." +msgstr "Kode copynya berubah, tapi fungsinya tetap berbeda." + +#: website/public/blog/reverse-engineering-models/index.html:509 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Use the returned values directly" +msgstr "Gunakan nilai yang dikembalikan secara langsung" + +#: website/public/blog/reverse-engineering-models/index.html:510 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Copying follows the original pattern, but temporary storage still differs." +msgstr "" +"Penyalinan mengikuti pola aslinya, namun penyimpanan sementara tetap " +"berbeda." + +#: website/public/blog/reverse-engineering-models/index.html:516 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the temporary results alive together" +msgstr "Jaga agar semua variabel sementara tetap valid pada waktu yang sama" + +#: website/public/blog/reverse-engineering-models/index.html:517 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"They get separate storage, but one still overlaps a value used later for " +"collision checks." +msgstr "" +"Mereka mendapatkan penyimpanan terpisah, namun masih tumpang tindih dengan " +"nilai yang digunakan nanti untuk pemeriksaan tabrakan." + +#: website/public/blog/reverse-engineering-models/index.html:523 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the collision-check value separate" +msgstr "Pisahkan nilai pemeriksaan tabrakan" + +#: website/public/blog/reverse-engineering-models/index.html:524 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A clean rebuild confirms the complete match." +msgstr "" +"Kompilasi ulang yang bersih dari sumber mengonfirmasi kecocokan yang " +"lengkap." + +#: website/public/blog/reverse-engineering-models/index.html:529 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Selected experiments from the attack-function reconstruction above, grouped " +"by what they taught us." +msgstr "" +"Eksperimen yang dipilih dari rekonstruksi fungsi serangan di atas, " +"dikelompokkan berdasarkan apa yang diajarkannya kepada kita." + +#: website/public/blog/reverse-engineering-models/index.html:533 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A training record that keeps this sequence can teach more than the final " +"source alone. It connects each proposed change to an observed result and the" +" next decision. Keeping ineffective variations matters too: they tell a " +"later agent which hypotheses have already been tested in that context." +msgstr "" +"Catatan pelatihan yang menjaga urutan ini dapat mengajarkan lebih dari " +"sekedar sumber akhir saja. Ini menghubungkan setiap perubahan yang diusulkan" +" dengan hasil yang diamati dan keputusan berikutnya. Mempertahankan variasi " +"yang tidak efektif juga penting: variasi tersebut akan memberi tahu agen " +"selanjutnya hipotesis mana yang telah diuji dalam konteks tersebut." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:540 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}I would want the model to learn how to use feedback.{ph1} A neutral " +"result is a reason to reconsider the variation; a changed mismatch may point" +" to another cause. When local edits stop being informative, the agent should" +" be able to revisit the source representation or inspect the evidence again." +msgstr "" +"{ph0}Saya ingin model mempelajari cara menggunakan masukan.{ph1} Hasil yang " +"netral adalah alasan untuk mempertimbangkan kembali variasi; ketidakcocokan " +"yang berubah mungkin menunjukkan penyebab lain. Ketika suntingan lokal tidak" +" lagi bersifat informatif, agen harus dapat mengunjungi kembali representasi" +" sumber atau memeriksa bukti lagi." + +#: website/public/blog/reverse-engineering-models/index.html:549 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Learning to make those choices requires autonomy. Sometimes the useful next " +"step is inspecting another caller rather than compiling another edit. A " +"rigid sequence of required rewrites would remove that choice, along with " +"part of the behavior we want the model to learn." +msgstr "" +"Belajar membuat pilihan tersebut membutuhkan otonomi. Terkadang langkah " +"berikutnya yang berguna adalah memeriksa pemanggil lain daripada " +"mengkompilasi hasil edit lainnya. Urutan penulisan ulang yang diperlukan " +"akan menghilangkan pilihan tersebut, bersama dengan bagian dari perilaku " +"yang kita ingin model pelajari." + +#: website/public/blog/reverse-engineering-models/index.html:558 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why RE is interesting for reinforcement learning" +msgstr "Mengapa rekayasa balik menarik untuk pembelajaran penguatan" + +#: website/public/blog/reverse-engineering-models/index.html:559 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Bounded reconstruction gives us a practical environment for studying these " +"choices. Once the target and compiler are pinned, many source experiments " +"produce quick, repeatable feedback. The agent can compile a candidate, " +"inspect the mismatch and use that observation to choose its next step." +msgstr "" +"Rekonstruksi dengan lingkup yang jelas memberi lingkungan praktis untuk " +"mempelajari pilihan ini. Setelah versi target dan kompiler ditetapkan, " +"banyak eksperimen kode sumber memberi umpan balik yang cepat dan dapat " +"diulang. Agen bisa mengompilasi kandidat, memeriksa perbedaan, lalu memilih " +"langkah berikutnya." + +#: website/public/blog/reverse-engineering-models/index.html:566 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reinforcement learning could train that decision-making process through the " +"feedback it receives. A candidate accepted by a defined check gives us a " +"concrete outcome, while the sequence of attempts lets us observe how much " +"work it took to reach it." +msgstr "" +"Pembelajaran penguatan dapat melatih proses pengambilan keputusan melalui " +"umpan balik yang diterimanya. Kandidat yang diterima melalui pemeriksaan " +"tertentu memberi kita hasil yang konkrit, sementara rangkaian upaya " +"memungkinkan kita mengamati berapa banyak usaha yang diperlukan untuk " +"mencapainya." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:572 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"There is already research supporting this direction. {ph0}LLM4Decompile{ph1}" +" studies models specialized for decompilation. {ph2}AutoDecompiler{ph3} " +"trains multi-turn refinement using compiler and execution feedback. Their " +"studies focus on functional recovery. For this work, I would use them as a " +"starting point for the related task of finding source that matches a " +"historical compiler's output." +msgstr "" +"Sudah ada penelitian yang mendukung arah ini. {ph0}LLM4Decompile{ph1} " +"mempelajari model yang dikhususkan untuk dekompilasi. " +"{ph2}AutoDecompiler{ph3} melatih penyempurnaan multi-putaran menggunakan " +"compiler dan masukan eksekusi. Studi mereka fokus pada pemulihan fungsional." +" Untuk pekerjaan ini, saya akan menggunakannya sebagai titik awal untuk " +"tugas terkait menemukan sumber yang cocok dengan keluaran kompiler historis." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:584 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To reach such a well-defined task, though, we may first need to establish " +"what a function owns or devise a new runtime measurement. In TH08, for " +"example, a {ph0}collision difference led back to how the original program " +"handled floating-point calculations{ph1}. Before the agent could test a " +"useful correction, it had to work out what to measure." +msgstr "" +"Namun, untuk mencapai tugas yang terdefinisi dengan baik, pertama-tama kita " +"perlu menetapkan sumber daya yang dimiliki suatu fungsi atau merancang cara " +"baru untuk mengamati perilaku program saat berjalan. Di TH08, misalnya, " +"{ph0}perbedaan tabrakan mengarah kembali ke cara program asli menangani " +"penghitungan floating-point{ph1}. Sebelum agen dapat menguji koreksi yang " +"berguna, agen harus menentukan apa yang diukur." + +#: website/public/blog/reverse-engineering-models/index.html:595 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This suggests another role for a broadly capable model: helping make the " +"problem checkable in the first place. Once that check exists, we can " +"investigate how far a learned search policy can get with it." +msgstr "" +"Hal ini menunjukkan adanya peran lain dari model yang berkemampuan luas: " +"membantu membuat masalah dapat diperiksa. Setelah pemeriksaan tersebut " +"dilakukan, kami dapat menyelidiki sejauh mana kebijakan penelusuran yang " +"dipelajari dapat diterapkan." + +#: website/public/blog/reverse-engineering-models/index.html:603 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "When the oracle is wrong" +msgstr "Ketika pemeriksa salah" + +#: website/public/blog/reverse-engineering-models/index.html:604 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Once we train against that check, its coverage becomes part of what the " +"model learns. I call the reference or checker an oracle. Even if it gives a " +"repeatable answer, we still need to understand what that answer covers." +msgstr "" +"Saat suatu pemeriksaan dipakai untuk pelatihan, cakupannya menjadi bagian " +"dari lingkungan belajar. Saya menyebut acuan atau alat pemeriksa itu oracle." +" Bahkan jawaban yang dapat diulang harus dipahami berdasarkan apa yang " +"benar-benar diperiksanya." + +#: website/public/blog/reverse-engineering-models/index.html:610 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"TH08 showed how much can hide outside that coverage. Some reconstructed " +"functions passed their exact comparisons with wrong floating constants. The " +"comparator adjusted relocated addresses before checking instructions, but it" +" checked the values behind those addresses only when they had been " +"explicitly recorded for verification. Other literal references could pass " +"without their values being examined." +msgstr "" +"TH08 menunjukkan seberapa banyak yang bisa disembunyikan di luar cakupan " +"itu. Beberapa fungsi yang direkonstruksi memberikan perbandingan yang tepat " +"dengan konstanta floating-point yang salah. Pembanding menyesuaikan alamat " +"yang direlokasi sebelum memeriksa instruksi, namun pembanding memeriksa " +"nilai di balik alamat tersebut hanya jika nilai tersebut telah dicatat " +"secara eksplisit untuk verifikasi. Referensi literal lainnya bisa lolos " +"tanpa diperiksa nilainya." + +#. {ph0}: +#. {ph1}: +#. {ph2}: 1.0f +#. {ph3}: 32.0f +#: website/public/blog/reverse-engineering-models/index.html:619 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To close that gap, a {ph0}wider audit{ph1} found incorrect constants in " +"functions that had already been accepted. In one calculation, the " +"reconstructed source added {ph2} where the original added {ph3}." +msgstr "" +"Untuk mengatasi kesenjangan ini, {ph0}audit yang lebih luas{ph1} menemukan " +"konstanta yang salah pada fungsi yang telah lolos verifikasi. Dalam satu " +"perhitungan, kode sumber yang direkonstruksi menambahkan {ph2}, sedangkan " +"kode sumber asli menambahkan {ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:629 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A learner rewarded by the old check could learn to produce the wrong " +"reconstruction.{ph1} We had not trained an RL model against this oracle; the" +" observed failure was in the check itself. But using that check as a reward " +"would have made the same blind spot part of the learning environment." +msgstr "" +"{ph0}Model yang diberi imbalan melalui pemeriksaan verifikasi lama dapat " +"belajar menghasilkan rekonstruksi yang salah.{ph1} Kami tidak melatih model " +"RL menggunakan pemeriksa tersebut. Kegagalan yang kami amati ada pada " +"pemeriksaan itu sendiri. Namun jika kita menggunakannya sebagai hadiah, " +"titik buta yang sama akan menjadi bagian dari lingkungan belajar." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:639 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A functional test can have a similar blind spot. {ph0}Differential-fuzzing " +"research on LLM decompilers{ph1} has found behavior differences beyond the " +"supplied tests. Passing those tests therefore needs to be read in terms of " +"the cases they exercise, just as an instruction match needs to be read in " +"terms of the bytes and data it checks." +msgstr "" +"Tes fungsional dapat memiliki titik buta serupa. {ph0}Penelitian yang " +"menggunakan fuzzing diferensial pada dekompiler LLM{ph1} menemukan perbedaan" +" perilaku di luar pengujian yang disediakan. Oleh karena itu, kita perlu " +"memahami tes kelulusan dalam kaitannya dengan kasus-kasus yang dilakukannya." +" Demikian pula, kecocokan instruksi harus dipahami dalam kaitannya dengan " +"byte dan data yang diverifikasi." + +#: website/public/blog/reverse-engineering-models/index.html:649 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before using either kind of check as a reward, I would test it with " +"deliberate failures. I would also keep final acceptance outside the " +"candidate's editable source, so the model has to satisfy the check through " +"its reconstruction. Otherwise, something like a byte array copied from the " +"target could satisfy a naive score without recovering useful source." +msgstr "" +"Sebelum menggunakan salah satu jenis cek verifikasi sebagai hadiah, saya " +"akan mengujinya dengan hasil yang sengaja salah. Saya juga akan memisahkan " +"penerimaan akhir dari kode sumber yang dapat diedit oleh agen AI. Model " +"harus lolos verifikasi dengan merekonstruksi program dengan benar. Jika " +"tidak, hanya dengan menyalin array byte dari target dapat menipu sistem " +"penilaian yang naif tanpa memulihkan kode sumber yang berguna." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:657 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Improving the learner and improving its oracle belong to the same " +"research problem.{ph1} As the learner gets better at optimizing the feedback" +" we give it, understanding that feedback becomes more important too." +msgstr "" +"{ph0}Memperbaiki model yang belajar dan memperbaiki pemeriksanya merupakan " +"bagian dari masalah penelitian yang sama.{ph1} Semakin baik model " +"mengoptimalkan umpan balik yang kita berikan, semakin penting memahami umpan" +" balik itu sendiri." + +#: website/public/blog/reverse-engineering-models/index.html:667 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A comparison worth running" +msgstr "Sebuah perbandingan yang layak untuk dijalankan" + +#: website/public/blog/reverse-engineering-models/index.html:668 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"With those checks in place, I would start the comparison on a fixed VC7.1 " +"task family. First, I would establish baselines for a strong general model " +"and a smaller one. Adding repository knowledge to the smaller model would " +"show what retrieval changes. From there, supervised specialization and an " +"RL-trained version would let us examine what training adds." +msgstr "" +"Dengan pemeriksaan tersebut, saya akan memulai perbandingan pada rangkaian " +"tugas VC7.1 yang tetap. Pertama, saya akan menetapkan dasar untuk model AI " +"tujuan umum yang kuat dan model yang lebih kecil. Menambahkan pengetahuan " +"repositori ke model yang lebih kecil akan menunjukkan perubahan pengambilan " +"apa. Dari sana, spesialisasi yang diawasi dan versi terlatih RL akan " +"memungkinkan kita memeriksa apa saja yang ditambahkan dalam pelatihan." + +#: website/public/blog/reverse-engineering-models/index.html:676 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To make those comparisons informative, each model would receive the same " +"target evidence and tools. For the training-specific comparisons, I would " +"keep the base model and available knowledge fixed as well. Repeated trials " +"would help distinguish a reliable search strategy from a lucky candidate." +msgstr "" +"Agar perbandingan tersebut informatif, setiap model akan menerima bukti dan " +"alat target yang sama. Untuk perbandingan khusus pelatihan, saya juga akan " +"mempertahankan model dasar dan pengetahuan yang tersedia. Uji coba berulang " +"akan membantu membedakan strategi pencarian yang andal dari kandidat solusi " +"yang berhasil secara kebetulan." + +#: website/public/blog/reverse-engineering-models/index.html:683 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would then separate two starting conditions: reconstructing a function " +"from its instructions, and closing the last few differences in maintained " +"source. A specialist might be excellent at the second task while a general " +"model does much of the work needed to reach it. Measuring them separately " +"would reveal that division of work." +msgstr "" +"Saya kemudian akan memisahkan dua kondisi awal: merekonstruksi suatu fungsi " +"dari instruksinya, dan menutup beberapa perbedaan terakhir dalam sumber yang" +" dipertahankan. Model AI khusus mungkin sangat baik dalam tugas kedua, " +"sementara model AI tujuan umum melakukan banyak pekerjaan yang diperlukan " +"untuk mencapainya. Mengukurnya secara terpisah akan mengungkapkan pembagian " +"kerja tersebut." + +#: website/public/blog/reverse-engineering-models/index.html:691 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both tasks need genuinely held-out examples. Because related Touhou games " +"share engine code, holding out a game's name would still allow nearly " +"identical functions to appear in training. I would instead hold out source " +"families and keep their accepted answers out of both training and retrieval." +msgstr "" +"Kedua tugas memerlukan contoh uji yang tidak ada dalam data pelatihan maupun" +" bahan retrieval. Game Touhou yang terkait berbagi kode engine. " +"Mengecualikan satu game saja masih bisa menyisakan fungsi yang hampir " +"identik dalam data pelatihan. Saya akan menyisihkan seluruh kelompok kode " +"yang terkait, dan memastikan jawaban terverifikasinya tidak masuk ke data " +"pelatihan ataupun pustaka retrieval." + +#: website/public/blog/reverse-engineering-models/index.html:698 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"On those examples, the outcome would be complete, independently validated " +"matches. Giving each model the same candidate or compile budget would help " +"compare search quality. Measuring elapsed time and total cost, including " +"compilation, would tell us whether that quality produces an efficient " +"workflow. Training costs would need their own accounting too." +msgstr "" +"Untuk contoh-contoh ini, hasilnya akan berupa kecocokan yang lengkap dan " +"terverifikasi secara independen. Anggaran yang sama untuk proposal kandidat " +"atau upaya kompilasi akan membantu membandingkan kualitas pencarian. " +"Mengukur total waktu dan biaya, termasuk kompilasi, akan menunjukkan apakah " +"kualitas tersebut menghasilkan alur kerja yang efisien. Biaya pelatihan " +"harus diukur secara terpisah." + +#: website/public/blog/reverse-engineering-models/index.html:706 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both views matter because a cheaper proposal can become an expensive " +"workflow if it causes more uninformative compiles. Conversely, a slower " +"model may pay for itself by finding a useful experiment sooner. The " +"comparison should capture those costs." +msgstr "" +"Kedua pengukuran itu penting. Proposal yang murah dapat menyebabkan alur " +"kerja yang mahal jika menyebabkan lebih banyak kompilasi yang tidak " +"memberikan informasi berguna. Sebaliknya, model yang lebih lambat mungkin " +"lebih ekonomis jika model tersebut menemukan eksperimen yang berguna lebih " +"cepat. Perbandingannya harus mencakup biaya-biaya ini." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:712 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Counting accepted results also keeps intermediate progress in perspective. " +"One TH09 {ph0}replay-save candidate{ph1} looked promising because the " +"compiled function had the expected size, but a full comparison still found " +"substantial differences. Changes in how the compiler arranges instructions " +"can also make a candidate look closer without resolving the underlying " +"problem. Such diagnostics can guide the next experiment, but I would " +"validate any reward for \"getting closer\" against the full acceptance " +"check." +msgstr "" +"Menghitung hasil yang diterima juga menjaga kemajuan sementara tetap dalam " +"perspektif. Salah satu TH09 {ph0}kandidat penyimpanan ulang{ph1} tampak " +"menjanjikan karena fungsi yang dikompilasi memiliki ukuran yang diharapkan, " +"namun perbandingan penuh masih menemukan perbedaan besar. Perubahan dalam " +"cara compiler menyusun instruksi juga dapat membuat kandidat tampak lebih " +"dekat dengan target tanpa menyelesaikan masalah mendasarnya. Diagnostik " +"semacam itu dapat memandu eksperimen berikutnya, namun saya akan memvalidasi" +" imbalan apa pun karena \"mendekati\" dengan pemeriksaan penerimaan penuh." + +#: website/public/blog/reverse-engineering-models/index.html:727 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would be interested in the pattern across these results. If a specialist " +"does well within a familiar compiler family, does that advantage survive a " +"toolchain change? And once both models have good retrieval, how much does " +"training still add? The answers would help us decide where each kind of " +"model is most useful." +msgstr "" +"Saya tertarik dengan pola hasil ini. Jika model AI khusus dapat berfungsi " +"dengan baik dalam rangkaian kompiler yang sudah dikenal, apakah keunggulan " +"tersebut dapat bertahan jika terjadi perubahan rantai alat? Dan ketika kedua" +" model AI memiliki pengambilan yang baik, berapa banyak lagi yang bisa " +"ditambahkan oleh pelatihan? Jawabannya akan membantu kita memutuskan model " +"mana yang paling berguna." + +#: website/public/blog/reverse-engineering-models/index.html:736 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What kind of agent does this suggest?" +msgstr "Agen AI macam apa yang disarankan di sini?" + +#: website/public/blog/reverse-engineering-models/index.html:737 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"If the models turn out to help at different stages, a useful arrangement " +"might be a broad agent with access to a specialist. Once a task becomes a " +"familiar compiler search, it could ask the specialist for candidates. If the" +" results stop making sense, the broader agent could return to the target " +"evidence or investigate the measurement itself." +msgstr "" +"Jika model-model tersebut terbukti membantu pada tahap-tahap yang berbeda, " +"pengaturan yang berguna mungkin adalah agen luas yang memiliki akses ke " +"model AI khusus. Setelah tugas menjadi pencarian kompiler yang familier, " +"tugas tersebut dapat menanyakan kandidat pada model AI khusus. Jika hasilnya" +" tidak masuk akal, agen yang lebih luas dapat kembali ke bukti target atau " +"menyelidiki pengukuran itu sendiri." + +#: website/public/blog/reverse-engineering-models/index.html:745 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In that arrangement, specialization becomes a tool the agent can choose as " +"the investigation develops. The agent keeps the freedom to change direction " +"that made the earlier breakthroughs possible." +msgstr "" +"Dalam pengaturan tersebut, spesialisasi menjadi alat yang dapat dipilih agen" +" seiring berkembangnya penyelidikan. Agen mempunyai kebebasan untuk mengubah" +" arah yang memungkinkan terobosan sebelumnya." + +#: website/public/blog/reverse-engineering-models/index.html:750 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This follows naturally from the industrial shift I described in the previous" +" essay. Being able to run many more experiments changes how we organize the " +"work. It also gives us a reason to ask which decisions need broad judgment " +"and which improve through repeated experience with a particular craft." +msgstr "" +"Hal ini tentu saja mengikuti pergeseran industri yang saya jelaskan di esai " +"sebelumnya. Mampu menjalankan lebih banyak eksperimen mengubah cara kami " +"mengatur pekerjaan. Hal ini juga memberi kita alasan untuk mempertanyakan " +"keputusan mana yang memerlukan pertimbangan luas dan mana yang ditingkatkan " +"melalui pengalaman berulang-ulang dengan bidang tertentu." + +#: website/public/blog/reverse-engineering-models/index.html:757 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For now, the strongest general model has earned its place in my workflow by " +"opening problems I had left unresolved. Those successes are what make me " +"want to examine the work more closely: how much depended on the model's " +"breadth, and how much could be handled by a model trained in this particular" +" kind of reconstruction?" +msgstr "" +"Untuk saat ini, model umum terkuat mendapat tempat dalam alur kerja saya " +"karena menyelesaikan masalah yang sebelumnya belum terpecahkan. Keberhasilan" +" itu membuat saya ingin mempelajari prosesnya: seberapa banyak bergantung " +"pada keluasan kemampuan model, dan seberapa banyak bisa dikerjakan oleh " +"model yang dilatih khusus untuk jenis rekonstruksi ini?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:765 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}The repositories give us a way to test the question.{ph1} I would like " +"to use that record to build the comparison and see what we learn." +msgstr "" +"{ph0}Repositori memberi kita cara untuk menguji pertanyaan tersebut.{ph1} " +"Saya ingin menggunakan data tersebut untuk membuat perbandingan dan melihat " +"apa yang kita pelajari." + +#: website/public/blog/reverse-engineering-models/index.html:775 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Evidence and model attribution" +msgstr "Bukti dan atribusi model" + +#: website/public/blog/reverse-engineering-models/index.html:783 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The examples describe historical function-level comparisons recorded in the " +"linked commits. Whole-game runtime checks are separate milestones. Source " +"and matching records were reviewed on 13 October 2026." +msgstr "" +"Contoh-contoh ini menjelaskan perbandingan tingkat fungsi historis yang " +"dicatat dalam penerapan tertaut. Pemeriksaan waktu proses keseluruhan " +"permainan adalah pencapaian yang terpisah. Sumber dan catatan pencocokan " +"ditinjau pada 13 Oktober 2026." + +#. {ph0}: gpt-dots +#. {ph1}: gpt-web +#: website/public/blog/reverse-engineering-models/index.html:789 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The TH09 label {ph0} identifies GPT-6-Astra. {ph1} changed with the model " +"used at the time, so that label alone does not identify an individual run. " +"Explicit model names in the other cited commits establish their recorded " +"attribution; the histories do not contain uniform per-run cost or effort " +"measurements." +msgstr "" +"Label TH09 {ph0} berarti GPT-6-Astra. Model yang terkait dengan {ph1} " +"berubah seiring waktu, sehingga label saja tidak dapat mengidentifikasi " +"model yang digunakan dalam proses tertentu. Nama model eksplisit dalam " +"commit lain yang dikutip memberikan atribusi yang tercatat. Riwayat ini " +"tidak mencakup pengukuran biaya atau upaya yang konsisten untuk setiap " +"proses." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#: website/public/blog/reverse-engineering-models/index.html:798 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 temporary lifetimes:{ph1} the {ph2}earlier context review{ph3} and" +" {ph4}complete cold reconstruction{ph5}." +msgstr "" +"{ph0}TH09 masa hidup objek sementara:{ph1} {ph2}tinjauan konteks " +"sebelumnya{ph3} dan {ph4}rekonstruksi lengkap yang diverifikasi melalui " +"kompilasi ulang yang bersih dari sumber{ph5}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:810 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 Front callback:{ph1} {ph2}non-exact diagnostic regions{ph3}, " +"{ph4}Astra's new graph{ph5} and its {ph6}accepted recompile replay{ph7}." +msgstr "" +"{ph0}TH09 Front callback:{ph1} {ph2}wilayah diagnostik yang tidak " +"cocok{ph3}, {ph4}Astra grafik aliran kontrol baru{ph5} dan {ph6}pemeriksaan " +"kompilasi ulang yang diterima{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:826 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Compiler knowledge:{ph1} {ph2}TH095 identifier-bucket reuse{ph3}, " +"{ph4}TH04 player source and linker order{ph5} and {ph6}Luna's TH09 enemy-" +"spawn closure{ph7}." +msgstr "" +"{ph0}Pengetahuan compiler:{ph1} {ph2}TH095 penggunaan kembali perilaku " +"identifier-bucket{ph3}, {ph4}TH04 kode sumber pemain dan urutan linker{ph5} " +"dan {ph6}Luna rekonstruksi lengkap fungsi penciptaan musuh TH09{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:842 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Checks and diagnostics:{ph1} {ph2}TH08 floating-literal audit{ph3}, " +"{ph4}TH08 numerical investigation{ph5} and {ph6}TH09 replay-save " +"reconstruction{ph7}." +msgstr "" +"{ph0}Pemeriksaan dan diagnostik:{ph1} {ph2}TH08 audit konstan floating-" +"point{ph3}, {ph4}TH08 investigasi numerik{ph5} dan {ph6}TH09 rekonstruksi " +"penyimpanan ulangan{ph7}." + +#: website/public/blog/reverse-engineering-models/index.html:859 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The proposed specialist and RL comparisons are future experiments. The " +"project histories motivate them; no trained specialist was evaluated for " +"this essay." +msgstr "" +"Perbandingan yang diusulkan antara model khusus dan pelatihan RL adalah " +"eksperimen di masa depan. Sejarah proyek memotivasi mereka. Tidak ada model " +"khusus terlatih yang dievaluasi untuk esai ini." + +#: website/public/blog/reverse-engineering-models/index.html:6 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model? | REA" +msgstr "Apakah rekayasa balik memerlukan model yang lebih besar? | REA" + +#: website/public/blog/reverse-engineering-models/index.html:7 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What Touhou reconstruction taught me about general models, specialist models" +" and learning to propose better hypotheses from compiler feedback." +msgstr "" +"Apa yang saya pelajari dari rekonstruksi Touhou tentang model umum, model " +"khusus, dan cara belajar mengajukan hipotesis yang lebih baik melalui umpan " +"balik kompiler." + +#: website/public/blog/reverse-engineering-models/index.html:240 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "On this page" +msgstr "Di halaman ini" + +#: website/public/blog/reverse-engineering-models/index.html:253 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Models and reverse engineering essay" +msgstr "Model dan esai rekayasa balik" + +#: website/public/blog/reverse-engineering-models/index.html:491 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Selected reconstruction experiments" +msgstr "Eksperimen rekonstruksi yang dipilih" + +#: website/public/blog/reverse-engineering-models/index.html:883 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Back to top" +msgstr "Kembali ke atas" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:776 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Detailed evidence notes{ph1} include the recorded comparisons and " +"experiment context." +msgstr "" +"{ph0}Catatan bukti terperinci{ph1} memuat perbandingan yang dicatat dan " +"konteks eksperimen." + +#~ msgctxt "page:blog/index.html" +#~ msgid "By N0zoM1z0 · 13 October 2026 · Draft for review" +#~ msgstr "Oleh N0zoM1z0 · 13 Oktober 2026 · Draf untuk ditinjau" + #~ msgctxt "page:index.html" #~ msgid "Setup guide" #~ msgstr "Panduan pemasangan" diff --git a/website/locales/it/messages.po b/website/locales/it/messages.po index 0c1aff1c9..516e09283 100644 --- a/website/locales/it/messages.po +++ b/website/locales/it/messages.po @@ -32,18 +32,18 @@ msgid "Thoughts on reverse engineering." msgstr "Riflessioni sul reverse engineering." #. {ph0}:
-#: website/public/blog/index.html:200 +#: website/public/blog/index.html:214 #, python-brace-format msgctxt "page:blog/index.html" msgid "Game reconstruction{ph0}Essay" msgstr "Gioco ricostruzione{ph0}Saggio" -#: website/public/blog/index.html:202 +#: website/public/blog/index.html:216 msgctxt "page:blog/index.html" msgid "The industrial revolution in reverse engineering" msgstr "La rivoluzione industriale nel reverse engineering" -#: website/public/blog/index.html:203 +#: website/public/blog/index.html:217 msgctxt "page:blog/index.html" msgid "" "What manual RE and reconstructing TH08, TH095 and TH04 taught me about " @@ -52,25 +52,26 @@ msgstr "" "Che manuale RE e ricostruire TH08, TH095 e TH04 mi ha insegnato circa gli " "agenti, la verifica e la conoscenza che composti attraverso i progetti." -#: website/public/blog/index.html:208 +#: website/public/blog/index.html:207 website/public/blog/index.html:222 msgctxt "page:blog/index.html" msgid "Read the article" msgstr "Leggi l'articolo" -#: website/public/blog/index.html:211 +#: website/public/blog/index.html:225 msgctxt "page:blog/index.html" msgid "By N0zoM1z0 · 10 October 2026" msgstr "Di N0zoM1z0 * 10 ottobre 2026" #. {ph0}: -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 +#: website/public/blog/reverse-engineering-models/index.html:883 #: website/public/blog/touhou-reconstruction/index.html:1080 #: website/public/examples/dino-lab/index.html:475 #: website/public/faq/index.html:573 #: website/public/first-investigation/index.html:523 #: website/public/get-started/index.html:882 -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 #: website/public/guides/index.html:282 #: website/public/guides/javascript/index.html:503 #: website/public/guides/native/index.html:509 website/public/index.html:1191 @@ -100,6 +101,8 @@ msgstr "" "e agenti AI." #: website/public/blog/index.html:75 website/public/blog/index.html:88 +#: website/public/blog/reverse-engineering-models/index.html:75 +#: website/public/blog/reverse-engineering-models/index.html:88 #: website/public/blog/touhou-reconstruction/index.html:75 #: website/public/blog/touhou-reconstruction/index.html:88 #: website/public/examples/dino-lab/index.html:75 @@ -136,6 +139,7 @@ msgstr "" "di codifica." #: website/public/blog/index.html:108 +#: website/public/blog/reverse-engineering-models/index.html:108 #: website/public/blog/touhou-reconstruction/index.html:108 #: website/public/examples/dino-lab/index.html:109 #: website/public/faq/index.html:108 @@ -156,6 +160,7 @@ msgid "REA home" msgstr "Home di REA" #: website/public/blog/index.html:112 +#: website/public/blog/reverse-engineering-models/index.html:112 #: website/public/blog/touhou-reconstruction/index.html:112 #: website/public/examples/dino-lab/index.html:113 #: website/public/faq/index.html:112 @@ -176,6 +181,7 @@ msgid "Main navigation" msgstr "Navigazione principale" #: website/public/blog/index.html:130 +#: website/public/blog/reverse-engineering-models/index.html:130 #: website/public/blog/touhou-reconstruction/index.html:130 #: website/public/examples/dino-lab/index.html:131 #: website/public/faq/index.html:130 @@ -196,6 +202,7 @@ msgid "REA on GitHub" msgstr "REA su GitHub" #: website/public/blog/index.html:150 +#: website/public/blog/reverse-engineering-models/index.html:150 #: website/public/blog/touhou-reconstruction/index.html:150 #: website/public/examples/dino-lab/index.html:151 #: website/public/faq/index.html:150 @@ -216,6 +223,7 @@ msgid "Join REA on Discord" msgstr "Unisciti a REA su Discord" #: website/public/blog/index.html:169 +#: website/public/blog/reverse-engineering-models/index.html:169 #: website/public/blog/touhou-reconstruction/index.html:169 #: website/public/examples/dino-lab/index.html:170 #: website/public/faq/index.html:169 @@ -235,7 +243,7 @@ msgctxt "shared" msgid "Follow REA on X" msgstr "Segui REA su X" -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 msgctxt "page:blog/index.html" msgid "Back to top" msgstr "Torna in alto" @@ -4809,7 +4817,7 @@ msgctxt "page:guides/browser/index.html" msgid "Copy read the loaded scripts" msgstr "Copia leggi gli script caricati" -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 msgctxt "page:guides/browser/index.html" msgid "Back to top" msgstr "Torna in alto" @@ -11712,6 +11720,7 @@ msgid "Same spacing: 256 ÷ 16 = 16 angle units = 22.5°" msgstr "Stessa spaziatura: 256 ÷ 16 = 16 unità angolari = 22,5°" #: website/public/blog/index.html:114 +#: website/public/blog/reverse-engineering-models/index.html:114 #: website/public/blog/touhou-reconstruction/index.html:114 #: website/public/examples/dino-lab/index.html:115 #: website/public/faq/index.html:114 @@ -11732,6 +11741,7 @@ msgid "Showcases" msgstr "Casi pratici" #: website/public/blog/index.html:115 +#: website/public/blog/reverse-engineering-models/index.html:115 #: website/public/blog/touhou-reconstruction/index.html:115 #: website/public/examples/dino-lab/index.html:116 #: website/public/faq/index.html:115 @@ -11752,6 +11762,7 @@ msgid "Blog" msgstr "Blog" #: website/public/blog/index.html:113 +#: website/public/blog/reverse-engineering-models/index.html:113 #: website/public/blog/touhou-reconstruction/index.html:113 #: website/public/examples/dino-lab/index.html:114 #: website/public/faq/index.html:113 @@ -11772,6 +11783,7 @@ msgid "Get started" msgstr "Inizia" #: website/public/blog/index.html:116 +#: website/public/blog/reverse-engineering-models/index.html:116 #: website/public/blog/touhou-reconstruction/index.html:116 #: website/public/examples/dino-lab/index.html:117 #: website/public/faq/index.html:116 @@ -11791,13 +11803,14 @@ msgctxt "shared" msgid "Guides" msgstr "Guide" -#: website/public/blog/index.html:222 +#: website/public/blog/index.html:236 +#: website/public/blog/reverse-engineering-models/index.html:875 #: website/public/blog/touhou-reconstruction/index.html:1072 #: website/public/examples/dino-lab/index.html:467 #: website/public/faq/index.html:565 #: website/public/first-investigation/index.html:515 #: website/public/get-started/index.html:874 -#: website/public/guides/browser/index.html:490 +#: website/public/guides/browser/index.html:499 #: website/public/guides/index.html:274 #: website/public/guides/javascript/index.html:495 #: website/public/guides/native/index.html:501 website/public/index.html:1183 @@ -11811,13 +11824,14 @@ msgctxt "shared" msgid "Source" msgstr "Codice sorgente" -#: website/public/blog/index.html:223 +#: website/public/blog/index.html:237 +#: website/public/blog/reverse-engineering-models/index.html:876 #: website/public/blog/touhou-reconstruction/index.html:1073 #: website/public/examples/dino-lab/index.html:468 #: website/public/faq/index.html:566 #: website/public/first-investigation/index.html:516 #: website/public/get-started/index.html:875 -#: website/public/guides/browser/index.html:491 +#: website/public/guides/browser/index.html:500 #: website/public/guides/index.html:275 #: website/public/guides/javascript/index.html:496 #: website/public/guides/native/index.html:502 website/public/index.html:1184 @@ -11831,13 +11845,14 @@ msgctxt "shared" msgid "MIT license" msgstr "Licenza MIT" -#: website/public/blog/index.html:224 +#: website/public/blog/index.html:238 +#: website/public/blog/reverse-engineering-models/index.html:877 #: website/public/blog/touhou-reconstruction/index.html:1074 #: website/public/examples/dino-lab/index.html:469 #: website/public/faq/index.html:567 #: website/public/first-investigation/index.html:517 #: website/public/get-started/index.html:876 -#: website/public/guides/browser/index.html:492 +#: website/public/guides/browser/index.html:501 #: website/public/guides/index.html:276 #: website/public/guides/javascript/index.html:497 #: website/public/guides/native/index.html:503 website/public/index.html:1185 @@ -11851,13 +11866,14 @@ msgctxt "shared" msgid "Join the community" msgstr "Unisciti alla community" -#: website/public/blog/index.html:225 +#: website/public/blog/index.html:239 +#: website/public/blog/reverse-engineering-models/index.html:878 #: website/public/blog/touhou-reconstruction/index.html:1075 #: website/public/examples/dino-lab/index.html:470 #: website/public/faq/index.html:568 #: website/public/first-investigation/index.html:518 #: website/public/get-started/index.html:877 -#: website/public/guides/browser/index.html:493 +#: website/public/guides/browser/index.html:502 #: website/public/guides/index.html:277 #: website/public/guides/javascript/index.html:498 #: website/public/guides/native/index.html:504 website/public/index.html:1186 @@ -12801,6 +12817,1418 @@ msgctxt "runtime" msgid "Switch to dark mode" msgstr "Passa alla modalità scura" +#. {ph0}: +#. {ph1}: +#: website/public/guides/browser/index.html:480 +#, python-brace-format +msgctxt "page:guides/browser/index.html" +msgid "" +"For HTTP request and response bodies, use {ph0}Fiddler Everywhere: SAZ " +"captures, MCP and public skills{ph1}. Export SAZ or HAR 1.2 for offline REA " +"capture Evidence, or connect Everywhere's MCP server to inspect captured " +"traffic with your agent." +msgstr "" +"Per esaminare i corpi delle richieste e delle risposte HTTP, usa " +"{ph0}Fiddler Everywhere: acquisizioni SAZ, MCP e skill pubbliche{ph1}. " +"Esporta SAZ o HAR 1.2 per ottenere Evidence delle acquisizioni in REA " +"offline, oppure collega il server MCP di Everywhere per esaminare il " +"traffico acquisito con il tuo agente." + +#. {ph0}:
+#: website/public/blog/index.html:200 +#, python-brace-format +msgctxt "page:blog/index.html" +msgid "Model specialization{ph0}Essay" +msgstr "Modelli specializzati{ph0}Saggio" + +#: website/public/blog/index.html:202 +msgctxt "page:blog/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "Il reverse engineering ha bisogno di un modello più grande?" + +#: website/public/blog/index.html:203 +msgctxt "page:blog/index.html" +msgid "" +"What Touhou reconstruction suggests about general models, specialist models " +"and learning from compiler feedback." +msgstr "" +"Cosa suggerisce la ricostruzione di Touhou sui modelli generali, quelli " +"specializzati e l’apprendimento dal feedback del compilatore." + +#: website/public/blog/index.html:210 +msgctxt "page:blog/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Da N0zoM1z0 · 13 ottobre 2026" + +#: website/public/blog/reverse-engineering-models/index.html:103 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Skip to content" +msgstr "Vai al contenuto" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:194 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Home{ph1} / {ph2}Blog{ph3} / Models and RE" +msgstr "{ph0}Home{ph1} / {ph2}Blog{ph3} / Modelli e reverse engineering" + +#: website/public/blog/reverse-engineering-models/index.html:197 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Essay" +msgstr "Saggio" + +# Reviewed for technical meaning and readability; preserve the distinction +# between historical evidence and proposed experiments. +#: website/public/blog/reverse-engineering-models/index.html:198 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "Il reverse engineering necessita di un modello più grande?" + +#: website/public/blog/reverse-engineering-models/index.html:199 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Da N0zoM1z0 · 13 ottobre 2026" + +#: website/public/blog/reverse-engineering-models/index.html:201 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Takeaway" +msgstr "Punti chiave" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:203 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Model capability changes the hypotheses an agent can find.{ph1} Later " +"sessions have solved problems that earlier work left unresolved." +msgstr "" +"{ph0}La capacità del modello influisce sulle ipotesi che un agente AI può " +"proporre.{ph1} Le sessioni successive hanno risolto problemi che il lavoro " +"precedente aveva lasciato irrisolti." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:211 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A familiar compiler gives specialization something concrete to " +"learn.{ph1} Some exact matches depend on narrow, reusable source patterns." +msgstr "" +"{ph0}Un compilatore familiare offre modelli concreti da apprendere per un " +"modello specializzato.{ph1} Alcune corrispondenze esatte del codice macchina" +" dipendono da modelli di codice sorgente specifici e riutilizzabili." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:218 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Give repository knowledge a fair comparison.{ph1} A smaller model with " +"good retrieval may already know enough to make progress." +msgstr "" +"{ph0}Includi la conoscenza del repository in un confronto equo.{ph1} Un " +"modello più piccolo con un buon recupero della conoscenza potrebbe già avere" +" informazioni sufficienti per fare progressi." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:223 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Train the next useful experiment, and test its oracle.{ph1} The value " +"of feedback depends on what the checker actually verifies." +msgstr "" +"{ph0}Addestra il modello per scegliere il prossimo esperimento utile e " +"testare il suo strumento di verifica.{ph1} Il feedback è utile solo nella " +"misura in cui lo strumento verifica le cose giuste." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:230 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Measure accepted reconstruction against time and cost.{ph1} That is the" +" result I want from the whole agent." +msgstr "" +"{ph0}Valuta i risultati verificati della ricostruzione insieme a tempi e " +"costi.{ph1} Questo è ciò che voglio misurare per il flusso di lavoro " +"completo dell'agente AI." + +#: website/public/blog/reverse-engineering-models/index.html:241 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "In this article" +msgstr "In questo articolo" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:243 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What better models changed{ph1}" +msgstr "{ph0}Che cosa hanno cambiato i modelli più capaci{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:244 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Why specialize?{ph1}" +msgstr "{ph0}Perché specializzarsi?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:245 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Where experience lives{ph1}" +msgstr "{ph0}Dove dovrebbe essere archiviata questa esperienza?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:246 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Learning the next experiment{ph1}" +msgstr "{ph0}Imparare il prossimo esperimento utile{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:247 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}RE and reinforcement learning{ph1}" +msgstr "{ph0}Reverse engineering e apprendimento per rinforzo{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:248 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}When the oracle is wrong{ph1}" +msgstr "{ph0}Quando il controllo di verifica è sbagliato{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:249 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}A comparison worth running{ph1}" +msgstr "{ph0}Un confronto che vale la pena fare{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:250 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What kind of agent?{ph1}" +msgstr "{ph0}Che tipo di agente AI suggerisce questo?{ph1}" + +#: website/public/blog/reverse-engineering-models/index.html:258 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What better models changed" +msgstr "Che cosa hanno cambiato i modelli più capaci" + +#: website/public/blog/reverse-engineering-models/index.html:259 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Since August 2026, I have been exploring how much reverse engineering an " +"agent can carry out with access to a program and a way to check its work. " +"Most of my experience comes from reconstructing games, especially Touhou. As" +" I have changed models, the repositories have kept a record of what each " +"session managed to recover." +msgstr "" +"Dall'agosto 2026, ho esplorato quanto reverse engineering può eseguire un " +"agente con l'accesso a un programma e un modo per verificarne il lavoro. La " +"maggior parte della mia esperienza deriva dalla ricostruzione di giochi, in " +"particolare Touhou. Man mano che ho cambiato modello, i repository hanno " +"tenuto traccia di ciò che è riuscita a recuperare ciascuna sessione." + +#: website/public/blog/reverse-engineering-models/index.html:267 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of that work aims at exact reconstruction: we compile the maintained " +"source under a pinned historical toolchain and check whether it reproduces " +"the target function. When a later model succeeds where earlier work stalled," +" the source changes show what made that progress possible." +msgstr "" +"Gran parte di questo lavoro mira alla ricostruzione esatta: compiliamo il " +"sorgente mantenuto in una toolchain storica con versioni fisse e " +"controlliamo se riproduce la funzione di destinazione. Quando un modello " +"successivo riesce dove il lavoro precedente si era bloccato, le modifiche " +"alla fonte mostrano cosa ha reso possibile quel progresso." + +#: website/public/blog/reverse-engineering-models/index.html:274 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The temporary values had to stay alive together" +msgstr "Le variabili temporanee necessitavano di vite sovrapposte" + +#: website/public/blog/reverse-engineering-models/index.html:275 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of a TH09 attack function's logic had already been reconstructed, but " +"the compiler still stored temporary results differently from the original. " +"The remaining problem was how long those results needed to stay alive and " +"when their storage could be reused." +msgstr "" +"Gran parte della logica della funzione di attacco TH09 era già stata " +"ricostruita, ma il compilatore memorizzava ancora i risultati temporanei in " +"modo diverso dall'originale. Il problema rimanente era quanto tempo " +"occorreva a quei risultati per rimanere in vita e quando la loro " +"conservazione poteva essere riutilizzata." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:282 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A {ph0}GPT-6-Sol checkpoint{ph1} recorded this mismatch after testing " +"several ordinary source arrangements. When {ph2}GPT-6.1-Sol returned to the " +"problem{ph3}, it found a useful hypothesis about how long those temporary " +"values needed to stay alive." +msgstr "" +"Un {ph0}GPT-6-Sol record di sessione{ph1} ha segnalato questa mancata " +"corrispondenza dopo aver testato diverse disposizioni ordinarie del codice " +"sorgente. Quando {ph2}GPT-6.1-Sol ha esaminato nuovamente il problema{ph3}, " +"ha proposto un'ipotesi utile sulla durata di queste variabili temporanee." + +#: website/public/blog/reverse-engineering-models/index.html:296 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The function produced temporary results and copied them into animation " +"positions. In C++, a temporary result stays alive until the expression using" +" it finishes. Grouping the real operations into one expression kept those " +"results alive together, so the compiler could no longer reuse the same " +"storage for all of them." +msgstr "" +"La funzione ha prodotto risultati temporanei e li ha copiati in posizioni di" +" animazione. In C++, un risultato temporaneo rimane attivo finché " +"l'espressione che lo utilizza non termina. Raggruppare le operazioni reali " +"in un'unica espressione ha mantenuto vivi insieme i risultati, quindi il " +"compilatore non poteva più riutilizzare la stessa memoria per tutti loro." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:303 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"One result still shared storage with a value used later for collision " +"checks. Giving that value its own lifetime resolved the remaining overlap. " +"After several targeted experiments, {ph0}a clean rebuild confirmed that the " +"full function matched{ph1}." +msgstr "" +"Un risultato ha ancora spazio di archiviazione condiviso con un valore " +"utilizzato successivamente per i controlli di collisione. Dare a quel valore" +" la propria durata ha risolto la rimanente sovrapposizione. Dopo diversi " +"esperimenti mirati, {ph0}una ricompilazione pulita dal sorgente ha " +"confermato che l'intera funzione corrispondeva a{ph1}." + +#: website/public/blog/reverse-engineering-models/index.html:312 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What interests me is how the discovery and the tests worked together. The " +"lifetime hypothesis gave the agent a new direction; compiler feedback helped" +" it refine that hypothesis into a source arrangement that matched." +msgstr "" +"Sono interessato a come la scoperta e i test si siano supportati a vicenda. " +"L’ipotesi sulla durata temporanea degli oggetti ha dato all’agente AI una " +"nuova direzione. Il feedback del compilatore ha aiutato a perfezionare " +"quell'ipotesi in una disposizione del codice sorgente che corrispondesse." + +#: website/public/blog/reverse-engineering-models/index.html:318 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A different control-flow graph" +msgstr "Un diverso grafico del flusso di controllo" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:319 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A second TH09 function shows why that new direction can matter. Here, the " +"obstacle was how several branches shared their final work. " +"{ph0}GPT-6.1-Sol's earlier work{ph1} got parts of the function to match, but" +" the branches still joined up differently from the original." +msgstr "" +"Una seconda funzione TH09 mostra perché questa nuova direzione può essere " +"importante. In questo caso, l’ostacolo era il modo in cui diversi rami " +"condividevano il loro lavoro finale. Il lavoro precedente di " +"{ph0}GPT-6.1-Sol{ph1} faceva corrispondere parti della funzione, ma i rami " +"si univano ancora in modo diverso dall'originale." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:330 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}GPT-6-Astra changed the surrounding control-flow graph{ph1}. The main " +"paths now led to a shared final update and return. In that source structure," +" the compiler combined the branches in the same way as the original. The " +"complete function matched, and a {ph2}separate rebuild check confirmed " +"it{ph3}." +msgstr "" +"{ph0}GPT-6-Astra ha cambiato il grafico del flusso di controllo " +"circostante{ph1}. I percorsi principali conducono ora ad un aggiornamento " +"finale condiviso e al ritorno. In quella struttura sorgente, il compilatore " +"ha combinato i rami nello stesso modo dell'originale. La funzione completa " +"corrispondeva e un {ph2}controllo di ricostruzione separato lo ha " +"confermato{ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:342 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Changing the representation opened a more productive search space.{ph1}" +" Both cases give me a reason to take general model capability seriously: an " +"agent can keep trying local variations for a long time before finding a " +"better way to express the surrounding work." +msgstr "" +"{ph0}La modifica della rappresentazione ha aperto uno spazio di ricerca più " +"produttivo.{ph1} Entrambi i casi mi danno un motivo per prendere sul serio " +"le capacità generali del modello: un agente può continuare a provare " +"variazioni locali per molto tempo prima di trovare un modo migliore per " +"esprimere il lavoro circostante." + +#: website/public/blog/reverse-engineering-models/index.html:351 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"At the same time, the later sessions started with earlier source and " +"recorded failures. That accumulated work was part of their advantage. The " +"history lets us inspect the discoveries and their checks; a controlled " +"comparison would help separate the effect of the model from the experience " +"it inherited." +msgstr "" +"Le sessioni successive hanno anche ereditato il codice precedente e i " +"tentativi falliti già documentati. Questo lavoro accumulato faceva parte del" +" loro vantaggio. La cronologia permette di esaminare le scoperte e le " +"verifiche; un confronto controllato aiuterebbe a separare il contributo del " +"modello da quello dell’esperienza ereditata." + +#: website/public/blog/reverse-engineering-models/index.html:358 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is where the question becomes interesting to me. Some of the progress " +"came from finding a new representation, while much of the remaining work " +"involved testing source forms against a familiar compiler. How much needs " +"broad reasoning ability, and how much could a model learn through repeated " +"experience with that compiler and code?" +msgstr "" +"Questo è ciò che rende la domanda interessante per me. Alcuni progressi sono" +" arrivati ​​dalla ricerca di una nuova rappresentanza. Gran parte del lavoro" +" rimanente prevedeva il test di diverse disposizioni del codice sorgente " +"rispetto a un compilatore familiare. Quanto di tutto ciò richiede un’ampia " +"capacità di ragionamento? Quanto potrebbe imparare un modello lavorando " +"ripetutamente con quel compilatore e quel codice?" + +#: website/public/blog/reverse-engineering-models/index.html:368 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why might a specialist do well?" +msgstr "Perché un modello di IA specializzato potrebbe funzionare bene?" + +#: website/public/blog/reverse-engineering-models/index.html:369 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Touhou gives us a concrete reason to expect that experience to help. Much of" +" the game code comes from the same author, ZUN, so recurring habits are a " +"reasonable starting assumption. The repositories back that up with useful " +"examples: related loops and helper arrangements have supplied hypotheses " +"across games." +msgstr "" +"Touhou ci dà una ragione concreta per aspettarci che quell'esperienza sia " +"d'aiuto. Gran parte del codice del gioco proviene dallo stesso autore, ZUN, " +"quindi le abitudini ricorrenti sono un presupposto di partenza ragionevole. " +"I repository lo confermano con esempi utili: cicli correlati e disposizioni " +"di supporto hanno fornito ipotesi attraverso i giochi." + +#: website/public/blog/reverse-engineering-models/index.html:376 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those source patterns become more specific when we include the compiler. " +"Under a particular version and set of options, small changes in source can " +"produce repeatable differences in machine code. Some of those differences " +"are surprisingly obscure." +msgstr "" +"Questi modelli di origine diventano più specifici quando includiamo il " +"compilatore. In una particolare versione e insieme di opzioni, piccole " +"modifiche all'origine possono produrre differenze ripetibili nel codice " +"macchina. Alcune di queste differenze sono sorprendentemente oscure." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: Supervisor::LoadDat +#. {ph4}: +#: website/public/blog/reverse-engineering-models/index.html:382 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In TH095, for example, {ph0}even the name used for a local variable could " +"change where VC7.1 stored it{ph1}. Moving the declaration alone did not " +"solve the layout, but an earlier background function had revealed a naming " +"pattern that affected the compiler's choice. That gave us a more promising " +"hypothesis for {ph2}the reconstruction of {ph3}{ph4}." +msgstr "" +"In TH095, ad esempio, {ph0}anche il nome di una variabile locale potrebbe " +"influenzare dove VC7.1 l'ha memorizzata{ph1}. Lo spostamento della sua " +"dichiarazione non ha risolto il layout di archiviazione. Tuttavia, una " +"precedente funzione in background aveva rivelato un modello di denominazione" +" che influenzava la scelta del compilatore. Ciò ha suggerito un'ipotesi più " +"promettente per {ph2}ricostruire {ph3}{ph4}." + +#: website/public/blog/reverse-engineering-models/index.html:396 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reusing that pattern for the file-size variable produced the required layout" +" without adding storage. The full function then matched." +msgstr "" +"Il riutilizzo di quel modello per la variabile della dimensione del file ha " +"prodotto il layout richiesto senza aggiungere spazio di archiviazione. La " +"funzione completa quindi corrispondeva." + +#: website/public/blog/reverse-engineering-models/index.html:401 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is the sort of narrow detail a general model may not know when it " +"arrives. A specialist trained on this compiler could encounter many such " +"cases and learn which source variations are worth testing first." +msgstr "" +"Questo è un dettaglio molto specifico che un modello di intelligenza " +"artificiale generica potrebbe non conoscere quando incontra per la prima " +"volta il problema. Un modello specializzato per questo compilatore potrebbe " +"vedere molti casi simili e apprendere quali variazioni del codice sorgente " +"vale la pena testare per prime." + +#: website/public/blog/reverse-engineering-models/index.html:407 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The opportunity is especially clear in final matching problems where we " +"already understand the behavior. We are then asking which ordinary source " +"form this compiler turns into the observed instructions. Specialization " +"could help the model propose those forms more efficiently, while the target " +"still determines which hypotheses survive." +msgstr "" +"L'opportunità è particolarmente chiara nei problemi di abbinamento finale in" +" cui ne comprendiamo già il comportamento. Ci stiamo quindi chiedendo quale " +"disposizione ordinaria del codice sorgente questo compilatore trasforma " +"nelle istruzioni osservate. La specializzazione potrebbe aiutare il modello " +"a proporre tali forme in modo più efficiente, mentre l’obiettivo determina " +"ancora quali ipotesi sopravvivono." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:415 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The useful patterns would have to retain their context, though. TH08 uses an" +" earlier Visual C++ version than TH095 and TH09. TH04 uses a different DOS-" +"era compiler, and its {ph0}player reconstruction{ph1} needed different " +"hypotheses about source patterns and how the compiled code was combined into" +" an executable. Shared authorship gives us a prior, but each target tells us" +" whether it applies. Distinguishing those contexts is part of what a " +"specialist would need to learn." +msgstr "" +"Gli schemi utili devono conservare il loro contesto. TH08 usa una versione " +"di Visual C++ precedente a quella di TH095 e TH09. TH04 usa un altro " +"compilatore dell’epoca DOS. La {ph0}ricostruzione del codice del " +"giocatore{ph1} richiedeva ipotesi diverse sugli schemi del sorgente e sul " +"collegamento del codice in un eseguibile. Lo stesso autore ci dà " +"un’aspettativa iniziale, ma ogni obiettivo indica se è applicabile. Un " +"modello specializzato dovrebbe imparare a distinguere questi contesti." + +#: website/public/blog/reverse-engineering-models/index.html:431 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Where should that experience live?" +msgstr "Dove dovrebbe essere archiviata questa esperienza?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:432 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before deciding to teach those patterns through model training, I would look" +" at the knowledge we already keep in the repository. In the {ph0}previous " +"essay{ph1}, I wrote about it becoming the project's working memory. That " +"memory includes compiler lessons, along with the source and failed " +"experiments that made them useful." +msgstr "" +"Prima di decidere di insegnare questi modelli attraverso l'addestramento dei" +" modelli, esaminerei la conoscenza che già conserviamo nel repository. Nel " +"{ph0}saggio precedente{ph1} ho scritto che diventa la memoria di lavoro del " +"progetto. Quella memoria include lezioni del compilatore, insieme alla fonte" +" e agli esperimenti falliti che li hanno resi utili." + +#: website/public/blog/reverse-engineering-models/index.html:441 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"By retrieving that record, an agent can start with a known result and " +"inspect how we reached it. It can also avoid alternatives that have already " +"failed in the same context. The starting point improves without changing the" +" model's weights." +msgstr "" +"Recuperando quel record, un agente può iniziare con un risultato noto e " +"verificare come lo abbiamo raggiunto. Può anche evitare alternative che " +"hanno già fallito nello stesso contesto. Il punto di partenza migliora senza" +" modificare i parametri addestrati del modello." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:447 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"When reconstructing TH09's enemy-spawn function, a {ph0}GPT-6-Luna-max " +"session{ph1} drew a hypothesis from TH08's corresponding source family. That" +" earlier source suggested how to structure the loop and handle failures " +"within it. TH09's own evidence supplied the game-specific details, and clean" +" rebuilds confirmed the complete match." +msgstr "" +"Durante la ricostruzione della funzione di TH09 per la creazione di nemici, " +"una sessione {ph0}GPT-6-Luna-max{ph1} ha preso un'ipotesi dal relativo " +"codice sorgente TH08. Il codice precedente suggeriva come organizzare il " +"ciclo e gestire gli errori al suo interno. Le prove di TH09 hanno fornito i " +"dettagli specifici del gioco. Le ricostruzioni dall'origine hanno confermato" +" che la funzione completa corrispondeva." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:458 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}\"The model did not know that compiler quirk\" and \"the model could " +"not use it\" are different problems.{ph1} Retrieval may solve the first, but" +" turning that fact into a useful candidate still takes work. And if the " +"candidate fails, the model must decide what the result means for its next " +"attempt. Knowing the relevant pattern is only the beginning of that process." +msgstr "" +"{ph0}\"Il modello non conosceva quella stranezza del compilatore\" e \"il " +"modello non poteva usarlo\" sono problemi diversi.{ph1} Il recupero può " +"risolvere il primo, ma trasformare questo fatto in un candidato utile " +"richiede ancora lavoro. E se il codice sorgente candidato fallisce, il " +"modello deve decidere cosa significherà il risultato per il suo prossimo " +"tentativo. Conoscere il modello rilevante è solo l’inizio di quel processo." + +#: website/public/blog/reverse-engineering-models/index.html:468 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For that reason, I would include a smaller general model with good retrieval" +" in any comparison. Giving it the same library of experience would help us " +"identify what training adds: better candidate generation, better decisions " +"after feedback, or both." +msgstr "" +"Per questo motivo, includerei in ogni confronto un modello di intelligenza " +"artificiale generico più piccolo con un buon recupero. Fornire la stessa " +"libreria di esperienze ci aiuterebbe a identificare ciò che la formazione " +"aggiunge: una migliore generazione di candidati, decisioni migliori dopo il " +"feedback o entrambi." + +#: website/public/blog/reverse-engineering-models/index.html:476 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Learning the next useful experiment" +msgstr "Imparare il prossimo esperimento utile" + +#: website/public/blog/reverse-engineering-models/index.html:477 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those decisions after feedback are what I would most like to study. In " +"agentic reconstruction, the model proposes an explanation and tests it " +"against the program. Even when the candidate fails, the result can narrow " +"the remaining possibilities and suggest a more useful next experiment." +msgstr "" +"Quelle decisioni dopo il feedback sono ciò che mi piacerebbe di più " +"studiare. Nella ricostruzione agentica, il modello propone una spiegazione e" +" la mette alla prova rispetto al programma. Anche quando il codice sorgente " +"candidato fallisce, il risultato può restringere le possibilità rimanenti e " +"suggerire un prossimo esperimento più utile." + +#: website/public/blog/reverse-engineering-models/index.html:484 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Consider the temporary-results example again. Its final source shows the " +"arrangement that worked, but the route to that source contains more " +"information. Each of the following experiments helped identify what still " +"needed to change." +msgstr "" +"Consideriamo nuovamente l'esempio dei risultati temporanei. La sua fonte " +"finale mostra la soluzione che ha funzionato, ma il percorso verso quella " +"fonte contiene più informazioni. Ciascuno dei seguenti esperimenti ha " +"contribuito a identificare ciò che ancora doveva cambiare." + +#: website/public/blog/reverse-engineering-models/index.html:497 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Experiment" +msgstr "Esperimento" + +#: website/public/blog/reverse-engineering-models/index.html:498 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What its result taught us" +msgstr "Cosa ci ha insegnato il suo risultato" + +#: website/public/blog/reverse-engineering-models/index.html:503 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Try a different way to copy the results" +msgstr "Prova un modo diverso per copiare i risultati" + +#: website/public/blog/reverse-engineering-models/index.html:504 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The copy code changes, but the function still differs." +msgstr "Il codice di copia cambia, ma la funzione cambia ancora." + +#: website/public/blog/reverse-engineering-models/index.html:509 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Use the returned values directly" +msgstr "Utilizzare direttamente i valori restituiti" + +#: website/public/blog/reverse-engineering-models/index.html:510 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Copying follows the original pattern, but temporary storage still differs." +msgstr "" +"La copia segue lo schema originale, ma l'archiviazione temporanea differisce" +" ancora." + +#: website/public/blog/reverse-engineering-models/index.html:516 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the temporary results alive together" +msgstr "Mantieni valide tutte le variabili temporanee contemporaneamente" + +#: website/public/blog/reverse-engineering-models/index.html:517 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"They get separate storage, but one still overlaps a value used later for " +"collision checks." +msgstr "" +"Ottengono un'archiviazione separata, ma uno si sovrappone comunque a un " +"valore utilizzato successivamente per i controlli di collisione." + +#: website/public/blog/reverse-engineering-models/index.html:523 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the collision-check value separate" +msgstr "Mantenere separato il valore del controllo collisioni" + +#: website/public/blog/reverse-engineering-models/index.html:524 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A clean rebuild confirms the complete match." +msgstr "" +"Una ricompilazione pulita dalla fonte conferma la corrispondenza completa." + +#: website/public/blog/reverse-engineering-models/index.html:529 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Selected experiments from the attack-function reconstruction above, grouped " +"by what they taught us." +msgstr "" +"Esperimenti selezionati dalla ricostruzione della funzione di attacco sopra," +" raggruppati in base a ciò che ci hanno insegnato." + +#: website/public/blog/reverse-engineering-models/index.html:533 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A training record that keeps this sequence can teach more than the final " +"source alone. It connects each proposed change to an observed result and the" +" next decision. Keeping ineffective variations matters too: they tell a " +"later agent which hypotheses have already been tested in that context." +msgstr "" +"Un registro dell'addestramento che mantenga questa sequenza può insegnare " +"più della sola fonte finale. Collega ogni modifica proposta a un risultato " +"osservato e alla decisione successiva. Anche mantenere le variazioni " +"inefficaci è importante: dicono a un agente successivo quali ipotesi sono " +"già state testate in quel contesto." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:540 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}I would want the model to learn how to use feedback.{ph1} A neutral " +"result is a reason to reconsider the variation; a changed mismatch may point" +" to another cause. When local edits stop being informative, the agent should" +" be able to revisit the source representation or inspect the evidence again." +msgstr "" +"{ph0}Vorrei che il modello imparasse come utilizzare il feedback.{ph1} Un " +"risultato neutro è un motivo per riconsiderare la variazione; una mancata " +"corrispondenza modificata può indicare un'altra causa. Quando le modifiche " +"locali smettono di essere informative, l'agente dovrebbe essere in grado di " +"rivisitare la rappresentazione originale o ispezionare nuovamente le prove." + +#: website/public/blog/reverse-engineering-models/index.html:549 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Learning to make those choices requires autonomy. Sometimes the useful next " +"step is inspecting another caller rather than compiling another edit. A " +"rigid sequence of required rewrites would remove that choice, along with " +"part of the behavior we want the model to learn." +msgstr "" +"Imparare a fare queste scelte richiede autonomia. A volte il passaggio " +"successivo utile è ispezionare un altro chiamante anziché compilare un'altra" +" modifica. Una rigida sequenza di riscritture richieste eliminerebbe quella " +"scelta, insieme a parte del comportamento che vogliamo che il modello " +"impari." + +#: website/public/blog/reverse-engineering-models/index.html:558 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why RE is interesting for reinforcement learning" +msgstr "" +"Perché il reverse engineering è interessante per l'apprendimento per " +"rinforzo" + +#: website/public/blog/reverse-engineering-models/index.html:559 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Bounded reconstruction gives us a practical environment for studying these " +"choices. Once the target and compiler are pinned, many source experiments " +"produce quick, repeatable feedback. The agent can compile a candidate, " +"inspect the mismatch and use that observation to choose its next step." +msgstr "" +"La ricostruzione limitata ci offre un ambiente pratico per studiare queste " +"scelte. Una volta fissate le versioni di destinazione e del compilatore, " +"molti esperimenti di origine producono un feedback rapido e ripetibile. " +"L'agente può compilare un candidato, ispezionare la mancata corrispondenza e" +" utilizzare tale osservazione per scegliere il passaggio successivo." + +#: website/public/blog/reverse-engineering-models/index.html:566 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reinforcement learning could train that decision-making process through the " +"feedback it receives. A candidate accepted by a defined check gives us a " +"concrete outcome, while the sequence of attempts lets us observe how much " +"work it took to reach it." +msgstr "" +"L’apprendimento per rinforzo potrebbe addestrare il processo decisionale " +"attraverso il feedback che riceve. Un candidato accettato con una verifica " +"definita ci dà un risultato concreto, mentre la sequenza dei tentativi ci " +"permette di osservare quanto lavoro è stato necessario per raggiungerlo." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:572 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"There is already research supporting this direction. {ph0}LLM4Decompile{ph1}" +" studies models specialized for decompilation. {ph2}AutoDecompiler{ph3} " +"trains multi-turn refinement using compiler and execution feedback. Their " +"studies focus on functional recovery. For this work, I would use them as a " +"starting point for the related task of finding source that matches a " +"historical compiler's output." +msgstr "" +"Esistono già ricerche a sostegno di questa direzione. " +"{ph0}LLM4Decompile{ph1} studia modelli specializzati per la decompilazione. " +"{ph2}AutoDecompiler{ph3} addestra il perfezionamento multi-turn utilizzando " +"il feedback del compilatore e dell'esecuzione. I loro studi si concentrano " +"sul recupero funzionale. Per questo lavoro, li userei come punto di partenza" +" per il compito correlato di trovare fonti che corrispondano all'output di " +"un compilatore storico." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:584 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To reach such a well-defined task, though, we may first need to establish " +"what a function owns or devise a new runtime measurement. In TH08, for " +"example, a {ph0}collision difference led back to how the original program " +"handled floating-point calculations{ph1}. Before the agent could test a " +"useful correction, it had to work out what to measure." +msgstr "" +"Per raggiungere un compito così ben definito, tuttavia, potremmo prima dover" +" stabilire quali risorse possiede una funzione o ideare un nuovo modo di " +"osservare il comportamento del programma durante l’esecuzione. In TH08, ad " +"esempio, una {ph0}differenza di collisione riportava al modo in cui il " +"programma originale gestiva i calcoli in virgola mobile{ph1}. Prima che " +"l'agente potesse testare una correzione utile, doveva capire cosa misurare." + +#: website/public/blog/reverse-engineering-models/index.html:595 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This suggests another role for a broadly capable model: helping make the " +"problem checkable in the first place. Once that check exists, we can " +"investigate how far a learned search policy can get with it." +msgstr "" +"Ciò suggerisce un altro ruolo per un modello ampiamente capace: in primo " +"luogo, contribuire a rendere il problema controllabile. Una volta effettuato" +" questo controllo, possiamo indagare fino a che punto può arrivare una " +"politica di ricerca appresa." + +#: website/public/blog/reverse-engineering-models/index.html:603 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "When the oracle is wrong" +msgstr "Quando il controllo di verifica è sbagliato" + +#: website/public/blog/reverse-engineering-models/index.html:604 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Once we train against that check, its coverage becomes part of what the " +"model learns. I call the reference or checker an oracle. Even if it gives a " +"repeatable answer, we still need to understand what that answer covers." +msgstr "" +"Quando una verifica viene usata per l’addestramento, anche la sua copertura " +"entra nell’ambiente di apprendimento. Chiamo oracle quel riferimento o " +"verificatore. Persino una risposta ripetibile va interpretata in base a ciò " +"che verifica davvero." + +#: website/public/blog/reverse-engineering-models/index.html:610 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"TH08 showed how much can hide outside that coverage. Some reconstructed " +"functions passed their exact comparisons with wrong floating constants. The " +"comparator adjusted relocated addresses before checking instructions, but it" +" checked the values behind those addresses only when they had been " +"explicitly recorded for verification. Other literal references could pass " +"without their values being examined." +msgstr "" +"TH08 ha mostrato quanto si può nascondere al di fuori di quella copertura. " +"Alcune funzioni ricostruite hanno superato i confronti esatti con costanti " +"in virgola mobile errate. Il comparatore aggiustava gli indirizzi trasferiti" +" prima di controllare le istruzioni, ma controllava i valori dietro quegli " +"indirizzi solo quando erano stati esplicitamente registrati per la verifica." +" Altri riferimenti letterali potrebbero passare senza che i loro valori " +"vengano esaminati." + +#. {ph0}: +#. {ph1}: +#. {ph2}: 1.0f +#. {ph3}: 32.0f +#: website/public/blog/reverse-engineering-models/index.html:619 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To close that gap, a {ph0}wider audit{ph1} found incorrect constants in " +"functions that had already been accepted. In one calculation, the " +"reconstructed source added {ph2} where the original added {ph3}." +msgstr "" +"Per colmare questa lacuna, un {ph0}controllo più ampio{ph1} ha rilevato " +"costanti errate nelle funzioni che avevano già superato la verifica. In un " +"calcolo, il codice sorgente ricostruito aggiungeva {ph2}, mentre l'originale" +" aggiungeva {ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:629 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A learner rewarded by the old check could learn to produce the wrong " +"reconstruction.{ph1} We had not trained an RL model against this oracle; the" +" observed failure was in the check itself. But using that check as a reward " +"would have made the same blind spot part of the learning environment." +msgstr "" +"{ph0}Un modello premiato dal vecchio controllo di verifica potrebbe imparare" +" a produrre una ricostruzione errata.{ph1} Non abbiamo addestrato un modello" +" RL utilizzando quel controllo. L'errore che abbiamo osservato riguardava il" +" controllo stesso. Ma se lo usassimo come ricompensa, lo stesso punto cieco " +"diventerebbe parte dell’ambiente di apprendimento." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:639 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A functional test can have a similar blind spot. {ph0}Differential-fuzzing " +"research on LLM decompilers{ph1} has found behavior differences beyond the " +"supplied tests. Passing those tests therefore needs to be read in terms of " +"the cases they exercise, just as an instruction match needs to be read in " +"terms of the bytes and data it checks." +msgstr "" +"Un test funzionale può avere un punto cieco simile. {ph0}La ricerca che " +"utilizza il fuzzing differenziale sui decompilatori LLM{ph1} ha rilevato " +"differenze comportamentali al di fuori dei test forniti. Dobbiamo quindi " +"comprendere una prova di superamento in termini dei casi che esercita. Allo " +"stesso modo, la corrispondenza di un'istruzione deve essere intesa in " +"termini di byte e dati che verifica." + +#: website/public/blog/reverse-engineering-models/index.html:649 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before using either kind of check as a reward, I would test it with " +"deliberate failures. I would also keep final acceptance outside the " +"candidate's editable source, so the model has to satisfy the check through " +"its reconstruction. Otherwise, something like a byte array copied from the " +"target could satisfy a naive score without recovering useful source." +msgstr "" +"Prima di utilizzare uno dei due tipi di controllo di verifica come " +"ricompensa, lo testerei con risultati deliberatamente errati. Vorrei anche " +"mantenere l'accettazione finale separata dal codice sorgente che l'agente AI" +" può modificare. Il modello deve superare la verifica ricostruendo " +"correttamente il programma. Altrimenti, la semplice copia di un array di " +"byte dalla destinazione potrebbe ingannare un ingenuo sistema di punteggio " +"senza recuperare codice sorgente utile." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:657 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Improving the learner and improving its oracle belong to the same " +"research problem.{ph1} As the learner gets better at optimizing the feedback" +" we give it, understanding that feedback becomes more important too." +msgstr "" +"{ph0}Migliorare lo studente e migliorare il suo strumento di verifica " +"appartengono allo stesso problema di ricerca.{ph1} Man mano che lo studente " +"migliora nell'ottimizzare il feedback che gli forniamo, anche comprendere " +"quel feedback diventa più importante." + +#: website/public/blog/reverse-engineering-models/index.html:667 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A comparison worth running" +msgstr "Un confronto che vale la pena fare" + +#: website/public/blog/reverse-engineering-models/index.html:668 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"With those checks in place, I would start the comparison on a fixed VC7.1 " +"task family. First, I would establish baselines for a strong general model " +"and a smaller one. Adding repository knowledge to the smaller model would " +"show what retrieval changes. From there, supervised specialization and an " +"RL-trained version would let us examine what training adds." +msgstr "" +"Con questi controlli in atto, inizierei il confronto su una famiglia di " +"attività VC7.1 fissa. Innanzitutto, stabilirei le linee di base per un forte" +" modello di intelligenza artificiale generale e uno più piccolo. L'aggiunta " +"della conoscenza del repository al modello più piccolo mostrerebbe quali " +"cambiamenti nel recupero. Da lì, la specializzazione supervisionata e una " +"versione addestrata con RL ci permetterebbero di esaminare cosa aggiunge la " +"formazione." + +#: website/public/blog/reverse-engineering-models/index.html:676 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To make those comparisons informative, each model would receive the same " +"target evidence and tools. For the training-specific comparisons, I would " +"keep the base model and available knowledge fixed as well. Repeated trials " +"would help distinguish a reliable search strategy from a lucky candidate." +msgstr "" +"Per rendere informativi questi confronti, ciascun modello riceverebbe le " +"stesse prove e strumenti target. Per i confronti specifici della formazione," +" manterrei fisso anche il modello base e le conoscenze disponibili. Prove " +"ripetute aiuterebbero a distinguere una strategia di ricerca affidabile da " +"una soluzione candidata che ha successo per caso." + +#: website/public/blog/reverse-engineering-models/index.html:683 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would then separate two starting conditions: reconstructing a function " +"from its instructions, and closing the last few differences in maintained " +"source. A specialist might be excellent at the second task while a general " +"model does much of the work needed to reach it. Measuring them separately " +"would reveal that division of work." +msgstr "" +"Separerei quindi due condizioni iniziali: ricostruire una funzione dalle sue" +" istruzioni e chiudere le ultime differenze nel sorgente mantenuto. Un " +"modello di IA specializzato potrebbe essere eccellente per il secondo " +"compito, mentre un modello di IA generico svolge gran parte del lavoro " +"necessario per raggiungerlo. Misurarli separatamente rivelerebbe quella " +"divisione del lavoro." + +#: website/public/blog/reverse-engineering-models/index.html:691 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both tasks need genuinely held-out examples. Because related Touhou games " +"share engine code, holding out a game's name would still allow nearly " +"identical functions to appear in training. I would instead hold out source " +"families and keep their accepted answers out of both training and retrieval." +msgstr "" +"Entrambi i compiti necessitano di esempi di test esclusi dall'addestramento " +"e dal recupero. I giochi Touhou correlati condividono il codice motore. " +"Escludere un gioco dall'allenamento potrebbe comunque lasciare funzioni " +"quasi identiche nel set di allenamento. Invece, escluderei interi gruppi di " +"codice sorgente correlato e manterrei le loro risposte verificate fuori sia " +"dai dati di addestramento che dalla libreria di recupero." + +#: website/public/blog/reverse-engineering-models/index.html:698 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"On those examples, the outcome would be complete, independently validated " +"matches. Giving each model the same candidate or compile budget would help " +"compare search quality. Measuring elapsed time and total cost, including " +"compilation, would tell us whether that quality produces an efficient " +"workflow. Training costs would need their own accounting too." +msgstr "" +"Per questi esempi, il risultato sarebbe corrispondenze complete e verificate" +" in modo indipendente. Budget uguali per le proposte dei candidati o i " +"tentativi di compilazione aiuterebbero a confrontare la qualità della " +"ricerca. La misurazione del tempo e dei costi totali, inclusa la " +"compilazione, mostrerebbe se tale qualità produce un flusso di lavoro " +"efficiente. I costi di formazione dovrebbero essere misurati separatamente." + +#: website/public/blog/reverse-engineering-models/index.html:706 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both views matter because a cheaper proposal can become an expensive " +"workflow if it causes more uninformative compiles. Conversely, a slower " +"model may pay for itself by finding a useful experiment sooner. The " +"comparison should capture those costs." +msgstr "" +"Entrambe le misurazioni contano. Una proposta economica può portare a un " +"flusso di lavoro costoso se comporta più compilazioni che non forniscono " +"informazioni utili. Al contrario, un modello più lento può essere più " +"economico se trova prima un esperimento utile. Il confronto dovrebbe " +"includere questi costi." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:712 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Counting accepted results also keeps intermediate progress in perspective. " +"One TH09 {ph0}replay-save candidate{ph1} looked promising because the " +"compiled function had the expected size, but a full comparison still found " +"substantial differences. Changes in how the compiler arranges instructions " +"can also make a candidate look closer without resolving the underlying " +"problem. Such diagnostics can guide the next experiment, but I would " +"validate any reward for \"getting closer\" against the full acceptance " +"check." +msgstr "" +"Il conteggio dei risultati accettati mantiene inoltre in prospettiva i " +"progressi intermedi. Un TH09 {ph0}candidato al replay-save{ph1} sembrava " +"promettente perché la funzione compilata aveva le dimensioni previste, ma un" +" confronto completo ha comunque rilevato differenze sostanziali. I " +"cambiamenti nel modo in cui il compilatore organizza le istruzioni possono " +"anche far sembrare un candidato più vicino al risultato atteso senza " +"risolvere il problema di fondo. Tale diagnostica può guidare il prossimo " +"esperimento, ma convaliderei qualsiasi ricompensa per \"avvicinarsi\" " +"rispetto al controllo completo di accettazione." + +#: website/public/blog/reverse-engineering-models/index.html:727 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would be interested in the pattern across these results. If a specialist " +"does well within a familiar compiler family, does that advantage survive a " +"toolchain change? And once both models have good retrieval, how much does " +"training still add? The answers would help us decide where each kind of " +"model is most useful." +msgstr "" +"Sarei interessato allo schema tra questi risultati. Se un modello di " +"intelligenza artificiale specializzato funziona bene all’interno di una " +"famiglia di compilatori familiare, tale vantaggio sopravvive a un " +"cambiamento nella toolchain? E una volta che entrambi i modelli di " +"intelligenza artificiale hanno un buon recupero, quanto aggiunge ancora " +"l’addestramento? Le risposte ci aiuterebbero a decidere dove ciascun tipo di" +" modello è più utile." + +#: website/public/blog/reverse-engineering-models/index.html:736 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What kind of agent does this suggest?" +msgstr "Che tipo di agente AI suggerisce questo?" + +#: website/public/blog/reverse-engineering-models/index.html:737 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"If the models turn out to help at different stages, a useful arrangement " +"might be a broad agent with access to a specialist. Once a task becomes a " +"familiar compiler search, it could ask the specialist for candidates. If the" +" results stop making sense, the broader agent could return to the target " +"evidence or investigate the measurement itself." +msgstr "" +"Se i modelli si rivelano utili in diverse fasi, una soluzione utile potrebbe" +" essere un ampio agente con accesso a un modello di intelligenza artificiale" +" specializzato. Una volta che un'attività diventa una ricerca familiare del " +"compilatore, potrebbe richiedere il modello di intelligenza artificiale " +"specializzato per i candidati. Se i risultati smettono di avere senso, " +"l’agente più ampio potrebbe tornare all’evidenza target o indagare sulla " +"misurazione stessa." + +#: website/public/blog/reverse-engineering-models/index.html:745 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In that arrangement, specialization becomes a tool the agent can choose as " +"the investigation develops. The agent keeps the freedom to change direction " +"that made the earlier breakthroughs possible." +msgstr "" +"In questo modo, la specializzazione diventa uno strumento che l’agente può " +"scegliere man mano che l’indagine si sviluppa. L'agente mantiene la libertà " +"di cambiare direzione che ha reso possibili i progressi precedenti." + +#: website/public/blog/reverse-engineering-models/index.html:750 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This follows naturally from the industrial shift I described in the previous" +" essay. Being able to run many more experiments changes how we organize the " +"work. It also gives us a reason to ask which decisions need broad judgment " +"and which improve through repeated experience with a particular craft." +msgstr "" +"Ciò deriva naturalmente dal cambiamento industriale che ho descritto nel " +"saggio precedente. Essere in grado di eseguire molti più esperimenti cambia " +"il modo in cui organizziamo il lavoro. Ci dà anche una ragione per chiederci" +" quali decisioni necessitano di un ampio giudizio e quali migliorano " +"attraverso l’esperienza ripetuta con una particolare imbarcazione." + +#: website/public/blog/reverse-engineering-models/index.html:757 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For now, the strongest general model has earned its place in my workflow by " +"opening problems I had left unresolved. Those successes are what make me " +"want to examine the work more closely: how much depended on the model's " +"breadth, and how much could be handled by a model trained in this particular" +" kind of reconstruction?" +msgstr "" +"Per ora il modello generale più forte si è guadagnato il suo posto nel mio " +"lavoro risolvendo problemi che avevo lasciato aperti. Sono proprio questi " +"successi a farmi studiare il processo: quanto dipendeva dall’ampiezza delle " +"sue capacità, e quanto potrebbe fare un modello addestrato specificamente " +"per questo tipo di ricostruzione?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:765 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}The repositories give us a way to test the question.{ph1} I would like " +"to use that record to build the comparison and see what we learn." +msgstr "" +"{ph0}I repository ci offrono un modo per testare la domanda.{ph1} Vorrei " +"utilizzare quel record per creare il confronto e vedere cosa impariamo." + +#: website/public/blog/reverse-engineering-models/index.html:775 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Evidence and model attribution" +msgstr "Evidenze e attribuzione del modello" + +#: website/public/blog/reverse-engineering-models/index.html:783 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The examples describe historical function-level comparisons recorded in the " +"linked commits. Whole-game runtime checks are separate milestones. Source " +"and matching records were reviewed on 13 October 2026." +msgstr "" +"Gli esempi descrivono i confronti storici a livello di funzione registrati " +"nei commit collegati. I controlli di runtime dell'intero gioco sono " +"traguardi separati. I record di origine e corrispondenza sono stati " +"esaminati il 13 ottobre 2026." + +#. {ph0}: gpt-dots +#. {ph1}: gpt-web +#: website/public/blog/reverse-engineering-models/index.html:789 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The TH09 label {ph0} identifies GPT-6-Astra. {ph1} changed with the model " +"used at the time, so that label alone does not identify an individual run. " +"Explicit model names in the other cited commits establish their recorded " +"attribution; the histories do not contain uniform per-run cost or effort " +"measurements." +msgstr "" +"L'etichetta TH09 {ph0} significa GPT-6-Astra. Il modello associato a {ph1} è" +" cambiato nel tempo, quindi l'etichetta da sola non può identificare il " +"modello utilizzato in una specifica esecuzione. I nomi espliciti dei modelli" +" negli altri commit citati forniscono l'attribuzione registrata. Queste " +"cronologie non includono misurazioni coerenti dei costi o dell'impegno per " +"ogni esecuzione." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#: website/public/blog/reverse-engineering-models/index.html:798 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 temporary lifetimes:{ph1} the {ph2}earlier context review{ph3} and" +" {ph4}complete cold reconstruction{ph5}." +msgstr "" +"{ph0}TH09 durata degli oggetti temporanei:{ph1} la {ph2}revisione del " +"contesto precedente{ph3} e la {ph4}ricostruzione completa verificata da una " +"ricompilazione pulita dall'origine{ph5}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:810 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 Front callback:{ph1} {ph2}non-exact diagnostic regions{ph3}, " +"{ph4}Astra's new graph{ph5} and its {ph6}accepted recompile replay{ph7}." +msgstr "" +"{ph0}TH09 Front callback:{ph1} {ph2}aree diagnostiche che non " +"corrispondevano{ph3}, il nuovo grafico del flusso di controllo di " +"{ph4}Astra{ph5} e il relativo {ph6}controllo di ricompilazione " +"accettato{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:826 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Compiler knowledge:{ph1} {ph2}TH095 identifier-bucket reuse{ph3}, " +"{ph4}TH04 player source and linker order{ph5} and {ph6}Luna's TH09 enemy-" +"spawn closure{ph7}." +msgstr "" +"{ph0}Conoscenze del compilatore:{ph1} {ph2}TH095 riutilizzo del " +"comportamento del bucket di identificatori{ph3}, {ph4}TH04 codice sorgente " +"del giocatore e ordine dei linker{ph5} e {ph6}Luna ricostruzione completa " +"della funzione di creazione del nemico di TH09{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:842 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Checks and diagnostics:{ph1} {ph2}TH08 floating-literal audit{ph3}, " +"{ph4}TH08 numerical investigation{ph5} and {ph6}TH09 replay-save " +"reconstruction{ph7}." +msgstr "" +"{ph0}Controlli e diagnostica:{ph1} {ph2}TH08 audit costante in virgola " +"mobile{ph3}, {ph4}TH08 indagine numerica{ph5} e {ph6}TH09 ricostruzione con " +"salvataggio replay{ph7}." + +#: website/public/blog/reverse-engineering-models/index.html:859 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The proposed specialist and RL comparisons are future experiments. The " +"project histories motivate them; no trained specialist was evaluated for " +"this essay." +msgstr "" +"I confronti proposti tra modelli specializzati e la formazione RL sono " +"esperimenti futuri. Le storie dei progetti li motivano. Per questo saggio " +"non è stato valutato alcun modello specializzato addestrato." + +#: website/public/blog/reverse-engineering-models/index.html:6 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model? | REA" +msgstr "Il reverse engineering necessita di un modello più grande? | REA" + +#: website/public/blog/reverse-engineering-models/index.html:7 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What Touhou reconstruction taught me about general models, specialist models" +" and learning to propose better hypotheses from compiler feedback." +msgstr "" +"Cosa mi ha insegnato la ricostruzione Touhou sui modelli generali, sui " +"modelli specialistici e sull'imparare a proporre ipotesi migliori dal " +"feedback del compilatore." + +#: website/public/blog/reverse-engineering-models/index.html:240 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "On this page" +msgstr "In questa pagina" + +#: website/public/blog/reverse-engineering-models/index.html:253 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Models and reverse engineering essay" +msgstr "Modelli e saggio di reverse engineering" + +#: website/public/blog/reverse-engineering-models/index.html:491 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Selected reconstruction experiments" +msgstr "Esperimenti di ricostruzione selezionati" + +#: website/public/blog/reverse-engineering-models/index.html:883 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Back to top" +msgstr "Torna in alto" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:776 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Detailed evidence notes{ph1} include the recorded comparisons and " +"experiment context." +msgstr "" +"Le {ph0}note dettagliate sulle prove{ph1} includono i confronti registrati e" +" il contesto degli esperimenti." + +#~ msgctxt "page:blog/index.html" +#~ msgid "By N0zoM1z0 · 13 October 2026 · Draft for review" +#~ msgstr "Di N0zoM1z0 · 13 ottobre 2026 · Bozza da rivedere" + #~ msgctxt "page:index.html" #~ msgid "Setup guide" #~ msgstr "Guida all’installazione" diff --git a/website/locales/ja/messages.po b/website/locales/ja/messages.po index 8f23848c6..17d784abd 100644 --- a/website/locales/ja/messages.po +++ b/website/locales/ja/messages.po @@ -32,18 +32,18 @@ msgid "Thoughts on reverse engineering." msgstr "リバースエンジニアリングに関する考え。" #. {ph0}:
-#: website/public/blog/index.html:200 +#: website/public/blog/index.html:214 #, python-brace-format msgctxt "page:blog/index.html" msgid "Game reconstruction{ph0}Essay" msgstr "ゲーム再構築{ph0}小論文" -#: website/public/blog/index.html:202 +#: website/public/blog/index.html:216 msgctxt "page:blog/index.html" msgid "The industrial revolution in reverse engineering" msgstr "リバースエンジニアリングにおける産業革命" -#: website/public/blog/index.html:203 +#: website/public/blog/index.html:217 msgctxt "page:blog/index.html" msgid "" "What manual RE and reconstructing TH08, TH095 and TH04 taught me about " @@ -51,25 +51,26 @@ msgid "" msgstr "" "どのようなマニュアル再とTH08を再構築,TH095とTH04は、エージェントについて教えてくれました,プロジェクト間で化合物の検証と知識." -#: website/public/blog/index.html:208 +#: website/public/blog/index.html:207 website/public/blog/index.html:222 msgctxt "page:blog/index.html" msgid "Read the article" msgstr "記事を読む" -#: website/public/blog/index.html:211 +#: website/public/blog/index.html:225 msgctxt "page:blog/index.html" msgid "By N0zoM1z0 · 10 October 2026" msgstr "によってN0zoM1z0•10 10月2026" #. {ph0}: -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 +#: website/public/blog/reverse-engineering-models/index.html:883 #: website/public/blog/touhou-reconstruction/index.html:1080 #: website/public/examples/dino-lab/index.html:475 #: website/public/faq/index.html:573 #: website/public/first-investigation/index.html:523 #: website/public/get-started/index.html:882 -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 #: website/public/guides/index.html:282 #: website/public/guides/javascript/index.html:503 #: website/public/guides/native/index.html:509 website/public/index.html:1191 @@ -97,6 +98,8 @@ msgid "" msgstr "REAとAIエージェントの実践的な経験に基づいて、リバースエンジニアリングに関する考え。" #: website/public/blog/index.html:75 website/public/blog/index.html:88 +#: website/public/blog/reverse-engineering-models/index.html:75 +#: website/public/blog/reverse-engineering-models/index.html:88 #: website/public/blog/touhou-reconstruction/index.html:75 #: website/public/blog/touhou-reconstruction/index.html:88 #: website/public/examples/dino-lab/index.html:75 @@ -131,6 +134,7 @@ msgid "" msgstr "REA—ソフトウェアがどのように動作するかを調べます。 コーディングエージェントとのリバースエンジニアリング。" #: website/public/blog/index.html:108 +#: website/public/blog/reverse-engineering-models/index.html:108 #: website/public/blog/touhou-reconstruction/index.html:108 #: website/public/examples/dino-lab/index.html:109 #: website/public/faq/index.html:108 @@ -151,6 +155,7 @@ msgid "REA home" msgstr "REA ホーム" #: website/public/blog/index.html:112 +#: website/public/blog/reverse-engineering-models/index.html:112 #: website/public/blog/touhou-reconstruction/index.html:112 #: website/public/examples/dino-lab/index.html:113 #: website/public/faq/index.html:112 @@ -171,6 +176,7 @@ msgid "Main navigation" msgstr "メインナビゲーション" #: website/public/blog/index.html:130 +#: website/public/blog/reverse-engineering-models/index.html:130 #: website/public/blog/touhou-reconstruction/index.html:130 #: website/public/examples/dino-lab/index.html:131 #: website/public/faq/index.html:130 @@ -191,6 +197,7 @@ msgid "REA on GitHub" msgstr "GitHub で REA を見る" #: website/public/blog/index.html:150 +#: website/public/blog/reverse-engineering-models/index.html:150 #: website/public/blog/touhou-reconstruction/index.html:150 #: website/public/examples/dino-lab/index.html:151 #: website/public/faq/index.html:150 @@ -211,6 +218,7 @@ msgid "Join REA on Discord" msgstr "REA の Discord に参加" #: website/public/blog/index.html:169 +#: website/public/blog/reverse-engineering-models/index.html:169 #: website/public/blog/touhou-reconstruction/index.html:169 #: website/public/examples/dino-lab/index.html:170 #: website/public/faq/index.html:169 @@ -230,7 +238,7 @@ msgctxt "shared" msgid "Follow REA on X" msgstr "X で REA をフォロー" -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 msgctxt "page:blog/index.html" msgid "Back to top" msgstr "ページ上部へ" @@ -4440,7 +4448,7 @@ msgctxt "page:guides/browser/index.html" msgid "Copy read the loaded scripts" msgstr "ロードされたスクリプトをコピーして読みます" -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 msgctxt "page:guides/browser/index.html" msgid "Back to top" msgstr "ページ上部へ" @@ -10886,6 +10894,7 @@ msgid "Same spacing: 256 ÷ 16 = 16 angle units = 22.5°" msgstr "同じ間隔:256÷16=16角度単位=22.5°" #: website/public/blog/index.html:114 +#: website/public/blog/reverse-engineering-models/index.html:114 #: website/public/blog/touhou-reconstruction/index.html:114 #: website/public/examples/dino-lab/index.html:115 #: website/public/faq/index.html:114 @@ -10906,6 +10915,7 @@ msgid "Showcases" msgstr "事例" #: website/public/blog/index.html:115 +#: website/public/blog/reverse-engineering-models/index.html:115 #: website/public/blog/touhou-reconstruction/index.html:115 #: website/public/examples/dino-lab/index.html:116 #: website/public/faq/index.html:115 @@ -10926,6 +10936,7 @@ msgid "Blog" msgstr "ブログ" #: website/public/blog/index.html:113 +#: website/public/blog/reverse-engineering-models/index.html:113 #: website/public/blog/touhou-reconstruction/index.html:113 #: website/public/examples/dino-lab/index.html:114 #: website/public/faq/index.html:113 @@ -10946,6 +10957,7 @@ msgid "Get started" msgstr "はじめる" #: website/public/blog/index.html:116 +#: website/public/blog/reverse-engineering-models/index.html:116 #: website/public/blog/touhou-reconstruction/index.html:116 #: website/public/examples/dino-lab/index.html:117 #: website/public/faq/index.html:116 @@ -10965,13 +10977,14 @@ msgctxt "shared" msgid "Guides" msgstr "ガイド" -#: website/public/blog/index.html:222 +#: website/public/blog/index.html:236 +#: website/public/blog/reverse-engineering-models/index.html:875 #: website/public/blog/touhou-reconstruction/index.html:1072 #: website/public/examples/dino-lab/index.html:467 #: website/public/faq/index.html:565 #: website/public/first-investigation/index.html:515 #: website/public/get-started/index.html:874 -#: website/public/guides/browser/index.html:490 +#: website/public/guides/browser/index.html:499 #: website/public/guides/index.html:274 #: website/public/guides/javascript/index.html:495 #: website/public/guides/native/index.html:501 website/public/index.html:1183 @@ -10985,13 +10998,14 @@ msgctxt "shared" msgid "Source" msgstr "ソースコード" -#: website/public/blog/index.html:223 +#: website/public/blog/index.html:237 +#: website/public/blog/reverse-engineering-models/index.html:876 #: website/public/blog/touhou-reconstruction/index.html:1073 #: website/public/examples/dino-lab/index.html:468 #: website/public/faq/index.html:566 #: website/public/first-investigation/index.html:516 #: website/public/get-started/index.html:875 -#: website/public/guides/browser/index.html:491 +#: website/public/guides/browser/index.html:500 #: website/public/guides/index.html:275 #: website/public/guides/javascript/index.html:496 #: website/public/guides/native/index.html:502 website/public/index.html:1184 @@ -11005,13 +11019,14 @@ msgctxt "shared" msgid "MIT license" msgstr "MIT ライセンス" -#: website/public/blog/index.html:224 +#: website/public/blog/index.html:238 +#: website/public/blog/reverse-engineering-models/index.html:877 #: website/public/blog/touhou-reconstruction/index.html:1074 #: website/public/examples/dino-lab/index.html:469 #: website/public/faq/index.html:567 #: website/public/first-investigation/index.html:517 #: website/public/get-started/index.html:876 -#: website/public/guides/browser/index.html:492 +#: website/public/guides/browser/index.html:501 #: website/public/guides/index.html:276 #: website/public/guides/javascript/index.html:497 #: website/public/guides/native/index.html:503 website/public/index.html:1185 @@ -11025,13 +11040,14 @@ msgctxt "shared" msgid "Join the community" msgstr "コミュニティに参加" -#: website/public/blog/index.html:225 +#: website/public/blog/index.html:239 +#: website/public/blog/reverse-engineering-models/index.html:878 #: website/public/blog/touhou-reconstruction/index.html:1075 #: website/public/examples/dino-lab/index.html:470 #: website/public/faq/index.html:568 #: website/public/first-investigation/index.html:518 #: website/public/get-started/index.html:877 -#: website/public/guides/browser/index.html:493 +#: website/public/guides/browser/index.html:502 #: website/public/guides/index.html:277 #: website/public/guides/javascript/index.html:498 #: website/public/guides/native/index.html:504 website/public/index.html:1186 @@ -11947,6 +11963,1187 @@ msgctxt "runtime" msgid "Switch to dark mode" msgstr "ダークモードに切り替える" +#. {ph0}: +#. {ph1}: +#: website/public/guides/browser/index.html:480 +#, python-brace-format +msgctxt "page:guides/browser/index.html" +msgid "" +"For HTTP request and response bodies, use {ph0}Fiddler Everywhere: SAZ " +"captures, MCP and public skills{ph1}. Export SAZ or HAR 1.2 for offline REA " +"capture Evidence, or connect Everywhere's MCP server to inspect captured " +"traffic with your agent." +msgstr "" +"HTTP リクエストとレスポンスの本文を調べるには、{ph0}Fiddler Everywhere:SAZ " +"キャプチャ、MCP、公開スキル{ph1}を利用してください。SAZ ??? HAR 1.2 をエクスポートすれば REA でオフラインのキャプチャ " +"Evidence を取得できます。また、Everywhere の MCP サーバーを接続して、エージェントと一緒に記録済みの通信を調べることもできます。" + +#. {ph0}:
+#: website/public/blog/index.html:200 +#, python-brace-format +msgctxt "page:blog/index.html" +msgid "Model specialization{ph0}Essay" +msgstr "モデルの特化{ph0}エッセイ" + +#: website/public/blog/index.html:202 +msgctxt "page:blog/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "リバースエンジニアリングに、より大きなモデルは必要か?" + +#: website/public/blog/index.html:203 +msgctxt "page:blog/index.html" +msgid "" +"What Touhou reconstruction suggests about general models, specialist models " +"and learning from compiler feedback." +msgstr "Touhou の再構築を通じて考える、汎用モデル、特化モデル、そしてコンパイラのフィードバックから学ぶ方法。" + +#: website/public/blog/index.html:210 +msgctxt "page:blog/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "N0zoM1z0 著 · 2026 年 10 月 13 日" + +#: website/public/blog/reverse-engineering-models/index.html:103 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Skip to content" +msgstr "コンテンツにスキップ" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:194 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Home{ph1} / {ph2}Blog{ph3} / Models and RE" +msgstr "{ph0}ホーム{ph1} / {ph2}ブログ{ph3} / モデルとリバース エンジニアリング" + +#: website/public/blog/reverse-engineering-models/index.html:197 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Essay" +msgstr "エッセイ" + +# Reviewed for technical meaning and readability; preserve the distinction +# between historical evidence and proposed experiments. +#: website/public/blog/reverse-engineering-models/index.html:198 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "リバースエンジニアリングに、より大きなモデルは必要か?" + +#: website/public/blog/reverse-engineering-models/index.html:199 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "N0zoM1z0 著 · 2026 年 10 月 13 日" + +#: website/public/blog/reverse-engineering-models/index.html:201 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Takeaway" +msgstr "要点" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:203 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Model capability changes the hypotheses an agent can find.{ph1} Later " +"sessions have solved problems that earlier work left unresolved." +msgstr "" +"{ph0}モデルの能力によって、エージェントが見つけられる仮説は変わる。{ph1} 後の作業では、それまで解決できなかった問題が解決された。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:211 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A familiar compiler gives specialization something concrete to " +"learn.{ph1} Some exact matches depend on narrow, reusable source patterns." +msgstr "" +"{ph0}使い慣れたコンパイラには、特化モデルが学べる具体的なパターンがある。{ph1} " +"完全一致の中には、細かな、再利用可能なソースコードのパターンに依存するものもある。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:218 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Give repository knowledge a fair comparison.{ph1} A smaller model with " +"good retrieval may already know enough to make progress." +msgstr "" +"{ph0}リポジトリに蓄積した知識も、公平に比較したい。{ph1} 小さなモデルでも、適切に知識を検索できれば、作業を進めるのに十分かもしれない。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:223 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Train the next useful experiment, and test its oracle.{ph1} The value " +"of feedback depends on what the checker actually verifies." +msgstr "" +"{ph0}次の有用な実験を選ぶ力を学習させ、その検証器も確かめる。{ph1} フィードバックの価値は、検証器が実際に何を確認しているかで決まる。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:230 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Measure accepted reconstruction against time and cost.{ph1} That is the" +" result I want from the whole agent." +msgstr "" +"{ph0}検証を通った再構築の成果を、時間とコストと合わせて評価する。{ph1} エージェントのワークフロー全体で測りたいのは、その結果だ。" + +#: website/public/blog/reverse-engineering-models/index.html:241 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "In this article" +msgstr "この記事の内容" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:243 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What better models changed{ph1}" +msgstr "{ph0}強いモデルで何が変わったか{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:244 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Why specialize?{ph1}" +msgstr "{ph0}なぜ特化を考えるのか{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:245 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Where experience lives{ph1}" +msgstr "{ph0}その経験はどこに蓄積すべきか?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:246 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Learning the next experiment{ph1}" +msgstr "{ph0}次の有用な実験を選ぶ力を学ぶ{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:247 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}RE and reinforcement learning{ph1}" +msgstr "{ph0}リバース エンジニアリングと強化学習{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:248 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}When the oracle is wrong{ph1}" +msgstr "{ph0}検証器が間違っていたら{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:249 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}A comparison worth running{ph1}" +msgstr "{ph0}実行する価値のある比較{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:250 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What kind of agent?{ph1}" +msgstr "{ph0}どんなエージェントが必要なのか?{ph1}" + +#: website/public/blog/reverse-engineering-models/index.html:258 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What better models changed" +msgstr "強いモデルで何が変わったか" + +#: website/public/blog/reverse-engineering-models/index.html:259 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Since August 2026, I have been exploring how much reverse engineering an " +"agent can carry out with access to a program and a way to check its work. " +"Most of my experience comes from reconstructing games, especially Touhou. As" +" I have changed models, the repositories have kept a record of what each " +"session managed to recover." +msgstr "" +"2026年8月から、プログラムにアクセスし、自分の作業を検証できるエージェントが、どこまでリバースエンジニアリングを進められるかを探ってきた。私の経験は主にゲーム、特に" +" Touhou(東方)の再構築から得たものだ。モデルを変えるたびに、各セッションで何を復元できたかがリポジトリに残った。" + +#: website/public/blog/reverse-engineering-models/index.html:267 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of that work aims at exact reconstruction: we compile the maintained " +"source under a pinned historical toolchain and check whether it reproduces " +"the target function. When a later model succeeds where earlier work stalled," +" the source changes show what made that progress possible." +msgstr "" +"その多くは完全一致を目指す再構築だ。バージョンを固定した当時のツールチェーンで、保守しているソースをコンパイルし、対象の関数を再現できるか確かめる。後のモデルが以前の行き詰まりを解消したとき、ソースの変更を見ると、何が進歩を可能にしたかがわかる。" + +#: website/public/blog/reverse-engineering-models/index.html:274 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The temporary values had to stay alive together" +msgstr "一時的な値を同時に有効にしておく必要があった" + +#: website/public/blog/reverse-engineering-models/index.html:275 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of a TH09 attack function's logic had already been reconstructed, but " +"the compiler still stored temporary results differently from the original. " +"The remaining problem was how long those results needed to stay alive and " +"when their storage could be reused." +msgstr "" +"TH09 " +"のある攻撃関数は、ロジックの大部分が復元済みだった。それでも、コンパイラが一時的な結果を格納する方法は原版と違っていた。残る問題は、その結果がいつまで有効であるべきか、そして格納領域をいつ再利用できるかだった。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:282 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A {ph0}GPT-6-Sol checkpoint{ph1} recorded this mismatch after testing " +"several ordinary source arrangements. When {ph2}GPT-6.1-Sol returned to the " +"problem{ph3}, it found a useful hypothesis about how long those temporary " +"values needed to stay alive." +msgstr "" +"いくつかの通常のソースコード配置をテストした後、{ph0}GPT-6-Sol セッション レコード{ph1} でこの不一致が報告されました。 " +"{ph2}GPT-6.1-Sol がこの問題を再度調査したとき{ph3}、これらの一時変数の存続期間に関する有用な仮説が提案されました。" + +#: website/public/blog/reverse-engineering-models/index.html:296 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The function produced temporary results and copied them into animation " +"positions. In C++, a temporary result stays alive until the expression using" +" it finishes. Grouping the real operations into one expression kept those " +"results alive together, so the compiler could no longer reuse the same " +"storage for all of them." +msgstr "" +"この関数は一時的な結果を作り、アニメーションの位置にコピーする。C++ " +"の一時的な結果は、それを使う式が終わるまで生存する。実際の処理を一つの式にまとめると、結果の生存期間が重なり、コンパイラはすべての結果に同じ格納領域を使い回せなくなる。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:303 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"One result still shared storage with a value used later for collision " +"checks. Giving that value its own lifetime resolved the remaining overlap. " +"After several targeted experiments, {ph0}a clean rebuild confirmed that the " +"full function matched{ph1}." +msgstr "" +"一つの結果だけは、後の衝突判定に使う値と領域を共有していた。その値に独立した生存期間を与えると、最後の重なりも解消された。いくつかの絞り込んだ実験の後、{ph0}クリーンビルドで関数全体の一致が確認された{ph1}。" + +#: website/public/blog/reverse-engineering-models/index.html:312 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What interests me is how the discovery and the tests worked together. The " +"lifetime hypothesis gave the agent a new direction; compiler feedback helped" +" it refine that hypothesis into a source arrangement that matched." +msgstr "" +"発見とテストがどのように相互にサポートし合っていたのかに興味があります。一時オブジェクトの存続期間に関する仮説は、AI " +"エージェントに新しい方向性を与えました。コンパイラーからのフィードバックにより、その仮説を改良して、一致するソースコードの配置を実現することができました。" + +#: website/public/blog/reverse-engineering-models/index.html:318 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A different control-flow graph" +msgstr "別の制御フロー グラフ" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:319 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A second TH09 function shows why that new direction can matter. Here, the " +"obstacle was how several branches shared their final work. " +"{ph0}GPT-6.1-Sol's earlier work{ph1} got parts of the function to match, but" +" the branches still joined up differently from the original." +msgstr "" +"2 番目の TH09 関数は、その新しい方向が重要である理由を示しています。ここでの障害は、複数のブランチが最終的な作業をどのように共有するかでした。 " +"{ph0}GPT-6.1-Sol の以前の作業{ph1} " +"では、関数の一部が一致していましたが、分岐の結合方法は依然としてオリジナルとは異なっていました。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:330 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}GPT-6-Astra changed the surrounding control-flow graph{ph1}. The main " +"paths now led to a shared final update and return. In that source structure," +" the compiler combined the branches in the same way as the original. The " +"complete function matched, and a {ph2}separate rebuild check confirmed " +"it{ph3}." +msgstr "" +"{ph0}GPT-6-Astra が周囲の制御フロー グラフを変更しました{ph1}。メイン " +"パスは共有された最終更新と復帰につながりました。そのソース構造では、コンパイラはオリジナルと同じ方法でブランチを結合しました。完全な関数は一致しており、{ph2}別のリビルド" +" チェックでそれが確認されました{ph3}。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:342 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Changing the representation opened a more productive search space.{ph1}" +" Both cases give me a reason to take general model capability seriously: an " +"agent can keep trying local variations for a long time before finding a " +"better way to express the surrounding work." +msgstr "" +"{ph0}表現を変えたことで、より有望な探索空間が開けた。{ph1} " +"この二つの例から、汎用モデルの能力は重要だと感じる。エージェントは局所的な変更を長く試した末に、周囲の処理を表すもっとよい方法を見つけることがある。" + +#: website/public/blog/reverse-engineering-models/index.html:351 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"At the same time, the later sessions started with earlier source and " +"recorded failures. That accumulated work was part of their advantage. The " +"history lets us inspect the discoveries and their checks; a controlled " +"comparison would help separate the effect of the model from the experience " +"it inherited." +msgstr "" +"ただし、後のセッションは、それまでのソースと失敗の記録を引き継いでいる。その蓄積も有利に働いた。履歴から発見と検証の過程は調べられるが、モデルの効果と引き継いだ経験の効果を分けるには、条件をそろえた比較が必要だ。" + +#: website/public/blog/reverse-engineering-models/index.html:358 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is where the question becomes interesting to me. Some of the progress " +"came from finding a new representation, while much of the remaining work " +"involved testing source forms against a familiar compiler. How much needs " +"broad reasoning ability, and how much could a model learn through repeated " +"experience with that compiler and code?" +msgstr "" +"これが私にとってこの質問が興味深い理由です。新しい表現を見つけることである程度の進歩が得られました。残りの作業の多くには、使い慣れたコンパイラに対してさまざまなソース" +" " +"コードの配置をテストすることが含まれていました。このうちどの程度の範囲で幅広い推論能力が必要でしょうか?モデルはそのコンパイラーとコードを繰り返し操作することでどれだけ学習できるでしょうか?" + +#: website/public/blog/reverse-engineering-models/index.html:368 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why might a specialist do well?" +msgstr "特化モデルがうまく働く可能性は?" + +#: website/public/blog/reverse-engineering-models/index.html:369 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Touhou gives us a concrete reason to expect that experience to help. Much of" +" the game code comes from the same author, ZUN, so recurring habits are a " +"reasonable starting assumption. The repositories back that up with useful " +"examples: related loops and helper arrangements have supplied hypotheses " +"across games." +msgstr "" +"Touhou には、経験の蓄積が役立つと考える具体的な理由がある。多くのゲームコードを ZUN " +"一人が書いているため、繰り返し現れる書き方を最初の仮説にできる。実際、リポジトリには、似たループや補助関数の構成が別のゲームの仮説につながった例がある。" + +#: website/public/blog/reverse-engineering-models/index.html:376 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those source patterns become more specific when we include the compiler. " +"Under a particular version and set of options, small changes in source can " +"produce repeatable differences in machine code. Some of those differences " +"are surprisingly obscure." +msgstr "" +"コンパイラを含めると、これらのソース " +"パターンがより具体的になります。特定のバージョンとオプションのセットでは、ソースに小さな変更を加えると、マシンコードに再現性のある差異が生じる可能性があります。これらの違いの中には、驚くほどわかりにくいものもあります。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: Supervisor::LoadDat +#. {ph4}: +#: website/public/blog/reverse-engineering-models/index.html:382 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In TH095, for example, {ph0}even the name used for a local variable could " +"change where VC7.1 stored it{ph1}. Moving the declaration alone did not " +"solve the layout, but an earlier background function had revealed a naming " +"pattern that affected the compiler's choice. That gave us a more promising " +"hypothesis for {ph2}the reconstruction of {ph3}{ph4}." +msgstr "" +"たとえば、TH095 では、{ph0}ローカル変数の名前であっても、VC7.1 " +"がそれを格納する場所に影響を与える可能性があります{ph1}。その宣言を移動してもストレージ " +"レイアウトは解決されませんでした。ただし、以前のバックグラウンド関数では、コンパイラの選択に影響を与える名前付けパターンが明らかになっていました。これは、{ph2}{ph3}{ph4}" +" を再構築するためのより有望な仮説を示唆しました。" + +#: website/public/blog/reverse-engineering-models/index.html:396 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reusing that pattern for the file-size variable produced the required layout" +" without adding storage. The full function then matched." +msgstr "そのパターンをファイルサイズの変数に使うと、格納領域を増やさずに必要な配置が得られた。続いて関数全体が一致した。" + +#: website/public/blog/reverse-engineering-models/index.html:401 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is the sort of narrow detail a general model may not know when it " +"arrives. A specialist trained on this compiler could encounter many such " +"cases and learn which source variations are worth testing first." +msgstr "" +"これは非常に具体的な詳細であり、汎用 AI " +"モデルが最初に問題に遭遇したときには認識できない可能性があります。このコンパイラに特化したモデルは、そのようなケースを多数確認し、どのソース " +"コードのバリエーションが最初にテストする価値があるかを学習できます。" + +#: website/public/blog/reverse-engineering-models/index.html:407 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The opportunity is especially clear in final matching problems where we " +"already understand the behavior. We are then asking which ordinary source " +"form this compiler turns into the observed instructions. Specialization " +"could help the model propose those forms more efficiently, while the target " +"still determines which hypotheses survive." +msgstr "" +"動作を理解したうえで最後の差分を詰める場面では、この可能性が特にはっきりする。問うのは、このコンパイラがどの通常の書き方を観測された命令に変換するかだ。特化した学習によって候補を効率よく出せるかもしれないが、どの仮説が正しいかは対象のプログラムが決める。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:415 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The useful patterns would have to retain their context, though. TH08 uses an" +" earlier Visual C++ version than TH095 and TH09. TH04 uses a different DOS-" +"era compiler, and its {ph0}player reconstruction{ph1} needed different " +"hypotheses about source patterns and how the compiled code was combined into" +" an executable. Shared authorship gives us a prior, but each target tells us" +" whether it applies. Distinguishing those contexts is part of what a " +"specialist would need to learn." +msgstr "" +"ただし、パターンには適用条件を残す必要がある。TH08 は TH095 や TH09 より古い Visual C++ を使う。TH04 は DOS " +"時代の別のコンパイラを使い、{ph0}プレイヤーのコードの再構築{ph1}では、書き方や実行ファイルへのリンク順序について別の仮説が必要だった。同じ作者であることは手がかりになるが、適用できるかは各対象で確かめる。その区別も特化モデルが学ぶべきことだ。" + +#: website/public/blog/reverse-engineering-models/index.html:431 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Where should that experience live?" +msgstr "その経験はどこに蓄積すべきか?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:432 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before deciding to teach those patterns through model training, I would look" +" at the knowledge we already keep in the repository. In the {ph0}previous " +"essay{ph1}, I wrote about it becoming the project's working memory. That " +"memory includes compiler lessons, along with the source and failed " +"experiments that made them useful." +msgstr "" +"モデルの学習でこうしたパターンを覚えさせる前に、まずリポジトリに保存している知識を見たい。{ph0}前のエッセイ{ph1}では、リポジトリがプロジェクトの作業記憶になっていると書いた。そこにはコンパイラから学んだことと、それを裏付けるソースや失敗した実験が残っている。" + +#: website/public/blog/reverse-engineering-models/index.html:441 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"By retrieving that record, an agent can start with a known result and " +"inspect how we reached it. It can also avoid alternatives that have already " +"failed in the same context. The starting point improves without changing the" +" model's weights." +msgstr "" +"そのレコードを取得することで、エージェントは既知の結果から開始し、どのようにしてその結果に到達したかを検査できます。また、同じコンテキストですでに失敗した代替案を回避することもできます。モデルのトレーニング済みパラメーターを変更することなく、開始点が改善されます。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:447 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"When reconstructing TH09's enemy-spawn function, a {ph0}GPT-6-Luna-max " +"session{ph1} drew a hypothesis from TH08's corresponding source family. That" +" earlier source suggested how to structure the loop and handle failures " +"within it. TH09's own evidence supplied the game-specific details, and clean" +" rebuilds confirmed the complete match." +msgstr "" +"TH09 の敵生成関数を再構築した{ph0}GPT-6-Luna-max のセッション{ph1}は、TH08 " +"の関連するソースから仮説を得た。以前のコードが、ループの構成と失敗時の処理の手がかりになった。TH09 " +"自体の証拠でゲーム固有の細部を補い、クリーンビルドで関数全体の一致を確認した。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:458 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}\"The model did not know that compiler quirk\" and \"the model could " +"not use it\" are different problems.{ph1} Retrieval may solve the first, but" +" turning that fact into a useful candidate still takes work. And if the " +"candidate fails, the model must decide what the result means for its next " +"attempt. Knowing the relevant pattern is only the beginning of that process." +msgstr "" +"{ph0}「モデルがそのコンパイラの癖を知らない」と「知っていても使えない」は別の問題だ。{ph1} " +"検索で前者を解決できても、知識を有用な候補にするには能力がいる。候補が失敗すれば、その結果が次の試みに何を意味するかも判断しなければならない。関連するパターンを知るのは、出発点にすぎない。" + +#: website/public/blog/reverse-engineering-models/index.html:468 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For that reason, I would include a smaller general model with good retrieval" +" in any comparison. Giving it the same library of experience would help us " +"identify what training adds: better candidate generation, better decisions " +"after feedback, or both." +msgstr "" +"だから比較には、知識を適切に検索できる小さな汎用モデルも加えたい。同じ経験のライブラリを与えれば、学習が何を加えるかを調べられる。候補を作る力なのか、フィードバック後の判断なのか、それとも両方なのか。" + +#: website/public/blog/reverse-engineering-models/index.html:476 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Learning the next useful experiment" +msgstr "次の有用な実験を選ぶ力を学ぶ" + +#: website/public/blog/reverse-engineering-models/index.html:477 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those decisions after feedback are what I would most like to study. In " +"agentic reconstruction, the model proposes an explanation and tests it " +"against the program. Even when the candidate fails, the result can narrow " +"the remaining possibilities and suggest a more useful next experiment." +msgstr "" +"私が最も研究したいのは、フィードバック後の判断だ。エージェントによる再構築では、モデルが説明を提案し、プログラムに照らして検証する。候補が失敗しても、可能性を絞り、次の有用な実験を考える材料になる。" + +#: website/public/blog/reverse-engineering-models/index.html:484 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Consider the temporary-results example again. Its final source shows the " +"arrangement that worked, but the route to that source contains more " +"information. Each of the following experiments helped identify what still " +"needed to change." +msgstr "" +"一時的な結果の例をもう一度考えてみましょう。最終的なソースには、機能した配置が示されていますが、そのソースへのルートにはさらに多くの情報が含まれています。以下の各実験は、まだ変更する必要があるものを特定するのに役立ちました。" + +#: website/public/blog/reverse-engineering-models/index.html:497 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Experiment" +msgstr "実験" + +#: website/public/blog/reverse-engineering-models/index.html:498 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What its result taught us" +msgstr "その結果が私たちに教えてくれたこと" + +#: website/public/blog/reverse-engineering-models/index.html:503 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Try a different way to copy the results" +msgstr "別の方法で結果をコピーしてみてください" + +#: website/public/blog/reverse-engineering-models/index.html:504 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The copy code changes, but the function still differs." +msgstr "コピーのコードは変わったが、関数にはまだ差分がある。" + +#: website/public/blog/reverse-engineering-models/index.html:509 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Use the returned values directly" +msgstr "戻り値を直接使用する" + +#: website/public/blog/reverse-engineering-models/index.html:510 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Copying follows the original pattern, but temporary storage still differs." +msgstr "コピーは元のパターンに従いますが、一時ストレージは依然として異なります。" + +#: website/public/blog/reverse-engineering-models/index.html:516 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the temporary results alive together" +msgstr "一時的な結果を同時に有効にしておく" + +#: website/public/blog/reverse-engineering-models/index.html:517 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"They get separate storage, but one still overlaps a value used later for " +"collision checks." +msgstr "これらは別々のストレージを取得しますが、1 つは、後で衝突チェックに使用される値と依然として重複します。" + +#: website/public/blog/reverse-engineering-models/index.html:523 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the collision-check value separate" +msgstr "衝突判定の値を独立させる" + +#: website/public/blog/reverse-engineering-models/index.html:524 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A clean rebuild confirms the complete match." +msgstr "ソースからクリーンに再コンパイルすると、完全に一致していることが確認されます。" + +#: website/public/blog/reverse-engineering-models/index.html:529 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Selected experiments from the attack-function reconstruction above, grouped " +"by what they taught us." +msgstr "上記の攻撃関数の再構築から選択した実験を、教えられた内容ごとにグループ化しました。" + +#: website/public/blog/reverse-engineering-models/index.html:533 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A training record that keeps this sequence can teach more than the final " +"source alone. It connects each proposed change to an observed result and the" +" next decision. Keeping ineffective variations matters too: they tell a " +"later agent which hypotheses have already been tested in that context." +msgstr "" +"この順序を維持したトレーニング記録は、最終的なソースだけよりも多くのことを教えることができます。提案された各変更を観察された結果と次の決定に結び付けます。効果のないバリエーションを維持することも重要です。バリエーションは、後のエージェントに、そのコンテキストでどの仮説がすでにテストされているかを伝えます。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:540 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}I would want the model to learn how to use feedback.{ph1} A neutral " +"result is a reason to reconsider the variation; a changed mismatch may point" +" to another cause. When local edits stop being informative, the agent should" +" be able to revisit the source representation or inspect the evidence again." +msgstr "" +"{ph0}モデルにはフィードバックの使い方を学んでほしい。{ph1} " +"結果が変わらなければ、その変更を考え直す理由になる。差分が変われば、別の原因が見えてくるかもしれない。局所的な編集から情報が得られなくなったら、全体の書き方を見直したり、証拠を再確認したりできるべきだ。" + +#: website/public/blog/reverse-engineering-models/index.html:549 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Learning to make those choices requires autonomy. Sometimes the useful next " +"step is inspecting another caller rather than compiling another edit. A " +"rigid sequence of required rewrites would remove that choice, along with " +"part of the behavior we want the model to learn." +msgstr "" +"これらの選択を行うことを学ぶには自主性が必要です。場合によっては、別の編集をコンパイルするのではなく、別の呼び出し元を検査することが次のステップとして役立つ場合があります。必要な書き換えを厳密に繰り返すと、モデルに学習させたい動作の一部とともに、その選択肢が削除されてしまいます。" + +#: website/public/blog/reverse-engineering-models/index.html:558 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why RE is interesting for reinforcement learning" +msgstr "リバースエンジニアリングが強化学習にとって興味深い理由" + +#: website/public/blog/reverse-engineering-models/index.html:559 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Bounded reconstruction gives us a practical environment for studying these " +"choices. Once the target and compiler are pinned, many source experiments " +"produce quick, repeatable feedback. The agent can compile a candidate, " +"inspect the mismatch and use that observation to choose its next step." +msgstr "" +"範囲を定めた再構築は、こうした判断を研究するための実用的な環境になる。対象とコンパイラのバージョンを固定すれば、多くのソースの実験で素早く再現可能なフィードバックを得られる。エージェントは候補をコンパイルし、差分を調べ、次の行動を選べる。" + +#: website/public/blog/reverse-engineering-models/index.html:566 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reinforcement learning could train that decision-making process through the " +"feedback it receives. A candidate accepted by a defined check gives us a " +"concrete outcome, while the sequence of attempts lets us observe how much " +"work it took to reach it." +msgstr "" +"強化学習なら、得られたフィードバックからこの判断を学習できるかもしれない。定義された検証を通った候補は具体的な成果になり、試行の記録から、そこに至るまでの労力も調べられる。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:572 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"There is already research supporting this direction. {ph0}LLM4Decompile{ph1}" +" studies models specialized for decompilation. {ph2}AutoDecompiler{ph3} " +"trains multi-turn refinement using compiler and execution feedback. Their " +"studies focus on functional recovery. For this work, I would use them as a " +"starting point for the related task of finding source that matches a " +"historical compiler's output." +msgstr "" +"この方向を支える研究はすでにある。{ph0}LLM4Decompile{ph1}は逆コンパイルに特化したモデルを研究している。{ph2}AutoDecompiler{ph3}はコンパイルと実行のフィードバックを使って複数ターンの改善を学習する。両者が扱うのは機能の復元だ。私はそこを出発点に、当時のコンパイラの出力に一致するソースを探すという関連した課題を研究したい。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:584 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To reach such a well-defined task, though, we may first need to establish " +"what a function owns or devise a new runtime measurement. In TH08, for " +"example, a {ph0}collision difference led back to how the original program " +"handled floating-point calculations{ph1}. Before the agent could test a " +"useful correction, it had to work out what to measure." +msgstr "" +"ただし、このように明確に定義されたタスクに到達するには、まず関数が所有するリソースを確立するか、実行中のプログラムの状態や振る舞いを観測する新しい方法を考案する必要があるかもしれません。たとえば、TH08" +" " +"では、{ph0}衝突の違いにより、元のプログラムでの浮動小数点計算の処理方法が問題になりました{ph1}。エージェントが有用な補正をテストする前に、何を測定するかを決定する必要がありました。" + +#: website/public/blog/reverse-engineering-models/index.html:595 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This suggests another role for a broadly capable model: helping make the " +"problem checkable in the first place. Once that check exists, we can " +"investigate how far a learned search policy can get with it." +msgstr "" +"ここから、幅広い能力を持つモデルのもう一つの役割が見える。まず問題を検証可能にすることだ。検証方法ができて初めて、学習した探索方針がどこまで進めるかを調べられる。" + +#: website/public/blog/reverse-engineering-models/index.html:603 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "When the oracle is wrong" +msgstr "検証器が間違っていたら" + +#: website/public/blog/reverse-engineering-models/index.html:604 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Once we train against that check, its coverage becomes part of what the " +"model learns. I call the reference or checker an oracle. Even if it gives a " +"repeatable answer, we still need to understand what that answer covers." +msgstr "" +"その検証を使って学習すると、検証範囲も学習環境の一部になる。私はこうした参照基準や検証器を oracle " +"と呼んでいる。答えが再現可能でも、それが何を確認した答えなのかを理解する必要がある。" + +#: website/public/blog/reverse-engineering-models/index.html:610 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"TH08 showed how much can hide outside that coverage. Some reconstructed " +"functions passed their exact comparisons with wrong floating constants. The " +"comparator adjusted relocated addresses before checking instructions, but it" +" checked the values behind those addresses only when they had been " +"explicitly recorded for verification. Other literal references could pass " +"without their values being examined." +msgstr "" +"TH08 " +"では、検証範囲の外に問題が隠れていた。誤った浮動小数点定数を使っていても、完全一致の比較を通る関数があった。比較器は命令を確認する前に再配置されたアドレスを調整するが、そのアドレスが指す値まで確認するのは、検証対象として明示的に登録された場合だけだった。ほかの定数への参照は、値を調べずに通ってしまう可能性があった。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: 1.0f +#. {ph3}: 32.0f +#: website/public/blog/reverse-engineering-models/index.html:619 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To close that gap, a {ph0}wider audit{ph1} found incorrect constants in " +"functions that had already been accepted. In one calculation, the " +"reconstructed source added {ph2} where the original added {ph3}." +msgstr "" +"このギャップに対処するために、{ph0}広範な監査{ph1}により、すでに検証に合格した関数内に不正な定数が見つかりました。ある計算では、再構成されたソース" +" コードでは {ph2} が追加されましたが、元のコードでは {ph3} が追加されました。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:629 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A learner rewarded by the old check could learn to produce the wrong " +"reconstruction.{ph1} We had not trained an RL model against this oracle; the" +" observed failure was in the check itself. But using that check as a reward " +"would have made the same blind spot part of the learning environment." +msgstr "" +"{ph0}古い検証結果で報酬を与えれば、モデルは誤った再構築を学習してしまうかもしれない。{ph1} この検証器で RL " +"モデルを訓練したわけではない。観測したのは検証そのものの不具合だ。ただ、それを報酬に使えば、同じ盲点が学習環境に入ることになる。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:639 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A functional test can have a similar blind spot. {ph0}Differential-fuzzing " +"research on LLM decompilers{ph1} has found behavior differences beyond the " +"supplied tests. Passing those tests therefore needs to be read in terms of " +"the cases they exercise, just as an instruction match needs to be read in " +"terms of the bytes and data it checks." +msgstr "" +"機能テストにも同様の盲点がある可能性があります。 {ph0}LLM " +"デコンパイラで差分ファジングを使用した研究{ph1}では、提供されたテスト以外の動作の違いが見つかりました。したがって、合格したテストを、実行されるケースの観点から理解する必要があります。同様に、命令の一致は、検証されるバイトとデータの観点から理解する必要があります。" + +#: website/public/blog/reverse-engineering-models/index.html:649 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before using either kind of check as a reward, I would test it with " +"deliberate failures. I would also keep final acceptance outside the " +"candidate's editable source, so the model has to satisfy the check through " +"its reconstruction. Otherwise, something like a byte array copied from the " +"target could satisfy a naive score without recovering useful source." +msgstr "" +"どちらの種類の検証チェックも報酬として使用する前に、意図的に間違った結果でテストします。また、最終的な承認は、AI エージェントが編集できるソース " +"コードとは別にしておきます。モデルはプログラムを正しく再構築することで検証に合格する必要があります。そうしないと、ターゲットからバイト配列をコピーするだけで、有用なソース" +" コードを回復せずに、単純なスコアリング システムを騙してしまう可能性があります。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:657 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Improving the learner and improving its oracle belong to the same " +"research problem.{ph1} As the learner gets better at optimizing the feedback" +" we give it, understanding that feedback becomes more important too." +msgstr "" +"{ph0}学習するモデルの改善と、その検証器の改善は同じ研究課題だ。{ph1} " +"モデルが与えられたフィードバックを上手に最適化するほど、そのフィードバックを理解することも重要になる。" + +#: website/public/blog/reverse-engineering-models/index.html:667 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A comparison worth running" +msgstr "実行する価値のある比較" + +#: website/public/blog/reverse-engineering-models/index.html:668 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"With those checks in place, I would start the comparison on a fixed VC7.1 " +"task family. First, I would establish baselines for a strong general model " +"and a smaller one. Adding repository knowledge to the smaller model would " +"show what retrieval changes. From there, supervised specialization and an " +"RL-trained version would let us examine what training adds." +msgstr "" +"これらのチェックを実施したら、固定の VC7.1 タスク ファミリで比較を開始します。まず、強力な汎用 AI モデルと小規模な AI " +"モデルのベースラインを確立します。リポジトリの知識をより小さなモデルに追加すると、取得によってどのような変化が生じるのかがわかります。そこから、教師付き特殊化と" +" RL でトレーニングされたバージョンを使用して、トレーニングによって追加される内容を検証できるようになります。" + +#: website/public/blog/reverse-engineering-models/index.html:676 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To make those comparisons informative, each model would receive the same " +"target evidence and tools. For the training-specific comparisons, I would " +"keep the base model and available knowledge fixed as well. Repeated trials " +"would help distinguish a reliable search strategy from a lucky candidate." +msgstr "" +"これらの比較を有益なものにするために、各モデルは同じターゲットの証拠とツールを受け取ります。トレーニング固有の比較の場合、基本モデルと利用可能な知識も固定したままにします。試行を繰り返すことは、信頼できる検索戦略と偶然成功した候補ソリューションを区別するのに役立ちます。" + +#: website/public/blog/reverse-engineering-models/index.html:683 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would then separate two starting conditions: reconstructing a function " +"from its instructions, and closing the last few differences in maintained " +"source. A specialist might be excellent at the second task while a general " +"model does much of the work needed to reach it. Measuring them separately " +"would reveal that division of work." +msgstr "" +"二つの出発点も分けたい。命令から関数を再構築する場合と、保守しているソースの最後の差分を詰める場合だ。特化モデルが後者に優れていても、そこまでの作業の多くは汎用モデルが担っているかもしれない。別々に測れば、その分担が見える。" + +#: website/public/blog/reverse-engineering-models/index.html:691 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both tasks need genuinely held-out examples. Because related Touhou games " +"share engine code, holding out a game's name would still allow nearly " +"identical functions to appear in training. I would instead hold out source " +"families and keep their accepted answers out of both training and retrieval." +msgstr "" +"どちらのタスクも、トレーニングと取得から除外されたテスト例が必要です。関連する Touhou ゲームはエンジン コードを共有します。トレーニングから 1" +" つのゲームを除外しても、トレーニング セットにほぼ同じ機能が残る可能性があります。代わりに、関連するソース " +"コードのグループ全体を除外し、検証済みの回答をトレーニング データと検索ライブラリの両方から除外します。" + +#: website/public/blog/reverse-engineering-models/index.html:698 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"On those examples, the outcome would be complete, independently validated " +"matches. Giving each model the same candidate or compile budget would help " +"compare search quality. Measuring elapsed time and total cost, including " +"compilation, would tell us whether that quality produces an efficient " +"workflow. Training costs would need their own accounting too." +msgstr "" +"測る成果は、関数全体が独立した検証を通って一致したことだ。候補数やコンパイル回数の予算をそろえれば探索の質を比べられる。コンパイルを含む総時間と総費用を測れば、それが効率的なワークフローにつながるかがわかる。学習費用は別に計上したい。" + +#: website/public/blog/reverse-engineering-models/index.html:706 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both views matter because a cheaper proposal can become an expensive " +"workflow if it causes more uninformative compiles. Conversely, a slower " +"model may pay for itself by finding a useful experiment sooner. The " +"comparison should capture those costs." +msgstr "" +"どちらの測定値も重要です。安価な提案により、有用な情報を提供しないコンパイルが増加すると、ワークフローが高価になる可能性があります。逆に、有用な実験がすぐに見つかった場合、遅いモデルの方が経済的になる可能性があります。比較にはこれらのコストも含める必要があります。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:712 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Counting accepted results also keeps intermediate progress in perspective. " +"One TH09 {ph0}replay-save candidate{ph1} looked promising because the " +"compiled function had the expected size, but a full comparison still found " +"substantial differences. Changes in how the compiler arranges instructions " +"can also make a candidate look closer without resolving the underlying " +"problem. Such diagnostics can guide the next experiment, but I would " +"validate any reward for \"getting closer\" against the full acceptance " +"check." +msgstr "" +"受け入れられた結果をカウントすることで、中間の進捗状況も把握できます。 1 つの TH09 " +"{ph0}再生保存候補{ph1}は、コンパイルされた関数が予想どおりのサイズを持っていたため有望に見えましたが、完全な比較では依然として大きな違いが見つかりました。コンパイラが命令を配置する方法を変更すると、根本的な問題が解決されずに、候補がより近くに見える可能性もあります。このような診断は次の実験を導くことができますが、私は「近づく」ことに対する報酬を完全な受け入れチェックと照らし合わせて検証します。" + +#: website/public/blog/reverse-engineering-models/index.html:727 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would be interested in the pattern across these results. If a specialist " +"does well within a familiar compiler family, does that advantage survive a " +"toolchain change? And once both models have good retrieval, how much does " +"training still add? The answers would help us decide where each kind of " +"model is most useful." +msgstr "" +"結果全体にどんな傾向があるかを知りたい。特化モデルが使い慣れたコンパイラで優れていても、ツールチェーンを変えた後にその強みは残るだろうか。両方が十分に知識を検索できるとき、学習はどれだけ価値を加えるのか。答えは、各モデルをどこで使うべきかを考える助けになる。" + +#: website/public/blog/reverse-engineering-models/index.html:736 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What kind of agent does this suggest?" +msgstr "どんなエージェントが必要なのか?" + +#: website/public/blog/reverse-engineering-models/index.html:737 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"If the models turn out to help at different stages, a useful arrangement " +"might be a broad agent with access to a specialist. Once a task becomes a " +"familiar compiler search, it could ask the specialist for candidates. If the" +" results stop making sense, the broader agent could return to the target " +"evidence or investigate the measurement itself." +msgstr "" +"モデルが段階ごとに違う強みを持つなら、汎用エージェントが特化モデルを呼び出せる構成が役立つかもしれない。問題が慣れたコンパイラでの探索になったら、特化モデルに候補を求める。結果が納得できなくなったら、汎用エージェントが対象の証拠や測定そのものを調べ直す。" + +#: website/public/blog/reverse-engineering-models/index.html:745 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In that arrangement, specialization becomes a tool the agent can choose as " +"the investigation develops. The agent keeps the freedom to change direction " +"that made the earlier breakthroughs possible." +msgstr "この構成では、特化モデルは調査の進み方に応じて選べる道具になる。エージェントには、以前の突破口を可能にした方向転換の自由が残る。" + +#: website/public/blog/reverse-engineering-models/index.html:750 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This follows naturally from the industrial shift I described in the previous" +" essay. Being able to run many more experiments changes how we organize the " +"work. It also gives us a reason to ask which decisions need broad judgment " +"and which improve through repeated experience with a particular craft." +msgstr "" +"これは、前回のエッセイで説明した産業の変化から自然に生じます。さらに多くの実験を実行できるようになると、作業の組織化の方法が変わります。また、どの決定には広範な判断が必要なのか、どの決定は特定の技術での繰り返しの経験によって改善されるのかを尋ねる理由も得られます。" + +#: website/public/blog/reverse-engineering-models/index.html:757 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For now, the strongest general model has earned its place in my workflow by " +"opening problems I had left unresolved. Those successes are what make me " +"want to examine the work more closely: how much depended on the model's " +"breadth, and how much could be handled by a model trained in this particular" +" kind of reconstruction?" +msgstr "" +"今のところ、未解決のまま残されていた問題を解決することで、最強の汎用 AI " +"モデルが私のワークフローにその地位を獲得しました。これらの成功により、私はこの作業をさらに詳しく調べてみたいと思うようになりました。どの程度がモデルの範囲に依存しているのか、また、この特定の種類の再構成でトレーニングされたモデルでどの程度の処理が可能でしょうか?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:765 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}The repositories give us a way to test the question.{ph1} I would like " +"to use that record to build the comparison and see what we learn." +msgstr "{ph0}リポジトリは、質問をテストする方法を提供します。{ph1}その記録を使用して比較を構築し、何が分かるかを確認したいと思います。" + +#: website/public/blog/reverse-engineering-models/index.html:775 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Evidence and model attribution" +msgstr "証拠とモデルの帰属" + +#: website/public/blog/reverse-engineering-models/index.html:783 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The examples describe historical function-level comparisons recorded in the " +"linked commits. Whole-game runtime checks are separate milestones. Source " +"and matching records were reviewed on 13 October 2026." +msgstr "" +"この例では、リンクされたコミットに記録された関数レベルの比較の履歴について説明します。ゲーム全体のランタイム " +"チェックは別のマイルストーンです。ソースと一致する記録は 2026 年 10 月 13 日に確認されました。" + +#. {ph0}: gpt-dots +#. {ph1}: gpt-web +#: website/public/blog/reverse-engineering-models/index.html:789 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The TH09 label {ph0} identifies GPT-6-Astra. {ph1} changed with the model " +"used at the time, so that label alone does not identify an individual run. " +"Explicit model names in the other cited commits establish their recorded " +"attribution; the histories do not contain uniform per-run cost or effort " +"measurements." +msgstr "" +"TH09 ラベル {ph0} は GPT-6-Astra を意味します。 {ph1} " +"に関連付けられたモデルは時間の経過とともに変化したため、ラベルだけでは特定の実行で使用されたモデルを識別できません。他の引用されたコミット内の明示的なモデル名は、記録された帰属を示します。これらの履歴には、実行ごとのコストや労力の一貫した測定値は含まれていません。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#: website/public/blog/reverse-engineering-models/index.html:798 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 temporary lifetimes:{ph1} the {ph2}earlier context review{ph3} and" +" {ph4}complete cold reconstruction{ph5}." +msgstr "" +"{ph0}TH09 一時オブジェクトの有効期間:{ph1} {ph2}以前のコンテキスト " +"レビュー{ph3}と、{ph4}ソースからのクリーンな再コンパイルによって検証された完全な再構築{ph5}。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:810 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 Front callback:{ph1} {ph2}non-exact diagnostic regions{ph3}, " +"{ph4}Astra's new graph{ph5} and its {ph6}accepted recompile replay{ph7}." +msgstr "" +"{ph0}TH09 Front コールバック:{ph1} {ph2}一致しなかった診断領域{ph3}、{ph4}Astra の新しい制御フロー " +"グラフ{ph5}、およびその {ph6}受け入れられた再コンパイル チェック{ph7}。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:826 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Compiler knowledge:{ph1} {ph2}TH095 identifier-bucket reuse{ph3}, " +"{ph4}TH04 player source and linker order{ph5} and {ph6}Luna's TH09 enemy-" +"spawn closure{ph7}." +msgstr "" +"{ph0}コンパイラーの知識:{ph1} {ph2}TH095 識別子バケットの動作の再利用{ph3}、{ph4}TH04 プレーヤーのソース " +"コードとリンカーの順序{ph5}、および {ph6}Luna による TH09 の敵作成関数の完全な再構築{ph7}。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:842 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Checks and diagnostics:{ph1} {ph2}TH08 floating-literal audit{ph3}, " +"{ph4}TH08 numerical investigation{ph5} and {ph6}TH09 replay-save " +"reconstruction{ph7}." +msgstr "" +"{ph0}チェックと診断:{ph1} {ph2}TH08 浮動小数点定数監査{ph3}、{ph4}TH08 数値調査{ph5}、および " +"{ph6}TH09 リプレイ保存再構築{ph7}。" + +#: website/public/blog/reverse-engineering-models/index.html:859 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The proposed specialist and RL comparisons are future experiments. The " +"project histories motivate them; no trained specialist was evaluated for " +"this essay." +msgstr "" +"特化モデルと RL " +"の比較は、これから行いたい実験だ。プロジェクトの履歴がその動機になっているが、このエッセイで学習済みの特化モデルを評価したわけではない。" + +#: website/public/blog/reverse-engineering-models/index.html:6 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model? | REA" +msgstr "リバースエンジニアリングに、より大きなモデルは必要か? | REA" + +#: website/public/blog/reverse-engineering-models/index.html:7 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What Touhou reconstruction taught me about general models, specialist models" +" and learning to propose better hypotheses from compiler feedback." +msgstr "Touhou の再構築から考える、汎用モデルと特化モデル、そしてコンパイラのフィードバックからよりよい仮説を学ぶ可能性。" + +#: website/public/blog/reverse-engineering-models/index.html:240 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "On this page" +msgstr "このページでは" + +#: website/public/blog/reverse-engineering-models/index.html:253 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Models and reverse engineering essay" +msgstr "モデルとリバース エンジニアリングのエッセイ" + +#: website/public/blog/reverse-engineering-models/index.html:491 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Selected reconstruction experiments" +msgstr "厳選された復元実験" + +#: website/public/blog/reverse-engineering-models/index.html:883 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Back to top" +msgstr "ページ上部へ" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:776 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Detailed evidence notes{ph1} include the recorded comparisons and " +"experiment context." +msgstr "{ph0}詳しい証拠の記録{ph1}には、比較結果と実験の背景をまとめている。" + +#~ msgctxt "page:blog/index.html" +#~ msgid "By N0zoM1z0 · 13 October 2026 · Draft for review" +#~ msgstr "N0zoM1z0 著 · 2026年10月13日 · レビュー用草稿" + #~ msgctxt "page:index.html" #~ msgid "Setup guide" #~ msgstr "セットアップガイド" diff --git a/website/locales/ko/messages.po b/website/locales/ko/messages.po index 62a18cf20..3d0354df4 100644 --- a/website/locales/ko/messages.po +++ b/website/locales/ko/messages.po @@ -32,43 +32,44 @@ msgid "Thoughts on reverse engineering." msgstr "리버스 엔지니어링에 대한 생각." #. {ph0}:
-#: website/public/blog/index.html:200 +#: website/public/blog/index.html:214 #, python-brace-format msgctxt "page:blog/index.html" msgid "Game reconstruction{ph0}Essay" msgstr "게임 재구성{ph0}에세이" -#: website/public/blog/index.html:202 +#: website/public/blog/index.html:216 msgctxt "page:blog/index.html" msgid "The industrial revolution in reverse engineering" msgstr "리버스 엔지니어링의 산업 혁명" -#: website/public/blog/index.html:203 +#: website/public/blog/index.html:217 msgctxt "page:blog/index.html" msgid "" "What manual RE and reconstructing TH08, TH095 and TH04 taught me about " "agents, verification and knowledge that compounds across projects." msgstr "어떤 매뉴얼 다시고 재구성TH08,TH095및TH04에 대해 가르쳐 에이전트,검증 및 지는 화합물에 걸쳐 프로젝트입니다." -#: website/public/blog/index.html:208 +#: website/public/blog/index.html:207 website/public/blog/index.html:222 msgctxt "page:blog/index.html" msgid "Read the article" msgstr "기사 읽기" -#: website/public/blog/index.html:211 +#: website/public/blog/index.html:225 msgctxt "page:blog/index.html" msgid "By N0zoM1z0 · 10 October 2026" msgstr "여N0zoM1z0·10 월 2026" #. {ph0}: -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 +#: website/public/blog/reverse-engineering-models/index.html:883 #: website/public/blog/touhou-reconstruction/index.html:1080 #: website/public/examples/dino-lab/index.html:475 #: website/public/faq/index.html:573 #: website/public/first-investigation/index.html:523 #: website/public/get-started/index.html:882 -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 #: website/public/guides/index.html:282 #: website/public/guides/javascript/index.html:503 #: website/public/guides/native/index.html:509 website/public/index.html:1191 @@ -96,6 +97,8 @@ msgid "" msgstr "리버스 엔지니어링에 대한 생각,실제 경험을 바탕으로 REA 그리고 인공지능 요원들도" #: website/public/blog/index.html:75 website/public/blog/index.html:88 +#: website/public/blog/reverse-engineering-models/index.html:75 +#: website/public/blog/reverse-engineering-models/index.html:88 #: website/public/blog/touhou-reconstruction/index.html:75 #: website/public/blog/touhou-reconstruction/index.html:88 #: website/public/examples/dino-lab/index.html:75 @@ -130,6 +133,7 @@ msgid "" msgstr "REA -소프트웨어가 어떻게 작동하는지 알아보십시오. 코딩 에이전트로 리버스 엔지니어링." #: website/public/blog/index.html:108 +#: website/public/blog/reverse-engineering-models/index.html:108 #: website/public/blog/touhou-reconstruction/index.html:108 #: website/public/examples/dino-lab/index.html:109 #: website/public/faq/index.html:108 @@ -150,6 +154,7 @@ msgid "REA home" msgstr "REA 홈" #: website/public/blog/index.html:112 +#: website/public/blog/reverse-engineering-models/index.html:112 #: website/public/blog/touhou-reconstruction/index.html:112 #: website/public/examples/dino-lab/index.html:113 #: website/public/faq/index.html:112 @@ -170,6 +175,7 @@ msgid "Main navigation" msgstr "주 메뉴" #: website/public/blog/index.html:130 +#: website/public/blog/reverse-engineering-models/index.html:130 #: website/public/blog/touhou-reconstruction/index.html:130 #: website/public/examples/dino-lab/index.html:131 #: website/public/faq/index.html:130 @@ -190,6 +196,7 @@ msgid "REA on GitHub" msgstr "GitHub에서 REA 보기" #: website/public/blog/index.html:150 +#: website/public/blog/reverse-engineering-models/index.html:150 #: website/public/blog/touhou-reconstruction/index.html:150 #: website/public/examples/dino-lab/index.html:151 #: website/public/faq/index.html:150 @@ -210,6 +217,7 @@ msgid "Join REA on Discord" msgstr "REA Discord 참여" #: website/public/blog/index.html:169 +#: website/public/blog/reverse-engineering-models/index.html:169 #: website/public/blog/touhou-reconstruction/index.html:169 #: website/public/examples/dino-lab/index.html:170 #: website/public/faq/index.html:169 @@ -229,7 +237,7 @@ msgctxt "shared" msgid "Follow REA on X" msgstr "X에서 REA 팔로우" -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 msgctxt "page:blog/index.html" msgid "Back to top" msgstr "맨 위로" @@ -4422,7 +4430,7 @@ msgctxt "page:guides/browser/index.html" msgid "Copy read the loaded scripts" msgstr "로드 된 스크립트 읽기 복사" -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 msgctxt "page:guides/browser/index.html" msgid "Back to top" msgstr "맨 위로" @@ -10852,6 +10860,7 @@ msgid "Same spacing: 256 ÷ 16 = 16 angle units = 22.5°" msgstr "같은 간격:256÷16=16 각도 단위=22.5°" #: website/public/blog/index.html:114 +#: website/public/blog/reverse-engineering-models/index.html:114 #: website/public/blog/touhou-reconstruction/index.html:114 #: website/public/examples/dino-lab/index.html:115 #: website/public/faq/index.html:114 @@ -10872,6 +10881,7 @@ msgid "Showcases" msgstr "사례" #: website/public/blog/index.html:115 +#: website/public/blog/reverse-engineering-models/index.html:115 #: website/public/blog/touhou-reconstruction/index.html:115 #: website/public/examples/dino-lab/index.html:116 #: website/public/faq/index.html:115 @@ -10892,6 +10902,7 @@ msgid "Blog" msgstr "블로그" #: website/public/blog/index.html:113 +#: website/public/blog/reverse-engineering-models/index.html:113 #: website/public/blog/touhou-reconstruction/index.html:113 #: website/public/examples/dino-lab/index.html:114 #: website/public/faq/index.html:113 @@ -10912,6 +10923,7 @@ msgid "Get started" msgstr "시작하기" #: website/public/blog/index.html:116 +#: website/public/blog/reverse-engineering-models/index.html:116 #: website/public/blog/touhou-reconstruction/index.html:116 #: website/public/examples/dino-lab/index.html:117 #: website/public/faq/index.html:116 @@ -10931,13 +10943,14 @@ msgctxt "shared" msgid "Guides" msgstr "가이드" -#: website/public/blog/index.html:222 +#: website/public/blog/index.html:236 +#: website/public/blog/reverse-engineering-models/index.html:875 #: website/public/blog/touhou-reconstruction/index.html:1072 #: website/public/examples/dino-lab/index.html:467 #: website/public/faq/index.html:565 #: website/public/first-investigation/index.html:515 #: website/public/get-started/index.html:874 -#: website/public/guides/browser/index.html:490 +#: website/public/guides/browser/index.html:499 #: website/public/guides/index.html:274 #: website/public/guides/javascript/index.html:495 #: website/public/guides/native/index.html:501 website/public/index.html:1183 @@ -10951,13 +10964,14 @@ msgctxt "shared" msgid "Source" msgstr "소스 코드" -#: website/public/blog/index.html:223 +#: website/public/blog/index.html:237 +#: website/public/blog/reverse-engineering-models/index.html:876 #: website/public/blog/touhou-reconstruction/index.html:1073 #: website/public/examples/dino-lab/index.html:468 #: website/public/faq/index.html:566 #: website/public/first-investigation/index.html:516 #: website/public/get-started/index.html:875 -#: website/public/guides/browser/index.html:491 +#: website/public/guides/browser/index.html:500 #: website/public/guides/index.html:275 #: website/public/guides/javascript/index.html:496 #: website/public/guides/native/index.html:502 website/public/index.html:1184 @@ -10971,13 +10985,14 @@ msgctxt "shared" msgid "MIT license" msgstr "MIT 라이선스" -#: website/public/blog/index.html:224 +#: website/public/blog/index.html:238 +#: website/public/blog/reverse-engineering-models/index.html:877 #: website/public/blog/touhou-reconstruction/index.html:1074 #: website/public/examples/dino-lab/index.html:469 #: website/public/faq/index.html:567 #: website/public/first-investigation/index.html:517 #: website/public/get-started/index.html:876 -#: website/public/guides/browser/index.html:492 +#: website/public/guides/browser/index.html:501 #: website/public/guides/index.html:276 #: website/public/guides/javascript/index.html:497 #: website/public/guides/native/index.html:503 website/public/index.html:1185 @@ -10991,13 +11006,14 @@ msgctxt "shared" msgid "Join the community" msgstr "커뮤니티 참여" -#: website/public/blog/index.html:225 +#: website/public/blog/index.html:239 +#: website/public/blog/reverse-engineering-models/index.html:878 #: website/public/blog/touhou-reconstruction/index.html:1075 #: website/public/examples/dino-lab/index.html:470 #: website/public/faq/index.html:568 #: website/public/first-investigation/index.html:518 #: website/public/get-started/index.html:877 -#: website/public/guides/browser/index.html:493 +#: website/public/guides/browser/index.html:502 #: website/public/guides/index.html:277 #: website/public/guides/javascript/index.html:498 #: website/public/guides/native/index.html:504 website/public/index.html:1186 @@ -11911,6 +11927,1238 @@ msgctxt "runtime" msgid "Switch to dark mode" msgstr "다크 모드로 전환" +#. {ph0}: +#. {ph1}: +#: website/public/guides/browser/index.html:480 +#, python-brace-format +msgctxt "page:guides/browser/index.html" +msgid "" +"For HTTP request and response bodies, use {ph0}Fiddler Everywhere: SAZ " +"captures, MCP and public skills{ph1}. Export SAZ or HAR 1.2 for offline REA " +"capture Evidence, or connect Everywhere's MCP server to inspect captured " +"traffic with your agent." +msgstr "" +"HTTP 요청과 응답 본문을 살펴보려면 {ph0}Fiddler Everywhere: SAZ 캡처, MCP 및 공개 스킬{ph1}을 " +"사용하세요. SAZ ?? HAR 1.2로 내보내면 REA에서 오프라인 캡처 Evidence를 얻을 수 있으며, Everywhere의 " +"MCP 서버를 연결하여 에이전트와 함께 캡처된 트래픽을 살펴볼 수도 있습니다." + +#. {ph0}:
+#: website/public/blog/index.html:200 +#, python-brace-format +msgctxt "page:blog/index.html" +msgid "Model specialization{ph0}Essay" +msgstr "전문 모델{ph0}에세이" + +#: website/public/blog/index.html:202 +msgctxt "page:blog/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "리버스 엔지니어링에는 더 큰 모델이 필요할까?" + +#: website/public/blog/index.html:203 +msgctxt "page:blog/index.html" +msgid "" +"What Touhou reconstruction suggests about general models, specialist models " +"and learning from compiler feedback." +msgstr "Touhou 재구축 경험을 바탕으로 범용 모델, 전문 모델, 컴파일러 피드백을 통한 학습을 생각해 봅니다." + +#: website/public/blog/index.html:210 +msgctxt "page:blog/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "작성자: N0zoM1z0 · 2026년 10월 13일" + +#: website/public/blog/reverse-engineering-models/index.html:103 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Skip to content" +msgstr "개인정보보호정책" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:194 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Home{ph1} / {ph2}Blog{ph3} / Models and RE" +msgstr "{ph0}홈{ph1} / {ph2}블로그{ph3} / 모델 및 리버스 엔지니어링" + +#: website/public/blog/reverse-engineering-models/index.html:197 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Essay" +msgstr "에세이" + +# Reviewed for technical meaning and readability; preserve the distinction +# between historical evidence and proposed experiments. +#: website/public/blog/reverse-engineering-models/index.html:198 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "리버스 엔지니어링에 더 큰 모델이 필요할까?" + +#: website/public/blog/reverse-engineering-models/index.html:199 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "작성자: N0zoM1z0 · 2026년 10월 13일" + +#: website/public/blog/reverse-engineering-models/index.html:201 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Takeaway" +msgstr "핵심 요점" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:203 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Model capability changes the hypotheses an agent can find.{ph1} Later " +"sessions have solved problems that earlier work left unresolved." +msgstr "" +"{ph0}모델 기능은 AI 에이전트가 제안할 수 있는 가설에 영향을 미칩니다.{ph1} 이후 세션에서는 이전 작업에서 해결되지 않은 " +"문제를 해결했습니다." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:211 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A familiar compiler gives specialization something concrete to " +"learn.{ph1} Some exact matches depend on narrow, reusable source patterns." +msgstr "" +"{ph0}익숙한 컴파일러는 특화 모델이 학습할 수 있는 구체적인 패턴을 제공합니다.{ph1} 일부 정확한 기계 코드 일치는 재사용 가능한" +" 특정 소스 코드 패턴에 따라 달라집니다." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:218 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Give repository knowledge a fair comparison.{ph1} A smaller model with " +"good retrieval may already know enough to make progress." +msgstr "" +"{ph0}공정한 비교에 저장소 지식을 포함합니다.{ph1} 지식 검색이 좋은 소규모 모델은 이미 진행하기에 충분한 정보를 가지고 있을 수" +" 있습니다." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:223 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Train the next useful experiment, and test its oracle.{ph1} The value " +"of feedback depends on what the checker actually verifies." +msgstr "" +"{ph0}다음 유용한 실험을 선택하도록 모델을 훈련시키고 검증 검사기를 테스트합니다.{ph1} 피드백은 검사기가 올바른 것을 검증하는 " +"정도까지만 유용합니다." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:230 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Measure accepted reconstruction against time and cost.{ph1} That is the" +" result I want from the whole agent." +msgstr "" +"{ph0}시간 및 비용과 함께 검증된 재구성 결과를 평가합니다.{ph1} 이것이 제가 전체 AI 에이전트 워크플로에 대해 측정하고 싶은 " +"것입니다." + +#: website/public/blog/reverse-engineering-models/index.html:241 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "In this article" +msgstr "이 기사 에서" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:243 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What better models changed{ph1}" +msgstr "{ph0}더 강한 모델이 바꾼 것{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:244 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Why specialize?{ph1}" +msgstr "{ph0}전문화해야 하는 이유{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:245 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Where experience lives{ph1}" +msgstr "{ph0}이 경험은 어디에 저장되어야 합니까?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:246 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Learning the next experiment{ph1}" +msgstr "{ph0}다음 유용한 실험 학습{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:247 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}RE and reinforcement learning{ph1}" +msgstr "{ph0}리버스 엔지니어링 및 강화 학습{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:248 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}When the oracle is wrong{ph1}" +msgstr "{ph0}검증체커가 잘못된 경우{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:249 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}A comparison worth running{ph1}" +msgstr "{ph0}실행할 가치가 있는 비교{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:250 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What kind of agent?{ph1}" +msgstr "{ph0}어떤 에이전트가 필요할까?{ph1}" + +#: website/public/blog/reverse-engineering-models/index.html:258 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What better models changed" +msgstr "더 강한 모델이 바꾼 것" + +#: website/public/blog/reverse-engineering-models/index.html:259 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Since August 2026, I have been exploring how much reverse engineering an " +"agent can carry out with access to a program and a way to check its work. " +"Most of my experience comes from reconstructing games, especially Touhou. As" +" I have changed models, the repositories have kept a record of what each " +"session managed to recover." +msgstr "" +"2026년 8월부터, 프로그램에 접근하고 자신의 작업을 검증할 수 있는 에이전트가 리버스 엔지니어링을 얼마나 수행할 수 있는지 탐구해 " +"왔다. 내 경험은 주로 게임, 특히 Touhou 재구성에서 나온다. 모델을 바꿔 가며 작업하는 동안 저장소에는 각 세션이 무엇을 " +"복원했는지 기록이 쌓였다." + +#: website/public/blog/reverse-engineering-models/index.html:267 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of that work aims at exact reconstruction: we compile the maintained " +"source under a pinned historical toolchain and check whether it reproduces " +"the target function. When a later model succeeds where earlier work stalled," +" the source changes show what made that progress possible." +msgstr "" +"이 작업의 상당 부분은 정확한 재구성을 목표로 한다. 버전을 고정한 당시의 툴체인으로 유지 관리 중인 소스 코드를 컴파일하고, 대상 " +"함수가 재현되는지 확인한다. 이전에 막혔던 문제를 나중 모델이 해결하면, 소스 변경을 통해 무엇이 진전을 가능하게 했는지 살펴볼 수 " +"있다." + +#: website/public/blog/reverse-engineering-models/index.html:274 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The temporary values had to stay alive together" +msgstr "임시 변수에는 수명이 겹치는 것이 필요했습니다." + +#: website/public/blog/reverse-engineering-models/index.html:275 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of a TH09 attack function's logic had already been reconstructed, but " +"the compiler still stored temporary results differently from the original. " +"The remaining problem was how long those results needed to stay alive and " +"when their storage could be reused." +msgstr "" +"TH09 공격 함수의 논리 대부분은 이미 재구성되었지만 컴파일러는 여전히 원본과 다르게 임시 결과를 저장했습니다. 남은 문제는 해당 " +"결과를 유지하는 데 필요한 기간과 해당 저장소를 언제 재사용할 수 있는지였습니다." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:282 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A {ph0}GPT-6-Sol checkpoint{ph1} recorded this mismatch after testing " +"several ordinary source arrangements. When {ph2}GPT-6.1-Sol returned to the " +"problem{ph3}, it found a useful hypothesis about how long those temporary " +"values needed to stay alive." +msgstr "" +"여러 가지 일반적인 소스 코드 배열을 테스트한 후 {ph0}GPT-6-Sol 세션 레코드{ph1}에서 이러한 불일치를 보고했습니다. " +"{ph2}GPT-6.1-Sol이 문제를 다시 조사했을 때{ph3} 이러한 임시 변수의 수명에 대한 유용한 가설을 제안했습니다." + +#: website/public/blog/reverse-engineering-models/index.html:296 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The function produced temporary results and copied them into animation " +"positions. In C++, a temporary result stays alive until the expression using" +" it finishes. Grouping the real operations into one expression kept those " +"results alive together, so the compiler could no longer reuse the same " +"storage for all of them." +msgstr "" +"이 함수는 임시 결과를 생성하고 이를 애니메이션 위치에 복사했습니다. C++에서 임시 결과는 이를 사용하는 표현식이 완료될 때까지 활성 " +"상태로 유지됩니다. 실제 작업을 하나의 표현식으로 그룹화하면 해당 결과가 함께 유지되므로 컴파일러는 더 이상 모든 작업에 대해 동일한 " +"저장소를 재사용할 수 없습니다." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:303 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"One result still shared storage with a value used later for collision " +"checks. Giving that value its own lifetime resolved the remaining overlap. " +"After several targeted experiments, {ph0}a clean rebuild confirmed that the " +"full function matched{ph1}." +msgstr "" +"한 결과는 나중에 충돌 확인을 위해 사용되는 값으로 여전히 저장 공간을 공유합니다. 해당 값에 고유한 수명을 부여하면 나머지 중복이 " +"해결되었습니다. 여러 번의 대상 실험 후 {ph0}소스에서 깔끔하게 재컴파일하여 전체 함수가 {ph1}과 일치함을 확인했습니다." + +#: website/public/blog/reverse-engineering-models/index.html:312 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What interests me is how the discovery and the tests worked together. The " +"lifetime hypothesis gave the agent a new direction; compiler feedback helped" +" it refine that hypothesis into a source arrangement that matched." +msgstr "" +"나는 발견과 테스트가 어떻게 서로를 지원하는지에 관심이 있습니다. 임시 개체 수명에 대한 가설은 AI 에이전트에 새로운 방향을 " +"제시했습니다. 컴파일러 피드백은 해당 가설을 일치하는 소스 코드 배열로 구체화하는 데 도움이 되었습니다." + +#: website/public/blog/reverse-engineering-models/index.html:318 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A different control-flow graph" +msgstr "다른 제어 흐름 그래프" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:319 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A second TH09 function shows why that new direction can matter. Here, the " +"obstacle was how several branches shared their final work. " +"{ph0}GPT-6.1-Sol's earlier work{ph1} got parts of the function to match, but" +" the branches still joined up differently from the original." +msgstr "" +"두 번째 TH09 함수는 새로운 방향이 중요한 이유를 보여줍니다. 여기서 장애물은 여러 지점이 최종 작업을 공유하는 방식이었습니다. " +"{ph0}GPT-6.1-Sol의 이전 작업{ph1}에서는 일치하는 함수의 일부를 얻었지만 분기는 여전히 원본과 다르게 결합되었습니다." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:330 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}GPT-6-Astra changed the surrounding control-flow graph{ph1}. The main " +"paths now led to a shared final update and return. In that source structure," +" the compiler combined the branches in the same way as the original. The " +"complete function matched, and a {ph2}separate rebuild check confirmed " +"it{ph3}." +msgstr "" +"{ph0}GPT-6-Astra이 주변 제어 흐름 그래프{ph1}를 변경했습니다. 이제 주요 경로는 공유된 최종 업데이트 및 복귀로 " +"이어졌습니다. 해당 소스 구조에서 컴파일러는 원본과 동일한 방식으로 분기를 결합했습니다. 전체 함수이 일치했으며 {ph2}별도의 재구축 " +"검사를 통해 이를 확인했습니다{ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:342 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Changing the representation opened a more productive search space.{ph1}" +" Both cases give me a reason to take general model capability seriously: an " +"agent can keep trying local variations for a long time before finding a " +"better way to express the surrounding work." +msgstr "" +"{ph0}표현을 변경하면 보다 생산적인 검색 공간이 열렸습니다.{ph1} 두 경우 모두 일반 모델 기능을 진지하게 받아들여야 하는 이유가" +" 됩니다. 에이전트는 주변 작업을 표현하는 더 나은 방법을 찾기 전에 오랫동안 지역 변형을 계속 시도할 수 있습니다." + +#: website/public/blog/reverse-engineering-models/index.html:351 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"At the same time, the later sessions started with earlier source and " +"recorded failures. That accumulated work was part of their advantage. The " +"history lets us inspect the discoveries and their checks; a controlled " +"comparison would help separate the effect of the model from the experience " +"it inherited." +msgstr "" +"다만 이후 세션은 이전 소스 코드와 실패 기록을 이어받았다. 그 축적도 이점의 일부였다. 이력에서 발견과 검증 과정을 확인할 수는 " +"있지만, 모델 자체의 효과와 물려받은 경험의 효과를 구분하려면 조건을 맞춘 비교가 필요하다." + +#: website/public/blog/reverse-engineering-models/index.html:358 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is where the question becomes interesting to me. Some of the progress " +"came from finding a new representation, while much of the remaining work " +"involved testing source forms against a familiar compiler. How much needs " +"broad reasoning ability, and how much could a model learn through repeated " +"experience with that compiler and code?" +msgstr "" +"이것이 나에게 그 질문을 흥미롭게 만드는 이유이다. 새로운 표현을 찾는 과정에서 약간의 진전이 있었습니다. 나머지 작업의 대부분은 익숙한" +" 컴파일러에 대해 다양한 소스 코드 배열을 테스트하는 것과 관련되었습니다. 이 중 얼마나 폭넓은 추론 능력이 필요한가요? 해당 컴파일러와" +" 코드를 반복적으로 사용하여 모델이 얼마나 학습할 수 있습니까?" + +#: website/public/blog/reverse-engineering-models/index.html:368 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why might a specialist do well?" +msgstr "특화된 AI 모델이 왜 좋은 성과를 낼 수 있을까요?" + +#: website/public/blog/reverse-engineering-models/index.html:369 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Touhou gives us a concrete reason to expect that experience to help. Much of" +" the game code comes from the same author, ZUN, so recurring habits are a " +"reasonable starting assumption. The repositories back that up with useful " +"examples: related loops and helper arrangements have supplied hypotheses " +"across games." +msgstr "" +"Touhou은 그 경험이 도움이 될 것으로 기대하는 구체적인 이유를 제공합니다. 대부분의 게임 코드는 동일한 작성자인 ZUN에서 " +"제공되므로 반복되는 습관은 합리적인 시작 가정입니다. 저장소는 유용한 예를 통해 이를 뒷받침합니다. 관련 루프 및 도우미 배열은 게임 " +"전반에 걸쳐 가설을 제공했습니다." + +#: website/public/blog/reverse-engineering-models/index.html:376 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those source patterns become more specific when we include the compiler. " +"Under a particular version and set of options, small changes in source can " +"produce repeatable differences in machine code. Some of those differences " +"are surprisingly obscure." +msgstr "" +"이러한 소스 패턴은 컴파일러를 포함하면 더욱 구체적이 됩니다. 특정 버전 및 옵션 세트에서는 소스의 작은 변경으로 인해 기계어 코드에 " +"반복 가능한 차이가 발생할 수 있습니다. 이러한 차이점 중 일부는 놀라울 정도로 모호합니다." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: Supervisor::LoadDat +#. {ph4}: +#: website/public/blog/reverse-engineering-models/index.html:382 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In TH095, for example, {ph0}even the name used for a local variable could " +"change where VC7.1 stored it{ph1}. Moving the declaration alone did not " +"solve the layout, but an earlier background function had revealed a naming " +"pattern that affected the compiler's choice. That gave us a more promising " +"hypothesis for {ph2}the reconstruction of {ph3}{ph4}." +msgstr "" +"예를 들어 TH095에서는 {ph0}로컬 변수의 이름도 VC7.1이 변수를 저장한 위치에 영향을 미칠 수 있습니다{ph1}. 선언을 " +"이동해도 스토리지 레이아웃이 해결되지 않았습니다. 그러나 이전 백그라운드 함수에서는 컴파일러의 선택에 영향을 미치는 명명 패턴이 " +"드러났습니다. 이는 {ph2}{ph3}{ph4} 재구성에 대한 더 유망한 가설을 제시했습니다." + +#: website/public/blog/reverse-engineering-models/index.html:396 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reusing that pattern for the file-size variable produced the required layout" +" without adding storage. The full function then matched." +msgstr "" +"파일 크기 변수에 해당 패턴을 재사용하면 저장소를 추가하지 않고도 필요한 레이아웃이 생성되었습니다. 그런 다음 전체 함수이 일치했습니다." + +#: website/public/blog/reverse-engineering-models/index.html:401 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is the sort of narrow detail a general model may not know when it " +"arrives. A specialist trained on this compiler could encounter many such " +"cases and learn which source variations are worth testing first." +msgstr "" +"이는 범용 AI 모델이 처음 문제에 직면했을 때 알 수 없는 매우 구체적인 세부 사항입니다. 이 컴파일러에 특화된 모델은 이러한 사례를 " +"많이 확인하고 먼저 테스트할 가치가 있는 소스 코드 변형을 학습할 수 있습니다." + +#: website/public/blog/reverse-engineering-models/index.html:407 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The opportunity is especially clear in final matching problems where we " +"already understand the behavior. We are then asking which ordinary source " +"form this compiler turns into the observed instructions. Specialization " +"could help the model propose those forms more efficiently, while the target " +"still determines which hypotheses survive." +msgstr "" +"우리가 이미 동작을 이해하고 있는 최종 매칭 문제에서 기회는 특히 분명합니다. 그런 다음 이 컴파일러가 관찰된 명령으로 변환하는 일반적인" +" 소스 코드 배열을 묻습니다. 전문화는 모델이 이러한 형태를 보다 효율적으로 제안하는 데 도움이 될 수 있으며, 목표는 여전히 어떤 " +"가설이 살아남는지 결정합니다." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:415 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The useful patterns would have to retain their context, though. TH08 uses an" +" earlier Visual C++ version than TH095 and TH09. TH04 uses a different DOS-" +"era compiler, and its {ph0}player reconstruction{ph1} needed different " +"hypotheses about source patterns and how the compiled code was combined into" +" an executable. Shared authorship gives us a prior, but each target tells us" +" whether it applies. Distinguishing those contexts is part of what a " +"specialist would need to learn." +msgstr "" +"유용한 패턴에는 적용 조건이 함께 남아야 한다. TH08은 TH095와 TH09보다 오래된 Visual C++ 버전을 사용한다. " +"TH04은 DOS 시대의 다른 컴파일러를 사용하며, {ph0}플레이어 코드 재구성{ph1}에는 소스 패턴과 실행 파일의 링크 방식에 관한" +" 다른 가설이 필요했다. 같은 저자라는 사실은 초기 가정을 제공하지만, 그것이 적용되는지는 각 대상에서 확인해야 한다. 이런 맥락을 " +"구별하는 것도 특화 모델이 배울 일이다." + +#: website/public/blog/reverse-engineering-models/index.html:431 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Where should that experience live?" +msgstr "이 경험은 어디에 저장되어야 합니까?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:432 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before deciding to teach those patterns through model training, I would look" +" at the knowledge we already keep in the repository. In the {ph0}previous " +"essay{ph1}, I wrote about it becoming the project's working memory. That " +"memory includes compiler lessons, along with the source and failed " +"experiments that made them useful." +msgstr "" +"모델 훈련을 통해 이러한 패턴을 가르치기로 결정하기 전에 우리가 이미 저장소에 보관하고 있는 지식을 살펴보겠습니다. {ph0}이전 " +"에세이{ph1}에서는 그것이 프로젝트의 작업 메모리가 되는 것에 대해 썼습니다. 그 메모리에는 컴파일러 교훈과 이를 유용하게 만든 소스 " +"및 실패한 실험이 포함됩니다." + +#: website/public/blog/reverse-engineering-models/index.html:441 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"By retrieving that record, an agent can start with a known result and " +"inspect how we reached it. It can also avoid alternatives that have already " +"failed in the same context. The starting point improves without changing the" +" model's weights." +msgstr "" +"에이전트는 해당 기록을 검색하여 알려진 결과로 시작하고 해당 기록에 어떻게 도달했는지 검사할 수 있습니다. 또한 동일한 맥락에서 이미 " +"실패한 대안을 피할 수도 있습니다. 모델의 훈련된 매개변수를 변경하지 않고도 시작점이 향상됩니다." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:447 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"When reconstructing TH09's enemy-spawn function, a {ph0}GPT-6-Luna-max " +"session{ph1} drew a hypothesis from TH08's corresponding source family. That" +" earlier source suggested how to structure the loop and handle failures " +"within it. TH09's own evidence supplied the game-specific details, and clean" +" rebuilds confirmed the complete match." +msgstr "" +"TH09의 적 생성 함수를 재구성할 때, {ph0}GPT-6-Luna-max 세션{ph1}은 TH08의 관련 소스 코드에서 가설을 " +"얻었다. 이전 코드는 루프의 구성과 내부 실패 처리에 대한 단서를 제공했다. TH09 자체의 증거로 게임별 세부 사항을 채웠고, 소스에서" +" 새로 빌드해 함수 전체의 일치를 확인했다." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:458 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}\"The model did not know that compiler quirk\" and \"the model could " +"not use it\" are different problems.{ph1} Retrieval may solve the first, but" +" turning that fact into a useful candidate still takes work. And if the " +"candidate fails, the model must decide what the result means for its next " +"attempt. Knowing the relevant pattern is only the beginning of that process." +msgstr "" +"{ph0}\"모델이 컴파일러의 이상한 점을 몰랐습니다\"와 \"모델이 이를 사용할 수 없었습니다\"는 다른 문제입니다.{ph1} 검색을 " +"통해 첫 번째 문제를 해결할 수 있지만 해당 사실을 유용한 후보로 바꾸는 작업은 여전히 필요합니다. 그리고 후보 소스 코드가 실패하면 " +"모델은 다음 시도에서 결과가 무엇을 의미하는지 결정해야 합니다. 관련 패턴을 아는 것은 해당 프로세스의 시작일뿐입니다." + +#: website/public/blog/reverse-engineering-models/index.html:468 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For that reason, I would include a smaller general model with good retrieval" +" in any comparison. Giving it the same library of experience would help us " +"identify what training adds: better candidate generation, better decisions " +"after feedback, or both." +msgstr "" +"그러한 이유로 저는 어떤 비교에서도 검색 성능이 좋은 더 작은 범용 AI 모델을 포함하겠습니다. 동일한 경험 라이브러리를 제공하면 교육이" +" 추가하는 것이 무엇인지 식별하는 데 도움이 됩니다. 즉, 더 나은 후보 코드 생성, 피드백 후 더 나은 결정 또는 둘 다입니다." + +#: website/public/blog/reverse-engineering-models/index.html:476 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Learning the next useful experiment" +msgstr "다음 유용한 실험 학습" + +#: website/public/blog/reverse-engineering-models/index.html:477 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those decisions after feedback are what I would most like to study. In " +"agentic reconstruction, the model proposes an explanation and tests it " +"against the program. Even when the candidate fails, the result can narrow " +"the remaining possibilities and suggest a more useful next experiment." +msgstr "" +"피드백 이후의 결정은 제가 가장 연구하고 싶은 것입니다. 에이전트 재구성에서 모델은 설명을 제안하고 이를 프로그램에 대해 테스트합니다. " +"후보 소스 코드가 실패하더라도 그 결과는 남은 가능성을 좁히고 보다 유용한 다음 실험을 제안할 수 있습니다." + +#: website/public/blog/reverse-engineering-models/index.html:484 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Consider the temporary-results example again. Its final source shows the " +"arrangement that worked, but the route to that source contains more " +"information. Each of the following experiments helped identify what still " +"needed to change." +msgstr "" +"임시 결과 예제를 다시 고려해보세요. 최종 소스에는 제대로 작동한 배열이 표시되지만 해당 소스로의 경로에는 더 많은 정보가 포함되어 " +"있습니다. 다음 각 실험은 여전히 ​​변경해야 할 사항을 식별하는 데 도움이 되었습니다." + +#: website/public/blog/reverse-engineering-models/index.html:497 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Experiment" +msgstr "실험" + +#: website/public/blog/reverse-engineering-models/index.html:498 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What its result taught us" +msgstr "그 결과가 우리에게 가르쳐준 것" + +#: website/public/blog/reverse-engineering-models/index.html:503 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Try a different way to copy the results" +msgstr "결과를 복사하는 다른 방법을 시도해 보세요." + +#: website/public/blog/reverse-engineering-models/index.html:504 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The copy code changes, but the function still differs." +msgstr "복사 코드는 변경되지만 기능은 여전히 다릅니다." + +#: website/public/blog/reverse-engineering-models/index.html:509 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Use the returned values directly" +msgstr "반환된 값을 직접 사용" + +#: website/public/blog/reverse-engineering-models/index.html:510 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Copying follows the original pattern, but temporary storage still differs." +msgstr "복사는 원래 패턴을 따르지만 임시 저장소는 여전히 다릅니다." + +#: website/public/blog/reverse-engineering-models/index.html:516 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the temporary results alive together" +msgstr "모든 임시 변수를 동시에 유효한 상태로 유지" + +#: website/public/blog/reverse-engineering-models/index.html:517 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"They get separate storage, but one still overlaps a value used later for " +"collision checks." +msgstr "별도의 저장소를 갖지만 하나는 나중에 충돌 검사에 사용되는 값과 여전히 겹칩니다." + +#: website/public/blog/reverse-engineering-models/index.html:523 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the collision-check value separate" +msgstr "충돌 검사 값을 별도로 유지하십시오." + +#: website/public/blog/reverse-engineering-models/index.html:524 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A clean rebuild confirms the complete match." +msgstr "소스에서 완전히 재컴파일하면 완전한 일치가 확인됩니다." + +#: website/public/blog/reverse-engineering-models/index.html:529 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Selected experiments from the attack-function reconstruction above, grouped " +"by what they taught us." +msgstr "위의 공격 함수 재구성에서 선택한 실험을 우리에게 가르친 내용을 기준으로 그룹화했습니다." + +#: website/public/blog/reverse-engineering-models/index.html:533 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A training record that keeps this sequence can teach more than the final " +"source alone. It connects each proposed change to an observed result and the" +" next decision. Keeping ineffective variations matters too: they tell a " +"later agent which hypotheses have already been tested in that context." +msgstr "" +"이 시퀀스를 유지하는 훈련 기록은 최종 소스보다 더 많은 것을 가르칠 수 있습니다. 이는 제안된 각 변경 사항을 관찰된 결과 및 다음 " +"결정에 연결합니다. 비효과적인 변형을 유지하는 것도 중요합니다. 이는 해당 맥락에서 어떤 가설이 이미 테스트되었는지 나중에 에이전트에게 " +"알려줍니다." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:540 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}I would want the model to learn how to use feedback.{ph1} A neutral " +"result is a reason to reconsider the variation; a changed mismatch may point" +" to another cause. When local edits stop being informative, the agent should" +" be able to revisit the source representation or inspect the evidence again." +msgstr "" +"{ph0}모델이 피드백을 사용하는 방법을 배우기를 원합니다.{ph1} 중립적인 결과는 변화를 재고해야 하는 이유입니다. 변경된 불일치는 " +"다른 원인을 가리킬 수도 있습니다. 로컬 편집이 더 이상 정보를 제공하지 않으면 에이전트는 소스 표현을 다시 방문하거나 증거를 다시 " +"검사할 수 있어야 합니다." + +#: website/public/blog/reverse-engineering-models/index.html:549 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Learning to make those choices requires autonomy. Sometimes the useful next " +"step is inspecting another caller rather than compiling another edit. A " +"rigid sequence of required rewrites would remove that choice, along with " +"part of the behavior we want the model to learn." +msgstr "" +"그러한 선택을 하는 방법을 배우려면 자율성이 필요합니다. 때때로 유용한 다음 단계는 다른 편집 내용을 컴파일하는 대신 다른 호출자를 " +"검사하는 것입니다. 필수 재작성의 엄격한 순서는 모델이 학습하기를 원하는 동작의 일부와 함께 해당 선택을 제거합니다." + +#: website/public/blog/reverse-engineering-models/index.html:558 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why RE is interesting for reinforcement learning" +msgstr "강화 학습에 리버스 엔지니어링이 흥미로운 이유" + +#: website/public/blog/reverse-engineering-models/index.html:559 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Bounded reconstruction gives us a practical environment for studying these " +"choices. Once the target and compiler are pinned, many source experiments " +"produce quick, repeatable feedback. The agent can compile a candidate, " +"inspect the mismatch and use that observation to choose its next step." +msgstr "" +"범위가 정해진 재구성은 이런 선택을 연구할 실용적인 환경을 제공한다. 대상과 컴파일러 버전을 고정하면 많은 소스 코드 실험에서 빠르고 " +"반복 가능한 피드백을 얻을 수 있다. 에이전트는 후보 코드를 컴파일하고 차이를 살펴본 뒤 다음 단계를 선택할 수 있다." + +#: website/public/blog/reverse-engineering-models/index.html:566 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reinforcement learning could train that decision-making process through the " +"feedback it receives. A candidate accepted by a defined check gives us a " +"concrete outcome, while the sequence of attempts lets us observe how much " +"work it took to reach it." +msgstr "" +"강화 학습은 이런 피드백을 통해 의사 결정 과정을 훈련할 수 있다. 정의된 검증을 통과한 후보는 구체적인 결과가 되고, 시도 기록은 그 " +"결과에 도달하는 데 얼마나 많은 작업이 필요했는지 보여 준다." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:572 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"There is already research supporting this direction. {ph0}LLM4Decompile{ph1}" +" studies models specialized for decompilation. {ph2}AutoDecompiler{ph3} " +"trains multi-turn refinement using compiler and execution feedback. Their " +"studies focus on functional recovery. For this work, I would use them as a " +"starting point for the related task of finding source that matches a " +"historical compiler's output." +msgstr "" +"이미 이 방향을 뒷받침하는 연구가 있다. {ph0}LLM4Decompile{ph1}은 디컴파일에 특화된 모델을 연구한다. " +"{ph2}AutoDecompiler{ph3}는 컴파일과 실행 피드백으로 여러 차례 개선하는 과정을 학습한다. 이 연구들은 기능 복원에 " +"초점을 맞춘다. 나는 이를 출발점으로 삼아, 당시 컴파일러의 출력과 일치하는 소스 코드를 찾는 관련 과제를 연구하고 싶다." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:584 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To reach such a well-defined task, though, we may first need to establish " +"what a function owns or devise a new runtime measurement. In TH08, for " +"example, a {ph0}collision difference led back to how the original program " +"handled floating-point calculations{ph1}. Before the agent could test a " +"useful correction, it had to work out what to measure." +msgstr "" +"그처럼 명확한 과제를 만들려면, 먼저 함수가 어떤 리소스를 책임지는지 확인하거나 실행 중인 프로그램의 동작을 관찰할 새로운 방법을 " +"설계해야 할 수 있다. TH08에서는 {ph0}충돌 판정의 차이를 추적하다가 원본 프로그램의 부동 소수점 계산 방식에 도달했다{ph1}." +" 에이전트는 유용한 수정을 검증하기 전에 무엇을 측정할지부터 알아내야 했다." + +#: website/public/blog/reverse-engineering-models/index.html:595 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This suggests another role for a broadly capable model: helping make the " +"problem checkable in the first place. Once that check exists, we can " +"investigate how far a learned search policy can get with it." +msgstr "" +"이는 광범위하게 역량을 갖춘 모델의 또 다른 역할을 제시합니다. 즉, 우선 문제를 확인할 수 있도록 돕는 것입니다. 해당 검사가 존재하면" +" 학습된 검색 정책이 얼마나 멀리까지 도달할 수 있는지 조사할 수 있습니다." + +#: website/public/blog/reverse-engineering-models/index.html:603 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "When the oracle is wrong" +msgstr "검증체커가 잘못된 경우" + +#: website/public/blog/reverse-engineering-models/index.html:604 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Once we train against that check, its coverage becomes part of what the " +"model learns. I call the reference or checker an oracle. Even if it gives a " +"repeatable answer, we still need to understand what that answer covers." +msgstr "" +"검증 결과를 학습에 사용하면 그 검증의 범위도 학습 환경의 일부가 된다. 나는 이런 참조 기준이나 검증기를 oracle이라고 부른다. " +"반복 가능한 답이라도, 실제로 무엇을 확인한 답인지 이해해야 한다." + +#: website/public/blog/reverse-engineering-models/index.html:610 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"TH08 showed how much can hide outside that coverage. Some reconstructed " +"functions passed their exact comparisons with wrong floating constants. The " +"comparator adjusted relocated addresses before checking instructions, but it" +" checked the values behind those addresses only when they had been " +"explicitly recorded for verification. Other literal references could pass " +"without their values being examined." +msgstr "" +"TH08은 해당 적용 범위 외부에 얼마나 많은 것을 숨길 수 있는지 보여주었습니다. 재구성된 일부 함수는 잘못된 부동 상수와의 정확한 " +"비교를 통과했습니다. 비교기는 명령을 확인하기 전에 재배치된 주소를 조정했지만 확인을 위해 명시적으로 기록된 경우에만 해당 주소 뒤에 " +"있는 값을 확인했습니다. 다른 리터럴 참조는 해당 값을 검사하지 않고 통과할 수 있습니다." + +#. {ph0}: +#. {ph1}: +#. {ph2}: 1.0f +#. {ph3}: 32.0f +#: website/public/blog/reverse-engineering-models/index.html:619 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To close that gap, a {ph0}wider audit{ph1} found incorrect constants in " +"functions that had already been accepted. In one calculation, the " +"reconstructed source added {ph2} where the original added {ph3}." +msgstr "" +"이러한 격차를 해소하기 위해 {ph0}광범위한 감사{ph1}를 통해 이미 검증을 통과한 함수에서 잘못된 상수가 발견되었습니다. 한 " +"계산에서는 재구성된 소스 코드에 {ph2}가 추가된 반면 원본 코드에는 {ph3}가 추가되었습니다." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:629 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A learner rewarded by the old check could learn to produce the wrong " +"reconstruction.{ph1} We had not trained an RL model against this oracle; the" +" observed failure was in the check itself. But using that check as a reward " +"would have made the same blind spot part of the learning environment." +msgstr "" +"{ph0}기존 검증으로 보상하면 모델이 잘못된 재구성을 학습할 수도 있다.{ph1} 이 검증기로 RL 모델을 훈련한 것은 아니다. 실제로" +" 관찰한 실패는 검증 자체의 오류였다. 하지만 이를 보상으로 사용했다면 같은 사각지대가 학습 환경에 들어갔을 것이다." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:639 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A functional test can have a similar blind spot. {ph0}Differential-fuzzing " +"research on LLM decompilers{ph1} has found behavior differences beyond the " +"supplied tests. Passing those tests therefore needs to be read in terms of " +"the cases they exercise, just as an instruction match needs to be read in " +"terms of the bytes and data it checks." +msgstr "" +"기능 테스트에도 비슷한 사각지대가 있을 수 있습니다. {ph0}LLM 디컴파일러에서 차등 퍼징을 사용한 연구{ph1}에서는 제공된 테스트" +" 외부의 동작 차이를 발견했습니다. 그러므로 우리는 그것이 수행되는 사례의 관점에서 합격 테스트를 이해해야 합니다. 마찬가지로 명령어 " +"일치는 확인하는 바이트와 데이터 측면에서 이해되어야 합니다." + +#: website/public/blog/reverse-engineering-models/index.html:649 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before using either kind of check as a reward, I would test it with " +"deliberate failures. I would also keep final acceptance outside the " +"candidate's editable source, so the model has to satisfy the check through " +"its reconstruction. Otherwise, something like a byte array copied from the " +"target could satisfy a naive score without recovering useful source." +msgstr "" +"어느 검증이든 보상으로 사용하기 전에는 의도적으로 틀린 결과를 넣어 시험해 보고 싶다. 최종 승인 기준도 에이전트가 수정할 수 있는 소스" +" 코드 밖에 두어야 한다. 올바르게 재구성해 검증을 통과하도록 해야 하기 때문이다. 그렇지 않으면 대상에서 복사한 바이트 배열이 유용한 " +"소스 복원 없이 단순한 채점 기준을 통과할 수 있다." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:657 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Improving the learner and improving its oracle belong to the same " +"research problem.{ph1} As the learner gets better at optimizing the feedback" +" we give it, understanding that feedback becomes more important too." +msgstr "" +"{ph0}학습 모델의 개선과 검증기의 개선은 같은 연구 문제다.{ph1} 모델이 우리가 주는 피드백을 더 잘 최적화할수록, 그 피드백 " +"자체를 이해하는 일도 중요해진다." + +#: website/public/blog/reverse-engineering-models/index.html:667 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A comparison worth running" +msgstr "실행할 가치가 있는 비교" + +#: website/public/blog/reverse-engineering-models/index.html:668 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"With those checks in place, I would start the comparison on a fixed VC7.1 " +"task family. First, I would establish baselines for a strong general model " +"and a smaller one. Adding repository knowledge to the smaller model would " +"show what retrieval changes. From there, supervised specialization and an " +"RL-trained version would let us examine what training adds." +msgstr "" +"이러한 점검을 마친 후 고정된 VC7.1 작업 계열에 대한 비교를 시작합니다. 첫째, 강력한 범용 AI 모델과 더 작은 모델에 대한 " +"기준선을 설정하겠습니다. 더 작은 모델에 저장소 지식을 추가하면 검색이 어떻게 변경되는지 확인할 수 있습니다. 거기에서 감독 전문화와 " +"RL 훈련 버전을 통해 훈련에 추가되는 것이 무엇인지 검토할 수 있습니다." + +#: website/public/blog/reverse-engineering-models/index.html:676 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To make those comparisons informative, each model would receive the same " +"target evidence and tools. For the training-specific comparisons, I would " +"keep the base model and available knowledge fixed as well. Repeated trials " +"would help distinguish a reliable search strategy from a lucky candidate." +msgstr "" +"이러한 비교를 유익하게 만들기 위해 각 모델은 동일한 대상 증거와 도구를 받게 됩니다. 훈련별 비교를 위해 기본 모델과 사용 가능한 " +"지식도 고정된 상태로 유지합니다. 반복적인 시도는 우연히 성공한 후보 솔루션과 신뢰할 수 있는 검색 전략을 구별하는 데 도움이 됩니다." + +#: website/public/blog/reverse-engineering-models/index.html:683 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would then separate two starting conditions: reconstructing a function " +"from its instructions, and closing the last few differences in maintained " +"source. A specialist might be excellent at the second task while a general " +"model does much of the work needed to reach it. Measuring them separately " +"would reveal that division of work." +msgstr "" +"그런 다음 두 가지 시작 조건을 분리합니다. 즉, 명령에서 함수를 재구성하는 것과 유지 관리되는 소스에서 마지막 몇 가지 차이점을 닫는 " +"것입니다. 특화 AI 모델은 두 번째 작업에 탁월할 수 있지만 범용 AI 모델은 두 번째 작업에 도달하는 데 필요한 많은 작업을 " +"수행합니다. 그것들을 개별적으로 측정하면 업무 분할이 드러날 것입니다." + +#: website/public/blog/reverse-engineering-models/index.html:691 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both tasks need genuinely held-out examples. Because related Touhou games " +"share engine code, holding out a game's name would still allow nearly " +"identical functions to appear in training. I would instead hold out source " +"families and keep their accepted answers out of both training and retrieval." +msgstr "" +"두 과제 모두 학습 데이터와 검색 자료에서 제외한 테스트 사례가 필요하다. 관련 Touhou 게임은 엔진 코드를 공유하므로, 게임 하나만" +" 제외해도 거의 같은 함수가 학습 데이터에 남을 수 있다. 대신 관련 소스 코드 계열 전체를 따로 남겨 두고, 검증된 정답을 학습 " +"데이터와 검색 라이브러리 양쪽에서 제외하겠다." + +#: website/public/blog/reverse-engineering-models/index.html:698 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"On those examples, the outcome would be complete, independently validated " +"matches. Giving each model the same candidate or compile budget would help " +"compare search quality. Measuring elapsed time and total cost, including " +"compilation, would tell us whether that quality produces an efficient " +"workflow. Training costs would need their own accounting too." +msgstr "" +"측정할 결과는 완전하고 독립적으로 검증된 일치다. 후보 제안 수나 컴파일 횟수의 예산을 같게 하면 탐색 품질을 비교할 수 있다. 컴파일을" +" 포함한 총 시간과 비용은 그 품질이 효율적인 워크플로로 이어지는지 보여 준다. 학습 비용은 별도로 계산해야 한다." + +#: website/public/blog/reverse-engineering-models/index.html:706 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both views matter because a cheaper proposal can become an expensive " +"workflow if it causes more uninformative compiles. Conversely, a slower " +"model may pay for itself by finding a useful experiment sooner. The " +"comparison should capture those costs." +msgstr "" +"두 측정 모두 중요합니다. 유용한 정보를 제공하지 않는 컴파일이 더 많이 발생하면 값싼 제안으로 인해 비용이 많이 드는 작업 흐름이 " +"발생할 수 있습니다. 반대로, 유용한 실험을 더 빨리 발견하면 느린 모델이 더 경제적일 수 있습니다. 비교에는 이러한 비용이 포함되어야 " +"합니다." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:712 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Counting accepted results also keeps intermediate progress in perspective. " +"One TH09 {ph0}replay-save candidate{ph1} looked promising because the " +"compiled function had the expected size, but a full comparison still found " +"substantial differences. Changes in how the compiler arranges instructions " +"can also make a candidate look closer without resolving the underlying " +"problem. Such diagnostics can guide the next experiment, but I would " +"validate any reward for \"getting closer\" against the full acceptance " +"check." +msgstr "" +"승인된 결과를 계산하면 중간 진행 상황을 관점에서 볼 수 있습니다. 하나의 TH09 {ph0}재생 저장 후보{ph1}는 컴파일된 함수의 " +"크기가 예상과 같았기 때문에 유망해 보였지만 전체 비교에서는 여전히 상당한 차이가 발견되었습니다. 컴파일러가 명령을 배열하는 방식을 " +"변경하면 근본적인 문제를 해결하지 않고도 후보가 더 가까워 보이게 만들 수도 있습니다. 이러한 진단은 다음 실험의 지침이 될 수 있지만 " +"전체 승인 확인에 대해 \"가까워지는\" 것에 대한 보상을 검증하겠습니다." + +#: website/public/blog/reverse-engineering-models/index.html:727 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would be interested in the pattern across these results. If a specialist " +"does well within a familiar compiler family, does that advantage survive a " +"toolchain change? And once both models have good retrieval, how much does " +"training still add? The answers would help us decide where each kind of " +"model is most useful." +msgstr "" +"나는 이러한 결과의 패턴에 관심이 있습니다. 전문화된 AI 모델이 친숙한 컴파일러 제품군 내에서 잘 작동한다면 툴체인 변경 후에도 해당 " +"장점이 유지됩니까? 두 AI 모델 모두 좋은 검색 결과를 얻은 후에도 훈련을 통해 얼마나 추가되나요? 답변은 각 모델 종류가 가장 유용한" +" 위치를 결정하는 데 도움이 됩니다." + +#: website/public/blog/reverse-engineering-models/index.html:736 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What kind of agent does this suggest?" +msgstr "어떤 에이전트가 필요할까?" + +#: website/public/blog/reverse-engineering-models/index.html:737 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"If the models turn out to help at different stages, a useful arrangement " +"might be a broad agent with access to a specialist. Once a task becomes a " +"familiar compiler search, it could ask the specialist for candidates. If the" +" results stop making sense, the broader agent could return to the target " +"evidence or investigate the measurement itself." +msgstr "" +"모델이 여러 단계에서 도움이 되는 것으로 밝혀지면 특화 AI 모델에 액세스할 수 있는 광범위한 에이전트가 유용한 배열이 될 수 있습니다." +" 작업이 익숙한 컴파일러 검색이 되면 특화 AI 모델에 후보를 요청할 수 있습니다. 결과가 더 이상 이해되지 않는 경우 더 넓은 범위의 " +"에이전트는 대상 증거로 돌아가거나 측정 자체를 조사할 수 있습니다." + +#: website/public/blog/reverse-engineering-models/index.html:745 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In that arrangement, specialization becomes a tool the agent can choose as " +"the investigation develops. The agent keeps the freedom to change direction " +"that made the earlier breakthroughs possible." +msgstr "" +"이러한 배열에서 전문화는 조사가 진행됨에 따라 에이전트가 선택할 수 있는 도구가 됩니다. 에이전트는 이전의 돌파구를 가능하게 했던 방향을" +" 자유롭게 변경할 수 있습니다." + +#: website/public/blog/reverse-engineering-models/index.html:750 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This follows naturally from the industrial shift I described in the previous" +" essay. Being able to run many more experiments changes how we organize the " +"work. It also gives us a reason to ask which decisions need broad judgment " +"and which improve through repeated experience with a particular craft." +msgstr "" +"이는 이전 에세이에서 설명한 산업 변화에서 자연스럽게 나온 것입니다. 더 많은 실험을 실행할 수 있게 되면 작업 구성 방식이 변경됩니다." +" 또한 어떤 결정에 폭 넓은 판단이 필요한지, 어떤 결정은 특정 기술에 대한 반복적인 경험을 통해 개선되는지 질문할 이유를 제공합니다." + +#: website/public/blog/reverse-engineering-models/index.html:757 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For now, the strongest general model has earned its place in my workflow by " +"opening problems I had left unresolved. Those successes are what make me " +"want to examine the work more closely: how much depended on the model's " +"breadth, and how much could be handled by a model trained in this particular" +" kind of reconstruction?" +msgstr "" +"지금까지 가장 강한 범용 모델은 내가 해결하지 못했던 문제를 풀어내며 워크플로에서 자리를 얻었다. 그 성공 때문에 작업을 더 자세히 " +"살펴보고 싶다. 얼마나 많은 부분이 모델의 폭넓은 능력에 의존했을까? 이 종류의 재구성을 전문적으로 학습한 모델은 어디까지 담당할 수 " +"있을까?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:765 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}The repositories give us a way to test the question.{ph1} I would like " +"to use that record to build the comparison and see what we learn." +msgstr "" +"{ph0}저장소는 질문을 테스트할 수 있는 방법을 제공합니다.{ph1} 해당 기록을 사용하여 비교를 구축하고 배운 내용을 확인하고 " +"싶습니다." + +#: website/public/blog/reverse-engineering-models/index.html:775 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Evidence and model attribution" +msgstr "증거 및 모델 속성" + +#: website/public/blog/reverse-engineering-models/index.html:783 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The examples describe historical function-level comparisons recorded in the " +"linked commits. Whole-game runtime checks are separate milestones. Source " +"and matching records were reviewed on 13 October 2026." +msgstr "" +"예제에서는 연결된 커밋에 기록된 기능 수준 비교 기록을 설명합니다. 전체 게임 런타임 확인은 별도의 마일스톤입니다. 출처 및 일치 기록은" +" 2026년 10월 13일에 검토되었습니다." + +#. {ph0}: gpt-dots +#. {ph1}: gpt-web +#: website/public/blog/reverse-engineering-models/index.html:789 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The TH09 label {ph0} identifies GPT-6-Astra. {ph1} changed with the model " +"used at the time, so that label alone does not identify an individual run. " +"Explicit model names in the other cited commits establish their recorded " +"attribution; the histories do not contain uniform per-run cost or effort " +"measurements." +msgstr "" +"TH09 라벨 {ph0}은 GPT-6-Astra을 의미합니다. {ph1}과(와) 관련된 모델은 시간이 지남에 따라 변경되었으므로 " +"라벨만으로는 특정 실행에 사용된 모델을 식별할 수 없습니다. 다른 인용 커밋의 명시적인 모델 이름은 기록된 속성을 제공합니다. 이러한 " +"기록에는 모든 실행에 대한 일관된 비용 또는 노력 측정이 포함되지 않습니다." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#: website/public/blog/reverse-engineering-models/index.html:798 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 temporary lifetimes:{ph1} the {ph2}earlier context review{ph3} and" +" {ph4}complete cold reconstruction{ph5}." +msgstr "" +"{ph0}TH09 임시 객체 수명:{ph1} {ph2}이전 컨텍스트 검토{ph3} 및 {ph4}소스에서 깔끔한 재컴파일을 통해 확인된 " +"완전한 재구성{ph5}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:810 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 Front callback:{ph1} {ph2}non-exact diagnostic regions{ph3}, " +"{ph4}Astra's new graph{ph5} and its {ph6}accepted recompile replay{ph7}." +msgstr "" +"{ph0}TH09 Front 콜백:{ph1} {ph2}일치하지 않는 진단 영역{ph3}, {ph4}Astra의 새로운 제어 흐름 " +"그래프{ph5} 및 {ph6}승인된 재컴파일 확인{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:826 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Compiler knowledge:{ph1} {ph2}TH095 identifier-bucket reuse{ph3}, " +"{ph4}TH04 player source and linker order{ph5} and {ph6}Luna's TH09 enemy-" +"spawn closure{ph7}." +msgstr "" +"{ph0}컴파일러 지식:{ph1} {ph2}TH095 식별자 버킷 동작 재사용{ph3}, {ph4}TH04 플레이어 소스 코드 및 링커 " +"순서{ph5} 및 {ph6}Luna의 TH09 적 생성 기능에 대한 완전한 재구성{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:842 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Checks and diagnostics:{ph1} {ph2}TH08 floating-literal audit{ph3}, " +"{ph4}TH08 numerical investigation{ph5} and {ph6}TH09 replay-save " +"reconstruction{ph7}." +msgstr "" +"{ph0}검사 및 진단:{ph1} {ph2}TH08 부동 소수점 상수 감사{ph3}, {ph4}TH08 수치 조사{ph5} 및 " +"{ph6}TH09 재생 저장 재구성{ph7}." + +#: website/public/blog/reverse-engineering-models/index.html:859 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The proposed specialist and RL comparisons are future experiments. The " +"project histories motivate them; no trained specialist was evaluated for " +"this essay." +msgstr "" +"제안한 특화 모델과 RL의 비교는 앞으로 해 볼 실험이다. 프로젝트 이력이 그 동기를 제공하지만, 이 글에서 학습된 특화 모델을 평가한 " +"것은 아니다." + +#: website/public/blog/reverse-engineering-models/index.html:6 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model? | REA" +msgstr "리버스 엔지니어링에 더 큰 모델이 필요할까? | REA" + +#: website/public/blog/reverse-engineering-models/index.html:7 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What Touhou reconstruction taught me about general models, specialist models" +" and learning to propose better hypotheses from compiler feedback." +msgstr "" +"Touhou 재구성을 통해 일반 모델, 전문 모델 및 컴파일러 피드백을 통해 더 나은 가설을 제안하는 방법에 대해 배운 내용." + +#: website/public/blog/reverse-engineering-models/index.html:240 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "On this page" +msgstr "이 페이지에" + +#: website/public/blog/reverse-engineering-models/index.html:253 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Models and reverse engineering essay" +msgstr "모델 및 리버스 엔지니어링 에세이" + +#: website/public/blog/reverse-engineering-models/index.html:491 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Selected reconstruction experiments" +msgstr "선정된 재구성 실험" + +#: website/public/blog/reverse-engineering-models/index.html:883 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Back to top" +msgstr "맨 위로" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:776 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Detailed evidence notes{ph1} include the recorded comparisons and " +"experiment context." +msgstr "{ph0}상세 증거 기록{ph1}에는 비교 결과와 실험 맥락이 담겨 있다." + +#~ msgctxt "page:blog/index.html" +#~ msgid "By N0zoM1z0 · 13 October 2026 · Draft for review" +#~ msgstr "작성: N0zoM1z0 · 2026년 10월 13일 · 검토용 초안" + #~ msgctxt "page:index.html" #~ msgid "Setup guide" #~ msgstr "설정 가이드" diff --git a/website/locales/messages.pot b/website/locales/messages.pot index e5f6fea82..4d41b3f46 100644 --- a/website/locales/messages.pot +++ b/website/locales/messages.pot @@ -12,6 +12,7 @@ msgid "Skip to content" msgstr "" #: website/public/blog/index.html:113 +#: website/public/blog/reverse-engineering-models/index.html:113 #: website/public/blog/touhou-reconstruction/index.html:113 #: website/public/examples/dino-lab/index.html:114 website/public/faq/index.html:113 #: website/public/first-investigation/index.html:113 @@ -29,6 +30,7 @@ msgid "Get started" msgstr "" #: website/public/blog/index.html:114 +#: website/public/blog/reverse-engineering-models/index.html:114 #: website/public/blog/touhou-reconstruction/index.html:114 #: website/public/examples/dino-lab/index.html:115 website/public/faq/index.html:114 #: website/public/first-investigation/index.html:114 @@ -46,6 +48,7 @@ msgid "Showcases" msgstr "" #: website/public/blog/index.html:115 +#: website/public/blog/reverse-engineering-models/index.html:115 #: website/public/blog/touhou-reconstruction/index.html:115 #: website/public/examples/dino-lab/index.html:116 website/public/faq/index.html:115 #: website/public/first-investigation/index.html:115 @@ -63,6 +66,7 @@ msgid "Blog" msgstr "" #: website/public/blog/index.html:116 +#: website/public/blog/reverse-engineering-models/index.html:116 #: website/public/blog/touhou-reconstruction/index.html:116 #: website/public/examples/dino-lab/index.html:117 website/public/faq/index.html:116 #: website/public/first-investigation/index.html:116 @@ -101,37 +105,62 @@ msgstr "" #: website/public/blog/index.html:200 #, python-brace-format msgctxt "page:blog/index.html" -msgid "Game reconstruction{ph0}Essay" +msgid "Model specialization{ph0}Essay" msgstr "" #: website/public/blog/index.html:202 msgctxt "page:blog/index.html" -msgid "The industrial revolution in reverse engineering" +msgid "Does reverse engineering need a bigger model?" msgstr "" #: website/public/blog/index.html:203 msgctxt "page:blog/index.html" msgid "" +"What Touhou reconstruction suggests about general models, specialist models and " +"learning from compiler feedback." +msgstr "" + +#: website/public/blog/index.html:207 website/public/blog/index.html:222 +msgctxt "page:blog/index.html" +msgid "Read the article" +msgstr "" + +#: website/public/blog/index.html:210 +msgctxt "page:blog/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "" + +#. {ph0}:
+#: website/public/blog/index.html:214 +#, python-brace-format +msgctxt "page:blog/index.html" +msgid "Game reconstruction{ph0}Essay" +msgstr "" + +#: website/public/blog/index.html:216 +msgctxt "page:blog/index.html" +msgid "The industrial revolution in reverse engineering" +msgstr "" + +#: website/public/blog/index.html:217 +msgctxt "page:blog/index.html" +msgid "" "What manual RE and reconstructing TH08, TH095 and TH04 taught me about agents, " "verification and knowledge that compounds across projects." msgstr "" -#: website/public/blog/index.html:208 -msgctxt "page:blog/index.html" -msgid "Read the article" -msgstr "" - -#: website/public/blog/index.html:211 +#: website/public/blog/index.html:225 msgctxt "page:blog/index.html" msgid "By N0zoM1z0 · 10 October 2026" msgstr "" -#: website/public/blog/index.html:222 +#: website/public/blog/index.html:236 +#: website/public/blog/reverse-engineering-models/index.html:875 #: website/public/blog/touhou-reconstruction/index.html:1072 #: website/public/examples/dino-lab/index.html:467 website/public/faq/index.html:565 #: website/public/first-investigation/index.html:515 #: website/public/get-started/index.html:874 -#: website/public/guides/browser/index.html:490 website/public/guides/index.html:274 +#: website/public/guides/browser/index.html:499 website/public/guides/index.html:274 #: website/public/guides/javascript/index.html:495 #: website/public/guides/native/index.html:501 website/public/index.html:1183 #: website/public/showcase/aegis/index.html:800 @@ -143,12 +172,13 @@ msgctxt "shared" msgid "Source" msgstr "" -#: website/public/blog/index.html:223 +#: website/public/blog/index.html:237 +#: website/public/blog/reverse-engineering-models/index.html:876 #: website/public/blog/touhou-reconstruction/index.html:1073 #: website/public/examples/dino-lab/index.html:468 website/public/faq/index.html:566 #: website/public/first-investigation/index.html:516 #: website/public/get-started/index.html:875 -#: website/public/guides/browser/index.html:491 website/public/guides/index.html:275 +#: website/public/guides/browser/index.html:500 website/public/guides/index.html:275 #: website/public/guides/javascript/index.html:496 #: website/public/guides/native/index.html:502 website/public/index.html:1184 #: website/public/showcase/aegis/index.html:801 @@ -160,12 +190,13 @@ msgctxt "shared" msgid "MIT license" msgstr "" -#: website/public/blog/index.html:224 +#: website/public/blog/index.html:238 +#: website/public/blog/reverse-engineering-models/index.html:877 #: website/public/blog/touhou-reconstruction/index.html:1074 #: website/public/examples/dino-lab/index.html:469 website/public/faq/index.html:567 #: website/public/first-investigation/index.html:517 #: website/public/get-started/index.html:876 -#: website/public/guides/browser/index.html:492 website/public/guides/index.html:276 +#: website/public/guides/browser/index.html:501 website/public/guides/index.html:276 #: website/public/guides/javascript/index.html:497 #: website/public/guides/native/index.html:503 website/public/index.html:1185 #: website/public/showcase/aegis/index.html:802 @@ -177,12 +208,13 @@ msgctxt "shared" msgid "Join the community" msgstr "" -#: website/public/blog/index.html:225 +#: website/public/blog/index.html:239 +#: website/public/blog/reverse-engineering-models/index.html:878 #: website/public/blog/touhou-reconstruction/index.html:1075 #: website/public/examples/dino-lab/index.html:470 website/public/faq/index.html:568 #: website/public/first-investigation/index.html:518 #: website/public/get-started/index.html:877 -#: website/public/guides/browser/index.html:493 website/public/guides/index.html:277 +#: website/public/guides/browser/index.html:502 website/public/guides/index.html:277 #: website/public/guides/javascript/index.html:498 #: website/public/guides/native/index.html:504 website/public/index.html:1186 #: website/public/showcase/aegis/index.html:803 @@ -196,12 +228,13 @@ msgstr "" #. {ph0}: -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 +#: website/public/blog/reverse-engineering-models/index.html:883 #: website/public/blog/touhou-reconstruction/index.html:1080 #: website/public/examples/dino-lab/index.html:475 website/public/faq/index.html:573 #: website/public/first-investigation/index.html:523 #: website/public/get-started/index.html:882 -#: website/public/guides/browser/index.html:498 website/public/guides/index.html:282 +#: website/public/guides/browser/index.html:507 website/public/guides/index.html:282 #: website/public/guides/javascript/index.html:503 #: website/public/guides/native/index.html:509 website/public/index.html:1191 #: website/public/showcase/aegis/index.html:808 @@ -227,6 +260,8 @@ msgid "" msgstr "" #: website/public/blog/index.html:75 website/public/blog/index.html:88 +#: website/public/blog/reverse-engineering-models/index.html:75 +#: website/public/blog/reverse-engineering-models/index.html:88 #: website/public/blog/touhou-reconstruction/index.html:75 #: website/public/blog/touhou-reconstruction/index.html:88 #: website/public/examples/dino-lab/index.html:75 @@ -253,6 +288,7 @@ msgid "REA — Find out how software works. Reverse engineering with your coding msgstr "" #: website/public/blog/index.html:108 +#: website/public/blog/reverse-engineering-models/index.html:108 #: website/public/blog/touhou-reconstruction/index.html:108 #: website/public/examples/dino-lab/index.html:109 website/public/faq/index.html:108 #: website/public/first-investigation/index.html:108 @@ -270,6 +306,7 @@ msgid "REA home" msgstr "" #: website/public/blog/index.html:112 +#: website/public/blog/reverse-engineering-models/index.html:112 #: website/public/blog/touhou-reconstruction/index.html:112 #: website/public/examples/dino-lab/index.html:113 website/public/faq/index.html:112 #: website/public/first-investigation/index.html:112 @@ -287,6 +324,7 @@ msgid "Main navigation" msgstr "" #: website/public/blog/index.html:130 +#: website/public/blog/reverse-engineering-models/index.html:130 #: website/public/blog/touhou-reconstruction/index.html:130 #: website/public/examples/dino-lab/index.html:131 website/public/faq/index.html:130 #: website/public/first-investigation/index.html:130 @@ -304,6 +342,7 @@ msgid "REA on GitHub" msgstr "" #: website/public/blog/index.html:150 +#: website/public/blog/reverse-engineering-models/index.html:150 #: website/public/blog/touhou-reconstruction/index.html:150 #: website/public/examples/dino-lab/index.html:151 website/public/faq/index.html:150 #: website/public/first-investigation/index.html:150 @@ -321,6 +360,7 @@ msgid "Join REA on Discord" msgstr "" #: website/public/blog/index.html:169 +#: website/public/blog/reverse-engineering-models/index.html:169 #: website/public/blog/touhou-reconstruction/index.html:169 #: website/public/examples/dino-lab/index.html:170 website/public/faq/index.html:169 #: website/public/first-investigation/index.html:169 @@ -337,11 +377,1005 @@ msgctxt "shared" msgid "Follow REA on X" msgstr "" -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 msgctxt "page:blog/index.html" msgid "Back to top" msgstr "" +#: website/public/blog/reverse-engineering-models/index.html:103 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Skip to content" +msgstr "" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:194 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Home{ph1} / {ph2}Blog{ph3} / Models and RE" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:197 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Essay" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:198 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:199 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:201 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Takeaway" +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:203 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Model capability changes the hypotheses an agent can find.{ph1} Later sessions " +"have solved problems that earlier work left unresolved." +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:211 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A familiar compiler gives specialization something concrete to learn.{ph1} Some " +"exact matches depend on narrow, reusable source patterns." +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:218 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Give repository knowledge a fair comparison.{ph1} A smaller model with good " +"retrieval may already know enough to make progress." +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:223 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Train the next useful experiment, and test its oracle.{ph1} The value of feedback" +" depends on what the checker actually verifies." +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:230 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Measure accepted reconstruction against time and cost.{ph1} That is the result I " +"want from the whole agent." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:241 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "In this article" +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:243 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What better models changed{ph1}" +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:244 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Why specialize?{ph1}" +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:245 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Where experience lives{ph1}" +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:246 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Learning the next experiment{ph1}" +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:247 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}RE and reinforcement learning{ph1}" +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:248 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}When the oracle is wrong{ph1}" +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:249 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}A comparison worth running{ph1}" +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:250 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What kind of agent?{ph1}" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:258 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What better models changed" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:259 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Since August 2026, I have been exploring how much reverse engineering an agent can " +"carry out with access to a program and a way to check its work. Most of my experience " +"comes from reconstructing games, especially Touhou. As I have changed models, the " +"repositories have kept a record of what each session managed to recover." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:267 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of that work aims at exact reconstruction: we compile the maintained source under" +" a pinned historical toolchain and check whether it reproduces the target function. " +"When a later model succeeds where earlier work stalled, the source changes show what " +"made that progress possible." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:274 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The temporary values had to stay alive together" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:275 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of a TH09 attack function's logic had already been reconstructed, but the " +"compiler still stored temporary results differently from the original. The remaining " +"problem was how long those results needed to stay alive and when their storage could " +"be reused." +msgstr "" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:282 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A {ph0}GPT-6-Sol checkpoint{ph1} recorded this mismatch after testing several ordinary" +" source arrangements. When {ph2}GPT-6.1-Sol returned to the problem{ph3}, it found a " +"useful hypothesis about how long those temporary values needed to stay alive." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:296 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The function produced temporary results and copied them into animation positions. In " +"C++, a temporary result stays alive until the expression using it finishes. Grouping " +"the real operations into one expression kept those results alive together, so the " +"compiler could no longer reuse the same storage for all of them." +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:303 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"One result still shared storage with a value used later for collision checks. Giving " +"that value its own lifetime resolved the remaining overlap. After several targeted " +"experiments, {ph0}a clean rebuild confirmed that the full function matched{ph1}." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:312 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What interests me is how the discovery and the tests worked together. The lifetime " +"hypothesis gave the agent a new direction; compiler feedback helped it refine that " +"hypothesis into a source arrangement that matched." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:318 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A different control-flow graph" +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:319 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A second TH09 function shows why that new direction can matter. Here, the obstacle was" +" how several branches shared their final work. {ph0}GPT-6.1-Sol's earlier work{ph1} " +"got parts of the function to match, but the branches still joined up differently from " +"the original." +msgstr "" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:330 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}GPT-6-Astra changed the surrounding control-flow graph{ph1}. The main paths now " +"led to a shared final update and return. In that source structure, the compiler " +"combined the branches in the same way as the original. The complete function matched, " +"and a {ph2}separate rebuild check confirmed it{ph3}." +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:342 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Changing the representation opened a more productive search space.{ph1} Both " +"cases give me a reason to take general model capability seriously: an agent can keep " +"trying local variations for a long time before finding a better way to express the " +"surrounding work." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:351 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"At the same time, the later sessions started with earlier source and recorded " +"failures. That accumulated work was part of their advantage. The history lets us " +"inspect the discoveries and their checks; a controlled comparison would help separate " +"the effect of the model from the experience it inherited." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:358 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is where the question becomes interesting to me. Some of the progress came from " +"finding a new representation, while much of the remaining work involved testing source" +" forms against a familiar compiler. How much needs broad reasoning ability, and how " +"much could a model learn through repeated experience with that compiler and code?" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:368 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why might a specialist do well?" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:369 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Touhou gives us a concrete reason to expect that experience to help. Much of the game " +"code comes from the same author, ZUN, so recurring habits are a reasonable starting " +"assumption. The repositories back that up with useful examples: related loops and " +"helper arrangements have supplied hypotheses across games." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:376 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those source patterns become more specific when we include the compiler. Under a " +"particular version and set of options, small changes in source can produce repeatable " +"differences in machine code. Some of those differences are surprisingly obscure." +msgstr "" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: Supervisor::LoadDat +#. {ph4}: +#: website/public/blog/reverse-engineering-models/index.html:382 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In TH095, for example, {ph0}even the name used for a local variable could change where" +" VC7.1 stored it{ph1}. Moving the declaration alone did not solve the layout, but an " +"earlier background function had revealed a naming pattern that affected the compiler's" +" choice. That gave us a more promising hypothesis for {ph2}the reconstruction of " +"{ph3}{ph4}." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:396 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reusing that pattern for the file-size variable produced the required layout without " +"adding storage. The full function then matched." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:401 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is the sort of narrow detail a general model may not know when it arrives. A " +"specialist trained on this compiler could encounter many such cases and learn which " +"source variations are worth testing first." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:407 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The opportunity is especially clear in final matching problems where we already " +"understand the behavior. We are then asking which ordinary source form this compiler " +"turns into the observed instructions. Specialization could help the model propose " +"those forms more efficiently, while the target still determines which hypotheses " +"survive." +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:415 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The useful patterns would have to retain their context, though. TH08 uses an earlier " +"Visual C++ version than TH095 and TH09. TH04 uses a different DOS-era compiler, and " +"its {ph0}player reconstruction{ph1} needed different hypotheses about source patterns " +"and how the compiled code was combined into an executable. Shared authorship gives us " +"a prior, but each target tells us whether it applies. Distinguishing those contexts is" +" part of what a specialist would need to learn." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:431 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Where should that experience live?" +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:432 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before deciding to teach those patterns through model training, I would look at the " +"knowledge we already keep in the repository. In the {ph0}previous essay{ph1}, I wrote " +"about it becoming the project's working memory. That memory includes compiler lessons," +" along with the source and failed experiments that made them useful." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:441 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"By retrieving that record, an agent can start with a known result and inspect how we " +"reached it. It can also avoid alternatives that have already failed in the same " +"context. The starting point improves without changing the model's weights." +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:447 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"When reconstructing TH09's enemy-spawn function, a {ph0}GPT-6-Luna-max session{ph1} " +"drew a hypothesis from TH08's corresponding source family. That earlier source " +"suggested how to structure the loop and handle failures within it. TH09's own evidence" +" supplied the game-specific details, and clean rebuilds confirmed the complete match." +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:458 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}\"The model did not know that compiler quirk\" and \"the model could not use it\"" +" are different problems.{ph1} Retrieval may solve the first, but turning that fact " +"into a useful candidate still takes work. And if the candidate fails, the model must " +"decide what the result means for its next attempt. Knowing the relevant pattern is " +"only the beginning of that process." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:468 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For that reason, I would include a smaller general model with good retrieval in any " +"comparison. Giving it the same library of experience would help us identify what " +"training adds: better candidate generation, better decisions after feedback, or both." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:476 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Learning the next useful experiment" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:477 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those decisions after feedback are what I would most like to study. In agentic " +"reconstruction, the model proposes an explanation and tests it against the program. " +"Even when the candidate fails, the result can narrow the remaining possibilities and " +"suggest a more useful next experiment." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:484 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Consider the temporary-results example again. Its final source shows the arrangement " +"that worked, but the route to that source contains more information. Each of the " +"following experiments helped identify what still needed to change." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:497 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Experiment" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:498 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What its result taught us" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:503 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Try a different way to copy the results" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:504 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The copy code changes, but the function still differs." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:509 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Use the returned values directly" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:510 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Copying follows the original pattern, but temporary storage still differs." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:516 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the temporary results alive together" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:517 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"They get separate storage, but one still overlaps a value used later for collision " +"checks." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:523 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the collision-check value separate" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:524 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A clean rebuild confirms the complete match." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:529 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Selected experiments from the attack-function reconstruction above, grouped by what " +"they taught us." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:533 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A training record that keeps this sequence can teach more than the final source alone." +" It connects each proposed change to an observed result and the next decision. Keeping" +" ineffective variations matters too: they tell a later agent which hypotheses have " +"already been tested in that context." +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:540 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}I would want the model to learn how to use feedback.{ph1} A neutral result is a " +"reason to reconsider the variation; a changed mismatch may point to another cause. " +"When local edits stop being informative, the agent should be able to revisit the " +"source representation or inspect the evidence again." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:549 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Learning to make those choices requires autonomy. Sometimes the useful next step is " +"inspecting another caller rather than compiling another edit. A rigid sequence of " +"required rewrites would remove that choice, along with part of the behavior we want " +"the model to learn." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:558 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why RE is interesting for reinforcement learning" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:559 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Bounded reconstruction gives us a practical environment for studying these choices. " +"Once the target and compiler are pinned, many source experiments produce quick, " +"repeatable feedback. The agent can compile a candidate, inspect the mismatch and use " +"that observation to choose its next step." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:566 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reinforcement learning could train that decision-making process through the feedback " +"it receives. A candidate accepted by a defined check gives us a concrete outcome, " +"while the sequence of attempts lets us observe how much work it took to reach it." +msgstr "" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:572 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"There is already research supporting this direction. {ph0}LLM4Decompile{ph1} studies " +"models specialized for decompilation. {ph2}AutoDecompiler{ph3} trains multi-turn " +"refinement using compiler and execution feedback. Their studies focus on functional " +"recovery. For this work, I would use them as a starting point for the related task of " +"finding source that matches a historical compiler's output." +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:584 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To reach such a well-defined task, though, we may first need to establish what a " +"function owns or devise a new runtime measurement. In TH08, for example, a " +"{ph0}collision difference led back to how the original program handled floating-point " +"calculations{ph1}. Before the agent could test a useful correction, it had to work out" +" what to measure." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:595 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This suggests another role for a broadly capable model: helping make the problem " +"checkable in the first place. Once that check exists, we can investigate how far a " +"learned search policy can get with it." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:603 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "When the oracle is wrong" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:604 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Once we train against that check, its coverage becomes part of what the model learns. " +"I call the reference or checker an oracle. Even if it gives a repeatable answer, we " +"still need to understand what that answer covers." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:610 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"TH08 showed how much can hide outside that coverage. Some reconstructed functions " +"passed their exact comparisons with wrong floating constants. The comparator adjusted " +"relocated addresses before checking instructions, but it checked the values behind " +"those addresses only when they had been explicitly recorded for verification. Other " +"literal references could pass without their values being examined." +msgstr "" + +#. {ph0}: +#. {ph1}: +#. {ph2}: 1.0f +#. {ph3}: 32.0f +#: website/public/blog/reverse-engineering-models/index.html:619 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To close that gap, a {ph0}wider audit{ph1} found incorrect constants in functions that" +" had already been accepted. In one calculation, the reconstructed source added {ph2} " +"where the original added {ph3}." +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:629 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A learner rewarded by the old check could learn to produce the wrong " +"reconstruction.{ph1} We had not trained an RL model against this oracle; the observed " +"failure was in the check itself. But using that check as a reward would have made the " +"same blind spot part of the learning environment." +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:639 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A functional test can have a similar blind spot. {ph0}Differential-fuzzing research on" +" LLM decompilers{ph1} has found behavior differences beyond the supplied tests. " +"Passing those tests therefore needs to be read in terms of the cases they exercise, " +"just as an instruction match needs to be read in terms of the bytes and data it " +"checks." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:649 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before using either kind of check as a reward, I would test it with deliberate " +"failures. I would also keep final acceptance outside the candidate's editable source, " +"so the model has to satisfy the check through its reconstruction. Otherwise, something" +" like a byte array copied from the target could satisfy a naive score without " +"recovering useful source." +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:657 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Improving the learner and improving its oracle belong to the same research " +"problem.{ph1} As the learner gets better at optimizing the feedback we give it, " +"understanding that feedback becomes more important too." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:667 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A comparison worth running" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:668 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"With those checks in place, I would start the comparison on a fixed VC7.1 task family." +" First, I would establish baselines for a strong general model and a smaller one. " +"Adding repository knowledge to the smaller model would show what retrieval changes. " +"From there, supervised specialization and an RL-trained version would let us examine " +"what training adds." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:676 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To make those comparisons informative, each model would receive the same target " +"evidence and tools. For the training-specific comparisons, I would keep the base model" +" and available knowledge fixed as well. Repeated trials would help distinguish a " +"reliable search strategy from a lucky candidate." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:683 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would then separate two starting conditions: reconstructing a function from its " +"instructions, and closing the last few differences in maintained source. A specialist " +"might be excellent at the second task while a general model does much of the work " +"needed to reach it. Measuring them separately would reveal that division of work." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:691 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both tasks need genuinely held-out examples. Because related Touhou games share engine" +" code, holding out a game's name would still allow nearly identical functions to " +"appear in training. I would instead hold out source families and keep their accepted " +"answers out of both training and retrieval." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:698 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"On those examples, the outcome would be complete, independently validated matches. " +"Giving each model the same candidate or compile budget would help compare search " +"quality. Measuring elapsed time and total cost, including compilation, would tell us " +"whether that quality produces an efficient workflow. Training costs would need their " +"own accounting too." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:706 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both views matter because a cheaper proposal can become an expensive workflow if it " +"causes more uninformative compiles. Conversely, a slower model may pay for itself by " +"finding a useful experiment sooner. The comparison should capture those costs." +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:712 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Counting accepted results also keeps intermediate progress in perspective. One TH09 " +"{ph0}replay-save candidate{ph1} looked promising because the compiled function had the" +" expected size, but a full comparison still found substantial differences. Changes in " +"how the compiler arranges instructions can also make a candidate look closer without " +"resolving the underlying problem. Such diagnostics can guide the next experiment, but " +"I would validate any reward for \"getting closer\" against the full acceptance check." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:727 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would be interested in the pattern across these results. If a specialist does well " +"within a familiar compiler family, does that advantage survive a toolchain change? And" +" once both models have good retrieval, how much does training still add? The answers " +"would help us decide where each kind of model is most useful." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:736 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What kind of agent does this suggest?" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:737 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"If the models turn out to help at different stages, a useful arrangement might be a " +"broad agent with access to a specialist. Once a task becomes a familiar compiler " +"search, it could ask the specialist for candidates. If the results stop making sense, " +"the broader agent could return to the target evidence or investigate the measurement " +"itself." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:745 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In that arrangement, specialization becomes a tool the agent can choose as the " +"investigation develops. The agent keeps the freedom to change direction that made the " +"earlier breakthroughs possible." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:750 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This follows naturally from the industrial shift I described in the previous essay. " +"Being able to run many more experiments changes how we organize the work. It also " +"gives us a reason to ask which decisions need broad judgment and which improve through" +" repeated experience with a particular craft." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:757 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For now, the strongest general model has earned its place in my workflow by opening " +"problems I had left unresolved. Those successes are what make me want to examine the " +"work more closely: how much depended on the model's breadth, and how much could be " +"handled by a model trained in this particular kind of reconstruction?" +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:765 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}The repositories give us a way to test the question.{ph1} I would like to use " +"that record to build the comparison and see what we learn." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:775 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Evidence and model attribution" +msgstr "" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:776 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Detailed evidence notes{ph1} include the recorded comparisons and experiment " +"context." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:783 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The examples describe historical function-level comparisons recorded in the linked " +"commits. Whole-game runtime checks are separate milestones. Source and matching " +"records were reviewed on 13 October 2026." +msgstr "" + +#. {ph0}: gpt-dots +#. {ph1}: gpt-web +#: website/public/blog/reverse-engineering-models/index.html:789 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The TH09 label {ph0} identifies GPT-6-Astra. {ph1} changed with the model used at the " +"time, so that label alone does not identify an individual run. Explicit model names in" +" the other cited commits establish their recorded attribution; the histories do not " +"contain uniform per-run cost or effort measurements." +msgstr "" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#: website/public/blog/reverse-engineering-models/index.html:798 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 temporary lifetimes:{ph1} the {ph2}earlier context review{ph3} and " +"{ph4}complete cold reconstruction{ph5}." +msgstr "" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:810 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 Front callback:{ph1} {ph2}non-exact diagnostic regions{ph3}, {ph4}Astra's " +"new graph{ph5} and its {ph6}accepted recompile replay{ph7}." +msgstr "" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:826 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Compiler knowledge:{ph1} {ph2}TH095 identifier-bucket reuse{ph3}, {ph4}TH04 " +"player source and linker order{ph5} and {ph6}Luna's TH09 enemy-spawn closure{ph7}." +msgstr "" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:842 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Checks and diagnostics:{ph1} {ph2}TH08 floating-literal audit{ph3}, {ph4}TH08 " +"numerical investigation{ph5} and {ph6}TH09 replay-save reconstruction{ph7}." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:859 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The proposed specialist and RL comparisons are future experiments. The project " +"histories motivate them; no trained specialist was evaluated for this essay." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:6 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model? | REA" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:7 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What Touhou reconstruction taught me about general models, specialist models and " +"learning to propose better hypotheses from compiler feedback." +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:240 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "On this page" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:253 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Models and reverse engineering essay" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:491 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Selected reconstruction experiments" +msgstr "" + +#: website/public/blog/reverse-engineering-models/index.html:883 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Back to top" +msgstr "" + #: website/public/blog/touhou-reconstruction/index.html:103 msgctxt "page:blog/touhou-reconstruction/index.html" msgid "Skip to content" @@ -4299,6 +5333,18 @@ msgid "" "workflow{ph1} to follow files and process boundaries." msgstr "" +#. {ph0}: +#. {ph1}: +#: website/public/guides/browser/index.html:480 +#, python-brace-format +msgctxt "page:guides/browser/index.html" +msgid "" +"For HTTP request and response bodies, use {ph0}Fiddler Everywhere: SAZ captures, MCP " +"and public skills{ph1}. Export SAZ or HAR 1.2 for offline REA capture Evidence, or " +"connect Everywhere's MCP server to inspect captured traffic with your agent." +msgstr "" + #: website/public/guides/browser/index.html:6 msgctxt "page:guides/browser/index.html" msgid "Browser and Network Analysis — REA" @@ -4346,7 +5392,7 @@ msgctxt "page:guides/browser/index.html" msgid "Copy read the loaded scripts" msgstr "" -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 msgctxt "page:guides/browser/index.html" msgid "Back to top" msgstr "" diff --git a/website/locales/pl/messages.po b/website/locales/pl/messages.po index 2c80f47bd..d97cd87f1 100644 --- a/website/locales/pl/messages.po +++ b/website/locales/pl/messages.po @@ -32,18 +32,18 @@ msgid "Thoughts on reverse engineering." msgstr "Myśli o inżynierii odwrotnej." #. {ph0}:
-#: website/public/blog/index.html:200 +#: website/public/blog/index.html:214 #, python-brace-format msgctxt "page:blog/index.html" msgid "Game reconstruction{ph0}Essay" msgstr "Rekonstrukcja gry{ph0}Esej" -#: website/public/blog/index.html:202 +#: website/public/blog/index.html:216 msgctxt "page:blog/index.html" msgid "The industrial revolution in reverse engineering" msgstr "Rewolucja przemysłowa w inżynierii odwrotnej" -#: website/public/blog/index.html:203 +#: website/public/blog/index.html:217 msgctxt "page:blog/index.html" msgid "" "What manual RE and reconstructing TH08, TH095 and TH04 taught me about " @@ -52,25 +52,26 @@ msgstr "" "Co Instrukcja RE i rekonstrukcji TH08, TH095 i TH04 nauczył mnie O agentów, " "weryfikacja i wiedza, że związki całej projektów." -#: website/public/blog/index.html:208 +#: website/public/blog/index.html:207 website/public/blog/index.html:222 msgctxt "page:blog/index.html" msgid "Read the article" msgstr "Przeczytaj artykuł" -#: website/public/blog/index.html:211 +#: website/public/blog/index.html:225 msgctxt "page:blog/index.html" msgid "By N0zoM1z0 · 10 October 2026" msgstr "Przez N0zoM1z0 * 10 października 2026" #. {ph0}: -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 +#: website/public/blog/reverse-engineering-models/index.html:883 #: website/public/blog/touhou-reconstruction/index.html:1080 #: website/public/examples/dino-lab/index.html:475 #: website/public/faq/index.html:573 #: website/public/first-investigation/index.html:523 #: website/public/get-started/index.html:882 -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 #: website/public/guides/index.html:282 #: website/public/guides/javascript/index.html:503 #: website/public/guides/native/index.html:509 website/public/index.html:1191 @@ -100,6 +101,8 @@ msgstr "" "doświadczenia z agentami REA i AI." #: website/public/blog/index.html:75 website/public/blog/index.html:88 +#: website/public/blog/reverse-engineering-models/index.html:75 +#: website/public/blog/reverse-engineering-models/index.html:88 #: website/public/blog/touhou-reconstruction/index.html:75 #: website/public/blog/touhou-reconstruction/index.html:88 #: website/public/examples/dino-lab/index.html:75 @@ -136,6 +139,7 @@ msgstr "" "agentem kodującym." #: website/public/blog/index.html:108 +#: website/public/blog/reverse-engineering-models/index.html:108 #: website/public/blog/touhou-reconstruction/index.html:108 #: website/public/examples/dino-lab/index.html:109 #: website/public/faq/index.html:108 @@ -156,6 +160,7 @@ msgid "REA home" msgstr "Strona główna REA" #: website/public/blog/index.html:112 +#: website/public/blog/reverse-engineering-models/index.html:112 #: website/public/blog/touhou-reconstruction/index.html:112 #: website/public/examples/dino-lab/index.html:113 #: website/public/faq/index.html:112 @@ -176,6 +181,7 @@ msgid "Main navigation" msgstr "Główna nawigacja" #: website/public/blog/index.html:130 +#: website/public/blog/reverse-engineering-models/index.html:130 #: website/public/blog/touhou-reconstruction/index.html:130 #: website/public/examples/dino-lab/index.html:131 #: website/public/faq/index.html:130 @@ -196,6 +202,7 @@ msgid "REA on GitHub" msgstr "REA na GitHub" #: website/public/blog/index.html:150 +#: website/public/blog/reverse-engineering-models/index.html:150 #: website/public/blog/touhou-reconstruction/index.html:150 #: website/public/examples/dino-lab/index.html:151 #: website/public/faq/index.html:150 @@ -216,6 +223,7 @@ msgid "Join REA on Discord" msgstr "Dołącz do REA na Discordzie" #: website/public/blog/index.html:169 +#: website/public/blog/reverse-engineering-models/index.html:169 #: website/public/blog/touhou-reconstruction/index.html:169 #: website/public/examples/dino-lab/index.html:170 #: website/public/faq/index.html:169 @@ -235,7 +243,7 @@ msgctxt "shared" msgid "Follow REA on X" msgstr "Obserwuj REA na X" -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 msgctxt "page:blog/index.html" msgid "Back to top" msgstr "Do góry" @@ -4754,7 +4762,7 @@ msgctxt "page:guides/browser/index.html" msgid "Copy read the loaded scripts" msgstr "Kopiuj przeczytaj załadowane Skrypty" -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 msgctxt "page:guides/browser/index.html" msgid "Back to top" msgstr "Do góry" @@ -11618,6 +11626,7 @@ msgid "Same spacing: 256 ÷ 16 = 16 angle units = 22.5°" msgstr "Ten sam odstęp: 256 ÷ 16 = 16 jednostek kąta = 22,5°" #: website/public/blog/index.html:114 +#: website/public/blog/reverse-engineering-models/index.html:114 #: website/public/blog/touhou-reconstruction/index.html:114 #: website/public/examples/dino-lab/index.html:115 #: website/public/faq/index.html:114 @@ -11638,6 +11647,7 @@ msgid "Showcases" msgstr "Przykłady" #: website/public/blog/index.html:115 +#: website/public/blog/reverse-engineering-models/index.html:115 #: website/public/blog/touhou-reconstruction/index.html:115 #: website/public/examples/dino-lab/index.html:116 #: website/public/faq/index.html:115 @@ -11658,6 +11668,7 @@ msgid "Blog" msgstr "Blog" #: website/public/blog/index.html:113 +#: website/public/blog/reverse-engineering-models/index.html:113 #: website/public/blog/touhou-reconstruction/index.html:113 #: website/public/examples/dino-lab/index.html:114 #: website/public/faq/index.html:113 @@ -11678,6 +11689,7 @@ msgid "Get started" msgstr "Pierwsze kroki" #: website/public/blog/index.html:116 +#: website/public/blog/reverse-engineering-models/index.html:116 #: website/public/blog/touhou-reconstruction/index.html:116 #: website/public/examples/dino-lab/index.html:117 #: website/public/faq/index.html:116 @@ -11697,13 +11709,14 @@ msgctxt "shared" msgid "Guides" msgstr "Poradniki" -#: website/public/blog/index.html:222 +#: website/public/blog/index.html:236 +#: website/public/blog/reverse-engineering-models/index.html:875 #: website/public/blog/touhou-reconstruction/index.html:1072 #: website/public/examples/dino-lab/index.html:467 #: website/public/faq/index.html:565 #: website/public/first-investigation/index.html:515 #: website/public/get-started/index.html:874 -#: website/public/guides/browser/index.html:490 +#: website/public/guides/browser/index.html:499 #: website/public/guides/index.html:274 #: website/public/guides/javascript/index.html:495 #: website/public/guides/native/index.html:501 website/public/index.html:1183 @@ -11717,13 +11730,14 @@ msgctxt "shared" msgid "Source" msgstr "Kod źródłowy" -#: website/public/blog/index.html:223 +#: website/public/blog/index.html:237 +#: website/public/blog/reverse-engineering-models/index.html:876 #: website/public/blog/touhou-reconstruction/index.html:1073 #: website/public/examples/dino-lab/index.html:468 #: website/public/faq/index.html:566 #: website/public/first-investigation/index.html:516 #: website/public/get-started/index.html:875 -#: website/public/guides/browser/index.html:491 +#: website/public/guides/browser/index.html:500 #: website/public/guides/index.html:275 #: website/public/guides/javascript/index.html:496 #: website/public/guides/native/index.html:502 website/public/index.html:1184 @@ -11737,13 +11751,14 @@ msgctxt "shared" msgid "MIT license" msgstr "Licencja MIT" -#: website/public/blog/index.html:224 +#: website/public/blog/index.html:238 +#: website/public/blog/reverse-engineering-models/index.html:877 #: website/public/blog/touhou-reconstruction/index.html:1074 #: website/public/examples/dino-lab/index.html:469 #: website/public/faq/index.html:567 #: website/public/first-investigation/index.html:517 #: website/public/get-started/index.html:876 -#: website/public/guides/browser/index.html:492 +#: website/public/guides/browser/index.html:501 #: website/public/guides/index.html:276 #: website/public/guides/javascript/index.html:497 #: website/public/guides/native/index.html:503 website/public/index.html:1185 @@ -11757,13 +11772,14 @@ msgctxt "shared" msgid "Join the community" msgstr "Dołącz do społeczności" -#: website/public/blog/index.html:225 +#: website/public/blog/index.html:239 +#: website/public/blog/reverse-engineering-models/index.html:878 #: website/public/blog/touhou-reconstruction/index.html:1075 #: website/public/examples/dino-lab/index.html:470 #: website/public/faq/index.html:568 #: website/public/first-investigation/index.html:518 #: website/public/get-started/index.html:877 -#: website/public/guides/browser/index.html:493 +#: website/public/guides/browser/index.html:502 #: website/public/guides/index.html:277 #: website/public/guides/javascript/index.html:498 #: website/public/guides/native/index.html:504 website/public/index.html:1186 @@ -12706,6 +12722,1401 @@ msgctxt "runtime" msgid "Switch to dark mode" msgstr "Przełącz na ciemny motyw" +#. {ph0}: +#. {ph1}: +#: website/public/guides/browser/index.html:480 +#, python-brace-format +msgctxt "page:guides/browser/index.html" +msgid "" +"For HTTP request and response bodies, use {ph0}Fiddler Everywhere: SAZ " +"captures, MCP and public skills{ph1}. Export SAZ or HAR 1.2 for offline REA " +"capture Evidence, or connect Everywhere's MCP server to inspect captured " +"traffic with your agent." +msgstr "" +"Aby zbadać treść żądań i odpowiedzi HTTP, użyj {ph0}Fiddler Everywhere: " +"zapisów SAZ, MCP i publicznych umiejętności{ph1}. Wyeksportuj SAZ lub HAR " +"1.2, aby uzyskać Evidence z zapisu w REA offline, lub połącz serwer MCP " +"Everywhere, aby analizować zapisany ruch z agentem." + +#. {ph0}:
+#: website/public/blog/index.html:200 +#, python-brace-format +msgctxt "page:blog/index.html" +msgid "Model specialization{ph0}Essay" +msgstr "Modele specjalistyczne{ph0}Esej" + +#: website/public/blog/index.html:202 +msgctxt "page:blog/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "Czy inżynieria odwrotna potrzebuje większego modelu?" + +#: website/public/blog/index.html:203 +msgctxt "page:blog/index.html" +msgid "" +"What Touhou reconstruction suggests about general models, specialist models " +"and learning from compiler feedback." +msgstr "" +"Co rekonstrukcja Touhou mówi o modelach ogólnych, modelach specjalistycznych" +" i uczeniu się na podstawie informacji zwrotnych z kompilatora." + +#: website/public/blog/index.html:210 +msgctxt "page:blog/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Przez N0zoM1z0 · 13 października 2026 r" + +#: website/public/blog/reverse-engineering-models/index.html:103 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Skip to content" +msgstr "Przejdź do treści" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:194 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Home{ph1} / {ph2}Blog{ph3} / Models and RE" +msgstr "" +"{ph0}Strona główna{ph1} / {ph2}Blog{ph3} / Modele i inżynieria odwrotna" + +#: website/public/blog/reverse-engineering-models/index.html:197 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Essay" +msgstr "Esej" + +# Reviewed for technical meaning and readability; preserve the distinction +# between historical evidence and proposed experiments. +#: website/public/blog/reverse-engineering-models/index.html:198 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "Czy inżynieria odwrotna potrzebuje większego modelu?" + +#: website/public/blog/reverse-engineering-models/index.html:199 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Przez N0zoM1z0 · 13 października 2026 r" + +#: website/public/blog/reverse-engineering-models/index.html:201 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Takeaway" +msgstr "Najważniejsze" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:203 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Model capability changes the hypotheses an agent can find.{ph1} Later " +"sessions have solved problems that earlier work left unresolved." +msgstr "" +"{ph0}Możliwości modelu wpływają na to, jakie hipotezy może postawić agent " +"AI.{ph1} Późniejsze sesje rozwiązały problemy, które nie zostały rozwiązane " +"wcześniej." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:211 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A familiar compiler gives specialization something concrete to " +"learn.{ph1} Some exact matches depend on narrow, reusable source patterns." +msgstr "" +"{ph0}Znany kompilator oferuje konkretne wzorce do nauki dla " +"wyspecjalizowanego modelu.{ph1} Niektóre dokładne dopasowania kodu " +"maszynowego zależą od konkretnych wzorców kodu źródłowego, które można " +"ponownie wykorzystać." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:218 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Give repository knowledge a fair comparison.{ph1} A smaller model with " +"good retrieval may already know enough to make progress." +msgstr "" +"{ph0}W rzetelnym porównaniu uwzględnij wiedzę o repozytorium.{ph1} Mniejszy " +"model z dobrym wyszukiwaniem wiedzy może już mieć wystarczającą ilość " +"informacji, aby poczynić postępy." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:223 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Train the next useful experiment, and test its oracle.{ph1} The value " +"of feedback depends on what the checker actually verifies." +msgstr "" +"{ph0}Wytrenuj model, aby wybrał kolejny przydatny eksperyment i przetestuj " +"jego moduł weryfikujący.{ph1} Opinia jest użyteczna tylko w takim stopniu, w" +" jakim moduł sprawdzający weryfikuje właściwe rzeczy." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:230 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Measure accepted reconstruction against time and cost.{ph1} That is the" +" result I want from the whole agent." +msgstr "" +"{ph0}Oceń zweryfikowane wyniki rekonstrukcji pod względem czasu i " +"kosztów.{ph1} Właśnie to chcę zmierzyć w przypadku pełnego przepływu pracy " +"agenta AI." + +#: website/public/blog/reverse-engineering-models/index.html:241 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "In this article" +msgstr "W tym artykule" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:243 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What better models changed{ph1}" +msgstr "{ph0}Co zmieniły silniejsze modele{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:244 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Why specialize?{ph1}" +msgstr "{ph0}Dlaczego się specjalizować?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:245 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Where experience lives{ph1}" +msgstr "{ph0}Gdzie należy przechowywać to doświadczenie?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:246 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Learning the next experiment{ph1}" +msgstr "{ph0}Nauka kolejnego przydatnego eksperymentu{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:247 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}RE and reinforcement learning{ph1}" +msgstr "{ph0}Inżynieria odwrotna i uczenie się przez wzmacnianie{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:248 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}When the oracle is wrong{ph1}" +msgstr "{ph0}Gdy osoba sprawdzająca myli się{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:249 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}A comparison worth running{ph1}" +msgstr "{ph0}Porównanie warte przeprowadzenia{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:250 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What kind of agent?{ph1}" +msgstr "{ph0}Jakiego rodzaju agenta AI to sugeruje?{ph1}" + +#: website/public/blog/reverse-engineering-models/index.html:258 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What better models changed" +msgstr "Co zmieniły silniejsze modele" + +#: website/public/blog/reverse-engineering-models/index.html:259 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Since August 2026, I have been exploring how much reverse engineering an " +"agent can carry out with access to a program and a way to check its work. " +"Most of my experience comes from reconstructing games, especially Touhou. As" +" I have changed models, the repositories have kept a record of what each " +"session managed to recover." +msgstr "" +"Od sierpnia 2026 roku badam, ile inżynierii wstecznej może dokonać agent " +"mając dostęp do programu i możliwość sprawdzenia jego działania. Większość " +"mojego doświadczenia pochodzi z rekonstrukcji gier, szczególnie Touhou. " +"Ponieważ zmieniałem modele, repozytoria prowadziły rejestr tego, co udało " +"się odzyskać w każdej sesji." + +#: website/public/blog/reverse-engineering-models/index.html:267 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of that work aims at exact reconstruction: we compile the maintained " +"source under a pinned historical toolchain and check whether it reproduces " +"the target function. When a later model succeeds where earlier work stalled," +" the source changes show what made that progress possible." +msgstr "" +"Duża część tych prac ma na celu dokładną rekonstrukcję: kompilujemy " +"utrzymywane źródło w ramach historycznego zestawu narzędzi ze stałymi " +"wersjami i sprawdzamy, czy odtwarza ono funkcję docelową. Kiedy późniejszy " +"model odniesie sukces tam, gdzie wcześniejsze prace utknęły w martwym " +"punkcie, zmiany źródłowe pokazują, co umożliwiło ten postęp." + +#: website/public/blog/reverse-engineering-models/index.html:274 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The temporary values had to stay alive together" +msgstr "Zmienne tymczasowe wymagały nakładających się okresów istnienia" + +#: website/public/blog/reverse-engineering-models/index.html:275 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of a TH09 attack function's logic had already been reconstructed, but " +"the compiler still stored temporary results differently from the original. " +"The remaining problem was how long those results needed to stay alive and " +"when their storage could be reused." +msgstr "" +"Duża część logiki funkcji ataku TH09 została już zrekonstruowana, ale " +"kompilator nadal przechowywał tymczasowe wyniki inaczej niż oryginał. " +"Pozostałym problemem było to, jak długo wyniki były potrzebne do zachowania " +"żywotności i kiedy można było ponownie wykorzystać ich pamięć." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:282 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A {ph0}GPT-6-Sol checkpoint{ph1} recorded this mismatch after testing " +"several ordinary source arrangements. When {ph2}GPT-6.1-Sol returned to the " +"problem{ph3}, it found a useful hypothesis about how long those temporary " +"values needed to stay alive." +msgstr "" +"{ph0}GPT-6-Sol rekord sesji{ph1} zgłosił tę niezgodność po przetestowaniu " +"kilku zwykłych ustaleń kodu źródłowego. Kiedy {ph2}GPT-6.1-Sol ponownie " +"zbadał problem{ph3}, zaproponował użyteczną hipotezę dotyczącą czasu życia " +"tych zmiennych tymczasowych." + +#: website/public/blog/reverse-engineering-models/index.html:296 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The function produced temporary results and copied them into animation " +"positions. In C++, a temporary result stays alive until the expression using" +" it finishes. Grouping the real operations into one expression kept those " +"results alive together, so the compiler could no longer reuse the same " +"storage for all of them." +msgstr "" +"Funkcja wygenerowała tymczasowe wyniki i skopiowała je do pozycji animacji. " +"W C++ tymczasowy wynik pozostaje aktywny aż do zakończenia wyrażenia, które " +"go używa. Grupowanie rzeczywistych operacji w jedno wyrażenie utrzymywało te" +" wyniki razem, więc kompilator nie mógł już ponownie używać tej samej " +"pamięci dla nich wszystkich." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:303 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"One result still shared storage with a value used later for collision " +"checks. Giving that value its own lifetime resolved the remaining overlap. " +"After several targeted experiments, {ph0}a clean rebuild confirmed that the " +"full function matched{ph1}." +msgstr "" +"Jeden wynik nadal współdzielił pamięć o wartości używanej później do " +"sprawdzania kolizji. Nadanie tej wartości własnego czasu życia rozwiązało " +"pozostałe nakładanie się. Po kilku ukierunkowanych eksperymentach " +"{ph0}czysta rekompilacja ze źródła potwierdziła, że ​​pełna funkcja jest " +"zgodna{ph1}." + +#: website/public/blog/reverse-engineering-models/index.html:312 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What interests me is how the discovery and the tests worked together. The " +"lifetime hypothesis gave the agent a new direction; compiler feedback helped" +" it refine that hypothesis into a source arrangement that matched." +msgstr "" +"Ciekawi mnie, jak odkrycie i testy wzajemnie się wspierały. Hipoteza " +"dotycząca czasu życia obiektów tymczasowych nadała agentowi AI nowy " +"kierunek. Informacje zwrotne od kompilatora pomogły mu udoskonalić tę " +"hipotezę w celu uzyskania pasującego układu kodu źródłowego." + +#: website/public/blog/reverse-engineering-models/index.html:318 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A different control-flow graph" +msgstr "Inny wykres przepływu sterowania" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:319 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A second TH09 function shows why that new direction can matter. Here, the " +"obstacle was how several branches shared their final work. " +"{ph0}GPT-6.1-Sol's earlier work{ph1} got parts of the function to match, but" +" the branches still joined up differently from the original." +msgstr "" +"Druga funkcja TH09 pokazuje, dlaczego ten nowy kierunek może mieć znaczenie." +" Tutaj przeszkodą był sposób, w jaki kilka oddziałów podzieliło się swoją " +"końcową pracą. We wcześniejszej pracy {ph0}GPT-6.1-Sol{ph1} części funkcji " +"pasowały, ale gałęzie nadal łączyły się inaczej niż w oryginale." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:330 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}GPT-6-Astra changed the surrounding control-flow graph{ph1}. The main " +"paths now led to a shared final update and return. In that source structure," +" the compiler combined the branches in the same way as the original. The " +"complete function matched, and a {ph2}separate rebuild check confirmed " +"it{ph3}." +msgstr "" +"{ph0}GPT-6-Astra zmienił otaczający wykres przepływu sterowania{ph1}. Główne" +" ścieżki prowadziły teraz do wspólnej ostatecznej aktualizacji i powrotu. W " +"tej strukturze źródłowej kompilator połączył gałęzie w taki sam sposób jak " +"oryginał. Pełna funkcja została zgodna i {ph2}potwierdziła to osobna " +"kontrola odbudowy{ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:342 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Changing the representation opened a more productive search space.{ph1}" +" Both cases give me a reason to take general model capability seriously: an " +"agent can keep trying local variations for a long time before finding a " +"better way to express the surrounding work." +msgstr "" +"{ph0}Zmiana reprezentacji otworzyła bardziej produktywną przestrzeń " +"poszukiwań.{ph1} Obydwa przypadki dają mi powód, aby poważnie traktować " +"możliwości modelu ogólnego: agent może przez długi czas testować lokalne " +"odmiany, zanim znajdzie lepszy sposób na wyrażenie otaczającej pracy." + +#: website/public/blog/reverse-engineering-models/index.html:351 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"At the same time, the later sessions started with earlier source and " +"recorded failures. That accumulated work was part of their advantage. The " +"history lets us inspect the discoveries and their checks; a controlled " +"comparison would help separate the effect of the model from the experience " +"it inherited." +msgstr "" +"Jednocześnie późniejsze sesje rozpoczynały się od wcześniejszych źródeł i " +"odnotowanych awarii. Ta skumulowana praca była częścią ich przewagi. " +"Historia pozwala nam na wgląd w odkrycia i ich sprawdzenie; kontrolowane " +"porównanie pomogłoby oddzielić wpływ modelu od odziedziczonego przez niego " +"doświadczenia." + +#: website/public/blog/reverse-engineering-models/index.html:358 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is where the question becomes interesting to me. Some of the progress " +"came from finding a new representation, while much of the remaining work " +"involved testing source forms against a familiar compiler. How much needs " +"broad reasoning ability, and how much could a model learn through repeated " +"experience with that compiler and code?" +msgstr "" +"Właśnie dlatego to pytanie jest dla mnie interesujące. Pewien postęp " +"nastąpił po znalezieniu nowej reprezentacji. Większość pozostałej pracy " +"polegała na testowaniu różnych rozwiązań kodu źródłowego w porównaniu ze " +"znanym kompilatorem. Ile z tego wymaga szerokiej zdolności rozumowania? Ile " +"model może się nauczyć, wielokrotnie pracując z tym kompilatorem i kodem?" + +#: website/public/blog/reverse-engineering-models/index.html:368 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why might a specialist do well?" +msgstr "" +"Dlaczego wyspecjalizowany model sztucznej inteligencji może się sprawdzić?" + +#: website/public/blog/reverse-engineering-models/index.html:369 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Touhou gives us a concrete reason to expect that experience to help. Much of" +" the game code comes from the same author, ZUN, so recurring habits are a " +"reasonable starting assumption. The repositories back that up with useful " +"examples: related loops and helper arrangements have supplied hypotheses " +"across games." +msgstr "" +"Touhou daje nam konkretny powód, aby oczekiwać, że to doświadczenie pomoże. " +"Duża część kodu gry pochodzi od tego samego autora, ZUN, więc powtarzające " +"się nawyki są rozsądnym założeniem wyjściowym. Repozytoria potwierdzają to " +"przydatnymi przykładami: powiązane pętle i układy pomocnicze dostarczyły " +"hipotez w różnych grach." + +#: website/public/blog/reverse-engineering-models/index.html:376 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those source patterns become more specific when we include the compiler. " +"Under a particular version and set of options, small changes in source can " +"produce repeatable differences in machine code. Some of those differences " +"are surprisingly obscure." +msgstr "" +"Te wzorce źródłowe stają się bardziej szczegółowe, gdy dołączymy kompilator." +" W przypadku określonej wersji i zestawu opcji niewielkie zmiany w źródle " +"mogą powodować powtarzalne różnice w kodzie maszynowym. Niektóre z tych " +"różnic są zaskakująco niejasne." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: Supervisor::LoadDat +#. {ph4}: +#: website/public/blog/reverse-engineering-models/index.html:382 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In TH095, for example, {ph0}even the name used for a local variable could " +"change where VC7.1 stored it{ph1}. Moving the declaration alone did not " +"solve the layout, but an earlier background function had revealed a naming " +"pattern that affected the compiler's choice. That gave us a more promising " +"hypothesis for {ph2}the reconstruction of {ph3}{ph4}." +msgstr "" +"Na przykład w TH095 {ph0}nawet nazwa zmiennej lokalnej może mieć wpływ na " +"miejsce przechowywania jej przez VC7.1{ph1}. Przeniesienie deklaracji nie " +"rozwiązało układu przechowywania. Jednak wcześniejsza funkcja tła ujawniła " +"wzorzec nazewnictwa, który wpłynął na wybór kompilatora. Sugerowało to " +"bardziej obiecującą hipotezę dotyczącą {ph2}rekonstrukcji {ph3}{ph4}." + +#: website/public/blog/reverse-engineering-models/index.html:396 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reusing that pattern for the file-size variable produced the required layout" +" without adding storage. The full function then matched." +msgstr "" +"Ponowne użycie tego wzorca dla zmiennej rozmiaru pliku pozwoliło uzyskać " +"wymagany układ bez dodawania pamięci. Pełna funkcja została wówczas " +"dopasowana." + +#: website/public/blog/reverse-engineering-models/index.html:401 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is the sort of narrow detail a general model may not know when it " +"arrives. A specialist trained on this compiler could encounter many such " +"cases and learn which source variations are worth testing first." +msgstr "" +"Jest to bardzo specyficzny szczegół, o którym model sztucznej inteligencji " +"ogólnego przeznaczenia może nie wiedzieć, kiedy po raz pierwszy napotka " +"problem. Model wyspecjalizowany dla tego kompilatora mógłby zobaczyć wiele " +"takich przypadków i dowiedzieć się, które odmiany kodu źródłowego warto " +"przetestować w pierwszej kolejności." + +#: website/public/blog/reverse-engineering-models/index.html:407 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The opportunity is especially clear in final matching problems where we " +"already understand the behavior. We are then asking which ordinary source " +"form this compiler turns into the observed instructions. Specialization " +"could help the model propose those forms more efficiently, while the target " +"still determines which hypotheses survive." +msgstr "" +"Możliwość jest szczególnie wyraźna w przypadku końcowych problemów z " +"dopasowywaniem, w których już rozumiemy zachowanie. Następnie pytamy, jaki " +"zwykły układ kodu źródłowego ten kompilator przekształca w obserwowane " +"instrukcje. Specjalizacja może pomóc modelowi w skuteczniejszym proponowaniu" +" tych form, podczas gdy cel nadal określa, które hipotezy przetrwają." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:415 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The useful patterns would have to retain their context, though. TH08 uses an" +" earlier Visual C++ version than TH095 and TH09. TH04 uses a different DOS-" +"era compiler, and its {ph0}player reconstruction{ph1} needed different " +"hypotheses about source patterns and how the compiled code was combined into" +" an executable. Shared authorship gives us a prior, but each target tells us" +" whether it applies. Distinguishing those contexts is part of what a " +"specialist would need to learn." +msgstr "" +"Przydatne wzorce muszą zachować swój kontekst. TH08 używa wcześniejszej " +"wersji Visual C++ niż TH095 i TH09. TH04 używa innego kompilatora z ery " +"DOS-u. Do jego {ph0}rekonstrukcji kodu gracza{ph1} potrzebne były różne " +"hipotezy dotyczące wzorców kodu źródłowego i sposobu łączenia skompilowanego" +" kodu z plikiem wykonywalnym. Posiadanie tego samego autora daje nam wstępne" +" oczekiwania, ale każdy cel mówi nam, czy ma to zastosowanie. Aby rozróżnić " +"te konteksty, potrzebny byłby wyspecjalizowany model." + +#: website/public/blog/reverse-engineering-models/index.html:431 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Where should that experience live?" +msgstr "Gdzie należy przechowywać to doświadczenie?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:432 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before deciding to teach those patterns through model training, I would look" +" at the knowledge we already keep in the repository. In the {ph0}previous " +"essay{ph1}, I wrote about it becoming the project's working memory. That " +"memory includes compiler lessons, along with the source and failed " +"experiments that made them useful." +msgstr "" +"Zanim zdecydowałbym się uczyć tych wzorców poprzez szkolenie modelowe, " +"sprawdziłbym wiedzę, którą już przechowujemy w repozytorium. W " +"{ph0}poprzednim eseju{ph1} pisałem, że stanie się ona pamięcią roboczą " +"projektu. Pamięć ta obejmuje lekcje kompilatora wraz ze źródłem i nieudanymi" +" eksperymentami, które uczyniły je użytecznymi." + +#: website/public/blog/reverse-engineering-models/index.html:441 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"By retrieving that record, an agent can start with a known result and " +"inspect how we reached it. It can also avoid alternatives that have already " +"failed in the same context. The starting point improves without changing the" +" model's weights." +msgstr "" +"Pobierając ten rekord, agent może zacząć od znanego wyniku i sprawdzić, w " +"jaki sposób do niego dotarliśmy. Można również uniknąć alternatyw, które już" +" zawiodły w tym samym kontekście. Punkt początkowy poprawia się bez zmiany " +"przeszkolonych parametrów modelu." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:447 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"When reconstructing TH09's enemy-spawn function, a {ph0}GPT-6-Luna-max " +"session{ph1} drew a hypothesis from TH08's corresponding source family. That" +" earlier source suggested how to structure the loop and handle failures " +"within it. TH09's own evidence supplied the game-specific details, and clean" +" rebuilds confirmed the complete match." +msgstr "" +"Podczas rekonstrukcji funkcji TH09 służącej do tworzenia wrogów, podczas " +"{ph0}GPT-6-Luna-max sesji{ph1} przyjęto hipotezę z powiązanego kodu " +"źródłowego TH08. Ten wcześniejszy kod sugerował, jak zorganizować pętlę i " +"poradzić sobie z błędami w niej występującymi. Zeznania TH09 dostarczyły " +"szczegółów specyficznych dla gry. Przebudowa ze źródła potwierdziła, że " +"​​cała funkcja jest zgodna." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:458 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}\"The model did not know that compiler quirk\" and \"the model could " +"not use it\" are different problems.{ph1} Retrieval may solve the first, but" +" turning that fact into a useful candidate still takes work. And if the " +"candidate fails, the model must decide what the result means for its next " +"attempt. Knowing the relevant pattern is only the beginning of that process." +msgstr "" +"{ph0}„Model nie wiedział, że dziwactwo kompilatora” i „model nie mógł go " +"użyć” to różne problemy.{ph1} Pobranie może rozwiązać pierwszy, ale " +"przekształcenie tego faktu w użyteczną kandydaturę nadal wymaga pracy. A " +"jeśli potencjalny kod źródłowy zawiedzie, model musi zdecydować, co oznacza " +"wynik przy następnej próbie. Znajomość odpowiedniego wzorca to dopiero " +"początek tego procesu." + +#: website/public/blog/reverse-engineering-models/index.html:468 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For that reason, I would include a smaller general model with good retrieval" +" in any comparison. Giving it the same library of experience would help us " +"identify what training adds: better candidate generation, better decisions " +"after feedback, or both." +msgstr "" +"Z tego powodu do każdego porównania uwzględniłbym mniejszy model sztucznej " +"inteligencji ogólnego przeznaczenia z dobrym wyszukiwaniem. Zapewnienie tej " +"samej biblioteki doświadczeń pomogłoby nam określić, co wnosi szkolenie: " +"lepsze generowanie kandydatów, lepsze decyzje po otrzymaniu informacji " +"zwrotnej lub jedno i drugie." + +#: website/public/blog/reverse-engineering-models/index.html:476 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Learning the next useful experiment" +msgstr "Nauka kolejnego przydatnego eksperymentu" + +#: website/public/blog/reverse-engineering-models/index.html:477 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those decisions after feedback are what I would most like to study. In " +"agentic reconstruction, the model proposes an explanation and tests it " +"against the program. Even when the candidate fails, the result can narrow " +"the remaining possibilities and suggest a more useful next experiment." +msgstr "" +"Najbardziej chciałbym się uczyć tych decyzji po otrzymaniu informacji " +"zwrotnej. W rekonstrukcji agentowej model proponuje wyjaśnienie i testuje je" +" w porównaniu z programem. Nawet jeśli potencjalny kod źródłowy zawiedzie, " +"wynik może zawęzić pozostałe możliwości i zasugerować bardziej przydatny " +"następny eksperyment." + +#: website/public/blog/reverse-engineering-models/index.html:484 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Consider the temporary-results example again. Its final source shows the " +"arrangement that worked, but the route to that source contains more " +"information. Each of the following experiments helped identify what still " +"needed to change." +msgstr "" +"Rozważmy ponownie przykład wyników tymczasowych. Ostateczne źródło pokazuje " +"układ, który zadziałał, ale trasa do tego źródła zawiera więcej informacji. " +"Każdy z poniższych eksperymentów pomógł określić, co należy jeszcze zmienić." + +#: website/public/blog/reverse-engineering-models/index.html:497 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Experiment" +msgstr "Eksperyment" + +#: website/public/blog/reverse-engineering-models/index.html:498 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What its result taught us" +msgstr "Czego nauczył nas jego wynik" + +#: website/public/blog/reverse-engineering-models/index.html:503 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Try a different way to copy the results" +msgstr "Wypróbuj inny sposób kopiowania wyników" + +#: website/public/blog/reverse-engineering-models/index.html:504 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The copy code changes, but the function still differs." +msgstr "Kod kopiowania zmienia się, ale funkcja nadal jest inna." + +#: website/public/blog/reverse-engineering-models/index.html:509 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Use the returned values directly" +msgstr "Użyj bezpośrednio zwróconych wartości" + +#: website/public/blog/reverse-engineering-models/index.html:510 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Copying follows the original pattern, but temporary storage still differs." +msgstr "" +"Kopiowanie odbywa się według oryginalnego wzorca, ale przechowywanie " +"tymczasowe nadal się różni." + +#: website/public/blog/reverse-engineering-models/index.html:516 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the temporary results alive together" +msgstr "Zachowaj ważność wszystkich zmiennych tymczasowych w tym samym czasie" + +#: website/public/blog/reverse-engineering-models/index.html:517 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"They get separate storage, but one still overlaps a value used later for " +"collision checks." +msgstr "" +"Dostają oddzielną pamięć, ale jedna z nich nadal pokrywa się z wartością " +"używaną później do sprawdzania kolizji." + +#: website/public/blog/reverse-engineering-models/index.html:523 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the collision-check value separate" +msgstr "Zachowaj wartość kontroli kolizji oddzielnie" + +#: website/public/blog/reverse-engineering-models/index.html:524 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A clean rebuild confirms the complete match." +msgstr "Czysta rekompilacja ze źródła potwierdza pełne dopasowanie." + +#: website/public/blog/reverse-engineering-models/index.html:529 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Selected experiments from the attack-function reconstruction above, grouped " +"by what they taught us." +msgstr "" +"Wybrane eksperymenty z powyższej rekonstrukcji funkcji ataku, pogrupowane " +"według tego, czego nas nauczyły." + +#: website/public/blog/reverse-engineering-models/index.html:533 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A training record that keeps this sequence can teach more than the final " +"source alone. It connects each proposed change to an observed result and the" +" next decision. Keeping ineffective variations matters too: they tell a " +"later agent which hypotheses have already been tested in that context." +msgstr "" +"Zapis szkoleniowy zachowujący tę sekwencję może nauczyć więcej niż samo " +"ostateczne źródło. Łączy każdą proponowaną zmianę z zaobserwowanym " +"rezultatem i następną decyzją. Utrzymywanie nieefektywnych odmian również ma" +" znaczenie: informują późniejszego agenta, które hipotezy zostały już " +"przetestowane w tym kontekście." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:540 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}I would want the model to learn how to use feedback.{ph1} A neutral " +"result is a reason to reconsider the variation; a changed mismatch may point" +" to another cause. When local edits stop being informative, the agent should" +" be able to revisit the source representation or inspect the evidence again." +msgstr "" +"{ph0}Chciałbym, żeby model nauczył się korzystać ze informacji " +"zwrotnej.{ph1} Neutralny wynik jest powodem do ponownego rozważenia odmiany;" +" zmieniona niezgodność może wskazywać na inną przyczynę. Kiedy lokalne " +"zmiany przestaną mieć charakter informacyjny, agent powinien mieć możliwość " +"ponownego przejrzenia reprezentacji źródłowej lub ponownego sprawdzenia " +"dowodów." + +#: website/public/blog/reverse-engineering-models/index.html:549 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Learning to make those choices requires autonomy. Sometimes the useful next " +"step is inspecting another caller rather than compiling another edit. A " +"rigid sequence of required rewrites would remove that choice, along with " +"part of the behavior we want the model to learn." +msgstr "" +"Nauka dokonywania takich wyborów wymaga autonomii. Czasami przydatnym " +"następnym krokiem jest sprawdzenie innego obiektu wywołującego, zamiast " +"kompilowania kolejnej edycji. Sztywna sekwencja wymaganych przeróbek usunie " +"ten wybór wraz z częścią zachowań, których ma się nauczyć model." + +#: website/public/blog/reverse-engineering-models/index.html:558 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why RE is interesting for reinforcement learning" +msgstr "" +"Dlaczego inżynieria odwrotna jest interesująca w uczeniu się przez " +"wzmacnianie" + +#: website/public/blog/reverse-engineering-models/index.html:559 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Bounded reconstruction gives us a practical environment for studying these " +"choices. Once the target and compiler are pinned, many source experiments " +"produce quick, repeatable feedback. The agent can compile a candidate, " +"inspect the mismatch and use that observation to choose its next step." +msgstr "" +"Rekonstrukcja o ustalonym zakresie daje praktyczne środowisko do badania " +"tych decyzji. Po ustaleniu wersji celu i kompilatora wiele eksperymentów z " +"kodem źródłowym zapewnia szybką, powtarzalną informację zwrotną. Agent może " +"skompilować kandydata, sprawdzić różnice i wybrać następny krok." + +#: website/public/blog/reverse-engineering-models/index.html:566 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reinforcement learning could train that decision-making process through the " +"feedback it receives. A candidate accepted by a defined check gives us a " +"concrete outcome, while the sequence of attempts lets us observe how much " +"work it took to reach it." +msgstr "" +"Uczenie się przez wzmacnianie mogłoby szkolić proces podejmowania decyzji na" +" podstawie otrzymywanych informacji zwrotnych. Kandydat przyjęty przez " +"określony sprawdzian daje nam konkretny wynik, a sekwencja prób pozwala " +"zaobserwować, ile pracy wymagało jego osiągnięcie." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:572 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"There is already research supporting this direction. {ph0}LLM4Decompile{ph1}" +" studies models specialized for decompilation. {ph2}AutoDecompiler{ph3} " +"trains multi-turn refinement using compiler and execution feedback. Their " +"studies focus on functional recovery. For this work, I would use them as a " +"starting point for the related task of finding source that matches a " +"historical compiler's output." +msgstr "" +"Istnieją już badania potwierdzające ten kierunek. {ph0}LLM4Decompile{ph1} " +"bada modele wyspecjalizowane w dekompilacji. {ph2}AutoDecompiler{ph3} " +"trenuje udoskonalanie w wielu turach, korzystając z informacji zwrotnych od " +"kompilatora i wykonania. Ich badania skupiają się na odzyskaniu funkcji. W " +"tej pracy użyłbym ich jako punktu wyjścia do powiązanego zadania " +"polegającego na znalezieniu źródła, które pasuje do danych wyjściowych " +"historycznego kompilatora." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:584 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To reach such a well-defined task, though, we may first need to establish " +"what a function owns or devise a new runtime measurement. In TH08, for " +"example, a {ph0}collision difference led back to how the original program " +"handled floating-point calculations{ph1}. Before the agent could test a " +"useful correction, it had to work out what to measure." +msgstr "" +"Aby jednak osiągnąć tak dobrze zdefiniowane zadanie, być może będziemy " +"musieli najpierw ustalić, jakie zasoby posiada funkcja, lub opracować nowy " +"sposób obserwowania zachowania programu podczas wykonania. Na przykład w " +"TH08 {ph0}różnica kolizji doprowadziła do tego, jak oryginalny program " +"obsługiwał obliczenia zmiennoprzecinkowe{ph1}. Zanim agent mógł przetestować" +" użyteczną korektę, musiał ustalić, co należy zmierzyć." + +#: website/public/blog/reverse-engineering-models/index.html:595 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This suggests another role for a broadly capable model: helping make the " +"problem checkable in the first place. Once that check exists, we can " +"investigate how far a learned search policy can get with it." +msgstr "" +"Sugeruje to inną rolę modelu o szerokich możliwościach: przede wszystkim " +"pomoc w sprawdzeniu problemu. Gdy już to sprawdzimy, będziemy mogli " +"sprawdzić, jak daleko może zajść w tym przypadku wyuczona polityka " +"wyszukiwania." + +#: website/public/blog/reverse-engineering-models/index.html:603 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "When the oracle is wrong" +msgstr "Gdy osoba sprawdzająca myli się" + +#: website/public/blog/reverse-engineering-models/index.html:604 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Once we train against that check, its coverage becomes part of what the " +"model learns. I call the reference or checker an oracle. Even if it gives a " +"repeatable answer, we still need to understand what that answer covers." +msgstr "" +"Gdy używamy sprawdzania podczas uczenia, jego zakres staje się częścią " +"środowiska uczenia. Taki wzorzec lub moduł sprawdzający nazywam wyrocznią. " +"Nawet powtarzalną odpowiedź trzeba rozumieć w granicach tego, co " +"rzeczywiście sprawdza." + +#: website/public/blog/reverse-engineering-models/index.html:610 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"TH08 showed how much can hide outside that coverage. Some reconstructed " +"functions passed their exact comparisons with wrong floating constants. The " +"comparator adjusted relocated addresses before checking instructions, but it" +" checked the values behind those addresses only when they had been " +"explicitly recorded for verification. Other literal references could pass " +"without their values being examined." +msgstr "" +"TH08 pokazał, jak wiele może ukryć się poza tym zasięgiem. Niektóre " +"zrekonstruowane funkcje przeszły dokładne porównania z nieprawidłowymi " +"stałymi zmiennoprzecinkowymi. Komparator korygował przeniesione adresy przed" +" sprawdzeniem instrukcji, ale sprawdzał wartości za tymi adresami tylko " +"wtedy, gdy zostały one wyraźnie zapisane w celu weryfikacji. Inne dosłowne " +"odniesienia można pominąć bez sprawdzania ich wartości." + +#. {ph0}: +#. {ph1}: +#. {ph2}: 1.0f +#. {ph3}: 32.0f +#: website/public/blog/reverse-engineering-models/index.html:619 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To close that gap, a {ph0}wider audit{ph1} found incorrect constants in " +"functions that had already been accepted. In one calculation, the " +"reconstructed source added {ph2} where the original added {ph3}." +msgstr "" +"Aby zaradzić tej luce, {ph0}szerszy audyt{ph1} wykrył nieprawidłowe stałe w " +"funkcjach, które przeszły już weryfikację. W jednym obliczeniu do " +"zrekonstruowanego kodu źródłowego dodano {ph2}, podczas gdy w oryginale " +"dodano {ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:629 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A learner rewarded by the old check could learn to produce the wrong " +"reconstruction.{ph1} We had not trained an RL model against this oracle; the" +" observed failure was in the check itself. But using that check as a reward " +"would have made the same blind spot part of the learning environment." +msgstr "" +"{ph0}Model nagrodzony starym testem weryfikacyjnym mógłby nauczyć się " +"tworzyć niepoprawną rekonstrukcję.{ph1} Nie trenowaliśmy modelu RL przy " +"użyciu tego modułu sprawdzającego. Niepowodzenie, które zaobserwowaliśmy, " +"dotyczyło samego sprawdzenia. Ale gdybyśmy użyli tego jako nagrody, ten sam " +"martwy punkt stałby się częścią środowiska uczenia się." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:639 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A functional test can have a similar blind spot. {ph0}Differential-fuzzing " +"research on LLM decompilers{ph1} has found behavior differences beyond the " +"supplied tests. Passing those tests therefore needs to be read in terms of " +"the cases they exercise, just as an instruction match needs to be read in " +"terms of the bytes and data it checks." +msgstr "" +"Test funkcjonalny może mieć podobny martwy punkt. {ph0}Badania " +"wykorzystujące różnicowe rozmycie w dekompilatorach LLM{ph1} wykazały " +"różnice w zachowaniu poza dostarczonymi testami. Dlatego też musimy rozumieć" +" zdanie egzaminu w kategoriach przypadków, w których występuje. Podobnie " +"dopasowanie instrukcji należy rozumieć w kategoriach bajtów i danych, które " +"sprawdza." + +#: website/public/blog/reverse-engineering-models/index.html:649 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before using either kind of check as a reward, I would test it with " +"deliberate failures. I would also keep final acceptance outside the " +"candidate's editable source, so the model has to satisfy the check through " +"its reconstruction. Otherwise, something like a byte array copied from the " +"target could satisfy a naive score without recovering useful source." +msgstr "" +"Zanim użyję jakiegokolwiek rodzaju kontroli weryfikacyjnej jako nagrody, " +"przetestowałbym ją z celowo błędnymi wynikami. Chciałbym również, aby " +"ostateczna akceptacja była oddzielona od kodu źródłowego, który agent AI " +"może edytować. Model musi przejść weryfikację poprzez poprawną rekonstrukcję" +" programu. W przeciwnym razie zwykłe skopiowanie tablicy bajtów z celu może " +"oszukać naiwny system oceniania bez odzyskiwania przydatnego kodu " +"źródłowego." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:657 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Improving the learner and improving its oracle belong to the same " +"research problem.{ph1} As the learner gets better at optimizing the feedback" +" we give it, understanding that feedback becomes more important too." +msgstr "" +"{ph0}Ulepszanie modelu i ulepszanie jego modułu sprawdzającego należą do " +"tego samego problemu badawczego.{ph1} Im lepiej model optymalizuje " +"otrzymywaną informację zwrotną, tym ważniejsze jest zrozumienie tej " +"informacji." + +#: website/public/blog/reverse-engineering-models/index.html:667 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A comparison worth running" +msgstr "Porównanie warte przeprowadzenia" + +#: website/public/blog/reverse-engineering-models/index.html:668 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"With those checks in place, I would start the comparison on a fixed VC7.1 " +"task family. First, I would establish baselines for a strong general model " +"and a smaller one. Adding repository knowledge to the smaller model would " +"show what retrieval changes. From there, supervised specialization and an " +"RL-trained version would let us examine what training adds." +msgstr "" +"Mając te kontrole, rozpocząłbym porównanie od ustalonej rodziny zadań VC7.1." +" Najpierw ustaliłbym podstawy dla silnego modelu sztucznej inteligencji " +"ogólnego przeznaczenia i mniejszego. Dodanie wiedzy o repozytorium do " +"mniejszego modelu pokazałoby, jakie zmiany w pobieraniu. Następnie " +"nadzorowana specjalizacja i wersja przeszkolona przez RL pozwolą nam " +"sprawdzić, co dodaje szkolenie." + +#: website/public/blog/reverse-engineering-models/index.html:676 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To make those comparisons informative, each model would receive the same " +"target evidence and tools. For the training-specific comparisons, I would " +"keep the base model and available knowledge fixed as well. Repeated trials " +"would help distinguish a reliable search strategy from a lucky candidate." +msgstr "" +"Aby porównania te miały charakter informacyjny, każdy model otrzymałby te " +"same docelowe dowody i narzędzia. W przypadku porównań specyficznych dla " +"szkoleń zachowałbym również stały model podstawowy i dostępną wiedzę. " +"Powtarzane próby pomogłyby odróżnić niezawodną strategię wyszukiwania od " +"potencjalnego rozwiązania, które odniosło sukces przez przypadek." + +#: website/public/blog/reverse-engineering-models/index.html:683 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would then separate two starting conditions: reconstructing a function " +"from its instructions, and closing the last few differences in maintained " +"source. A specialist might be excellent at the second task while a general " +"model does much of the work needed to reach it. Measuring them separately " +"would reveal that division of work." +msgstr "" +"Następnie oddzieliłbym dwa warunki początkowe: rekonstrukcję funkcji na " +"podstawie jej instrukcji i zamknięcie kilku ostatnich różnic w utrzymywanym " +"źródle. Wyspecjalizowany model sztucznej inteligencji może doskonale " +"sprawdzić się w drugim zadaniu, podczas gdy model sztucznej inteligencji " +"ogólnego przeznaczenia wykonuje większość pracy potrzebnej do jego " +"osiągnięcia. Zmierzenie ich osobno ujawniłoby ten podział pracy." + +#: website/public/blog/reverse-engineering-models/index.html:691 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both tasks need genuinely held-out examples. Because related Touhou games " +"share engine code, holding out a game's name would still allow nearly " +"identical functions to appear in training. I would instead hold out source " +"families and keep their accepted answers out of both training and retrieval." +msgstr "" +"Obydwa zadania wymagają przykładów testowych wyłączonych ze szkolenia i " +"wyszukiwania. Powiązane gry Touhou udostępniają kod silnika. Wykluczenie " +"jednej gry z treningu może w dalszym ciągu pozostawić prawie identyczne " +"funkcje w zbiorze treningowym. Zamiast tego wykluczyłbym całe grupy " +"powiązanego kodu źródłowego i trzymał ich zweryfikowane odpowiedzi poza " +"danymi szkoleniowymi i biblioteką pobierania." + +#: website/public/blog/reverse-engineering-models/index.html:698 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"On those examples, the outcome would be complete, independently validated " +"matches. Giving each model the same candidate or compile budget would help " +"compare search quality. Measuring elapsed time and total cost, including " +"compilation, would tell us whether that quality produces an efficient " +"workflow. Training costs would need their own accounting too." +msgstr "" +"W przypadku tych przykładów wynikiem będą kompletne, niezależnie " +"zweryfikowane dopasowania. Równe budżety na propozycje kandydatów lub próby " +"kompilacji pomogłyby w porównaniu jakości wyszukiwania. Pomiar całkowitego " +"czasu i kosztów, w tym kompilacji, pokazałby, czy ta jakość zapewnia " +"efektywny przepływ pracy. Koszty szkoleń należy mierzyć osobno." + +#: website/public/blog/reverse-engineering-models/index.html:706 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both views matter because a cheaper proposal can become an expensive " +"workflow if it causes more uninformative compiles. Conversely, a slower " +"model may pay for itself by finding a useful experiment sooner. The " +"comparison should capture those costs." +msgstr "" +"Obydwa pomiary mają znaczenie. Tania propozycja może prowadzić do " +"kosztownego przepływu pracy, jeśli powoduje więcej kompilacji, które nie " +"dostarczają żadnych przydatnych informacji. I odwrotnie, wolniejszy model " +"może być bardziej ekonomiczny, jeśli wcześniej znajdzie przydatny " +"eksperyment. Porównanie powinno uwzględniać te koszty." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:712 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Counting accepted results also keeps intermediate progress in perspective. " +"One TH09 {ph0}replay-save candidate{ph1} looked promising because the " +"compiled function had the expected size, but a full comparison still found " +"substantial differences. Changes in how the compiler arranges instructions " +"can also make a candidate look closer without resolving the underlying " +"problem. Such diagnostics can guide the next experiment, but I would " +"validate any reward for \"getting closer\" against the full acceptance " +"check." +msgstr "" +"Liczenie zaakceptowanych wyników pozwala również zachować perspektywę " +"postępu pośredniego. Jeden TH09 {ph0}kandydat do zapisania powtórki{ph1} " +"wyglądał obiecująco, ponieważ skompilowana funkcja miała oczekiwany rozmiar," +" ale pełne porównanie nadal wykazało znaczne różnice. Zmiany w sposobie, w " +"jaki kompilator organizuje instrukcje, mogą również sprawić, że kandydat " +"wydaje się bliższy celowi bez rozwiązania podstawowego problemu. Taka " +"diagnostyka może pomóc w kolejnym eksperymencie, ale każdą nagrodę za " +"„zbliżenie się” porównałbym z pełną kontrolą akceptacji." + +#: website/public/blog/reverse-engineering-models/index.html:727 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would be interested in the pattern across these results. If a specialist " +"does well within a familiar compiler family, does that advantage survive a " +"toolchain change? And once both models have good retrieval, how much does " +"training still add? The answers would help us decide where each kind of " +"model is most useful." +msgstr "" +"Byłbym zainteresowany wzorcem tych wyników. Jeśli wyspecjalizowany model " +"sztucznej inteligencji radzi sobie dobrze w znanej rodzinie kompilatorów, " +"czy ta zaleta przetrwa zmianę w zestawie narzędzi? A kiedy oba modele " +"sztucznej inteligencji będą dobrze odzyskiwane, ile jeszcze doda szkolenie? " +"Odpowiedzi pomogłyby nam zdecydować, gdzie każdy rodzaj modelu jest " +"najbardziej przydatny." + +#: website/public/blog/reverse-engineering-models/index.html:736 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What kind of agent does this suggest?" +msgstr "Jakiego rodzaju agenta AI to sugeruje?" + +#: website/public/blog/reverse-engineering-models/index.html:737 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"If the models turn out to help at different stages, a useful arrangement " +"might be a broad agent with access to a specialist. Once a task becomes a " +"familiar compiler search, it could ask the specialist for candidates. If the" +" results stop making sense, the broader agent could return to the target " +"evidence or investigate the measurement itself." +msgstr "" +"Jeśli modele pomagają na różnych etapach, przydatny może być agent ogólny z " +"dostępem do modelu specjalistycznego. Gdy zadanie sprowadza się do znanego " +"problemu z kompilatorem, może poprosić specjalistę o kandydatów. Jeśli " +"wyniki przestają mieć sens, agent ogólny może wrócić do danych celu albo " +"zbadać sam pomiar." + +#: website/public/blog/reverse-engineering-models/index.html:745 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In that arrangement, specialization becomes a tool the agent can choose as " +"the investigation develops. The agent keeps the freedom to change direction " +"that made the earlier breakthroughs possible." +msgstr "" +"W takim układzie specjalizacja staje się narzędziem, które agent może wybrać" +" w miarę rozwoju dochodzenia. Agent zachowuje swobodę zmiany kierunku, co " +"umożliwiło wcześniejsze przełomy." + +#: website/public/blog/reverse-engineering-models/index.html:750 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This follows naturally from the industrial shift I described in the previous" +" essay. Being able to run many more experiments changes how we organize the " +"work. It also gives us a reason to ask which decisions need broad judgment " +"and which improve through repeated experience with a particular craft." +msgstr "" +"Wynika to naturalnie z przemian w przemyśle, które opisałem w poprzednim " +"eseju. Możliwość przeprowadzenia znacznie większej liczby eksperymentów " +"zmienia sposób organizacji pracy. Daje nam to również powód, aby zadać sobie" +" pytanie, które decyzje wymagają szerokiego osądu, a które można poprawić " +"dzięki wielokrotnemu doświadczeniu z konkretnym rzemiosłem." + +#: website/public/blog/reverse-engineering-models/index.html:757 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For now, the strongest general model has earned its place in my workflow by " +"opening problems I had left unresolved. Those successes are what make me " +"want to examine the work more closely: how much depended on the model's " +"breadth, and how much could be handled by a model trained in this particular" +" kind of reconstruction?" +msgstr "" +"Na razie najsilniejszy model ogólny zasłużył na miejsce w moim przepływie " +"pracy, rozwiązując problemy, których wcześniej nie rozwiązałem. Te sukcesy " +"skłaniają mnie do zbadania procesu: jaka część zależała od szerokości jego " +"możliwości, a jaką mógłby wykonać model uczony specjalnie do takiej " +"rekonstrukcji?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:765 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}The repositories give us a way to test the question.{ph1} I would like " +"to use that record to build the comparison and see what we learn." +msgstr "" +"{ph0}Repozytoria umożliwiają nam przetestowanie pytania.{ph1} Chciałbym użyć" +" tego rekordu do stworzenia porównania i zobaczenia, czego się dowiemy." + +#: website/public/blog/reverse-engineering-models/index.html:775 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Evidence and model attribution" +msgstr "Dowody i modelowe przypisanie" + +#: website/public/blog/reverse-engineering-models/index.html:783 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The examples describe historical function-level comparisons recorded in the " +"linked commits. Whole-game runtime checks are separate milestones. Source " +"and matching records were reviewed on 13 October 2026." +msgstr "" +"Przykłady opisują historyczne porównania poziomów funkcji zarejestrowane w " +"połączonych zatwierdzeniach. Sprawdzanie czasu działania całej gry to osobne" +" kamienie milowe. Dokumenty źródłowe i pasujące sprawdzono 13 października " +"2026 r." + +#. {ph0}: gpt-dots +#. {ph1}: gpt-web +#: website/public/blog/reverse-engineering-models/index.html:789 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The TH09 label {ph0} identifies GPT-6-Astra. {ph1} changed with the model " +"used at the time, so that label alone does not identify an individual run. " +"Explicit model names in the other cited commits establish their recorded " +"attribution; the histories do not contain uniform per-run cost or effort " +"measurements." +msgstr "" +"Etykieta TH09 {ph0} oznacza GPT-6-Astra. Model powiązany z {ph1} zmieniał " +"się z biegiem czasu, więc sama etykieta nie może zidentyfikować modelu " +"użytego w konkretnym przebiegu. Wyraźne nazwy modeli w innych cytowanych " +"zatwierdzeniach zapewniają zarejestrowaną atrybucję. Historie te nie " +"obejmują spójnych pomiarów kosztów i wysiłku dla każdego przebiegu." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#: website/public/blog/reverse-engineering-models/index.html:798 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 temporary lifetimes:{ph1} the {ph2}earlier context review{ph3} and" +" {ph4}complete cold reconstruction{ph5}." +msgstr "" +"{ph0}TH09 okresy istnienia obiektów tymczasowych:{ph1} {ph2}wcześniejszy " +"przegląd kontekstu{ph3} i {ph4}pełna rekonstrukcja zweryfikowana przez " +"czystą rekompilację ze źródła{ph5}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:810 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 Front callback:{ph1} {ph2}non-exact diagnostic regions{ph3}, " +"{ph4}Astra's new graph{ph5} and its {ph6}accepted recompile replay{ph7}." +msgstr "" +"{ph0}Callback Front w TH09:{ph1} {ph2}obszary diagnostyczne bez pełnej " +"zgodności{ph3}, {ph4}nowy graf przepływu sterowania Astry{ph5} i " +"{ph6}zaakceptowane sprawdzenie ponownej kompilacji{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:826 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Compiler knowledge:{ph1} {ph2}TH095 identifier-bucket reuse{ph3}, " +"{ph4}TH04 player source and linker order{ph5} and {ph6}Luna's TH09 enemy-" +"spawn closure{ph7}." +msgstr "" +"{ph0}Wiedza o kompilatorze:{ph1} {ph2}ponowne użycie zachowania grup " +"identyfikatorów w TH095{ph3}, {ph4}kod gracza TH04 i kolejność " +"linkowania{ph5} oraz {ph6}pełna rekonstrukcja funkcji tworzenia przeciwników" +" TH09 przez Lunę{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:842 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Checks and diagnostics:{ph1} {ph2}TH08 floating-literal audit{ph3}, " +"{ph4}TH08 numerical investigation{ph5} and {ph6}TH09 replay-save " +"reconstruction{ph7}." +msgstr "" +"{ph0}Kontrola i diagnostyka:{ph1} {ph2}TH08 audyt stałej " +"zmiennoprzecinkowej{ph3}, {ph4}TH08 badanie numeryczne{ph5} i {ph6}TH09 " +"rekonstrukcja z zachowaniem powtórki{ph7}." + +#: website/public/blog/reverse-engineering-models/index.html:859 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The proposed specialist and RL comparisons are future experiments. The " +"project histories motivate them; no trained specialist was evaluated for " +"this essay." +msgstr "" +"Proponowane porównania wyspecjalizowanych modeli i szkolenia RL to przyszłe " +"eksperymenty. Motywują ich historie projektów. Na potrzeby tego eseju nie " +"oceniano żadnego wyszkolonego specjalistycznego modelu." + +#: website/public/blog/reverse-engineering-models/index.html:6 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model? | REA" +msgstr "Czy inżynieria odwrotna potrzebuje większego modelu? | REA" + +#: website/public/blog/reverse-engineering-models/index.html:7 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What Touhou reconstruction taught me about general models, specialist models" +" and learning to propose better hypotheses from compiler feedback." +msgstr "" +"Czego rekonstrukcja Touhou nauczyła mnie o modelach ogólnych, modelach " +"specjalistycznych i nauce proponowania lepszych hipotez na podstawie " +"informacji zwrotnych od kompilatora." + +#: website/public/blog/reverse-engineering-models/index.html:240 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "On this page" +msgstr "Na tej stronie" + +#: website/public/blog/reverse-engineering-models/index.html:253 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Models and reverse engineering essay" +msgstr "Modele i esej o inżynierii odwrotnej" + +#: website/public/blog/reverse-engineering-models/index.html:491 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Selected reconstruction experiments" +msgstr "Wybrane eksperymenty rekonstrukcyjne" + +#: website/public/blog/reverse-engineering-models/index.html:883 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Back to top" +msgstr "Do góry" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:776 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Detailed evidence notes{ph1} include the recorded comparisons and " +"experiment context." +msgstr "" +"{ph0}Szczegółowe notatki o dowodach{ph1} zawierają zapisane porównania i " +"kontekst eksperymentów." + +#~ msgctxt "page:blog/index.html" +#~ msgid "By N0zoM1z0 · 13 October 2026 · Draft for review" +#~ msgstr "Autor: N0zoM1z0 · 13 października 2026 · Szkic do przejrzenia" + #~ msgctxt "page:index.html" #~ msgid "Setup guide" #~ msgstr "Instrukcja instalacji" diff --git a/website/locales/pt-BR/messages.po b/website/locales/pt-BR/messages.po index 0e1b8a3a3..c1e2d5d2b 100644 --- a/website/locales/pt-BR/messages.po +++ b/website/locales/pt-BR/messages.po @@ -32,18 +32,18 @@ msgid "Thoughts on reverse engineering." msgstr "Reflexões sobre engenharia reversa." #. {ph0}:
-#: website/public/blog/index.html:200 +#: website/public/blog/index.html:214 #, python-brace-format msgctxt "page:blog/index.html" msgid "Game reconstruction{ph0}Essay" msgstr "Reconstrução do jogo{ph0}Ensaio" -#: website/public/blog/index.html:202 +#: website/public/blog/index.html:216 msgctxt "page:blog/index.html" msgid "The industrial revolution in reverse engineering" msgstr "A Revolução industrial na engenharia reversa" -#: website/public/blog/index.html:203 +#: website/public/blog/index.html:217 msgctxt "page:blog/index.html" msgid "" "What manual RE and reconstructing TH08, TH095 and TH04 taught me about " @@ -52,25 +52,26 @@ msgstr "" "O manual RE e reconstrução TH08, TH095 e TH04 me ensinou sobre agentes, " "verificação e conhecimento que compostos entre projetos." -#: website/public/blog/index.html:208 +#: website/public/blog/index.html:207 website/public/blog/index.html:222 msgctxt "page:blog/index.html" msgid "Read the article" msgstr "Leia o artigo" -#: website/public/blog/index.html:211 +#: website/public/blog/index.html:225 msgctxt "page:blog/index.html" msgid "By N0zoM1z0 · 10 October 2026" msgstr "Por N0zoM1z0 * 10 de outubro de 2026" #. {ph0}: -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 +#: website/public/blog/reverse-engineering-models/index.html:883 #: website/public/blog/touhou-reconstruction/index.html:1080 #: website/public/examples/dino-lab/index.html:475 #: website/public/faq/index.html:573 #: website/public/first-investigation/index.html:523 #: website/public/get-started/index.html:882 -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 #: website/public/guides/index.html:282 #: website/public/guides/javascript/index.html:503 #: website/public/guides/native/index.html:509 website/public/index.html:1191 @@ -100,6 +101,8 @@ msgstr "" "agentes REA e AI." #: website/public/blog/index.html:75 website/public/blog/index.html:88 +#: website/public/blog/reverse-engineering-models/index.html:75 +#: website/public/blog/reverse-engineering-models/index.html:88 #: website/public/blog/touhou-reconstruction/index.html:75 #: website/public/blog/touhou-reconstruction/index.html:88 #: website/public/examples/dino-lab/index.html:75 @@ -136,6 +139,7 @@ msgstr "" "codificação." #: website/public/blog/index.html:108 +#: website/public/blog/reverse-engineering-models/index.html:108 #: website/public/blog/touhou-reconstruction/index.html:108 #: website/public/examples/dino-lab/index.html:109 #: website/public/faq/index.html:108 @@ -156,6 +160,7 @@ msgid "REA home" msgstr "Página inicial do REA" #: website/public/blog/index.html:112 +#: website/public/blog/reverse-engineering-models/index.html:112 #: website/public/blog/touhou-reconstruction/index.html:112 #: website/public/examples/dino-lab/index.html:113 #: website/public/faq/index.html:112 @@ -176,6 +181,7 @@ msgid "Main navigation" msgstr "Navegação principal" #: website/public/blog/index.html:130 +#: website/public/blog/reverse-engineering-models/index.html:130 #: website/public/blog/touhou-reconstruction/index.html:130 #: website/public/examples/dino-lab/index.html:131 #: website/public/faq/index.html:130 @@ -196,6 +202,7 @@ msgid "REA on GitHub" msgstr "REA no GitHub" #: website/public/blog/index.html:150 +#: website/public/blog/reverse-engineering-models/index.html:150 #: website/public/blog/touhou-reconstruction/index.html:150 #: website/public/examples/dino-lab/index.html:151 #: website/public/faq/index.html:150 @@ -216,6 +223,7 @@ msgid "Join REA on Discord" msgstr "Participe do Discord do REA" #: website/public/blog/index.html:169 +#: website/public/blog/reverse-engineering-models/index.html:169 #: website/public/blog/touhou-reconstruction/index.html:169 #: website/public/examples/dino-lab/index.html:170 #: website/public/faq/index.html:169 @@ -235,7 +243,7 @@ msgctxt "shared" msgid "Follow REA on X" msgstr "Siga o REA no X" -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 msgctxt "page:blog/index.html" msgid "Back to top" msgstr "Voltar ao topo" @@ -4793,7 +4801,7 @@ msgctxt "page:guides/browser/index.html" msgid "Copy read the loaded scripts" msgstr "Copiar ler os scripts carregados" -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 msgctxt "page:guides/browser/index.html" msgid "Back to top" msgstr "Voltar ao topo" @@ -11695,6 +11703,7 @@ msgid "Same spacing: 256 ÷ 16 = 16 angle units = 22.5°" msgstr "Mesmo espaçamento: 256 16 = 16 unidades angulares = 22,5" #: website/public/blog/index.html:114 +#: website/public/blog/reverse-engineering-models/index.html:114 #: website/public/blog/touhou-reconstruction/index.html:114 #: website/public/examples/dino-lab/index.html:115 #: website/public/faq/index.html:114 @@ -11715,6 +11724,7 @@ msgid "Showcases" msgstr "Casos práticos" #: website/public/blog/index.html:115 +#: website/public/blog/reverse-engineering-models/index.html:115 #: website/public/blog/touhou-reconstruction/index.html:115 #: website/public/examples/dino-lab/index.html:116 #: website/public/faq/index.html:115 @@ -11735,6 +11745,7 @@ msgid "Blog" msgstr "Blog" #: website/public/blog/index.html:113 +#: website/public/blog/reverse-engineering-models/index.html:113 #: website/public/blog/touhou-reconstruction/index.html:113 #: website/public/examples/dino-lab/index.html:114 #: website/public/faq/index.html:113 @@ -11755,6 +11766,7 @@ msgid "Get started" msgstr "Começar" #: website/public/blog/index.html:116 +#: website/public/blog/reverse-engineering-models/index.html:116 #: website/public/blog/touhou-reconstruction/index.html:116 #: website/public/examples/dino-lab/index.html:117 #: website/public/faq/index.html:116 @@ -11774,13 +11786,14 @@ msgctxt "shared" msgid "Guides" msgstr "Guias" -#: website/public/blog/index.html:222 +#: website/public/blog/index.html:236 +#: website/public/blog/reverse-engineering-models/index.html:875 #: website/public/blog/touhou-reconstruction/index.html:1072 #: website/public/examples/dino-lab/index.html:467 #: website/public/faq/index.html:565 #: website/public/first-investigation/index.html:515 #: website/public/get-started/index.html:874 -#: website/public/guides/browser/index.html:490 +#: website/public/guides/browser/index.html:499 #: website/public/guides/index.html:274 #: website/public/guides/javascript/index.html:495 #: website/public/guides/native/index.html:501 website/public/index.html:1183 @@ -11794,13 +11807,14 @@ msgctxt "shared" msgid "Source" msgstr "Código-fonte" -#: website/public/blog/index.html:223 +#: website/public/blog/index.html:237 +#: website/public/blog/reverse-engineering-models/index.html:876 #: website/public/blog/touhou-reconstruction/index.html:1073 #: website/public/examples/dino-lab/index.html:468 #: website/public/faq/index.html:566 #: website/public/first-investigation/index.html:516 #: website/public/get-started/index.html:875 -#: website/public/guides/browser/index.html:491 +#: website/public/guides/browser/index.html:500 #: website/public/guides/index.html:275 #: website/public/guides/javascript/index.html:496 #: website/public/guides/native/index.html:502 website/public/index.html:1184 @@ -11814,13 +11828,14 @@ msgctxt "shared" msgid "MIT license" msgstr "Licença MIT" -#: website/public/blog/index.html:224 +#: website/public/blog/index.html:238 +#: website/public/blog/reverse-engineering-models/index.html:877 #: website/public/blog/touhou-reconstruction/index.html:1074 #: website/public/examples/dino-lab/index.html:469 #: website/public/faq/index.html:567 #: website/public/first-investigation/index.html:517 #: website/public/get-started/index.html:876 -#: website/public/guides/browser/index.html:492 +#: website/public/guides/browser/index.html:501 #: website/public/guides/index.html:276 #: website/public/guides/javascript/index.html:497 #: website/public/guides/native/index.html:503 website/public/index.html:1185 @@ -11834,13 +11849,14 @@ msgctxt "shared" msgid "Join the community" msgstr "Participe da comunidade" -#: website/public/blog/index.html:225 +#: website/public/blog/index.html:239 +#: website/public/blog/reverse-engineering-models/index.html:878 #: website/public/blog/touhou-reconstruction/index.html:1075 #: website/public/examples/dino-lab/index.html:470 #: website/public/faq/index.html:568 #: website/public/first-investigation/index.html:518 #: website/public/get-started/index.html:877 -#: website/public/guides/browser/index.html:493 +#: website/public/guides/browser/index.html:502 #: website/public/guides/index.html:277 #: website/public/guides/javascript/index.html:498 #: website/public/guides/native/index.html:504 website/public/index.html:1186 @@ -12780,6 +12796,1404 @@ msgctxt "runtime" msgid "Switch to dark mode" msgstr "Alternar para o modo escuro" +#. {ph0}: +#. {ph1}: +#: website/public/guides/browser/index.html:480 +#, python-brace-format +msgctxt "page:guides/browser/index.html" +msgid "" +"For HTTP request and response bodies, use {ph0}Fiddler Everywhere: SAZ " +"captures, MCP and public skills{ph1}. Export SAZ or HAR 1.2 for offline REA " +"capture Evidence, or connect Everywhere's MCP server to inspect captured " +"traffic with your agent." +msgstr "" +"Para examinar os corpos das requisições e respostas HTTP, use {ph0}Fiddler " +"Everywhere: capturas SAZ, MCP e habilidades públicas{ph1}. Exporte SAZ ou " +"HAR 1.2 para obter Evidence das capturas no REA offline, ou conecte o " +"servidor MCP do Everywhere para inspecionar o tráfego capturado com seu " +"agente." + +#. {ph0}:
+#: website/public/blog/index.html:200 +#, python-brace-format +msgctxt "page:blog/index.html" +msgid "Model specialization{ph0}Essay" +msgstr "Modelos especializados{ph0}Ensaio" + +#: website/public/blog/index.html:202 +msgctxt "page:blog/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "A engenharia reversa precisa de um modelo maior?" + +#: website/public/blog/index.html:203 +msgctxt "page:blog/index.html" +msgid "" +"What Touhou reconstruction suggests about general models, specialist models " +"and learning from compiler feedback." +msgstr "" +"O que a reconstrução de Touhou sugere sobre modelos gerais, modelos " +"especializados e o aprendizado com o retorno do compilador." + +#: website/public/blog/index.html:210 +msgctxt "page:blog/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Por N0zoM1z0 · 13 de outubro de 2026" + +#: website/public/blog/reverse-engineering-models/index.html:103 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Skip to content" +msgstr "Ir para o conteúdo" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:194 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Home{ph1} / {ph2}Blog{ph3} / Models and RE" +msgstr "" +"{ph0}Página inicial{ph1} / {ph2}Blog{ph3} / Modelos e engenharia reversa" + +#: website/public/blog/reverse-engineering-models/index.html:197 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Essay" +msgstr "Ensaio" + +# Reviewed for technical meaning and readability; preserve the distinction +# between historical evidence and proposed experiments. +#: website/public/blog/reverse-engineering-models/index.html:198 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "A engenharia reversa precisa de um modelo maior?" + +#: website/public/blog/reverse-engineering-models/index.html:199 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Por N0zoM1z0 · 13 de outubro de 2026" + +#: website/public/blog/reverse-engineering-models/index.html:201 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Takeaway" +msgstr "Pontos principais" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:203 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Model capability changes the hypotheses an agent can find.{ph1} Later " +"sessions have solved problems that earlier work left unresolved." +msgstr "" +"{ph0}A capacidade do modelo afeta quais hipóteses um agente de IA pode " +"propor.{ph1} Sessões posteriores resolveram problemas que trabalhos " +"anteriores haviam deixado sem solução." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:211 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A familiar compiler gives specialization something concrete to " +"learn.{ph1} Some exact matches depend on narrow, reusable source patterns." +msgstr "" +"{ph0}Um compilador familiar oferece padrões concretos para um modelo " +"especializado aprender.{ph1} Algumas correspondências exatas de código de " +"máquina dependem de padrões de código-fonte específicos e reutilizáveis." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:218 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Give repository knowledge a fair comparison.{ph1} A smaller model with " +"good retrieval may already know enough to make progress." +msgstr "" +"{ph0}Inclua o conhecimento do repositório em uma comparação justa.{ph1} Um " +"modelo menor com boa recuperação de conhecimento pode já ter informações " +"suficientes para progredir." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:223 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Train the next useful experiment, and test its oracle.{ph1} The value " +"of feedback depends on what the checker actually verifies." +msgstr "" +"{ph0}Treine o modelo para escolher o próximo experimento útil e teste seu " +"verificador.{ph1} O feedback é útil apenas na medida em que o verificador " +"verifica as coisas certas." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:230 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Measure accepted reconstruction against time and cost.{ph1} That is the" +" result I want from the whole agent." +msgstr "" +"{ph0}Avalie os resultados verificados da reconstrução juntamente com o tempo" +" e o custo.{ph1} É isso que quero medir para o fluxo de trabalho completo do" +" agente de IA." + +#: website/public/blog/reverse-engineering-models/index.html:241 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "In this article" +msgstr "Neste artigo" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:243 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What better models changed{ph1}" +msgstr "{ph0}O que os modelos mais capazes mudaram{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:244 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Why specialize?{ph1}" +msgstr "{ph0}Por que se especializar?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:245 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Where experience lives{ph1}" +msgstr "{ph0}Onde essa experiência deve ser armazenada?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:246 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Learning the next experiment{ph1}" +msgstr "{ph0}Aprendendo o próximo experimento útil{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:247 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}RE and reinforcement learning{ph1}" +msgstr "{ph0}Engenharia reversa e aprendizagem por reforço{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:248 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}When the oracle is wrong{ph1}" +msgstr "{ph0}Quando o verificador está errado{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:249 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}A comparison worth running{ph1}" +msgstr "{ph0}Uma comparação que vale a pena fazer{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:250 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What kind of agent?{ph1}" +msgstr "{ph0}Que tipo de agente de IA isso sugere?{ph1}" + +#: website/public/blog/reverse-engineering-models/index.html:258 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What better models changed" +msgstr "O que os modelos mais capazes mudaram" + +#: website/public/blog/reverse-engineering-models/index.html:259 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Since August 2026, I have been exploring how much reverse engineering an " +"agent can carry out with access to a program and a way to check its work. " +"Most of my experience comes from reconstructing games, especially Touhou. As" +" I have changed models, the repositories have kept a record of what each " +"session managed to recover." +msgstr "" +"Desde agosto de 2026 venho explorando o quanto de engenharia reversa um " +"agente pode realizar com acesso a um programa e uma forma de verificar seu " +"funcionamento. A maior parte da minha experiência vem da reconstrução de " +"jogos, especialmente Touhou. Como mudei de modelo, os repositórios " +"mantiveram um registro do que cada sessão conseguiu recuperar." + +#: website/public/blog/reverse-engineering-models/index.html:267 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of that work aims at exact reconstruction: we compile the maintained " +"source under a pinned historical toolchain and check whether it reproduces " +"the target function. When a later model succeeds where earlier work stalled," +" the source changes show what made that progress possible." +msgstr "" +"Grande parte desse trabalho visa a reconstrução exata: compilamos a fonte " +"mantida sob um conjunto de ferramentas histórico com versões fixas e " +"verificamos se ela reproduz a função alvo. Quando um modelo posterior é bem-" +"sucedido onde o trabalho anterior estagnou, as mudanças na origem mostram o " +"que tornou esse progresso possível." + +#: website/public/blog/reverse-engineering-models/index.html:274 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The temporary values had to stay alive together" +msgstr "As variáveis temporárias precisavam de vidas sobrepostas" + +#: website/public/blog/reverse-engineering-models/index.html:275 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of a TH09 attack function's logic had already been reconstructed, but " +"the compiler still stored temporary results differently from the original. " +"The remaining problem was how long those results needed to stay alive and " +"when their storage could be reused." +msgstr "" +"Grande parte da lógica da função de ataque TH09 já havia sido reconstruída, " +"mas o compilador ainda armazenava resultados temporários de maneira " +"diferente do original. O problema restante era quanto tempo esses resultados" +" precisavam para permanecer vivos e quando seu armazenamento poderia ser " +"reutilizado." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:282 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A {ph0}GPT-6-Sol checkpoint{ph1} recorded this mismatch after testing " +"several ordinary source arrangements. When {ph2}GPT-6.1-Sol returned to the " +"problem{ph3}, it found a useful hypothesis about how long those temporary " +"values needed to stay alive." +msgstr "" +"Um {ph0}GPT-6-Sol registro de sessão{ph1} relatou essa incompatibilidade " +"após testar vários arranjos comuns de código-fonte. Quando {ph2}GPT-6.1-Sol " +"examinou o problema novamente{ph3}, propôs uma hipótese útil sobre o tempo " +"de vida dessas variáveis ​​temporárias." + +#: website/public/blog/reverse-engineering-models/index.html:296 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The function produced temporary results and copied them into animation " +"positions. In C++, a temporary result stays alive until the expression using" +" it finishes. Grouping the real operations into one expression kept those " +"results alive together, so the compiler could no longer reuse the same " +"storage for all of them." +msgstr "" +"A função produziu resultados temporários e os copiou em posições de " +"animação. Em C++, um resultado temporário permanece ativo até que a " +"expressão que o utiliza termine. O agrupamento das operações reais em uma " +"expressão manteve esses resultados vivos juntos, de modo que o compilador " +"não pudesse mais reutilizar o mesmo armazenamento para todos eles." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:303 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"One result still shared storage with a value used later for collision " +"checks. Giving that value its own lifetime resolved the remaining overlap. " +"After several targeted experiments, {ph0}a clean rebuild confirmed that the " +"full function matched{ph1}." +msgstr "" +"Um resultado ainda compartilha o armazenamento com um valor usado " +"posteriormente para verificações de colisão. Dar a esse valor seu próprio " +"tempo de vida resolveu a sobreposição restante. Após vários experimentos " +"direcionados, {ph0}uma recompilação limpa da fonte confirmou que a função " +"completa correspondia a{ph1}." + +#: website/public/blog/reverse-engineering-models/index.html:312 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What interests me is how the discovery and the tests worked together. The " +"lifetime hypothesis gave the agent a new direction; compiler feedback helped" +" it refine that hypothesis into a source arrangement that matched." +msgstr "" +"Estou interessado em saber como a descoberta e os testes se apoiaram. A " +"hipótese sobre a vida útil dos objetos temporários deu ao agente de IA uma " +"nova direção. O feedback do compilador ajudou a refinar essa hipótese em um " +"arranjo de código-fonte correspondente." + +#: website/public/blog/reverse-engineering-models/index.html:318 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A different control-flow graph" +msgstr "Um gráfico de fluxo de controle diferente" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:319 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A second TH09 function shows why that new direction can matter. Here, the " +"obstacle was how several branches shared their final work. " +"{ph0}GPT-6.1-Sol's earlier work{ph1} got parts of the function to match, but" +" the branches still joined up differently from the original." +msgstr "" +"Uma segunda função TH09 mostra por que essa nova direção pode ser " +"importante. Aqui, o obstáculo foi a forma como diversas filiais " +"compartilharam seu trabalho final. O trabalho anterior de " +"{ph0}GPT-6.1-Sol{ph1} fez com que partes da função correspondessem, mas as " +"ramificações ainda se uniram de maneira diferente do original." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:330 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}GPT-6-Astra changed the surrounding control-flow graph{ph1}. The main " +"paths now led to a shared final update and return. In that source structure," +" the compiler combined the branches in the same way as the original. The " +"complete function matched, and a {ph2}separate rebuild check confirmed " +"it{ph3}." +msgstr "" +"{ph0}GPT-6-Astra alterou o gráfico de fluxo de controle circundante{ph1}. Os" +" principais caminhos levaram agora a uma atualização e retorno final " +"partilhado. Nessa estrutura de origem, o compilador combinou as ramificações" +" da mesma forma que o original. A função completa correspondeu e uma " +"{ph2}verificação de reconstrução separada confirmou isso{ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:342 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Changing the representation opened a more productive search space.{ph1}" +" Both cases give me a reason to take general model capability seriously: an " +"agent can keep trying local variations for a long time before finding a " +"better way to express the surrounding work." +msgstr "" +"{ph0}Mudar a representação abriu um espaço de pesquisa mais produtivo.{ph1} " +"Ambos os casos me dão um motivo para levar a sério a capacidade geral do " +"modelo: um agente pode continuar tentando variações locais por um longo " +"tempo antes de encontrar uma maneira melhor de expressar o trabalho " +"circundante." + +#: website/public/blog/reverse-engineering-models/index.html:351 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"At the same time, the later sessions started with earlier source and " +"recorded failures. That accumulated work was part of their advantage. The " +"history lets us inspect the discoveries and their checks; a controlled " +"comparison would help separate the effect of the model from the experience " +"it inherited." +msgstr "" +"As sessões posteriores também herdaram o código anterior e os experimentos " +"fracassados já registrados. Esse trabalho acumulado fazia parte da vantagem " +"delas. O histórico permite examinar as descobertas e suas verificações; uma " +"comparação controlada ajudaria a separar o efeito do modelo do efeito da " +"experiência herdada." + +#: website/public/blog/reverse-engineering-models/index.html:358 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is where the question becomes interesting to me. Some of the progress " +"came from finding a new representation, while much of the remaining work " +"involved testing source forms against a familiar compiler. How much needs " +"broad reasoning ability, and how much could a model learn through repeated " +"experience with that compiler and code?" +msgstr "" +"É isso que torna a questão interessante para mim. Algum progresso veio da " +"descoberta de uma nova representação. Grande parte do trabalho restante " +"envolveu testar diferentes arranjos de código-fonte em um compilador " +"familiar. Quanto disso requer ampla capacidade de raciocínio? Quanto um " +"modelo poderia aprender trabalhando repetidamente com esse compilador e " +"código?" + +#: website/public/blog/reverse-engineering-models/index.html:368 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why might a specialist do well?" +msgstr "Por que um modelo especializado de IA pode funcionar bem?" + +#: website/public/blog/reverse-engineering-models/index.html:369 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Touhou gives us a concrete reason to expect that experience to help. Much of" +" the game code comes from the same author, ZUN, so recurring habits are a " +"reasonable starting assumption. The repositories back that up with useful " +"examples: related loops and helper arrangements have supplied hypotheses " +"across games." +msgstr "" +"Touhou nos dá uma razão concreta para esperar que essa experiência ajude. " +"Grande parte do código do jogo vem do mesmo autor, ZUN, portanto, hábitos " +"recorrentes são uma suposição inicial razoável. Os repositórios apoiam isso " +"com exemplos úteis: loops relacionados e arranjos auxiliares forneceram " +"hipóteses em todos os jogos." + +#: website/public/blog/reverse-engineering-models/index.html:376 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those source patterns become more specific when we include the compiler. " +"Under a particular version and set of options, small changes in source can " +"produce repeatable differences in machine code. Some of those differences " +"are surprisingly obscure." +msgstr "" +"Esses padrões de origem tornam-se mais específicos quando incluímos o " +"compilador. Sob uma determinada versão e conjunto de opções, pequenas " +"alterações na fonte podem produzir diferenças repetíveis no código de " +"máquina. Algumas dessas diferenças são surpreendentemente obscuras." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: Supervisor::LoadDat +#. {ph4}: +#: website/public/blog/reverse-engineering-models/index.html:382 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In TH095, for example, {ph0}even the name used for a local variable could " +"change where VC7.1 stored it{ph1}. Moving the declaration alone did not " +"solve the layout, but an earlier background function had revealed a naming " +"pattern that affected the compiler's choice. That gave us a more promising " +"hypothesis for {ph2}the reconstruction of {ph3}{ph4}." +msgstr "" +"Em TH095, por exemplo, {ph0}até mesmo o nome de uma variável local pode " +"afetar o local onde VC7.1 a armazenou{ph1}. Mover sua declaração não " +"resolveu o layout de armazenamento. No entanto, uma função anterior revelou " +"um padrão de nomenclatura que afetou a escolha do compilador. Isso sugeriu " +"uma hipótese mais promissora para {ph2}reconstruir {ph3}{ph4}." + +#: website/public/blog/reverse-engineering-models/index.html:396 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reusing that pattern for the file-size variable produced the required layout" +" without adding storage. The full function then matched." +msgstr "" +"A reutilização desse padrão para a variável de tamanho de arquivo produziu o" +" layout necessário sem adicionar armazenamento. A função completa então " +"correspondeu." + +#: website/public/blog/reverse-engineering-models/index.html:401 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is the sort of narrow detail a general model may not know when it " +"arrives. A specialist trained on this compiler could encounter many such " +"cases and learn which source variations are worth testing first." +msgstr "" +"Este é um detalhe muito específico que um modelo de IA de uso geral pode não" +" saber quando encontra o problema pela primeira vez. Um modelo especializado" +" para este compilador poderia ver muitos desses casos e aprender quais " +"variações do código-fonte valem a pena testar primeiro." + +#: website/public/blog/reverse-engineering-models/index.html:407 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The opportunity is especially clear in final matching problems where we " +"already understand the behavior. We are then asking which ordinary source " +"form this compiler turns into the observed instructions. Specialization " +"could help the model propose those forms more efficiently, while the target " +"still determines which hypotheses survive." +msgstr "" +"A oportunidade é especialmente clara em problemas de correspondência final " +"onde já entendemos o comportamento. Estamos então perguntando qual arranjo " +"comum de código-fonte este compilador transforma nas instruções observadas. " +"A especialização poderia ajudar o modelo a propor essas formas de forma mais" +" eficiente, enquanto o alvo ainda determina quais hipóteses sobreviverão." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:415 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The useful patterns would have to retain their context, though. TH08 uses an" +" earlier Visual C++ version than TH095 and TH09. TH04 uses a different DOS-" +"era compiler, and its {ph0}player reconstruction{ph1} needed different " +"hypotheses about source patterns and how the compiled code was combined into" +" an executable. Shared authorship gives us a prior, but each target tells us" +" whether it applies. Distinguishing those contexts is part of what a " +"specialist would need to learn." +msgstr "" +"Padrões úteis devem manter seu contexto. TH08 usa uma versão Visual C++ " +"anterior a TH095 e TH09. TH04 usa um compilador diferente da era DOS. Sua " +"{ph0}reconstrução do código do jogador{ph1} precisava de diferentes " +"hipóteses sobre os padrões do código-fonte e como o código compilado era " +"vinculado a um executável. Ter o mesmo autor nos dá uma expectativa inicial," +" mas cada alvo nos diz se se aplica. Um modelo especializado precisaria " +"distinguir esses contextos." + +#: website/public/blog/reverse-engineering-models/index.html:431 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Where should that experience live?" +msgstr "Onde essa experiência deve ser armazenada?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:432 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before deciding to teach those patterns through model training, I would look" +" at the knowledge we already keep in the repository. In the {ph0}previous " +"essay{ph1}, I wrote about it becoming the project's working memory. That " +"memory includes compiler lessons, along with the source and failed " +"experiments that made them useful." +msgstr "" +"Antes de decidir ensinar esses padrões por meio do treinamento de modelos, " +"eu examinaria o conhecimento que já mantemos no repositório. No {ph0}ensaio " +"anterior{ph1}, escrevi sobre isso se tornar a memória de trabalho do " +"projeto. Essa memória inclui lições de compilador, juntamente com a fonte e " +"os experimentos fracassados ​​que os tornaram úteis." + +#: website/public/blog/reverse-engineering-models/index.html:441 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"By retrieving that record, an agent can start with a known result and " +"inspect how we reached it. It can also avoid alternatives that have already " +"failed in the same context. The starting point improves without changing the" +" model's weights." +msgstr "" +"Ao recuperar esse registro, um agente pode começar com um resultado " +"conhecido e inspecionar como o alcançamos. Também pode evitar alternativas " +"que já falharam no mesmo contexto. O ponto inicial melhora sem alterar os " +"parâmetros treinados do modelo." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:447 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"When reconstructing TH09's enemy-spawn function, a {ph0}GPT-6-Luna-max " +"session{ph1} drew a hypothesis from TH08's corresponding source family. That" +" earlier source suggested how to structure the loop and handle failures " +"within it. TH09's own evidence supplied the game-specific details, and clean" +" rebuilds confirmed the complete match." +msgstr "" +"Ao reconstruir a função de TH09 para criar inimigos, uma {ph0}sessão " +"GPT-6-Luna-max{ph1} tirou uma hipótese do código-fonte TH08 relacionado. " +"Esse código anterior sugeria como organizar o loop e lidar com falhas dentro" +" dele. As próprias evidências de TH09 forneceram detalhes específicos do " +"jogo. As reconstruções da fonte confirmaram que a função completa " +"correspondia." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:458 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}\"The model did not know that compiler quirk\" and \"the model could " +"not use it\" are different problems.{ph1} Retrieval may solve the first, but" +" turning that fact into a useful candidate still takes work. And if the " +"candidate fails, the model must decide what the result means for its next " +"attempt. Knowing the relevant pattern is only the beginning of that process." +msgstr "" +"{ph0}\"O modelo não sabia daquela peculiaridade do compilador\" e \"o modelo" +" não pôde usá-lo\" são problemas diferentes.{ph1} A recuperação pode " +"resolver o primeiro, mas transformar esse fato em um candidato útil ainda dá" +" trabalho. E se o código-fonte candidato falhar, o modelo deverá decidir o " +"que o resultado significa para sua próxima tentativa. Conhecer o padrão " +"relevante é apenas o começo desse processo." + +#: website/public/blog/reverse-engineering-models/index.html:468 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For that reason, I would include a smaller general model with good retrieval" +" in any comparison. Giving it the same library of experience would help us " +"identify what training adds: better candidate generation, better decisions " +"after feedback, or both." +msgstr "" +"Por esse motivo, eu incluiria um modelo menor de IA de uso geral com boa " +"recuperação em qualquer comparação. Fornecer a mesma biblioteca de " +"experiência nos ajudaria a identificar o que o treinamento agrega: melhor " +"geração de candidatos, melhores decisões após feedback ou ambos." + +#: website/public/blog/reverse-engineering-models/index.html:476 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Learning the next useful experiment" +msgstr "Aprendendo o próximo experimento útil" + +#: website/public/blog/reverse-engineering-models/index.html:477 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those decisions after feedback are what I would most like to study. In " +"agentic reconstruction, the model proposes an explanation and tests it " +"against the program. Even when the candidate fails, the result can narrow " +"the remaining possibilities and suggest a more useful next experiment." +msgstr "" +"Essas decisões após feedback são o que eu mais gostaria de estudar. Na " +"reconstrução agente, o modelo propõe uma explicação e a testa em relação ao " +"programa. Mesmo quando o código-fonte candidato falha, o resultado pode " +"restringir as possibilidades restantes e sugerir um próximo experimento mais" +" útil." + +#: website/public/blog/reverse-engineering-models/index.html:484 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Consider the temporary-results example again. Its final source shows the " +"arrangement that worked, but the route to that source contains more " +"information. Each of the following experiments helped identify what still " +"needed to change." +msgstr "" +"Considere novamente o exemplo dos resultados temporários. Sua fonte final " +"mostra o arranjo que funcionou, mas o caminho para essa fonte contém mais " +"informações. Cada um dos experimentos a seguir ajudou a identificar o que " +"ainda precisava ser mudado." + +#: website/public/blog/reverse-engineering-models/index.html:497 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Experiment" +msgstr "Experimentar" + +#: website/public/blog/reverse-engineering-models/index.html:498 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What its result taught us" +msgstr "O que seu resultado nos ensinou" + +#: website/public/blog/reverse-engineering-models/index.html:503 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Try a different way to copy the results" +msgstr "Tente uma maneira diferente de copiar os resultados" + +#: website/public/blog/reverse-engineering-models/index.html:504 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The copy code changes, but the function still differs." +msgstr "O código de cópia muda, mas a função ainda é diferente." + +#: website/public/blog/reverse-engineering-models/index.html:509 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Use the returned values directly" +msgstr "Use os valores retornados diretamente" + +#: website/public/blog/reverse-engineering-models/index.html:510 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Copying follows the original pattern, but temporary storage still differs." +msgstr "" +"A cópia segue o padrão original, mas o armazenamento temporário ainda é " +"diferente." + +#: website/public/blog/reverse-engineering-models/index.html:516 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the temporary results alive together" +msgstr "Mantenha todas as variáveis temporárias válidas ao mesmo tempo" + +#: website/public/blog/reverse-engineering-models/index.html:517 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"They get separate storage, but one still overlaps a value used later for " +"collision checks." +msgstr "" +"Eles obtêm armazenamento separado, mas ainda se sobrepõe a um valor usado " +"posteriormente para verificações de colisão." + +#: website/public/blog/reverse-engineering-models/index.html:523 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the collision-check value separate" +msgstr "Mantenha o valor de verificação de colisão separado" + +#: website/public/blog/reverse-engineering-models/index.html:524 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A clean rebuild confirms the complete match." +msgstr "Uma recompilação limpa da fonte confirma a correspondência completa." + +#: website/public/blog/reverse-engineering-models/index.html:529 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Selected experiments from the attack-function reconstruction above, grouped " +"by what they taught us." +msgstr "" +"Experimentos selecionados da reconstrução da função de ataque acima, " +"agrupados pelo que nos ensinaram." + +#: website/public/blog/reverse-engineering-models/index.html:533 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A training record that keeps this sequence can teach more than the final " +"source alone. It connects each proposed change to an observed result and the" +" next decision. Keeping ineffective variations matters too: they tell a " +"later agent which hypotheses have already been tested in that context." +msgstr "" +"Um registro de treinamento que mantenha essa sequência pode ensinar mais do " +"que apenas a fonte final. Ele conecta cada mudança proposta a um resultado " +"observado e à próxima decisão. Manter variações ineficazes também é " +"importante: elas informam a um agente posterior quais hipóteses já foram " +"testadas naquele contexto." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:540 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}I would want the model to learn how to use feedback.{ph1} A neutral " +"result is a reason to reconsider the variation; a changed mismatch may point" +" to another cause. When local edits stop being informative, the agent should" +" be able to revisit the source representation or inspect the evidence again." +msgstr "" +"{ph0}Eu gostaria que o modelo aprendesse como usar o feedback.{ph1} Um " +"resultado neutro é um motivo para reconsiderar a variação; uma " +"incompatibilidade alterada pode apontar para outra causa. Quando as edições " +"locais deixarem de ser informativas, o agente deverá ser capaz de revisitar " +"a representação da fonte ou inspecionar as evidências novamente." + +#: website/public/blog/reverse-engineering-models/index.html:549 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Learning to make those choices requires autonomy. Sometimes the useful next " +"step is inspecting another caller rather than compiling another edit. A " +"rigid sequence of required rewrites would remove that choice, along with " +"part of the behavior we want the model to learn." +msgstr "" +"Aprender a fazer essas escolhas requer autonomia. Às vezes, o próximo passo " +"útil é inspecionar outro chamador em vez de compilar outra edição. Uma " +"sequência rígida de reescritas necessárias eliminaria essa escolha, " +"juntamente com parte do comportamento que queremos que o modelo aprenda." + +#: website/public/blog/reverse-engineering-models/index.html:558 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why RE is interesting for reinforcement learning" +msgstr "" +"Por que a engenharia reversa é interessante para aprendizagem por reforço" + +#: website/public/blog/reverse-engineering-models/index.html:559 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Bounded reconstruction gives us a practical environment for studying these " +"choices. Once the target and compiler are pinned, many source experiments " +"produce quick, repeatable feedback. The agent can compile a candidate, " +"inspect the mismatch and use that observation to choose its next step." +msgstr "" +"Uma reconstrução com escopo definido oferece um ambiente prático para " +"estudar essas escolhas. Com as versões do alvo e do compilador fixadas, " +"muitos experimentos de código-fonte dão feedback rápido e repetível. O " +"agente pode compilar um candidato, examinar as diferenças e escolher o " +"próximo passo." + +#: website/public/blog/reverse-engineering-models/index.html:566 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reinforcement learning could train that decision-making process through the " +"feedback it receives. A candidate accepted by a defined check gives us a " +"concrete outcome, while the sequence of attempts lets us observe how much " +"work it took to reach it." +msgstr "" +"A aprendizagem por reforço poderia treinar esse processo de tomada de " +"decisão por meio do feedback que recebe. Um candidato aceito por meio de uma" +" verificação definida nos dá um resultado concreto, enquanto a sequência de " +"tentativas nos permite observar quanto trabalho foi necessário para alcançá-" +"lo." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:572 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"There is already research supporting this direction. {ph0}LLM4Decompile{ph1}" +" studies models specialized for decompilation. {ph2}AutoDecompiler{ph3} " +"trains multi-turn refinement using compiler and execution feedback. Their " +"studies focus on functional recovery. For this work, I would use them as a " +"starting point for the related task of finding source that matches a " +"historical compiler's output." +msgstr "" +"Já existem pesquisas apoiando essa direção. {ph0}LLM4Decompile{ph1} estuda " +"modelos especializados para descompilação. {ph2}AutoDecompiler{ph3} treina o" +" refinamento em várias rodadas usando feedback do compilador e de execução. " +"Seus estudos se concentram na recuperação funcional. Para este trabalho, eu " +"os usaria como ponto de partida para a tarefa relacionada de encontrar uma " +"fonte que corresponda à saída de um compilador histórico." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:584 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To reach such a well-defined task, though, we may first need to establish " +"what a function owns or devise a new runtime measurement. In TH08, for " +"example, a {ph0}collision difference led back to how the original program " +"handled floating-point calculations{ph1}. Before the agent could test a " +"useful correction, it had to work out what to measure." +msgstr "" +"Porém, para alcançar uma tarefa tão bem definida, talvez precisemos primeiro" +" estabelecer quais recursos uma função possui ou criar uma nova forma de " +"observar o comportamento do programa durante a execução. Em TH08, por " +"exemplo, uma {ph0}diferença de colisão levou de volta à forma como o " +"programa original lidava com cálculos de ponto flutuante{ph1}. Antes que o " +"agente pudesse testar uma correção útil, ele precisava decidir o que medir." + +#: website/public/blog/reverse-engineering-models/index.html:595 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This suggests another role for a broadly capable model: helping make the " +"problem checkable in the first place. Once that check exists, we can " +"investigate how far a learned search policy can get with it." +msgstr "" +"Isto sugere outro papel para um modelo amplamente capaz: em primeiro lugar, " +"ajudar a tornar o problema verificável. Uma vez que essa verificação exista," +" podemos investigar até onde uma política de pesquisa aprendida pode chegar " +"com ela." + +#: website/public/blog/reverse-engineering-models/index.html:603 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "When the oracle is wrong" +msgstr "Quando o verificador está errado" + +#: website/public/blog/reverse-engineering-models/index.html:604 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Once we train against that check, its coverage becomes part of what the " +"model learns. I call the reference or checker an oracle. Even if it gives a " +"repeatable answer, we still need to understand what that answer covers." +msgstr "" +"Quando usamos uma verificação no treinamento, sua cobertura passa a fazer " +"parte do ambiente de aprendizagem. Chamo essa referência ou verificador de " +"oracle. Mesmo uma resposta repetível precisa ser entendida pelos aspectos " +"que ela realmente verifica." + +#: website/public/blog/reverse-engineering-models/index.html:610 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"TH08 showed how much can hide outside that coverage. Some reconstructed " +"functions passed their exact comparisons with wrong floating constants. The " +"comparator adjusted relocated addresses before checking instructions, but it" +" checked the values behind those addresses only when they had been " +"explicitly recorded for verification. Other literal references could pass " +"without their values being examined." +msgstr "" +"TH08 mostrou o quanto pode se esconder fora dessa cobertura. Algumas funções" +" reconstruídas passaram em suas comparações exatas com constantes flutuantes" +" erradas. O comparador ajustou os endereços realocados antes de verificar as" +" instruções, mas verificou os valores por trás desses endereços somente " +"quando eles foram explicitamente registrados para verificação. Outras " +"referências literais poderiam passar sem que seus valores fossem examinados." + +#. {ph0}: +#. {ph1}: +#. {ph2}: 1.0f +#. {ph3}: 32.0f +#: website/public/blog/reverse-engineering-models/index.html:619 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To close that gap, a {ph0}wider audit{ph1} found incorrect constants in " +"functions that had already been accepted. In one calculation, the " +"reconstructed source added {ph2} where the original added {ph3}." +msgstr "" +"Para resolver essa lacuna, uma {ph0}auditoria mais ampla{ph1} encontrou " +"constantes incorretas em funções que já haviam passado na verificação. Em um" +" cálculo, o código-fonte reconstruído adicionou {ph2}, enquanto o original " +"adicionou {ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:629 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A learner rewarded by the old check could learn to produce the wrong " +"reconstruction.{ph1} We had not trained an RL model against this oracle; the" +" observed failure was in the check itself. But using that check as a reward " +"would have made the same blind spot part of the learning environment." +msgstr "" +"{ph0}Um modelo recompensado pela verificação antiga poderia aprender a " +"produzir uma reconstrução incorreta.{ph1} Não treinamos um modelo RL usando " +"esse verificador. A falha que observamos estava na própria verificação. Mas " +"se usássemos isso como recompensa, o mesmo ponto cego passaria a fazer parte" +" do ambiente de aprendizagem." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:639 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A functional test can have a similar blind spot. {ph0}Differential-fuzzing " +"research on LLM decompilers{ph1} has found behavior differences beyond the " +"supplied tests. Passing those tests therefore needs to be read in terms of " +"the cases they exercise, just as an instruction match needs to be read in " +"terms of the bytes and data it checks." +msgstr "" +"Um teste funcional pode ter um ponto cego semelhante. {ph0}Pesquisas usando " +"difusão diferencial em descompiladores LLM{ph1} encontraram diferenças " +"comportamentais fora dos testes fornecidos. Portanto, precisamos entender um" +" teste aprovado em termos dos casos que ele exerce. Da mesma forma, uma " +"correspondência de instrução deve ser entendida em termos dos bytes e dos " +"dados que ela verifica." + +#: website/public/blog/reverse-engineering-models/index.html:649 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before using either kind of check as a reward, I would test it with " +"deliberate failures. I would also keep final acceptance outside the " +"candidate's editable source, so the model has to satisfy the check through " +"its reconstruction. Otherwise, something like a byte array copied from the " +"target could satisfy a naive score without recovering useful source." +msgstr "" +"Antes de usar qualquer tipo de verificação como recompensa, eu testaria com " +"resultados deliberadamente incorretos. Eu também manteria a aceitação final " +"separada do código-fonte que o agente de IA pode editar. O modelo deve " +"passar na verificação reconstruindo o programa corretamente. Caso contrário," +" simplesmente copiar uma matriz de bytes do destino poderia enganar um " +"sistema de pontuação ingênuo sem recuperar o código-fonte útil." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:657 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Improving the learner and improving its oracle belong to the same " +"research problem.{ph1} As the learner gets better at optimizing the feedback" +" we give it, understanding that feedback becomes more important too." +msgstr "" +"{ph0}Aprimorar o aluno e melhorar seu verificador pertencem ao mesmo " +"problema de pesquisa.{ph1} À medida que o aluno melhora na otimização do " +"feedback que lhe damos, compreender esse feedback também se torna mais " +"importante." + +#: website/public/blog/reverse-engineering-models/index.html:667 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A comparison worth running" +msgstr "Uma comparação que vale a pena fazer" + +#: website/public/blog/reverse-engineering-models/index.html:668 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"With those checks in place, I would start the comparison on a fixed VC7.1 " +"task family. First, I would establish baselines for a strong general model " +"and a smaller one. Adding repository knowledge to the smaller model would " +"show what retrieval changes. From there, supervised specialization and an " +"RL-trained version would let us examine what training adds." +msgstr "" +"Com essas verificações implementadas, eu iniciaria a comparação em uma " +"família de tarefas fixa VC7.1. Primeiro, eu estabeleceria linhas de base " +"para um modelo forte de IA de uso geral e um modelo menor. Adicionar " +"conhecimento do repositório ao modelo menor mostraria quais mudanças na " +"recuperação. A partir daí, a especialização supervisionada e uma versão " +"treinada em RL nos permitiriam examinar o que o treinamento acrescenta." + +#: website/public/blog/reverse-engineering-models/index.html:676 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To make those comparisons informative, each model would receive the same " +"target evidence and tools. For the training-specific comparisons, I would " +"keep the base model and available knowledge fixed as well. Repeated trials " +"would help distinguish a reliable search strategy from a lucky candidate." +msgstr "" +"Para tornar essas comparações informativas, cada modelo receberia as mesmas " +"evidências e ferramentas alvo. Para comparações específicas de treinamento, " +"eu também manteria o modelo base e o conhecimento disponível fixos. Testes " +"repetidos ajudariam a distinguir uma estratégia de busca confiável de uma " +"solução candidata que tivesse sucesso por acaso." + +#: website/public/blog/reverse-engineering-models/index.html:683 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would then separate two starting conditions: reconstructing a function " +"from its instructions, and closing the last few differences in maintained " +"source. A specialist might be excellent at the second task while a general " +"model does much of the work needed to reach it. Measuring them separately " +"would reveal that division of work." +msgstr "" +"Eu separaria então duas condições iniciais: reconstruir uma função a partir " +"de suas instruções e fechar as últimas diferenças na fonte mantida. Um " +"modelo de IA especializado pode ser excelente na segunda tarefa, enquanto um" +" modelo de IA de uso geral faz grande parte do trabalho necessário para " +"alcançá-la. Medi-los separadamente revelaria essa divisão de trabalho." + +#: website/public/blog/reverse-engineering-models/index.html:691 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both tasks need genuinely held-out examples. Because related Touhou games " +"share engine code, holding out a game's name would still allow nearly " +"identical functions to appear in training. I would instead hold out source " +"families and keep their accepted answers out of both training and retrieval." +msgstr "" +"Ambas as tarefas precisam de exemplos de teste excluídos do treinamento e " +"recuperação. Jogos Touhou relacionados compartilham o código do mecanismo. " +"Excluir um jogo do treinamento ainda poderia deixar funções quase idênticas " +"no conjunto de treinamento. Em vez disso, eu excluiria grupos inteiros de " +"código-fonte relacionado e manteria suas respostas verificadas fora dos " +"dados de treinamento e da biblioteca de recuperação." + +#: website/public/blog/reverse-engineering-models/index.html:698 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"On those examples, the outcome would be complete, independently validated " +"matches. Giving each model the same candidate or compile budget would help " +"compare search quality. Measuring elapsed time and total cost, including " +"compilation, would tell us whether that quality produces an efficient " +"workflow. Training costs would need their own accounting too." +msgstr "" +"Para estes exemplos, o resultado seria correspondências completas e " +"verificadas de forma independente. Orçamentos iguais para propostas de " +"candidatos ou tentativas de compilação ajudariam a comparar a qualidade da " +"pesquisa. Medir o tempo e o custo totais, incluindo a compilação, mostraria " +"se essa qualidade produz um fluxo de trabalho eficiente. Os custos de " +"formação devem ser medidos separadamente." + +#: website/public/blog/reverse-engineering-models/index.html:706 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both views matter because a cheaper proposal can become an expensive " +"workflow if it causes more uninformative compiles. Conversely, a slower " +"model may pay for itself by finding a useful experiment sooner. The " +"comparison should capture those costs." +msgstr "" +"Ambas as medidas são importantes. Uma proposta barata pode levar a um fluxo " +"de trabalho caro se causar mais compilações que não fornecem informações " +"úteis. Por outro lado, um modelo mais lento pode ser mais econômico se " +"encontrar uma experiência útil mais cedo. A comparação deve incluir esses " +"custos." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:712 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Counting accepted results also keeps intermediate progress in perspective. " +"One TH09 {ph0}replay-save candidate{ph1} looked promising because the " +"compiled function had the expected size, but a full comparison still found " +"substantial differences. Changes in how the compiler arranges instructions " +"can also make a candidate look closer without resolving the underlying " +"problem. Such diagnostics can guide the next experiment, but I would " +"validate any reward for \"getting closer\" against the full acceptance " +"check." +msgstr "" +"A contagem dos resultados aceitos também mantém o progresso intermediário em" +" perspectiva. Um TH09 {ph0}candidato para salvar repetição{ph1} parecia " +"promissor porque a função compilada tinha o tamanho esperado, mas uma " +"comparação completa ainda encontrou diferenças substanciais. Mudanças na " +"forma como o compilador organiza as instruções também podem fazer o " +"candidato parecer mais próximo do alvo, sem resolver o problema subjacente. " +"Esses diagnósticos podem orientar o próximo experimento, mas eu validaria " +"qualquer recompensa por “chegar mais perto” em relação à verificação de " +"aceitação total." + +#: website/public/blog/reverse-engineering-models/index.html:727 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would be interested in the pattern across these results. If a specialist " +"does well within a familiar compiler family, does that advantage survive a " +"toolchain change? And once both models have good retrieval, how much does " +"training still add? The answers would help us decide where each kind of " +"model is most useful." +msgstr "" +"Eu estaria interessado no padrão desses resultados. Se um modelo de IA " +"especializado funciona bem em uma família de compiladores familiar, essa " +"vantagem sobreviverá a uma mudança no conjunto de ferramentas? E uma vez que" +" ambos os modelos de IA tenham uma boa recuperação, quanto o treinamento " +"ainda acrescenta? As respostas nos ajudariam a decidir onde cada tipo de " +"modelo é mais útil." + +#: website/public/blog/reverse-engineering-models/index.html:736 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What kind of agent does this suggest?" +msgstr "Que tipo de agente de IA isso sugere?" + +#: website/public/blog/reverse-engineering-models/index.html:737 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"If the models turn out to help at different stages, a useful arrangement " +"might be a broad agent with access to a specialist. Once a task becomes a " +"familiar compiler search, it could ask the specialist for candidates. If the" +" results stop making sense, the broader agent could return to the target " +"evidence or investigate the measurement itself." +msgstr "" +"Se os modelos acabarem por ajudar em diferentes fases, um acordo útil poderá" +" ser um agente amplo com acesso a um modelo de IA especializado. Depois que " +"uma tarefa se torna uma pesquisa familiar do compilador, ela pode solicitar " +"candidatos ao modelo de IA especializado. Se os resultados deixarem de fazer" +" sentido, o agente mais amplo poderá retornar à evidência alvo ou investigar" +" a própria medição." + +#: website/public/blog/reverse-engineering-models/index.html:745 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In that arrangement, specialization becomes a tool the agent can choose as " +"the investigation develops. The agent keeps the freedom to change direction " +"that made the earlier breakthroughs possible." +msgstr "" +"Nesse arranjo, a especialização torna-se uma ferramenta que o agente pode " +"escolher à medida que a investigação se desenvolve. O agente mantém a " +"liberdade de mudar a direção que tornou possíveis os avanços anteriores." + +#: website/public/blog/reverse-engineering-models/index.html:750 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This follows naturally from the industrial shift I described in the previous" +" essay. Being able to run many more experiments changes how we organize the " +"work. It also gives us a reason to ask which decisions need broad judgment " +"and which improve through repeated experience with a particular craft." +msgstr "" +"Isto decorre naturalmente da mudança industrial que descrevi no ensaio " +"anterior. Ser capaz de realizar muito mais experimentos muda a forma como " +"organizamos o trabalho. Também nos dá uma razão para perguntar quais " +"decisões necessitam de um julgamento amplo e quais melhoram através da " +"experiência repetida com uma determinada arte." + +#: website/public/blog/reverse-engineering-models/index.html:757 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For now, the strongest general model has earned its place in my workflow by " +"opening problems I had left unresolved. Those successes are what make me " +"want to examine the work more closely: how much depended on the model's " +"breadth, and how much could be handled by a model trained in this particular" +" kind of reconstruction?" +msgstr "" +"Por enquanto, o modelo geral mais forte conquistou seu lugar no meu trabalho" +" ao resolver problemas que eu havia deixado em aberto. Esses sucessos me " +"fazem querer estudar o processo: quanto dependia da amplitude do modelo e " +"quanto poderia ser feito por um modelo treinado especificamente para esse " +"tipo de reconstrução?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:765 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}The repositories give us a way to test the question.{ph1} I would like " +"to use that record to build the comparison and see what we learn." +msgstr "" +"{ph0}Os repositórios nos fornecem uma maneira de testar a pergunta.{ph1} " +"Gostaria de usar esse registro para construir a comparação e ver o que " +"aprendemos." + +#: website/public/blog/reverse-engineering-models/index.html:775 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Evidence and model attribution" +msgstr "Evidências e atribuição de modelo" + +#: website/public/blog/reverse-engineering-models/index.html:783 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The examples describe historical function-level comparisons recorded in the " +"linked commits. Whole-game runtime checks are separate milestones. Source " +"and matching records were reviewed on 13 October 2026." +msgstr "" +"Os exemplos descrevem comparações históricas em nível de função registradas " +"nos commits vinculados. As verificações de tempo de execução de todo o jogo " +"são marcos separados. Os registros de origem e correspondentes foram " +"revisados em 13 de outubro de 2026." + +#. {ph0}: gpt-dots +#. {ph1}: gpt-web +#: website/public/blog/reverse-engineering-models/index.html:789 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The TH09 label {ph0} identifies GPT-6-Astra. {ph1} changed with the model " +"used at the time, so that label alone does not identify an individual run. " +"Explicit model names in the other cited commits establish their recorded " +"attribution; the histories do not contain uniform per-run cost or effort " +"measurements." +msgstr "" +"O rótulo TH09 {ph0} significa GPT-6-Astra. O modelo associado a {ph1} mudou " +"ao longo do tempo, de modo que o rótulo por si só não consegue identificar o" +" modelo usado em uma execução específica. Nomes de modelos explícitos nos " +"outros commits citados fornecem a atribuição registrada. Esses históricos " +"não incluem medições consistentes de custo ou esforço para cada execução." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#: website/public/blog/reverse-engineering-models/index.html:798 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 temporary lifetimes:{ph1} the {ph2}earlier context review{ph3} and" +" {ph4}complete cold reconstruction{ph5}." +msgstr "" +"{ph0}TH09 tempos de vida do objeto temporário:{ph1} a {ph2}revisão de " +"contexto anterior{ph3} e a {ph4}reconstrução completa verificada por uma " +"recompilação limpa da origem{ph5}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:810 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 Front callback:{ph1} {ph2}non-exact diagnostic regions{ph3}, " +"{ph4}Astra's new graph{ph5} and its {ph6}accepted recompile replay{ph7}." +msgstr "" +"{ph0}TH09 Front retorno de chamada:{ph1} {ph2}regiões de diagnóstico que não" +" correspondem{ph3}, {ph4}novo gráfico de fluxo de controle de Astra{ph5} e " +"sua {ph6}verificação de recompilação aceita{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:826 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Compiler knowledge:{ph1} {ph2}TH095 identifier-bucket reuse{ph3}, " +"{ph4}TH04 player source and linker order{ph5} and {ph6}Luna's TH09 enemy-" +"spawn closure{ph7}." +msgstr "" +"{ph0}Conhecimento do compilador:{ph1} {ph2}TH095 reutilização do " +"comportamento do bloco de identificadores{ph3}, {ph4}TH04 código-fonte do " +"jogador e ordem do vinculador{ph5} e {ph6}reconstrução completa de Luna da " +"função de criação de inimigo de TH09{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:842 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Checks and diagnostics:{ph1} {ph2}TH08 floating-literal audit{ph3}, " +"{ph4}TH08 numerical investigation{ph5} and {ph6}TH09 replay-save " +"reconstruction{ph7}." +msgstr "" +"{ph0}Verificações e diagnósticos:{ph1} {ph2}TH08 auditoria de constante de " +"ponto flutuante{ph3}, {ph4}TH08 investigação numérica{ph5} e {ph6}TH09 " +"reconstrução para salvar replay{ph7}." + +#: website/public/blog/reverse-engineering-models/index.html:859 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The proposed specialist and RL comparisons are future experiments. The " +"project histories motivate them; no trained specialist was evaluated for " +"this essay." +msgstr "" +"As comparações propostas de modelos especializados e treinamento RL são " +"experimentos futuros. As histórias do projeto os motivam. Nenhum modelo " +"especializado treinado foi avaliado para este ensaio." + +#: website/public/blog/reverse-engineering-models/index.html:6 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model? | REA" +msgstr "A engenharia reversa precisa de um modelo maior? | REA" + +#: website/public/blog/reverse-engineering-models/index.html:7 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What Touhou reconstruction taught me about general models, specialist models" +" and learning to propose better hypotheses from compiler feedback." +msgstr "" +"O que a reconstrução Touhou me ensinou sobre modelos gerais, modelos " +"especializados e como aprender a propor melhores hipóteses a partir do " +"feedback do compilador." + +#: website/public/blog/reverse-engineering-models/index.html:240 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "On this page" +msgstr "Nesta página" + +#: website/public/blog/reverse-engineering-models/index.html:253 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Models and reverse engineering essay" +msgstr "Modelos e ensaio de engenharia reversa" + +#: website/public/blog/reverse-engineering-models/index.html:491 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Selected reconstruction experiments" +msgstr "Experimentos de reconstrução selecionados" + +#: website/public/blog/reverse-engineering-models/index.html:883 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Back to top" +msgstr "Voltar ao topo" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:776 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Detailed evidence notes{ph1} include the recorded comparisons and " +"experiment context." +msgstr "" +"As {ph0}notas detalhadas de evidências{ph1} incluem as comparações " +"registradas e o contexto dos experimentos." + +#~ msgctxt "page:blog/index.html" +#~ msgid "By N0zoM1z0 · 13 October 2026 · Draft for review" +#~ msgstr "Por N0zoM1z0 · 13 de outubro de 2026 · Rascunho para revisão" + #~ msgctxt "page:index.html" #~ msgid "Setup guide" #~ msgstr "Guia de instalação" diff --git a/website/locales/ru/messages.po b/website/locales/ru/messages.po index 4349d04ef..32b973c33 100644 --- a/website/locales/ru/messages.po +++ b/website/locales/ru/messages.po @@ -32,18 +32,18 @@ msgid "Thoughts on reverse engineering." msgstr "Мысли о реверс-инжиниринге." #. {ph0}:
-#: website/public/blog/index.html:200 +#: website/public/blog/index.html:214 #, python-brace-format msgctxt "page:blog/index.html" msgid "Game reconstruction{ph0}Essay" msgstr "Реконструкция игры{ph0}Сочинение" -#: website/public/blog/index.html:202 +#: website/public/blog/index.html:216 msgctxt "page:blog/index.html" msgid "The industrial revolution in reverse engineering" msgstr "Промышленная революция в области реверс-инжиниринга" -#: website/public/blog/index.html:203 +#: website/public/blog/index.html:217 msgctxt "page:blog/index.html" msgid "" "What manual RE and reconstructing TH08, TH095 and TH04 taught me about " @@ -52,25 +52,26 @@ msgstr "" "Чему меня научило руководство по обновлению и реконструкции TH08, TH095 и " "TH04 об агентах, проверке и знаниях, которые используются в разных проектах." -#: website/public/blog/index.html:208 +#: website/public/blog/index.html:207 website/public/blog/index.html:222 msgctxt "page:blog/index.html" msgid "Read the article" msgstr "Прочтите статью" -#: website/public/blog/index.html:211 +#: website/public/blog/index.html:225 msgctxt "page:blog/index.html" msgid "By N0zoM1z0 · 10 October 2026" msgstr "Автор: N0zoM1z0 · 10 октября 2026" #. {ph0}: -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 +#: website/public/blog/reverse-engineering-models/index.html:883 #: website/public/blog/touhou-reconstruction/index.html:1080 #: website/public/examples/dino-lab/index.html:475 #: website/public/faq/index.html:573 #: website/public/first-investigation/index.html:523 #: website/public/get-started/index.html:882 -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 #: website/public/guides/index.html:282 #: website/public/guides/javascript/index.html:503 #: website/public/guides/native/index.html:509 website/public/index.html:1191 @@ -100,6 +101,8 @@ msgstr "" "REA и агентами искусственного интеллекта." #: website/public/blog/index.html:75 website/public/blog/index.html:88 +#: website/public/blog/reverse-engineering-models/index.html:75 +#: website/public/blog/reverse-engineering-models/index.html:88 #: website/public/blog/touhou-reconstruction/index.html:75 #: website/public/blog/touhou-reconstruction/index.html:88 #: website/public/examples/dino-lab/index.html:75 @@ -136,6 +139,7 @@ msgstr "" " с помощью вашего программиста." #: website/public/blog/index.html:108 +#: website/public/blog/reverse-engineering-models/index.html:108 #: website/public/blog/touhou-reconstruction/index.html:108 #: website/public/examples/dino-lab/index.html:109 #: website/public/faq/index.html:108 @@ -156,6 +160,7 @@ msgid "REA home" msgstr "Главная REA" #: website/public/blog/index.html:112 +#: website/public/blog/reverse-engineering-models/index.html:112 #: website/public/blog/touhou-reconstruction/index.html:112 #: website/public/examples/dino-lab/index.html:113 #: website/public/faq/index.html:112 @@ -176,6 +181,7 @@ msgid "Main navigation" msgstr "Основная навигация" #: website/public/blog/index.html:130 +#: website/public/blog/reverse-engineering-models/index.html:130 #: website/public/blog/touhou-reconstruction/index.html:130 #: website/public/examples/dino-lab/index.html:131 #: website/public/faq/index.html:130 @@ -196,6 +202,7 @@ msgid "REA on GitHub" msgstr "REA на GitHub" #: website/public/blog/index.html:150 +#: website/public/blog/reverse-engineering-models/index.html:150 #: website/public/blog/touhou-reconstruction/index.html:150 #: website/public/examples/dino-lab/index.html:151 #: website/public/faq/index.html:150 @@ -216,6 +223,7 @@ msgid "Join REA on Discord" msgstr "Присоединиться к REA в Discord" #: website/public/blog/index.html:169 +#: website/public/blog/reverse-engineering-models/index.html:169 #: website/public/blog/touhou-reconstruction/index.html:169 #: website/public/examples/dino-lab/index.html:170 #: website/public/faq/index.html:169 @@ -235,7 +243,7 @@ msgctxt "shared" msgid "Follow REA on X" msgstr "Подписаться на REA в X" -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 msgctxt "page:blog/index.html" msgid "Back to top" msgstr "Наверх" @@ -4804,7 +4812,7 @@ msgctxt "page:guides/browser/index.html" msgid "Copy read the loaded scripts" msgstr "Копировать читать загруженные скрипты" -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 msgctxt "page:guides/browser/index.html" msgid "Back to top" msgstr "Наверх" @@ -11732,6 +11740,7 @@ msgid "Same spacing: 256 ÷ 16 = 16 angle units = 22.5°" msgstr "Одинаковое расстояние: 256 ÷ 16 = 16 угловых единиц = 22,5°" #: website/public/blog/index.html:114 +#: website/public/blog/reverse-engineering-models/index.html:114 #: website/public/blog/touhou-reconstruction/index.html:114 #: website/public/examples/dino-lab/index.html:115 #: website/public/faq/index.html:114 @@ -11752,6 +11761,7 @@ msgid "Showcases" msgstr "Примеры" #: website/public/blog/index.html:115 +#: website/public/blog/reverse-engineering-models/index.html:115 #: website/public/blog/touhou-reconstruction/index.html:115 #: website/public/examples/dino-lab/index.html:116 #: website/public/faq/index.html:115 @@ -11772,6 +11782,7 @@ msgid "Blog" msgstr "Блог" #: website/public/blog/index.html:113 +#: website/public/blog/reverse-engineering-models/index.html:113 #: website/public/blog/touhou-reconstruction/index.html:113 #: website/public/examples/dino-lab/index.html:114 #: website/public/faq/index.html:113 @@ -11792,6 +11803,7 @@ msgid "Get started" msgstr "Начать" #: website/public/blog/index.html:116 +#: website/public/blog/reverse-engineering-models/index.html:116 #: website/public/blog/touhou-reconstruction/index.html:116 #: website/public/examples/dino-lab/index.html:117 #: website/public/faq/index.html:116 @@ -11811,13 +11823,14 @@ msgctxt "shared" msgid "Guides" msgstr "Руководства" -#: website/public/blog/index.html:222 +#: website/public/blog/index.html:236 +#: website/public/blog/reverse-engineering-models/index.html:875 #: website/public/blog/touhou-reconstruction/index.html:1072 #: website/public/examples/dino-lab/index.html:467 #: website/public/faq/index.html:565 #: website/public/first-investigation/index.html:515 #: website/public/get-started/index.html:874 -#: website/public/guides/browser/index.html:490 +#: website/public/guides/browser/index.html:499 #: website/public/guides/index.html:274 #: website/public/guides/javascript/index.html:495 #: website/public/guides/native/index.html:501 website/public/index.html:1183 @@ -11831,13 +11844,14 @@ msgctxt "shared" msgid "Source" msgstr "Исходный код" -#: website/public/blog/index.html:223 +#: website/public/blog/index.html:237 +#: website/public/blog/reverse-engineering-models/index.html:876 #: website/public/blog/touhou-reconstruction/index.html:1073 #: website/public/examples/dino-lab/index.html:468 #: website/public/faq/index.html:566 #: website/public/first-investigation/index.html:516 #: website/public/get-started/index.html:875 -#: website/public/guides/browser/index.html:491 +#: website/public/guides/browser/index.html:500 #: website/public/guides/index.html:275 #: website/public/guides/javascript/index.html:496 #: website/public/guides/native/index.html:502 website/public/index.html:1184 @@ -11851,13 +11865,14 @@ msgctxt "shared" msgid "MIT license" msgstr "Лицензия MIT" -#: website/public/blog/index.html:224 +#: website/public/blog/index.html:238 +#: website/public/blog/reverse-engineering-models/index.html:877 #: website/public/blog/touhou-reconstruction/index.html:1074 #: website/public/examples/dino-lab/index.html:469 #: website/public/faq/index.html:567 #: website/public/first-investigation/index.html:517 #: website/public/get-started/index.html:876 -#: website/public/guides/browser/index.html:492 +#: website/public/guides/browser/index.html:501 #: website/public/guides/index.html:276 #: website/public/guides/javascript/index.html:497 #: website/public/guides/native/index.html:503 website/public/index.html:1185 @@ -11871,13 +11886,14 @@ msgctxt "shared" msgid "Join the community" msgstr "Вступить в сообщество" -#: website/public/blog/index.html:225 +#: website/public/blog/index.html:239 +#: website/public/blog/reverse-engineering-models/index.html:878 #: website/public/blog/touhou-reconstruction/index.html:1075 #: website/public/examples/dino-lab/index.html:470 #: website/public/faq/index.html:568 #: website/public/first-investigation/index.html:518 #: website/public/get-started/index.html:877 -#: website/public/guides/browser/index.html:493 +#: website/public/guides/browser/index.html:502 #: website/public/guides/index.html:277 #: website/public/guides/javascript/index.html:498 #: website/public/guides/native/index.html:504 website/public/index.html:1186 @@ -12822,6 +12838,1403 @@ msgctxt "runtime" msgid "Switch to dark mode" msgstr "Переключить на тёмную тему" +#. {ph0}: +#. {ph1}: +#: website/public/guides/browser/index.html:480 +#, python-brace-format +msgctxt "page:guides/browser/index.html" +msgid "" +"For HTTP request and response bodies, use {ph0}Fiddler Everywhere: SAZ " +"captures, MCP and public skills{ph1}. Export SAZ or HAR 1.2 for offline REA " +"capture Evidence, or connect Everywhere's MCP server to inspect captured " +"traffic with your agent." +msgstr "" +"Для изучения тел запросов и ответов HTTP используйте {ph0}Fiddler " +"Everywhere: захваты SAZ, MCP и общедоступные навыки{ph1}. Экспортируйте SAZ " +"??? HAR 1.2 для получения Evidence захвата в REA без подключения к сети или " +"подключите MCP-сервер Everywhere, чтобы исследовать записанный трафик вместе" +" с агентом." + +#. {ph0}:
+#: website/public/blog/index.html:200 +#, python-brace-format +msgctxt "page:blog/index.html" +msgid "Model specialization{ph0}Essay" +msgstr "Специализация моделей{ph0}Эссе" + +#: website/public/blog/index.html:202 +msgctxt "page:blog/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "Нужна ли для обратной разработки более крупная модель?" + +#: website/public/blog/index.html:203 +msgctxt "page:blog/index.html" +msgid "" +"What Touhou reconstruction suggests about general models, specialist models " +"and learning from compiler feedback." +msgstr "" +"Что реконструкция Touhou помогает понять об универсальных и " +"специализированных моделях и обучении на обратной связи от компилятора." + +#: website/public/blog/index.html:210 +msgctxt "page:blog/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Автор N0zoM1z0 · 13 октября 2026 г." + +#: website/public/blog/reverse-engineering-models/index.html:103 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Skip to content" +msgstr "Перейти к содержанию" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:194 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Home{ph1} / {ph2}Blog{ph3} / Models and RE" +msgstr "" +"{ph0}Главная страница{ph1} / {ph2}Блог{ph3} / Модели и реверс-инжиниринг" + +#: website/public/blog/reverse-engineering-models/index.html:197 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Essay" +msgstr "Эссе" + +# Reviewed for technical meaning and readability; preserve the distinction +# between historical evidence and proposed experiments. +#: website/public/blog/reverse-engineering-models/index.html:198 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "Нужна ли реверс-инжинирингу модель большего размера?" + +#: website/public/blog/reverse-engineering-models/index.html:199 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Автор N0zoM1z0 · 13 октября 2026 г." + +#: website/public/blog/reverse-engineering-models/index.html:201 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Takeaway" +msgstr "Главное" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:203 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Model capability changes the hypotheses an agent can find.{ph1} Later " +"sessions have solved problems that earlier work left unresolved." +msgstr "" +"{ph0}Возможности модели влияют на то, какие гипотезы может предложить ИИ-" +"агент.{ph1} На последующих занятиях решались проблемы, которые ранее " +"оставались нерешенными." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:211 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A familiar compiler gives specialization something concrete to " +"learn.{ph1} Some exact matches depend on narrow, reusable source patterns." +msgstr "" +"{ph0}Знакомый компилятор предлагает конкретные шаблоны для изучения " +"специализированной модели.{ph1} Некоторые точные совпадения машинного кода " +"зависят от конкретных многократно используемых шаблонов исходного кода." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:218 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Give repository knowledge a fair comparison.{ph1} A smaller model with " +"good retrieval may already know enough to make progress." +msgstr "" +"{ph0}Включите знания репозитория в справедливое сравнение.{ph1} Меньшая " +"модель с хорошим извлечением знаний может уже иметь достаточно информации " +"для достижения прогресса." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:223 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Train the next useful experiment, and test its oracle.{ph1} The value " +"of feedback depends on what the checker actually verifies." +msgstr "" +"{ph0}Обучите модель выбирать следующий полезный эксперимент и протестируйте " +"ее средство проверки.{ph1} Обратная связь полезна только в той степени, в " +"которой средство проверки проверяет правильные вещи." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:230 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Measure accepted reconstruction against time and cost.{ph1} That is the" +" result I want from the whole agent." +msgstr "" +"{ph0}Оцените проверенные результаты реконструкции, а также время и " +"стоимость.{ph1} Это то, что я хочу измерить для всего рабочего процесса ИИ-" +"агента." + +#: website/public/blog/reverse-engineering-models/index.html:241 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "In this article" +msgstr "В этой статье" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:243 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What better models changed{ph1}" +msgstr "{ph0}Что изменили более сильные модели{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:244 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Why specialize?{ph1}" +msgstr "{ph0}Зачем специализироваться?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:245 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Where experience lives{ph1}" +msgstr "{ph0}Где следует хранить этот опыт?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:246 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Learning the next experiment{ph1}" +msgstr "{ph0}Изучаем следующий полезный эксперимент{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:247 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}RE and reinforcement learning{ph1}" +msgstr "{ph0}Обратное проектирование и обучение с подкреплением{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:248 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}When the oracle is wrong{ph1}" +msgstr "{ph0}Когда программа проверки ошибается{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:249 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}A comparison worth running{ph1}" +msgstr "{ph0}Сравнение, которое стоит провести{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:250 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What kind of agent?{ph1}" +msgstr "{ph0}О каком ИИ-агенте идет речь?{ph1}" + +#: website/public/blog/reverse-engineering-models/index.html:258 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What better models changed" +msgstr "Что изменили более сильные модели" + +#: website/public/blog/reverse-engineering-models/index.html:259 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Since August 2026, I have been exploring how much reverse engineering an " +"agent can carry out with access to a program and a way to check its work. " +"Most of my experience comes from reconstructing games, especially Touhou. As" +" I have changed models, the repositories have kept a record of what each " +"session managed to recover." +msgstr "" +"С августа 2026 года я изучаю, какой объем реверс-инжиниринга может провести " +"агент, имея доступ к программе и способ проверки ее работы. Большая часть " +"моего опыта связана с реконструкцией игр, особенно Touhou. Поскольку я менял" +" модели, в репозиториях фиксировалось, что удалось восстановить в каждом " +"сеансе." + +#: website/public/blog/reverse-engineering-models/index.html:267 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of that work aims at exact reconstruction: we compile the maintained " +"source under a pinned historical toolchain and check whether it reproduces " +"the target function. When a later model succeeds where earlier work stalled," +" the source changes show what made that progress possible." +msgstr "" +"Большая часть этой работы направлена на точную реконструкцию: мы компилируем" +" поддерживаемый исходный код с помощью исторической цепочки инструментов с " +"фиксированными версиями и проверяем, воспроизводит ли он целевую функцию. " +"Когда более поздняя модель оказывается успешной там, где предыдущие работы " +"застопорились, изменения в исходном коде показывают, что сделало этот " +"прогресс возможным." + +#: website/public/blog/reverse-engineering-models/index.html:274 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The temporary values had to stay alive together" +msgstr "Временным переменным требовалось перекрывающееся время жизни." + +#: website/public/blog/reverse-engineering-models/index.html:275 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of a TH09 attack function's logic had already been reconstructed, but " +"the compiler still stored temporary results differently from the original. " +"The remaining problem was how long those results needed to stay alive and " +"when their storage could be reused." +msgstr "" +"Большая часть логики функции атаки TH09 уже была реконструирована, но " +"компилятор по-прежнему сохранял временные результаты иначе, чем оригинал. " +"Оставшаяся проблема заключалась в том, как долго эти результаты должны " +"оставаться живыми и когда их хранилище можно будет использовать повторно." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:282 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A {ph0}GPT-6-Sol checkpoint{ph1} recorded this mismatch after testing " +"several ordinary source arrangements. When {ph2}GPT-6.1-Sol returned to the " +"problem{ph3}, it found a useful hypothesis about how long those temporary " +"values needed to stay alive." +msgstr "" +"Запись сеанса {ph0}GPT-6-Sol{ph1} сообщила об этом несоответствии после " +"тестирования нескольких обычных схем исходного кода. Когда {ph2}GPT-6.1-Sol " +"снова изучил проблему{ph3}, он предложил полезную гипотезу о времени жизни " +"этих временных переменных." + +#: website/public/blog/reverse-engineering-models/index.html:296 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The function produced temporary results and copied them into animation " +"positions. In C++, a temporary result stays alive until the expression using" +" it finishes. Grouping the real operations into one expression kept those " +"results alive together, so the compiler could no longer reuse the same " +"storage for all of them." +msgstr "" +"Функция выдавала временные результаты и копировала их в позиции анимации. В " +"C++ временный результат остается активным до тех пор, пока выражение, " +"использующее его, не завершится. Группировка реальных операций в одно " +"выражение сохраняла эти результаты вместе, поэтому компилятор больше не мог " +"повторно использовать одно и то же хранилище для всех них." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:303 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"One result still shared storage with a value used later for collision " +"checks. Giving that value its own lifetime resolved the remaining overlap. " +"After several targeted experiments, {ph0}a clean rebuild confirmed that the " +"full function matched{ph1}." +msgstr "" +"Один из результатов по-прежнему разделял хранилище со значением, которое " +"позже использовалось для проверок на коллизии. Предоставление этому значению" +" собственного времени жизни решило оставшееся перекрытие. После нескольких " +"целевых экспериментов {ph0}чистая перекомпиляция из исходного кода " +"подтвердила, что полная функция соответствует{ph1}." + +#: website/public/blog/reverse-engineering-models/index.html:312 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What interests me is how the discovery and the tests worked together. The " +"lifetime hypothesis gave the agent a new direction; compiler feedback helped" +" it refine that hypothesis into a source arrangement that matched." +msgstr "" +"Меня интересует, как открытие и испытания поддержали друг друга. Гипотеза о " +"времени жизни временных объектов дала агенту ИИ новое направление. Отзывы " +"компилятора помогли усовершенствовать эту гипотезу и привести ее в " +"соответствие с исходным кодом." + +#: website/public/blog/reverse-engineering-models/index.html:318 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A different control-flow graph" +msgstr "Другой граф потока управления" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:319 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A second TH09 function shows why that new direction can matter. Here, the " +"obstacle was how several branches shared their final work. " +"{ph0}GPT-6.1-Sol's earlier work{ph1} got parts of the function to match, but" +" the branches still joined up differently from the original." +msgstr "" +"Вторая функция TH09 показывает, почему это новое направление может иметь " +"значение. Здесь препятствием было то, как несколько филиалов делились своей " +"итоговой работой. В более ранней работе {ph0}GPT-6.1-Sol{ph1} части функции " +"совпадали, но ветви по-прежнему соединялись иначе, чем в оригинале." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:330 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}GPT-6-Astra changed the surrounding control-flow graph{ph1}. The main " +"paths now led to a shared final update and return. In that source structure," +" the compiler combined the branches in the same way as the original. The " +"complete function matched, and a {ph2}separate rebuild check confirmed " +"it{ph3}." +msgstr "" +"{ph0}GPT-6-Astra изменил окружающий граф потока управления{ph1}. Основные " +"пути теперь вели к общему финальному обновлению и возврату. В этой исходной " +"структуре компилятор объединил ветки так же, как в оригинале. Полная функция" +" совпала, и {ph2}отдельная проверка перестроения подтвердила это{ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:342 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Changing the representation opened a more productive search space.{ph1}" +" Both cases give me a reason to take general model capability seriously: an " +"agent can keep trying local variations for a long time before finding a " +"better way to express the surrounding work." +msgstr "" +"{ph0}Изменение представления открыло более продуктивное пространство " +"поиска.{ph1} Оба случая дают мне повод серьезно отнестись к общим " +"возможностям модели: агент может долго пробовать локальные варианты, прежде " +"чем найти лучший способ выразить окружающую работу." + +#: website/public/blog/reverse-engineering-models/index.html:351 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"At the same time, the later sessions started with earlier source and " +"recorded failures. That accumulated work was part of their advantage. The " +"history lets us inspect the discoveries and their checks; a controlled " +"comparison would help separate the effect of the model from the experience " +"it inherited." +msgstr "" +"В то же время более поздние сеансы начинались с более раннего источника и " +"фиксировали сбои. Этот накопленный труд был частью их преимущества. История " +"позволяет нам проверять открытия и их проверки; контролируемое сравнение " +"поможет отделить эффект модели от унаследованного ею опыта." + +#: website/public/blog/reverse-engineering-models/index.html:358 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is where the question becomes interesting to me. Some of the progress " +"came from finding a new representation, while much of the remaining work " +"involved testing source forms against a familiar compiler. How much needs " +"broad reasoning ability, and how much could a model learn through repeated " +"experience with that compiler and code?" +msgstr "" +"Именно это делает вопрос для меня интересным. Некоторый прогресс был " +"достигнут благодаря поиску нового представительства. Большая часть " +"оставшейся работы заключалась в тестировании различных вариантов исходного " +"кода на знакомом компиляторе. Насколько это требует широких умственных " +"способностей? Насколько многому может научиться модель, неоднократно работая" +" с этим компилятором и кодом?" + +#: website/public/blog/reverse-engineering-models/index.html:368 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why might a specialist do well?" +msgstr "Почему специализированная модель ИИ может преуспеть?" + +#: website/public/blog/reverse-engineering-models/index.html:369 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Touhou gives us a concrete reason to expect that experience to help. Much of" +" the game code comes from the same author, ZUN, so recurring habits are a " +"reasonable starting assumption. The repositories back that up with useful " +"examples: related loops and helper arrangements have supplied hypotheses " +"across games." +msgstr "" +"Touhou дает нам конкретную причину ожидать, что этот опыт поможет. Большая " +"часть игрового кода принадлежит одному и тому же автору, ZUN, поэтому " +"повторяющиеся привычки — разумное исходное предположение. Репозитории " +"подтверждают это полезными примерами: связанные циклы и вспомогательные " +"механизмы служат источником гипотез для разных игр." + +#: website/public/blog/reverse-engineering-models/index.html:376 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those source patterns become more specific when we include the compiler. " +"Under a particular version and set of options, small changes in source can " +"produce repeatable differences in machine code. Some of those differences " +"are surprisingly obscure." +msgstr "" +"Эти исходные шаблоны становятся более конкретными, когда мы подключаем " +"компилятор. В конкретной версии и наборе опций небольшие изменения в " +"исходном коде могут привести к повторяющимся различиям в машинном коде. " +"Некоторые из этих различий на удивление неясны." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: Supervisor::LoadDat +#. {ph4}: +#: website/public/blog/reverse-engineering-models/index.html:382 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In TH095, for example, {ph0}even the name used for a local variable could " +"change where VC7.1 stored it{ph1}. Moving the declaration alone did not " +"solve the layout, but an earlier background function had revealed a naming " +"pattern that affected the compiler's choice. That gave us a more promising " +"hypothesis for {ph2}the reconstruction of {ph3}{ph4}." +msgstr "" +"Например, в TH095 {ph0}даже имя локальной переменной может влиять на то, где" +" VC7.1 ее хранит{ph1}. Перемещение его декларации не решило проблему с " +"расположением хранилища. Однако более ранняя фоновая функция выявила шаблон " +"именования, который повлиял на выбор компилятора. Это предложило более " +"многообещающую гипотезу для {ph2}реконструкции {ph3}{ph4}." + +#: website/public/blog/reverse-engineering-models/index.html:396 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reusing that pattern for the file-size variable produced the required layout" +" without adding storage. The full function then matched." +msgstr "" +"Повторное использование этого шаблона для переменной размера файла позволило" +" получить требуемый макет без увеличения объема памяти. Затем полная функция" +" совпала." + +#: website/public/blog/reverse-engineering-models/index.html:401 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is the sort of narrow detail a general model may not know when it " +"arrives. A specialist trained on this compiler could encounter many such " +"cases and learn which source variations are worth testing first." +msgstr "" +"Это очень специфическая деталь, о которой модель ИИ общего назначения может " +"не знать, когда впервые сталкивается с проблемой. Модель, специализированная" +" для этого компилятора, могла бы увидеть множество таких случаев и " +"определить, какие варианты исходного кода стоит протестировать в первую " +"очередь." + +#: website/public/blog/reverse-engineering-models/index.html:407 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The opportunity is especially clear in final matching problems where we " +"already understand the behavior. We are then asking which ordinary source " +"form this compiler turns into the observed instructions. Specialization " +"could help the model propose those forms more efficiently, while the target " +"still determines which hypotheses survive." +msgstr "" +"Эта возможность особенно очевидна в задачах окончательного сопоставления, " +"где мы уже понимаем поведение. Затем мы спрашиваем, какую обычную структуру " +"исходного кода этот компилятор превращает в наблюдаемые инструкции. " +"Специализация может помочь модели более эффективно предлагать эти формы, в " +"то время как цель по-прежнему определяет, какие гипотезы выживут." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:415 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The useful patterns would have to retain their context, though. TH08 uses an" +" earlier Visual C++ version than TH095 and TH09. TH04 uses a different DOS-" +"era compiler, and its {ph0}player reconstruction{ph1} needed different " +"hypotheses about source patterns and how the compiled code was combined into" +" an executable. Shared authorship gives us a prior, but each target tells us" +" whether it applies. Distinguishing those contexts is part of what a " +"specialist would need to learn." +msgstr "" +"Полезные шаблоны должны сохранять свой контекст. TH08 использует более " +"раннюю версию Visual C++, чем TH095 и TH09. TH04 использует другой " +"компилятор эпохи DOS. Его {ph0}реконструкция кода игрока{ph1} требовала " +"различных гипотез о шаблонах исходного кода и о том, как скомпилированный " +"код связывается с исполняемым файлом. Наличие одного и того же автора дает " +"нам первоначальное ожидание, но каждая цель говорит нам, применимо ли оно. " +"Специализированная модель должна будет различать эти контексты." + +#: website/public/blog/reverse-engineering-models/index.html:431 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Where should that experience live?" +msgstr "Где следует хранить этот опыт?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:432 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before deciding to teach those patterns through model training, I would look" +" at the knowledge we already keep in the repository. In the {ph0}previous " +"essay{ph1}, I wrote about it becoming the project's working memory. That " +"memory includes compiler lessons, along with the source and failed " +"experiments that made them useful." +msgstr "" +"Прежде чем принять решение об обучении этим шаблонам посредством обучения " +"моделей, я бы посмотрел на знания, которые мы уже храним в репозитории. В " +"{ph0}предыдущем эссе{ph1} я писал о том, что оно стало рабочей памятью " +"проекта. Эта память включает в себя уроки компилятора, а также исходники и " +"неудачные эксперименты, которые сделали их полезными." + +#: website/public/blog/reverse-engineering-models/index.html:441 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"By retrieving that record, an agent can start with a known result and " +"inspect how we reached it. It can also avoid alternatives that have already " +"failed in the same context. The starting point improves without changing the" +" model's weights." +msgstr "" +"Получив эту запись, агент может начать с известного результата и проверить, " +"как мы его достигли. Это также позволяет избежать альтернатив, которые уже " +"потерпели неудачу в том же контексте. Начальная точка улучшается без " +"изменения обученных параметров модели." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:447 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"When reconstructing TH09's enemy-spawn function, a {ph0}GPT-6-Luna-max " +"session{ph1} drew a hypothesis from TH08's corresponding source family. That" +" earlier source suggested how to structure the loop and handle failures " +"within it. TH09's own evidence supplied the game-specific details, and clean" +" rebuilds confirmed the complete match." +msgstr "" +"При реконструкции функции TH09 по созданию врагов сеанс {ph0}GPT-6-Luna-" +"max{ph1} взял гипотезу из соответствующего исходного кода TH08. Этот более " +"ранний код предлагал, как организовать цикл и обрабатывать сбои внутри него." +" Собственные доказательства TH09 предоставили подробности, касающиеся " +"конкретной игры. Перестройки из исходного кода подтвердили, что вся функция " +"совпадает." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:458 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}\"The model did not know that compiler quirk\" and \"the model could " +"not use it\" are different problems.{ph1} Retrieval may solve the first, but" +" turning that fact into a useful candidate still takes work. And if the " +"candidate fails, the model must decide what the result means for its next " +"attempt. Knowing the relevant pattern is only the beginning of that process." +msgstr "" +"{ph0}\"Модель не знала об этой особенности компилятора\" и \"модель не могла" +" ее использовать\" — это разные проблемы.{ph1} Извлечение может решить " +"первую, но превращение этого факта в полезного кандидата все равно требует " +"работы. И если исходный код-кандидат терпит неудачу, модель должна решить, " +"что означает результат для ее следующей попытки. Знание соответствующей " +"закономерности — это только начало этого процесса." + +#: website/public/blog/reverse-engineering-models/index.html:468 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For that reason, I would include a smaller general model with good retrieval" +" in any comparison. Giving it the same library of experience would help us " +"identify what training adds: better candidate generation, better decisions " +"after feedback, or both." +msgstr "" +"По этой причине я бы включил в любое сравнение меньшую модель ИИ общего " +"назначения с хорошей воспроизводимостью. Предоставление ему той же " +"библиотеки опыта поможет нам определить, что дает обучение: лучшее " +"генерирование кандидатов, лучшие решения после обратной связи или и то, и " +"другое." + +#: website/public/blog/reverse-engineering-models/index.html:476 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Learning the next useful experiment" +msgstr "Изучаем следующий полезный эксперимент" + +#: website/public/blog/reverse-engineering-models/index.html:477 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those decisions after feedback are what I would most like to study. In " +"agentic reconstruction, the model proposes an explanation and tests it " +"against the program. Even when the candidate fails, the result can narrow " +"the remaining possibilities and suggest a more useful next experiment." +msgstr "" +"Эти решения после получения обратной связи — это то, что мне больше всего " +"хотелось бы изучить. При агентной реконструкции модель предлагает объяснение" +" и проверяет его на соответствие программе. Даже если исходный код-кандидат " +"дает сбой, результат может сузить оставшиеся возможности и предложить более " +"полезный следующий эксперимент." + +#: website/public/blog/reverse-engineering-models/index.html:484 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Consider the temporary-results example again. Its final source shows the " +"arrangement that worked, but the route to that source contains more " +"information. Each of the following experiments helped identify what still " +"needed to change." +msgstr "" +"Снова рассмотрим пример временных результатов. Его окончательный источник " +"показывает сработавшую схему, но путь к этому источнику содержит больше " +"информации. Каждый из следующих экспериментов помог определить, что еще " +"необходимо изменить." + +#: website/public/blog/reverse-engineering-models/index.html:497 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Experiment" +msgstr "Эксперимент" + +#: website/public/blog/reverse-engineering-models/index.html:498 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What its result taught us" +msgstr "Чему нас научил его результат" + +#: website/public/blog/reverse-engineering-models/index.html:503 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Try a different way to copy the results" +msgstr "Попробуйте другой способ скопировать результаты" + +#: website/public/blog/reverse-engineering-models/index.html:504 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The copy code changes, but the function still differs." +msgstr "Код копирования меняется, но функция по-прежнему отличается." + +#: website/public/blog/reverse-engineering-models/index.html:509 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Use the returned values directly" +msgstr "Используйте возвращаемые значения напрямую" + +#: website/public/blog/reverse-engineering-models/index.html:510 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Copying follows the original pattern, but temporary storage still differs." +msgstr "" +"Копирование следует исходному шаблону, но временное хранилище все равно " +"отличается." + +#: website/public/blog/reverse-engineering-models/index.html:516 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the temporary results alive together" +msgstr "Сохраняйте все временные переменные действительными одновременно" + +#: website/public/blog/reverse-engineering-models/index.html:517 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"They get separate storage, but one still overlaps a value used later for " +"collision checks." +msgstr "" +"Они получают отдельное хранилище, но одно из них по-прежнему перекрывает " +"значение, используемое позже для проверок коллизий." + +#: website/public/blog/reverse-engineering-models/index.html:523 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the collision-check value separate" +msgstr "Держите значение проверки столкновений отдельно" + +#: website/public/blog/reverse-engineering-models/index.html:524 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A clean rebuild confirms the complete match." +msgstr "" +"Чистая перекомпиляция из исходного кода подтверждает полное совпадение." + +#: website/public/blog/reverse-engineering-models/index.html:529 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Selected experiments from the attack-function reconstruction above, grouped " +"by what they taught us." +msgstr "" +"Избранные эксперименты из приведенной выше реконструкции атакующей функции, " +"сгруппированные по тому, чему они нас научили." + +#: website/public/blog/reverse-engineering-models/index.html:533 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A training record that keeps this sequence can teach more than the final " +"source alone. It connects each proposed change to an observed result and the" +" next decision. Keeping ineffective variations matters too: they tell a " +"later agent which hypotheses have already been tested in that context." +msgstr "" +"Запись обучения, в которой сохраняется эта последовательность, может научить" +" большему, чем один только окончательный источник. Он связывает каждое " +"предлагаемое изменение с наблюдаемым результатом и следующим решением. " +"Сохранение неэффективных вариантов также имеет значение: они сообщают " +"следующему агенту, какие гипотезы уже были проверены в этом контексте." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:540 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}I would want the model to learn how to use feedback.{ph1} A neutral " +"result is a reason to reconsider the variation; a changed mismatch may point" +" to another cause. When local edits stop being informative, the agent should" +" be able to revisit the source representation or inspect the evidence again." +msgstr "" +"{ph0}Я бы хотел, чтобы модель научилась использовать обратную связь.{ph1} " +"Нейтральный результат — повод пересмотреть вариант; измененное " +"несоответствие может указывать на другую причину. Когда локальные изменения " +"перестают быть информативными, агент должен иметь возможность вернуться к " +"исходному представлению или снова проверить доказательства." + +#: website/public/blog/reverse-engineering-models/index.html:549 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Learning to make those choices requires autonomy. Sometimes the useful next " +"step is inspecting another caller rather than compiling another edit. A " +"rigid sequence of required rewrites would remove that choice, along with " +"part of the behavior we want the model to learn." +msgstr "" +"Чтобы научиться делать такой выбор, требуется автономия. Иногда полезным " +"следующим шагом является проверка другого вызывающего объекта, а не " +"компиляция еще одного редактирования. Жесткая последовательность необходимых" +" переписываний устранит этот выбор вместе с частью поведения, которому мы " +"хотим, чтобы модель научилась." + +#: website/public/blog/reverse-engineering-models/index.html:558 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why RE is interesting for reinforcement learning" +msgstr "Чем реверс-инжиниринг интересен для обучения с подкреплением" + +#: website/public/blog/reverse-engineering-models/index.html:559 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Bounded reconstruction gives us a practical environment for studying these " +"choices. Once the target and compiler are pinned, many source experiments " +"produce quick, repeatable feedback. The agent can compile a candidate, " +"inspect the mismatch and use that observation to choose its next step." +msgstr "" +"Реконструкция с заданными границами даёт практическую среду для изучения " +"этих решений. Когда версии цели и компилятора зафиксированы, многие " +"эксперименты с исходным кодом дают быструю, воспроизводимую обратную связь. " +"Агент может скомпилировать кандидат, изучить различия и выбрать следующий " +"шаг." + +#: website/public/blog/reverse-engineering-models/index.html:566 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reinforcement learning could train that decision-making process through the " +"feedback it receives. A candidate accepted by a defined check gives us a " +"concrete outcome, while the sequence of attempts lets us observe how much " +"work it took to reach it." +msgstr "" +"Обучение с подкреплением могло бы тренировать этот процесс принятия решений " +"посредством получаемой обратной связи. Кандидат, принятый посредством " +"определенной проверки, дает нам конкретный результат, а последовательность " +"попыток позволяет нам наблюдать, сколько работы потребовалось для его " +"достижения." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:572 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"There is already research supporting this direction. {ph0}LLM4Decompile{ph1}" +" studies models specialized for decompilation. {ph2}AutoDecompiler{ph3} " +"trains multi-turn refinement using compiler and execution feedback. Their " +"studies focus on functional recovery. For this work, I would use them as a " +"starting point for the related task of finding source that matches a " +"historical compiler's output." +msgstr "" +"Уже есть исследования, подтверждающие это направление. " +"{ph0}LLM4Decompile{ph1} изучает модели, предназначенные для декомпиляции. " +"{ph2}AutoDecompiler{ph3} обучает многоэтапному уточнению с использованием " +"компилятора и обратной связи при выполнении. Их исследования сосредоточены " +"на функциональном восстановлении. В этой работе я бы использовал их в " +"качестве отправной точки для решения соответствующей задачи по поиску " +"источника, соответствующего выводам исторического компилятора." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:584 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To reach such a well-defined task, though, we may first need to establish " +"what a function owns or devise a new runtime measurement. In TH08, for " +"example, a {ph0}collision difference led back to how the original program " +"handled floating-point calculations{ph1}. Before the agent could test a " +"useful correction, it had to work out what to measure." +msgstr "" +"Однако для решения такой четко определенной задачи нам, возможно, сначала " +"придется установить, какими ресурсами владеет функция, или разработать новый" +" способ наблюдения за поведением программы во время выполнения. Например, в " +"TH08 {ph0}разность коллизий привела к тому, как исходная программа " +"обрабатывала вычисления с плавающей запятой{ph1}. Прежде чем агент смог " +"протестировать полезную поправку, он должен был решить, что именно измерять." + +#: website/public/blog/reverse-engineering-models/index.html:595 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This suggests another role for a broadly capable model: helping make the " +"problem checkable in the first place. Once that check exists, we can " +"investigate how far a learned search policy can get with it." +msgstr "" +"Это предполагает еще одну роль модели с широкими возможностями: в первую " +"очередь помочь сделать проблему проверяемой. Как только эта проверка " +"появится, мы сможем выяснить, насколько далеко может зайти изученная " +"политика поиска." + +#: website/public/blog/reverse-engineering-models/index.html:603 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "When the oracle is wrong" +msgstr "Когда программа проверки ошибается" + +#: website/public/blog/reverse-engineering-models/index.html:604 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Once we train against that check, its coverage becomes part of what the " +"model learns. I call the reference or checker an oracle. Even if it gives a " +"repeatable answer, we still need to understand what that answer covers." +msgstr "" +"Когда проверка используется для обучения, её охват становится частью среды " +"обучения. Я называю такой эталон или проверяющий инструмент оракулом. Даже " +"воспроизводимый ответ нужно понимать в пределах того, что он действительно " +"проверяет." + +#: website/public/blog/reverse-engineering-models/index.html:610 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"TH08 showed how much can hide outside that coverage. Some reconstructed " +"functions passed their exact comparisons with wrong floating constants. The " +"comparator adjusted relocated addresses before checking instructions, but it" +" checked the values behind those addresses only when they had been " +"explicitly recorded for verification. Other literal references could pass " +"without their values being examined." +msgstr "" +"TH08 показал, как много может скрываться за пределами этого покрытия. " +"Некоторые реконструированные функции прошли точное сравнение с неправильными" +" плавающими константами. Компаратор корректировал перемещенные адреса перед " +"проверкой инструкций, но проверял значения, стоящие за этими адресами, " +"только тогда, когда они были явно записаны для проверки. Другие буквальные " +"ссылки могут пройти без проверки их значений." + +#. {ph0}: +#. {ph1}: +#. {ph2}: 1.0f +#. {ph3}: 32.0f +#: website/public/blog/reverse-engineering-models/index.html:619 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To close that gap, a {ph0}wider audit{ph1} found incorrect constants in " +"functions that had already been accepted. In one calculation, the " +"reconstructed source added {ph2} where the original added {ph3}." +msgstr "" +"Чтобы устранить этот пробел, {ph0}более широкий аудит{ph1} обнаружил " +"неправильные константы в функциях, которые уже прошли проверку. В одном " +"расчете в реконструированный исходный код добавлен {ph2}, тогда как в " +"исходный добавлен {ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:629 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A learner rewarded by the old check could learn to produce the wrong " +"reconstruction.{ph1} We had not trained an RL model against this oracle; the" +" observed failure was in the check itself. But using that check as a reward " +"would have made the same blind spot part of the learning environment." +msgstr "" +"{ph0}Модель, получившая награду в результате старой проверки, могла " +"научиться производить неверную реконструкцию.{ph1} Мы не обучали модель RL с" +" помощью этой проверки. Сбой, который мы наблюдали, заключался в самой " +"проверке. Но если бы мы использовали это в качестве награды, то же самое " +"слепое пятно стало бы частью среды обучения." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:639 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A functional test can have a similar blind spot. {ph0}Differential-fuzzing " +"research on LLM decompilers{ph1} has found behavior differences beyond the " +"supplied tests. Passing those tests therefore needs to be read in terms of " +"the cases they exercise, just as an instruction match needs to be read in " +"terms of the bytes and data it checks." +msgstr "" +"Функциональный тест может иметь аналогичное «слепое пятно». " +"{ph0}Исследование с использованием дифференциального фаззинга в " +"декомпиляторах LLM{ph1} выявило различия в поведении за пределами " +"предоставленных тестов. Поэтому нам необходимо понимать прохождение теста с " +"точки зрения случаев, в которых он проводится. Точно так же совпадение " +"инструкций следует понимать с точки зрения байтов и данных, которые оно " +"проверяет." + +#: website/public/blog/reverse-engineering-models/index.html:649 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before using either kind of check as a reward, I would test it with " +"deliberate failures. I would also keep final acceptance outside the " +"candidate's editable source, so the model has to satisfy the check through " +"its reconstruction. Otherwise, something like a byte array copied from the " +"target could satisfy a naive score without recovering useful source." +msgstr "" +"Прежде чем использовать какой-либо вид проверки в качестве награды, я бы " +"протестировал ее с заведомо неверными результатами. Я также хотел бы хранить" +" окончательное принятие отдельно от исходного кода, который может " +"редактировать агент ИИ. Модель должна пройти проверку путем правильного " +"восстановления программы. В противном случае простое копирование массива " +"байтов из целевого объекта может обмануть наивную систему оценки без " +"восстановления полезного исходного кода." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:657 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Improving the learner and improving its oracle belong to the same " +"research problem.{ph1} As the learner gets better at optimizing the feedback" +" we give it, understanding that feedback becomes more important too." +msgstr "" +"{ph0}Улучшение модели и улучшение её проверяющего инструмента — части одной " +"исследовательской задачи.{ph1} Чем лучше модель оптимизирует получаемую " +"обратную связь, тем важнее понимать саму эту связь." + +#: website/public/blog/reverse-engineering-models/index.html:667 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A comparison worth running" +msgstr "Сравнение, которое стоит провести" + +#: website/public/blog/reverse-engineering-models/index.html:668 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"With those checks in place, I would start the comparison on a fixed VC7.1 " +"task family. First, I would establish baselines for a strong general model " +"and a smaller one. Adding repository knowledge to the smaller model would " +"show what retrieval changes. From there, supervised specialization and an " +"RL-trained version would let us examine what training adds." +msgstr "" +"Имея эти проверки, я бы начал сравнение с фиксированного семейства задач " +"VC7.1. Во-первых, я бы установил базовые показатели для сильной модели ИИ " +"общего назначения и для более мелкой модели. Добавление знаний о репозитории" +" в меньшую модель покажет, какие изменения происходят в извлечении. Отсюда " +"контролируемая специализация и версия, обученная RL, позволят нам изучить, " +"что добавляет обучение." + +#: website/public/blog/reverse-engineering-models/index.html:676 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To make those comparisons informative, each model would receive the same " +"target evidence and tools. For the training-specific comparisons, I would " +"keep the base model and available knowledge fixed as well. Repeated trials " +"would help distinguish a reliable search strategy from a lucky candidate." +msgstr "" +"Чтобы сделать эти сравнения информативными, каждая модель должна получить " +"одни и те же целевые доказательства и инструменты. Для сравнений по " +"конкретному обучению я бы также оставил фиксированной базовую модель и " +"доступные знания. Повторные испытания помогут отличить надежную стратегию " +"поиска от возможного решения, которое окажется успешным случайно." + +#: website/public/blog/reverse-engineering-models/index.html:683 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would then separate two starting conditions: reconstructing a function " +"from its instructions, and closing the last few differences in maintained " +"source. A specialist might be excellent at the second task while a general " +"model does much of the work needed to reach it. Measuring them separately " +"would reveal that division of work." +msgstr "" +"Затем я бы разделил два начальных условия: восстановление функции из ее " +"инструкций и закрытие последних нескольких различий в поддерживаемом " +"исходном коде. Специализированная модель ИИ может превосходно справиться со " +"второй задачей, в то время как модель ИИ общего назначения выполняет большую" +" часть работы, необходимой для ее достижения. Измерение их по отдельности " +"выявило бы такое разделение работы." + +#: website/public/blog/reverse-engineering-models/index.html:691 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both tasks need genuinely held-out examples. Because related Touhou games " +"share engine code, holding out a game's name would still allow nearly " +"identical functions to appear in training. I would instead hold out source " +"families and keep their accepted answers out of both training and retrieval." +msgstr "" +"Обе задачи требуют тестовых примеров, исключенных из обучения и поиска. " +"Связанные игры Touhou имеют общий код движка. Исключение одной игры из " +"обучения может привести к тому, что в обучающем наборе останутся почти " +"идентичные функции. Вместо этого я бы исключил целые группы связанного " +"исходного кода и исключил бы их проверенные ответы как из обучающих данных, " +"так и из библиотеки поиска." + +#: website/public/blog/reverse-engineering-models/index.html:698 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"On those examples, the outcome would be complete, independently validated " +"matches. Giving each model the same candidate or compile budget would help " +"compare search quality. Measuring elapsed time and total cost, including " +"compilation, would tell us whether that quality produces an efficient " +"workflow. Training costs would need their own accounting too." +msgstr "" +"В этих примерах результатом будут полные, независимо проверенные совпадения." +" Равные бюджеты для предложений-кандидатов или попыток компиляции помогут " +"сравнить качество поиска. Измерение общего времени и затрат, включая " +"компиляцию, покажет, обеспечивает ли это качество эффективный рабочий " +"процесс. Затраты на обучение следует оценивать отдельно." + +#: website/public/blog/reverse-engineering-models/index.html:706 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both views matter because a cheaper proposal can become an expensive " +"workflow if it causes more uninformative compiles. Conversely, a slower " +"model may pay for itself by finding a useful experiment sooner. The " +"comparison should capture those costs." +msgstr "" +"Оба измерения имеют значение. Дешевое предложение может привести к " +"дорогостоящему рабочему процессу, если оно приведет к увеличению количества " +"компиляций, не содержащих полезной информации. И наоборот, более медленная " +"модель может быть более экономичной, если она быстрее обнаружит полезный " +"эксперимент. Сравнение должно включать эти затраты." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:712 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Counting accepted results also keeps intermediate progress in perspective. " +"One TH09 {ph0}replay-save candidate{ph1} looked promising because the " +"compiled function had the expected size, but a full comparison still found " +"substantial differences. Changes in how the compiler arranges instructions " +"can also make a candidate look closer without resolving the underlying " +"problem. Such diagnostics can guide the next experiment, but I would " +"validate any reward for \"getting closer\" against the full acceptance " +"check." +msgstr "" +"Подсчет принятых результатов также позволяет учитывать промежуточный " +"прогресс. Один TH09 {ph0}кандидат на воспроизведение-сохранение{ph1} " +"выглядел многообещающе, поскольку скомпилированная функция имела ожидаемый " +"размер, но полное сравнение все же выявило существенные различия. Изменения " +"в том, как компилятор организует инструкции, также могут создать " +"впечатление, что кандидат ближе к цели, не решая при этом основную проблему." +" Такая диагностика может помочь в следующем эксперименте, но я бы сопоставил" +" любую награду за «приближение» с полной приемочной проверкой." + +#: website/public/blog/reverse-engineering-models/index.html:727 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would be interested in the pattern across these results. If a specialist " +"does well within a familiar compiler family, does that advantage survive a " +"toolchain change? And once both models have good retrieval, how much does " +"training still add? The answers would help us decide where each kind of " +"model is most useful." +msgstr "" +"Меня бы заинтересовала закономерность этих результатов. Если " +"специализированная модель ИИ хорошо работает в знакомом семействе " +"компиляторов, сохранится ли это преимущество при изменении инструментальной " +"цепочки? И как только обе модели ИИ будут хорошо воспроизводиться, сколько " +"еще добавится обучение? Ответы помогут нам решить, где тот или иной тип " +"модели наиболее полезен." + +#: website/public/blog/reverse-engineering-models/index.html:736 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What kind of agent does this suggest?" +msgstr "О каком ИИ-агенте идет речь?" + +#: website/public/blog/reverse-engineering-models/index.html:737 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"If the models turn out to help at different stages, a useful arrangement " +"might be a broad agent with access to a specialist. Once a task becomes a " +"familiar compiler search, it could ask the specialist for candidates. If the" +" results stop making sense, the broader agent could return to the target " +"evidence or investigate the measurement itself." +msgstr "" +"Если модели окажутся полезными на разных этапах, полезным механизмом может " +"стать широкий агент с доступом к специализированной модели ИИ. Как только " +"задача становится привычным поиском компилятора, он может запросить " +"кандидатов у специализированной модели ИИ. Если результаты перестанут иметь " +"смысл, более широкий агент может вернуться к целевым данным или исследовать " +"само измерение." + +#: website/public/blog/reverse-engineering-models/index.html:745 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In that arrangement, specialization becomes a tool the agent can choose as " +"the investigation develops. The agent keeps the freedom to change direction " +"that made the earlier breakthroughs possible." +msgstr "" +"В таком случае специализация становится инструментом, который агент может " +"выбирать по мере развития расследования. Агент сохраняет свободу менять " +"направление, которое сделало возможными предыдущие прорывы." + +#: website/public/blog/reverse-engineering-models/index.html:750 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This follows naturally from the industrial shift I described in the previous" +" essay. Being able to run many more experiments changes how we organize the " +"work. It also gives us a reason to ask which decisions need broad judgment " +"and which improve through repeated experience with a particular craft." +msgstr "" +"Это естественным образом следует из промышленного сдвига, который я описал в" +" предыдущем эссе. Возможность проводить больше экспериментов меняет то, как " +"мы организуем работу. Это также дает нам повод задаться вопросом, какие " +"решения требуют широкого суждения, а какие улучшаются за счет повторного " +"опыта работы с конкретным ремеслом." + +#: website/public/blog/reverse-engineering-models/index.html:757 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For now, the strongest general model has earned its place in my workflow by " +"opening problems I had left unresolved. Those successes are what make me " +"want to examine the work more closely: how much depended on the model's " +"breadth, and how much could be handled by a model trained in this particular" +" kind of reconstruction?" +msgstr "" +"Пока самая сильная универсальная модель заслужила место в моём рабочем " +"процессе, решив проблемы, которые прежде оставались нерешёнными. Эти успехи " +"побуждают меня изучить процесс внимательнее: какая часть зависела от широты " +"её возможностей, а какую могла бы выполнить модель, обученная именно такой " +"реконструкции?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:765 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}The repositories give us a way to test the question.{ph1} I would like " +"to use that record to build the comparison and see what we learn." +msgstr "" +"{ph0}Репозитории дают нам возможность проверить вопрос.{ph1} Я хотел бы " +"использовать эту запись для сравнения и посмотреть, что мы узнаем." + +#: website/public/blog/reverse-engineering-models/index.html:775 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Evidence and model attribution" +msgstr "Доказательства и атрибуция моделей" + +#: website/public/blog/reverse-engineering-models/index.html:783 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The examples describe historical function-level comparisons recorded in the " +"linked commits. Whole-game runtime checks are separate milestones. Source " +"and matching records were reviewed on 13 October 2026." +msgstr "" +"В примерах описаны исторические сравнения на уровне функций, записанные в " +"связанных коммитах. Проверки во время выполнения всей игры — это отдельные " +"этапы. Источник и соответствующие записи были проверены 13 октября 2026 " +"года." + +#. {ph0}: gpt-dots +#. {ph1}: gpt-web +#: website/public/blog/reverse-engineering-models/index.html:789 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The TH09 label {ph0} identifies GPT-6-Astra. {ph1} changed with the model " +"used at the time, so that label alone does not identify an individual run. " +"Explicit model names in the other cited commits establish their recorded " +"attribution; the histories do not contain uniform per-run cost or effort " +"measurements." +msgstr "" +"Метка TH09 {ph0} означает GPT-6-Astra. Модель, связанная с {ph1}, со " +"временем изменилась, поэтому сама по себе метка не может идентифицировать " +"модель, использованную в конкретном прогоне. Явные названия моделей в других" +" цитируемых коммитах обеспечивают записанную атрибуцию. Эти истории не " +"включают в себя последовательные измерения затрат или усилий для каждого " +"запуска." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#: website/public/blog/reverse-engineering-models/index.html:798 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 temporary lifetimes:{ph1} the {ph2}earlier context review{ph3} and" +" {ph4}complete cold reconstruction{ph5}." +msgstr "" +"{ph0}TH09 время жизни временных объектов:{ph1} {ph2}предыдущая проверка " +"контекста{ph3} и {ph4}полная реконструкция, подтвержденная чистой " +"перекомпиляцией из исходного кода{ph5}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:810 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 Front callback:{ph1} {ph2}non-exact diagnostic regions{ph3}, " +"{ph4}Astra's new graph{ph5} and its {ph6}accepted recompile replay{ph7}." +msgstr "" +"Обратный вызов {ph0}TH09 Front:{ph1} {ph2}диагностические области, которые " +"не соответствуют{ph3}, {ph4}Astra новому графу потока управления{ph5} и его " +"{ph6}принятой проверке перекомпиляции{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:826 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Compiler knowledge:{ph1} {ph2}TH095 identifier-bucket reuse{ph3}, " +"{ph4}TH04 player source and linker order{ph5} and {ph6}Luna's TH09 enemy-" +"spawn closure{ph7}." +msgstr "" +"{ph0}Знание компилятора:{ph1} {ph2}TH095 повторное использование поведения " +"сегмента идентификаторов{ph3}, {ph4}TH04 исходный код игрока и порядок " +"компоновщика{ph5} и полная реконструкция {ph6}Luna функции создания врагов " +"TH09{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:842 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Checks and diagnostics:{ph1} {ph2}TH08 floating-literal audit{ph3}, " +"{ph4}TH08 numerical investigation{ph5} and {ph6}TH09 replay-save " +"reconstruction{ph7}." +msgstr "" +"{ph0}Проверки и диагностика:{ph1} {ph2}TH08 аудит констант с плавающей " +"запятой{ph3}, {ph4}TH08 численное исследование{ph5} и {ph6}TH09 " +"реконструкция с сохранением повторов{ph7}." + +#: website/public/blog/reverse-engineering-models/index.html:859 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The proposed specialist and RL comparisons are future experiments. The " +"project histories motivate them; no trained specialist was evaluated for " +"this essay." +msgstr "" +"Предлагаемые сравнения специализированных моделей и обучение RL — это " +"будущие эксперименты. Их мотивирует история проектов. Для этого эссе не " +"оценивалась обученная специализированная модель." + +#: website/public/blog/reverse-engineering-models/index.html:6 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model? | REA" +msgstr "Нужна ли реверс-инжинирингу модель большего размера? | REA" + +#: website/public/blog/reverse-engineering-models/index.html:7 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What Touhou reconstruction taught me about general models, specialist models" +" and learning to propose better hypotheses from compiler feedback." +msgstr "" +"Чему реконструкция Touhou научила меня об общих моделях, специализированных " +"моделях и умении предлагать лучшие гипотезы на основе отзывов компилятора." + +#: website/public/blog/reverse-engineering-models/index.html:240 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "On this page" +msgstr "На этой странице" + +#: website/public/blog/reverse-engineering-models/index.html:253 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Models and reverse engineering essay" +msgstr "Эссе «Модели и реверс-инжиниринг»" + +#: website/public/blog/reverse-engineering-models/index.html:491 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Selected reconstruction experiments" +msgstr "Избранные эксперименты по реконструкции" + +#: website/public/blog/reverse-engineering-models/index.html:883 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Back to top" +msgstr "Наверх" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:776 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Detailed evidence notes{ph1} include the recorded comparisons and " +"experiment context." +msgstr "" +"{ph0}Подробные записи о доказательствах{ph1} содержат результаты сравнений и" +" контекст экспериментов." + +#~ msgctxt "page:blog/index.html" +#~ msgid "By N0zoM1z0 · 13 October 2026 · Draft for review" +#~ msgstr "Автор: N0zoM1z0 · 13 октября 2026 г. · Черновик для обсуждения" + #~ msgctxt "page:index.html" #~ msgid "Setup guide" #~ msgstr "Руководство по установке" diff --git a/website/locales/th/messages.po b/website/locales/th/messages.po index 2fab386ea..45c67209b 100644 --- a/website/locales/th/messages.po +++ b/website/locales/th/messages.po @@ -32,18 +32,18 @@ msgid "Thoughts on reverse engineering." msgstr "ความคิดเกี่ยวกับวิศวกรรมย้อนกลับ" #. {ph0}:
-#: website/public/blog/index.html:200 +#: website/public/blog/index.html:214 #, python-brace-format msgctxt "page:blog/index.html" msgid "Game reconstruction{ph0}Essay" msgstr "ฟื้นฟูเกม{ph0}เรียงความ" -#: website/public/blog/index.html:202 +#: website/public/blog/index.html:216 msgctxt "page:blog/index.html" msgid "The industrial revolution in reverse engineering" msgstr "การปฏิวัติอุตสาหกรรมในวิศวกรรมย้อนกลับ" -#: website/public/blog/index.html:203 +#: website/public/blog/index.html:217 msgctxt "page:blog/index.html" msgid "" "What manual RE and reconstructing TH08, TH095 and TH04 taught me about " @@ -52,25 +52,26 @@ msgstr "" "สิ่งที่คู่มือใหม่และ " "reconstructingTH08,TH095และTH04สอนฉันเกี่ยวกับเจ้าหน้าที่ปลายทางและความรู้นั่นรประกอบที่อีกฝั่งยิ่งสนง.โครงการ." -#: website/public/blog/index.html:208 +#: website/public/blog/index.html:207 website/public/blog/index.html:222 msgctxt "page:blog/index.html" msgid "Read the article" msgstr "อ่านบทความ" -#: website/public/blog/index.html:211 +#: website/public/blog/index.html:225 msgctxt "page:blog/index.html" msgid "By N0zoM1z0 · 10 October 2026" msgstr "โดย N0zoM1z0 ·10 ตุลาคม 2026" #. {ph0}: -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 +#: website/public/blog/reverse-engineering-models/index.html:883 #: website/public/blog/touhou-reconstruction/index.html:1080 #: website/public/examples/dino-lab/index.html:475 #: website/public/faq/index.html:573 #: website/public/first-investigation/index.html:523 #: website/public/get-started/index.html:882 -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 #: website/public/guides/index.html:282 #: website/public/guides/javascript/index.html:503 #: website/public/guides/native/index.html:509 website/public/index.html:1191 @@ -100,6 +101,8 @@ msgstr "" "และเจ้าหน้าที่ปัญญาประดิษฐ์" #: website/public/blog/index.html:75 website/public/blog/index.html:88 +#: website/public/blog/reverse-engineering-models/index.html:75 +#: website/public/blog/reverse-engineering-models/index.html:88 #: website/public/blog/touhou-reconstruction/index.html:75 #: website/public/blog/touhou-reconstruction/index.html:88 #: website/public/examples/dino-lab/index.html:75 @@ -135,6 +138,7 @@ msgstr "" "REA-หาวิธีการทำงานของซอฟต์แวร์ วิศวกรรมย้อนกลับกับตัวแทนการเขียนโค้ดของคุณ" #: website/public/blog/index.html:108 +#: website/public/blog/reverse-engineering-models/index.html:108 #: website/public/blog/touhou-reconstruction/index.html:108 #: website/public/examples/dino-lab/index.html:109 #: website/public/faq/index.html:108 @@ -155,6 +159,7 @@ msgid "REA home" msgstr "หน้าแรก REA" #: website/public/blog/index.html:112 +#: website/public/blog/reverse-engineering-models/index.html:112 #: website/public/blog/touhou-reconstruction/index.html:112 #: website/public/examples/dino-lab/index.html:113 #: website/public/faq/index.html:112 @@ -175,6 +180,7 @@ msgid "Main navigation" msgstr "เมนูหลัก" #: website/public/blog/index.html:130 +#: website/public/blog/reverse-engineering-models/index.html:130 #: website/public/blog/touhou-reconstruction/index.html:130 #: website/public/examples/dino-lab/index.html:131 #: website/public/faq/index.html:130 @@ -195,6 +201,7 @@ msgid "REA on GitHub" msgstr "REA บน GitHub" #: website/public/blog/index.html:150 +#: website/public/blog/reverse-engineering-models/index.html:150 #: website/public/blog/touhou-reconstruction/index.html:150 #: website/public/examples/dino-lab/index.html:151 #: website/public/faq/index.html:150 @@ -215,6 +222,7 @@ msgid "Join REA on Discord" msgstr "เข้าร่วม REA บน Discord" #: website/public/blog/index.html:169 +#: website/public/blog/reverse-engineering-models/index.html:169 #: website/public/blog/touhou-reconstruction/index.html:169 #: website/public/examples/dino-lab/index.html:170 #: website/public/faq/index.html:169 @@ -234,7 +242,7 @@ msgctxt "shared" msgid "Follow REA on X" msgstr "ติดตาม REA บน X" -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 msgctxt "page:blog/index.html" msgid "Back to top" msgstr "กลับขึ้นด้านบน" @@ -4674,7 +4682,7 @@ msgctxt "page:guides/browser/index.html" msgid "Copy read the loaded scripts" msgstr "คัดลอกอ่านสคริปต์ที่โหลด" -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 msgctxt "page:guides/browser/index.html" msgid "Back to top" msgstr "กลับขึ้นด้านบน" @@ -11376,6 +11384,7 @@ msgid "Same spacing: 256 ÷ 16 = 16 angle units = 22.5°" msgstr "ระยะห่างเดียวกัน:256÷16=16 หน่วยมุม=22.5°" #: website/public/blog/index.html:114 +#: website/public/blog/reverse-engineering-models/index.html:114 #: website/public/blog/touhou-reconstruction/index.html:114 #: website/public/examples/dino-lab/index.html:115 #: website/public/faq/index.html:114 @@ -11396,6 +11405,7 @@ msgid "Showcases" msgstr "ตัวอย่าง" #: website/public/blog/index.html:115 +#: website/public/blog/reverse-engineering-models/index.html:115 #: website/public/blog/touhou-reconstruction/index.html:115 #: website/public/examples/dino-lab/index.html:116 #: website/public/faq/index.html:115 @@ -11416,6 +11426,7 @@ msgid "Blog" msgstr "บล็อก" #: website/public/blog/index.html:113 +#: website/public/blog/reverse-engineering-models/index.html:113 #: website/public/blog/touhou-reconstruction/index.html:113 #: website/public/examples/dino-lab/index.html:114 #: website/public/faq/index.html:113 @@ -11436,6 +11447,7 @@ msgid "Get started" msgstr "เริ่มต้น" #: website/public/blog/index.html:116 +#: website/public/blog/reverse-engineering-models/index.html:116 #: website/public/blog/touhou-reconstruction/index.html:116 #: website/public/examples/dino-lab/index.html:117 #: website/public/faq/index.html:116 @@ -11455,13 +11467,14 @@ msgctxt "shared" msgid "Guides" msgstr "คู่มือ" -#: website/public/blog/index.html:222 +#: website/public/blog/index.html:236 +#: website/public/blog/reverse-engineering-models/index.html:875 #: website/public/blog/touhou-reconstruction/index.html:1072 #: website/public/examples/dino-lab/index.html:467 #: website/public/faq/index.html:565 #: website/public/first-investigation/index.html:515 #: website/public/get-started/index.html:874 -#: website/public/guides/browser/index.html:490 +#: website/public/guides/browser/index.html:499 #: website/public/guides/index.html:274 #: website/public/guides/javascript/index.html:495 #: website/public/guides/native/index.html:501 website/public/index.html:1183 @@ -11475,13 +11488,14 @@ msgctxt "shared" msgid "Source" msgstr "ซอร์สโค้ด" -#: website/public/blog/index.html:223 +#: website/public/blog/index.html:237 +#: website/public/blog/reverse-engineering-models/index.html:876 #: website/public/blog/touhou-reconstruction/index.html:1073 #: website/public/examples/dino-lab/index.html:468 #: website/public/faq/index.html:566 #: website/public/first-investigation/index.html:516 #: website/public/get-started/index.html:875 -#: website/public/guides/browser/index.html:491 +#: website/public/guides/browser/index.html:500 #: website/public/guides/index.html:275 #: website/public/guides/javascript/index.html:496 #: website/public/guides/native/index.html:502 website/public/index.html:1184 @@ -11495,13 +11509,14 @@ msgctxt "shared" msgid "MIT license" msgstr "สัญญาอนุญาต MIT" -#: website/public/blog/index.html:224 +#: website/public/blog/index.html:238 +#: website/public/blog/reverse-engineering-models/index.html:877 #: website/public/blog/touhou-reconstruction/index.html:1074 #: website/public/examples/dino-lab/index.html:469 #: website/public/faq/index.html:567 #: website/public/first-investigation/index.html:517 #: website/public/get-started/index.html:876 -#: website/public/guides/browser/index.html:492 +#: website/public/guides/browser/index.html:501 #: website/public/guides/index.html:276 #: website/public/guides/javascript/index.html:497 #: website/public/guides/native/index.html:503 website/public/index.html:1185 @@ -11515,13 +11530,14 @@ msgctxt "shared" msgid "Join the community" msgstr "เข้าร่วมชุมชน" -#: website/public/blog/index.html:225 +#: website/public/blog/index.html:239 +#: website/public/blog/reverse-engineering-models/index.html:878 #: website/public/blog/touhou-reconstruction/index.html:1075 #: website/public/examples/dino-lab/index.html:470 #: website/public/faq/index.html:568 #: website/public/first-investigation/index.html:518 #: website/public/get-started/index.html:877 -#: website/public/guides/browser/index.html:493 +#: website/public/guides/browser/index.html:502 #: website/public/guides/index.html:277 #: website/public/guides/javascript/index.html:498 #: website/public/guides/native/index.html:504 website/public/index.html:1186 @@ -12459,6 +12475,1374 @@ msgctxt "runtime" msgid "Switch to dark mode" msgstr "เปลี่ยนเป็นโหมดมืด" +#. {ph0}: +#. {ph1}: +#: website/public/guides/browser/index.html:480 +#, python-brace-format +msgctxt "page:guides/browser/index.html" +msgid "" +"For HTTP request and response bodies, use {ph0}Fiddler Everywhere: SAZ " +"captures, MCP and public skills{ph1}. Export SAZ or HAR 1.2 for offline REA " +"capture Evidence, or connect Everywhere's MCP server to inspect captured " +"traffic with your agent." +msgstr "" +"หากต้องการตรวจสอบเนื้อหาคำขอและคำตอบ HTTP ให้ใช้ {ph0}Fiddler Everywhere: " +"ข้อมูลที่บันทึกในรูปแบบ SAZ, MCP และทักษะสาธารณะ{ph1} ส่งออกเป็น SAZ ???? " +"HAR 1.2 เพื่อรับ Evidence จากข้อมูลที่บันทึกใน REA แบบออฟไลน์ " +"หรือเชื่อมต่อเซิร์ฟเวอร์ MCP ของ Everywhere " +"เพื่อตรวจสอบทราฟฟิกที่บันทึกไว้ร่วมกับเอเจนต์ของคุณ" + +#. {ph0}:
+#: website/public/blog/index.html:200 +#, python-brace-format +msgctxt "page:blog/index.html" +msgid "Model specialization{ph0}Essay" +msgstr "โมเดลเฉพาะทาง{ph0}บทความ" + +#: website/public/blog/index.html:202 +msgctxt "page:blog/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "การวิศวกรรมย้อนกลับจำเป็นต้องใช้โมเดลที่ใหญ่ขึ้นไหม?" + +#: website/public/blog/index.html:203 +msgctxt "page:blog/index.html" +msgid "" +"What Touhou reconstruction suggests about general models, specialist models " +"and learning from compiler feedback." +msgstr "" +"ข้อคิดจากการสร้าง Touhou ขึ้นใหม่ เกี่ยวกับโมเดลทั่วไป โมเดลเฉพาะทาง " +"และการเรียนรู้จากผลตอบกลับของคอมไพเลอร์" + +#: website/public/blog/index.html:210 +msgctxt "page:blog/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "โดย N0zoM1z0 · 13 ตุลาคม 2026" + +#: website/public/blog/reverse-engineering-models/index.html:103 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Skip to content" +msgstr "ข้ามไปที่เนื้อหา" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:194 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Home{ph1} / {ph2}Blog{ph3} / Models and RE" +msgstr "{ph0}Home{ph1} / {ph2}Blog{ph3} / โมเดลและวิศวกรรมย้อนกลับ" + +#: website/public/blog/reverse-engineering-models/index.html:197 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Essay" +msgstr "เรียงความ" + +# Reviewed for technical meaning and readability; preserve the distinction +# between historical evidence and proposed experiments. +#: website/public/blog/reverse-engineering-models/index.html:198 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "วิศวกรรมย้อนกลับจำเป็นต้องมีโมเดลที่ใหญ่กว่านี้หรือไม่?" + +#: website/public/blog/reverse-engineering-models/index.html:199 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "โดย N0zoM1z0 · 13 ตุลาคม 2026" + +#: website/public/blog/reverse-engineering-models/index.html:201 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Takeaway" +msgstr "ประเด็นสำคัญ" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:203 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Model capability changes the hypotheses an agent can find.{ph1} Later " +"sessions have solved problems that earlier work left unresolved." +msgstr "" +"{ph0}ความสามารถของโมเดลส่งผลต่อสมมติฐานที่เอเจนต์ AI สามารถเสนอได้{ph1} " +"เซสชันต่อมาได้แก้ไขปัญหาที่งานก่อนหน้านี้ยังไม่ได้รับการแก้ไข" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:211 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A familiar compiler gives specialization something concrete to " +"learn.{ph1} Some exact matches depend on narrow, reusable source patterns." +msgstr "" +"{ph0}คอมไพเลอร์ที่คุ้นเคยนำเสนอรูปแบบที่เป็นรูปธรรมสำหรับโมเดลเฉพาะทางในการเรียนรู้{ph1}" +" " +"การจับคู่โค้ดเครื่องที่แน่นอนบางรายการขึ้นอยู่กับรูปแบบซอร์สโค้ดเฉพาะที่สามารถนำมาใช้ซ้ำได้" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:218 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Give repository knowledge a fair comparison.{ph1} A smaller model with " +"good retrieval may already know enough to make progress." +msgstr "" +"{ph0}รวมความรู้ของพื้นที่เก็บข้อมูลไว้ในการเปรียบเทียบที่ยุติธรรม{ph1} " +"โมเดลขนาดเล็กที่มีการดึงความรู้ที่ดีอาจมีข้อมูลเพียงพอที่จะดำเนินการต่อไป" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:223 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Train the next useful experiment, and test its oracle.{ph1} The value " +"of feedback depends on what the checker actually verifies." +msgstr "" +"{ph0}ฝึกแบบจำลองเพื่อเลือกการทดลองที่มีประโยชน์ถัดไป " +"และทดสอบเครื่องมือตรวจสอบการยืนยัน{ph1} " +"ความคิดเห็นจะมีประโยชน์เฉพาะในขอบเขตที่ผู้ตรวจสอบตรวจสอบสิ่งที่ถูกต้องเท่านั้น" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:230 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Measure accepted reconstruction against time and cost.{ph1} That is the" +" result I want from the whole agent." +msgstr "" +"{ph0}ประเมินผลการฟื้นฟูที่ได้รับการยืนยันควบคู่ไปกับเวลาและต้นทุน{ph1} " +"นั่นคือสิ่งที่ฉันต้องการวัดสำหรับเวิร์กโฟลว์เอเจนต์ AI ที่สมบูรณ์" + +#: website/public/blog/reverse-engineering-models/index.html:241 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "In this article" +msgstr "ในบทความนี้" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:243 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What better models changed{ph1}" +msgstr "{ph0}โมเดลที่เก่งขึ้นเปลี่ยนอะไรบ้าง?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:244 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Why specialize?{ph1}" +msgstr "{ph0}ทำไมจึงมีความเชี่ยวชาญ?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:245 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Where experience lives{ph1}" +msgstr "{ph0}ประสบการณ์นี้ควรเก็บไว้ที่ไหน?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:246 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Learning the next experiment{ph1}" +msgstr "{ph0}เรียนรู้การทดลองที่มีประโยชน์ครั้งต่อไป{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:247 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}RE and reinforcement learning{ph1}" +msgstr "{ph0}วิศวกรรมย้อนกลับและการเสริมกำลัง{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:248 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}When the oracle is wrong{ph1}" +msgstr "{ph0}เมื่อตัวตรวจสอบความถูกต้องผิดพลาด{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:249 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}A comparison worth running{ph1}" +msgstr "{ph0}การเปรียบเทียบที่คุ้มค่า{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:250 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What kind of agent?{ph1}" +msgstr "{ph0}สิ่งนี้แนะนำเอเจนต์ AI ประเภทใด{ph1}" + +#: website/public/blog/reverse-engineering-models/index.html:258 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What better models changed" +msgstr "โมเดลที่เก่งขึ้นเปลี่ยนอะไรบ้าง?" + +#: website/public/blog/reverse-engineering-models/index.html:259 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Since August 2026, I have been exploring how much reverse engineering an " +"agent can carry out with access to a program and a way to check its work. " +"Most of my experience comes from reconstructing games, especially Touhou. As" +" I have changed models, the repositories have kept a record of what each " +"session managed to recover." +msgstr "" +"ตั้งแต่เดือนสิงหาคม 2026 " +"ฉันได้สำรวจว่าตัวแทนสามารถดำเนินการวิศวกรรมย้อนกลับได้มากเพียงใดโดยมีสิทธิ์เข้าถึงโปรแกรมและวิธีการตรวจสอบงาน" +" ประสบการณ์ส่วนใหญ่ของฉันมาจากการสร้างเกมขึ้นใหม่ โดยเฉพาะ Touhou " +"ขณะที่ฉันเปลี่ยนโมเดล " +"พื้นที่เก็บข้อมูลได้เก็บบันทึกว่าแต่ละเซสชันจัดการเพื่อกู้คืนอะไรบ้าง" + +#: website/public/blog/reverse-engineering-models/index.html:267 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of that work aims at exact reconstruction: we compile the maintained " +"source under a pinned historical toolchain and check whether it reproduces " +"the target function. When a later model succeeds where earlier work stalled," +" the source changes show what made that progress possible." +msgstr "" +"งานส่วนใหญ่มีเป้าหมายให้ซอร์สโค้ดที่สร้างขึ้นคอมไพล์แล้วตรงกับต้นฉบับ " +"เราใช้ชุดเครื่องมือจากยุคนั้นโดยกำหนดเวอร์ชันไว้แน่นอน " +"แล้วตรวจว่าผลลัพธ์ตรงกับฟังก์ชันเป้าหมายหรือไม่ " +"เมื่อโมเดลรุ่นใหม่แก้จุดที่งานก่อนหน้าติดอยู่ได้ " +"การเปลี่ยนแปลงซอร์สโค้ดช่วยให้เห็นว่าอะไรทำให้ก้าวหน้าได้" + +#: website/public/blog/reverse-engineering-models/index.html:274 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The temporary values had to stay alive together" +msgstr "ตัวแปรชั่วคราวจำเป็นต้องมีอายุการใช้งานที่ทับซ้อนกัน" + +#: website/public/blog/reverse-engineering-models/index.html:275 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of a TH09 attack function's logic had already been reconstructed, but " +"the compiler still stored temporary results differently from the original. " +"The remaining problem was how long those results needed to stay alive and " +"when their storage could be reused." +msgstr "" +"ลอจิกของฟังก์ชันการโจมตี TH09 ส่วนใหญ่ได้ถูกสร้างขึ้นใหม่แล้ว " +"แต่คอมไพเลอร์ยังคงเก็บผลลัพธ์ชั่วคราวแตกต่างไปจากต้นฉบับ " +"ปัญหาที่เหลืออยู่คือผลลัพธ์เหล่านั้นต้องใช้เวลานานเท่าใดจึงจะคงอยู่ได้ " +"และเมื่อใดจึงจะสามารถนำพื้นที่จัดเก็บกลับมาใช้ใหม่ได้" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:282 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A {ph0}GPT-6-Sol checkpoint{ph1} recorded this mismatch after testing " +"several ordinary source arrangements. When {ph2}GPT-6.1-Sol returned to the " +"problem{ph3}, it found a useful hypothesis about how long those temporary " +"values needed to stay alive." +msgstr "" +"บันทึกเซสชัน {ph0}GPT-6-Sol{ph1} " +"รายงานความไม่ตรงกันนี้หลังจากทดสอบการจัดเตรียมซอร์สโค้ดทั่วไปหลายรายการ " +"เมื่อ {ph2}GPT-6.1-Sol ตรวจสอบปัญหาอีกครั้ง{ph3} " +"ก็เสนอสมมติฐานที่เป็นประโยชน์เกี่ยวกับช่วงอายุของตัวแปรชั่วคราวเหล่านี้" + +#: website/public/blog/reverse-engineering-models/index.html:296 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The function produced temporary results and copied them into animation " +"positions. In C++, a temporary result stays alive until the expression using" +" it finishes. Grouping the real operations into one expression kept those " +"results alive together, so the compiler could no longer reuse the same " +"storage for all of them." +msgstr "" +"ฟังก์ชันนี้สร้างผลลัพธ์ชั่วคราวและคัดลอกไปยังตำแหน่งแอนิเมชัน ในภาษา C++ " +"ผลลัพธ์ชั่วคราวจะคงอยู่จนกว่านิพจน์ที่ใช้จะเสร็จสิ้น " +"การจัดกลุ่มการดำเนินการจริงให้เป็นนิพจน์เดียวทำให้ผลลัพธ์เหล่านั้นคงอยู่ร่วมกัน" +" " +"ดังนั้นคอมไพลเลอร์จึงไม่สามารถใช้พื้นที่เก็บข้อมูลเดียวกันซ้ำได้อีกต่อไปสำหรับการดำเนินการทั้งหมด" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:303 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"One result still shared storage with a value used later for collision " +"checks. Giving that value its own lifetime resolved the remaining overlap. " +"After several targeted experiments, {ph0}a clean rebuild confirmed that the " +"full function matched{ph1}." +msgstr "" +"ผลลัพธ์หนึ่งรายการยังคงแชร์พื้นที่เก็บข้อมูลด้วยค่าที่ใช้ในภายหลังสำหรับการตรวจสอบการชนกัน" +" การให้คุณค่านั้นตลอดอายุการใช้งานจะช่วยแก้ไขการทับซ้อนที่เหลือ " +"หลังจากการทดลองที่กำหนดเป้าหมายหลายครั้ง " +"{ph0}การคอมไพล์ใหม่ทั้งหมดจากแหล่งที่มายืนยันว่าฟังก์ชันทั้งหมดตรงกัน{ph1}" + +#: website/public/blog/reverse-engineering-models/index.html:312 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What interests me is how the discovery and the tests worked together. The " +"lifetime hypothesis gave the agent a new direction; compiler feedback helped" +" it refine that hypothesis into a source arrangement that matched." +msgstr "" +"ฉันสนใจว่าการค้นพบและการทดสอบสนับสนุนซึ่งกันและกันอย่างไร " +"สมมติฐานเกี่ยวกับอายุการใช้งานของวัตถุชั่วคราวทำให้เอเจนต์ AI มีทิศทางใหม่ " +"คำติชมของคอมไพเลอร์ช่วยปรับแต่งสมมติฐานนั้นให้กลายเป็นการจัดเรียงซอร์สโค้ดที่ตรงกัน" + +#: website/public/blog/reverse-engineering-models/index.html:318 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A different control-flow graph" +msgstr "กราฟโฟลว์ควบคุมที่แตกต่างกัน" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:319 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A second TH09 function shows why that new direction can matter. Here, the " +"obstacle was how several branches shared their final work. " +"{ph0}GPT-6.1-Sol's earlier work{ph1} got parts of the function to match, but" +" the branches still joined up differently from the original." +msgstr "" +"ฟังก์ชัน TH09 ตัวที่สองแสดงให้เห็นว่าเหตุใดทิศทางใหม่จึงมีความสำคัญ " +"อุปสรรคอยู่ที่ว่าหลายสาขาจะแบ่งปันผลงานชิ้นสุดท้ายของพวกเขาอย่างไร " +"งานก่อนหน้าของ {ph0}GPT-6.1-Sol{ph1} มีฟังก์ชันบางส่วนที่เข้ากัน " +"แต่กิ่งก้านยังคงเชื่อมกันแตกต่างไปจากต้นฉบับ" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:330 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}GPT-6-Astra changed the surrounding control-flow graph{ph1}. The main " +"paths now led to a shared final update and return. In that source structure," +" the compiler combined the branches in the same way as the original. The " +"complete function matched, and a {ph2}separate rebuild check confirmed " +"it{ph3}." +msgstr "" +"{ph0}GPT-6-Astra เปลี่ยนกราฟโฟลว์ควบคุมโดยรอบ{ph1} " +"เส้นทางหลักตอนนี้นำไปสู่การอัปเดตและการกลับมาครั้งสุดท้ายที่ใช้ร่วมกัน " +"ในโครงสร้างต้นทางนั้น คอมไพลเลอร์จะรวมสาขาต่างๆ ในลักษณะเดียวกับต้นฉบับ " +"ฟังก์ชันที่สมบูรณ์ตรงกัน และ " +"{ph2}การตรวจสอบการสร้างใหม่แยกกันยืนยันแล้ว{ph3}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:342 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Changing the representation opened a more productive search space.{ph1}" +" Both cases give me a reason to take general model capability seriously: an " +"agent can keep trying local variations for a long time before finding a " +"better way to express the surrounding work." +msgstr "" +"{ph0}การเปลี่ยนการแสดงทำให้พื้นที่การค้นหามีประสิทธิผลมากขึ้น{ph1} " +"ทั้งสองกรณีให้เหตุผลแก่ฉันในการให้ความสำคัญกับความสามารถของโมเดลทั่วไปอย่างจริงจัง:" +" " +"ตัวแทนสามารถลองใช้รูปแบบท้องถิ่นต่อไปเป็นเวลานานก่อนที่จะพบวิธีที่ดีกว่าในการแสดงงานโดยรอบ" + +#: website/public/blog/reverse-engineering-models/index.html:351 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"At the same time, the later sessions started with earlier source and " +"recorded failures. That accumulated work was part of their advantage. The " +"history lets us inspect the discoveries and their checks; a controlled " +"comparison would help separate the effect of the model from the experience " +"it inherited." +msgstr "" +"ในเวลาเดียวกัน " +"เซสชันต่อมาเริ่มต้นด้วยแหล่งที่มาก่อนหน้าและความล้มเหลวที่บันทึกไว้ " +"งานที่สะสมไว้นั้นเป็นส่วนหนึ่งของความได้เปรียบของพวกเขา " +"ประวัติศาสตร์ช่วยให้เราตรวจสอบการค้นพบและการตรวจสอบของพวกเขา " +"การเปรียบเทียบแบบควบคุมจะช่วยแยกผลกระทบของแบบจำลองออกจากประสบการณ์ที่สืบทอดมา" + +#: website/public/blog/reverse-engineering-models/index.html:358 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is where the question becomes interesting to me. Some of the progress " +"came from finding a new representation, while much of the remaining work " +"involved testing source forms against a familiar compiler. How much needs " +"broad reasoning ability, and how much could a model learn through repeated " +"experience with that compiler and code?" +msgstr "" +"นี่คือสิ่งที่ทำให้คำถามน่าสนใจสำหรับฉัน " +"ความคืบหน้าบางอย่างมาจากการค้นหาตัวแทนใหม่ " +"งานที่เหลือส่วนใหญ่เกี่ยวข้องกับการทดสอบการจัดเตรียมซอร์สโค้ดที่แตกต่างกันกับคอมไพเลอร์ที่คุ้นเคย" +" สิ่งนี้ต้องใช้ความสามารถในการให้เหตุผลแบบกว้างๆ มากน้อยเพียงใด? " +"โมเดลสามารถเรียนรู้ได้มากเพียงใดจากการทำงานซ้ำๆ กับคอมไพเลอร์และโค้ดนั้น" + +#: website/public/blog/reverse-engineering-models/index.html:368 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why might a specialist do well?" +msgstr "เหตุใดโมเดล AI เฉพาะทางจึงทำได้ดี" + +#: website/public/blog/reverse-engineering-models/index.html:369 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Touhou gives us a concrete reason to expect that experience to help. Much of" +" the game code comes from the same author, ZUN, so recurring habits are a " +"reasonable starting assumption. The repositories back that up with useful " +"examples: related loops and helper arrangements have supplied hypotheses " +"across games." +msgstr "" +"Touhou ให้เหตุผลที่ชัดเจนแก่เราในการคาดหวังว่าประสบการณ์นั้นจะช่วยได้ " +"รหัสเกมส่วนใหญ่มาจากผู้เขียนคนเดียวกัน ZUN ดังนั้นนิสัยที่เกิดขึ้นซ้ำๆ " +"จึงเป็นสมมติฐานเริ่มต้นที่สมเหตุสมผล " +"แหล่งเก็บข้อมูลสนับสนุนด้วยตัวอย่างที่เป็นประโยชน์: " +"ลูปที่เกี่ยวข้องและการจัดเตรียมตัวช่วยได้ให้สมมติฐานแก่เกมต่างๆ" + +#: website/public/blog/reverse-engineering-models/index.html:376 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those source patterns become more specific when we include the compiler. " +"Under a particular version and set of options, small changes in source can " +"produce repeatable differences in machine code. Some of those differences " +"are surprisingly obscure." +msgstr "" +"รูปแบบแหล่งที่มาเหล่านั้นมีความเฉพาะเจาะจงมากขึ้นเมื่อเรารวมคอมไพเลอร์ " +"ภายใต้เวอร์ชันและชุดตัวเลือกเฉพาะ การเปลี่ยนแปลงเล็กๆ น้อยๆ " +"ในแหล่งที่มาอาจทำให้เกิดความแตกต่างที่ทำซ้ำได้ในรหัสเครื่อง " +"ความแตกต่างบางประการนั้นคลุมเครืออย่างน่าประหลาดใจ" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: Supervisor::LoadDat +#. {ph4}: +#: website/public/blog/reverse-engineering-models/index.html:382 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In TH095, for example, {ph0}even the name used for a local variable could " +"change where VC7.1 stored it{ph1}. Moving the declaration alone did not " +"solve the layout, but an earlier background function had revealed a naming " +"pattern that affected the compiler's choice. That gave us a more promising " +"hypothesis for {ph2}the reconstruction of {ph3}{ph4}." +msgstr "" +"ตัวอย่างเช่น ใน TH095 {ph0}แม้แต่ชื่อของตัวแปรโลคัลก็อาจส่งผลต่อตำแหน่งที่ " +"VC7.1 เก็บไว้{ph1} การย้ายการประกาศไม่ได้ช่วยแก้ปัญหารูปแบบการจัดเก็บข้อมูล " +"อย่างไรก็ตาม " +"ฟังก์ชันพื้นหลังก่อนหน้านี้ได้เปิดเผยรูปแบบการตั้งชื่อที่ส่งผลต่อการเลือกของคอมไพเลอร์" +" นั่นเสนอสมมติฐานที่มีแนวโน้มมากขึ้นสำหรับ {ph2}การสร้าง {ph3}{ph4} ใหม่" + +#: website/public/blog/reverse-engineering-models/index.html:396 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reusing that pattern for the file-size variable produced the required layout" +" without adding storage. The full function then matched." +msgstr "" +"การใช้รูปแบบนั้นซ้ำสำหรับตัวแปรขนาดไฟล์ทำให้เกิดเค้าโครงที่ต้องการโดยไม่ต้องเพิ่มพื้นที่เก็บข้อมูล" +" ฟังก์ชั่นครบถ้วนจึงเข้าคู่กัน" + +#: website/public/blog/reverse-engineering-models/index.html:401 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is the sort of narrow detail a general model may not know when it " +"arrives. A specialist trained on this compiler could encounter many such " +"cases and learn which source variations are worth testing first." +msgstr "" +"นี่เป็นรายละเอียดเฉพาะเจาะจงมากที่โมเดล AI " +"เอนกประสงค์อาจไม่ทราบเมื่อพบปัญหาครั้งแรก " +"โมเดลเฉพาะสำหรับคอมไพลเลอร์นี้สามารถเห็นกรณีดังกล่าวได้มากมาย " +"และเรียนรู้ว่าซอร์สโค้ดรูปแบบใดที่ควรค่าแก่การทดสอบก่อน" + +#: website/public/blog/reverse-engineering-models/index.html:407 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The opportunity is especially clear in final matching problems where we " +"already understand the behavior. We are then asking which ordinary source " +"form this compiler turns into the observed instructions. Specialization " +"could help the model propose those forms more efficiently, while the target " +"still determines which hypotheses survive." +msgstr "" +"โอกาสมีความชัดเจนโดยเฉพาะอย่างยิ่งในปัญหาการจับคู่ขั้นสุดท้ายที่เราเข้าใจพฤติกรรมอยู่แล้ว" +" " +"จากนั้นเราจะถามว่าการจัดเรียงซอร์สโค้ดธรรมดาใดที่คอมไพเลอร์นี้เปลี่ยนเป็นคำสั่งที่สังเกตได้" +" " +"ความเชี่ยวชาญพิเศษสามารถช่วยแบบจำลองเสนอรูปแบบเหล่านั้นได้อย่างมีประสิทธิภาพมากขึ้น" +" ในขณะที่เป้าหมายยังคงกำหนดว่าสมมติฐานใดที่จะอยู่รอดได้" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:415 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The useful patterns would have to retain their context, though. TH08 uses an" +" earlier Visual C++ version than TH095 and TH09. TH04 uses a different DOS-" +"era compiler, and its {ph0}player reconstruction{ph1} needed different " +"hypotheses about source patterns and how the compiled code was combined into" +" an executable. Shared authorship gives us a prior, but each target tells us" +" whether it applies. Distinguishing those contexts is part of what a " +"specialist would need to learn." +msgstr "" +"รูปแบบที่ใช้ได้ต้องมีบริบทกำกับ TH08 ใช้ Visual C++ รุ่นเก่ากว่า TH095 และ " +"TH09 ส่วน TH04 ใช้คอมไพเลอร์อีกตัวหนึ่งจากยุค DOS " +"{ph0}การสร้างโค้ดผู้เล่นขึ้นใหม่{ph1} จึงต้องใช้สมมติฐานต่างออกไป " +"ทั้งเรื่องรูปแบบซอร์สโค้ดและการลิงก์เป็นไฟล์ปฏิบัติการ " +"ผู้เขียนคนเดียวกันให้ข้อสันนิษฐานเริ่มต้น " +"แต่ต้องตรวจในแต่ละเป้าหมายว่าใช้ได้หรือไม่ " +"โมเดลเฉพาะทางต้องเรียนรู้ความแตกต่างเหล่านี้" + +#: website/public/blog/reverse-engineering-models/index.html:431 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Where should that experience live?" +msgstr "ประสบการณ์นี้ควรเก็บไว้ที่ไหน?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:432 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before deciding to teach those patterns through model training, I would look" +" at the knowledge we already keep in the repository. In the {ph0}previous " +"essay{ph1}, I wrote about it becoming the project's working memory. That " +"memory includes compiler lessons, along with the source and failed " +"experiments that made them useful." +msgstr "" +"ก่อนที่จะตัดสินใจสอนรูปแบบเหล่านั้นผ่านการฝึกโมเดล " +"ฉันจะดูความรู้ที่เราเก็บไว้ในที่เก็บอยู่แล้ว ใน {ph0}เรียงความก่อนหน้า{ph1} " +"ฉันเขียนเกี่ยวกับการที่กลายเป็นความทรงจำในการทำงานของโครงการ " +"หน่วยความจำดังกล่าวประกอบด้วยบทเรียนเกี่ยวกับคอมไพเลอร์ " +"พร้อมด้วยแหล่งที่มาและการทดลองที่ล้มเหลวซึ่งทำให้มีประโยชน์" + +#: website/public/blog/reverse-engineering-models/index.html:441 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"By retrieving that record, an agent can start with a known result and " +"inspect how we reached it. It can also avoid alternatives that have already " +"failed in the same context. The starting point improves without changing the" +" model's weights." +msgstr "" +"ด้วยการเรียกบันทึกนั้น " +"ตัวแทนสามารถเริ่มต้นด้วยผลลัพธ์ที่ทราบและตรวจสอบว่าเราเข้าถึงได้อย่างไร " +"นอกจากนี้ยังสามารถหลีกเลี่ยงทางเลือกอื่นที่ล้มเหลวในบริบทเดียวกันได้ " +"จุดเริ่มต้นจะดีขึ้นโดยไม่ต้องเปลี่ยนพารามิเตอร์ที่ได้รับการฝึกของโมเดล" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:447 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"When reconstructing TH09's enemy-spawn function, a {ph0}GPT-6-Luna-max " +"session{ph1} drew a hypothesis from TH08's corresponding source family. That" +" earlier source suggested how to structure the loop and handle failures " +"within it. TH09's own evidence supplied the game-specific details, and clean" +" rebuilds confirmed the complete match." +msgstr "" +"ในขณะที่สร้างฟังก์ชันของ TH09 ขึ้นมาใหม่เพื่อสร้างศัตรู เซสชัน " +"{ph0}GPT-6-Luna-max{ph1} ได้ตั้งสมมุติฐานจากซอร์สโค้ด TH08 ที่เกี่ยวข้อง " +"โค้ดก่อนหน้านี้แนะนำวิธีจัดระเบียบลูปและจัดการกับความล้มเหลวภายใน หลักฐานของ" +" TH09 ให้รายละเอียดเฉพาะเกม " +"การสร้างใหม่จากแหล่งที่มายืนยันว่าฟังก์ชันทั้งหมดตรงกัน" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:458 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}\"The model did not know that compiler quirk\" and \"the model could " +"not use it\" are different problems.{ph1} Retrieval may solve the first, but" +" turning that fact into a useful candidate still takes work. And if the " +"candidate fails, the model must decide what the result means for its next " +"attempt. Knowing the relevant pattern is only the beginning of that process." +msgstr "" +"{ph0}\"โมเดลไม่ทราบว่านิสัยแปลกๆ ของคอมไพเลอร์\" และ " +"\"โมเดลไม่สามารถใช้งานได้\" เป็นปัญหาที่แตกต่างกัน {ph1} " +"การดึงข้อมูลอาจแก้ปัญหาได้ในวิธีแรก " +"แต่การเปลี่ยนข้อเท็จจริงดังกล่าวให้กลายเป็นตัวเลือกที่มีประโยชน์ยังคงต้องใช้เวลา" +" และหากซอร์สโค้ดของผู้สมัครล้มเหลว " +"โมเดลจะต้องตัดสินใจว่าผลลัพธ์มีความหมายอย่างไรสำหรับความพยายามครั้งถัดไป " +"การรู้รูปแบบที่เกี่ยวข้องเป็นเพียงจุดเริ่มต้นของกระบวนการนั้นเท่านั้น" + +#: website/public/blog/reverse-engineering-models/index.html:468 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For that reason, I would include a smaller general model with good retrieval" +" in any comparison. Giving it the same library of experience would help us " +"identify what training adds: better candidate generation, better decisions " +"after feedback, or both." +msgstr "" +"ด้วยเหตุผลดังกล่าว ฉันจึงรวมโมเดล AI " +"อเนกประสงค์ที่มีขนาดเล็กกว่าพร้อมการดึงข้อมูลที่ดีไว้ด้วยในการเปรียบเทียบ " +"การให้คลังประสบการณ์แบบเดียวกันจะช่วยให้เราระบุได้ว่าการฝึกอบรมใดที่เพิ่มเข้ามา" +" เช่น การสร้างผู้สมัครที่ดีขึ้น การตัดสินใจที่ดีขึ้นหลังจากได้รับคำติชม " +"หรือทั้งสองอย่าง" + +#: website/public/blog/reverse-engineering-models/index.html:476 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Learning the next useful experiment" +msgstr "เรียนรู้การทดลองที่มีประโยชน์ครั้งต่อไป" + +#: website/public/blog/reverse-engineering-models/index.html:477 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those decisions after feedback are what I would most like to study. In " +"agentic reconstruction, the model proposes an explanation and tests it " +"against the program. Even when the candidate fails, the result can narrow " +"the remaining possibilities and suggest a more useful next experiment." +msgstr "" +"การตัดสินใจเหล่านั้นหลังจากได้รับคำติชมคือสิ่งที่ฉันอยากศึกษามากที่สุด " +"ในการฟื้นฟูแบบเอเจนต์ แบบจำลองจะเสนอคำอธิบายและทดสอบกับโปรแกรม " +"แม้ว่าซอร์สโค้ดของผู้สมัครจะล้มเหลว " +"ผลลัพธ์ก็สามารถจำกัดความเป็นไปได้ที่เหลืออยู่ให้แคบลง " +"และแนะนำการทดลองถัดไปที่มีประโยชน์มากขึ้น" + +#: website/public/blog/reverse-engineering-models/index.html:484 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Consider the temporary-results example again. Its final source shows the " +"arrangement that worked, but the route to that source contains more " +"information. Each of the following experiments helped identify what still " +"needed to change." +msgstr "" +"ลองพิจารณาตัวอย่างผลลัพธ์ชั่วคราวอีกครั้ง " +"แหล่งที่มาสุดท้ายแสดงการจัดเตรียมที่ได้ผล " +"แต่เส้นทางไปยังแหล่งข้อมูลนั้นมีข้อมูลเพิ่มเติม " +"การทดลองแต่ละรายการต่อไปนี้ช่วยระบุสิ่งที่ยังจำเป็นต้องเปลี่ยนแปลง" + +#: website/public/blog/reverse-engineering-models/index.html:497 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Experiment" +msgstr "การทดลอง" + +#: website/public/blog/reverse-engineering-models/index.html:498 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What its result taught us" +msgstr "ผลลัพธ์ของมันสอนอะไรเรา" + +#: website/public/blog/reverse-engineering-models/index.html:503 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Try a different way to copy the results" +msgstr "ลองวิธีอื่นในการคัดลอกผลลัพธ์" + +#: website/public/blog/reverse-engineering-models/index.html:504 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The copy code changes, but the function still differs." +msgstr "รหัสการคัดลอกเปลี่ยนไป แต่ฟังก์ชันยังคงแตกต่างออกไป" + +#: website/public/blog/reverse-engineering-models/index.html:509 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Use the returned values directly" +msgstr "ใช้ค่าที่ส่งคืนโดยตรง" + +#: website/public/blog/reverse-engineering-models/index.html:510 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Copying follows the original pattern, but temporary storage still differs." +msgstr "" +"การคัดลอกเป็นไปตามรูปแบบดั้งเดิม แต่การจัดเก็บชั่วคราวยังคงแตกต่างออกไป" + +#: website/public/blog/reverse-engineering-models/index.html:516 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the temporary results alive together" +msgstr "เก็บตัวแปรชั่วคราวทั้งหมดให้ถูกต้องในเวลาเดียวกัน" + +#: website/public/blog/reverse-engineering-models/index.html:517 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"They get separate storage, but one still overlaps a value used later for " +"collision checks." +msgstr "" +"พวกเขาได้รับพื้นที่เก็บข้อมูลแยกต่างหาก " +"แต่ยังมีอีกหนึ่งค่าที่ทับซ้อนกันกับค่าที่ใช้ในภายหลังสำหรับการตรวจสอบการชนกัน" + +#: website/public/blog/reverse-engineering-models/index.html:523 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the collision-check value separate" +msgstr "แยกค่าการตรวจสอบการชนกันออกจากกัน" + +#: website/public/blog/reverse-engineering-models/index.html:524 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A clean rebuild confirms the complete match." +msgstr "การคอมไพล์ใหม่ทั้งหมดจากแหล่งที่มายืนยันการจับคู่ที่สมบูรณ์" + +#: website/public/blog/reverse-engineering-models/index.html:529 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Selected experiments from the attack-function reconstruction above, grouped " +"by what they taught us." +msgstr "" +"การทดลองที่เลือกจากการสร้างฟังก์ชันการโจมตีขึ้นใหม่ด้านบน " +"โดยจัดกลุ่มตามสิ่งที่พวกเขาสอนเรา" + +#: website/public/blog/reverse-engineering-models/index.html:533 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A training record that keeps this sequence can teach more than the final " +"source alone. It connects each proposed change to an observed result and the" +" next decision. Keeping ineffective variations matters too: they tell a " +"later agent which hypotheses have already been tested in that context." +msgstr "" +"บันทึกการฝึกอบรมที่เก็บลำดับนี้สามารถสอนได้มากกว่าแหล่งข้อมูลสุดท้ายเพียงอย่างเดียว" +" " +"โดยเชื่อมโยงการเปลี่ยนแปลงที่เสนอแต่ละรายการกับผลลัพธ์ที่สังเกตได้และการตัดสินใจครั้งต่อไป" +" การรักษารูปแบบที่ไม่มีประสิทธิภาพก็มีความสำคัญเช่นกัน " +"โดยจะแจ้งตัวแทนในภายหลังว่าสมมติฐานใดได้รับการทดสอบแล้วในบริบทนั้น" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:540 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}I would want the model to learn how to use feedback.{ph1} A neutral " +"result is a reason to reconsider the variation; a changed mismatch may point" +" to another cause. When local edits stop being informative, the agent should" +" be able to revisit the source representation or inspect the evidence again." +msgstr "" +"{ph0}ฉันอยากให้โมเดลเรียนรู้วิธีใช้คำติชม{ph1} " +"ผลลัพธ์ที่เป็นกลางคือเหตุผลที่ต้องพิจารณารูปแบบใหม่อีกครั้ง " +"ความไม่ตรงกันที่เปลี่ยนแปลงอาจชี้ไปที่สาเหตุอื่น " +"เมื่อการแก้ไขเฉพาะที่หยุดให้ข้อมูล " +"ตัวแทนควรสามารถทบทวนการนำเสนอแหล่งที่มาหรือตรวจสอบหลักฐานอีกครั้ง" + +#: website/public/blog/reverse-engineering-models/index.html:549 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Learning to make those choices requires autonomy. Sometimes the useful next " +"step is inspecting another caller rather than compiling another edit. A " +"rigid sequence of required rewrites would remove that choice, along with " +"part of the behavior we want the model to learn." +msgstr "" +"การเรียนรู้ที่จะเลือกขั้นตอนเหล่านี้ต้องมีอิสระในการทำงาน " +"บางครั้งขั้นตอนที่มีประโยชน์คือการตรวจฟังก์ชันอื่นที่เรียกฟังก์ชันนี้ " +"แทนที่จะแก้โค้ดแล้วคอมไพล์อีกครั้ง " +"ลำดับการเขียนโค้ดใหม่ที่บังคับตายตัวจะตัดทางเลือกนี้ออก " +"รวมถึงพฤติกรรมบางส่วนที่เราอยากให้โมเดลเรียนรู้" + +#: website/public/blog/reverse-engineering-models/index.html:558 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why RE is interesting for reinforcement learning" +msgstr "เหตุใดวิศวกรรมย้อนกลับจึงน่าสนใจสำหรับการเรียนรู้แบบเสริมกำลัง" + +#: website/public/blog/reverse-engineering-models/index.html:559 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Bounded reconstruction gives us a practical environment for studying these " +"choices. Once the target and compiler are pinned, many source experiments " +"produce quick, repeatable feedback. The agent can compile a candidate, " +"inspect the mismatch and use that observation to choose its next step." +msgstr "" +"การสร้างโค้ดขึ้นใหม่ในขอบเขตที่ชัดเจนเป็นสภาพแวดล้อมที่ใช้ศึกษาการเลือกเหล่านี้ได้จริง" +" เมื่อกำหนดเป้าหมายและเวอร์ชันคอมไพเลอร์ไว้แน่นอน " +"การทดลองซอร์สโค้ดหลายแบบจะให้ผลตอบกลับรวดเร็วและทำซ้ำได้ " +"เอเจนต์สามารถคอมไพล์โค้ดที่เสนอ ตรวจส่วนที่ยังต่าง แล้วเลือกขั้นตอนถัดไป" + +#: website/public/blog/reverse-engineering-models/index.html:566 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reinforcement learning could train that decision-making process through the " +"feedback it receives. A candidate accepted by a defined check gives us a " +"concrete outcome, while the sequence of attempts lets us observe how much " +"work it took to reach it." +msgstr "" +"การเรียนรู้แบบเสริมกำลังสามารถฝึกกระบวนการตัดสินใจผ่านคำติชมที่ได้รับ " +"ผู้สมัครที่ยอมรับโดยเช็คที่กำหนดจะให้ผลลัพธ์ที่เป็นรูปธรรมแก่เรา " +"ในขณะที่ลำดับของความพยายามช่วยให้เราสังเกตได้ว่าต้องใช้ความพยายามมากเพียงใดจึงจะบรรลุผลนั้น" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:572 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"There is already research supporting this direction. {ph0}LLM4Decompile{ph1}" +" studies models specialized for decompilation. {ph2}AutoDecompiler{ph3} " +"trains multi-turn refinement using compiler and execution feedback. Their " +"studies focus on functional recovery. For this work, I would use them as a " +"starting point for the related task of finding source that matches a " +"historical compiler's output." +msgstr "" +"มีงานวิจัยสนับสนุนแนวทางนี้แล้ว {ph0}LLM4Decompile{ph1} " +"ศึกษาโมเดลที่เชี่ยวชาญการดีคอมไพล์ ส่วน {ph2}AutoDecompiler{ph3} " +"ฝึกการปรับปรุงหลายรอบโดยใช้ผลจากคอมไพเลอร์และการรันโปรแกรม " +"งานเหล่านี้เน้นการกู้คืนพฤติกรรมของโปรแกรม " +"ผมจะใช้เป็นจุดเริ่มต้นสำหรับงานที่เกี่ยวข้อง " +"คือการหาซอร์สโค้ดที่ให้ผลลัพธ์ตรงกับคอมไพเลอร์จากยุคเดิม" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:584 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To reach such a well-defined task, though, we may first need to establish " +"what a function owns or devise a new runtime measurement. In TH08, for " +"example, a {ph0}collision difference led back to how the original program " +"handled floating-point calculations{ph1}. Before the agent could test a " +"useful correction, it had to work out what to measure." +msgstr "" +"เพื่อให้บรรลุภารกิจที่กำหนดไว้อย่างชัดเจน " +"เราอาจจำเป็นต้องกำหนดทรัพยากรที่ฟังก์ชันเป็นเจ้าของก่อนหรือออกแบบวิธีใหม่เพื่อสังเกตพฤติกรรมของโปรแกรมขณะทำงาน" +" ตัวอย่างเช่น ใน TH08 ความแตกต่าง {ph0}การชนกัน " +"ทำให้ย้อนกลับไปถึงวิธีที่โปรแกรมดั้งเดิมจัดการการคำนวณจุดลอยตัว{ph1} " +"ก่อนที่ตัวแทนจะทดสอบการแก้ไขที่เป็นประโยชน์ได้ ก็ต้องพิจารณาว่าจะวัดอะไรก่อน" + +#: website/public/blog/reverse-engineering-models/index.html:595 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This suggests another role for a broadly capable model: helping make the " +"problem checkable in the first place. Once that check exists, we can " +"investigate how far a learned search policy can get with it." +msgstr "" +"สิ่งนี้ชี้ให้เห็นอีกบทบาทหนึ่งของแบบจำลองที่มีความสามารถในวงกว้าง: " +"ช่วยให้สามารถตรวจสอบปัญหาได้ตั้งแต่แรก เมื่อมีการตรวจสอบแล้ว " +"เราจะสามารถตรวจสอบได้ว่านโยบายการค้นหาที่เรียนรู้จะสามารถนำมาใช้ได้ไกลเพียงใด" + +#: website/public/blog/reverse-engineering-models/index.html:603 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "When the oracle is wrong" +msgstr "เมื่อตัวตรวจสอบความถูกต้องผิดพลาด" + +#: website/public/blog/reverse-engineering-models/index.html:604 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Once we train against that check, its coverage becomes part of what the " +"model learns. I call the reference or checker an oracle. Even if it gives a " +"repeatable answer, we still need to understand what that answer covers." +msgstr "" +"เมื่อใช้ผลตรวจสำหรับการฝึก " +"ขอบเขตของการตรวจก็กลายเป็นส่วนหนึ่งของสภาพแวดล้อมการเรียนรู้ " +"ผมเรียกข้อมูลอ้างอิงหรือเครื่องมือตรวจนี้ว่า oracle แม้คำตอบจะทำซ้ำได้ " +"เราก็ต้องเข้าใจว่ามันตรวจสอบอะไรจริง ๆ" + +#: website/public/blog/reverse-engineering-models/index.html:610 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"TH08 showed how much can hide outside that coverage. Some reconstructed " +"functions passed their exact comparisons with wrong floating constants. The " +"comparator adjusted relocated addresses before checking instructions, but it" +" checked the values behind those addresses only when they had been " +"explicitly recorded for verification. Other literal references could pass " +"without their values being examined." +msgstr "" +"TH08 แสดงให้เห็นว่าปัญหาอาจซ่อนอยู่นอกขอบเขตการตรวจ " +"บางฟังก์ชันใช้ค่าคงที่แบบ floating-point ที่ผิด " +"แต่ยังผ่านการเปรียบเทียบแบบตรงกันทุกส่วน " +"ตัวเปรียบเทียบปรับแอดเดรสที่ถูกย้ายก่อนตรวจคำสั่ง " +"แต่ตรวจค่าที่แอดเดรสนั้นชี้ไปเฉพาะกรณีที่บันทึกไว้ให้ตรวจอย่างชัดเจน " +"การอ้างอิงค่าคงที่อื่นจึงอาจผ่านโดยไม่ได้ตรวจค่า" + +#. {ph0}: +#. {ph1}: +#. {ph2}: 1.0f +#. {ph3}: 32.0f +#: website/public/blog/reverse-engineering-models/index.html:619 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To close that gap, a {ph0}wider audit{ph1} found incorrect constants in " +"functions that had already been accepted. In one calculation, the " +"reconstructed source added {ph2} where the original added {ph3}." +msgstr "" +"เพื่อแก้ไขช่องว่างนี้ {ph0}การตรวจสอบที่กว้างขึ้น{ph1} " +"พบค่าคงที่ที่ไม่ถูกต้องในฟังก์ชันที่ผ่านการตรวจสอบแล้ว ในการคำนวณครั้งหนึ่ง " +"ซอร์สโค้ดที่สร้างขึ้นใหม่ได้เพิ่ม {ph2} ในขณะที่ซอร์สโค้ดดั้งเดิมเพิ่ม {ph3}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:629 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A learner rewarded by the old check could learn to produce the wrong " +"reconstruction.{ph1} We had not trained an RL model against this oracle; the" +" observed failure was in the check itself. But using that check as a reward " +"would have made the same blind spot part of the learning environment." +msgstr "" +"{ph0}โมเดลที่ได้รับรางวัลจากการตรวจสอบยืนยันแบบเก่าสามารถเรียนรู้ที่จะสร้างการสร้างใหม่ที่ไม่ถูกต้อง{ph1}" +" เราไม่ได้ฝึกโมเดล RL โดยใช้ตัวตรวจสอบนั้น " +"ความล้มเหลวที่เราสังเกตเห็นคือในการตรวจสอบนั่นเอง แต่ถ้าเราใช้เป็นรางวัล " +"จุดบอดเดียวกันก็จะกลายเป็นส่วนหนึ่งของสภาพแวดล้อมการเรียนรู้" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:639 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A functional test can have a similar blind spot. {ph0}Differential-fuzzing " +"research on LLM decompilers{ph1} has found behavior differences beyond the " +"supplied tests. Passing those tests therefore needs to be read in terms of " +"the cases they exercise, just as an instruction match needs to be read in " +"terms of the bytes and data it checks." +msgstr "" +"การทดสอบการทำงานอาจมีจุดบอดที่คล้ายกัน " +"{ph0}การวิจัยโดยใช้การแยกฟัซซีแบบดิฟเฟอเรนเชียลกับตัวถอดรหัส LLM{ph1} " +"พบความแตกต่างด้านพฤติกรรมนอกเหนือจากการทดสอบที่ให้มา " +"ดังนั้นเราจึงจำเป็นต้องเข้าใจการทดสอบที่ผ่านในแง่ของกรณีที่ผ่านการทดสอบ " +"ในทำนองเดียวกัน การจับคู่คำสั่งจะต้องเข้าใจในแง่ของไบต์และข้อมูลที่ตรวจสอบ" + +#: website/public/blog/reverse-engineering-models/index.html:649 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before using either kind of check as a reward, I would test it with " +"deliberate failures. I would also keep final acceptance outside the " +"candidate's editable source, so the model has to satisfy the check through " +"its reconstruction. Otherwise, something like a byte array copied from the " +"target could satisfy a naive score without recovering useful source." +msgstr "" +"ก่อนที่จะใช้การตรวจสอบยืนยันประเภทใดประเภทหนึ่งเป็นรางวัล " +"ฉันจะทดสอบด้วยผลลัพธ์ที่ไม่ถูกต้องโดยเจตนา " +"ฉันยังจะแยกการยอมรับขั้นสุดท้ายออกจากซอร์สโค้ดที่เอเจนต์ AI สามารถแก้ไขได้ " +"โมเดลจะต้องผ่านการตรวจสอบโดยการสร้างโปรแกรมใหม่อย่างถูกต้อง มิฉะนั้น " +"การคัดลอกอาร์เรย์ไบต์จากเป้าหมายอาจหลอกระบบการให้คะแนนที่ไร้เดียงสาโดยไม่ต้องกู้คืนซอร์สโค้ดที่มีประโยชน์" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:657 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Improving the learner and improving its oracle belong to the same " +"research problem.{ph1} As the learner gets better at optimizing the feedback" +" we give it, understanding that feedback becomes more important too." +msgstr "" +"{ph0}การปรับปรุงโมเดลและการปรับปรุงเครื่องมือตรวจเป็นส่วนของปัญหาวิจัยเดียวกัน{ph1}" +" ยิ่งโมเดลเก่งในการปรับตัวตามผลตอบกลับที่เราให้ " +"ก็ยิ่งสำคัญที่เราต้องเข้าใจผลตอบกลับนั้น" + +#: website/public/blog/reverse-engineering-models/index.html:667 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A comparison worth running" +msgstr "การเปรียบเทียบที่คุ้มค่า" + +#: website/public/blog/reverse-engineering-models/index.html:668 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"With those checks in place, I would start the comparison on a fixed VC7.1 " +"task family. First, I would establish baselines for a strong general model " +"and a smaller one. Adding repository knowledge to the smaller model would " +"show what retrieval changes. From there, supervised specialization and an " +"RL-trained version would let us examine what training adds." +msgstr "" +"เมื่อตรวจสอบแล้ว ฉันจะเริ่มการเปรียบเทียบในกลุ่มงาน VC7.1 ที่คงที่ ก่อนอื่น " +"ฉันจะสร้างพื้นฐานสำหรับโมเดล AI อเนกประสงค์ที่แข็งแกร่งและโมเดลที่เล็กกว่า " +"การเพิ่มความรู้เกี่ยวกับพื้นที่เก็บข้อมูลให้กับโมเดลขนาดเล็กจะแสดงให้เห็นว่าการเปลี่ยนแปลงในการดึงข้อมูลมีอะไรบ้าง" +" จากนั้น ความเชี่ยวชาญพิเศษภายใต้การดูแลและเวอร์ชันที่ได้รับการฝึกอบรม RL " +"จะให้เราตรวจสอบได้ว่าการฝึกอบรมใดบ้างที่เพิ่มเข้ามา" + +#: website/public/blog/reverse-engineering-models/index.html:676 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To make those comparisons informative, each model would receive the same " +"target evidence and tools. For the training-specific comparisons, I would " +"keep the base model and available knowledge fixed as well. Repeated trials " +"would help distinguish a reliable search strategy from a lucky candidate." +msgstr "" +"เพื่อให้การเปรียบเทียบเหล่านี้เป็นข้อมูล " +"แต่ละแบบจำลองจะได้รับหลักฐานและเครื่องมือเป้าหมายที่เหมือนกัน " +"สำหรับการเปรียบเทียบเฉพาะการฝึกอบรม " +"ฉันจะเก็บโมเดลพื้นฐานและความรู้ที่มีอยู่ไว้คงที่เช่นกัน การทดลองซ้ำๆ " +"จะช่วยแยกแยะกลยุทธ์การค้นหาที่เชื่อถือได้จากโซลูชันผู้สมัครที่ประสบความสำเร็จโดยบังเอิญ" + +#: website/public/blog/reverse-engineering-models/index.html:683 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would then separate two starting conditions: reconstructing a function " +"from its instructions, and closing the last few differences in maintained " +"source. A specialist might be excellent at the second task while a general " +"model does much of the work needed to reach it. Measuring them separately " +"would reveal that division of work." +msgstr "" +"จากนั้นฉันจะแยกเงื่อนไขเริ่มต้นสองเงื่อนไข: สร้างฟังก์ชันใหม่จากคำแนะนำ " +"และปิดความแตกต่างสองสามข้อสุดท้ายในแหล่งที่มาที่ได้รับการบำรุงรักษา โมเดล AI" +" เฉพาะทางอาจทำงานได้ดีในงานที่สอง ในขณะที่โมเดล AI " +"เอนกประสงค์ทำงานส่วนใหญ่ที่จำเป็นในการเข้าถึงโมเดลนั้น " +"การวัดแยกกันจะเผยให้เห็นการแบ่งส่วนงานนั้น" + +#: website/public/blog/reverse-engineering-models/index.html:691 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both tasks need genuinely held-out examples. Because related Touhou games " +"share engine code, holding out a game's name would still allow nearly " +"identical functions to appear in training. I would instead hold out source " +"families and keep their accepted answers out of both training and retrieval." +msgstr "" +"ทั้งสองงานต้องมีตัวอย่างทดสอบที่ไม่อยู่ในข้อมูลฝึกและแหล่งข้อมูลสำหรับค้นคืน" +" เกม Touhou ที่เกี่ยวข้องใช้โค้ดเอนจินร่วมกัน " +"การกันเกมหนึ่งออกจากข้อมูลฝึกจึงอาจยังเหลือฟังก์ชันที่เกือบเหมือนกันในชุดฝึก" +" ผมจะกันซอร์สโค้ดที่เกี่ยวข้องทั้งกลุ่มออก " +"และไม่ให้คำตอบที่ผ่านการตรวจแล้วอยู่ในข้อมูลฝึกหรือคลังสำหรับค้นคืน" + +#: website/public/blog/reverse-engineering-models/index.html:698 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"On those examples, the outcome would be complete, independently validated " +"matches. Giving each model the same candidate or compile budget would help " +"compare search quality. Measuring elapsed time and total cost, including " +"compilation, would tell us whether that quality produces an efficient " +"workflow. Training costs would need their own accounting too." +msgstr "" +"ผลที่วัดควรเป็นการตรงกันครบทั้งฟังก์ชันซึ่งผ่านการตรวจอย่างอิสระ " +"ให้แต่ละโมเดลมีงบจำนวนโค้ดที่เสนอหรือจำนวนครั้งที่คอมไพล์เท่ากันเพื่อเปรียบเทียบคุณภาพการค้นหา" +" แล้ววัดเวลารวมและต้นทุนรวมซึ่งรวมการคอมไพล์ด้วย " +"เพื่อดูว่าคุณภาพนั้นทำให้งานมีประสิทธิภาพจริงหรือไม่ " +"ต้นทุนการฝึกต้องนับแยกต่างหาก" + +#: website/public/blog/reverse-engineering-models/index.html:706 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both views matter because a cheaper proposal can become an expensive " +"workflow if it causes more uninformative compiles. Conversely, a slower " +"model may pay for itself by finding a useful experiment sooner. The " +"comparison should capture those costs." +msgstr "" +"การวัดทั้งสองมีความสำคัญ ข้อเสนอราคาถูกอาจนำไปสู่ขั้นตอนการทำงานที่มีราคาแพง" +" หากทำให้เกิดการคอมไพล์มากขึ้นซึ่งไม่ได้ให้ข้อมูลที่เป็นประโยชน์ " +"ในทางกลับกัน " +"โมเดลที่ช้ากว่าอาจประหยัดกว่าหากพบว่าการทดสอบมีประโยชน์เร็วกว่านี้ " +"การเปรียบเทียบควรรวมต้นทุนเหล่านี้ด้วย" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:712 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Counting accepted results also keeps intermediate progress in perspective. " +"One TH09 {ph0}replay-save candidate{ph1} looked promising because the " +"compiled function had the expected size, but a full comparison still found " +"substantial differences. Changes in how the compiler arranges instructions " +"can also make a candidate look closer without resolving the underlying " +"problem. Such diagnostics can guide the next experiment, but I would " +"validate any reward for \"getting closer\" against the full acceptance " +"check." +msgstr "" +"การนับผลลัพธ์ที่ยอมรับยังช่วยรักษาความก้าวหน้าระดับกลางในมุมมองด้วย TH09 " +"{ph0}บันทึกการเล่นซ้ำ-บันทึกผู้สมัคร{ph1} " +"หนึ่งตัวดูมีแนวโน้มดีเนื่องจากฟังก์ชันที่คอมไพล์มีขนาดตามที่คาดหวัง " +"แต่การเปรียบเทียบแบบเต็มยังคงพบความแตกต่างอย่างมาก " +"การเปลี่ยนแปลงวิธีที่คอมไพลเลอร์จัดเรียงคำสั่งยังสามารถทำให้ผู้สมัครมองเข้าไปใกล้มากขึ้นโดยไม่ต้องแก้ไขปัญหาที่ซ่อนอยู่" +" การวินิจฉัยดังกล่าวสามารถเป็นแนวทางในการทดสอบครั้งต่อไปได้ " +"แต่ฉันจะตรวจสอบรางวัลสำหรับการ \"เข้าใกล้\" " +"เทียบกับการตรวจสอบการยอมรับทั้งหมด" + +#: website/public/blog/reverse-engineering-models/index.html:727 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would be interested in the pattern across these results. If a specialist " +"does well within a familiar compiler family, does that advantage survive a " +"toolchain change? And once both models have good retrieval, how much does " +"training still add? The answers would help us decide where each kind of " +"model is most useful." +msgstr "" +"ฉันจะสนใจรูปแบบของผลลัพธ์เหล่านี้ หากโมเดล AI " +"เฉพาะทางทำงานได้ดีในตระกูลคอมไพเลอร์ที่คุ้นเคย " +"ข้อได้เปรียบนั้นจะรอดพ้นจากการเปลี่ยนแปลง toolchain หรือไม่ และเมื่อ AI " +"ทั้งสองโมเดลดึงข้อมูลได้ดีแล้ว การฝึกยังเพิ่มอีกเท่าไหร่? " +"คำตอบจะช่วยให้เราตัดสินใจได้ว่าโมเดลแต่ละประเภทมีประโยชน์มากที่สุดที่ใด" + +#: website/public/blog/reverse-engineering-models/index.html:736 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What kind of agent does this suggest?" +msgstr "สิ่งนี้แนะนำเอเจนต์ AI ประเภทใด" + +#: website/public/blog/reverse-engineering-models/index.html:737 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"If the models turn out to help at different stages, a useful arrangement " +"might be a broad agent with access to a specialist. Once a task becomes a " +"familiar compiler search, it could ask the specialist for candidates. If the" +" results stop making sense, the broader agent could return to the target " +"evidence or investigate the measurement itself." +msgstr "" +"หากโมเดลต่างๆ กลายเป็นความช่วยเหลือในขั้นตอนต่างๆ " +"การจัดการที่เป็นประโยชน์อาจเป็นตัวแทนในวงกว้างที่สามารถเข้าถึงโมเดล AI " +"เฉพาะทางได้ เมื่องานกลายมาเป็นการค้นหาคอมไพเลอร์ที่คุ้นเคย " +"งานนั้นก็สามารถถามโมเดล AI เฉพาะทางสำหรับผู้สมัครได้ หากผลลัพธ์ไม่สมเหตุสมผล" +" สารในวงกว้างสามารถกลับไปยังหลักฐานเป้าหมายหรือตรวจสอบการวัดเองได้" + +#: website/public/blog/reverse-engineering-models/index.html:745 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In that arrangement, specialization becomes a tool the agent can choose as " +"the investigation develops. The agent keeps the freedom to change direction " +"that made the earlier breakthroughs possible." +msgstr "" +"ในการจัดการดังกล่าว " +"ความเชี่ยวชาญเฉพาะทางจะกลายเป็นเครื่องมือที่ตัวแทนสามารถเลือกได้ในขณะที่การสืบสวนพัฒนาขึ้น" +" ตัวแทนรักษาอิสระในการเปลี่ยนทิศทางที่ทำให้ความก้าวหน้าก่อนหน้านี้เป็นไปได้" + +#: website/public/blog/reverse-engineering-models/index.html:750 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This follows naturally from the industrial shift I described in the previous" +" essay. Being able to run many more experiments changes how we organize the " +"work. It also gives us a reason to ask which decisions need broad judgment " +"and which improve through repeated experience with a particular craft." +msgstr "" +"สิ่งนี้เป็นไปตามธรรมชาติจากการเปลี่ยนแปลงทางอุตสาหกรรมที่ฉันอธิบายไว้ในบทความก่อนหน้านี้" +" ความสามารถในการทำการทดสอบอื่นๆ " +"อีกมากมายได้เปลี่ยนแปลงวิธีที่เราจัดระเบียบงาน " +"นอกจากนี้ยังให้เหตุผลแก่เราในการถามว่าการตัดสินใจใดต้องใช้วิจารณญาณกว้างๆ " +"และการตัดสินใจครั้งใดจะปรับปรุงผ่านประสบการณ์ซ้ำแล้วซ้ำเล่ากับงานฝีมือเฉพาะอย่าง" + +#: website/public/blog/reverse-engineering-models/index.html:757 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For now, the strongest general model has earned its place in my workflow by " +"opening problems I had left unresolved. Those successes are what make me " +"want to examine the work more closely: how much depended on the model's " +"breadth, and how much could be handled by a model trained in this particular" +" kind of reconstruction?" +msgstr "" +"ตอนนี้โมเดลทั่วไปที่เก่งที่สุดมีที่ยืนในวิธีทำงานของผม " +"เพราะแก้ปัญหาที่ก่อนหน้านี้ผมยังแก้ไม่ได้ " +"ความสำเร็จเหล่านี้ทำให้ผมอยากศึกษางานให้ละเอียดขึ้น " +"ว่ามากน้อยแค่ไหนต้องอาศัยความสามารถที่กว้างของโมเดล " +"และมากน้อยแค่ไหนอาจให้โมเดลที่ฝึกเฉพาะงานสร้างโค้ดแบบนี้ทำได้" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:765 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}The repositories give us a way to test the question.{ph1} I would like " +"to use that record to build the comparison and see what we learn." +msgstr "" +"{ph0}ที่เก็บข้อมูลช่วยให้เรามีวิธีทดสอบคำถาม{ph1} " +"ฉันต้องการใช้บันทึกนั้นเพื่อสร้างการเปรียบเทียบและดูสิ่งที่เราเรียนรู้" + +#: website/public/blog/reverse-engineering-models/index.html:775 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Evidence and model attribution" +msgstr "หลักฐานและการระบุแหล่งที่มาของแบบจำลอง" + +#: website/public/blog/reverse-engineering-models/index.html:783 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The examples describe historical function-level comparisons recorded in the " +"linked commits. Whole-game runtime checks are separate milestones. Source " +"and matching records were reviewed on 13 October 2026." +msgstr "" +"ตัวอย่างจะอธิบายการเปรียบเทียบระดับฟังก์ชันในอดีตที่บันทึกไว้ในการคอมมิตที่เชื่อมโยง" +" การตรวจสอบรันไทม์ทั้งเกมถือเป็นเหตุการณ์สำคัญที่แยกจากกัน " +"แหล่งที่มาและบันทึกการจับคู่ได้รับการตรวจสอบเมื่อวันที่ 13 ตุลาคม พ.ศ. 2569" + +#. {ph0}: gpt-dots +#. {ph1}: gpt-web +#: website/public/blog/reverse-engineering-models/index.html:789 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The TH09 label {ph0} identifies GPT-6-Astra. {ph1} changed with the model " +"used at the time, so that label alone does not identify an individual run. " +"Explicit model names in the other cited commits establish their recorded " +"attribution; the histories do not contain uniform per-run cost or effort " +"measurements." +msgstr "" +"ป้ายกำกับ TH09 {ph0} หมายถึง GPT-6-Astra โมเดลที่เกี่ยวข้องกับ {ph1} " +"มีการเปลี่ยนแปลงเมื่อเวลาผ่านไป " +"ดังนั้นป้ายกำกับเพียงอย่างเดียวจึงไม่สามารถระบุโมเดลที่ใช้ในการดำเนินการเฉพาะได้" +" ชื่อโมเดลที่ชัดเจนในคอมมิตที่อ้างถึงอื่นๆ ให้การระบุแหล่งที่มาที่บันทึกไว้ " +"ประวัติเหล่านี้ไม่ได้รวมการวัดต้นทุนหรือความพยายามอย่างสม่ำเสมอสำหรับการวิ่งทุกครั้ง" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#: website/public/blog/reverse-engineering-models/index.html:798 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 temporary lifetimes:{ph1} the {ph2}earlier context review{ph3} and" +" {ph4}complete cold reconstruction{ph5}." +msgstr "" +"{ph0}TH09 อายุการใช้งานของวัตถุชั่วคราว:{ph1} " +"{ph2}การตรวจสอบบริบทก่อนหน้า{ph3} และ " +"{ph4}การสร้างใหม่เสร็จสมบูรณ์ได้รับการตรวจสอบโดยการคอมไพล์ใหม่ทั้งหมดจากแหล่งที่มา{ph5}" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:810 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 Front callback:{ph1} {ph2}non-exact diagnostic regions{ph3}, " +"{ph4}Astra's new graph{ph5} and its {ph6}accepted recompile replay{ph7}." +msgstr "" +"{ph0}คอลแบ็ก Front ของ TH09:{ph1} {ph2}บริเวณที่ตรวจพบว่ายังไม่ตรง{ph3}, " +"{ph4}กราฟการควบคุมใหม่ของ Astra{ph5} และ " +"{ph6}การตรวจคอมไพล์ซ้ำที่ผ่านการยอมรับ{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:826 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Compiler knowledge:{ph1} {ph2}TH095 identifier-bucket reuse{ph3}, " +"{ph4}TH04 player source and linker order{ph5} and {ph6}Luna's TH09 enemy-" +"spawn closure{ph7}." +msgstr "" +"{ph0}ความรู้เกี่ยวกับคอมไพเลอร์:{ph1} {ph2}TH095 " +"นำพฤติกรรมตัวระบุ-บัคเก็ตกลับมาใช้ใหม่{ph3}, {ph4}TH04 " +"ซอร์สโค้ดของผู้เล่นและลำดับลิงก์เกอร์{ph5} และ {ph6}Luna " +"การสร้างศัตรูใหม่ของ TH09 ใหม่ทั้งหมด ฟังก์ชั่น{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:842 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Checks and diagnostics:{ph1} {ph2}TH08 floating-literal audit{ph3}, " +"{ph4}TH08 numerical investigation{ph5} and {ph6}TH09 replay-save " +"reconstruction{ph7}." +msgstr "" +"{ph0}การตรวจสอบและการวินิจฉัย:{ph1} {ph2}TH08 " +"การตรวจสอบคงที่แบบจุดลอยตัว{ph3}, {ph4}TH08 การตรวจสอบเชิงตัวเลข{ph5} และ " +"{ph6}TH09 การสร้างใหม่เพื่อประหยัดการเล่นซ้ำ{ph7}" + +#: website/public/blog/reverse-engineering-models/index.html:859 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The proposed specialist and RL comparisons are future experiments. The " +"project histories motivate them; no trained specialist was evaluated for " +"this essay." +msgstr "" +"การเปรียบเทียบที่เสนอของรุ่นพิเศษและการฝึกอบรม RL เป็นการทดลองในอนาคต " +"ประวัติโครงการเป็นแรงบันดาลใจให้พวกเขา " +"ไม่มีการประเมินแบบจำลองเฉพาะทางที่ได้รับการฝึกอบรมสำหรับบทความนี้" + +#: website/public/blog/reverse-engineering-models/index.html:6 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model? | REA" +msgstr "วิศวกรรมย้อนกลับจำเป็นต้องมีโมเดลที่ใหญ่กว่านี้หรือไม่? | REA" + +#: website/public/blog/reverse-engineering-models/index.html:7 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What Touhou reconstruction taught me about general models, specialist models" +" and learning to propose better hypotheses from compiler feedback." +msgstr "" +"สิ่งที่ Touhou การสร้างใหม่สอนฉันเกี่ยวกับโมเดลทั่วไป โมเดลผู้เชี่ยวชาญ " +"และการเรียนรู้ที่จะเสนอสมมติฐานที่ดีกว่าจากคำติชมของคอมไพเลอร์" + +#: website/public/blog/reverse-engineering-models/index.html:240 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "On this page" +msgstr "ในหน้านี้" + +#: website/public/blog/reverse-engineering-models/index.html:253 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Models and reverse engineering essay" +msgstr "แบบจำลองและเรียงความวิศวกรรมย้อนกลับ" + +#: website/public/blog/reverse-engineering-models/index.html:491 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Selected reconstruction experiments" +msgstr "การทดลองสร้างใหม่ที่เลือก" + +#: website/public/blog/reverse-engineering-models/index.html:883 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Back to top" +msgstr "กลับขึ้นด้านบน" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:776 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Detailed evidence notes{ph1} include the recorded comparisons and " +"experiment context." +msgstr "" +"{ph0}บันทึกหลักฐานโดยละเอียด{ph1} มีผลการเปรียบเทียบและบริบทของการทดลอง" + +#~ msgctxt "page:blog/index.html" +#~ msgid "By N0zoM1z0 · 13 October 2026 · Draft for review" +#~ msgstr "โดย N0zoM1z0 · 13 ตุลาคม 2026 · ฉบับร่างสำหรับตรวจทาน" + #~ msgctxt "page:index.html" #~ msgid "Setup guide" #~ msgstr "คู่มือการติดตั้ง" diff --git a/website/locales/tr/messages.po b/website/locales/tr/messages.po index 7de2f7e5a..9d71134ed 100644 --- a/website/locales/tr/messages.po +++ b/website/locales/tr/messages.po @@ -32,18 +32,18 @@ msgid "Thoughts on reverse engineering." msgstr "Tersine mühendislik üzerine düşünceler." #. {ph0}:
-#: website/public/blog/index.html:200 +#: website/public/blog/index.html:214 #, python-brace-format msgctxt "page:blog/index.html" msgid "Game reconstruction{ph0}Essay" msgstr "Oyun rekonstrüksiyonu{ph0}Deneme" -#: website/public/blog/index.html:202 +#: website/public/blog/index.html:216 msgctxt "page:blog/index.html" msgid "The industrial revolution in reverse engineering" msgstr "Tersine mühendislikte endüstri devrimi" -#: website/public/blog/index.html:203 +#: website/public/blog/index.html:217 msgctxt "page:blog/index.html" msgid "" "What manual RE and reconstructing TH08, TH095 and TH04 taught me about " @@ -53,25 +53,26 @@ msgstr "" "yapılandırılması bana projeler arasında birleşen aracılar, doğrulama ve " "bilgi hakkında bilgi verdi." -#: website/public/blog/index.html:208 +#: website/public/blog/index.html:207 website/public/blog/index.html:222 msgctxt "page:blog/index.html" msgid "Read the article" msgstr "Makaleyi okuyun" -#: website/public/blog/index.html:211 +#: website/public/blog/index.html:225 msgctxt "page:blog/index.html" msgid "By N0zoM1z0 · 10 October 2026" msgstr "Başlangıç N0zoM1z0 * 10 Ekim 2026" #. {ph0}: -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 +#: website/public/blog/reverse-engineering-models/index.html:883 #: website/public/blog/touhou-reconstruction/index.html:1080 #: website/public/examples/dino-lab/index.html:475 #: website/public/faq/index.html:573 #: website/public/first-investigation/index.html:523 #: website/public/get-started/index.html:882 -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 #: website/public/guides/index.html:282 #: website/public/guides/javascript/index.html:503 #: website/public/guides/native/index.html:509 website/public/index.html:1191 @@ -101,6 +102,8 @@ msgstr "" "deneyimlerden yararlanma." #: website/public/blog/index.html:75 website/public/blog/index.html:88 +#: website/public/blog/reverse-engineering-models/index.html:75 +#: website/public/blog/reverse-engineering-models/index.html:88 #: website/public/blog/touhou-reconstruction/index.html:75 #: website/public/blog/touhou-reconstruction/index.html:88 #: website/public/examples/dino-lab/index.html:75 @@ -137,6 +140,7 @@ msgstr "" "mühendislik yapın." #: website/public/blog/index.html:108 +#: website/public/blog/reverse-engineering-models/index.html:108 #: website/public/blog/touhou-reconstruction/index.html:108 #: website/public/examples/dino-lab/index.html:109 #: website/public/faq/index.html:108 @@ -157,6 +161,7 @@ msgid "REA home" msgstr "REA ana sayfası" #: website/public/blog/index.html:112 +#: website/public/blog/reverse-engineering-models/index.html:112 #: website/public/blog/touhou-reconstruction/index.html:112 #: website/public/examples/dino-lab/index.html:113 #: website/public/faq/index.html:112 @@ -177,6 +182,7 @@ msgid "Main navigation" msgstr "Ana menü" #: website/public/blog/index.html:130 +#: website/public/blog/reverse-engineering-models/index.html:130 #: website/public/blog/touhou-reconstruction/index.html:130 #: website/public/examples/dino-lab/index.html:131 #: website/public/faq/index.html:130 @@ -197,6 +203,7 @@ msgid "REA on GitHub" msgstr "GitHub’da REA" #: website/public/blog/index.html:150 +#: website/public/blog/reverse-engineering-models/index.html:150 #: website/public/blog/touhou-reconstruction/index.html:150 #: website/public/examples/dino-lab/index.html:151 #: website/public/faq/index.html:150 @@ -217,6 +224,7 @@ msgid "Join REA on Discord" msgstr "REA Discord topluluğuna katıl" #: website/public/blog/index.html:169 +#: website/public/blog/reverse-engineering-models/index.html:169 #: website/public/blog/touhou-reconstruction/index.html:169 #: website/public/examples/dino-lab/index.html:170 #: website/public/faq/index.html:169 @@ -236,7 +244,7 @@ msgctxt "shared" msgid "Follow REA on X" msgstr "REA’yı X’te takip et" -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 msgctxt "page:blog/index.html" msgid "Back to top" msgstr "Başa dön" @@ -4754,7 +4762,7 @@ msgctxt "page:guides/browser/index.html" msgid "Copy read the loaded scripts" msgstr "Kopyala yüklenen komut dosyalarını oku" -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 msgctxt "page:guides/browser/index.html" msgid "Back to top" msgstr "Başa dön" @@ -11605,6 +11613,7 @@ msgid "Same spacing: 256 ÷ 16 = 16 angle units = 22.5°" msgstr "Aynı aralık: 256 ÷ 16 = 16 açı birimi = 22,5 °" #: website/public/blog/index.html:114 +#: website/public/blog/reverse-engineering-models/index.html:114 #: website/public/blog/touhou-reconstruction/index.html:114 #: website/public/examples/dino-lab/index.html:115 #: website/public/faq/index.html:114 @@ -11625,6 +11634,7 @@ msgid "Showcases" msgstr "Örnekler" #: website/public/blog/index.html:115 +#: website/public/blog/reverse-engineering-models/index.html:115 #: website/public/blog/touhou-reconstruction/index.html:115 #: website/public/examples/dino-lab/index.html:116 #: website/public/faq/index.html:115 @@ -11645,6 +11655,7 @@ msgid "Blog" msgstr "Blog" #: website/public/blog/index.html:113 +#: website/public/blog/reverse-engineering-models/index.html:113 #: website/public/blog/touhou-reconstruction/index.html:113 #: website/public/examples/dino-lab/index.html:114 #: website/public/faq/index.html:113 @@ -11665,6 +11676,7 @@ msgid "Get started" msgstr "Başlangıç" #: website/public/blog/index.html:116 +#: website/public/blog/reverse-engineering-models/index.html:116 #: website/public/blog/touhou-reconstruction/index.html:116 #: website/public/examples/dino-lab/index.html:117 #: website/public/faq/index.html:116 @@ -11684,13 +11696,14 @@ msgctxt "shared" msgid "Guides" msgstr "Rehberler" -#: website/public/blog/index.html:222 +#: website/public/blog/index.html:236 +#: website/public/blog/reverse-engineering-models/index.html:875 #: website/public/blog/touhou-reconstruction/index.html:1072 #: website/public/examples/dino-lab/index.html:467 #: website/public/faq/index.html:565 #: website/public/first-investigation/index.html:515 #: website/public/get-started/index.html:874 -#: website/public/guides/browser/index.html:490 +#: website/public/guides/browser/index.html:499 #: website/public/guides/index.html:274 #: website/public/guides/javascript/index.html:495 #: website/public/guides/native/index.html:501 website/public/index.html:1183 @@ -11704,13 +11717,14 @@ msgctxt "shared" msgid "Source" msgstr "Kaynak kod" -#: website/public/blog/index.html:223 +#: website/public/blog/index.html:237 +#: website/public/blog/reverse-engineering-models/index.html:876 #: website/public/blog/touhou-reconstruction/index.html:1073 #: website/public/examples/dino-lab/index.html:468 #: website/public/faq/index.html:566 #: website/public/first-investigation/index.html:516 #: website/public/get-started/index.html:875 -#: website/public/guides/browser/index.html:491 +#: website/public/guides/browser/index.html:500 #: website/public/guides/index.html:275 #: website/public/guides/javascript/index.html:496 #: website/public/guides/native/index.html:502 website/public/index.html:1184 @@ -11724,13 +11738,14 @@ msgctxt "shared" msgid "MIT license" msgstr "MIT lisansı" -#: website/public/blog/index.html:224 +#: website/public/blog/index.html:238 +#: website/public/blog/reverse-engineering-models/index.html:877 #: website/public/blog/touhou-reconstruction/index.html:1074 #: website/public/examples/dino-lab/index.html:469 #: website/public/faq/index.html:567 #: website/public/first-investigation/index.html:517 #: website/public/get-started/index.html:876 -#: website/public/guides/browser/index.html:492 +#: website/public/guides/browser/index.html:501 #: website/public/guides/index.html:276 #: website/public/guides/javascript/index.html:497 #: website/public/guides/native/index.html:503 website/public/index.html:1185 @@ -11744,13 +11759,14 @@ msgctxt "shared" msgid "Join the community" msgstr "Topluluğa katıl" -#: website/public/blog/index.html:225 +#: website/public/blog/index.html:239 +#: website/public/blog/reverse-engineering-models/index.html:878 #: website/public/blog/touhou-reconstruction/index.html:1075 #: website/public/examples/dino-lab/index.html:470 #: website/public/faq/index.html:568 #: website/public/first-investigation/index.html:518 #: website/public/get-started/index.html:877 -#: website/public/guides/browser/index.html:493 +#: website/public/guides/browser/index.html:502 #: website/public/guides/index.html:277 #: website/public/guides/javascript/index.html:498 #: website/public/guides/native/index.html:504 website/public/index.html:1186 @@ -12688,6 +12704,1394 @@ msgctxt "runtime" msgid "Switch to dark mode" msgstr "Koyu temaya geç" +#. {ph0}: +#. {ph1}: +#: website/public/guides/browser/index.html:480 +#, python-brace-format +msgctxt "page:guides/browser/index.html" +msgid "" +"For HTTP request and response bodies, use {ph0}Fiddler Everywhere: SAZ " +"captures, MCP and public skills{ph1}. Export SAZ or HAR 1.2 for offline REA " +"capture Evidence, or connect Everywhere's MCP server to inspect captured " +"traffic with your agent." +msgstr "" +"HTTP istek ve yanıt gövdelerini incelemek için {ph0}Fiddler Everywhere: SAZ " +"kayıtları, MCP ve herkese açık beceriler{ph1} rehberini kullanın. REA ile " +"çevrimdışı kayıt Evidence elde etmek için SAZ veya HAR 1.2 dışa aktarın veya" +" kaydedilen trafiği ajanınızla incelemek için Everywhere MCP sunucusunu " +"bağlayın." + +#. {ph0}:
+#: website/public/blog/index.html:200 +#, python-brace-format +msgctxt "page:blog/index.html" +msgid "Model specialization{ph0}Essay" +msgstr "Uzmanlaşmış modeller{ph0}Deneme" + +#: website/public/blog/index.html:202 +msgctxt "page:blog/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "Tersine mühendislik için daha büyük bir model gerekli mi?" + +#: website/public/blog/index.html:203 +msgctxt "page:blog/index.html" +msgid "" +"What Touhou reconstruction suggests about general models, specialist models " +"and learning from compiler feedback." +msgstr "" +"Touhou yeniden yapılandırma deneyiminden genel amaçlı modeller, uzmanlaşmış " +"modeller ve derleyici geri bildiriminden öğrenme üzerine düşünceler." + +#: website/public/blog/index.html:210 +msgctxt "page:blog/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Yazan N0zoM1z0 · 13 Ekim 2026" + +#: website/public/blog/reverse-engineering-models/index.html:103 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Skip to content" +msgstr "İçeriğe geç" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:194 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Home{ph1} / {ph2}Blog{ph3} / Models and RE" +msgstr "" +"{ph0}Ana Sayfa{ph1} / {ph2}Blog{ph3} / Modeller ve tersine mühendislik" + +#: website/public/blog/reverse-engineering-models/index.html:197 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Essay" +msgstr "Deneme" + +# Reviewed for technical meaning and readability; preserve the distinction +# between historical evidence and proposed experiments. +#: website/public/blog/reverse-engineering-models/index.html:198 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "Tersine mühendisliğin daha büyük bir modele ihtiyacı var mı?" + +#: website/public/blog/reverse-engineering-models/index.html:199 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Yazan N0zoM1z0 · 13 Ekim 2026" + +#: website/public/blog/reverse-engineering-models/index.html:201 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Takeaway" +msgstr "Önemli noktalar" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:203 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Model capability changes the hypotheses an agent can find.{ph1} Later " +"sessions have solved problems that earlier work left unresolved." +msgstr "" +"{ph0}Model yeteneği, bir yapay zeka aracısının hangi hipotezleri " +"önerebileceğini etkiler.{ph1} Daha sonraki oturumlarda, daha önceki " +"çalışmaların çözümsüz bıraktığı sorunlar çözüldü." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:211 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A familiar compiler gives specialization something concrete to " +"learn.{ph1} Some exact matches depend on narrow, reusable source patterns." +msgstr "" +"{ph0}Tanıdık bir derleyici, özel bir modelin öğrenilmesi için somut modeller" +" sunar.{ph1} Bazı tam makine kodu eşleşmeleri, belirli, yeniden " +"kullanılabilir kaynak kodu modellerine bağlıdır." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:218 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Give repository knowledge a fair comparison.{ph1} A smaller model with " +"good retrieval may already know enough to make progress." +msgstr "" +"{ph0}Adil bir karşılaştırmaya veri deposu bilgisini dahil edin.{ph1} İyi " +"bilgi erişimine sahip daha küçük bir model, ilerleme kaydetmek için zaten " +"yeterli bilgiye sahip olabilir." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:223 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Train the next useful experiment, and test its oracle.{ph1} The value " +"of feedback depends on what the checker actually verifies." +msgstr "" +"{ph0}Modeli, bir sonraki faydalı deneyi seçecek şekilde eğitin ve doğrulama " +"denetleyicisini test edin.{ph1} Geri bildirim, yalnızca denetleyicinin doğru" +" şeyleri doğruladığı ölçüde faydalıdır." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:230 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Measure accepted reconstruction against time and cost.{ph1} That is the" +" result I want from the whole agent." +msgstr "" +"{ph0}Doğrulanmış yeniden yapılandırma sonuçlarını zaman ve maliyetle " +"birlikte değerlendirin.{ph1} Yapay zeka aracısı iş akışının tamamı için " +"ölçmek istediğim şey bu." + +#: website/public/blog/reverse-engineering-models/index.html:241 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "In this article" +msgstr "Bu makalede" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:243 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What better models changed{ph1}" +msgstr "{ph0}Daha güçlü modeller neyi değiştirdi?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:244 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Why specialize?{ph1}" +msgstr "{ph0}Neden uzmanlaşalım?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:245 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Where experience lives{ph1}" +msgstr "{ph0}Bu deneyim nerede saklanmalı?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:246 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Learning the next experiment{ph1}" +msgstr "{ph0}Bir sonraki faydalı deneyi öğrenmek{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:247 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}RE and reinforcement learning{ph1}" +msgstr "{ph0}Tersine mühendislik ve takviyeli öğrenme{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:248 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}When the oracle is wrong{ph1}" +msgstr "{ph0}Doğrulama denetleyicisi yanlış olduğunda{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:249 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}A comparison worth running{ph1}" +msgstr "{ph0}Çalıştırılmaya değer bir karşılaştırma{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:250 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What kind of agent?{ph1}" +msgstr "{ph0}Bu ne tür bir yapay zeka ajanını akla getiriyor?{ph1}" + +#: website/public/blog/reverse-engineering-models/index.html:258 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What better models changed" +msgstr "Daha güçlü modeller neyi değiştirdi?" + +#: website/public/blog/reverse-engineering-models/index.html:259 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Since August 2026, I have been exploring how much reverse engineering an " +"agent can carry out with access to a program and a way to check its work. " +"Most of my experience comes from reconstructing games, especially Touhou. As" +" I have changed models, the repositories have kept a record of what each " +"session managed to recover." +msgstr "" +"Ağustos 2026'dan bu yana, bir aracının bir programa erişim ve onun " +"çalışmasını kontrol etmenin bir yolu ile ne kadar tersine mühendislik " +"gerçekleştirebileceğini araştırıyorum. Deneyimlerimin çoğu, özellikle Touhou" +" olmak üzere oyunların yeniden yapılandırılmasından geliyor. Modelleri " +"değiştirdiğimde, depolar her oturumda kurtarılanların kaydını tuttu." + +#: website/public/blog/reverse-engineering-models/index.html:267 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of that work aims at exact reconstruction: we compile the maintained " +"source under a pinned historical toolchain and check whether it reproduces " +"the target function. When a later model succeeds where earlier work stalled," +" the source changes show what made that progress possible." +msgstr "" +"Bu çalışmanın büyük bir kısmı tam olarak yeniden yapılanmayı amaçlamaktadır:" +" Bakımı yapılan kaynağı, sabit versiyonlara sahip tarihsel bir araç zinciri " +"altında derliyoruz ve hedef işlevi yeniden üretip üretmediğini kontrol " +"ediyoruz. Önceki çalışmaların durduğu yerde daha sonraki bir model başarılı " +"olduğunda, kaynak değişiklikleri bu ilerlemeyi neyin mümkün kıldığını " +"gösterir." + +#: website/public/blog/reverse-engineering-models/index.html:274 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The temporary values had to stay alive together" +msgstr "Geçici değişkenler örtüşen yaşam sürelerine ihtiyaç duyuyordu" + +#: website/public/blog/reverse-engineering-models/index.html:275 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of a TH09 attack function's logic had already been reconstructed, but " +"the compiler still stored temporary results differently from the original. " +"The remaining problem was how long those results needed to stay alive and " +"when their storage could be reused." +msgstr "" +"TH09 saldırı fonksiyonunun mantığının büyük bir kısmı zaten yeniden " +"oluşturulmuştu, ancak derleyici hala geçici sonuçları orijinalinden farklı " +"şekilde saklıyordu. Geriye kalan sorun, bu sonuçların ne kadar süreyle canlı" +" kalması gerektiği ve depolarının ne zaman yeniden kullanılabileceğiydi." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:282 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A {ph0}GPT-6-Sol checkpoint{ph1} recorded this mismatch after testing " +"several ordinary source arrangements. When {ph2}GPT-6.1-Sol returned to the " +"problem{ph3}, it found a useful hypothesis about how long those temporary " +"values needed to stay alive." +msgstr "" +"Bir {ph0}GPT-6-Sol oturum kaydı{ph1}, birkaç sıradan kaynak kodu " +"düzenlemesini test ettikten sonra bu uyumsuzluğu bildirdi. {ph2}GPT-6.1-Sol " +"sorunu tekrar incelediğinde{ph3}, bu geçici değişkenlerin yaşam süreleri " +"hakkında yararlı bir hipotez öne sürdü." + +#: website/public/blog/reverse-engineering-models/index.html:296 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The function produced temporary results and copied them into animation " +"positions. In C++, a temporary result stays alive until the expression using" +" it finishes. Grouping the real operations into one expression kept those " +"results alive together, so the compiler could no longer reuse the same " +"storage for all of them." +msgstr "" +"İşlev geçici sonuçlar üretti ve bunları animasyon konumlarına kopyaladı. " +"C++'da geçici bir sonuç, onu kullanan ifade bitene kadar canlı kalır. Gerçek" +" işlemleri tek bir ifadede gruplandırmak bu sonuçları bir arada canlı tuttu," +" böylece derleyici artık aynı depolamayı hepsi için yeniden kullanamıyordu." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:303 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"One result still shared storage with a value used later for collision " +"checks. Giving that value its own lifetime resolved the remaining overlap. " +"After several targeted experiments, {ph0}a clean rebuild confirmed that the " +"full function matched{ph1}." +msgstr "" +"Sonuçlardan biri, daha sonra çarpışma kontrolleri için kullanılan bir " +"değerle depolamayı hâlâ paylaşıyordu. Bu değere kendi ömrünü vermek, kalan " +"örtüşmeyi çözdü. Birkaç hedefli denemeden sonra, {ph0}kaynaktan yapılan " +"temiz bir yeniden derleme, tam işlevin eşleştiğini doğruladı{ph1}." + +#: website/public/blog/reverse-engineering-models/index.html:312 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What interests me is how the discovery and the tests worked together. The " +"lifetime hypothesis gave the agent a new direction; compiler feedback helped" +" it refine that hypothesis into a source arrangement that matched." +msgstr "" +"Keşif ve testlerin birbirini nasıl desteklediğiyle ilgileniyorum. Geçici " +"nesne yaşam süreleri hakkındaki hipotez, yapay zeka ajanına yeni bir yön " +"verdi. Derleyici geri bildirimi, bu hipotezi eşleşen bir kaynak kodu " +"düzenlemesine dönüştürmeye yardımcı oldu." + +#: website/public/blog/reverse-engineering-models/index.html:318 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A different control-flow graph" +msgstr "Farklı bir kontrol akışı grafiği" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:319 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A second TH09 function shows why that new direction can matter. Here, the " +"obstacle was how several branches shared their final work. " +"{ph0}GPT-6.1-Sol's earlier work{ph1} got parts of the function to match, but" +" the branches still joined up differently from the original." +msgstr "" +"İkinci bir TH09 işlevi, bu yeni yönün neden önemli olabileceğini gösterir. " +"Buradaki engel, birden fazla şubenin nihai çalışmalarını paylaşmasıydı. " +"{ph0}GPT-6.1-Sol'nin önceki çalışması{ph1} işlevin bazı kısımlarını " +"eşleştirdi ancak dallar yine de orijinalinden farklı bir şekilde birleşti." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:330 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}GPT-6-Astra changed the surrounding control-flow graph{ph1}. The main " +"paths now led to a shared final update and return. In that source structure," +" the compiler combined the branches in the same way as the original. The " +"complete function matched, and a {ph2}separate rebuild check confirmed " +"it{ph3}." +msgstr "" +"{ph0}GPT-6-Astra çevredeki kontrol akışı grafiğini değiştirdi{ph1}. Ana " +"yollar artık paylaşılan bir son güncellemeye ve geri dönüşe yol açıyordu. Bu" +" kaynak yapısında derleyici, dalları orijinaldekiyle aynı şekilde " +"birleştirdi. İşlevin tamamı eşleşti ve {ph2}ayrı bir yeniden oluşturma " +"kontrolü bunu doğruladı{ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:342 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Changing the representation opened a more productive search space.{ph1}" +" Both cases give me a reason to take general model capability seriously: an " +"agent can keep trying local variations for a long time before finding a " +"better way to express the surrounding work." +msgstr "" +"{ph0}Temsili değiştirmek daha verimli bir arama alanı açtı.{ph1} Her iki " +"durum da bana genel model yeteneğini ciddiye almam için bir neden veriyor: " +"Bir temsilci, çevredeki işi ifade etmenin daha iyi bir yolunu bulmadan önce " +"yerel varyasyonları uzun süre denemeye devam edebilir." + +#: website/public/blog/reverse-engineering-models/index.html:351 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"At the same time, the later sessions started with earlier source and " +"recorded failures. That accumulated work was part of their advantage. The " +"history lets us inspect the discoveries and their checks; a controlled " +"comparison would help separate the effect of the model from the experience " +"it inherited." +msgstr "" +"Sonraki oturumlar önceki kaynak kodu ve kaydedilmiş başarısız denemeleri de " +"devraldı. Bu birikim avantajlarının bir parçasıydı. Geçmiş, keşifleri ve " +"doğrulamalarını incelememizi sağlar; kontrollü bir karşılaştırma ise modelin" +" etkisini devraldığı deneyimin etkisinden ayırmaya yardımcı olur." + +#: website/public/blog/reverse-engineering-models/index.html:358 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is where the question becomes interesting to me. Some of the progress " +"came from finding a new representation, while much of the remaining work " +"involved testing source forms against a familiar compiler. How much needs " +"broad reasoning ability, and how much could a model learn through repeated " +"experience with that compiler and code?" +msgstr "" +"Soruyu benim için ilginç kılan da bu. Yeni bir temsil bulma konusunda bir " +"miktar ilerleme sağlandı. Geriye kalan çalışmaların çoğu, farklı kaynak kodu" +" düzenlemelerinin tanıdık bir derleyiciye karşı test edilmesini içeriyordu. " +"Bunun ne kadarı geniş muhakeme yeteneği gerektiriyor? Bir model, o derleyici" +" ve kodla tekrar tekrar çalışarak ne kadar öğrenebilir?" + +#: website/public/blog/reverse-engineering-models/index.html:368 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why might a specialist do well?" +msgstr "Özel bir yapay zeka modeli neden başarılı olabilir?" + +#: website/public/blog/reverse-engineering-models/index.html:369 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Touhou gives us a concrete reason to expect that experience to help. Much of" +" the game code comes from the same author, ZUN, so recurring habits are a " +"reasonable starting assumption. The repositories back that up with useful " +"examples: related loops and helper arrangements have supplied hypotheses " +"across games." +msgstr "" +"Touhou bize bu deneyimin yardımcı olmasını beklememiz için somut bir neden " +"veriyor. Oyun kodunun büyük bir kısmı aynı yazardan (ZUN) geliyor, " +"dolayısıyla yinelenen alışkanlıklar makul bir başlangıç ​​varsayımıdır. " +"Depolar bunu yararlı örneklerle desteklemektedir: İlgili döngüler ve " +"yardımcı düzenlemeler, oyunlar arasında hipotezler sağlamıştır." + +#: website/public/blog/reverse-engineering-models/index.html:376 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those source patterns become more specific when we include the compiler. " +"Under a particular version and set of options, small changes in source can " +"produce repeatable differences in machine code. Some of those differences " +"are surprisingly obscure." +msgstr "" +"Derleyiciyi dahil ettiğimizde bu kaynak kalıpları daha spesifik hale gelir. " +"Belirli bir sürüm ve seçenekler kümesi altında, kaynaktaki küçük " +"değişiklikler makine kodunda tekrarlanabilir farklılıklara neden olabilir. " +"Bu farklılıkların bazıları şaşırtıcı derecede belirsizdir." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: Supervisor::LoadDat +#. {ph4}: +#: website/public/blog/reverse-engineering-models/index.html:382 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In TH095, for example, {ph0}even the name used for a local variable could " +"change where VC7.1 stored it{ph1}. Moving the declaration alone did not " +"solve the layout, but an earlier background function had revealed a naming " +"pattern that affected the compiler's choice. That gave us a more promising " +"hypothesis for {ph2}the reconstruction of {ph3}{ph4}." +msgstr "" +"Örneğin TH095'de, {ph0}yerel bir değişkenin adı bile VC7.1'nin onu nerede " +"sakladığını etkileyebilir{ph1}. Bildiriminin taşınması depolama düzenini " +"çözmedi. Ancak daha önceki bir arka plan işlevi, derleyicinin seçimini " +"etkileyen bir adlandırma modelini ortaya çıkarmıştı. Bu, {ph2}yeniden inşa " +"etmek{ph3}{ph4} için daha umut verici bir hipotez önerdi." + +#: website/public/blog/reverse-engineering-models/index.html:396 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reusing that pattern for the file-size variable produced the required layout" +" without adding storage. The full function then matched." +msgstr "" +"Dosya boyutu değişkeni için bu kalıbın yeniden kullanılması, depolama alanı " +"eklemeden gerekli düzeni oluşturdu. Daha sonra tam işlev eşleşti." + +#: website/public/blog/reverse-engineering-models/index.html:401 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is the sort of narrow detail a general model may not know when it " +"arrives. A specialist trained on this compiler could encounter many such " +"cases and learn which source variations are worth testing first." +msgstr "" +"Bu, genel amaçlı bir yapay zeka modelinin sorunla ilk karşılaştığında " +"bilemeyeceği çok özel bir ayrıntıdır. Bu derleyici için uzmanlaşmış bir " +"model, bu tür birçok durumu görebilir ve hangi kaynak kodu varyasyonlarının " +"ilk önce test edilmeye değer olduğunu öğrenebilir." + +#: website/public/blog/reverse-engineering-models/index.html:407 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The opportunity is especially clear in final matching problems where we " +"already understand the behavior. We are then asking which ordinary source " +"form this compiler turns into the observed instructions. Specialization " +"could help the model propose those forms more efficiently, while the target " +"still determines which hypotheses survive." +msgstr "" +"Davranışı zaten anladığımız son eşleştirme problemlerinde bu fırsat " +"özellikle açıktır. Daha sonra bu derleyicinin hangi sıradan kaynak kodu " +"düzenlemesini gözlemlenen talimatlara dönüştürdüğünü soruyoruz. Uzmanlaşma, " +"modelin bu formları daha verimli bir şekilde önermesine yardımcı olurken, " +"hedef hâlâ hangi hipotezlerin hayatta kalacağını belirliyor." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:415 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The useful patterns would have to retain their context, though. TH08 uses an" +" earlier Visual C++ version than TH095 and TH09. TH04 uses a different DOS-" +"era compiler, and its {ph0}player reconstruction{ph1} needed different " +"hypotheses about source patterns and how the compiled code was combined into" +" an executable. Shared authorship gives us a prior, but each target tells us" +" whether it applies. Distinguishing those contexts is part of what a " +"specialist would need to learn." +msgstr "" +"Yararlı kalıplar bağlamlarını korumalıdır. TH08, TH095 ve TH09’dan daha eski" +" bir Visual C++ sürümü kullanır. TH04 ise DOS dönemine ait başka bir " +"derleyici kullanır. {ph0}Oyuncu kodunun yeniden yapılandırılması{ph1}, " +"kaynak kalıpları ve kodun çalıştırılabilir dosyaya nasıl bağlandığı hakkında" +" farklı hipotezler gerektirdi. Aynı yazar bize bir başlangıç varsayımı " +"verir; bunun geçerli olup olmadığını her hedefte kontrol ederiz. Uzman model" +" bu bağlamları ayırt etmeyi öğrenmelidir." + +#: website/public/blog/reverse-engineering-models/index.html:431 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Where should that experience live?" +msgstr "Bu deneyim nerede saklanmalı?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:432 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before deciding to teach those patterns through model training, I would look" +" at the knowledge we already keep in the repository. In the {ph0}previous " +"essay{ph1}, I wrote about it becoming the project's working memory. That " +"memory includes compiler lessons, along with the source and failed " +"experiments that made them useful." +msgstr "" +"Bu kalıpları model eğitimi yoluyla öğretmeye karar vermeden önce, " +"halihazırda depoda tuttuğumuz bilgilere bakardım. {ph0}Önceki makalemde{ph1}" +" bunun projenin çalışma belleği haline geldiğini yazmıştım. Bu hafıza, " +"derleyici derslerini, kaynağı ve onları faydalı kılan başarısız deneyleri " +"içerir." + +#: website/public/blog/reverse-engineering-models/index.html:441 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"By retrieving that record, an agent can start with a known result and " +"inspect how we reached it. It can also avoid alternatives that have already " +"failed in the same context. The starting point improves without changing the" +" model's weights." +msgstr "" +"Bir aracı, bu kaydı alarak bilinen bir sonuçla başlayabilir ve bu sonuca " +"nasıl ulaştığımızı inceleyebilir. Aynı bağlamda halihazırda başarısız olan " +"alternatiflerden de kaçınılabilir. Başlangıç ​​noktası, modelin eğitilmiş " +"parametrelerini değiştirmeden iyileşir." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:447 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"When reconstructing TH09's enemy-spawn function, a {ph0}GPT-6-Luna-max " +"session{ph1} drew a hypothesis from TH08's corresponding source family. That" +" earlier source suggested how to structure the loop and handle failures " +"within it. TH09's own evidence supplied the game-specific details, and clean" +" rebuilds confirmed the complete match." +msgstr "" +"TH09'nin düşman yaratma işlevi yeniden yapılandırılırken, bir " +"{ph0}GPT-6-Luna-max oturumu{ph1} ilgili TH08 kaynak kodundan bir hipotez " +"aldı. Önceki kod, döngünün nasıl organize edileceğini ve içindeki arızaların" +" nasıl ele alınacağını öneriyordu. TH09'in kendi kanıtları oyuna özgü " +"ayrıntıları sağladı. Kaynaktan yapılan yeniden derlemeler, işlevin tamamının" +" eşleştiğini doğruladı." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:458 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}\"The model did not know that compiler quirk\" and \"the model could " +"not use it\" are different problems.{ph1} Retrieval may solve the first, but" +" turning that fact into a useful candidate still takes work. And if the " +"candidate fails, the model must decide what the result means for its next " +"attempt. Knowing the relevant pattern is only the beginning of that process." +msgstr "" +"{ph0}\"Model, derleyici tuhaflığını bilmiyordu\" ve \"model bunu " +"kullanamadı\" farklı sorunlardır.{ph1} Geri getirme ilk sorunu çözebilir, " +"ancak bu gerçeği yararlı bir adaya dönüştürmek yine de çalışma gerektirir. " +"Aday kaynak kodu başarısız olursa modelin bir sonraki denemesinde sonucun ne" +" anlama geldiğine karar vermesi gerekir. İlgili modeli bilmek bu sürecin " +"yalnızca başlangıcıdır." + +#: website/public/blog/reverse-engineering-models/index.html:468 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For that reason, I would include a smaller general model with good retrieval" +" in any comparison. Giving it the same library of experience would help us " +"identify what training adds: better candidate generation, better decisions " +"after feedback, or both." +msgstr "" +"Bu nedenle, herhangi bir karşılaştırmaya, iyi geri çağırma özelliğine sahip," +" daha küçük, genel amaçlı bir yapay zeka modelini dahil ederim. Aynı deneyim" +" kitaplığını ona vermek, eğitimin ne gibi katkılar sağladığını belirlememize" +" yardımcı olacaktır: daha iyi aday oluşturma, geri bildirim sonrasında daha " +"iyi kararlar veya her ikisi." + +#: website/public/blog/reverse-engineering-models/index.html:476 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Learning the next useful experiment" +msgstr "Bir sonraki faydalı deneyi öğrenmek" + +#: website/public/blog/reverse-engineering-models/index.html:477 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those decisions after feedback are what I would most like to study. In " +"agentic reconstruction, the model proposes an explanation and tests it " +"against the program. Even when the candidate fails, the result can narrow " +"the remaining possibilities and suggest a more useful next experiment." +msgstr "" +"Geri bildirim sonrasında verilen kararlar, üzerinde çalışmayı en çok " +"istediğim şeylerdir. Failsel yeniden yapılanmada model bir açıklama önerir " +"ve bunu programa karşı test eder. Aday kaynak kodu başarısız olsa bile " +"sonuç, kalan olasılıkları daraltabilir ve bir sonraki deneyin daha yararlı " +"olmasını önerebilir." + +#: website/public/blog/reverse-engineering-models/index.html:484 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Consider the temporary-results example again. Its final source shows the " +"arrangement that worked, but the route to that source contains more " +"information. Each of the following experiments helped identify what still " +"needed to change." +msgstr "" +"Geçici sonuçlar örneğini tekrar düşünün. Son kaynağı işe yarayan düzenlemeyi" +" gösteriyor ancak bu kaynağa giden yol daha fazla bilgi içeriyor. Aşağıdaki " +"deneylerin her biri, hala neyin değişmesi gerektiğinin belirlenmesine " +"yardımcı oldu." + +#: website/public/blog/reverse-engineering-models/index.html:497 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Experiment" +msgstr "Deney" + +#: website/public/blog/reverse-engineering-models/index.html:498 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What its result taught us" +msgstr "Sonucunun bize öğrettikleri" + +#: website/public/blog/reverse-engineering-models/index.html:503 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Try a different way to copy the results" +msgstr "Sonuçları kopyalamanın farklı bir yolunu deneyin" + +#: website/public/blog/reverse-engineering-models/index.html:504 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The copy code changes, but the function still differs." +msgstr "Kopya kodu değişir ancak işlev hâlâ farklıdır." + +#: website/public/blog/reverse-engineering-models/index.html:509 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Use the returned values directly" +msgstr "Döndürülen değerleri doğrudan kullanın" + +#: website/public/blog/reverse-engineering-models/index.html:510 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Copying follows the original pattern, but temporary storage still differs." +msgstr "" +"Kopyalama orijinal düzeni takip eder ancak geçici depolama yine de " +"farklıdır." + +#: website/public/blog/reverse-engineering-models/index.html:516 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the temporary results alive together" +msgstr "Tüm geçici değişkenleri aynı anda geçerli tutun" + +#: website/public/blog/reverse-engineering-models/index.html:517 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"They get separate storage, but one still overlaps a value used later for " +"collision checks." +msgstr "" +"Ayrı bir depolamaya sahipler, ancak bir tanesi daha sonra çarpışma " +"kontrolleri için kullanılan bir değerle örtüşüyor." + +#: website/public/blog/reverse-engineering-models/index.html:523 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the collision-check value separate" +msgstr "Çarpışma kontrolü değerini ayrı tutun" + +#: website/public/blog/reverse-engineering-models/index.html:524 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A clean rebuild confirms the complete match." +msgstr "" +"Kaynaktan alınan temiz bir yeniden derleme, eşleşmenin tamamını doğrular." + +#: website/public/blog/reverse-engineering-models/index.html:529 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Selected experiments from the attack-function reconstruction above, grouped " +"by what they taught us." +msgstr "" +"Yukarıdaki saldırı fonksiyonunun yeniden yapılandırılmasından bize " +"öğrettiklerine göre gruplandırılmış seçilmiş deneyler." + +#: website/public/blog/reverse-engineering-models/index.html:533 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A training record that keeps this sequence can teach more than the final " +"source alone. It connects each proposed change to an observed result and the" +" next decision. Keeping ineffective variations matters too: they tell a " +"later agent which hypotheses have already been tested in that context." +msgstr "" +"Bu sırayı tutan bir eğitim kaydı, tek başına nihai kaynaktan daha fazlasını " +"öğretebilir. Önerilen her değişikliği gözlemlenen bir sonuca ve bir sonraki " +"karara bağlar. Etkisiz varyasyonları korumak da önemlidir: Daha sonraki bir " +"temsilciye, o bağlamda hangi hipotezlerin zaten test edildiğini söylerler." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:540 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}I would want the model to learn how to use feedback.{ph1} A neutral " +"result is a reason to reconsider the variation; a changed mismatch may point" +" to another cause. When local edits stop being informative, the agent should" +" be able to revisit the source representation or inspect the evidence again." +msgstr "" +"{ph0}Modelin geri bildirimi nasıl kullanacağını öğrenmesini isterim.{ph1} " +"Nötr bir sonuç, varyasyonun yeniden değerlendirilmesi için bir nedendir; " +"değişen bir uyumsuzluk başka bir nedene işaret edebilir. Yerel düzenlemeler " +"bilgilendirici olmayı bıraktığında, aracının kaynak temsilini yeniden gözden" +" geçirebilmesi veya kanıtları yeniden inceleyebilmesi gerekir." + +#: website/public/blog/reverse-engineering-models/index.html:549 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Learning to make those choices requires autonomy. Sometimes the useful next " +"step is inspecting another caller rather than compiling another edit. A " +"rigid sequence of required rewrites would remove that choice, along with " +"part of the behavior we want the model to learn." +msgstr "" +"Bu seçimleri yapmayı öğrenmek özerklik gerektirir. Bazen bir sonraki yararlı" +" adım, başka bir düzenleme derlemek yerine başka bir arayanı incelemektir. " +"Gerekli yeniden yazma işlemlerinin katı bir dizisi, modelin öğrenmesini " +"istediğimiz davranışın bir kısmıyla birlikte bu seçimi de ortadan " +"kaldıracaktır." + +#: website/public/blog/reverse-engineering-models/index.html:558 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why RE is interesting for reinforcement learning" +msgstr "Takviyeli öğrenme için tersine mühendislik neden ilgi çekicidir?" + +#: website/public/blog/reverse-engineering-models/index.html:559 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Bounded reconstruction gives us a practical environment for studying these " +"choices. Once the target and compiler are pinned, many source experiments " +"produce quick, repeatable feedback. The agent can compile a candidate, " +"inspect the mismatch and use that observation to choose its next step." +msgstr "" +"Sınırlı yeniden yapılandırma bize bu seçimleri incelemek için pratik bir " +"ortam sağlar. Hedef ve derleyici sürümleri sabitlendikten sonra birçok " +"kaynak denemesi hızlı, tekrarlanabilir geri bildirim üretir. Temsilci bir " +"adayı derleyebilir, uyumsuzluğu inceleyebilir ve bu gözlemi bir sonraki " +"adımını seçmek için kullanabilir." + +#: website/public/blog/reverse-engineering-models/index.html:566 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reinforcement learning could train that decision-making process through the " +"feedback it receives. A candidate accepted by a defined check gives us a " +"concrete outcome, while the sequence of attempts lets us observe how much " +"work it took to reach it." +msgstr "" +"Takviyeli öğrenme, aldığı geri bildirim yoluyla karar verme sürecini " +"eğitebilir. Belirli bir kontrolle kabul edilen bir aday bize somut bir sonuç" +" verirken, denemelerin sırası bu sonuca ulaşmak için ne kadar çaba " +"harcandığını gözlemlememize olanak tanır." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:572 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"There is already research supporting this direction. {ph0}LLM4Decompile{ph1}" +" studies models specialized for decompilation. {ph2}AutoDecompiler{ph3} " +"trains multi-turn refinement using compiler and execution feedback. Their " +"studies focus on functional recovery. For this work, I would use them as a " +"starting point for the related task of finding source that matches a " +"historical compiler's output." +msgstr "" +"Bu yönü destekleyen araştırmalar zaten var. {ph0}LLM4Decompile{ph1} kaynak " +"koda dönüştürme için uzmanlaşmış modelleri inceliyor. " +"{ph2}AutoDecompiler{ph3}, derleyici ve yürütme geri bildirimini kullanarak " +"çok turlu iyileştirmeyi eğitir. Çalışmaları fonksiyonel iyileşmeye " +"odaklanıyor. Bu çalışma için bunları, tarihsel bir derleyicinin çıktısıyla " +"eşleşen kaynağı bulma görevi için bir başlangıç ​​noktası olarak " +"kullanacağım." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:584 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To reach such a well-defined task, though, we may first need to establish " +"what a function owns or devise a new runtime measurement. In TH08, for " +"example, a {ph0}collision difference led back to how the original program " +"handled floating-point calculations{ph1}. Before the agent could test a " +"useful correction, it had to work out what to measure." +msgstr "" +"Böylesine iyi tanımlanmış bir göreve ulaşmak için öncelikle bir fonksiyonun " +"hangi kaynaklara sahip olduğunu belirlememiz veya programın çalışırkenki " +"davranışını gözlemlemek için yeni bir yöntem tasarlamamız gerekebilir. " +"Örneğin TH08'de, bir {ph0}çarpışma farkı, orijinal programın kayan nokta " +"hesaplamalarını{ph1} nasıl işlediğini ortaya çıkardı. Aracının yararlı bir " +"düzeltmeyi test edebilmesi için önce neyi ölçeceğini bulması gerekiyordu." + +#: website/public/blog/reverse-engineering-models/index.html:595 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This suggests another role for a broadly capable model: helping make the " +"problem checkable in the first place. Once that check exists, we can " +"investigate how far a learned search policy can get with it." +msgstr "" +"Bu, geniş kapsamlı bir model için başka bir rol öneriyor: ilk etapta sorunun" +" kontrol edilebilir olmasına yardımcı olmak. Bu kontrol bir kez mevcut " +"olduğunda, öğrenilmiş bir arama politikasının bununla ne kadar ileri " +"gidebileceğini araştırabiliriz." + +#: website/public/blog/reverse-engineering-models/index.html:603 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "When the oracle is wrong" +msgstr "Doğrulama denetleyicisi yanlış olduğunda" + +#: website/public/blog/reverse-engineering-models/index.html:604 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Once we train against that check, its coverage becomes part of what the " +"model learns. I call the reference or checker an oracle. Even if it gives a " +"repeatable answer, we still need to understand what that answer covers." +msgstr "" +"Bir kontrolü eğitimde kullandığımızda, kapsamı da öğrenme ortamının bir " +"parçası olur. Böyle bir referansa veya doğrulayıcıya oracle diyorum. " +"Tekrarlanabilir bir yanıt bile gerçekte neyi doğruladığına göre " +"anlaşılmalıdır." + +#: website/public/blog/reverse-engineering-models/index.html:610 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"TH08 showed how much can hide outside that coverage. Some reconstructed " +"functions passed their exact comparisons with wrong floating constants. The " +"comparator adjusted relocated addresses before checking instructions, but it" +" checked the values behind those addresses only when they had been " +"explicitly recorded for verification. Other literal references could pass " +"without their values being examined." +msgstr "" +"TH08 bu kapsamın dışında ne kadar şeyin saklanabileceğini gösterdi. Yeniden " +"oluşturulan bazı işlevler, tam karşılaştırmalarını yanlış kayan sabitlerle " +"geçti. Karşılaştırıcı, talimatları kontrol etmeden önce yeniden " +"konumlandırılan adresleri ayarladı, ancak bu adreslerin arkasındaki " +"değerleri yalnızca doğrulama için açıkça kaydedildiklerinde kontrol etti. " +"Diğer gerçek referanslar, değerleri incelenmeden geçebilir." + +#. {ph0}: +#. {ph1}: +#. {ph2}: 1.0f +#. {ph3}: 32.0f +#: website/public/blog/reverse-engineering-models/index.html:619 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To close that gap, a {ph0}wider audit{ph1} found incorrect constants in " +"functions that had already been accepted. In one calculation, the " +"reconstructed source added {ph2} where the original added {ph3}." +msgstr "" +"Bu boşluğu gidermek için, {ph0}daha geniş bir denetim{ph1}, doğrulamayı " +"zaten geçmiş işlevlerde hatalı sabitler buldu. Bir hesaplamada, yeniden " +"oluşturulan kaynak kodu {ph2}'yi eklerken, orijinali {ph3}'ü ekledi." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:629 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A learner rewarded by the old check could learn to produce the wrong " +"reconstruction.{ph1} We had not trained an RL model against this oracle; the" +" observed failure was in the check itself. But using that check as a reward " +"would have made the same blind spot part of the learning environment." +msgstr "" +"{ph0}Eski doğrulama denetimiyle ödüllendirilen bir model, hatalı bir yeniden" +" yapılandırma üretmeyi öğrenebilir.{ph1} Bu denetleyiciyi kullanarak bir RL " +"modelini eğitmedik. Gözlemlediğimiz başarısızlık kontrolün kendisindeydi. " +"Ancak bunu bir ödül olarak kullanırsak aynı kör nokta öğrenme ortamının bir " +"parçası haline gelecektir." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:639 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A functional test can have a similar blind spot. {ph0}Differential-fuzzing " +"research on LLM decompilers{ph1} has found behavior differences beyond the " +"supplied tests. Passing those tests therefore needs to be read in terms of " +"the cases they exercise, just as an instruction match needs to be read in " +"terms of the bytes and data it checks." +msgstr "" +"Fonksiyonel bir testin de benzer bir kör noktası olabilir. {ph0}LLM kod " +"çözücüler üzerinde diferansiyel bulanıklaştırmayı kullanan araştırma{ph1}, " +"sağlanan testlerin dışında davranışsal farklılıklar buldu. Bu nedenle geçme " +"testini, uyguladığı durumlar açısından anlamamız gerekir. Benzer şekilde, " +"bir talimat eşleşmesinin, doğruladığı baytlar ve veriler açısından " +"anlaşılması gerekir." + +#: website/public/blog/reverse-engineering-models/index.html:649 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before using either kind of check as a reward, I would test it with " +"deliberate failures. I would also keep final acceptance outside the " +"candidate's editable source, so the model has to satisfy the check through " +"its reconstruction. Otherwise, something like a byte array copied from the " +"target could satisfy a naive score without recovering useful source." +msgstr "" +"Her iki doğrulama kontrolünü de ödül olarak kullanmadan önce, bunu kasıtlı " +"olarak yanlış sonuçlarla test ederim. Ayrıca nihai kabulü, yapay zeka " +"aracısının düzenleyebileceği kaynak kodundan ayrı tutacağım. Model, programı" +" doğru bir şekilde yeniden oluşturarak doğrulamayı geçmelidir. Aksi " +"takdirde, hedeften bir bayt dizisinin basitçe kopyalanması, yararlı kaynak " +"kodunu kurtarmadan saf bir puanlama sistemini kandırabilir." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:657 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Improving the learner and improving its oracle belong to the same " +"research problem.{ph1} As the learner gets better at optimizing the feedback" +" we give it, understanding that feedback becomes more important too." +msgstr "" +"{ph0}Öğreneni geliştirmek ve doğrulama denetleyicisini geliştirmek aynı " +"araştırma probleminin bir parçasıdır.{ph1} Öğrenci, ona verdiğimiz geri " +"bildirimi optimize etme konusunda daha iyi hale geldikçe, geri bildirimin " +"anlaşılması da daha önemli hale gelir." + +#: website/public/blog/reverse-engineering-models/index.html:667 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A comparison worth running" +msgstr "Çalıştırılmaya değer bir karşılaştırma" + +#: website/public/blog/reverse-engineering-models/index.html:668 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"With those checks in place, I would start the comparison on a fixed VC7.1 " +"task family. First, I would establish baselines for a strong general model " +"and a smaller one. Adding repository knowledge to the smaller model would " +"show what retrieval changes. From there, supervised specialization and an " +"RL-trained version would let us examine what training adds." +msgstr "" +"Bu kontroller uygulandığında, karşılaştırmaya sabit bir VC7.1 görev ailesi " +"üzerinde başlayacağım. İlk olarak, güçlü bir genel amaçlı yapay zeka modeli " +"ve daha küçük bir model için temeller oluşturacaktım. Daha küçük modele depo" +" bilgisinin eklenmesi, geri almanın ne gibi değişikliklere yol açtığını " +"gösterecektir. Buradan itibaren, denetimli uzmanlık ve RL ile eğitilmiş bir " +"sürüm, eğitimin neleri eklediğini incelememize olanak tanır." + +#: website/public/blog/reverse-engineering-models/index.html:676 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To make those comparisons informative, each model would receive the same " +"target evidence and tools. For the training-specific comparisons, I would " +"keep the base model and available knowledge fixed as well. Repeated trials " +"would help distinguish a reliable search strategy from a lucky candidate." +msgstr "" +"Bu karşılaştırmaları bilgilendirici kılmak için her model aynı hedef " +"kanıtları ve araçları alacaktır. Eğitime özel karşılaştırmalar için temel " +"modeli ve mevcut bilgiyi de sabit tutardım. Tekrarlanan denemeler, güvenilir" +" bir arama stratejisini şans eseri başarılı olan aday bir çözümden ayırmaya " +"yardımcı olacaktır." + +#: website/public/blog/reverse-engineering-models/index.html:683 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would then separate two starting conditions: reconstructing a function " +"from its instructions, and closing the last few differences in maintained " +"source. A specialist might be excellent at the second task while a general " +"model does much of the work needed to reach it. Measuring them separately " +"would reveal that division of work." +msgstr "" +"Daha sonra iki başlangıç koşulunu ayıracağım: bir fonksiyonu talimatlarından" +" yeniden oluşturmak ve korunan kaynaktaki son birkaç farklılığı kapatmak. " +"Özel bir yapay zeka modeli ikinci görevde mükemmel olabilirken, genel amaçlı" +" bir yapay zeka modeli, ona ulaşmak için gereken işin çoğunu yapar. Bunları " +"ayrı ayrı ölçmek o iş bölümünü ortaya çıkaracaktır." + +#: website/public/blog/reverse-engineering-models/index.html:691 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both tasks need genuinely held-out examples. Because related Touhou games " +"share engine code, holding out a game's name would still allow nearly " +"identical functions to appear in training. I would instead hold out source " +"families and keep their accepted answers out of both training and retrieval." +msgstr "" +"Her iki görevin de eğitim ve erişim dışında bırakılan test örneklerine " +"ihtiyacı vardır. İlgili Touhou oyunları motor kodunu paylaşır. Bir oyunu " +"eğitimden hariç tutmak, eğitim setinde neredeyse aynı işlevlerin kalmasına " +"neden olabilir. Bunun yerine, ilgili kaynak kodu gruplarının tamamını hariç " +"tutar ve doğrulanmış yanıtlarını hem eğitim verilerinin hem de erişim " +"kütüphanesinin dışında tutardım." + +#: website/public/blog/reverse-engineering-models/index.html:698 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"On those examples, the outcome would be complete, independently validated " +"matches. Giving each model the same candidate or compile budget would help " +"compare search quality. Measuring elapsed time and total cost, including " +"compilation, would tell us whether that quality produces an efficient " +"workflow. Training costs would need their own accounting too." +msgstr "" +"Bu örnekler için sonuç, eksiksiz, bağımsız olarak doğrulanmış eşleşmeler " +"olacaktır. Aday teklifleri veya derleme girişimleri için eşit bütçeler, " +"arama kalitesinin karşılaştırılmasına yardımcı olacaktır. Derleme de dahil " +"olmak üzere toplam süre ve maliyetin ölçülmesi, bu kalitenin verimli bir iş " +"akışı üretip üretmediğini gösterecektir. Eğitim maliyetleri ayrı olarak " +"ölçülmelidir." + +#: website/public/blog/reverse-engineering-models/index.html:706 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both views matter because a cheaper proposal can become an expensive " +"workflow if it causes more uninformative compiles. Conversely, a slower " +"model may pay for itself by finding a useful experiment sooner. The " +"comparison should capture those costs." +msgstr "" +"Her iki ölçüm de önemlidir. Ucuz bir teklif, hiçbir yararlı bilgi sağlamayan" +" daha fazla derlemeye neden olursa pahalı bir iş akışına yol açabilir. " +"Tersine, daha yavaş bir model, yararlı bir deneyi daha erken bulursa daha " +"ekonomik olabilir. Karşılaştırma bu maliyetleri içermelidir." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:712 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Counting accepted results also keeps intermediate progress in perspective. " +"One TH09 {ph0}replay-save candidate{ph1} looked promising because the " +"compiled function had the expected size, but a full comparison still found " +"substantial differences. Changes in how the compiler arranges instructions " +"can also make a candidate look closer without resolving the underlying " +"problem. Such diagnostics can guide the next experiment, but I would " +"validate any reward for \"getting closer\" against the full acceptance " +"check." +msgstr "" +"Kabul edilen sonuçların sayılması aynı zamanda ara ilerlemeyi de perspektif " +"içinde tutar. Bir TH09 {ph0}tekrar oynatma-kaydetme adayı{ph1}, derlenmiş " +"işlevin beklenen boyuta sahip olması nedeniyle umut verici görünüyordu, " +"ancak tam bir karşılaştırma yine de önemli farklılıklar buldu. Derleyicinin " +"talimatları düzenleme biçimindeki değişiklikler, altta yatan sorunu çözmeden" +" adayın daha yakından bakmasını da sağlayabilir. Bu tür teşhisler bir " +"sonraki deneye rehberlik edebilir, ancak \"yaklaşmanın\" herhangi bir " +"ödülünü tam kabul kontrolüne göre doğrularım." + +#: website/public/blog/reverse-engineering-models/index.html:727 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would be interested in the pattern across these results. If a specialist " +"does well within a familiar compiler family, does that advantage survive a " +"toolchain change? And once both models have good retrieval, how much does " +"training still add? The answers would help us decide where each kind of " +"model is most useful." +msgstr "" +"Bu sonuçlardaki model ilgimi çeker. Özel bir yapay zeka modeli tanıdık bir " +"derleyici ailesinde başarılı olursa, bu avantaj bir takım zinciri " +"değişikliğinden sonra da ayakta kalabilir mi? Her iki yapay zeka modeli de " +"iyi bir erişime sahip olduğunda, eğitimin hâlâ ne kadar katkısı olur? " +"Cevaplar, her model türünün en yararlı olduğu yerin neresi olduğuna karar " +"vermemize yardımcı olacaktır." + +#: website/public/blog/reverse-engineering-models/index.html:736 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What kind of agent does this suggest?" +msgstr "Bu ne tür bir yapay zeka ajanını akla getiriyor?" + +#: website/public/blog/reverse-engineering-models/index.html:737 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"If the models turn out to help at different stages, a useful arrangement " +"might be a broad agent with access to a specialist. Once a task becomes a " +"familiar compiler search, it could ask the specialist for candidates. If the" +" results stop making sense, the broader agent could return to the target " +"evidence or investigate the measurement itself." +msgstr "" +"Modellerin farklı aşamalarda yardımcı olduğu ortaya çıkarsa, özel bir yapay " +"zeka modeline erişimi olan geniş bir temsilci yararlı bir düzenleme " +"olabilir. Bir görev tanıdık bir derleyici araması haline geldiğinde, adaylar" +" için özel yapay zeka modelini isteyebilir. Sonuçların artık bir anlamı " +"kalmazsa, daha geniş kapsamlı temsilci hedef kanıtlara geri dönebilir veya " +"ölçümün kendisini araştırabilir." + +#: website/public/blog/reverse-engineering-models/index.html:745 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In that arrangement, specialization becomes a tool the agent can choose as " +"the investigation develops. The agent keeps the freedom to change direction " +"that made the earlier breakthroughs possible." +msgstr "" +"Bu düzenlemede uzmanlaşma, soruşturma geliştikçe temsilcinin seçebileceği " +"bir araç haline gelir. Temsilci, daha önceki atılımları mümkün kılan yön " +"değiştirme özgürlüğünü elinde tutar." + +#: website/public/blog/reverse-engineering-models/index.html:750 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This follows naturally from the industrial shift I described in the previous" +" essay. Being able to run many more experiments changes how we organize the " +"work. It also gives us a reason to ask which decisions need broad judgment " +"and which improve through repeated experience with a particular craft." +msgstr "" +"Bu, doğal olarak önceki makalede anlattığım endüstriyel değişimin sonucudur." +" Daha fazla deney yürütebilmek, işi organize etme şeklimizi değiştirir. Aynı" +" zamanda bize, hangi kararların geniş muhakeme gerektirdiğini ve " +"hangilerinin belirli bir zanaatta tekrarlanan deneyimlerle geliştirildiğini " +"sormamız için bir neden de verir." + +#: website/public/blog/reverse-engineering-models/index.html:757 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For now, the strongest general model has earned its place in my workflow by " +"opening problems I had left unresolved. Those successes are what make me " +"want to examine the work more closely: how much depended on the model's " +"breadth, and how much could be handled by a model trained in this particular" +" kind of reconstruction?" +msgstr "" +"Şimdilik en güçlü genel model, daha önce çözemediğim sorunları çözerek iş " +"akışımda yerini kazandı. Bu başarılar süreci daha yakından incelemek " +"istememe yol açıyor: Ne kadarı modelin geniş yeteneklerine bağlıydı? Ne " +"kadarı bu yeniden yapılandırma türüne özel olarak eğitilmiş bir model " +"tarafından yapılabilirdi?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:765 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}The repositories give us a way to test the question.{ph1} I would like " +"to use that record to build the comparison and see what we learn." +msgstr "" +"{ph0}Depolar bize soruyu test etmemiz için bir yol sağlıyor.{ph1} Bu kaydı " +"karşılaştırmayı oluşturmak ve ne öğrendiğimizi görmek için kullanmak " +"istiyorum." + +#: website/public/blog/reverse-engineering-models/index.html:775 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Evidence and model attribution" +msgstr "Kanıt ve model ilişkilendirme" + +#: website/public/blog/reverse-engineering-models/index.html:783 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The examples describe historical function-level comparisons recorded in the " +"linked commits. Whole-game runtime checks are separate milestones. Source " +"and matching records were reviewed on 13 October 2026." +msgstr "" +"Örnekler, bağlantılı taahhütlerde kaydedilen geçmiş işlev düzeyi " +"karşılaştırmalarını açıklamaktadır. Tüm oyun çalışma zamanı kontrolleri ayrı" +" kilometre taşlarıdır. Kaynak ve eşleşen kayıtlar 13 Ekim 2026'da incelendi." + +#. {ph0}: gpt-dots +#. {ph1}: gpt-web +#: website/public/blog/reverse-engineering-models/index.html:789 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The TH09 label {ph0} identifies GPT-6-Astra. {ph1} changed with the model " +"used at the time, so that label alone does not identify an individual run. " +"Explicit model names in the other cited commits establish their recorded " +"attribution; the histories do not contain uniform per-run cost or effort " +"measurements." +msgstr "" +"TH09 etiketi {ph0}, GPT-6-Astra anlamına gelir. {ph1} ile ilişkilendirilen " +"model zaman içinde değiştiği için etiket tek başına belirli bir çalıştırmada" +" kullanılan modeli tanımlayamaz. Belirtilen diğer taahhütlerdeki açık model " +"adları, kaydedilen atıfları sağlar. Bu geçmişler, her çalıştırma için " +"tutarlı maliyet veya çaba ölçümlerini içermez." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#: website/public/blog/reverse-engineering-models/index.html:798 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 temporary lifetimes:{ph1} the {ph2}earlier context review{ph3} and" +" {ph4}complete cold reconstruction{ph5}." +msgstr "" +"{ph0}TH09 geçici nesne yaşam süreleri:{ph1} {ph2}önceki bağlam " +"incelemesi{ph3} ve {ph4}kaynaktan temiz bir yeniden derlemeyle doğrulanan " +"tam yeniden yapılandırma{ph5}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:810 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 Front callback:{ph1} {ph2}non-exact diagnostic regions{ph3}, " +"{ph4}Astra's new graph{ph5} and its {ph6}accepted recompile replay{ph7}." +msgstr "" +"{ph0}TH09 Front geri arama:{ph1} {ph2}eşleşmeyen tanılama bölgeleri{ph3}, " +"{ph4}Astra'nin yeni kontrol akışı grafiği{ph5} ve onun {ph6}kabul edilen " +"yeniden derleme kontrolü{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:826 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Compiler knowledge:{ph1} {ph2}TH095 identifier-bucket reuse{ph3}, " +"{ph4}TH04 player source and linker order{ph5} and {ph6}Luna's TH09 enemy-" +"spawn closure{ph7}." +msgstr "" +"{ph0}Derleyici bilgisi:{ph1} {ph2}TH095 tanımlayıcı grubu davranışının " +"yeniden kullanımı{ph3}, {ph4}TH04 oyuncu kaynak kodu ve bağlayıcı " +"sırası{ph5} ve {ph6}Luna'nin TH09'nin düşman oluşturma işlevini tamamen " +"yeniden yapılandırması{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:842 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Checks and diagnostics:{ph1} {ph2}TH08 floating-literal audit{ph3}, " +"{ph4}TH08 numerical investigation{ph5} and {ph6}TH09 replay-save " +"reconstruction{ph7}." +msgstr "" +"{ph0}Kontroller ve teşhisler:{ph1} {ph2}TH08 kayan nokta sabiti " +"denetimi{ph3}, {ph4}TH08 sayısal inceleme{ph5} ve {ph6}TH09 yeniden " +"oynatmayı kaydeden yeniden yapılandırma{ph7}." + +#: website/public/blog/reverse-engineering-models/index.html:859 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The proposed specialist and RL comparisons are future experiments. The " +"project histories motivate them; no trained specialist was evaluated for " +"this essay." +msgstr "" +"Özel modellerin ve RL eğitiminin önerilen karşılaştırmaları gelecekteki " +"deneylerdir. Proje geçmişleri onları motive ediyor. Bu makale için hiçbir " +"eğitimli özel model değerlendirilmemiştir." + +#: website/public/blog/reverse-engineering-models/index.html:6 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model? | REA" +msgstr "Tersine mühendisliğin daha büyük bir modele ihtiyacı var mı? | REA" + +#: website/public/blog/reverse-engineering-models/index.html:7 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What Touhou reconstruction taught me about general models, specialist models" +" and learning to propose better hypotheses from compiler feedback." +msgstr "" +"Touhou yeniden yapılandırma bana genel modeller, uzman modeller ve derleyici" +" geri bildirimlerinden daha iyi hipotezler önermeyi öğrenme hakkında neler " +"öğretti?" + +#: website/public/blog/reverse-engineering-models/index.html:240 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "On this page" +msgstr "Bu sayfada" + +#: website/public/blog/reverse-engineering-models/index.html:253 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Models and reverse engineering essay" +msgstr "Modeller ve tersine mühendislik makalesi" + +#: website/public/blog/reverse-engineering-models/index.html:491 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Selected reconstruction experiments" +msgstr "Seçilmiş yeniden yapılanma deneyleri" + +#: website/public/blog/reverse-engineering-models/index.html:883 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Back to top" +msgstr "Başa dön" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:776 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Detailed evidence notes{ph1} include the recorded comparisons and " +"experiment context." +msgstr "" +"{ph0}Ayrıntılı kanıt notları{ph1}, kaydedilen karşılaştırmaları ve " +"deneylerin bağlamını içerir." + +#~ msgctxt "page:blog/index.html" +#~ msgid "By N0zoM1z0 · 13 October 2026 · Draft for review" +#~ msgstr "N0zoM1z0 · 13 Ekim 2026 · İnceleme taslağı" + #~ msgctxt "page:index.html" #~ msgid "Setup guide" #~ msgstr "Kurulum rehberi" diff --git a/website/locales/uk/messages.po b/website/locales/uk/messages.po index ddd250d24..55afe918d 100644 --- a/website/locales/uk/messages.po +++ b/website/locales/uk/messages.po @@ -32,18 +32,18 @@ msgid "Thoughts on reverse engineering." msgstr "Думки про реверс-інжиніринг." #. {ph0}:
-#: website/public/blog/index.html:200 +#: website/public/blog/index.html:214 #, python-brace-format msgctxt "page:blog/index.html" msgid "Game reconstruction{ph0}Essay" msgstr "Реконструкція гри{ph0}Вигадування" -#: website/public/blog/index.html:202 +#: website/public/blog/index.html:216 msgctxt "page:blog/index.html" msgid "The industrial revolution in reverse engineering" msgstr "Промислова революція в області реверс-інжинірингу" -#: website/public/blog/index.html:203 +#: website/public/blog/index.html:217 msgctxt "page:blog/index.html" msgid "" "What manual RE and reconstructing TH08, TH095 and TH04 taught me about " @@ -52,25 +52,26 @@ msgstr "" "Чому мене навчив посібник з оновлення та реконструкції TH08, TH095 та TH04 " "про агентів, перевірку та знання, які використовуються в різних проектах." -#: website/public/blog/index.html:208 +#: website/public/blog/index.html:207 website/public/blog/index.html:222 msgctxt "page:blog/index.html" msgid "Read the article" msgstr "Прочитайте статтю" -#: website/public/blog/index.html:211 +#: website/public/blog/index.html:225 msgctxt "page:blog/index.html" msgid "By N0zoM1z0 · 10 October 2026" msgstr "Автор: N0zoM1z0 · 10 жовтня 2026" #. {ph0}: -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 +#: website/public/blog/reverse-engineering-models/index.html:883 #: website/public/blog/touhou-reconstruction/index.html:1080 #: website/public/examples/dino-lab/index.html:475 #: website/public/faq/index.html:573 #: website/public/first-investigation/index.html:523 #: website/public/get-started/index.html:882 -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 #: website/public/guides/index.html:282 #: website/public/guides/javascript/index.html:503 #: website/public/guides/native/index.html:509 website/public/index.html:1191 @@ -100,6 +101,8 @@ msgstr "" " і агентами штучного інтелекту." #: website/public/blog/index.html:75 website/public/blog/index.html:88 +#: website/public/blog/reverse-engineering-models/index.html:75 +#: website/public/blog/reverse-engineering-models/index.html:88 #: website/public/blog/touhou-reconstruction/index.html:75 #: website/public/blog/touhou-reconstruction/index.html:88 #: website/public/examples/dino-lab/index.html:75 @@ -136,6 +139,7 @@ msgstr "" "допомогою вашого програміста." #: website/public/blog/index.html:108 +#: website/public/blog/reverse-engineering-models/index.html:108 #: website/public/blog/touhou-reconstruction/index.html:108 #: website/public/examples/dino-lab/index.html:109 #: website/public/faq/index.html:108 @@ -156,6 +160,7 @@ msgid "REA home" msgstr "Головна REA" #: website/public/blog/index.html:112 +#: website/public/blog/reverse-engineering-models/index.html:112 #: website/public/blog/touhou-reconstruction/index.html:112 #: website/public/examples/dino-lab/index.html:113 #: website/public/faq/index.html:112 @@ -176,6 +181,7 @@ msgid "Main navigation" msgstr "Основна навігація" #: website/public/blog/index.html:130 +#: website/public/blog/reverse-engineering-models/index.html:130 #: website/public/blog/touhou-reconstruction/index.html:130 #: website/public/examples/dino-lab/index.html:131 #: website/public/faq/index.html:130 @@ -196,6 +202,7 @@ msgid "REA on GitHub" msgstr "REA на GitHub" #: website/public/blog/index.html:150 +#: website/public/blog/reverse-engineering-models/index.html:150 #: website/public/blog/touhou-reconstruction/index.html:150 #: website/public/examples/dino-lab/index.html:151 #: website/public/faq/index.html:150 @@ -216,6 +223,7 @@ msgid "Join REA on Discord" msgstr "Приєднатися до REA в Discord" #: website/public/blog/index.html:169 +#: website/public/blog/reverse-engineering-models/index.html:169 #: website/public/blog/touhou-reconstruction/index.html:169 #: website/public/examples/dino-lab/index.html:170 #: website/public/faq/index.html:169 @@ -235,7 +243,7 @@ msgctxt "shared" msgid "Follow REA on X" msgstr "Стежити за REA в X" -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 msgctxt "page:blog/index.html" msgid "Back to top" msgstr "Нагору" @@ -4780,7 +4788,7 @@ msgctxt "page:guides/browser/index.html" msgid "Copy read the loaded scripts" msgstr "Копіювати читати завантажені скрипти" -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 msgctxt "page:guides/browser/index.html" msgid "Back to top" msgstr "Нагору" @@ -11684,6 +11692,7 @@ msgid "Same spacing: 256 ÷ 16 = 16 angle units = 22.5°" msgstr "Однакова відстань: 256 ÷ 16 = 16 кутових одиниць = 22,5°" #: website/public/blog/index.html:114 +#: website/public/blog/reverse-engineering-models/index.html:114 #: website/public/blog/touhou-reconstruction/index.html:114 #: website/public/examples/dino-lab/index.html:115 #: website/public/faq/index.html:114 @@ -11704,6 +11713,7 @@ msgid "Showcases" msgstr "Приклади" #: website/public/blog/index.html:115 +#: website/public/blog/reverse-engineering-models/index.html:115 #: website/public/blog/touhou-reconstruction/index.html:115 #: website/public/examples/dino-lab/index.html:116 #: website/public/faq/index.html:115 @@ -11724,6 +11734,7 @@ msgid "Blog" msgstr "Блог" #: website/public/blog/index.html:113 +#: website/public/blog/reverse-engineering-models/index.html:113 #: website/public/blog/touhou-reconstruction/index.html:113 #: website/public/examples/dino-lab/index.html:114 #: website/public/faq/index.html:113 @@ -11744,6 +11755,7 @@ msgid "Get started" msgstr "Почати" #: website/public/blog/index.html:116 +#: website/public/blog/reverse-engineering-models/index.html:116 #: website/public/blog/touhou-reconstruction/index.html:116 #: website/public/examples/dino-lab/index.html:117 #: website/public/faq/index.html:116 @@ -11763,13 +11775,14 @@ msgctxt "shared" msgid "Guides" msgstr "Посібники" -#: website/public/blog/index.html:222 +#: website/public/blog/index.html:236 +#: website/public/blog/reverse-engineering-models/index.html:875 #: website/public/blog/touhou-reconstruction/index.html:1072 #: website/public/examples/dino-lab/index.html:467 #: website/public/faq/index.html:565 #: website/public/first-investigation/index.html:515 #: website/public/get-started/index.html:874 -#: website/public/guides/browser/index.html:490 +#: website/public/guides/browser/index.html:499 #: website/public/guides/index.html:274 #: website/public/guides/javascript/index.html:495 #: website/public/guides/native/index.html:501 website/public/index.html:1183 @@ -11783,13 +11796,14 @@ msgctxt "shared" msgid "Source" msgstr "Вихідний код" -#: website/public/blog/index.html:223 +#: website/public/blog/index.html:237 +#: website/public/blog/reverse-engineering-models/index.html:876 #: website/public/blog/touhou-reconstruction/index.html:1073 #: website/public/examples/dino-lab/index.html:468 #: website/public/faq/index.html:566 #: website/public/first-investigation/index.html:516 #: website/public/get-started/index.html:875 -#: website/public/guides/browser/index.html:491 +#: website/public/guides/browser/index.html:500 #: website/public/guides/index.html:275 #: website/public/guides/javascript/index.html:496 #: website/public/guides/native/index.html:502 website/public/index.html:1184 @@ -11803,13 +11817,14 @@ msgctxt "shared" msgid "MIT license" msgstr "Ліцензія MIT" -#: website/public/blog/index.html:224 +#: website/public/blog/index.html:238 +#: website/public/blog/reverse-engineering-models/index.html:877 #: website/public/blog/touhou-reconstruction/index.html:1074 #: website/public/examples/dino-lab/index.html:469 #: website/public/faq/index.html:567 #: website/public/first-investigation/index.html:517 #: website/public/get-started/index.html:876 -#: website/public/guides/browser/index.html:492 +#: website/public/guides/browser/index.html:501 #: website/public/guides/index.html:276 #: website/public/guides/javascript/index.html:497 #: website/public/guides/native/index.html:503 website/public/index.html:1185 @@ -11823,13 +11838,14 @@ msgctxt "shared" msgid "Join the community" msgstr "Приєднатися до спільноти" -#: website/public/blog/index.html:225 +#: website/public/blog/index.html:239 +#: website/public/blog/reverse-engineering-models/index.html:878 #: website/public/blog/touhou-reconstruction/index.html:1075 #: website/public/examples/dino-lab/index.html:470 #: website/public/faq/index.html:568 #: website/public/first-investigation/index.html:518 #: website/public/get-started/index.html:877 -#: website/public/guides/browser/index.html:493 +#: website/public/guides/browser/index.html:502 #: website/public/guides/index.html:277 #: website/public/guides/javascript/index.html:498 #: website/public/guides/native/index.html:504 website/public/index.html:1186 @@ -12773,6 +12789,1390 @@ msgctxt "runtime" msgid "Switch to dark mode" msgstr "Перемкнути на темну тему" +#. {ph0}: +#. {ph1}: +#: website/public/guides/browser/index.html:480 +#, python-brace-format +msgctxt "page:guides/browser/index.html" +msgid "" +"For HTTP request and response bodies, use {ph0}Fiddler Everywhere: SAZ " +"captures, MCP and public skills{ph1}. Export SAZ or HAR 1.2 for offline REA " +"capture Evidence, or connect Everywhere's MCP server to inspect captured " +"traffic with your agent." +msgstr "" +"Для дослідження тіл запитів і відповідей HTTP використовуйте {ph0}Fiddler " +"Everywhere: захоплення SAZ, MCP і загальнодоступні навички{ph1}. Експортуйте" +" SAZ ??? HAR 1.2 для отримання Evidence захоплення в REA без підключення до " +"мережі або підключіть MCP-сервер Everywhere, щоб досліджувати записаний " +"трафік разом з агентом." + +#. {ph0}:
+#: website/public/blog/index.html:200 +#, python-brace-format +msgctxt "page:blog/index.html" +msgid "Model specialization{ph0}Essay" +msgstr "Спеціалізація моделей{ph0}Есе" + +#: website/public/blog/index.html:202 +msgctxt "page:blog/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "Чи потрібна для зворотної розробки більша модель?" + +#: website/public/blog/index.html:203 +msgctxt "page:blog/index.html" +msgid "" +"What Touhou reconstruction suggests about general models, specialist models " +"and learning from compiler feedback." +msgstr "" +"Що реконструкція Touhou допомагає зрозуміти про універсальні й " +"спеціалізовані моделі та навчання на зворотному зв’язку від компілятора." + +#: website/public/blog/index.html:210 +msgctxt "page:blog/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Автор N0zoM1z0 · 13 жовтня 2026 р" + +#: website/public/blog/reverse-engineering-models/index.html:103 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Skip to content" +msgstr "Перейти до змісту" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:194 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Home{ph1} / {ph2}Blog{ph3} / Models and RE" +msgstr "" +"{ph0}Домашня сторінка{ph1} / {ph2}Блог{ph3} / Моделі та зворотне " +"проектування" + +#: website/public/blog/reverse-engineering-models/index.html:197 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Essay" +msgstr "Есе" + +# Reviewed for technical meaning and readability; preserve the distinction +# between historical evidence and proposed experiments. +#: website/public/blog/reverse-engineering-models/index.html:198 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "Чи потрібна для зворотного проектування більша модель?" + +#: website/public/blog/reverse-engineering-models/index.html:199 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Автор N0zoM1z0 · 13 жовтня 2026 р" + +#: website/public/blog/reverse-engineering-models/index.html:201 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Takeaway" +msgstr "Головне" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:203 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Model capability changes the hypotheses an agent can find.{ph1} Later " +"sessions have solved problems that earlier work left unresolved." +msgstr "" +"{ph0}Можливості моделі впливають на те, які гіпотези може запропонувати " +"агент штучного інтелекту.{ph1} Пізніші сесії вирішували проблеми, які " +"попередня робота залишила невирішеними." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:211 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A familiar compiler gives specialization something concrete to " +"learn.{ph1} Some exact matches depend on narrow, reusable source patterns." +msgstr "" +"{ph0}Знайомий компілятор пропонує конкретні шаблони для вивчення " +"спеціалізованої моделі.{ph1} Деякі точні збіги машинного коду залежать від " +"конкретних шаблонів вихідного коду, які можна багаторазово використовувати." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:218 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Give repository knowledge a fair comparison.{ph1} A smaller model with " +"good retrieval may already know enough to make progress." +msgstr "" +"{ph0}Включіть знання репозиторію в справедливе порівняння.{ph1} Менша модель" +" із хорошим пошуком знань може вже мати достатньо інформації для прогресу." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:223 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Train the next useful experiment, and test its oracle.{ph1} The value " +"of feedback depends on what the checker actually verifies." +msgstr "" +"{ph0}Навчіть модель вибирати наступний корисний експеримент і протестуйте її" +" засіб перевірки.{ph1} Зворотній зв’язок корисний лише в тій мірі, в якій " +"засіб перевірки перевіряє правильні речі." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:230 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Measure accepted reconstruction against time and cost.{ph1} That is the" +" result I want from the whole agent." +msgstr "" +"{ph0}Оцінка перевірених результатів реконструкції разом із часом і " +"вартістю.{ph1} Це те, що я хочу виміряти для повного робочого процесу AI-" +"agent." + +#: website/public/blog/reverse-engineering-models/index.html:241 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "In this article" +msgstr "У цій статті" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:243 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What better models changed{ph1}" +msgstr "{ph0}Що змінили сильніші моделі{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:244 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Why specialize?{ph1}" +msgstr "{ph0}Навіщо спеціалізуватися?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:245 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Where experience lives{ph1}" +msgstr "{ph0}Де зберігати цей досвід?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:246 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Learning the next experiment{ph1}" +msgstr "{ph0}Вивчення наступного корисного експерименту{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:247 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}RE and reinforcement learning{ph1}" +msgstr "{ph0}Зворотне проектування та навчання з посиленням{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:248 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}When the oracle is wrong{ph1}" +msgstr "{ph0}Коли засіб перевірки помиляється{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:249 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}A comparison worth running{ph1}" +msgstr "{ph0}Варте порівняння{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:250 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What kind of agent?{ph1}" +msgstr "{ph0}Що це за агент ШІ?{ph1}" + +#: website/public/blog/reverse-engineering-models/index.html:258 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What better models changed" +msgstr "Що змінили сильніші моделі" + +#: website/public/blog/reverse-engineering-models/index.html:259 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Since August 2026, I have been exploring how much reverse engineering an " +"agent can carry out with access to a program and a way to check its work. " +"Most of my experience comes from reconstructing games, especially Touhou. As" +" I have changed models, the repositories have kept a record of what each " +"session managed to recover." +msgstr "" +"З серпня 2026 року я досліджував, який обсяг зворотного проектування може " +"виконати агент, маючи доступ до програми та спосіб перевірки її роботи. " +"Більшість мого досвіду отримано від реконструкції ігор, особливо Touhou. " +"Оскільки я змінював моделі, репозиторії зберігали записи про те, що вдалося " +"відновити за кожну сесію." + +#: website/public/blog/reverse-engineering-models/index.html:267 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of that work aims at exact reconstruction: we compile the maintained " +"source under a pinned historical toolchain and check whether it reproduces " +"the target function. When a later model succeeds where earlier work stalled," +" the source changes show what made that progress possible." +msgstr "" +"Значна частина цієї роботи спрямована на точну реконструкцію: ми компілюємо " +"підтримуваний джерело в ланцюжку історичних інструментів із фіксованими " +"версіями та перевіряємо, чи він відтворює цільову функцію. Коли пізніша " +"модель досягає успіху там, де попередня робота зупинилася, зміни джерела " +"показують, що зробило цей прогрес можливим." + +#: website/public/blog/reverse-engineering-models/index.html:274 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The temporary values had to stay alive together" +msgstr "Тимчасові змінні потребували часу існування, що збігався" + +#: website/public/blog/reverse-engineering-models/index.html:275 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of a TH09 attack function's logic had already been reconstructed, but " +"the compiler still stored temporary results differently from the original. " +"The remaining problem was how long those results needed to stay alive and " +"when their storage could be reused." +msgstr "" +"Значна частина логіки функції атаки TH09 вже була реконструйована, але " +"компілятор усе ще зберігав тимчасові результати, відмінні від оригіналу. " +"Проблема, що залишилася, полягала в тому, як довго ці результати залишалися " +"живими та коли їхнє сховище можна було використовувати повторно." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:282 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A {ph0}GPT-6-Sol checkpoint{ph1} recorded this mismatch after testing " +"several ordinary source arrangements. When {ph2}GPT-6.1-Sol returned to the " +"problem{ph3}, it found a useful hypothesis about how long those temporary " +"values needed to stay alive." +msgstr "" +"{ph0}GPT-6-Sol запис сеансу{ph1} повідомив про цю невідповідність після " +"тестування кількох звичайних механізмів вихідного коду. Коли " +"{ph2}GPT-6.1-Sol знову дослідив проблему{ph3}, він запропонував корисну " +"гіпотезу щодо тривалості життя цих тимчасових змінних." + +#: website/public/blog/reverse-engineering-models/index.html:296 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The function produced temporary results and copied them into animation " +"positions. In C++, a temporary result stays alive until the expression using" +" it finishes. Grouping the real operations into one expression kept those " +"results alive together, so the compiler could no longer reuse the same " +"storage for all of them." +msgstr "" +"Функція створила тимчасові результати та скопіювала їх у позиції анімації. У" +" C++ тимчасовий результат залишається активним до завершення виразу, який " +"його використовує. Групування реальних операцій в один вираз зберігало ці " +"результати разом, тому компілятор більше не міг повторно використовувати те " +"саме сховище для всіх них." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:303 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"One result still shared storage with a value used later for collision " +"checks. Giving that value its own lifetime resolved the remaining overlap. " +"After several targeted experiments, {ph0}a clean rebuild confirmed that the " +"full function matched{ph1}." +msgstr "" +"Один результат все ще ділився пам’яттю зі значенням, яке використовується " +"пізніше для перевірок на конфлікти. Надання цьому значенню власного часу " +"життя вирішило решту накладання. Після кількох цілеспрямованих експериментів" +" {ph0}чиста повторна компіляція з джерела підтвердила, що повна функція " +"відповідає{ph1}." + +#: website/public/blog/reverse-engineering-models/index.html:312 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What interests me is how the discovery and the tests worked together. The " +"lifetime hypothesis gave the agent a new direction; compiler feedback helped" +" it refine that hypothesis into a source arrangement that matched." +msgstr "" +"Мені цікаво, як відкриття та випробування підтримували одне одного. Гіпотеза" +" про тривалість життя тимчасових об’єктів дала агенту ШІ новий напрямок. " +"Відгуки компілятора допомогли уточнити цю гіпотезу у вихідному коді, який " +"відповідав." + +#: website/public/blog/reverse-engineering-models/index.html:318 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A different control-flow graph" +msgstr "Інший графік потоку керування" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:319 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A second TH09 function shows why that new direction can matter. Here, the " +"obstacle was how several branches shared their final work. " +"{ph0}GPT-6.1-Sol's earlier work{ph1} got parts of the function to match, but" +" the branches still joined up differently from the original." +msgstr "" +"Друга функція TH09 показує, чому цей новий напрямок може мати значення. Тут " +"перешкодою було те, що кілька філій поділилися своєю остаточною роботою. " +"Попередня робота {ph0}GPT-6.1-Sol{ph1} мала відповідні частини функції, але " +"гілки все одно з’єднувалися інакше, ніж у оригіналі." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:330 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}GPT-6-Astra changed the surrounding control-flow graph{ph1}. The main " +"paths now led to a shared final update and return. In that source structure," +" the compiler combined the branches in the same way as the original. The " +"complete function matched, and a {ph2}separate rebuild check confirmed " +"it{ph3}." +msgstr "" +"{ph0}GPT-6-Astra змінив навколишній графік потоку керування{ph1}. Основні " +"шляхи тепер вели до спільного остаточного оновлення та повернення. У цій " +"вихідній структурі компілятор об’єднав гілки так само, як і в оригіналі. " +"Повна функція відповідала, і {ph2}окрема перевірка відновлення підтвердила " +"це{ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:342 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Changing the representation opened a more productive search space.{ph1}" +" Both cases give me a reason to take general model capability seriously: an " +"agent can keep trying local variations for a long time before finding a " +"better way to express the surrounding work." +msgstr "" +"{ph0}Зміна подання відкрила більш продуктивний простір для пошуку.{ph1} " +"Обидва випадки дають мені підстави серйозно поставитися до загальної " +"можливості моделі: агент може довго пробувати місцеві варіації, перш ніж " +"знайти кращий спосіб вираження навколишньої роботи." + +#: website/public/blog/reverse-engineering-models/index.html:351 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"At the same time, the later sessions started with earlier source and " +"recorded failures. That accumulated work was part of their advantage. The " +"history lets us inspect the discoveries and their checks; a controlled " +"comparison would help separate the effect of the model from the experience " +"it inherited." +msgstr "" +"У той же час пізніші сеанси починалися з попередніх джерел і записаних " +"збоїв. Ця накопичена робота була частиною їхньої переваги. Історія дозволяє " +"нам ознайомитися з відкриттями та їх перевірками; контрольоване порівняння " +"допомогло б відокремити ефект моделі від досвіду, який вона успадкувала." + +#: website/public/blog/reverse-engineering-models/index.html:358 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is where the question becomes interesting to me. Some of the progress " +"came from finding a new representation, while much of the remaining work " +"involved testing source forms against a familiar compiler. How much needs " +"broad reasoning ability, and how much could a model learn through repeated " +"experience with that compiler and code?" +msgstr "" +"Ось що робить питання цікавим для мене. Певний прогрес став завдяки пошуку " +"нового представництва. Значна частина роботи, що залишилася, включала " +"тестування різних механізмів вихідного коду на знайомому компіляторі. " +"Наскільки для цього потрібна широка здатність міркувати? Скільки може " +"навчитися модель, постійно працюючи з цим компілятором і кодом?" + +#: website/public/blog/reverse-engineering-models/index.html:368 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why might a specialist do well?" +msgstr "Чому спеціалізована модель ШІ може працювати добре?" + +#: website/public/blog/reverse-engineering-models/index.html:369 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Touhou gives us a concrete reason to expect that experience to help. Much of" +" the game code comes from the same author, ZUN, so recurring habits are a " +"reasonable starting assumption. The repositories back that up with useful " +"examples: related loops and helper arrangements have supplied hypotheses " +"across games." +msgstr "" +"Touhou дає нам конкретну причину очікувати, що цей досвід допоможе. Значна " +"частина ігрового коду походить від того самого автора, ZUN, тому повторювані" +" звички є розумним вихідним припущенням. Репозиторії підтверджують це " +"корисними прикладами: пов’язані цикли та допоміжні механізми надають " +"гіпотези для різних ігор." + +#: website/public/blog/reverse-engineering-models/index.html:376 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those source patterns become more specific when we include the compiler. " +"Under a particular version and set of options, small changes in source can " +"produce repeatable differences in machine code. Some of those differences " +"are surprisingly obscure." +msgstr "" +"Ці вихідні шаблони стають більш конкретними, коли ми включаємо компілятор. " +"За певної версії та набору параметрів невеликі зміни у джерелі можуть " +"призвести до повторюваних відмінностей у машинному коді. Деякі з цих " +"відмінностей напрочуд незрозумілі." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: Supervisor::LoadDat +#. {ph4}: +#: website/public/blog/reverse-engineering-models/index.html:382 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In TH095, for example, {ph0}even the name used for a local variable could " +"change where VC7.1 stored it{ph1}. Moving the declaration alone did not " +"solve the layout, but an earlier background function had revealed a naming " +"pattern that affected the compiler's choice. That gave us a more promising " +"hypothesis for {ph2}the reconstruction of {ph3}{ph4}." +msgstr "" +"У TH095, наприклад, {ph0}навіть ім’я локальної змінної могло вплинути на те," +" де VC7.1 її зберігати{ph1}. Переміщення його оголошення не вирішило макета " +"сховища. Однак більш рання фонова функція виявила шаблон іменування, який " +"вплинув на вибір компілятора. Це запропонувало більш перспективну гіпотезу " +"для {ph2}реконструкції {ph3}{ph4}." + +#: website/public/blog/reverse-engineering-models/index.html:396 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reusing that pattern for the file-size variable produced the required layout" +" without adding storage. The full function then matched." +msgstr "" +"Повторне використання цього шаблону для змінної розміру файлу створило " +"необхідний макет без додавання пам’яті. Тоді повна функція збігалася." + +#: website/public/blog/reverse-engineering-models/index.html:401 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is the sort of narrow detail a general model may not know when it " +"arrives. A specialist trained on this compiler could encounter many such " +"cases and learn which source variations are worth testing first." +msgstr "" +"Це дуже конкретна деталь, яку модель ШІ загального призначення може не " +"знати, коли вперше стикається з проблемою. Спеціалізована модель для цього " +"компілятора може побачити багато таких випадків і дізнатися, які варіації " +"вихідного коду варто перевірити в першу чергу." + +#: website/public/blog/reverse-engineering-models/index.html:407 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The opportunity is especially clear in final matching problems where we " +"already understand the behavior. We are then asking which ordinary source " +"form this compiler turns into the observed instructions. Specialization " +"could help the model propose those forms more efficiently, while the target " +"still determines which hypotheses survive." +msgstr "" +"Можливість особливо чітка в остаточних задачах зіставлення, де ми вже " +"розуміємо поведінку. Тоді ми запитуємо, яку звичайну структуру вихідного " +"коду цей компілятор перетворює на спостережувані інструкції. Спеціалізація " +"може допомогти моделі запропонувати ці форми ефективніше, тоді як ціль все " +"ще визначає, які гіпотези виживуть." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:415 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The useful patterns would have to retain their context, though. TH08 uses an" +" earlier Visual C++ version than TH095 and TH09. TH04 uses a different DOS-" +"era compiler, and its {ph0}player reconstruction{ph1} needed different " +"hypotheses about source patterns and how the compiled code was combined into" +" an executable. Shared authorship gives us a prior, but each target tells us" +" whether it applies. Distinguishing those contexts is part of what a " +"specialist would need to learn." +msgstr "" +"Корисні шаблони повинні зберігати свій контекст. TH08 використовує попередню" +" версію Visual C++, ніж TH095 і TH09. TH04 використовує компілятор, " +"відмінний від епохи DOS. Його {ph0}реконструкція коду гравця{ph1} " +"потребувала різних гіпотез щодо шаблонів вихідного коду та того, як " +"скомпільований код зв’язувався у виконуваний файл. Наявність того самого " +"автора дає нам початкове очікування, але кожна ціль говорить нам, чи вона " +"застосовна. Спеціалізована модель повинна розрізняти ці контексти." + +#: website/public/blog/reverse-engineering-models/index.html:431 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Where should that experience live?" +msgstr "Де зберігати цей досвід?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:432 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before deciding to teach those patterns through model training, I would look" +" at the knowledge we already keep in the repository. In the {ph0}previous " +"essay{ph1}, I wrote about it becoming the project's working memory. That " +"memory includes compiler lessons, along with the source and failed " +"experiments that made them useful." +msgstr "" +"Перш ніж вирішувати навчати цим шаблонам за допомогою моделювання, я б " +"подивився на знання, які ми вже зберігаємо в репозиторії. У " +"{ph0}попередньому есе{ph1} я писав про те, що це стало робочою пам’яттю " +"проекту. Ця пам’ять містить уроки компілятора, а також джерело та невдалі " +"експерименти, які зробили їх корисними." + +#: website/public/blog/reverse-engineering-models/index.html:441 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"By retrieving that record, an agent can start with a known result and " +"inspect how we reached it. It can also avoid alternatives that have already " +"failed in the same context. The starting point improves without changing the" +" model's weights." +msgstr "" +"Отримавши цей запис, агент може почати з відомого результату та перевірити, " +"як ми його досягли. Він також може уникнути альтернатив, які вже зазнали " +"невдачі в тому ж контексті. Початкова точка покращується без зміни навчених " +"параметрів моделі." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:447 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"When reconstructing TH09's enemy-spawn function, a {ph0}GPT-6-Luna-max " +"session{ph1} drew a hypothesis from TH08's corresponding source family. That" +" earlier source suggested how to structure the loop and handle failures " +"within it. TH09's own evidence supplied the game-specific details, and clean" +" rebuilds confirmed the complete match." +msgstr "" +"Під час реконструкції функції TH09 для створення ворогів {ph0}GPT-6-Luna-max" +" сеанс{ph1} взяв гіпотезу з відповідного вихідного коду TH08. Цей попередній" +" код пропонував, як організувати цикл і обробляти збої в ньому. Власні " +"свідчення TH09 надали деталі гри. Перебудови з джерела підтвердили, що вся " +"функція відповідає." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:458 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}\"The model did not know that compiler quirk\" and \"the model could " +"not use it\" are different problems.{ph1} Retrieval may solve the first, but" +" turning that fact into a useful candidate still takes work. And if the " +"candidate fails, the model must decide what the result means for its next " +"attempt. Knowing the relevant pattern is only the beginning of that process." +msgstr "" +"{ph0}«Модель не знала про цю примху компілятора» та «модель не могла її " +"використати» — це різні проблеми.{ph1} Отримання може вирішити першу, але " +"для того, щоб перетворити цей факт на корисну кандидатуру, все ще потрібно " +"попрацювати. І якщо вихідний код кандидата зазнає невдачі, модель повинна " +"вирішити, що означає результат для її наступної спроби. Знання відповідного " +"шаблону – це лише початок цього процесу." + +#: website/public/blog/reverse-engineering-models/index.html:468 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For that reason, I would include a smaller general model with good retrieval" +" in any comparison. Giving it the same library of experience would help us " +"identify what training adds: better candidate generation, better decisions " +"after feedback, or both." +msgstr "" +"З цієї причини я б включив у будь-яке порівняння меншу модель ШІ загального " +"призначення з хорошим пошуком. Надання такої ж бібліотеки досвіду допомогло " +"б нам визначити, що додає навчання: краще формування кандидатів, кращі " +"рішення після зворотного зв’язку чи те й інше." + +#: website/public/blog/reverse-engineering-models/index.html:476 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Learning the next useful experiment" +msgstr "Вивчення наступного корисного експерименту" + +#: website/public/blog/reverse-engineering-models/index.html:477 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those decisions after feedback are what I would most like to study. In " +"agentic reconstruction, the model proposes an explanation and tests it " +"against the program. Even when the candidate fails, the result can narrow " +"the remaining possibilities and suggest a more useful next experiment." +msgstr "" +"Я б найбільше хотів вивчити ці рішення після зворотного зв’язку. При " +"агентній реконструкції модель пропонує пояснення та перевіряє його на " +"програму. Навіть якщо вихідний код кандидата не вдається, результат може " +"звузити можливості, що залишилися, і запропонувати більш корисний наступний " +"експеримент." + +#: website/public/blog/reverse-engineering-models/index.html:484 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Consider the temporary-results example again. Its final source shows the " +"arrangement that worked, but the route to that source contains more " +"information. Each of the following experiments helped identify what still " +"needed to change." +msgstr "" +"Знову розглянемо приклад із тимчасовими результатами. Його остаточне джерело" +" показує схему, яка спрацювала, але шлях до цього джерела містить більше " +"інформації. Кожен із наступних експериментів допоміг визначити, що ще " +"потрібно змінити." + +#: website/public/blog/reverse-engineering-models/index.html:497 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Experiment" +msgstr "Експериментуйте" + +#: website/public/blog/reverse-engineering-models/index.html:498 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What its result taught us" +msgstr "Чого нас навчив його результат" + +#: website/public/blog/reverse-engineering-models/index.html:503 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Try a different way to copy the results" +msgstr "Спробуйте інший спосіб скопіювати результати" + +#: website/public/blog/reverse-engineering-models/index.html:504 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The copy code changes, but the function still differs." +msgstr "Код копіювання змінюється, але функція все ще відрізняється." + +#: website/public/blog/reverse-engineering-models/index.html:509 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Use the returned values directly" +msgstr "Використовуйте повернуті значення безпосередньо" + +#: website/public/blog/reverse-engineering-models/index.html:510 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Copying follows the original pattern, but temporary storage still differs." +msgstr "" +"Копіювання відбувається за оригінальним шаблоном, але тимчасове зберігання " +"все ще відрізняється." + +#: website/public/blog/reverse-engineering-models/index.html:516 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the temporary results alive together" +msgstr "Зберігайте всі тимчасові змінні дійсними одночасно" + +#: website/public/blog/reverse-engineering-models/index.html:517 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"They get separate storage, but one still overlaps a value used later for " +"collision checks." +msgstr "" +"Вони отримують окреме сховище, але одне все одно перекриває значення, яке " +"буде використано пізніше для перевірок на конфлікти." + +#: website/public/blog/reverse-engineering-models/index.html:523 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the collision-check value separate" +msgstr "Зберігайте значення перевірки зіткнень окремо" + +#: website/public/blog/reverse-engineering-models/index.html:524 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A clean rebuild confirms the complete match." +msgstr "Чиста перекомпіляція з джерела підтверджує повний збіг." + +#: website/public/blog/reverse-engineering-models/index.html:529 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Selected experiments from the attack-function reconstruction above, grouped " +"by what they taught us." +msgstr "" +"Вибрані експерименти з наведеної вище реконструкції функції атаки, " +"згруповані за тим, чому вони нас навчили." + +#: website/public/blog/reverse-engineering-models/index.html:533 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A training record that keeps this sequence can teach more than the final " +"source alone. It connects each proposed change to an observed result and the" +" next decision. Keeping ineffective variations matters too: they tell a " +"later agent which hypotheses have already been tested in that context." +msgstr "" +"Навчальний запис, який зберігає цю послідовність, може навчити більше, ніж " +"одне лише кінцеве джерело. Він пов’язує кожну запропоновану зміну з " +"результатом спостереження та наступним рішенням. Збереження неефективних " +"варіацій також має значення: вони повідомляють пізнішому агенту, які " +"гіпотези вже були перевірені в цьому контексті." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:540 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}I would want the model to learn how to use feedback.{ph1} A neutral " +"result is a reason to reconsider the variation; a changed mismatch may point" +" to another cause. When local edits stop being informative, the agent should" +" be able to revisit the source representation or inspect the evidence again." +msgstr "" +"{ph0}Я хотів би, щоб модель навчилася використовувати зворотній " +"зв’язок.{ph1} Нейтральний результат є причиною переглянути варіант; змінена " +"невідповідність може вказувати на іншу причину. Коли локальні редагування " +"перестають бути інформативними, агент повинен мати можливість переглянути " +"вихідне представлення або знову перевірити докази." + +#: website/public/blog/reverse-engineering-models/index.html:549 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Learning to make those choices requires autonomy. Sometimes the useful next " +"step is inspecting another caller rather than compiling another edit. A " +"rigid sequence of required rewrites would remove that choice, along with " +"part of the behavior we want the model to learn." +msgstr "" +"Щоб навчитися робити такий вибір, потрібна автономія. Іноді корисним " +"наступним кроком є ​​перевірка іншої функції, що викликає цю функцію, а не " +"компіляція іншого редагування. Жорстка послідовність необхідних переписів " +"усуне цей вибір разом із частиною поведінки, яку ми хочемо, щоб модель " +"навчила." + +#: website/public/blog/reverse-engineering-models/index.html:558 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why RE is interesting for reinforcement learning" +msgstr "Чому зворотне проектування цікаве для навчання з підкріпленням" + +#: website/public/blog/reverse-engineering-models/index.html:559 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Bounded reconstruction gives us a practical environment for studying these " +"choices. Once the target and compiler are pinned, many source experiments " +"produce quick, repeatable feedback. The agent can compile a candidate, " +"inspect the mismatch and use that observation to choose its next step." +msgstr "" +"Реконструкція з визначеними межами дає практичне середовище для вивчення цих" +" рішень. Коли версії цілі й компілятора зафіксовані, багато експериментів із" +" вихідним кодом дають швидкий, відтворюваний зворотний зв’язок. Агент може " +"скомпілювати кандидата, перевірити відмінності та вибрати наступний крок." + +#: website/public/blog/reverse-engineering-models/index.html:566 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reinforcement learning could train that decision-making process through the " +"feedback it receives. A candidate accepted by a defined check gives us a " +"concrete outcome, while the sequence of attempts lets us observe how much " +"work it took to reach it." +msgstr "" +"Навчання з підкріпленням може тренувати цей процес прийняття рішень за " +"допомогою зворотного зв’язку, який він отримує. Кандидат, прийнятий за " +"певною перевіркою, дає нам конкретний результат, а послідовність спроб " +"дозволяє спостерігати, скільки роботи знадобилося для його досягнення." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:572 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"There is already research supporting this direction. {ph0}LLM4Decompile{ph1}" +" studies models specialized for decompilation. {ph2}AutoDecompiler{ph3} " +"trains multi-turn refinement using compiler and execution feedback. Their " +"studies focus on functional recovery. For this work, I would use them as a " +"starting point for the related task of finding source that matches a " +"historical compiler's output." +msgstr "" +"У цьому напрямку вже є дослідження. {ph0}LLM4Decompile{ph1} вивчає моделі, " +"що спеціалізуються на декомпіляції. {ph2}AutoDecompiler{ph3} тренує " +"багатоповоротне вдосконалення за допомогою компілятора та відгуків про " +"виконання. Їх дослідження зосереджені на функціональному відновленні. Для " +"цієї роботи я б використав їх як відправну точку для пов’язаного завдання " +"пошуку джерела, яке відповідає історичним результатам компілятора." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:584 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To reach such a well-defined task, though, we may first need to establish " +"what a function owns or devise a new runtime measurement. In TH08, for " +"example, a {ph0}collision difference led back to how the original program " +"handled floating-point calculations{ph1}. Before the agent could test a " +"useful correction, it had to work out what to measure." +msgstr "" +"Однак, щоб досягти такого чітко визначеного завдання, нам може спочатку " +"знадобитися встановити, якими ресурсами володіє функція, або розробити новий" +" спосіб спостереження за поведінкою програми під час виконання. У TH08, " +"наприклад, {ph0}різниця зіткнень призвела до того, як оригінальна програма " +"обробляла обчислення з плаваючою комою{ph1}. Перш ніж агент міг перевірити " +"корисну поправку, він повинен був визначити, що вимірювати." + +#: website/public/blog/reverse-engineering-models/index.html:595 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This suggests another role for a broadly capable model: helping make the " +"problem checkable in the first place. Once that check exists, we can " +"investigate how far a learned search policy can get with it." +msgstr "" +"Це говорить про іншу роль широкофункціональної моделі: допомогти зробити " +"проблему перевіреною в першу чергу. Щойно така перевірка з’явиться, ми " +"зможемо дослідити, наскільки далеко може зайти вивчена політика пошуку з " +"нею." + +#: website/public/blog/reverse-engineering-models/index.html:603 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "When the oracle is wrong" +msgstr "Коли засіб перевірки помиляється" + +#: website/public/blog/reverse-engineering-models/index.html:604 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Once we train against that check, its coverage becomes part of what the " +"model learns. I call the reference or checker an oracle. Even if it gives a " +"repeatable answer, we still need to understand what that answer covers." +msgstr "" +"Коли перевірка використовується для навчання, її охоплення стає частиною " +"навчального середовища. Такий еталон або засіб перевірки я називаю оракулом." +" Навіть відтворювану відповідь потрібно розуміти в межах того, що вона " +"справді перевіряє." + +#: website/public/blog/reverse-engineering-models/index.html:610 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"TH08 showed how much can hide outside that coverage. Some reconstructed " +"functions passed their exact comparisons with wrong floating constants. The " +"comparator adjusted relocated addresses before checking instructions, but it" +" checked the values behind those addresses only when they had been " +"explicitly recorded for verification. Other literal references could pass " +"without their values being examined." +msgstr "" +"TH08 показав, як багато можна приховати за межами цього покриття. Деякі " +"реконструйовані функції пройшли точне порівняння з неправильними плаваючими " +"константами. Компаратор скоригував переміщені адреси перед перевіркою " +"інструкцій, але він перевірив значення за цими адресами лише тоді, коли вони" +" були явно записані для перевірки. Інші літеральні посилання можуть пройти " +"без перевірки їхніх значень." + +#. {ph0}: +#. {ph1}: +#. {ph2}: 1.0f +#. {ph3}: 32.0f +#: website/public/blog/reverse-engineering-models/index.html:619 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To close that gap, a {ph0}wider audit{ph1} found incorrect constants in " +"functions that had already been accepted. In one calculation, the " +"reconstructed source added {ph2} where the original added {ph3}." +msgstr "" +"Щоб усунути цю прогалину, {ph0}ширший аудит{ph1} виявив неправильні " +"константи у функціях, які вже пройшли перевірку. В одному обчисленні " +"реконструйований вихідний код додав {ph2}, тоді як оригінальний додав {ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:629 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A learner rewarded by the old check could learn to produce the wrong " +"reconstruction.{ph1} We had not trained an RL model against this oracle; the" +" observed failure was in the check itself. But using that check as a reward " +"would have made the same blind spot part of the learning environment." +msgstr "" +"{ph0}Модель, нагороджена старою перевіркою перевірки, може навчитися " +"виробляти неправильну реконструкцію.{ph1} Ми не навчали модель RL за " +"допомогою цього засобу перевірки. Помилка, яку ми помітили, була в самій " +"перевірці. Але якби ми використовували це як винагороду, та сама сліпа пляма" +" стала б частиною навчального середовища." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:639 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A functional test can have a similar blind spot. {ph0}Differential-fuzzing " +"research on LLM decompilers{ph1} has found behavior differences beyond the " +"supplied tests. Passing those tests therefore needs to be read in terms of " +"the cases they exercise, just as an instruction match needs to be read in " +"terms of the bytes and data it checks." +msgstr "" +"Функціональний тест може мати подібну сліпу пляму. {ph0}Дослідження з " +"використанням диференціального фаззингу на декомпіляторах LLM{ph1} виявили " +"поведінкові відмінності поза наданими тестами. Тому ми повинні розуміти " +"проходження тесту з точки зору випадків, які він виконує. Подібним чином " +"збіг інструкцій слід розуміти з точки зору байтів і даних, які він " +"перевіряє." + +#: website/public/blog/reverse-engineering-models/index.html:649 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before using either kind of check as a reward, I would test it with " +"deliberate failures. I would also keep final acceptance outside the " +"candidate's editable source, so the model has to satisfy the check through " +"its reconstruction. Otherwise, something like a byte array copied from the " +"target could satisfy a naive score without recovering useful source." +msgstr "" +"Перш ніж використовувати будь-який вид перевірки підтвердження як " +"винагороду, я б перевірив його з навмисно неправильними результатами. Я " +"також хотів би зберегти остаточне прийняття окремо від вихідного коду, який " +"агент ШІ може редагувати. Модель повинна пройти перевірку шляхом коректної " +"реконструкції програми. Інакше просте копіювання байтового масиву з " +"цільового об’єкта може обдурити наївну систему підрахунку балів без " +"відновлення корисного вихідного коду." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:657 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Improving the learner and improving its oracle belong to the same " +"research problem.{ph1} As the learner gets better at optimizing the feedback" +" we give it, understanding that feedback becomes more important too." +msgstr "" +"{ph0}Поліпшення моделі й поліпшення її засобу перевірки — частини однієї " +"дослідницької проблеми.{ph1} Що краще модель оптимізує наданий зворотний " +"зв’язок, то важливіше розуміти сам цей зв’язок." + +#: website/public/blog/reverse-engineering-models/index.html:667 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A comparison worth running" +msgstr "Варте порівняння" + +#: website/public/blog/reverse-engineering-models/index.html:668 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"With those checks in place, I would start the comparison on a fixed VC7.1 " +"task family. First, I would establish baselines for a strong general model " +"and a smaller one. Adding repository knowledge to the smaller model would " +"show what retrieval changes. From there, supervised specialization and an " +"RL-trained version would let us examine what training adds." +msgstr "" +"За наявності цих перевірок я б почав порівняння з фіксованої сім’ї завдань " +"VC7.1. По-перше, я б визначив базові лінії для сильної моделі ШІ загального " +"призначення та меншої. Додавання знань про репозиторій до меншої моделі " +"покаже, які зміни в пошуку. Звідти контрольована спеціалізація та версія з " +"навчанням RL дозволять нам перевірити, що додає навчання." + +#: website/public/blog/reverse-engineering-models/index.html:676 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To make those comparisons informative, each model would receive the same " +"target evidence and tools. For the training-specific comparisons, I would " +"keep the base model and available knowledge fixed as well. Repeated trials " +"would help distinguish a reliable search strategy from a lucky candidate." +msgstr "" +"Щоб зробити ці порівняння інформативними, кожна модель отримуватиме однакові" +" цільові дані та інструменти. Для порівнянь, пов’язаних із навчанням, я б " +"також залишив базову модель і наявні знання фіксованими. Повторні " +"випробування допомогли б відрізнити надійну стратегію пошуку від " +"альтернативного рішення, яке вдалось випадково." + +#: website/public/blog/reverse-engineering-models/index.html:683 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would then separate two starting conditions: reconstructing a function " +"from its instructions, and closing the last few differences in maintained " +"source. A specialist might be excellent at the second task while a general " +"model does much of the work needed to reach it. Measuring them separately " +"would reveal that division of work." +msgstr "" +"Потім я б розділив дві початкові умови: реконструкція функції з її " +"інструкцій і закриття кількох останніх відмінностей у підтримуваному " +"джерелі. Спеціалізована модель штучного інтелекту може бути чудовою для " +"другого завдання, тоді як модель загального призначення виконує велику " +"частину роботи, необхідної для її досягнення. Вимірювання їх окремо виявило " +"б такий розподіл роботи." + +#: website/public/blog/reverse-engineering-models/index.html:691 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both tasks need genuinely held-out examples. Because related Touhou games " +"share engine code, holding out a game's name would still allow nearly " +"identical functions to appear in training. I would instead hold out source " +"families and keep their accepted answers out of both training and retrieval." +msgstr "" +"Для обох завдань потрібні тестові приклади, яких немає ані в навчальних " +"даних, ані в матеріалах для пошуку. Пов’язані ігри Touhou мають спільний код" +" рушія. Якщо відкласти лише одну гру, майже однакові функції все одно можуть" +" потрапити до навчання. Я б натомість відкладав цілі групи спорідненого " +"вихідного коду й виключав їхні перевірені відповіді як із навчальних даних, " +"так і з бібліотеки пошуку." + +#: website/public/blog/reverse-engineering-models/index.html:698 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"On those examples, the outcome would be complete, independently validated " +"matches. Giving each model the same candidate or compile budget would help " +"compare search quality. Measuring elapsed time and total cost, including " +"compilation, would tell us whether that quality produces an efficient " +"workflow. Training costs would need their own accounting too." +msgstr "" +"Для цих прикладів результатом будуть повні, незалежно перевірені збіги. " +"Рівні бюджети для пропозицій кандидатів або спроб компіляції допоможуть " +"порівняти якість пошуку. Вимірювання загального часу та вартості, включаючи " +"компіляцію, покаже, чи забезпечує така якість ефективний робочий процес. " +"Витрати на навчання слід вимірювати окремо." + +#: website/public/blog/reverse-engineering-models/index.html:706 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both views matter because a cheaper proposal can become an expensive " +"workflow if it causes more uninformative compiles. Conversely, a slower " +"model may pay for itself by finding a useful experiment sooner. The " +"comparison should capture those costs." +msgstr "" +"Обидва вимірювання мають значення. Дешева пропозиція може призвести до " +"дорогого робочого процесу, якщо це спричинить більше компіляцій, які не " +"надають корисної інформації. І навпаки, повільніша модель може бути " +"економнішою, якщо швидше знайде корисний експеримент. Порівняння повинно " +"включати ці витрати." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:712 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Counting accepted results also keeps intermediate progress in perspective. " +"One TH09 {ph0}replay-save candidate{ph1} looked promising because the " +"compiled function had the expected size, but a full comparison still found " +"substantial differences. Changes in how the compiler arranges instructions " +"can also make a candidate look closer without resolving the underlying " +"problem. Such diagnostics can guide the next experiment, but I would " +"validate any reward for \"getting closer\" against the full acceptance " +"check." +msgstr "" +"Підрахунок прийнятих результатів також зберігає проміжний прогрес у " +"перспективі. Один TH09 {ph0}кандидат для повторного збереження{ph1} виглядав" +" багатообіцяючим, оскільки скомпільована функція мала очікуваний розмір, але" +" повне порівняння все одно виявило значні відмінності. Зміни в тому, як " +"компілятор впорядковує інструкції, також можуть створити враження, що " +"кандидат ближчий до цілі, не вирішуючи основної проблеми. Така діагностика " +"може керувати наступним експериментом, але я б перевірив будь-яку винагороду" +" за «наближення» до повної перевірки прийняття." + +#: website/public/blog/reverse-engineering-models/index.html:727 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would be interested in the pattern across these results. If a specialist " +"does well within a familiar compiler family, does that advantage survive a " +"toolchain change? And once both models have good retrieval, how much does " +"training still add? The answers would help us decide where each kind of " +"model is most useful." +msgstr "" +"Мені було б цікаво побачити закономірності цих результатів. Якщо " +"спеціалізована модель штучного інтелекту добре справляється зі знайомим " +"сімейством компіляторів, чи збережеться ця перевага після зміни " +"інструментального ланцюжка? І коли обидві моделі штучного інтелекту мають " +"хороший пошук, скільки ще додає навчання? Відповіді допоможуть нам вирішити," +" де кожен тип моделі є найбільш корисним." + +#: website/public/blog/reverse-engineering-models/index.html:736 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What kind of agent does this suggest?" +msgstr "Що це за агент ШІ?" + +#: website/public/blog/reverse-engineering-models/index.html:737 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"If the models turn out to help at different stages, a useful arrangement " +"might be a broad agent with access to a specialist. Once a task becomes a " +"familiar compiler search, it could ask the specialist for candidates. If the" +" results stop making sense, the broader agent could return to the target " +"evidence or investigate the measurement itself." +msgstr "" +"Якщо виявиться, що моделі допомагають на різних етапах, корисною " +"домовленістю може бути широкий агент із доступом до спеціалізованої моделі " +"ШІ. Коли завдання стає звичним пошуком компілятора, воно може запитати " +"спеціалізовану модель AI для кандидатів. Якщо результати перестають мати " +"сенс, ширший агент може повернутися до цільових доказів або дослідити саме " +"вимірювання." + +#: website/public/blog/reverse-engineering-models/index.html:745 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In that arrangement, specialization becomes a tool the agent can choose as " +"the investigation develops. The agent keeps the freedom to change direction " +"that made the earlier breakthroughs possible." +msgstr "" +"У такій домовленості спеціалізація стає інструментом, який агент може " +"вибрати під час розслідування. Агент зберігає свободу змінювати напрямок, що" +" уможливило попередні прориви." + +#: website/public/blog/reverse-engineering-models/index.html:750 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This follows naturally from the industrial shift I described in the previous" +" essay. Being able to run many more experiments changes how we organize the " +"work. It also gives us a reason to ask which decisions need broad judgment " +"and which improve through repeated experience with a particular craft." +msgstr "" +"Це природно випливає з промислових змін, які я описав у попередньому есе. " +"Можливість проводити багато інших експериментів змінює спосіб організації " +"роботи. Це також дає нам привід запитати, які рішення потребують широкого " +"оцінювання, а які покращуються через повторний досвід роботи з певним " +"ремеслом." + +#: website/public/blog/reverse-engineering-models/index.html:757 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For now, the strongest general model has earned its place in my workflow by " +"opening problems I had left unresolved. Those successes are what make me " +"want to examine the work more closely: how much depended on the model's " +"breadth, and how much could be handled by a model trained in this particular" +" kind of reconstruction?" +msgstr "" +"Наразі найсильніша універсальна модель заслужила місце в моєму робочому " +"процесі, розв’язавши проблеми, які раніше залишалися нерозв’язаними. Ці " +"успіхи спонукають мене дослідити процес: яка частина залежала від широти її " +"можливостей, а яку могла б виконати модель, навчена саме такої " +"реконструкції?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:765 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}The repositories give us a way to test the question.{ph1} I would like " +"to use that record to build the comparison and see what we learn." +msgstr "" +"{ph0}Репозиторії дають нам спосіб перевірити запитання.{ph1} Я хотів би " +"використати цей запис для порівняння та побачити, що ми дізнаємося." + +#: website/public/blog/reverse-engineering-models/index.html:775 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Evidence and model attribution" +msgstr "Докази та атрибуція моделі" + +#: website/public/blog/reverse-engineering-models/index.html:783 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The examples describe historical function-level comparisons recorded in the " +"linked commits. Whole-game runtime checks are separate milestones. Source " +"and matching records were reviewed on 13 October 2026." +msgstr "" +"Приклади описують історичні порівняння на рівні функцій, записані у " +"зв’язаних комітах. Перевірки виконання всієї гри є окремими етапами. Вихідні" +" та відповідні записи було переглянуто 13 жовтня 2026 року." + +#. {ph0}: gpt-dots +#. {ph1}: gpt-web +#: website/public/blog/reverse-engineering-models/index.html:789 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The TH09 label {ph0} identifies GPT-6-Astra. {ph1} changed with the model " +"used at the time, so that label alone does not identify an individual run. " +"Explicit model names in the other cited commits establish their recorded " +"attribution; the histories do not contain uniform per-run cost or effort " +"measurements." +msgstr "" +"Мітка TH09 {ph0} означає GPT-6-Astra. Модель, пов’язана з {ph1}, змінювалася" +" з часом, тому лише ця мітка не може ідентифікувати модель, яка " +"використовується в певному циклі. Явні назви моделей в інших цитованих " +"комітах забезпечують записане присвоєння. Ці історії не містять узгоджених " +"вимірювань витрат або зусиль для кожного запуску." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#: website/public/blog/reverse-engineering-models/index.html:798 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 temporary lifetimes:{ph1} the {ph2}earlier context review{ph3} and" +" {ph4}complete cold reconstruction{ph5}." +msgstr "" +"{ph0}TH09 терміни життя тимчасових об’єктів:{ph1} {ph2}попередній перегляд " +"контексту{ph3} та {ph4}повна реконструкція, підтверджена чистою " +"перекомпіляцією з джерела{ph5}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:810 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 Front callback:{ph1} {ph2}non-exact diagnostic regions{ph3}, " +"{ph4}Astra's new graph{ph5} and its {ph6}accepted recompile replay{ph7}." +msgstr "" +"{ph0}TH09 Front зворотний виклик:{ph1} {ph2}діагностичні області, які не " +"збігаються{ph3}, {ph4}Astra новий графік потоку керування{ph5} та його " +"{ph6}перевірка прийнятої повторної компіляції{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:826 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Compiler knowledge:{ph1} {ph2}TH095 identifier-bucket reuse{ph3}, " +"{ph4}TH04 player source and linker order{ph5} and {ph6}Luna's TH09 enemy-" +"spawn closure{ph7}." +msgstr "" +"{ph0}Знання компілятора:{ph1} {ph2}TH095 повторне використання поведінки " +"сегмента ідентифікатора{ph3}, {ph4}TH04 вихідний код гравця та порядок " +"зв’язування{ph5} та повна реконструкція {ph6}Luna функції створення ворогів " +"TH09{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:842 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Checks and diagnostics:{ph1} {ph2}TH08 floating-literal audit{ph3}, " +"{ph4}TH08 numerical investigation{ph5} and {ph6}TH09 replay-save " +"reconstruction{ph7}." +msgstr "" +"{ph0}Перевірки та діагностика:{ph1} {ph2}TH08 постійний аудит із плаваючою " +"комою{ph3}, {ph4}TH08 числове дослідження{ph5} та {ph6}TH09 реконструкція зі" +" збереженням повтору{ph7}." + +#: website/public/blog/reverse-engineering-models/index.html:859 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The proposed specialist and RL comparisons are future experiments. The " +"project histories motivate them; no trained specialist was evaluated for " +"this essay." +msgstr "" +"Пропоновані порівняння спеціалізованих моделей і навчання RL є майбутніми " +"експериментами. Історії проекту мотивують їх. Жодна навчена спеціалізована " +"модель не оцінювалася для цього есе." + +#: website/public/blog/reverse-engineering-models/index.html:6 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model? | REA" +msgstr "Чи потрібна для зворотного проектування більша модель? | REA" + +#: website/public/blog/reverse-engineering-models/index.html:7 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What Touhou reconstruction taught me about general models, specialist models" +" and learning to propose better hypotheses from compiler feedback." +msgstr "" +"Чого навчила мене реконструкція Touhou щодо загальних моделей, спеціальних " +"моделей і навчання пропонувати кращі гіпотези на основі відгуків " +"компілятора." + +#: website/public/blog/reverse-engineering-models/index.html:240 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "On this page" +msgstr "На цій сторінці" + +#: website/public/blog/reverse-engineering-models/index.html:253 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Models and reverse engineering essay" +msgstr "Есе про моделі та зворотне проектування" + +#: website/public/blog/reverse-engineering-models/index.html:491 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Selected reconstruction experiments" +msgstr "Вибрані досліди реконструкції" + +#: website/public/blog/reverse-engineering-models/index.html:883 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Back to top" +msgstr "Нагору" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:776 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Detailed evidence notes{ph1} include the recorded comparisons and " +"experiment context." +msgstr "" +"{ph0}Докладні записи доказів{ph1} містять результати порівнянь і контекст " +"експериментів." + +#~ msgctxt "page:blog/index.html" +#~ msgid "By N0zoM1z0 · 13 October 2026 · Draft for review" +#~ msgstr "Автор: N0zoM1z0 · 13 жовтня 2026 р. · Чернетка для обговорення" + #~ msgctxt "page:index.html" #~ msgid "Setup guide" #~ msgstr "Посібник зі встановлення" diff --git a/website/locales/vi/messages.po b/website/locales/vi/messages.po index 3aa144d2f..dac24ee02 100644 --- a/website/locales/vi/messages.po +++ b/website/locales/vi/messages.po @@ -32,18 +32,18 @@ msgid "Thoughts on reverse engineering." msgstr "Suy nghĩ về kỹ thuật đảo ngược." #. {ph0}:
-#: website/public/blog/index.html:200 +#: website/public/blog/index.html:214 #, python-brace-format msgctxt "page:blog/index.html" msgid "Game reconstruction{ph0}Essay" msgstr "Tái thiết trò chơi{ph0}Bài luận" -#: website/public/blog/index.html:202 +#: website/public/blog/index.html:216 msgctxt "page:blog/index.html" msgid "The industrial revolution in reverse engineering" msgstr "Cuộc cách mạng công nghiệp trong kỹ thuật đảo ngược" -#: website/public/blog/index.html:203 +#: website/public/blog/index.html:217 msgctxt "page:blog/index.html" msgid "" "What manual RE and reconstructing TH08, TH095 and TH04 taught me about " @@ -52,25 +52,26 @@ msgstr "" "NHỮNG gì HƯỚNG DẪN SỬ DỤNG LẠI VÀ tái TẠO TH08, TH095 và TH04 đã dạy tôi về " "các agent, xác minh và kiến thức mà các hợp chất trong các dự án." -#: website/public/blog/index.html:208 +#: website/public/blog/index.html:207 website/public/blog/index.html:222 msgctxt "page:blog/index.html" msgid "Read the article" msgstr "Đọc bài viết" -#: website/public/blog/index.html:211 +#: website/public/blog/index.html:225 msgctxt "page:blog/index.html" msgid "By N0zoM1z0 · 10 October 2026" msgstr "Bởi N0zoM1z0 * 10 tháng mười 2026" #. {ph0}: -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 +#: website/public/blog/reverse-engineering-models/index.html:883 #: website/public/blog/touhou-reconstruction/index.html:1080 #: website/public/examples/dino-lab/index.html:475 #: website/public/faq/index.html:573 #: website/public/first-investigation/index.html:523 #: website/public/get-started/index.html:882 -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 #: website/public/guides/index.html:282 #: website/public/guides/javascript/index.html:503 #: website/public/guides/native/index.html:509 website/public/index.html:1191 @@ -100,6 +101,8 @@ msgstr "" "REA và AI." #: website/public/blog/index.html:75 website/public/blog/index.html:88 +#: website/public/blog/reverse-engineering-models/index.html:75 +#: website/public/blog/reverse-engineering-models/index.html:88 #: website/public/blog/touhou-reconstruction/index.html:75 #: website/public/blog/touhou-reconstruction/index.html:88 #: website/public/examples/dino-lab/index.html:75 @@ -136,6 +139,7 @@ msgstr "" "mã hóa của bạn." #: website/public/blog/index.html:108 +#: website/public/blog/reverse-engineering-models/index.html:108 #: website/public/blog/touhou-reconstruction/index.html:108 #: website/public/examples/dino-lab/index.html:109 #: website/public/faq/index.html:108 @@ -156,6 +160,7 @@ msgid "REA home" msgstr "Trang chủ REA" #: website/public/blog/index.html:112 +#: website/public/blog/reverse-engineering-models/index.html:112 #: website/public/blog/touhou-reconstruction/index.html:112 #: website/public/examples/dino-lab/index.html:113 #: website/public/faq/index.html:112 @@ -176,6 +181,7 @@ msgid "Main navigation" msgstr "Điều hướng chính" #: website/public/blog/index.html:130 +#: website/public/blog/reverse-engineering-models/index.html:130 #: website/public/blog/touhou-reconstruction/index.html:130 #: website/public/examples/dino-lab/index.html:131 #: website/public/faq/index.html:130 @@ -196,6 +202,7 @@ msgid "REA on GitHub" msgstr "REA TRÊN GitHub" #: website/public/blog/index.html:150 +#: website/public/blog/reverse-engineering-models/index.html:150 #: website/public/blog/touhou-reconstruction/index.html:150 #: website/public/examples/dino-lab/index.html:151 #: website/public/faq/index.html:150 @@ -216,6 +223,7 @@ msgid "Join REA on Discord" msgstr "Tham gia Discord của REA" #: website/public/blog/index.html:169 +#: website/public/blog/reverse-engineering-models/index.html:169 #: website/public/blog/touhou-reconstruction/index.html:169 #: website/public/examples/dino-lab/index.html:170 #: website/public/faq/index.html:169 @@ -235,7 +243,7 @@ msgctxt "shared" msgid "Follow REA on X" msgstr "Theo DÕI REA Trên X" -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 msgctxt "page:blog/index.html" msgid "Back to top" msgstr "Về đầu trang" @@ -4764,7 +4772,7 @@ msgctxt "page:guides/browser/index.html" msgid "Copy read the loaded scripts" msgstr "Sao chép đọc các tập lệnh đã tải" -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 msgctxt "page:guides/browser/index.html" msgid "Back to top" msgstr "Về đầu trang" @@ -11594,6 +11602,7 @@ msgid "Same spacing: 256 ÷ 16 = 16 angle units = 22.5°" msgstr "Khoảng cách tương tự: 256 ÷ 16 = 16 đơn vị góc = 22,5°" #: website/public/blog/index.html:114 +#: website/public/blog/reverse-engineering-models/index.html:114 #: website/public/blog/touhou-reconstruction/index.html:114 #: website/public/examples/dino-lab/index.html:115 #: website/public/faq/index.html:114 @@ -11614,6 +11623,7 @@ msgid "Showcases" msgstr "Ví dụ thực tế" #: website/public/blog/index.html:115 +#: website/public/blog/reverse-engineering-models/index.html:115 #: website/public/blog/touhou-reconstruction/index.html:115 #: website/public/examples/dino-lab/index.html:116 #: website/public/faq/index.html:115 @@ -11634,6 +11644,7 @@ msgid "Blog" msgstr "Blog" #: website/public/blog/index.html:113 +#: website/public/blog/reverse-engineering-models/index.html:113 #: website/public/blog/touhou-reconstruction/index.html:113 #: website/public/examples/dino-lab/index.html:114 #: website/public/faq/index.html:113 @@ -11654,6 +11665,7 @@ msgid "Get started" msgstr "Bắt đầu" #: website/public/blog/index.html:116 +#: website/public/blog/reverse-engineering-models/index.html:116 #: website/public/blog/touhou-reconstruction/index.html:116 #: website/public/examples/dino-lab/index.html:117 #: website/public/faq/index.html:116 @@ -11673,13 +11685,14 @@ msgctxt "shared" msgid "Guides" msgstr "Hướng dẫn" -#: website/public/blog/index.html:222 +#: website/public/blog/index.html:236 +#: website/public/blog/reverse-engineering-models/index.html:875 #: website/public/blog/touhou-reconstruction/index.html:1072 #: website/public/examples/dino-lab/index.html:467 #: website/public/faq/index.html:565 #: website/public/first-investigation/index.html:515 #: website/public/get-started/index.html:874 -#: website/public/guides/browser/index.html:490 +#: website/public/guides/browser/index.html:499 #: website/public/guides/index.html:274 #: website/public/guides/javascript/index.html:495 #: website/public/guides/native/index.html:501 website/public/index.html:1183 @@ -11693,13 +11706,14 @@ msgctxt "shared" msgid "Source" msgstr "Mã nguồn" -#: website/public/blog/index.html:223 +#: website/public/blog/index.html:237 +#: website/public/blog/reverse-engineering-models/index.html:876 #: website/public/blog/touhou-reconstruction/index.html:1073 #: website/public/examples/dino-lab/index.html:468 #: website/public/faq/index.html:566 #: website/public/first-investigation/index.html:516 #: website/public/get-started/index.html:875 -#: website/public/guides/browser/index.html:491 +#: website/public/guides/browser/index.html:500 #: website/public/guides/index.html:275 #: website/public/guides/javascript/index.html:496 #: website/public/guides/native/index.html:502 website/public/index.html:1184 @@ -11713,13 +11727,14 @@ msgctxt "shared" msgid "MIT license" msgstr "Giấy phép MIT" -#: website/public/blog/index.html:224 +#: website/public/blog/index.html:238 +#: website/public/blog/reverse-engineering-models/index.html:877 #: website/public/blog/touhou-reconstruction/index.html:1074 #: website/public/examples/dino-lab/index.html:469 #: website/public/faq/index.html:567 #: website/public/first-investigation/index.html:517 #: website/public/get-started/index.html:876 -#: website/public/guides/browser/index.html:492 +#: website/public/guides/browser/index.html:501 #: website/public/guides/index.html:276 #: website/public/guides/javascript/index.html:497 #: website/public/guides/native/index.html:503 website/public/index.html:1185 @@ -11733,13 +11748,14 @@ msgctxt "shared" msgid "Join the community" msgstr "Tham gia cộng đồng" -#: website/public/blog/index.html:225 +#: website/public/blog/index.html:239 +#: website/public/blog/reverse-engineering-models/index.html:878 #: website/public/blog/touhou-reconstruction/index.html:1075 #: website/public/examples/dino-lab/index.html:470 #: website/public/faq/index.html:568 #: website/public/first-investigation/index.html:518 #: website/public/get-started/index.html:877 -#: website/public/guides/browser/index.html:493 +#: website/public/guides/browser/index.html:502 #: website/public/guides/index.html:277 #: website/public/guides/javascript/index.html:498 #: website/public/guides/native/index.html:504 website/public/index.html:1186 @@ -12675,6 +12691,1389 @@ msgctxt "runtime" msgid "Switch to dark mode" msgstr "Chuyển sang giao diện tối" +#. {ph0}: +#. {ph1}: +#: website/public/guides/browser/index.html:480 +#, python-brace-format +msgctxt "page:guides/browser/index.html" +msgid "" +"For HTTP request and response bodies, use {ph0}Fiddler Everywhere: SAZ " +"captures, MCP and public skills{ph1}. Export SAZ or HAR 1.2 for offline REA " +"capture Evidence, or connect Everywhere's MCP server to inspect captured " +"traffic with your agent." +msgstr "" +"Để kiểm tra phần thân yêu cầu và phản hồi HTTP, hãy dùng {ph0}Fiddler " +"Everywhere: bản ghi SAZ, MCP và kỹ năng công khai{ph1}. Xuất SAZ ho?c HAR " +"1.2 để lấy Evidence từ bản ghi trong REA ngoại tuyến, hoặc kết nối máy chủ " +"MCP của Everywhere để kiểm tra lưu lượng đã ghi cùng trợ lý của bạn." + +#. {ph0}:
+#: website/public/blog/index.html:200 +#, python-brace-format +msgctxt "page:blog/index.html" +msgid "Model specialization{ph0}Essay" +msgstr "Mô hình chuyên biệt{ph0}Bài viết" + +#: website/public/blog/index.html:202 +msgctxt "page:blog/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "Kỹ thuật đảo ngược có cần mô hình lớn hơn không?" + +#: website/public/blog/index.html:203 +msgctxt "page:blog/index.html" +msgid "" +"What Touhou reconstruction suggests about general models, specialist models " +"and learning from compiler feedback." +msgstr "" +"Những suy nghĩ từ việc tái dựng Touhou về mô hình tổng quát, mô hình chuyên " +"biệt và cách học từ phản hồi của trình biên dịch." + +#: website/public/blog/index.html:210 +msgctxt "page:blog/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Bởi N0zoM1z0 · 13 tháng 10 năm 2026" + +#: website/public/blog/reverse-engineering-models/index.html:103 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Skip to content" +msgstr "Bỏ qua nội dung" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:194 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Home{ph1} / {ph2}Blog{ph3} / Models and RE" +msgstr "{ph0}Home{ph1} / {ph2}Blog{ph3} / Mô hình và kỹ thuật đảo ngược" + +#: website/public/blog/reverse-engineering-models/index.html:197 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Essay" +msgstr "Tiểu luận" + +# Reviewed for technical meaning and readability; preserve the distinction +# between historical evidence and proposed experiments. +#: website/public/blog/reverse-engineering-models/index.html:198 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "Kỹ thuật đảo ngược có cần một mô hình lớn hơn không?" + +#: website/public/blog/reverse-engineering-models/index.html:199 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "Bởi N0zoM1z0 · 13 tháng 10 năm 2026" + +#: website/public/blog/reverse-engineering-models/index.html:201 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Takeaway" +msgstr "Điểm chính" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:203 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Model capability changes the hypotheses an agent can find.{ph1} Later " +"sessions have solved problems that earlier work left unresolved." +msgstr "" +"{ph0}Khả năng của mô hình ảnh hưởng đến những giả thuyết mà tác nhân AI có " +"thể đề xuất.{ph1} Các phiên sau đã giải quyết các vấn đề mà công việc trước " +"đó chưa được giải quyết." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:211 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A familiar compiler gives specialization something concrete to " +"learn.{ph1} Some exact matches depend on narrow, reusable source patterns." +msgstr "" +"{ph0}Một trình biên dịch quen thuộc cung cấp các mẫu cụ thể để một mô hình " +"chuyên biệt học.{ph1} Một số kết quả khớp mã máy chính xác phụ thuộc vào các" +" mẫu mã nguồn cụ thể, có thể tái sử dụng." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:218 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Give repository knowledge a fair comparison.{ph1} A smaller model with " +"good retrieval may already know enough to make progress." +msgstr "" +"{ph0}Bao gồm kiến thức kho lưu trữ trong một so sánh công bằng.{ph1} Một mô " +"hình nhỏ hơn với khả năng truy xuất kiến thức tốt có thể đã có đủ thông tin " +"để đạt được tiến bộ." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:223 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Train the next useful experiment, and test its oracle.{ph1} The value " +"of feedback depends on what the checker actually verifies." +msgstr "" +"{ph0}Huấn luyện mô hình để chọn thử nghiệm hữu ích tiếp theo và kiểm tra " +"công cụ xác minh của nó.{ph1} Phản hồi chỉ hữu ích trong phạm vi công cụ xác" +" minh những điều đúng." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:230 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Measure accepted reconstruction against time and cost.{ph1} That is the" +" result I want from the whole agent." +msgstr "" +"{ph0}Đánh giá kết quả tái thiết đã được xác minh cùng với thời gian và chi " +"phí.{ph1} Đó là những gì tôi muốn đo lường cho quy trình làm việc hoàn chỉnh" +" của tác nhân AI." + +#: website/public/blog/reverse-engineering-models/index.html:241 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "In this article" +msgstr "Trong bài viết này" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:243 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What better models changed{ph1}" +msgstr "{ph0}Mô hình mạnh hơn đã thay đổi điều gì?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:244 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Why specialize?{ph1}" +msgstr "{ph0}Tại sao nên chuyên môn hóa?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:245 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Where experience lives{ph1}" +msgstr "{ph0}Kinh nghiệm này nên được lưu trữ ở đâu?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:246 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Learning the next experiment{ph1}" +msgstr "{ph0}Học thí nghiệm hữu ích tiếp theo{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:247 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}RE and reinforcement learning{ph1}" +msgstr "{ph0}Học kỹ thuật đảo ngược và tăng cường{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:248 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}When the oracle is wrong{ph1}" +msgstr "{ph0}Khi công cụ xác minh sai{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:249 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}A comparison worth running{ph1}" +msgstr "{ph0}Một so sánh đáng chạy{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:250 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What kind of agent?{ph1}" +msgstr "{ph0}Điều này gợi ý loại tác nhân AI nào?{ph1}" + +#: website/public/blog/reverse-engineering-models/index.html:258 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What better models changed" +msgstr "Mô hình mạnh hơn đã thay đổi điều gì?" + +#: website/public/blog/reverse-engineering-models/index.html:259 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Since August 2026, I have been exploring how much reverse engineering an " +"agent can carry out with access to a program and a way to check its work. " +"Most of my experience comes from reconstructing games, especially Touhou. As" +" I have changed models, the repositories have kept a record of what each " +"session managed to recover." +msgstr "" +"Kể từ tháng 8 năm 2026, tôi đã khám phá mức độ kỹ thuật đảo ngược mà một tác" +" nhân có thể thực hiện khi có quyền truy cập vào một chương trình và cách " +"kiểm tra công việc của nó. Phần lớn kinh nghiệm của tôi đến từ việc xây dựng" +" lại trò chơi, đặc biệt là Touhou. Khi tôi thay đổi mô hình, các kho lưu trữ" +" đã lưu giữ một bản ghi về những gì mỗi phiên đã khôi phục được." + +#: website/public/blog/reverse-engineering-models/index.html:267 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of that work aims at exact reconstruction: we compile the maintained " +"source under a pinned historical toolchain and check whether it reproduces " +"the target function. When a later model succeeds where earlier work stalled," +" the source changes show what made that progress possible." +msgstr "" +"Phần lớn công việc hướng tới tái dựng chính xác: biên dịch mã nguồn đang " +"được duy trì bằng chuỗi công cụ lịch sử có phiên bản cố định, rồi kiểm tra " +"xem nó có tái tạo hàm đích hay không. Khi mô hình sau giải quyết được điểm " +"mà công việc trước bị mắc kẹt, thay đổi trong mã nguồn cho thấy điều gì đã " +"giúp tiến triển." + +#: website/public/blog/reverse-engineering-models/index.html:274 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The temporary values had to stay alive together" +msgstr "Các biến tạm thời cần có thời gian tồn tại chồng chéo" + +#: website/public/blog/reverse-engineering-models/index.html:275 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of a TH09 attack function's logic had already been reconstructed, but " +"the compiler still stored temporary results differently from the original. " +"The remaining problem was how long those results needed to stay alive and " +"when their storage could be reused." +msgstr "" +"Phần lớn logic của hàm tấn công TH09 đã được xây dựng lại, nhưng trình biên " +"dịch vẫn lưu trữ các kết quả tạm thời khác với kết quả ban đầu. Vấn đề còn " +"lại là những kết quả đó cần tồn tại trong bao lâu và khi nào bộ nhớ của " +"chúng có thể được sử dụng lại." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:282 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A {ph0}GPT-6-Sol checkpoint{ph1} recorded this mismatch after testing " +"several ordinary source arrangements. When {ph2}GPT-6.1-Sol returned to the " +"problem{ph3}, it found a useful hypothesis about how long those temporary " +"values needed to stay alive." +msgstr "" +"Bản ghi phiên {ph0}GPT-6-Sol{ph1} đã báo cáo sự không khớp này sau khi kiểm " +"tra một số cách sắp xếp mã nguồn thông thường. Khi {ph2}GPT-6.1-Sol kiểm tra" +" lại vấn đề{ph3}, nó đã đề xuất một giả thuyết hữu ích về thời gian tồn tại " +"của các biến tạm thời này." + +#: website/public/blog/reverse-engineering-models/index.html:296 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The function produced temporary results and copied them into animation " +"positions. In C++, a temporary result stays alive until the expression using" +" it finishes. Grouping the real operations into one expression kept those " +"results alive together, so the compiler could no longer reuse the same " +"storage for all of them." +msgstr "" +"Hàm này tạo ra các kết quả tạm thời và sao chép chúng vào các vị trí hoạt " +"ảnh. Trong C++, kết quả tạm thời vẫn tồn tại cho đến khi biểu thức sử dụng " +"nó kết thúc. Việc nhóm các hoạt động thực tế thành một biểu thức giúp giữ " +"cho các kết quả đó tồn tại cùng nhau, do đó trình biên dịch không thể sử " +"dụng lại cùng một bộ lưu trữ cho tất cả chúng nữa." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:303 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"One result still shared storage with a value used later for collision " +"checks. Giving that value its own lifetime resolved the remaining overlap. " +"After several targeted experiments, {ph0}a clean rebuild confirmed that the " +"full function matched{ph1}." +msgstr "" +"Một kết quả vẫn được chia sẻ bộ nhớ với giá trị được sử dụng sau này để kiểm" +" tra xung đột. Việc đưa ra giá trị đó trong thời gian tồn tại của chính nó " +"đã giải quyết được sự chồng chéo còn lại. Sau một số thử nghiệm có mục tiêu," +" {ph0}việc biên dịch lại rõ ràng từ nguồn đã xác nhận rằng hàm đầy đủ khớp " +"với{ph1}." + +#: website/public/blog/reverse-engineering-models/index.html:312 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What interests me is how the discovery and the tests worked together. The " +"lifetime hypothesis gave the agent a new direction; compiler feedback helped" +" it refine that hypothesis into a source arrangement that matched." +msgstr "" +"Tôi quan tâm đến việc khám phá và thử nghiệm hỗ trợ lẫn nhau như thế nào. " +"Giả thuyết về thời gian tồn tại của đối tượng tạm thời đã mang lại cho tác " +"nhân AI một hướng đi mới. Phản hồi của trình biên dịch đã giúp nó tinh chỉnh" +" giả thuyết đó thành một cách sắp xếp mã nguồn phù hợp." + +#: website/public/blog/reverse-engineering-models/index.html:318 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A different control-flow graph" +msgstr "Biểu đồ luồng điều khiển khác" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:319 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A second TH09 function shows why that new direction can matter. Here, the " +"obstacle was how several branches shared their final work. " +"{ph0}GPT-6.1-Sol's earlier work{ph1} got parts of the function to match, but" +" the branches still joined up differently from the original." +msgstr "" +"Hàm TH09 thứ hai cho biết tại sao hướng đi mới đó lại quan trọng. Ở đây, trở" +" ngại là làm thế nào một số chi nhánh chia sẻ công việc cuối cùng của họ. " +"Công việc trước đó của {ph0}GPT-6.1-Sol{ph1} có các phần của hàm khớp với " +"nhau, nhưng các nhánh vẫn được nối với nhau khác với ban đầu." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:330 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}GPT-6-Astra changed the surrounding control-flow graph{ph1}. The main " +"paths now led to a shared final update and return. In that source structure," +" the compiler combined the branches in the same way as the original. The " +"complete function matched, and a {ph2}separate rebuild check confirmed " +"it{ph3}." +msgstr "" +"{ph0}GPT-6-Astra đã thay đổi biểu đồ luồng điều khiển xung quanh{ph1}. Các " +"đường dẫn chính hiện đã dẫn đến bản cập nhật cuối cùng được chia sẻ và quay " +"trở lại. Trong cấu trúc nguồn đó, trình biên dịch kết hợp các nhánh giống " +"như bản gốc. Chức năng hoàn chỉnh đã khớp và một cuộc kiểm tra xây dựng lại " +"riêng biệt {ph2} đã xác nhận điều đó{ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:342 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Changing the representation opened a more productive search space.{ph1}" +" Both cases give me a reason to take general model capability seriously: an " +"agent can keep trying local variations for a long time before finding a " +"better way to express the surrounding work." +msgstr "" +"{ph0}Việc thay đổi cách biểu diễn đã mở ra một không gian tìm kiếm hiệu quả " +"hơn.{ph1} Cả hai trường hợp đều cho tôi lý do để xem xét nghiêm túc khả năng" +" của mô hình chung: một tác nhân có thể tiếp tục thử các biến thể cục bộ " +"trong một thời gian dài trước khi tìm ra cách tốt hơn để thể hiện công việc " +"xung quanh." + +#: website/public/blog/reverse-engineering-models/index.html:351 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"At the same time, the later sessions started with earlier source and " +"recorded failures. That accumulated work was part of their advantage. The " +"history lets us inspect the discoveries and their checks; a controlled " +"comparison would help separate the effect of the model from the experience " +"it inherited." +msgstr "" +"Đồng thời, các phiên sau bắt đầu với nguồn trước đó và ghi lại lỗi. Công " +"việc tích lũy đó là một phần lợi thế của họ. Lịch sử cho phép chúng ta kiểm " +"tra những khám phá và kiểm tra chúng; một so sánh có kiểm soát sẽ giúp tách " +"biệt tác động của mô hình khỏi trải nghiệm mà nó kế thừa." + +#: website/public/blog/reverse-engineering-models/index.html:358 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is where the question becomes interesting to me. Some of the progress " +"came from finding a new representation, while much of the remaining work " +"involved testing source forms against a familiar compiler. How much needs " +"broad reasoning ability, and how much could a model learn through repeated " +"experience with that compiler and code?" +msgstr "" +"Đây là điều làm cho câu hỏi trở nên thú vị đối với tôi. Một số tiến bộ đến " +"từ việc tìm kiếm một đại diện mới. Phần lớn công việc còn lại liên quan đến " +"việc kiểm tra các cách sắp xếp mã nguồn khác nhau dựa trên một trình biên " +"dịch quen thuộc. Bao nhiêu trong số này đòi hỏi khả năng suy luận rộng? Một " +"mô hình có thể học được bao nhiêu khi làm việc liên tục với trình biên dịch " +"và mã đó?" + +#: website/public/blog/reverse-engineering-models/index.html:368 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why might a specialist do well?" +msgstr "Tại sao một mô hình AI chuyên biệt có thể hoạt động tốt?" + +#: website/public/blog/reverse-engineering-models/index.html:369 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Touhou gives us a concrete reason to expect that experience to help. Much of" +" the game code comes from the same author, ZUN, so recurring habits are a " +"reasonable starting assumption. The repositories back that up with useful " +"examples: related loops and helper arrangements have supplied hypotheses " +"across games." +msgstr "" +"Touhou cho chúng ta lý do cụ thể để mong đợi trải nghiệm đó sẽ hữu ích. Phần" +" lớn mã trò chơi đều đến từ cùng một tác giả, ZUN, vì vậy những thói quen " +"lặp đi lặp lại là một giả định khởi đầu hợp lý. Các kho lưu trữ hỗ trợ điều " +"đó bằng các ví dụ hữu ích: các vòng lặp liên quan và cách sắp xếp trợ giúp " +"đã cung cấp các giả thuyết cho các trò chơi." + +#: website/public/blog/reverse-engineering-models/index.html:376 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those source patterns become more specific when we include the compiler. " +"Under a particular version and set of options, small changes in source can " +"produce repeatable differences in machine code. Some of those differences " +"are surprisingly obscure." +msgstr "" +"Các mẫu nguồn đó trở nên cụ thể hơn khi chúng tôi đưa trình biên dịch vào. " +"Trong một phiên bản và tập hợp các tùy chọn cụ thể, những thay đổi nhỏ về " +"nguồn có thể tạo ra những khác biệt lặp lại trong mã máy. Một số trong những" +" khác biệt đó mơ hồ một cách đáng ngạc nhiên." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: Supervisor::LoadDat +#. {ph4}: +#: website/public/blog/reverse-engineering-models/index.html:382 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In TH095, for example, {ph0}even the name used for a local variable could " +"change where VC7.1 stored it{ph1}. Moving the declaration alone did not " +"solve the layout, but an earlier background function had revealed a naming " +"pattern that affected the compiler's choice. That gave us a more promising " +"hypothesis for {ph2}the reconstruction of {ph3}{ph4}." +msgstr "" +"Ví dụ: trong TH095, {ph0}ngay cả tên biến cục bộ cũng có thể ảnh hưởng đến " +"vị trí VC7.1 lưu trữ nó{ph1}. Việc di chuyển khai báo của nó không giải " +"quyết được bố cục lưu trữ. Tuy nhiên, hàm nền trước đó đã tiết lộ kiểu đặt " +"tên ảnh hưởng đến sự lựa chọn của trình biên dịch. Điều đó gợi ý một giả " +"thuyết hứa hẹn hơn cho việc {ph2}tái tạo {ph3}{ph4}." + +#: website/public/blog/reverse-engineering-models/index.html:396 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reusing that pattern for the file-size variable produced the required layout" +" without adding storage. The full function then matched." +msgstr "" +"Việc sử dụng lại mẫu đó cho biến kích thước tệp sẽ tạo ra bố cục cần thiết " +"mà không cần thêm bộ nhớ. Các chức năng đầy đủ sau đó phù hợp." + +#: website/public/blog/reverse-engineering-models/index.html:401 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is the sort of narrow detail a general model may not know when it " +"arrives. A specialist trained on this compiler could encounter many such " +"cases and learn which source variations are worth testing first." +msgstr "" +"Đây là một chi tiết rất cụ thể mà một mô hình AI đa năng có thể không biết " +"khi lần đầu gặp sự cố. Một mô hình chuyên dụng cho trình biên dịch này có " +"thể xem nhiều trường hợp như vậy và tìm hiểu xem biến thể mã nguồn nào đáng " +"để thử nghiệm trước tiên." + +#: website/public/blog/reverse-engineering-models/index.html:407 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The opportunity is especially clear in final matching problems where we " +"already understand the behavior. We are then asking which ordinary source " +"form this compiler turns into the observed instructions. Specialization " +"could help the model propose those forms more efficiently, while the target " +"still determines which hypotheses survive." +msgstr "" +"Cơ hội đặc biệt rõ ràng trong các bài toán so khớp cuối cùng khi chúng ta đã" +" hiểu rõ hành vi. Sau đó, chúng tôi sẽ hỏi trình biên dịch này sẽ chuyển sự " +"sắp xếp mã nguồn thông thường nào thành các lệnh máy được quan sát. Sự " +"chuyên môn hóa có thể giúp mô hình đề xuất các hình thức đó hiệu quả hơn, " +"trong khi mục tiêu vẫn xác định giả thuyết nào tồn tại." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:415 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The useful patterns would have to retain their context, though. TH08 uses an" +" earlier Visual C++ version than TH095 and TH09. TH04 uses a different DOS-" +"era compiler, and its {ph0}player reconstruction{ph1} needed different " +"hypotheses about source patterns and how the compiled code was combined into" +" an executable. Shared authorship gives us a prior, but each target tells us" +" whether it applies. Distinguishing those contexts is part of what a " +"specialist would need to learn." +msgstr "" +"Các mẫu hữu ích phải đi kèm bối cảnh áp dụng. TH08 dùng phiên bản Visual C++" +" cũ hơn TH095 và TH09. TH04 dùng một trình biên dịch khác thuộc thời DOS. " +"{ph0}Việc tái dựng mã điều khiển người chơi{ph1} cần những giả thuyết khác " +"về cách viết mã và cách liên kết thành tệp thực thi. Cùng tác giả là một giả" +" định ban đầu; từng mục tiêu cho biết giả định đó có phù hợp không. Mô hình " +"chuyên biệt cần học cách phân biệt các bối cảnh này." + +#: website/public/blog/reverse-engineering-models/index.html:431 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Where should that experience live?" +msgstr "Kinh nghiệm này nên được lưu trữ ở đâu?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:432 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before deciding to teach those patterns through model training, I would look" +" at the knowledge we already keep in the repository. In the {ph0}previous " +"essay{ph1}, I wrote about it becoming the project's working memory. That " +"memory includes compiler lessons, along with the source and failed " +"experiments that made them useful." +msgstr "" +"Trước khi quyết định dạy những mẫu đó thông qua đào tạo mô hình, tôi sẽ xem " +"xét kiến thức mà chúng tôi đã lưu giữ trong kho. Trong {ph0}bài luận " +"trước{ph1}, tôi đã viết về việc nó trở thành bộ nhớ làm việc của dự án. Bộ " +"nhớ đó bao gồm các bài học về trình biên dịch, cùng với nguồn và các thử " +"nghiệm thất bại khiến chúng trở nên hữu ích." + +#: website/public/blog/reverse-engineering-models/index.html:441 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"By retrieving that record, an agent can start with a known result and " +"inspect how we reached it. It can also avoid alternatives that have already " +"failed in the same context. The starting point improves without changing the" +" model's weights." +msgstr "" +"Bằng cách truy xuất bản ghi đó, nhân viên hỗ trợ có thể bắt đầu với kết quả " +"đã biết và kiểm tra cách chúng tôi đạt được kết quả đó. Nó cũng có thể tránh" +" được những lựa chọn thay thế đã thất bại trong cùng bối cảnh. Điểm bắt đầu " +"được cải thiện mà không thay đổi các tham số đã huấn luyện của mô hình." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:447 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"When reconstructing TH09's enemy-spawn function, a {ph0}GPT-6-Luna-max " +"session{ph1} drew a hypothesis from TH08's corresponding source family. That" +" earlier source suggested how to structure the loop and handle failures " +"within it. TH09's own evidence supplied the game-specific details, and clean" +" rebuilds confirmed the complete match." +msgstr "" +"Trong khi xây dựng lại chức năng tạo kẻ thù của TH09, phiên {ph0}GPT-6-Luna-" +"max{ph1} đã lấy một giả thuyết từ mã nguồn TH08 có liên quan. Mã trước đó đã" +" đề xuất cách tổ chức vòng lặp và xử lý các lỗi bên trong nó. Bằng chứng " +"riêng của TH09 đã cung cấp chi tiết cụ thể về trò chơi. Việc xây dựng lại từ" +" nguồn đã xác nhận rằng toàn bộ hàm khớp với bản gốc." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:458 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}\"The model did not know that compiler quirk\" and \"the model could " +"not use it\" are different problems.{ph1} Retrieval may solve the first, but" +" turning that fact into a useful candidate still takes work. And if the " +"candidate fails, the model must decide what the result means for its next " +"attempt. Knowing the relevant pattern is only the beginning of that process." +msgstr "" +"{ph0}\"Mô hình không biết lỗi của trình biên dịch\" và \"mô hình không thể " +"sử dụng nó\" là các vấn đề khác nhau.{ph1} Việc truy xuất có thể giải quyết " +"được vấn đề đầu tiên, nhưng việc biến thực tế đó thành một mã ứng viên hữu " +"ích vẫn cần nhiều công sức. Và nếu mã nguồn mã ứng viên không thành công, mô" +" hình phải quyết định kết quả có ý nghĩa gì cho lần thử tiếp theo. Biết mẫu " +"có liên quan chỉ là bước khởi đầu của quá trình đó." + +#: website/public/blog/reverse-engineering-models/index.html:468 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For that reason, I would include a smaller general model with good retrieval" +" in any comparison. Giving it the same library of experience would help us " +"identify what training adds: better candidate generation, better decisions " +"after feedback, or both." +msgstr "" +"Vì lý do đó, tôi sẽ đưa vào một mô hình AI có mục đích chung nhỏ hơn với khả" +" năng truy xuất tốt trong mọi so sánh. Cung cấp cho nó cùng một thư viện " +"kinh nghiệm sẽ giúp chúng tôi xác định nội dung đào tạo bổ sung: tạo ứng " +"viên tốt hơn, đưa ra quyết định tốt hơn sau khi phản hồi hoặc cả hai." + +#: website/public/blog/reverse-engineering-models/index.html:476 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Learning the next useful experiment" +msgstr "Học thí nghiệm hữu ích tiếp theo" + +#: website/public/blog/reverse-engineering-models/index.html:477 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those decisions after feedback are what I would most like to study. In " +"agentic reconstruction, the model proposes an explanation and tests it " +"against the program. Even when the candidate fails, the result can narrow " +"the remaining possibilities and suggest a more useful next experiment." +msgstr "" +"Những quyết định sau khi được phản hồi là điều tôi muốn nghiên cứu nhất. " +"Trong tái cấu trúc tác nhân, mô hình đề xuất một lời giải thích và kiểm tra " +"nó dựa trên chương trình. Ngay cả khi mã nguồn ứng viên không thành công, " +"kết quả có thể thu hẹp các khả năng còn lại và đề xuất thử nghiệm tiếp theo " +"hữu ích hơn." + +#: website/public/blog/reverse-engineering-models/index.html:484 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Consider the temporary-results example again. Its final source shows the " +"arrangement that worked, but the route to that source contains more " +"information. Each of the following experiments helped identify what still " +"needed to change." +msgstr "" +"Hãy xem xét lại ví dụ về kết quả tạm thời. Nguồn cuối cùng của nó cho thấy " +"sự sắp xếp đã có hiệu quả nhưng đường dẫn đến nguồn đó chứa nhiều thông tin " +"hơn. Mỗi thử nghiệm sau đây đã giúp xác định những gì vẫn cần thay đổi." + +#: website/public/blog/reverse-engineering-models/index.html:497 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Experiment" +msgstr "Thử nghiệm" + +#: website/public/blog/reverse-engineering-models/index.html:498 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What its result taught us" +msgstr "Kết quả của nó đã dạy chúng ta điều gì" + +#: website/public/blog/reverse-engineering-models/index.html:503 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Try a different way to copy the results" +msgstr "Hãy thử một cách khác để sao chép kết quả" + +#: website/public/blog/reverse-engineering-models/index.html:504 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The copy code changes, but the function still differs." +msgstr "Mã sao chép thay đổi nhưng chức năng vẫn khác." + +#: website/public/blog/reverse-engineering-models/index.html:509 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Use the returned values directly" +msgstr "Sử dụng trực tiếp các giá trị được trả về" + +#: website/public/blog/reverse-engineering-models/index.html:510 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Copying follows the original pattern, but temporary storage still differs." +msgstr "Sao chép theo mẫu ban đầu, nhưng lưu trữ tạm thời vẫn khác." + +#: website/public/blog/reverse-engineering-models/index.html:516 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the temporary results alive together" +msgstr "Giữ tất cả các biến tạm thời hợp lệ cùng một lúc" + +#: website/public/blog/reverse-engineering-models/index.html:517 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"They get separate storage, but one still overlaps a value used later for " +"collision checks." +msgstr "" +"Chúng có bộ lưu trữ riêng biệt, nhưng một bộ vẫn chồng lên một giá trị được " +"sử dụng sau này để kiểm tra va chạm." + +#: website/public/blog/reverse-engineering-models/index.html:523 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the collision-check value separate" +msgstr "Giữ riêng giá trị kiểm tra va chạm" + +#: website/public/blog/reverse-engineering-models/index.html:524 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A clean rebuild confirms the complete match." +msgstr "" +"Việc biên dịch lại rõ ràng từ nguồn xác nhận sự trùng khớp hoàn chỉnh." + +#: website/public/blog/reverse-engineering-models/index.html:529 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Selected experiments from the attack-function reconstruction above, grouped " +"by what they taught us." +msgstr "" +"Các thử nghiệm được chọn từ quá trình tái cấu trúc chức năng tấn công ở " +"trên, được nhóm theo những gì chúng đã dạy chúng ta." + +#: website/public/blog/reverse-engineering-models/index.html:533 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A training record that keeps this sequence can teach more than the final " +"source alone. It connects each proposed change to an observed result and the" +" next decision. Keeping ineffective variations matters too: they tell a " +"later agent which hypotheses have already been tested in that context." +msgstr "" +"Một bản ghi huấn luyện lưu giữ trình tự này có thể dạy nhiều hơn chỉ riêng " +"nguồn cuối cùng. Nó kết nối từng thay đổi được đề xuất với kết quả được quan" +" sát và quyết định tiếp theo. Việc giữ lại các biến thể không hiệu quả cũng " +"rất quan trọng: chúng cho tác nhân sau này biết những giả thuyết nào đã được" +" kiểm tra trong bối cảnh đó." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:540 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}I would want the model to learn how to use feedback.{ph1} A neutral " +"result is a reason to reconsider the variation; a changed mismatch may point" +" to another cause. When local edits stop being informative, the agent should" +" be able to revisit the source representation or inspect the evidence again." +msgstr "" +"{ph0}Tôi muốn mô hình học cách sử dụng phản hồi.{ph1} Kết quả trung lập là " +"lý do để xem xét lại biến thể; sự không phù hợp đã thay đổi có thể chỉ ra " +"một nguyên nhân khác. Khi các chỉnh sửa cục bộ không còn mang tính thông " +"tin, nhân viên sẽ có thể xem lại phần trình bày nguồn hoặc kiểm tra lại bằng" +" chứng." + +#: website/public/blog/reverse-engineering-models/index.html:549 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Learning to make those choices requires autonomy. Sometimes the useful next " +"step is inspecting another caller rather than compiling another edit. A " +"rigid sequence of required rewrites would remove that choice, along with " +"part of the behavior we want the model to learn." +msgstr "" +"Học cách đưa ra những lựa chọn đó đòi hỏi sự tự chủ. Đôi khi, bước tiếp theo" +" hữu ích là kiểm tra người gọi khác thay vì biên dịch một bản chỉnh sửa " +"khác. Một chuỗi các lần viết lại bắt buộc một cách cứng nhắc sẽ loại bỏ lựa " +"chọn đó, cùng với một phần hành vi mà chúng ta muốn mô hình học." + +#: website/public/blog/reverse-engineering-models/index.html:558 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why RE is interesting for reinforcement learning" +msgstr "Tại sao kỹ thuật đảo ngược lại thú vị cho việc học tăng cường" + +#: website/public/blog/reverse-engineering-models/index.html:559 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Bounded reconstruction gives us a practical environment for studying these " +"choices. Once the target and compiler are pinned, many source experiments " +"produce quick, repeatable feedback. The agent can compile a candidate, " +"inspect the mismatch and use that observation to choose its next step." +msgstr "" +"Tái dựng trong phạm vi xác định là môi trường thực tế để nghiên cứu các lựa " +"chọn này. Khi phiên bản của mục tiêu và trình biên dịch được cố định, nhiều " +"thử nghiệm mã nguồn cho phản hồi nhanh và có thể lặp lại. Tác nhân có thể " +"biên dịch mã ứng viên, xem khác biệt rồi chọn bước tiếp theo." + +#: website/public/blog/reverse-engineering-models/index.html:566 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reinforcement learning could train that decision-making process through the " +"feedback it receives. A candidate accepted by a defined check gives us a " +"concrete outcome, while the sequence of attempts lets us observe how much " +"work it took to reach it." +msgstr "" +"Học tăng cường có thể rèn luyện quá trình ra quyết định đó thông qua phản " +"hồi mà nó nhận được. Một mã ứng viên được chấp nhận bởi một cuộc kiểm tra " +"xác định sẽ cho chúng ta một kết quả cụ thể, trong khi trình tự các lần thử " +"cho phép chúng ta quan sát xem cần bao nhiêu công sức để đạt được kết quả " +"đó." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:572 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"There is already research supporting this direction. {ph0}LLM4Decompile{ph1}" +" studies models specialized for decompilation. {ph2}AutoDecompiler{ph3} " +"trains multi-turn refinement using compiler and execution feedback. Their " +"studies focus on functional recovery. For this work, I would use them as a " +"starting point for the related task of finding source that matches a " +"historical compiler's output." +msgstr "" +"Đã có nghiên cứu hỗ trợ hướng này. {ph0}LLM4Decompile{ph1} nghiên cứu các mô" +" hình chuyên dùng cho việc dịch ngược. {ph2}AutoDecompiler{ph3} đào tạo sàng" +" lọc nhiều lượt bằng cách sử dụng phản hồi của trình biên dịch và thực thi. " +"Nghiên cứu của họ tập trung vào phục hồi chức năng. Đối với công việc này, " +"tôi sẽ sử dụng chúng làm điểm khởi đầu cho nhiệm vụ liên quan là tìm nguồn " +"phù hợp với đầu ra của trình biên dịch lịch sử." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:584 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To reach such a well-defined task, though, we may first need to establish " +"what a function owns or devise a new runtime measurement. In TH08, for " +"example, a {ph0}collision difference led back to how the original program " +"handled floating-point calculations{ph1}. Before the agent could test a " +"useful correction, it had to work out what to measure." +msgstr "" +"Tuy nhiên, để đạt được nhiệm vụ được xác định rõ ràng như vậy, trước tiên " +"chúng ta có thể cần thiết lập tài nguyên nào mà hàm sở hữu hoặc nghĩ ra cách" +" mới để quan sát hành vi của chương trình khi chạy. Ví dụ: trong TH08, sự " +"khác biệt về va chạm {ph0} dẫn đến cách chương trình ban đầu xử lý các phép " +"tính dấu phẩy động{ph1}. Trước khi tác nhân có thể kiểm tra một sự điều " +"chỉnh hữu ích, nó phải tìm ra những gì cần đo lường." + +#: website/public/blog/reverse-engineering-models/index.html:595 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This suggests another role for a broadly capable model: helping make the " +"problem checkable in the first place. Once that check exists, we can " +"investigate how far a learned search policy can get with it." +msgstr "" +"Điều này gợi ý một vai trò khác của một mô hình có khả năng rộng rãi: giúp " +"làm cho vấn đề có thể được kiểm tra ngay từ đầu. Khi kiểm tra đó tồn tại, " +"chúng tôi có thể điều tra xem chính sách tìm kiếm đã học có thể đi được bao " +"xa." + +#: website/public/blog/reverse-engineering-models/index.html:603 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "When the oracle is wrong" +msgstr "Khi công cụ xác minh sai" + +#: website/public/blog/reverse-engineering-models/index.html:604 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Once we train against that check, its coverage becomes part of what the " +"model learns. I call the reference or checker an oracle. Even if it gives a " +"repeatable answer, we still need to understand what that answer covers." +msgstr "" +"Khi dùng một phép kiểm tra để huấn luyện, phạm vi kiểm tra cũng trở thành " +"một phần của môi trường học. Tôi gọi chuẩn tham chiếu hoặc công cụ kiểm tra " +"đó là oracle. Ngay cả câu trả lời lặp lại được cũng cần được hiểu theo những" +" gì nó thực sự xác minh." + +#: website/public/blog/reverse-engineering-models/index.html:610 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"TH08 showed how much can hide outside that coverage. Some reconstructed " +"functions passed their exact comparisons with wrong floating constants. The " +"comparator adjusted relocated addresses before checking instructions, but it" +" checked the values behind those addresses only when they had been " +"explicitly recorded for verification. Other literal references could pass " +"without their values being examined." +msgstr "" +"TH08 cho thấy vấn đề có thể ẩn ngoài phạm vi kiểm tra. Một số hàm tái dựng " +"dùng hằng số dấu phẩy động sai nhưng vẫn vượt qua so sánh chính xác. Bộ so " +"sánh điều chỉnh địa chỉ tái định vị trước khi kiểm tra lệnh máy, nhưng chỉ " +"kiểm tra giá trị tại các địa chỉ đó nếu chúng đã được ghi nhận rõ ràng để " +"xác minh. Những tham chiếu hằng số khác có thể vượt qua mà không được kiểm " +"tra giá trị." + +#. {ph0}: +#. {ph1}: +#. {ph2}: 1.0f +#. {ph3}: 32.0f +#: website/public/blog/reverse-engineering-models/index.html:619 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To close that gap, a {ph0}wider audit{ph1} found incorrect constants in " +"functions that had already been accepted. In one calculation, the " +"reconstructed source added {ph2} where the original added {ph3}." +msgstr "" +"Để giải quyết khoảng trống này, {ph0}kiểm tra rộng hơn{ph1} đã tìm thấy các " +"hằng số không chính xác trong các hàm đã vượt qua quá trình xác minh. Trong " +"một phép tính, mã nguồn được xây dựng lại đã thêm {ph2}, trong khi mã nguồn " +"ban đầu được thêm {ph3}." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:629 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A learner rewarded by the old check could learn to produce the wrong " +"reconstruction.{ph1} We had not trained an RL model against this oracle; the" +" observed failure was in the check itself. But using that check as a reward " +"would have made the same blind spot part of the learning environment." +msgstr "" +"{ph0}Mô hình được khen thưởng bằng kiểm tra xác minh cũ có thể học cách tạo " +"ra bản dựng lại không chính xác.{ph1} Chúng tôi không đào tạo mô hình RL " +"bằng trình kiểm tra đó. Sự thất bại mà chúng tôi quan sát thấy nằm ở chính " +"việc kiểm tra. Nhưng nếu chúng ta sử dụng nó như một phần thưởng thì điểm mù" +" đó sẽ trở thành một phần của môi trường học tập." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:639 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A functional test can have a similar blind spot. {ph0}Differential-fuzzing " +"research on LLM decompilers{ph1} has found behavior differences beyond the " +"supplied tests. Passing those tests therefore needs to be read in terms of " +"the cases they exercise, just as an instruction match needs to be read in " +"terms of the bytes and data it checks." +msgstr "" +"Một bài kiểm tra chức năng có thể có điểm mù tương tự. {ph0}Nghiên cứu sử " +"dụng tính năng làm mờ vi phân trên bộ giải mã LLM{ph1} đã tìm thấy những " +"khác biệt về hành vi bên ngoài các thử nghiệm được cung cấp. Do đó, chúng ta" +" cần hiểu một bài kiểm tra đạt theo các trường hợp mà nó thực hiện. Tương " +"tự, một lệnh khớp phải được hiểu theo byte và dữ liệu mà nó xác minh." + +#: website/public/blog/reverse-engineering-models/index.html:649 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before using either kind of check as a reward, I would test it with " +"deliberate failures. I would also keep final acceptance outside the " +"candidate's editable source, so the model has to satisfy the check through " +"its reconstruction. Otherwise, something like a byte array copied from the " +"target could satisfy a naive score without recovering useful source." +msgstr "" +"Trước khi sử dụng một trong hai loại kiểm tra xác minh làm phần thưởng, tôi " +"sẽ kiểm tra nó với kết quả cố tình sai. Tôi cũng sẽ tách biệt sự chấp nhận " +"cuối cùng với mã nguồn mà tác nhân AI có thể chỉnh sửa. Mô hình phải vượt " +"qua quá trình xác minh bằng cách xây dựng lại chương trình một cách chính " +"xác. Mặt khác, chỉ cần sao chép một mảng byte từ mục tiêu có thể đánh lừa " +"một hệ thống tính điểm ngây thơ mà không khôi phục được mã nguồn hữu ích." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:657 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Improving the learner and improving its oracle belong to the same " +"research problem.{ph1} As the learner gets better at optimizing the feedback" +" we give it, understanding that feedback becomes more important too." +msgstr "" +"{ph0}Cải thiện mô hình học và cải thiện công cụ xác minh là hai phần của " +"cùng một vấn đề nghiên cứu.{ph1} Mô hình càng giỏi tối ưu phản hồi được cung" +" cấp, chúng ta càng cần hiểu chính phản hồi đó." + +#: website/public/blog/reverse-engineering-models/index.html:667 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A comparison worth running" +msgstr "Một so sánh đáng chạy" + +#: website/public/blog/reverse-engineering-models/index.html:668 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"With those checks in place, I would start the comparison on a fixed VC7.1 " +"task family. First, I would establish baselines for a strong general model " +"and a smaller one. Adding repository knowledge to the smaller model would " +"show what retrieval changes. From there, supervised specialization and an " +"RL-trained version would let us examine what training adds." +msgstr "" +"Với những bước kiểm tra đó, tôi sẽ bắt đầu so sánh trên nhóm nhiệm vụ VC7.1 " +"cố định. Đầu tiên, tôi sẽ thiết lập các đường cơ sở cho một mô hình AI có " +"mục đích chung mạnh mẽ và một mô hình nhỏ hơn. Việc bổ sung kiến ​​thức về " +"kho lưu trữ vào mô hình nhỏ hơn sẽ cho thấy những thay đổi trong quá trình " +"truy xuất. Từ đó, chuyên môn hóa được giám sát và phiên bản được đào tạo RL " +"sẽ cho phép chúng tôi kiểm tra nội dung đào tạo bổ sung." + +#: website/public/blog/reverse-engineering-models/index.html:676 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To make those comparisons informative, each model would receive the same " +"target evidence and tools. For the training-specific comparisons, I would " +"keep the base model and available knowledge fixed as well. Repeated trials " +"would help distinguish a reliable search strategy from a lucky candidate." +msgstr "" +"Để làm cho những so sánh đó trở nên hữu ích, mỗi mô hình sẽ nhận được các " +"bằng chứng và công cụ mục tiêu giống nhau. Đối với các so sánh dành riêng " +"cho đào tạo, tôi cũng sẽ giữ nguyên mô hình cơ sở và kiến ​​thức có sẵn. Các" +" thử nghiệm lặp đi lặp lại sẽ giúp phân biệt một chiến lược tìm kiếm đáng " +"tin cậy với một giải pháp ứng viên tình cờ thành công." + +#: website/public/blog/reverse-engineering-models/index.html:683 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would then separate two starting conditions: reconstructing a function " +"from its instructions, and closing the last few differences in maintained " +"source. A specialist might be excellent at the second task while a general " +"model does much of the work needed to reach it. Measuring them separately " +"would reveal that division of work." +msgstr "" +"Sau đó, tôi sẽ tách hai điều kiện bắt đầu: xây dựng lại một hàm từ các lệnh " +"máy của nó và đóng một số khác biệt cuối cùng trong nguồn được duy trì. Một " +"mô hình AI chuyên dụng có thể thực hiện xuất sắc nhiệm vụ thứ hai trong khi " +"một mô hình AI có mục đích chung thực hiện phần lớn công việc cần thiết để " +"đạt được nhiệm vụ đó. Việc đo lường chúng một cách riêng biệt sẽ tiết lộ sự " +"phân chia công việc đó." + +#: website/public/blog/reverse-engineering-models/index.html:691 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both tasks need genuinely held-out examples. Because related Touhou games " +"share engine code, holding out a game's name would still allow nearly " +"identical functions to appear in training. I would instead hold out source " +"families and keep their accepted answers out of both training and retrieval." +msgstr "" +"Cả hai nhiệm vụ đều cần loại trừ các mẫu thử nghiệm khỏi quá trình đào tạo " +"và truy xuất. Trò chơi Touhou liên quan chia sẻ mã công cụ. Việc loại trừ " +"một trò chơi khỏi quá trình huấn luyện vẫn có thể để lại các chức năng gần " +"như giống hệt nhau trong tập huấn luyện. Thay vào đó, tôi sẽ loại trừ toàn " +"bộ các nhóm mã nguồn có liên quan và loại bỏ các câu trả lời đã được xác " +"minh của chúng khỏi cả dữ liệu huấn luyện và thư viện truy xuất." + +#: website/public/blog/reverse-engineering-models/index.html:698 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"On those examples, the outcome would be complete, independently validated " +"matches. Giving each model the same candidate or compile budget would help " +"compare search quality. Measuring elapsed time and total cost, including " +"compilation, would tell us whether that quality produces an efficient " +"workflow. Training costs would need their own accounting too." +msgstr "" +"Kết quả cần đo là các lần khớp hoàn chỉnh, được xác minh độc lập. Giới hạn " +"số mã ứng viên hoặc số lần biên dịch bằng nhau giúp so sánh chất lượng tìm " +"kiếm. Đo tổng thời gian và chi phí, kể cả biên dịch, cho biết chất lượng đó " +"có tạo ra quy trình hiệu quả không. Chi phí huấn luyện cần tính riêng." + +#: website/public/blog/reverse-engineering-models/index.html:706 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both views matter because a cheaper proposal can become an expensive " +"workflow if it causes more uninformative compiles. Conversely, a slower " +"model may pay for itself by finding a useful experiment sooner. The " +"comparison should capture those costs." +msgstr "" +"Cả hai phép đo đều quan trọng. Một đề xuất rẻ tiền có thể dẫn đến một quy " +"trình làm việc tốn kém nếu nó tạo ra nhiều lần biên dịch không cung cấp " +"thông tin hữu ích. Ngược lại, một mô hình chậm hơn có thể tiết kiệm hơn nếu " +"nó tìm thấy một thử nghiệm hữu ích sớm hơn. Việc so sánh nên bao gồm những " +"chi phí này." + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:712 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Counting accepted results also keeps intermediate progress in perspective. " +"One TH09 {ph0}replay-save candidate{ph1} looked promising because the " +"compiled function had the expected size, but a full comparison still found " +"substantial differences. Changes in how the compiler arranges instructions " +"can also make a candidate look closer without resolving the underlying " +"problem. Such diagnostics can guide the next experiment, but I would " +"validate any reward for \"getting closer\" against the full acceptance " +"check." +msgstr "" +"Việc đếm các kết quả được chấp nhận cũng giúp duy trì tiến độ trung gian. " +"Một TH09 {ph0}mã ứng viên lưu lại phát lại{ph1} có vẻ đầy hứa hẹn vì hàm " +"được biên dịch có kích thước như mong đợi nhưng so sánh đầy đủ vẫn tìm thấy " +"những khác biệt đáng kể. Những thay đổi trong cách trình biên dịch sắp xếp " +"các lệnh máy cũng có thể khiến mã ứng viên có vẻ gần hơn mà không giải quyết" +" được vấn đề cơ bản. Chẩn đoán như vậy có thể hướng dẫn thử nghiệm tiếp " +"theo, nhưng tôi sẽ xác thực bất kỳ phần thưởng nào cho việc \"tiến gần hơn\"" +" so với kiểm tra chấp nhận hoàn toàn." + +#: website/public/blog/reverse-engineering-models/index.html:727 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would be interested in the pattern across these results. If a specialist " +"does well within a familiar compiler family, does that advantage survive a " +"toolchain change? And once both models have good retrieval, how much does " +"training still add? The answers would help us decide where each kind of " +"model is most useful." +msgstr "" +"Tôi sẽ quan tâm đến mô hình trên các kết quả này. Nếu một mô hình AI chuyên " +"dụng hoạt động tốt trong dòng trình biên dịch quen thuộc, liệu lợi thế đó có" +" tồn tại được khi thay đổi chuỗi công cụ không? Và một khi cả hai mô hình AI" +" đều có khả năng truy xuất tốt thì việc đào tạo vẫn bổ sung thêm bao nhiêu? " +"Câu trả lời sẽ giúp chúng ta quyết định xem mỗi loại mô hình sẽ hữu ích nhất" +" ở đâu." + +#: website/public/blog/reverse-engineering-models/index.html:736 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What kind of agent does this suggest?" +msgstr "Điều này gợi ý loại tác nhân AI nào?" + +#: website/public/blog/reverse-engineering-models/index.html:737 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"If the models turn out to help at different stages, a useful arrangement " +"might be a broad agent with access to a specialist. Once a task becomes a " +"familiar compiler search, it could ask the specialist for candidates. If the" +" results stop making sense, the broader agent could return to the target " +"evidence or investigate the measurement itself." +msgstr "" +"Nếu các mô hình tỏ ra hữu ích ở các giai đoạn khác nhau, thì sự sắp xếp hữu " +"ích có thể là một tác nhân rộng rãi có quyền truy cập vào một mô hình AI " +"chuyên dụng. Khi một nhiệm vụ trở thành một công cụ tìm kiếm trình biên dịch" +" quen thuộc, nó có thể yêu cầu mô hình AI chuyên biệt dành cho ứng viên. Nếu" +" kết quả không còn có ý nghĩa nữa, tác nhân rộng hơn có thể quay lại bằng " +"chứng mục tiêu hoặc tự điều tra phép đo." + +#: website/public/blog/reverse-engineering-models/index.html:745 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In that arrangement, specialization becomes a tool the agent can choose as " +"the investigation develops. The agent keeps the freedom to change direction " +"that made the earlier breakthroughs possible." +msgstr "" +"Trong sự sắp xếp đó, sự chuyên môn hóa trở thành một công cụ mà đặc vụ có " +"thể lựa chọn khi cuộc điều tra phát triển. Người đại diện có quyền tự do " +"thay đổi hướng đi để có thể đạt được những đột phá trước đó." + +#: website/public/blog/reverse-engineering-models/index.html:750 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This follows naturally from the industrial shift I described in the previous" +" essay. Being able to run many more experiments changes how we organize the " +"work. It also gives us a reason to ask which decisions need broad judgment " +"and which improve through repeated experience with a particular craft." +msgstr "" +"Điều này diễn ra một cách tự nhiên từ sự chuyển dịch công nghiệp mà tôi đã " +"mô tả trong bài viết trước. Việc có thể chạy nhiều thử nghiệm hơn sẽ thay " +"đổi cách chúng ta tổ chức công việc. Nó cũng cho chúng ta lý do để hỏi những" +" quyết định nào cần được đánh giá rộng rãi và quyết định nào được cải thiện " +"thông qua kinh nghiệm lặp đi lặp lại với một nghề cụ thể." + +#: website/public/blog/reverse-engineering-models/index.html:757 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For now, the strongest general model has earned its place in my workflow by " +"opening problems I had left unresolved. Those successes are what make me " +"want to examine the work more closely: how much depended on the model's " +"breadth, and how much could be handled by a model trained in this particular" +" kind of reconstruction?" +msgstr "" +"Hiện tại, mô hình tổng quát mạnh nhất đã có chỗ đứng trong quy trình của tôi" +" vì giải quyết được những vấn đề trước đó còn bỏ ngỏ. Những thành công này " +"khiến tôi muốn xem kỹ quá trình: bao nhiêu phần phụ thuộc vào năng lực rộng " +"của mô hình, và bao nhiêu phần có thể giao cho một mô hình được huấn luyện " +"riêng cho kiểu tái dựng này?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:765 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}The repositories give us a way to test the question.{ph1} I would like " +"to use that record to build the comparison and see what we learn." +msgstr "" +"{ph0}Các kho lưu trữ cung cấp cho chúng tôi một cách để kiểm tra câu " +"hỏi.{ph1} Tôi muốn sử dụng hồ sơ đó để xây dựng sự so sánh và xem những gì " +"chúng tôi học được." + +#: website/public/blog/reverse-engineering-models/index.html:775 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Evidence and model attribution" +msgstr "Bằng chứng và quy kết mô hình" + +#: website/public/blog/reverse-engineering-models/index.html:783 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The examples describe historical function-level comparisons recorded in the " +"linked commits. Whole-game runtime checks are separate milestones. Source " +"and matching records were reviewed on 13 October 2026." +msgstr "" +"Các ví dụ mô tả các so sánh cấp độ chức năng lịch sử được ghi lại trong các " +"cam kết được liên kết. Kiểm tra thời gian chạy toàn bộ trò chơi là các cột " +"mốc riêng biệt. Nguồn và hồ sơ trùng khớp đã được xem xét vào ngày 13 tháng " +"10 năm 2026." + +#. {ph0}: gpt-dots +#. {ph1}: gpt-web +#: website/public/blog/reverse-engineering-models/index.html:789 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The TH09 label {ph0} identifies GPT-6-Astra. {ph1} changed with the model " +"used at the time, so that label alone does not identify an individual run. " +"Explicit model names in the other cited commits establish their recorded " +"attribution; the histories do not contain uniform per-run cost or effort " +"measurements." +msgstr "" +"Nhãn TH09 {ph0} có nghĩa là GPT-6-Astra. Mô hình được liên kết với {ph1} đã " +"thay đổi theo thời gian, do đó, riêng nhãn đó không thể xác định được mô " +"hình được sử dụng trong một lần chạy cụ thể. Tên mô hình rõ ràng trong các " +"cam kết được trích dẫn khác cung cấp thông tin ghi nhận. Những lịch sử này " +"không bao gồm các phép đo nhất quán về chi phí hoặc nỗ lực cho mỗi lần chạy." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#: website/public/blog/reverse-engineering-models/index.html:798 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 temporary lifetimes:{ph1} the {ph2}earlier context review{ph3} and" +" {ph4}complete cold reconstruction{ph5}." +msgstr "" +"{ph0}TH09 thời gian tồn tại của đối tượng tạm thời:{ph1} {ph2}xem xét ngữ " +"cảnh trước đó{ph3} và {ph4} việc xây dựng lại hoàn chỉnh được xác minh bằng " +"cách biên dịch lại rõ ràng từ nguồn{ph5}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:810 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 Front callback:{ph1} {ph2}non-exact diagnostic regions{ph3}, " +"{ph4}Astra's new graph{ph5} and its {ph6}accepted recompile replay{ph7}." +msgstr "" +"{ph0}Callback Front của TH09:{ph1} {ph2}các vùng chẩn đoán chưa khớp{ph3}, " +"{ph4}đồ thị luồng điều khiển mới của Astra{ph5} và {ph6}phép kiểm tra biên " +"dịch lại đã được chấp nhận{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:826 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Compiler knowledge:{ph1} {ph2}TH095 identifier-bucket reuse{ph3}, " +"{ph4}TH04 player source and linker order{ph5} and {ph6}Luna's TH09 enemy-" +"spawn closure{ph7}." +msgstr "" +"{ph0}Kiến thức về trình biên dịch:{ph1} {ph2}TH095 tái sử dụng hành vi của " +"nhóm nhận dạng{ph3}, {ph4}TH04 mã nguồn người chơi và thứ tự trình liên " +"kết{ph5} và {ph6}Luna tái thiết hoàn toàn chức năng tạo kẻ thù của " +"TH09{ph7}." + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:842 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Checks and diagnostics:{ph1} {ph2}TH08 floating-literal audit{ph3}, " +"{ph4}TH08 numerical investigation{ph5} and {ph6}TH09 replay-save " +"reconstruction{ph7}." +msgstr "" +"{ph0}Kiểm tra và chẩn đoán:{ph1} {ph2}kiểm tra hằng số dấu phẩy động của " +"TH08{ph3}, {ph4}điều tra số học của TH08{ph5} và {ph6}tái dựng chức năng lưu" +" replay của TH09{ph7}." + +#: website/public/blog/reverse-engineering-models/index.html:859 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The proposed specialist and RL comparisons are future experiments. The " +"project histories motivate them; no trained specialist was evaluated for " +"this essay." +msgstr "" +"Những so sánh được đề xuất giữa các mô hình chuyên dụng và chương trình đào " +"tạo RL là những thử nghiệm trong tương lai. Lịch sử dự án thúc đẩy họ. Không" +" có mô hình chuyên ngành được đào tạo nào được đánh giá cho bài tiểu luận " +"này." + +#: website/public/blog/reverse-engineering-models/index.html:6 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model? | REA" +msgstr "Kỹ thuật đảo ngược có cần một mô hình lớn hơn không? | REA" + +#: website/public/blog/reverse-engineering-models/index.html:7 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What Touhou reconstruction taught me about general models, specialist models" +" and learning to propose better hypotheses from compiler feedback." +msgstr "" +"Việc tái xây dựng Touhou đã dạy tôi điều gì về các mô hình chung, mô hình " +"chuyên biệt và việc học cách đề xuất các giả thuyết tốt hơn từ phản hồi của " +"trình biên dịch." + +#: website/public/blog/reverse-engineering-models/index.html:240 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "On this page" +msgstr "Trên trang này" + +#: website/public/blog/reverse-engineering-models/index.html:253 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Models and reverse engineering essay" +msgstr "Mô hình và tiểu luận kỹ thuật đảo ngược" + +#: website/public/blog/reverse-engineering-models/index.html:491 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Selected reconstruction experiments" +msgstr "Các thí nghiệm tái thiết được chọn" + +#: website/public/blog/reverse-engineering-models/index.html:883 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Back to top" +msgstr "Về đầu trang" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:776 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Detailed evidence notes{ph1} include the recorded comparisons and " +"experiment context." +msgstr "" +"{ph0}Ghi chú bằng chứng chi tiết{ph1} bao gồm các kết quả so sánh và bối " +"cảnh thử nghiệm." + +#~ msgctxt "page:blog/index.html" +#~ msgid "By N0zoM1z0 · 13 October 2026 · Draft for review" +#~ msgstr "N0zoM1z0 · 13 tháng 10 năm 2026 · Bản nháp để góp ý" + #~ msgctxt "page:index.html" #~ msgid "Setup guide" #~ msgstr "Hướng dẫn cài đặt" diff --git a/website/locales/zh-CN/messages.po b/website/locales/zh-CN/messages.po index b9350e625..9a4933ad7 100644 --- a/website/locales/zh-CN/messages.po +++ b/website/locales/zh-CN/messages.po @@ -32,43 +32,44 @@ msgid "Thoughts on reverse engineering." msgstr "逆向工程的思考。" #. {ph0}:
-#: website/public/blog/index.html:200 +#: website/public/blog/index.html:214 #, python-brace-format msgctxt "page:blog/index.html" msgid "Game reconstruction{ph0}Essay" msgstr "游戏重建{ph0}论文" -#: website/public/blog/index.html:202 +#: website/public/blog/index.html:216 msgctxt "page:blog/index.html" msgid "The industrial revolution in reverse engineering" msgstr "逆向工程的工业革命" -#: website/public/blog/index.html:203 +#: website/public/blog/index.html:217 msgctxt "page:blog/index.html" msgid "" "What manual RE and reconstructing TH08, TH095 and TH04 taught me about " "agents, verification and knowledge that compounds across projects." msgstr "什么手动重新和重建TH08,TH095和TH04教我关于编程助手,验证和知识,跨项目的化合物。" -#: website/public/blog/index.html:208 +#: website/public/blog/index.html:207 website/public/blog/index.html:222 msgctxt "page:blog/index.html" msgid "Read the article" msgstr "阅读文章" -#: website/public/blog/index.html:211 +#: website/public/blog/index.html:225 msgctxt "page:blog/index.html" msgid "By N0zoM1z0 · 10 October 2026" msgstr "通过N0zoM1z0*10十月2026" #. {ph0}: -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 +#: website/public/blog/reverse-engineering-models/index.html:883 #: website/public/blog/touhou-reconstruction/index.html:1080 #: website/public/examples/dino-lab/index.html:475 #: website/public/faq/index.html:573 #: website/public/first-investigation/index.html:523 #: website/public/get-started/index.html:882 -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 #: website/public/guides/index.html:282 #: website/public/guides/javascript/index.html:503 #: website/public/guides/native/index.html:509 website/public/index.html:1191 @@ -96,6 +97,8 @@ msgid "" msgstr "对逆向工程的思考,借鉴与 REA 和AI编程助手。" #: website/public/blog/index.html:75 website/public/blog/index.html:88 +#: website/public/blog/reverse-engineering-models/index.html:75 +#: website/public/blog/reverse-engineering-models/index.html:88 #: website/public/blog/touhou-reconstruction/index.html:75 #: website/public/blog/touhou-reconstruction/index.html:88 #: website/public/examples/dino-lab/index.html:75 @@ -130,6 +133,7 @@ msgid "" msgstr "REA-了解软件的工作原理。 逆向工程与你的编程助手." #: website/public/blog/index.html:108 +#: website/public/blog/reverse-engineering-models/index.html:108 #: website/public/blog/touhou-reconstruction/index.html:108 #: website/public/examples/dino-lab/index.html:109 #: website/public/faq/index.html:108 @@ -150,6 +154,7 @@ msgid "REA home" msgstr "REA 首页" #: website/public/blog/index.html:112 +#: website/public/blog/reverse-engineering-models/index.html:112 #: website/public/blog/touhou-reconstruction/index.html:112 #: website/public/examples/dino-lab/index.html:113 #: website/public/faq/index.html:112 @@ -170,6 +175,7 @@ msgid "Main navigation" msgstr "主导航" #: website/public/blog/index.html:130 +#: website/public/blog/reverse-engineering-models/index.html:130 #: website/public/blog/touhou-reconstruction/index.html:130 #: website/public/examples/dino-lab/index.html:131 #: website/public/faq/index.html:130 @@ -190,6 +196,7 @@ msgid "REA on GitHub" msgstr "在 GitHub 查看 REA" #: website/public/blog/index.html:150 +#: website/public/blog/reverse-engineering-models/index.html:150 #: website/public/blog/touhou-reconstruction/index.html:150 #: website/public/examples/dino-lab/index.html:151 #: website/public/faq/index.html:150 @@ -210,6 +217,7 @@ msgid "Join REA on Discord" msgstr "加入 REA 的 Discord 社区" #: website/public/blog/index.html:169 +#: website/public/blog/reverse-engineering-models/index.html:169 #: website/public/blog/touhou-reconstruction/index.html:169 #: website/public/examples/dino-lab/index.html:170 #: website/public/faq/index.html:169 @@ -229,7 +237,7 @@ msgctxt "shared" msgid "Follow REA on X" msgstr "在 X 上关注 REA" -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 msgctxt "page:blog/index.html" msgid "Back to top" msgstr "回到顶部" @@ -4237,7 +4245,7 @@ msgctxt "page:guides/browser/index.html" msgid "Copy read the loaded scripts" msgstr "复制读取加载的脚本" -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 msgctxt "page:guides/browser/index.html" msgid "Back to top" msgstr "回到顶部" @@ -10615,6 +10623,7 @@ msgid "Same spacing: 256 ÷ 16 = 16 angle units = 22.5°" msgstr "相同间距:256÷16=16角度单位=22.5°" #: website/public/blog/index.html:114 +#: website/public/blog/reverse-engineering-models/index.html:114 #: website/public/blog/touhou-reconstruction/index.html:114 #: website/public/examples/dino-lab/index.html:115 #: website/public/faq/index.html:114 @@ -10635,6 +10644,7 @@ msgid "Showcases" msgstr "案例展示" #: website/public/blog/index.html:115 +#: website/public/blog/reverse-engineering-models/index.html:115 #: website/public/blog/touhou-reconstruction/index.html:115 #: website/public/examples/dino-lab/index.html:116 #: website/public/faq/index.html:115 @@ -10655,6 +10665,7 @@ msgid "Blog" msgstr "博客" #: website/public/blog/index.html:113 +#: website/public/blog/reverse-engineering-models/index.html:113 #: website/public/blog/touhou-reconstruction/index.html:113 #: website/public/examples/dino-lab/index.html:114 #: website/public/faq/index.html:113 @@ -10675,6 +10686,7 @@ msgid "Get started" msgstr "开始使用" #: website/public/blog/index.html:116 +#: website/public/blog/reverse-engineering-models/index.html:116 #: website/public/blog/touhou-reconstruction/index.html:116 #: website/public/examples/dino-lab/index.html:117 #: website/public/faq/index.html:116 @@ -10694,13 +10706,14 @@ msgctxt "shared" msgid "Guides" msgstr "指南" -#: website/public/blog/index.html:222 +#: website/public/blog/index.html:236 +#: website/public/blog/reverse-engineering-models/index.html:875 #: website/public/blog/touhou-reconstruction/index.html:1072 #: website/public/examples/dino-lab/index.html:467 #: website/public/faq/index.html:565 #: website/public/first-investigation/index.html:515 #: website/public/get-started/index.html:874 -#: website/public/guides/browser/index.html:490 +#: website/public/guides/browser/index.html:499 #: website/public/guides/index.html:274 #: website/public/guides/javascript/index.html:495 #: website/public/guides/native/index.html:501 website/public/index.html:1183 @@ -10714,13 +10727,14 @@ msgctxt "shared" msgid "Source" msgstr "源代码" -#: website/public/blog/index.html:223 +#: website/public/blog/index.html:237 +#: website/public/blog/reverse-engineering-models/index.html:876 #: website/public/blog/touhou-reconstruction/index.html:1073 #: website/public/examples/dino-lab/index.html:468 #: website/public/faq/index.html:566 #: website/public/first-investigation/index.html:516 #: website/public/get-started/index.html:875 -#: website/public/guides/browser/index.html:491 +#: website/public/guides/browser/index.html:500 #: website/public/guides/index.html:275 #: website/public/guides/javascript/index.html:496 #: website/public/guides/native/index.html:502 website/public/index.html:1184 @@ -10734,13 +10748,14 @@ msgctxt "shared" msgid "MIT license" msgstr "MIT 许可证" -#: website/public/blog/index.html:224 +#: website/public/blog/index.html:238 +#: website/public/blog/reverse-engineering-models/index.html:877 #: website/public/blog/touhou-reconstruction/index.html:1074 #: website/public/examples/dino-lab/index.html:469 #: website/public/faq/index.html:567 #: website/public/first-investigation/index.html:517 #: website/public/get-started/index.html:876 -#: website/public/guides/browser/index.html:492 +#: website/public/guides/browser/index.html:501 #: website/public/guides/index.html:276 #: website/public/guides/javascript/index.html:497 #: website/public/guides/native/index.html:503 website/public/index.html:1185 @@ -10754,13 +10769,14 @@ msgctxt "shared" msgid "Join the community" msgstr "加入社区" -#: website/public/blog/index.html:225 +#: website/public/blog/index.html:239 +#: website/public/blog/reverse-engineering-models/index.html:878 #: website/public/blog/touhou-reconstruction/index.html:1075 #: website/public/examples/dino-lab/index.html:470 #: website/public/faq/index.html:568 #: website/public/first-investigation/index.html:518 #: website/public/get-started/index.html:877 -#: website/public/guides/browser/index.html:493 +#: website/public/guides/browser/index.html:502 #: website/public/guides/index.html:277 #: website/public/guides/javascript/index.html:498 #: website/public/guides/native/index.html:504 website/public/index.html:1186 @@ -11666,6 +11682,1161 @@ msgctxt "runtime" msgid "Switch to dark mode" msgstr "切换到深色模式" +#. {ph0}: +#. {ph1}: +#: website/public/guides/browser/index.html:480 +#, python-brace-format +msgctxt "page:guides/browser/index.html" +msgid "" +"For HTTP request and response bodies, use {ph0}Fiddler Everywhere: SAZ " +"captures, MCP and public skills{ph1}. Export SAZ or HAR 1.2 for offline REA " +"capture Evidence, or connect Everywhere's MCP server to inspect captured " +"traffic with your agent." +msgstr "" +"要检查 HTTP 请求和响应正文,请使用{ph0}Fiddler Everywhere:SAZ 捕获、MCP 和公开技能{ph1}。导出 SAZ ? " +"HAR 1.2 可在 REA 中离线获取捕获 Evidence,也可以连接 Everywhere 的 MCP 服务器,让智能体检查已捕获的流量。" + +#. {ph0}:
+#: website/public/blog/index.html:200 +#, python-brace-format +msgctxt "page:blog/index.html" +msgid "Model specialization{ph0}Essay" +msgstr "专用模型{ph0}随笔" + +#: website/public/blog/index.html:202 +msgctxt "page:blog/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "逆向工程需要更大的模型吗?" + +#: website/public/blog/index.html:203 +msgctxt "page:blog/index.html" +msgid "" +"What Touhou reconstruction suggests about general models, specialist models " +"and learning from compiler feedback." +msgstr "从 Touhou 重建的经验,探讨通用模型、专用模型,以及如何从编译器反馈中学习。" + +#: website/public/blog/index.html:210 +msgctxt "page:blog/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "作者 N0zoM1z0 · 2026 年 10 月 13 日" + +#: website/public/blog/reverse-engineering-models/index.html:103 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Skip to content" +msgstr "跳转到内容" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:194 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Home{ph1} / {ph2}Blog{ph3} / Models and RE" +msgstr "{ph0}首页{ph1} / {ph2}博客{ph3} / 模型与逆向工程" + +#: website/public/blog/reverse-engineering-models/index.html:197 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Essay" +msgstr "随笔" + +# Reviewed for technical meaning and readability; preserve the distinction +# between historical evidence and proposed experiments. +#: website/public/blog/reverse-engineering-models/index.html:198 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "逆向工程需要更大的模型吗?" + +#: website/public/blog/reverse-engineering-models/index.html:199 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "作者 N0zoM1z0 · 2026 年 10 月 13 日" + +#: website/public/blog/reverse-engineering-models/index.html:201 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Takeaway" +msgstr "要点" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:203 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Model capability changes the hypotheses an agent can find.{ph1} Later " +"sessions have solved problems that earlier work left unresolved." +msgstr "{ph0}模型能力会影响 agent 能提出哪些假设。{ph1} 后来的工作解决了早期尝试未能解决的问题。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:211 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A familiar compiler gives specialization something concrete to " +"learn.{ph1} Some exact matches depend on narrow, reusable source patterns." +msgstr "{ph0}熟悉的编译器为专用模型提供了具体的学习对象。{ph1} 有些精确匹配依赖细小但可以复用的源码模式。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:218 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Give repository knowledge a fair comparison.{ph1} A smaller model with " +"good retrieval may already know enough to make progress." +msgstr "{ph0}也要公平评估代码仓库里的知识。{ph1} 较小的模型配合良好的检索,可能已经掌握了足以推进工作的知识。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:223 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Train the next useful experiment, and test its oracle.{ph1} The value " +"of feedback depends on what the checker actually verifies." +msgstr "{ph0}训练模型选择下一个有价值的实验,同时检验它使用的验证器。{ph1} 反馈是否有用,取决于验证器究竟检查了什么。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:230 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Measure accepted reconstruction against time and cost.{ph1} That is the" +" result I want from the whole agent." +msgstr "{ph0}结合时间和成本,评估通过验证的重建成果。{ph1} 这是我希望从整个 agent 工作流中获得的结果。" + +#: website/public/blog/reverse-engineering-models/index.html:241 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "In this article" +msgstr "在本文中" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:243 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What better models changed{ph1}" +msgstr "{ph0}更强的模型改变了什么{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:244 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Why specialize?{ph1}" +msgstr "{ph0}为什么考虑专用模型?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:245 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Where experience lives{ph1}" +msgstr "{ph0}这些经验应该存在哪里?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:246 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Learning the next experiment{ph1}" +msgstr "{ph0}学习选择下一个有价值的实验{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:247 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}RE and reinforcement learning{ph1}" +msgstr "{ph0}逆向工程与强化学习{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:248 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}When the oracle is wrong{ph1}" +msgstr "{ph0}当验证器出错时{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:249 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}A comparison worth running{ph1}" +msgstr "{ph0}值得做的对照实验{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:250 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What kind of agent?{ph1}" +msgstr "{ph0}这意味着怎样的 agent?{ph1}" + +#: website/public/blog/reverse-engineering-models/index.html:258 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What better models changed" +msgstr "更强的模型改变了什么" + +#: website/public/blog/reverse-engineering-models/index.html:259 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Since August 2026, I have been exploring how much reverse engineering an " +"agent can carry out with access to a program and a way to check its work. " +"Most of my experience comes from reconstructing games, especially Touhou. As" +" I have changed models, the repositories have kept a record of what each " +"session managed to recover." +msgstr "" +"从 2026 年 8 月开始,我一直在探索:如果给 agent " +"访问程序的能力,以及验证自身工作的手段,它能独立完成多少逆向工作?我的经验主要来自游戏重建,尤其是东方系列。随着我更换模型,代码仓库记录了每次工作还原出的内容。" + +#: website/public/blog/reverse-engineering-models/index.html:267 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of that work aims at exact reconstruction: we compile the maintained " +"source under a pinned historical toolchain and check whether it reproduces " +"the target function. When a later model succeeds where earlier work stalled," +" the source changes show what made that progress possible." +msgstr "" +"这些工作很大一部分追求精确重建:用固定版本的历史工具链编译我们维护的源码,检查它能否复现目标函数。当后来的模型解决了早期工作卡住的问题,我们可以从源码变化中看出,是什么促成了这次进展。" + +#: website/public/blog/reverse-engineering-models/index.html:274 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The temporary values had to stay alive together" +msgstr "临时值需要同时保持有效" + +#: website/public/blog/reverse-engineering-models/index.html:275 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of a TH09 attack function's logic had already been reconstructed, but " +"the compiler still stored temporary results differently from the original. " +"The remaining problem was how long those results needed to stay alive and " +"when their storage could be reused." +msgstr "" +"TH09 " +"中一个攻击函数的大部分逻辑已经还原,但编译器对临时结果的存储安排仍然与原版不同。剩下的问题在于:这些结果的生命周期应该多长,编译器何时可以复用它们的存储空间?" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:282 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A {ph0}GPT-6-Sol checkpoint{ph1} recorded this mismatch after testing " +"several ordinary source arrangements. When {ph2}GPT-6.1-Sol returned to the " +"problem{ph3}, it found a useful hypothesis about how long those temporary " +"values needed to stay alive." +msgstr "" +"{ph0}GPT-6-Sol 的一次工作记录{ph1}表明,尝试几种常见的源码写法后,差异仍未消除。后来,{ph2}GPT-6.1-Sol " +"重新研究这个问题{ph3},提出了一个有价值的假设:关键可能在于这些临时值的生命周期。" + +#: website/public/blog/reverse-engineering-models/index.html:296 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The function produced temporary results and copied them into animation " +"positions. In C++, a temporary result stays alive until the expression using" +" it finishes. Grouping the real operations into one expression kept those " +"results alive together, so the compiler could no longer reuse the same " +"storage for all of them." +msgstr "" +"这个函数会产生临时结果,再把它们复制到动画的位置数据中。在 C++ " +"中,临时结果会一直存在,直到使用它的表达式执行完毕。把实际操作放进同一个表达式,就能让这些结果同时存活,编译器也就不能为它们全部复用同一块存储空间。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:303 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"One result still shared storage with a value used later for collision " +"checks. Giving that value its own lifetime resolved the remaining overlap. " +"After several targeted experiments, {ph0}a clean rebuild confirmed that the " +"full function matched{ph1}." +msgstr "" +"其中一个结果仍然与稍后用于碰撞检查的值共享存储。给那个值单独的生命周期后,最后一处重叠也消失了。经过几次有针对性的实验,{ph0}一次干净的重新构建确认,整个函数都已匹配{ph1}。" + +#: website/public/blog/reverse-engineering-models/index.html:312 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What interests me is how the discovery and the tests worked together. The " +"lifetime hypothesis gave the agent a new direction; compiler feedback helped" +" it refine that hypothesis into a source arrangement that matched." +msgstr "" +"我感兴趣的是,发现与验证如何共同推进工作。生命周期的假设给 agent 指明了新方向;编译器的反馈又帮助它逐步调整源码,直到找到匹配的写法。" + +#: website/public/blog/reverse-engineering-models/index.html:318 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A different control-flow graph" +msgstr "换一种控制流结构" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:319 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A second TH09 function shows why that new direction can matter. Here, the " +"obstacle was how several branches shared their final work. " +"{ph0}GPT-6.1-Sol's earlier work{ph1} got parts of the function to match, but" +" the branches still joined up differently from the original." +msgstr "" +"TH09 的另一个函数说明了,新方向为什么重要。这次的难点是多个分支如何共享末尾的操作。{ph0}GPT-6.1-Sol " +"之前的工作{ph1}已经让函数的一些部分匹配,但分支汇合的方式仍与原版不同。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:330 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}GPT-6-Astra changed the surrounding control-flow graph{ph1}. The main " +"paths now led to a shared final update and return. In that source structure," +" the compiler combined the branches in the same way as the original. The " +"complete function matched, and a {ph2}separate rebuild check confirmed " +"it{ph3}." +msgstr "" +"{ph0}GPT-6-Astra " +"改写了外围的控制流图{ph1},让主要路径汇入共同的最终更新与返回。在这种源码结构下,编译器合并分支的方式与原版一致。整个函数实现了匹配,{ph2}另一次独立的重新构建也确认了这一结果{ph3}。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:342 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Changing the representation opened a more productive search space.{ph1}" +" Both cases give me a reason to take general model capability seriously: an " +"agent can keep trying local variations for a long time before finding a " +"better way to express the surrounding work." +msgstr "" +"{ph0}换一种表达方式,打开了更有价值的搜索空间。{ph1} 这两个例子让我认真看待通用模型的能力:agent " +"可能花很长时间尝试局部变化,才找到一种更好的整体写法。" + +#: website/public/blog/reverse-engineering-models/index.html:351 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"At the same time, the later sessions started with earlier source and " +"recorded failures. That accumulated work was part of their advantage. The " +"history lets us inspect the discoveries and their checks; a controlled " +"comparison would help separate the effect of the model from the experience " +"it inherited." +msgstr "" +"不过,后来的工作也继承了之前的源码和失败记录。这些积累本身就是优势的一部分。历史记录让我们可以检查发现及其验证过程;要区分模型能力和继承经验各自带来的帮助,还需要对照实验。" + +#: website/public/blog/reverse-engineering-models/index.html:358 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is where the question becomes interesting to me. Some of the progress " +"came from finding a new representation, while much of the remaining work " +"involved testing source forms against a familiar compiler. How much needs " +"broad reasoning ability, and how much could a model learn through repeated " +"experience with that compiler and code?" +msgstr "" +"这正是我觉得有趣的地方。有些进展来自新的表达方式,剩下的大量工作则是在熟悉的编译器上试验不同写法。其中有多少需要广泛的推理能力?又有多少可以通过反复接触这个编译器和这些代码来学会?" + +#: website/public/blog/reverse-engineering-models/index.html:368 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why might a specialist do well?" +msgstr "为什么专用模型可能做得很好?" + +#: website/public/blog/reverse-engineering-models/index.html:369 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Touhou gives us a concrete reason to expect that experience to help. Much of" +" the game code comes from the same author, ZUN, so recurring habits are a " +"reasonable starting assumption. The repositories back that up with useful " +"examples: related loops and helper arrangements have supplied hypotheses " +"across games." +msgstr "" +"东方系列给了我们一个具体的理由,去期待经验积累能发挥作用。大量游戏代码出自同一个作者 " +"ZUN,因此,把反复出现的编程习惯作为初始假设是合理的。代码仓库也提供了实际例子:相似的循环和辅助函数组织方式,曾为其他游戏的重建提供假设。" + +#: website/public/blog/reverse-engineering-models/index.html:376 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those source patterns become more specific when we include the compiler. " +"Under a particular version and set of options, small changes in source can " +"produce repeatable differences in machine code. Some of those differences " +"are surprisingly obscure." +msgstr "把编译器也考虑进去,这些源码模式就更加具体了。在固定版本和选项下,源码的小变化可能稳定地产生不同的机器码。其中一些规律藏得很深。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: Supervisor::LoadDat +#. {ph4}: +#: website/public/blog/reverse-engineering-models/index.html:382 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In TH095, for example, {ph0}even the name used for a local variable could " +"change where VC7.1 stored it{ph1}. Moving the declaration alone did not " +"solve the layout, but an earlier background function had revealed a naming " +"pattern that affected the compiler's choice. That gave us a more promising " +"hypothesis for {ph2}the reconstruction of {ph3}{ph4}." +msgstr "" +"以 TH095 为例,{ph0}连局部变量的名字,都可能改变 VC7.1 " +"为它安排的存储位置{ph1}。只移动声明并不能解决布局问题,但先前重建的一个背景函数揭示了会影响编译器选择的命名模式。这为{ph2}重建 " +"{ph3}{ph4}提供了更有希望的假设。" + +#: website/public/blog/reverse-engineering-models/index.html:396 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reusing that pattern for the file-size variable produced the required layout" +" without adding storage. The full function then matched." +msgstr "把这个模式用于表示文件大小的变量,不增加额外存储就得到了所需布局。随后,整个函数实现了匹配。" + +#: website/public/blog/reverse-engineering-models/index.html:401 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is the sort of narrow detail a general model may not know when it " +"arrives. A specialist trained on this compiler could encounter many such " +"cases and learn which source variations are worth testing first." +msgstr "这类细节非常具体,通用模型第一次接触时未必知道。而针对这个编译器训练的专用模型,可以接触许多类似案例,学会哪些源码变化值得优先尝试。" + +#: website/public/blog/reverse-engineering-models/index.html:407 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The opportunity is especially clear in final matching problems where we " +"already understand the behavior. We are then asking which ordinary source " +"form this compiler turns into the observed instructions. Specialization " +"could help the model propose those forms more efficiently, while the target " +"still determines which hypotheses survive." +msgstr "" +"当我们已经理解函数行为,只剩最后的匹配差异时,这个机会尤其明显。此时的问题是:这个编译器会把哪一种普通写法编译成眼前的指令?专用训练可能让模型更高效地提出这些写法,而最终由目标程序决定哪些假设成立。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:415 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The useful patterns would have to retain their context, though. TH08 uses an" +" earlier Visual C++ version than TH095 and TH09. TH04 uses a different DOS-" +"era compiler, and its {ph0}player reconstruction{ph1} needed different " +"hypotheses about source patterns and how the compiled code was combined into" +" an executable. Shared authorship gives us a prior, but each target tells us" +" whether it applies. Distinguishing those contexts is part of what a " +"specialist would need to learn." +msgstr "" +"但这些模式必须保留适用的上下文。TH08 使用的 Visual C++ 版本早于 TH095 和 TH09。TH04 用的是另一种 DOS " +"时代的编译器,其{ph0}玩家逻辑重建{ph1}需要不同的假设,既涉及源码写法,也涉及编译后的代码如何链接成可执行文件。同一作者为我们提供了先验,具体目标则告诉我们它是否适用。区分这些情况,也是专用模型需要学会的事。" + +#: website/public/blog/reverse-engineering-models/index.html:431 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Where should that experience live?" +msgstr "这些经验应该存在哪里?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:432 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before deciding to teach those patterns through model training, I would look" +" at the knowledge we already keep in the repository. In the {ph0}previous " +"essay{ph1}, I wrote about it becoming the project's working memory. That " +"memory includes compiler lessons, along with the source and failed " +"experiments that made them useful." +msgstr "" +"在决定通过训练让模型学会这些模式之前,我会先看看代码仓库里已有的知识。在{ph0}上一篇文章{ph1}中,我写过:仓库正在成为项目的工作记忆。这里保存着对编译器的认识,也保存着相关源码和失败实验,让这些经验有据可查。" + +#: website/public/blog/reverse-engineering-models/index.html:441 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"By retrieving that record, an agent can start with a known result and " +"inspect how we reached it. It can also avoid alternatives that have already " +"failed in the same context. The starting point improves without changing the" +" model's weights." +msgstr "" +"检索这些记录后,agent 可以从已知结果出发,看看我们是如何得到它的,也可以避开在相同条件下已经失败的方案。模型的参数没有改变,工作的起点却提高了。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:447 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"When reconstructing TH09's enemy-spawn function, a {ph0}GPT-6-Luna-max " +"session{ph1} drew a hypothesis from TH08's corresponding source family. That" +" earlier source suggested how to structure the loop and handle failures " +"within it. TH09's own evidence supplied the game-specific details, and clean" +" rebuilds confirmed the complete match." +msgstr "" +"重建 TH09 的敌人生成函数时,{ph0}GPT-6-Luna-max 的一次工作{ph1}从 TH08 " +"对应的源码中找到了假设。之前的写法提示了循环该如何组织,以及内部失败该如何处理。TH09 " +"自身的证据补上了游戏特有的细节,干净的重新构建确认了整个函数的匹配。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:458 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}\"The model did not know that compiler quirk\" and \"the model could " +"not use it\" are different problems.{ph1} Retrieval may solve the first, but" +" turning that fact into a useful candidate still takes work. And if the " +"candidate fails, the model must decide what the result means for its next " +"attempt. Knowing the relevant pattern is only the beginning of that process." +msgstr "" +"{ph0}“模型不知道这个编译器特性”和“模型不会利用它”,是两个不同的问题。{ph1} " +"检索可能解决前者,但把知识变成有价值的候选源码,仍然需要能力。如果候选方案失败了,模型还得判断这次结果对下一次尝试意味着什么。知道相关模式,只是这个过程的起点。" + +#: website/public/blog/reverse-engineering-models/index.html:468 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For that reason, I would include a smaller general model with good retrieval" +" in any comparison. Giving it the same library of experience would help us " +"identify what training adds: better candidate generation, better decisions " +"after feedback, or both." +msgstr "" +"所以,在任何对照实验中,我都会加入一个具备良好检索能力的小型通用模型。给它相同的经验库,才能看清训练究竟增加了什么:更好的候选方案、更好的反馈后决策,还是两者都有?" + +#: website/public/blog/reverse-engineering-models/index.html:476 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Learning the next useful experiment" +msgstr "学习选择下一个有价值的实验" + +#: website/public/blog/reverse-engineering-models/index.html:477 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those decisions after feedback are what I would most like to study. In " +"agentic reconstruction, the model proposes an explanation and tests it " +"against the program. Even when the candidate fails, the result can narrow " +"the remaining possibilities and suggest a more useful next experiment." +msgstr "" +"反馈之后如何决策,是我最想研究的部分。在 agent " +"驱动的重建中,模型提出解释,再用程序来验证。即使候选方案失败了,结果也可能缩小剩余的可能性,提示下一步更值得尝试什么。" + +#: website/public/blog/reverse-engineering-models/index.html:484 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Consider the temporary-results example again. Its final source shows the " +"arrangement that worked, but the route to that source contains more " +"information. Each of the following experiments helped identify what still " +"needed to change." +msgstr "再看前面的临时结果例子。最终源码展示了成功的写法,但得到它的过程包含更多信息。下面的每次实验,都帮助我们确定了还有什么需要改变。" + +#: website/public/blog/reverse-engineering-models/index.html:497 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Experiment" +msgstr "实验" + +#: website/public/blog/reverse-engineering-models/index.html:498 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What its result taught us" +msgstr "结果告诉了我们什么" + +#: website/public/blog/reverse-engineering-models/index.html:503 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Try a different way to copy the results" +msgstr "尝试另一种复制结果的方式" + +#: website/public/blog/reverse-engineering-models/index.html:504 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The copy code changes, but the function still differs." +msgstr "复制代码改变了,但函数仍有差异。" + +#: website/public/blog/reverse-engineering-models/index.html:509 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Use the returned values directly" +msgstr "直接使用返回值" + +#: website/public/blog/reverse-engineering-models/index.html:510 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Copying follows the original pattern, but temporary storage still differs." +msgstr "复制方式与原版一致,但临时存储布局仍有差异。" + +#: website/public/blog/reverse-engineering-models/index.html:516 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the temporary results alive together" +msgstr "让临时结果同时保持有效" + +#: website/public/blog/reverse-engineering-models/index.html:517 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"They get separate storage, but one still overlaps a value used later for " +"collision checks." +msgstr "它们得到了各自的存储空间,但其中一个仍与稍后用于碰撞检查的值重叠。" + +#: website/public/blog/reverse-engineering-models/index.html:523 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the collision-check value separate" +msgstr "让碰撞检查的值拥有独立存储" + +#: website/public/blog/reverse-engineering-models/index.html:524 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A clean rebuild confirms the complete match." +msgstr "干净的重新构建确认了整个函数的匹配。" + +#: website/public/blog/reverse-engineering-models/index.html:529 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Selected experiments from the attack-function reconstruction above, grouped " +"by what they taught us." +msgstr "选自前面攻击函数的重建实验,按每次得到的认识整理。" + +#: website/public/blog/reverse-engineering-models/index.html:533 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A training record that keeps this sequence can teach more than the final " +"source alone. It connects each proposed change to an observed result and the" +" next decision. Keeping ineffective variations matters too: they tell a " +"later agent which hypotheses have already been tested in that context." +msgstr "" +"保存这个过程的训练记录,比单独保存最终源码能教会模型更多东西。它把每个改动、观察到的结果和下一步决策联系起来。无效的尝试也值得保留:后来的 agent " +"可以据此知道,在这个上下文中,哪些假设已经检验过。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:540 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}I would want the model to learn how to use feedback.{ph1} A neutral " +"result is a reason to reconsider the variation; a changed mismatch may point" +" to another cause. When local edits stop being informative, the agent should" +" be able to revisit the source representation or inspect the evidence again." +msgstr "" +"{ph0}我希望模型学会如何利用反馈。{ph1} " +"结果没有变化,可能意味着该重新考虑这个改动;差异发生变化,则可能指向另一个原因。当局部修改不再提供有效信息时,agent " +"应该可以重新考虑源码的整体表达,或者回头检查证据。" + +#: website/public/blog/reverse-engineering-models/index.html:549 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Learning to make those choices requires autonomy. Sometimes the useful next " +"step is inspecting another caller rather than compiling another edit. A " +"rigid sequence of required rewrites would remove that choice, along with " +"part of the behavior we want the model to learn." +msgstr "" +"学会这些选择,需要自主性。有时,更有价值的下一步是检查另一个调用者,而不是再修改一次源码并编译。如果强制按照固定顺序改写,就会剥夺这种选择,也失去一部分我们希望模型学会的行为。" + +#: website/public/blog/reverse-engineering-models/index.html:558 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why RE is interesting for reinforcement learning" +msgstr "为什么逆向工程值得用来探索强化学习" + +#: website/public/blog/reverse-engineering-models/index.html:559 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Bounded reconstruction gives us a practical environment for studying these " +"choices. Once the target and compiler are pinned, many source experiments " +"produce quick, repeatable feedback. The agent can compile a candidate, " +"inspect the mismatch and use that observation to choose its next step." +msgstr "" +"范围明确的重建任务,为研究这些选择提供了一个实际环境。目标和编译器版本固定之后,许多源码实验都能快速给出可重复的反馈。agent " +"可以编译候选源码、检查差异,再根据观察选择下一步。" + +#: website/public/blog/reverse-engineering-models/index.html:566 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reinforcement learning could train that decision-making process through the " +"feedback it receives. A candidate accepted by a defined check gives us a " +"concrete outcome, while the sequence of attempts lets us observe how much " +"work it took to reach it." +msgstr "" +"强化学习可以利用这些反馈,训练模型的决策过程。候选源码通过明确定义的检查,就给出了一个具体成果;而尝试的过程,让我们可以看到实现它花了多少功夫。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:572 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"There is already research supporting this direction. {ph0}LLM4Decompile{ph1}" +" studies models specialized for decompilation. {ph2}AutoDecompiler{ph3} " +"trains multi-turn refinement using compiler and execution feedback. Their " +"studies focus on functional recovery. For this work, I would use them as a " +"starting point for the related task of finding source that matches a " +"historical compiler's output." +msgstr "" +"已有研究支持这个方向。{ph0}LLM4Decompile{ph1}研究专门用于反编译的模型。{ph2}AutoDecompiler{ph3}利用编译和执行反馈,训练多轮改进能力。这些研究关注的是功能还原。对于我们的工作,我会以它们为起点,研究一个相关任务:找到能复现历史编译器输出的源码。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:584 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To reach such a well-defined task, though, we may first need to establish " +"what a function owns or devise a new runtime measurement. In TH08, for " +"example, a {ph0}collision difference led back to how the original program " +"handled floating-point calculations{ph1}. Before the agent could test a " +"useful correction, it had to work out what to measure." +msgstr "" +"不过,要得到这样一个定义明确的任务,有时得先确定函数负责哪些资源,或者设计新的方法观察程序运行时的行为。在 TH08 " +"中,{ph0}一次碰撞差异最终追溯到原版处理浮点计算的方式{ph1}。agent 必须先弄清楚该测量什么,才能验证有价值的修正。" + +#: website/public/blog/reverse-engineering-models/index.html:595 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This suggests another role for a broadly capable model: helping make the " +"problem checkable in the first place. Once that check exists, we can " +"investigate how far a learned search policy can get with it." +msgstr "这提示了能力全面的模型的另一个作用:先把问题变得可验证。验证手段建立起来后,我们才可以研究,学出来的搜索策略能走多远。" + +#: website/public/blog/reverse-engineering-models/index.html:603 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "When the oracle is wrong" +msgstr "当验证器出错时" + +#: website/public/blog/reverse-engineering-models/index.html:604 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Once we train against that check, its coverage becomes part of what the " +"model learns. I call the reference or checker an oracle. Even if it gives a " +"repeatable answer, we still need to understand what that answer covers." +msgstr "" +"一旦用某项检查来训练模型,它的覆盖范围就会成为学习环境的一部分。我把这种参考标准或验证器称为 " +"oracle。即使它能给出可重复的答案,我们也必须知道,这个答案覆盖了什么。" + +#: website/public/blog/reverse-engineering-models/index.html:610 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"TH08 showed how much can hide outside that coverage. Some reconstructed " +"functions passed their exact comparisons with wrong floating constants. The " +"comparator adjusted relocated addresses before checking instructions, but it" +" checked the values behind those addresses only when they had been " +"explicitly recorded for verification. Other literal references could pass " +"without their values being examined." +msgstr "" +"TH08 " +"展示了检查范围之外可能藏着多少问题。一些重建函数使用了错误的浮点常量,却仍然通过了精确比较。比较器会先调整重定位地址,再检查指令;但只有明确登记过的常量,才会进一步检查地址指向的值。其他常量引用,即使值没被检查,也可能通过。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: 1.0f +#. {ph3}: 32.0f +#: website/public/blog/reverse-engineering-models/index.html:619 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To close that gap, a {ph0}wider audit{ph1} found incorrect constants in " +"functions that had already been accepted. In one calculation, the " +"reconstructed source added {ph2} where the original added {ph3}." +msgstr "" +"为了补上这个漏洞,{ph0}一次更全面的审计{ph1}在已经通过检查的函数中找到了错误常量。有一个计算中,重建源码加的是 {ph2},而原版加的是 " +"{ph3}。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:629 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A learner rewarded by the old check could learn to produce the wrong " +"reconstruction.{ph1} We had not trained an RL model against this oracle; the" +" observed failure was in the check itself. But using that check as a reward " +"would have made the same blind spot part of the learning environment." +msgstr "" +"{ph0}如果用旧检查的结果作为奖励,模型可能学会生成错误的重建源码。{ph1} 我们没有用这个验证器训练 RL " +"模型;实际观察到的是检查本身的漏洞。但如果把它用作奖励,同一个盲点也会进入学习环境。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:639 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A functional test can have a similar blind spot. {ph0}Differential-fuzzing " +"research on LLM decompilers{ph1} has found behavior differences beyond the " +"supplied tests. Passing those tests therefore needs to be read in terms of " +"the cases they exercise, just as an instruction match needs to be read in " +"terms of the bytes and data it checks." +msgstr "" +"功能测试也可能存在类似盲点。{ph0}针对 LLM " +"反编译器的差分模糊测试研究{ph1}发现,在已有测试之外,程序行为仍可能不同。因此,理解“测试通过”时,必须看它实际覆盖了哪些情况;理解“指令匹配”时,也必须看它检查了哪些字节和数据。" + +#: website/public/blog/reverse-engineering-models/index.html:649 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before using either kind of check as a reward, I would test it with " +"deliberate failures. I would also keep final acceptance outside the " +"candidate's editable source, so the model has to satisfy the check through " +"its reconstruction. Otherwise, something like a byte array copied from the " +"target could satisfy a naive score without recovering useful source." +msgstr "" +"在把任何一种检查用作奖励之前,我都会用刻意构造的错误来测试它。我还会把最终验收机制放在候选源码可修改的范围之外,让模型通过正确重建来满足检查。否则,直接复制目标程序的字节数组,也可能骗过简单的评分,却没有还原出有用的源码。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:657 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Improving the learner and improving its oracle belong to the same " +"research problem.{ph1} As the learner gets better at optimizing the feedback" +" we give it, understanding that feedback becomes more important too." +msgstr "{ph0}改进模型和改进它的验证器,属于同一个研究问题。{ph1} 模型越擅长优化我们给它的反馈,我们就越需要理解反馈本身。" + +#: website/public/blog/reverse-engineering-models/index.html:667 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A comparison worth running" +msgstr "值得做的对照实验" + +#: website/public/blog/reverse-engineering-models/index.html:668 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"With those checks in place, I would start the comparison on a fixed VC7.1 " +"task family. First, I would establish baselines for a strong general model " +"and a smaller one. Adding repository knowledge to the smaller model would " +"show what retrieval changes. From there, supervised specialization and an " +"RL-trained version would let us examine what training adds." +msgstr "" +"具备这些检查之后,我会从一组固定的 VC7.1 " +"任务开始比较。先为强通用模型和小型通用模型建立基线,再给小模型加入仓库知识,观察检索带来的变化。接着比较监督训练的专用模型,以及在此基础上进行 RL " +"训练的版本,看看训练增加了什么。" + +#: website/public/blog/reverse-engineering-models/index.html:676 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To make those comparisons informative, each model would receive the same " +"target evidence and tools. For the training-specific comparisons, I would " +"keep the base model and available knowledge fixed as well. Repeated trials " +"would help distinguish a reliable search strategy from a lucky candidate." +msgstr "" +"为了让比较有意义,每个模型都应获得相同的目标证据和工具。专门比较训练效果时,还要固定基础模型和可用知识。重复实验有助于区分可靠的搜索策略与偶然成功的候选方案。" + +#: website/public/blog/reverse-engineering-models/index.html:683 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would then separate two starting conditions: reconstructing a function " +"from its instructions, and closing the last few differences in maintained " +"source. A specialist might be excellent at the second task while a general " +"model does much of the work needed to reach it. Measuring them separately " +"would reveal that division of work." +msgstr "" +"我还会区分两种起点:从指令开始重建函数,以及在已有源码中消除最后几处差异。专用模型可能很擅长后者,而通用模型承担了到达这个阶段的大部分工作。分别评估,才能看清这种分工。" + +#: website/public/blog/reverse-engineering-models/index.html:691 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both tasks need genuinely held-out examples. Because related Touhou games " +"share engine code, holding out a game's name would still allow nearly " +"identical functions to appear in training. I would instead hold out source " +"families and keep their accepted answers out of both training and retrieval." +msgstr "" +"两类任务都需要真正未见过的测试案例。东方系列的相关游戏会共享引擎代码,只留出某个游戏,仍可能让几乎相同的函数进入训练集。我会改为按相关源码家族留出测试集,并把它们通过验证的答案同时排除在训练数据和检索库之外。" + +#: website/public/blog/reverse-engineering-models/index.html:698 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"On those examples, the outcome would be complete, independently validated " +"matches. Giving each model the same candidate or compile budget would help " +"compare search quality. Measuring elapsed time and total cost, including " +"compilation, would tell us whether that quality produces an efficient " +"workflow. Training costs would need their own accounting too." +msgstr "" +"在这些案例上,要统计的是完整、独立验证过的匹配成果。给每个模型相同的候选数量或编译次数预算,可以比较搜索质量。记录总耗时和总成本,包括编译,才能知道这种质量是否转化成了高效的工作流。训练成本也要单独计算。" + +#: website/public/blog/reverse-engineering-models/index.html:706 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both views matter because a cheaper proposal can become an expensive " +"workflow if it causes more uninformative compiles. Conversely, a slower " +"model may pay for itself by finding a useful experiment sooner. The " +"comparison should capture those costs." +msgstr "" +"两种角度都重要。一个便宜的提议,如果导致大量无效编译,整个工作流仍可能很贵。反过来,较慢的模型如果更早找到有价值的实验,也可能更划算。对照实验应当反映这些成本。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:712 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Counting accepted results also keeps intermediate progress in perspective. " +"One TH09 {ph0}replay-save candidate{ph1} looked promising because the " +"compiled function had the expected size, but a full comparison still found " +"substantial differences. Changes in how the compiler arranges instructions " +"can also make a candidate look closer without resolving the underlying " +"problem. Such diagnostics can guide the next experiment, but I would " +"validate any reward for \"getting closer\" against the full acceptance " +"check." +msgstr "" +"只统计通过验收的结果,也能帮助我们合理看待中间进展。TH09 " +"的一个{ph0}录像保存候选方案{ph1}看起来很有希望,因为编译后函数的大小符合预期,但完整比较仍发现大量差异。编译器改变指令排列,也可能让候选源码看起来更接近,却没有解决根本问题。这些诊断可以指导下一步,但若要奖励“变得更接近”,我会先对照完整验收检查,确认这种奖励有效。" + +#: website/public/blog/reverse-engineering-models/index.html:727 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would be interested in the pattern across these results. If a specialist " +"does well within a familiar compiler family, does that advantage survive a " +"toolchain change? And once both models have good retrieval, how much does " +"training still add? The answers would help us decide where each kind of " +"model is most useful." +msgstr "" +"我想看这些结果呈现出什么规律。如果专用模型在熟悉的编译器上表现很好,换了工具链后,这种优势还在吗?当双方都拥有良好的检索能力时,训练还能增加多少价值?这些答案能帮助我们决定,每类模型最适合用在哪里。" + +#: website/public/blog/reverse-engineering-models/index.html:736 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What kind of agent does this suggest?" +msgstr "这意味着怎样的 agent?" + +#: website/public/blog/reverse-engineering-models/index.html:737 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"If the models turn out to help at different stages, a useful arrangement " +"might be a broad agent with access to a specialist. Once a task becomes a " +"familiar compiler search, it could ask the specialist for candidates. If the" +" results stop making sense, the broader agent could return to the target " +"evidence or investigate the measurement itself." +msgstr "" +"如果不同模型在不同阶段各有所长,一种有价值的安排可能是:让能力全面的 agent " +"可以调用专用模型。当任务变成一种熟悉的编译器搜索时,它可以请专用模型提出候选方案。如果结果开始说不通,通用 agent " +"可以回头检查目标证据,或者调查测量本身。" + +#: website/public/blog/reverse-engineering-models/index.html:745 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In that arrangement, specialization becomes a tool the agent can choose as " +"the investigation develops. The agent keeps the freedom to change direction " +"that made the earlier breakthroughs possible." +msgstr "" +"在这样的安排中,专用模型成为 agent 随着调查进展可以选择的工具。agent 仍然拥有改变方向的自由,而这正是前面那些突破发生的重要条件。" + +#: website/public/blog/reverse-engineering-models/index.html:750 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This follows naturally from the industrial shift I described in the previous" +" essay. Being able to run many more experiments changes how we organize the " +"work. It also gives us a reason to ask which decisions need broad judgment " +"and which improve through repeated experience with a particular craft." +msgstr "" +"这自然延续了上一篇文章中谈到的工业化转变。能够开展更多实验,会改变我们组织工作的方式,也让我们有理由追问:哪些决策需要全面的判断,哪些可以通过反复积累某项具体技艺来改善?" + +#: website/public/blog/reverse-engineering-models/index.html:757 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For now, the strongest general model has earned its place in my workflow by " +"opening problems I had left unresolved. Those successes are what make me " +"want to examine the work more closely: how much depended on the model's " +"breadth, and how much could be handled by a model trained in this particular" +" kind of reconstruction?" +msgstr "" +"目前,最强的通用模型解决了我过去没能解决的问题,因此在我的工作流中占据了一席之地。正是这些成功,让我想更仔细地研究过程:有多少依赖模型能力的广度?又有多少可以交给专门学习这类重建的模型?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:765 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}The repositories give us a way to test the question.{ph1} I would like " +"to use that record to build the comparison and see what we learn." +msgstr "{ph0}这些代码仓库让我们有机会验证这个问题。{ph1} 我想利用这些记录建立对照实验,看看我们能学到什么。" + +#: website/public/blog/reverse-engineering-models/index.html:775 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Evidence and model attribution" +msgstr "证据与模型归属" + +#: website/public/blog/reverse-engineering-models/index.html:783 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The examples describe historical function-level comparisons recorded in the " +"linked commits. Whole-game runtime checks are separate milestones. Source " +"and matching records were reviewed on 13 October 2026." +msgstr "" +"这些例子来自链接提交中记录的历史函数级比较。整款游戏的运行时检查属于另一个里程碑。相关源码与匹配记录核查于 2026 年 10 月 13 日。" + +#. {ph0}: gpt-dots +#. {ph1}: gpt-web +#: website/public/blog/reverse-engineering-models/index.html:789 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The TH09 label {ph0} identifies GPT-6-Astra. {ph1} changed with the model " +"used at the time, so that label alone does not identify an individual run. " +"Explicit model names in the other cited commits establish their recorded " +"attribution; the histories do not contain uniform per-run cost or effort " +"measurements." +msgstr "" +"TH09 中的标签 {ph0} 对应 GPT-6-Astra。{ph1} " +"会随当时使用的模型变化,因此仅凭这个标签无法确定某次运行的具体模型。其他引用提交中的明确模型名,提供了对应的归属记录;这些历史记录并没有统一的单次运行成本或投入统计。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#: website/public/blog/reverse-engineering-models/index.html:798 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 temporary lifetimes:{ph1} the {ph2}earlier context review{ph3} and" +" {ph4}complete cold reconstruction{ph5}." +msgstr "{ph0}TH09 临时值生命周期:{ph1}{ph2}早期上下文检查{ph3}与{ph4}完整的干净重建{ph5}。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:810 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 Front callback:{ph1} {ph2}non-exact diagnostic regions{ph3}, " +"{ph4}Astra's new graph{ph5} and its {ph6}accepted recompile replay{ph7}." +msgstr "" +"{ph0}TH09 Front 回调:{ph1}{ph2}未匹配的诊断区域{ph3}、{ph4}Astra " +"提出的新控制流图{ph5}与{ph6}通过验收的重新编译检查{ph7}。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:826 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Compiler knowledge:{ph1} {ph2}TH095 identifier-bucket reuse{ph3}, " +"{ph4}TH04 player source and linker order{ph5} and {ph6}Luna's TH09 enemy-" +"spawn closure{ph7}." +msgstr "" +"{ph0}编译器知识:{ph1}{ph2}TH095 标识符分桶规律的复用{ph3}、{ph4}TH04 " +"玩家源码与链接顺序{ph5}、{ph6}Luna 完成的 TH09 敌人生成函数匹配{ph7}。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:842 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Checks and diagnostics:{ph1} {ph2}TH08 floating-literal audit{ph3}, " +"{ph4}TH08 numerical investigation{ph5} and {ph6}TH09 replay-save " +"reconstruction{ph7}." +msgstr "" +"{ph0}检查与诊断:{ph1}{ph2}TH08 浮点常量审计{ph3}、{ph4}TH08 数值问题调查{ph5}、{ph6}TH09 " +"录像保存重建{ph7}。" + +#: website/public/blog/reverse-engineering-models/index.html:859 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The proposed specialist and RL comparisons are future experiments. The " +"project histories motivate them; no trained specialist was evaluated for " +"this essay." +msgstr "文中提出的专用模型与 RL 对照,属于未来实验。项目历史为这些想法提供了依据;本文尚未评估经过训练的专用模型。" + +#: website/public/blog/reverse-engineering-models/index.html:6 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model? | REA" +msgstr "逆向工程需要更大的模型吗? | REA" + +#: website/public/blog/reverse-engineering-models/index.html:7 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What Touhou reconstruction taught me about general models, specialist models" +" and learning to propose better hypotheses from compiler feedback." +msgstr "东方重建让我如何思考通用模型、专用模型,以及怎样从编译器反馈中学会提出更好的假设。" + +#: website/public/blog/reverse-engineering-models/index.html:240 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "On this page" +msgstr "在本页" + +#: website/public/blog/reverse-engineering-models/index.html:253 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Models and reverse engineering essay" +msgstr "模型与逆向工程随笔" + +#: website/public/blog/reverse-engineering-models/index.html:491 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Selected reconstruction experiments" +msgstr "重建实验选例" + +#: website/public/blog/reverse-engineering-models/index.html:883 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Back to top" +msgstr "回到顶部" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:776 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Detailed evidence notes{ph1} include the recorded comparisons and " +"experiment context." +msgstr "{ph0}详细证据记录{ph1}包含历史比较结果与实验背景。" + +#~ msgctxt "page:blog/index.html" +#~ msgid "By N0zoM1z0 · 13 October 2026 · Draft for review" +#~ msgstr "作者 N0zoM1z0 · 2026 年 10 月 13 日 · 待审草稿" + #~ msgctxt "page:index.html" #~ msgid "Setup guide" #~ msgstr "安装指南" diff --git a/website/locales/zh-TW/messages.po b/website/locales/zh-TW/messages.po index 98984f130..af7b68bcd 100644 --- a/website/locales/zh-TW/messages.po +++ b/website/locales/zh-TW/messages.po @@ -32,43 +32,44 @@ msgid "Thoughts on reverse engineering." msgstr "逆向工程的思考。" #. {ph0}:
-#: website/public/blog/index.html:200 +#: website/public/blog/index.html:214 #, python-brace-format msgctxt "page:blog/index.html" msgid "Game reconstruction{ph0}Essay" msgstr "遊戲重建{ph0}論文" -#: website/public/blog/index.html:202 +#: website/public/blog/index.html:216 msgctxt "page:blog/index.html" msgid "The industrial revolution in reverse engineering" msgstr "逆向工程的工業革命" -#: website/public/blog/index.html:203 +#: website/public/blog/index.html:217 msgctxt "page:blog/index.html" msgid "" "What manual RE and reconstructing TH08, TH095 and TH04 taught me about " "agents, verification and knowledge that compounds across projects." msgstr "什麼手動重新和重建TH08,TH095和TH04教我關於程式設計助手,驗證和知識,跨專案的化合物。" -#: website/public/blog/index.html:208 +#: website/public/blog/index.html:207 website/public/blog/index.html:222 msgctxt "page:blog/index.html" msgid "Read the article" msgstr "閱讀文章" -#: website/public/blog/index.html:211 +#: website/public/blog/index.html:225 msgctxt "page:blog/index.html" msgid "By N0zoM1z0 · 10 October 2026" msgstr "透過N0zoM1z0*10十月2026" #. {ph0}: -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 +#: website/public/blog/reverse-engineering-models/index.html:883 #: website/public/blog/touhou-reconstruction/index.html:1080 #: website/public/examples/dino-lab/index.html:475 #: website/public/faq/index.html:573 #: website/public/first-investigation/index.html:523 #: website/public/get-started/index.html:882 -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 #: website/public/guides/index.html:282 #: website/public/guides/javascript/index.html:503 #: website/public/guides/native/index.html:509 website/public/index.html:1191 @@ -96,6 +97,8 @@ msgid "" msgstr "對逆向工程的思考,借鑑與 REA 和AI程式設計助手。" #: website/public/blog/index.html:75 website/public/blog/index.html:88 +#: website/public/blog/reverse-engineering-models/index.html:75 +#: website/public/blog/reverse-engineering-models/index.html:88 #: website/public/blog/touhou-reconstruction/index.html:75 #: website/public/blog/touhou-reconstruction/index.html:88 #: website/public/examples/dino-lab/index.html:75 @@ -130,6 +133,7 @@ msgid "" msgstr "REA-瞭解軟體的工作原理。 逆向工程與你的程式設計助手." #: website/public/blog/index.html:108 +#: website/public/blog/reverse-engineering-models/index.html:108 #: website/public/blog/touhou-reconstruction/index.html:108 #: website/public/examples/dino-lab/index.html:109 #: website/public/faq/index.html:108 @@ -150,6 +154,7 @@ msgid "REA home" msgstr "REA 首頁" #: website/public/blog/index.html:112 +#: website/public/blog/reverse-engineering-models/index.html:112 #: website/public/blog/touhou-reconstruction/index.html:112 #: website/public/examples/dino-lab/index.html:113 #: website/public/faq/index.html:112 @@ -170,6 +175,7 @@ msgid "Main navigation" msgstr "主導航" #: website/public/blog/index.html:130 +#: website/public/blog/reverse-engineering-models/index.html:130 #: website/public/blog/touhou-reconstruction/index.html:130 #: website/public/examples/dino-lab/index.html:131 #: website/public/faq/index.html:130 @@ -190,6 +196,7 @@ msgid "REA on GitHub" msgstr "在 GitHub 檢視 REA" #: website/public/blog/index.html:150 +#: website/public/blog/reverse-engineering-models/index.html:150 #: website/public/blog/touhou-reconstruction/index.html:150 #: website/public/examples/dino-lab/index.html:151 #: website/public/faq/index.html:150 @@ -210,6 +217,7 @@ msgid "Join REA on Discord" msgstr "加入 REA 的 Discord 社群" #: website/public/blog/index.html:169 +#: website/public/blog/reverse-engineering-models/index.html:169 #: website/public/blog/touhou-reconstruction/index.html:169 #: website/public/examples/dino-lab/index.html:170 #: website/public/faq/index.html:169 @@ -229,7 +237,7 @@ msgctxt "shared" msgid "Follow REA on X" msgstr "在 X 上關注 REA" -#: website/public/blog/index.html:230 +#: website/public/blog/index.html:244 msgctxt "page:blog/index.html" msgid "Back to top" msgstr "回到頂部" @@ -4252,7 +4260,7 @@ msgctxt "page:guides/browser/index.html" msgid "Copy read the loaded scripts" msgstr "複製讀取載入的指令碼" -#: website/public/guides/browser/index.html:498 +#: website/public/guides/browser/index.html:507 msgctxt "page:guides/browser/index.html" msgid "Back to top" msgstr "回到頂部" @@ -10634,6 +10642,7 @@ msgid "Same spacing: 256 ÷ 16 = 16 angle units = 22.5°" msgstr "相同間距:256÷16=16角度單位=22.5°" #: website/public/blog/index.html:114 +#: website/public/blog/reverse-engineering-models/index.html:114 #: website/public/blog/touhou-reconstruction/index.html:114 #: website/public/examples/dino-lab/index.html:115 #: website/public/faq/index.html:114 @@ -10654,6 +10663,7 @@ msgid "Showcases" msgstr "案例展示" #: website/public/blog/index.html:115 +#: website/public/blog/reverse-engineering-models/index.html:115 #: website/public/blog/touhou-reconstruction/index.html:115 #: website/public/examples/dino-lab/index.html:116 #: website/public/faq/index.html:115 @@ -10674,6 +10684,7 @@ msgid "Blog" msgstr "部落格" #: website/public/blog/index.html:113 +#: website/public/blog/reverse-engineering-models/index.html:113 #: website/public/blog/touhou-reconstruction/index.html:113 #: website/public/examples/dino-lab/index.html:114 #: website/public/faq/index.html:113 @@ -10694,6 +10705,7 @@ msgid "Get started" msgstr "開始使用" #: website/public/blog/index.html:116 +#: website/public/blog/reverse-engineering-models/index.html:116 #: website/public/blog/touhou-reconstruction/index.html:116 #: website/public/examples/dino-lab/index.html:117 #: website/public/faq/index.html:116 @@ -10713,13 +10725,14 @@ msgctxt "shared" msgid "Guides" msgstr "指南" -#: website/public/blog/index.html:222 +#: website/public/blog/index.html:236 +#: website/public/blog/reverse-engineering-models/index.html:875 #: website/public/blog/touhou-reconstruction/index.html:1072 #: website/public/examples/dino-lab/index.html:467 #: website/public/faq/index.html:565 #: website/public/first-investigation/index.html:515 #: website/public/get-started/index.html:874 -#: website/public/guides/browser/index.html:490 +#: website/public/guides/browser/index.html:499 #: website/public/guides/index.html:274 #: website/public/guides/javascript/index.html:495 #: website/public/guides/native/index.html:501 website/public/index.html:1183 @@ -10733,13 +10746,14 @@ msgctxt "shared" msgid "Source" msgstr "原始碼" -#: website/public/blog/index.html:223 +#: website/public/blog/index.html:237 +#: website/public/blog/reverse-engineering-models/index.html:876 #: website/public/blog/touhou-reconstruction/index.html:1073 #: website/public/examples/dino-lab/index.html:468 #: website/public/faq/index.html:566 #: website/public/first-investigation/index.html:516 #: website/public/get-started/index.html:875 -#: website/public/guides/browser/index.html:491 +#: website/public/guides/browser/index.html:500 #: website/public/guides/index.html:275 #: website/public/guides/javascript/index.html:496 #: website/public/guides/native/index.html:502 website/public/index.html:1184 @@ -10753,13 +10767,14 @@ msgctxt "shared" msgid "MIT license" msgstr "MIT 許可證" -#: website/public/blog/index.html:224 +#: website/public/blog/index.html:238 +#: website/public/blog/reverse-engineering-models/index.html:877 #: website/public/blog/touhou-reconstruction/index.html:1074 #: website/public/examples/dino-lab/index.html:469 #: website/public/faq/index.html:567 #: website/public/first-investigation/index.html:517 #: website/public/get-started/index.html:876 -#: website/public/guides/browser/index.html:492 +#: website/public/guides/browser/index.html:501 #: website/public/guides/index.html:276 #: website/public/guides/javascript/index.html:497 #: website/public/guides/native/index.html:503 website/public/index.html:1185 @@ -10773,13 +10788,14 @@ msgctxt "shared" msgid "Join the community" msgstr "加入社群" -#: website/public/blog/index.html:225 +#: website/public/blog/index.html:239 +#: website/public/blog/reverse-engineering-models/index.html:878 #: website/public/blog/touhou-reconstruction/index.html:1075 #: website/public/examples/dino-lab/index.html:470 #: website/public/faq/index.html:568 #: website/public/first-investigation/index.html:518 #: website/public/get-started/index.html:877 -#: website/public/guides/browser/index.html:493 +#: website/public/guides/browser/index.html:502 #: website/public/guides/index.html:277 #: website/public/guides/javascript/index.html:498 #: website/public/guides/native/index.html:504 website/public/index.html:1186 @@ -11686,6 +11702,1161 @@ msgctxt "runtime" msgid "Switch to dark mode" msgstr "切換至深色模式" +#. {ph0}: +#. {ph1}: +#: website/public/guides/browser/index.html:480 +#, python-brace-format +msgctxt "page:guides/browser/index.html" +msgid "" +"For HTTP request and response bodies, use {ph0}Fiddler Everywhere: SAZ " +"captures, MCP and public skills{ph1}. Export SAZ or HAR 1.2 for offline REA " +"capture Evidence, or connect Everywhere's MCP server to inspect captured " +"traffic with your agent." +msgstr "" +"若要檢查 HTTP 請求與回應本文,請使用{ph0}Fiddler Everywhere:SAZ 擷取、MCP 與公開技能{ph1}。匯出 SAZ ? " +"HAR 1.2 可在 REA 中離線取得擷取 Evidence,也可以連接 Everywhere 的 MCP 伺服器,讓代理檢查已擷取的流量。" + +#. {ph0}:
+#: website/public/blog/index.html:200 +#, python-brace-format +msgctxt "page:blog/index.html" +msgid "Model specialization{ph0}Essay" +msgstr "專用模型{ph0}隨筆" + +#: website/public/blog/index.html:202 +msgctxt "page:blog/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "逆向工程需要更大的模型嗎?" + +#: website/public/blog/index.html:203 +msgctxt "page:blog/index.html" +msgid "" +"What Touhou reconstruction suggests about general models, specialist models " +"and learning from compiler feedback." +msgstr "從 Touhou 重建的經驗,探討通用模型、專用模型,以及如何從編譯器回饋中學習。" + +#: website/public/blog/index.html:210 +msgctxt "page:blog/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "作者 N0zoM1z0 · 2026 年 10 月 13 日" + +#: website/public/blog/reverse-engineering-models/index.html:103 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Skip to content" +msgstr "跳轉到內容" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:194 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Home{ph1} / {ph2}Blog{ph3} / Models and RE" +msgstr "{ph0}首頁{ph1} / {ph2}部落格{ph3} / 模型與逆向工程" + +#: website/public/blog/reverse-engineering-models/index.html:197 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Essay" +msgstr "隨筆" + +# Reviewed for technical meaning and readability; preserve the distinction +# between historical evidence and proposed experiments. +#: website/public/blog/reverse-engineering-models/index.html:198 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model?" +msgstr "逆向工程需要更大的模型嗎?" + +#: website/public/blog/reverse-engineering-models/index.html:199 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "By N0zoM1z0 · 13 October 2026" +msgstr "作者 N0zoM1z0 · 2026 年 10 月 13 日" + +#: website/public/blog/reverse-engineering-models/index.html:201 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Takeaway" +msgstr "要點" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:203 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Model capability changes the hypotheses an agent can find.{ph1} Later " +"sessions have solved problems that earlier work left unresolved." +msgstr "{ph0}模型能力會影響 agent 能提出哪些假設。{ph1} 後來的工作解決了早期嘗試未能解決的問題。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:211 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A familiar compiler gives specialization something concrete to " +"learn.{ph1} Some exact matches depend on narrow, reusable source patterns." +msgstr "{ph0}熟悉的編譯器為專用模型提供了具體的學習對象。{ph1} 有些精確匹配依賴細小但可以複用的原始碼模式。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:218 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Give repository knowledge a fair comparison.{ph1} A smaller model with " +"good retrieval may already know enough to make progress." +msgstr "{ph0}也要公平評估程式碼倉庫裡的知識。{ph1} 較小的模型配合良好的檢索,可能已經掌握了足以推進工作的知識。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:223 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Train the next useful experiment, and test its oracle.{ph1} The value " +"of feedback depends on what the checker actually verifies." +msgstr "{ph0}訓練模型選擇下一個有價值的實驗,同時檢驗它使用的驗證器。{ph1} 回饋是否有用,取決於驗證器究竟檢查了什麼。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:230 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Measure accepted reconstruction against time and cost.{ph1} That is the" +" result I want from the whole agent." +msgstr "{ph0}結合時間和成本,評估透過驗證的重建成果。{ph1} 這是我希望從整個 agent 工作流程中獲得的結果。" + +#: website/public/blog/reverse-engineering-models/index.html:241 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "In this article" +msgstr "在本文中" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:243 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What better models changed{ph1}" +msgstr "{ph0}更強的模型改變了什麼{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:244 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Why specialize?{ph1}" +msgstr "{ph0}為什麼考慮專用模型?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:245 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Where experience lives{ph1}" +msgstr "{ph0}這些經驗應該存在哪裡?{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:246 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}Learning the next experiment{ph1}" +msgstr "{ph0}學習選擇下一個有價值的實驗{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:247 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}RE and reinforcement learning{ph1}" +msgstr "{ph0}逆向工程與強化學習{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:248 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}When the oracle is wrong{ph1}" +msgstr "{ph0}當驗證器出錯時{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:249 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}A comparison worth running{ph1}" +msgstr "{ph0}值得做的對照實驗{ph1}" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:250 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "{ph0}What kind of agent?{ph1}" +msgstr "{ph0}這意味著怎樣的 agent?{ph1}" + +#: website/public/blog/reverse-engineering-models/index.html:258 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What better models changed" +msgstr "更強的模型改變了什麼" + +#: website/public/blog/reverse-engineering-models/index.html:259 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Since August 2026, I have been exploring how much reverse engineering an " +"agent can carry out with access to a program and a way to check its work. " +"Most of my experience comes from reconstructing games, especially Touhou. As" +" I have changed models, the repositories have kept a record of what each " +"session managed to recover." +msgstr "" +"從 2026 年 8 月開始,我一直在探索:如果給 agent " +"訪問程式的能力,以及驗證自身工作的手段,它能獨立完成多少逆向工作?我的經驗主要來自遊戲重建,尤其是東方系列。隨著我更換模型,程式碼倉庫記錄了每次工作還原出的內容。" + +#: website/public/blog/reverse-engineering-models/index.html:267 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of that work aims at exact reconstruction: we compile the maintained " +"source under a pinned historical toolchain and check whether it reproduces " +"the target function. When a later model succeeds where earlier work stalled," +" the source changes show what made that progress possible." +msgstr "" +"這些工作很大一部分追求精確重建:用固定版本的歷史工具鏈編譯我們維護的原始碼,檢查它能否復現目標函式。當後來的模型解決了早期工作卡住的問題,我們可以從原始碼變化中看出,是什麼促成了這次進展。" + +#: website/public/blog/reverse-engineering-models/index.html:274 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The temporary values had to stay alive together" +msgstr "臨時值需要同時保持有效" + +#: website/public/blog/reverse-engineering-models/index.html:275 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Much of a TH09 attack function's logic had already been reconstructed, but " +"the compiler still stored temporary results differently from the original. " +"The remaining problem was how long those results needed to stay alive and " +"when their storage could be reused." +msgstr "" +"TH09 " +"中一個攻擊函式的大部分邏輯已經還原,但編譯器對臨時結果的儲存安排仍然與原版不同。剩下的問題在於:這些結果的生命週期應該多長,編譯器何時可以複用它們的儲存空間?" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:282 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A {ph0}GPT-6-Sol checkpoint{ph1} recorded this mismatch after testing " +"several ordinary source arrangements. When {ph2}GPT-6.1-Sol returned to the " +"problem{ph3}, it found a useful hypothesis about how long those temporary " +"values needed to stay alive." +msgstr "" +"{ph0}GPT-6-Sol 的一次工作記錄{ph1}表明,嘗試幾種常見的原始碼寫法後,差異仍未消除。後來,{ph2}GPT-6.1-Sol " +"重新研究這個問題{ph3},提出了一個有價值的假設:關鍵可能在於這些臨時值的生命週期。" + +#: website/public/blog/reverse-engineering-models/index.html:296 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The function produced temporary results and copied them into animation " +"positions. In C++, a temporary result stays alive until the expression using" +" it finishes. Grouping the real operations into one expression kept those " +"results alive together, so the compiler could no longer reuse the same " +"storage for all of them." +msgstr "" +"這個函式會產生臨時結果,再把它們複製到動畫的位置資料中。在 C++ " +"中,臨時結果會一直存在,直到使用它的表示式執行完畢。把實際操作放進同一個表示式,就能讓這些結果同時存活,編譯器也就不能為它們全部複用同一塊儲存空間。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:303 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"One result still shared storage with a value used later for collision " +"checks. Giving that value its own lifetime resolved the remaining overlap. " +"After several targeted experiments, {ph0}a clean rebuild confirmed that the " +"full function matched{ph1}." +msgstr "" +"其中一個結果仍然與稍後用於碰撞檢查的值共享儲存。給那個值單獨的生命週期後,最後一處重疊也消失了。經過幾次有針對性的實驗,{ph0}一次乾淨的重新構建確認,整個函式都已匹配{ph1}。" + +#: website/public/blog/reverse-engineering-models/index.html:312 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What interests me is how the discovery and the tests worked together. The " +"lifetime hypothesis gave the agent a new direction; compiler feedback helped" +" it refine that hypothesis into a source arrangement that matched." +msgstr "" +"我感興趣的是,發現與驗證如何共同推進工作。生命週期的假設給 agent 指明瞭新方向;編譯器的回饋又幫助它逐步調整原始碼,直到找到匹配的寫法。" + +#: website/public/blog/reverse-engineering-models/index.html:318 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A different control-flow graph" +msgstr "換一種控制流結構" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:319 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A second TH09 function shows why that new direction can matter. Here, the " +"obstacle was how several branches shared their final work. " +"{ph0}GPT-6.1-Sol's earlier work{ph1} got parts of the function to match, but" +" the branches still joined up differently from the original." +msgstr "" +"TH09 的另一個函式說明了,新方向為什麼重要。這次的難點是多個分支如何共享末尾的操作。{ph0}GPT-6.1-Sol " +"之前的工作{ph1}已經讓函式的一些部分匹配,但分支匯合的方式仍與原版不同。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:330 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}GPT-6-Astra changed the surrounding control-flow graph{ph1}. The main " +"paths now led to a shared final update and return. In that source structure," +" the compiler combined the branches in the same way as the original. The " +"complete function matched, and a {ph2}separate rebuild check confirmed " +"it{ph3}." +msgstr "" +"{ph0}GPT-6-Astra " +"改寫了外圍的控制流圖{ph1},讓主要路徑匯入共同的最終更新與返回。在這種原始碼結構下,編譯器合併分支的方式與原版一致。整個函式實現了匹配,{ph2}另一次獨立的重新構建也確認了這一結果{ph3}。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:342 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Changing the representation opened a more productive search space.{ph1}" +" Both cases give me a reason to take general model capability seriously: an " +"agent can keep trying local variations for a long time before finding a " +"better way to express the surrounding work." +msgstr "" +"{ph0}換一種表達方式,開啟了更有價值的搜尋空間。{ph1} 這兩個例子讓我認真看待通用模型的能力:agent " +"可能花很長時間嘗試局部變化,才找到一種更好的整體寫法。" + +#: website/public/blog/reverse-engineering-models/index.html:351 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"At the same time, the later sessions started with earlier source and " +"recorded failures. That accumulated work was part of their advantage. The " +"history lets us inspect the discoveries and their checks; a controlled " +"comparison would help separate the effect of the model from the experience " +"it inherited." +msgstr "" +"不過,後來的工作也繼承了之前的原始碼和失敗記錄。這些積累本身就是優勢的一部分。歷史記錄讓我們可以檢查發現及其驗證過程;要區分模型能力和繼承經驗各自帶來的幫助,還需要對照實驗。" + +#: website/public/blog/reverse-engineering-models/index.html:358 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is where the question becomes interesting to me. Some of the progress " +"came from finding a new representation, while much of the remaining work " +"involved testing source forms against a familiar compiler. How much needs " +"broad reasoning ability, and how much could a model learn through repeated " +"experience with that compiler and code?" +msgstr "" +"這正是我覺得有趣的地方。有些進展來自新的表達方式,剩下的大量工作則是在熟悉的編譯器上試驗不同寫法。其中有多少需要廣泛的推理能力?又有多少可以透過反覆接觸這個編譯器和這些程式碼來學會?" + +#: website/public/blog/reverse-engineering-models/index.html:368 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why might a specialist do well?" +msgstr "為什麼專用模型可能做得很好?" + +#: website/public/blog/reverse-engineering-models/index.html:369 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Touhou gives us a concrete reason to expect that experience to help. Much of" +" the game code comes from the same author, ZUN, so recurring habits are a " +"reasonable starting assumption. The repositories back that up with useful " +"examples: related loops and helper arrangements have supplied hypotheses " +"across games." +msgstr "" +"東方系列給了我們一個具體的理由,去期待經驗積累能發揮作用。大量遊戲程式碼出自同一個作者 " +"ZUN,因此,把反覆出現的程式設計習慣作為初始假設是合理的。程式碼倉庫也提供了實際例子:相似的迴圈和輔助函式組織方式,曾為其他遊戲的重建提供假設。" + +#: website/public/blog/reverse-engineering-models/index.html:376 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those source patterns become more specific when we include the compiler. " +"Under a particular version and set of options, small changes in source can " +"produce repeatable differences in machine code. Some of those differences " +"are surprisingly obscure." +msgstr "把編譯器也考慮進去,這些原始碼模式就更加具體了。在固定版本和選項下,原始碼的小變化可能穩定地產生不同的機器碼。其中一些規律藏得很深。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: Supervisor::LoadDat +#. {ph4}: +#: website/public/blog/reverse-engineering-models/index.html:382 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In TH095, for example, {ph0}even the name used for a local variable could " +"change where VC7.1 stored it{ph1}. Moving the declaration alone did not " +"solve the layout, but an earlier background function had revealed a naming " +"pattern that affected the compiler's choice. That gave us a more promising " +"hypothesis for {ph2}the reconstruction of {ph3}{ph4}." +msgstr "" +"以 TH095 為例,{ph0}連局部變數的名字,都可能改變 VC7.1 " +"為它安排的儲存位置{ph1}。只移動宣告並不能解決佈局問題,但先前重建的一個背景函式揭示了會影響編譯器選擇的命名模式。這為{ph2}重建 " +"{ph3}{ph4}提供了更有希望的假設。" + +#: website/public/blog/reverse-engineering-models/index.html:396 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reusing that pattern for the file-size variable produced the required layout" +" without adding storage. The full function then matched." +msgstr "把這個模式用於表示檔案大小的變數,不增加額外儲存就得到了所需佈局。隨後,整個函式實現了匹配。" + +#: website/public/blog/reverse-engineering-models/index.html:401 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This is the sort of narrow detail a general model may not know when it " +"arrives. A specialist trained on this compiler could encounter many such " +"cases and learn which source variations are worth testing first." +msgstr "這類細節非常具體,通用模型第一次接觸時未必知道。而針對這個編譯器訓練的專用模型,可以接觸許多類似案例,學會哪些原始碼變化值得優先嚐試。" + +#: website/public/blog/reverse-engineering-models/index.html:407 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The opportunity is especially clear in final matching problems where we " +"already understand the behavior. We are then asking which ordinary source " +"form this compiler turns into the observed instructions. Specialization " +"could help the model propose those forms more efficiently, while the target " +"still determines which hypotheses survive." +msgstr "" +"當我們已經理解函式行為,只剩最後的匹配差異時,這個機會尤其明顯。此時的問題是:這個編譯器會把哪一種普通寫法編譯成眼前的指令?專用訓練可能讓模型更高效地提出這些寫法,而最終由目標程式決定哪些假設成立。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:415 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The useful patterns would have to retain their context, though. TH08 uses an" +" earlier Visual C++ version than TH095 and TH09. TH04 uses a different DOS-" +"era compiler, and its {ph0}player reconstruction{ph1} needed different " +"hypotheses about source patterns and how the compiled code was combined into" +" an executable. Shared authorship gives us a prior, but each target tells us" +" whether it applies. Distinguishing those contexts is part of what a " +"specialist would need to learn." +msgstr "" +"但這些模式必須保留適用的上下文。TH08 使用的 Visual C++ 版本早於 TH095 和 TH09。TH04 用的是另一種 DOS " +"時代的編譯器,其{ph0}玩家邏輯重建{ph1}需要不同的假設,既涉及原始碼寫法,也涉及編譯後的程式碼如何連結成可執行檔案。同一作者為我們提供了先驗,具體目標則告訴我們它是否適用。區分這些情況,也是專用模型需要學會的事。" + +#: website/public/blog/reverse-engineering-models/index.html:431 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Where should that experience live?" +msgstr "這些經驗應該存在哪裡?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:432 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before deciding to teach those patterns through model training, I would look" +" at the knowledge we already keep in the repository. In the {ph0}previous " +"essay{ph1}, I wrote about it becoming the project's working memory. That " +"memory includes compiler lessons, along with the source and failed " +"experiments that made them useful." +msgstr "" +"在決定透過訓練讓模型學會這些模式之前,我會先看看程式碼倉庫裡已有的知識。在{ph0}上一篇文章{ph1}中,我寫過:倉庫正在成為專案的工作記憶。這裡儲存著對編譯器的認識,也儲存著相關原始碼和失敗實驗,讓這些經驗有據可查。" + +#: website/public/blog/reverse-engineering-models/index.html:441 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"By retrieving that record, an agent can start with a known result and " +"inspect how we reached it. It can also avoid alternatives that have already " +"failed in the same context. The starting point improves without changing the" +" model's weights." +msgstr "" +"檢索這些記錄後,agent 可以從已知結果出發,看看我們是如何得到它的,也可以避開在相同條件下已經失敗的方案。模型的參數沒有改變,工作的起點卻提高了。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:447 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"When reconstructing TH09's enemy-spawn function, a {ph0}GPT-6-Luna-max " +"session{ph1} drew a hypothesis from TH08's corresponding source family. That" +" earlier source suggested how to structure the loop and handle failures " +"within it. TH09's own evidence supplied the game-specific details, and clean" +" rebuilds confirmed the complete match." +msgstr "" +"重建 TH09 的敵人生成函式時,{ph0}GPT-6-Luna-max 的一次工作{ph1}從 TH08 " +"對應的原始碼中找到了假設。之前的寫法提示了迴圈該如何組織,以及內部失敗該如何處理。TH09 " +"自身的證據補上了遊戲特有的細節,乾淨的重新構建確認了整個函式的匹配。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:458 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}\"The model did not know that compiler quirk\" and \"the model could " +"not use it\" are different problems.{ph1} Retrieval may solve the first, but" +" turning that fact into a useful candidate still takes work. And if the " +"candidate fails, the model must decide what the result means for its next " +"attempt. Knowing the relevant pattern is only the beginning of that process." +msgstr "" +"{ph0}“模型不知道這個編譯器特性”和“模型不會利用它”,是兩個不同的問題。{ph1} " +"檢索可能解決前者,但把知識變成有價值的候選原始碼,仍然需要能力。如果候選方案失敗了,模型還得判斷這次結果對下一次嘗試意味著什麼。知道相關模式,只是這個過程的起點。" + +#: website/public/blog/reverse-engineering-models/index.html:468 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For that reason, I would include a smaller general model with good retrieval" +" in any comparison. Giving it the same library of experience would help us " +"identify what training adds: better candidate generation, better decisions " +"after feedback, or both." +msgstr "" +"所以,在任何對照實驗中,我都會加入一個具備良好檢索能力的小型通用模型。給它相同的經驗庫,才能看清訓練究竟增加了什麼:更好的候選方案、更好的回饋後決策,還是兩者都有?" + +#: website/public/blog/reverse-engineering-models/index.html:476 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Learning the next useful experiment" +msgstr "學習選擇下一個有價值的實驗" + +#: website/public/blog/reverse-engineering-models/index.html:477 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Those decisions after feedback are what I would most like to study. In " +"agentic reconstruction, the model proposes an explanation and tests it " +"against the program. Even when the candidate fails, the result can narrow " +"the remaining possibilities and suggest a more useful next experiment." +msgstr "" +"回饋之後如何決策,是我最想研究的部分。在 agent " +"驅動的重建中,模型提出解釋,再用程式來驗證。即使候選方案失敗了,結果也可能縮小剩餘的可能性,提示下一步更值得嘗試什麼。" + +#: website/public/blog/reverse-engineering-models/index.html:484 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Consider the temporary-results example again. Its final source shows the " +"arrangement that worked, but the route to that source contains more " +"information. Each of the following experiments helped identify what still " +"needed to change." +msgstr "再看前面的臨時結果例子。最終原始碼展示了成功的寫法,但得到它的過程包含更多資訊。下面的每次實驗,都幫助我們確定了還有什麼需要改變。" + +#: website/public/blog/reverse-engineering-models/index.html:497 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Experiment" +msgstr "實驗" + +#: website/public/blog/reverse-engineering-models/index.html:498 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What its result taught us" +msgstr "結果告訴了我們什麼" + +#: website/public/blog/reverse-engineering-models/index.html:503 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Try a different way to copy the results" +msgstr "嘗試另一種複製結果的方式" + +#: website/public/blog/reverse-engineering-models/index.html:504 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "The copy code changes, but the function still differs." +msgstr "複製程式碼改變了,但函式仍有差異。" + +#: website/public/blog/reverse-engineering-models/index.html:509 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Use the returned values directly" +msgstr "直接使用回傳值" + +#: website/public/blog/reverse-engineering-models/index.html:510 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Copying follows the original pattern, but temporary storage still differs." +msgstr "複製方式與原版一致,但臨時儲存佈局仍有差異。" + +#: website/public/blog/reverse-engineering-models/index.html:516 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the temporary results alive together" +msgstr "讓臨時結果同時保持有效" + +#: website/public/blog/reverse-engineering-models/index.html:517 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"They get separate storage, but one still overlaps a value used later for " +"collision checks." +msgstr "它們得到了各自的儲存空間,但其中一個仍與稍後用於碰撞檢查的值重疊。" + +#: website/public/blog/reverse-engineering-models/index.html:523 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Keep the collision-check value separate" +msgstr "讓碰撞檢查的值擁有獨立儲存" + +#: website/public/blog/reverse-engineering-models/index.html:524 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A clean rebuild confirms the complete match." +msgstr "乾淨的重新構建確認了整個函式的匹配。" + +#: website/public/blog/reverse-engineering-models/index.html:529 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Selected experiments from the attack-function reconstruction above, grouped " +"by what they taught us." +msgstr "選自前面攻擊函式的重建實驗,按每次得到的認識整理。" + +#: website/public/blog/reverse-engineering-models/index.html:533 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A training record that keeps this sequence can teach more than the final " +"source alone. It connects each proposed change to an observed result and the" +" next decision. Keeping ineffective variations matters too: they tell a " +"later agent which hypotheses have already been tested in that context." +msgstr "" +"儲存這個過程的訓練記錄,比單獨儲存最終原始碼能教會模型更多東西。它把每個改動、觀察到的結果和下一步決策聯絡起來。無效的嘗試也值得保留:後來的 agent" +" 可以據此知道,在這個上下文中,哪些假設已經檢驗過。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:540 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}I would want the model to learn how to use feedback.{ph1} A neutral " +"result is a reason to reconsider the variation; a changed mismatch may point" +" to another cause. When local edits stop being informative, the agent should" +" be able to revisit the source representation or inspect the evidence again." +msgstr "" +"{ph0}我希望模型學會如何利用回饋。{ph1} " +"結果沒有變化,可能意味著該重新考慮這個改動;差異發生變化,則可能指向另一個原因。當局部修改不再提供有效資訊時,agent " +"應該可以重新考慮原始碼的整體表達,或者回頭檢查證據。" + +#: website/public/blog/reverse-engineering-models/index.html:549 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Learning to make those choices requires autonomy. Sometimes the useful next " +"step is inspecting another caller rather than compiling another edit. A " +"rigid sequence of required rewrites would remove that choice, along with " +"part of the behavior we want the model to learn." +msgstr "" +"學會這些選擇,需要自主性。有時,更有價值的下一步是檢查另一個呼叫者,而不是再修改一次原始碼並編譯。如果強制按照固定順序改寫,就會剝奪這種選擇,也失去一部分我們希望模型學會的行為。" + +#: website/public/blog/reverse-engineering-models/index.html:558 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Why RE is interesting for reinforcement learning" +msgstr "為什麼逆向工程值得用來探索強化學習" + +#: website/public/blog/reverse-engineering-models/index.html:559 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Bounded reconstruction gives us a practical environment for studying these " +"choices. Once the target and compiler are pinned, many source experiments " +"produce quick, repeatable feedback. The agent can compile a candidate, " +"inspect the mismatch and use that observation to choose its next step." +msgstr "" +"範圍明確的重建任務,為研究這些選擇提供了一個實際環境。目標和編譯器版本固定之後,許多原始碼實驗都能快速給出可重複的回饋。agent " +"可以編譯候選原始碼、檢查差異,再根據觀察選擇下一步。" + +#: website/public/blog/reverse-engineering-models/index.html:566 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Reinforcement learning could train that decision-making process through the " +"feedback it receives. A candidate accepted by a defined check gives us a " +"concrete outcome, while the sequence of attempts lets us observe how much " +"work it took to reach it." +msgstr "" +"強化學習可以利用這些回饋,訓練模型的決策過程。候選原始碼透過明確定義的檢查,就給出了一個具體成果;而嘗試的過程,讓我們可以看到實現它花了多少功夫。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#: website/public/blog/reverse-engineering-models/index.html:572 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"There is already research supporting this direction. {ph0}LLM4Decompile{ph1}" +" studies models specialized for decompilation. {ph2}AutoDecompiler{ph3} " +"trains multi-turn refinement using compiler and execution feedback. Their " +"studies focus on functional recovery. For this work, I would use them as a " +"starting point for the related task of finding source that matches a " +"historical compiler's output." +msgstr "" +"已有研究支援這個方向。{ph0}LLM4Decompile{ph1}研究專門用於反編譯的模型。{ph2}AutoDecompiler{ph3}利用編譯和執行回饋,訓練多輪改進能力。這些研究關注的是功能還原。對於我們的工作,我會以它們為起點,研究一個相關任務:找到能復現歷史編譯器輸出的原始碼。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:584 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To reach such a well-defined task, though, we may first need to establish " +"what a function owns or devise a new runtime measurement. In TH08, for " +"example, a {ph0}collision difference led back to how the original program " +"handled floating-point calculations{ph1}. Before the agent could test a " +"useful correction, it had to work out what to measure." +msgstr "" +"不過,要得到這樣一個定義明確的任務,有時得先確定函式負責哪些資源,或者設計新的方法觀察程式執行時的行為。在 TH08 " +"中,{ph0}一次碰撞差異最終追溯到原版處理浮點計算的方式{ph1}。agent 必須先弄清楚該測量什麼,才能驗證有價值的修正。" + +#: website/public/blog/reverse-engineering-models/index.html:595 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This suggests another role for a broadly capable model: helping make the " +"problem checkable in the first place. Once that check exists, we can " +"investigate how far a learned search policy can get with it." +msgstr "這提示了能力全面的模型的另一個作用:先把問題變得可驗證。驗證手段建立起來後,我們才可以研究,學出來的搜尋策略能走多遠。" + +#: website/public/blog/reverse-engineering-models/index.html:603 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "When the oracle is wrong" +msgstr "當驗證器出錯時" + +#: website/public/blog/reverse-engineering-models/index.html:604 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Once we train against that check, its coverage becomes part of what the " +"model learns. I call the reference or checker an oracle. Even if it gives a " +"repeatable answer, we still need to understand what that answer covers." +msgstr "" +"一旦用某項檢查來訓練模型,它的覆蓋範圍就會成為學習環境的一部分。我把這種參考標準或驗證器稱為 " +"oracle。即使它能給出可重複的答案,我們也必須知道,這個答案覆蓋了什麼。" + +#: website/public/blog/reverse-engineering-models/index.html:610 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"TH08 showed how much can hide outside that coverage. Some reconstructed " +"functions passed their exact comparisons with wrong floating constants. The " +"comparator adjusted relocated addresses before checking instructions, but it" +" checked the values behind those addresses only when they had been " +"explicitly recorded for verification. Other literal references could pass " +"without their values being examined." +msgstr "" +"TH08 " +"展示了檢查範圍之外可能藏著多少問題。一些重建函式使用了錯誤的浮點常量,卻仍然透過了精確比較。比較器會先調整重定位地址,再檢查指令;但只有明確登記過的常量,才會進一步檢查地址指向的值。其他常量引用,即使值沒被檢查,也可能透過。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: 1.0f +#. {ph3}: 32.0f +#: website/public/blog/reverse-engineering-models/index.html:619 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To close that gap, a {ph0}wider audit{ph1} found incorrect constants in " +"functions that had already been accepted. In one calculation, the " +"reconstructed source added {ph2} where the original added {ph3}." +msgstr "" +"為了補上這個漏洞,{ph0}一次更全面的審計{ph1}在已經透過檢查的函式中找到了錯誤常量。有一個計算中,重建原始碼加的是 {ph2},而原版加的是 " +"{ph3}。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:629 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}A learner rewarded by the old check could learn to produce the wrong " +"reconstruction.{ph1} We had not trained an RL model against this oracle; the" +" observed failure was in the check itself. But using that check as a reward " +"would have made the same blind spot part of the learning environment." +msgstr "" +"{ph0}如果用舊檢查的結果作為獎勵,模型可能學會生成錯誤的重建原始碼。{ph1} 我們沒有用這個驗證器訓練 RL " +"模型;實際觀察到的是檢查本身的漏洞。但如果把它用作獎勵,同一個盲點也會進入學習環境。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:639 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"A functional test can have a similar blind spot. {ph0}Differential-fuzzing " +"research on LLM decompilers{ph1} has found behavior differences beyond the " +"supplied tests. Passing those tests therefore needs to be read in terms of " +"the cases they exercise, just as an instruction match needs to be read in " +"terms of the bytes and data it checks." +msgstr "" +"功能測試也可能存在類似盲點。{ph0}針對 LLM " +"反編譯器的差分模糊測試研究{ph1}發現,在已有測試之外,程式行為仍可能不同。因此,理解“測試透過”時,必須看它實際覆蓋了哪些情況;理解“指令匹配”時,也必須看它檢查了哪些位元組和資料。" + +#: website/public/blog/reverse-engineering-models/index.html:649 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Before using either kind of check as a reward, I would test it with " +"deliberate failures. I would also keep final acceptance outside the " +"candidate's editable source, so the model has to satisfy the check through " +"its reconstruction. Otherwise, something like a byte array copied from the " +"target could satisfy a naive score without recovering useful source." +msgstr "" +"在把任何一種檢查用作獎勵之前,我都會用刻意構造的錯誤來測試它。我還會把最終驗收機制放在候選原始碼可修改的範圍之外,讓模型透過正確重建來滿足檢查。否則,直接複製目標程式的位元組陣列,也可能騙過簡單的評分,卻沒有還原出有用的原始碼。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:657 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Improving the learner and improving its oracle belong to the same " +"research problem.{ph1} As the learner gets better at optimizing the feedback" +" we give it, understanding that feedback becomes more important too." +msgstr "{ph0}改進模型和改進它的驗證器,屬於同一個研究問題。{ph1} 模型越擅長最佳化我們給它的回饋,我們就越需要理解回饋本身。" + +#: website/public/blog/reverse-engineering-models/index.html:667 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "A comparison worth running" +msgstr "值得做的對照實驗" + +#: website/public/blog/reverse-engineering-models/index.html:668 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"With those checks in place, I would start the comparison on a fixed VC7.1 " +"task family. First, I would establish baselines for a strong general model " +"and a smaller one. Adding repository knowledge to the smaller model would " +"show what retrieval changes. From there, supervised specialization and an " +"RL-trained version would let us examine what training adds." +msgstr "" +"具備這些檢查之後,我會從一組固定的 VC7.1 " +"任務開始比較。先為強通用模型和小型通用模型建立基線,再給小模型加入倉庫知識,觀察檢索帶來的變化。接著比較監督訓練的專用模型,以及在此基礎上進行 RL " +"訓練的版本,看看訓練增加了什麼。" + +#: website/public/blog/reverse-engineering-models/index.html:676 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"To make those comparisons informative, each model would receive the same " +"target evidence and tools. For the training-specific comparisons, I would " +"keep the base model and available knowledge fixed as well. Repeated trials " +"would help distinguish a reliable search strategy from a lucky candidate." +msgstr "" +"為了讓比較有意義,每個模型都應獲得相同的目標證據和工具。專門比較訓練效果時,還要固定基礎模型和可用知識。重複實驗有助於區分可靠的搜尋策略與偶然成功的候選方案。" + +#: website/public/blog/reverse-engineering-models/index.html:683 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would then separate two starting conditions: reconstructing a function " +"from its instructions, and closing the last few differences in maintained " +"source. A specialist might be excellent at the second task while a general " +"model does much of the work needed to reach it. Measuring them separately " +"would reveal that division of work." +msgstr "" +"我還會區分兩種起點:從指令開始重建函式,以及在已有原始碼中消除最後幾處差異。專用模型可能很擅長後者,而通用模型承擔了到達這個階段的大部分工作。分別評估,才能看清這種分工。" + +#: website/public/blog/reverse-engineering-models/index.html:691 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both tasks need genuinely held-out examples. Because related Touhou games " +"share engine code, holding out a game's name would still allow nearly " +"identical functions to appear in training. I would instead hold out source " +"families and keep their accepted answers out of both training and retrieval." +msgstr "" +"兩類任務都需要真正未見過的測試案例。東方系列的相關遊戲會共享引擎程式碼,只留出某個遊戲,仍可能讓幾乎相同的函式進入訓練集。我會改為按相關原始碼家族留出測試集,並把它們透過驗證的答案同時排除在訓練資料和檢索庫之外。" + +#: website/public/blog/reverse-engineering-models/index.html:698 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"On those examples, the outcome would be complete, independently validated " +"matches. Giving each model the same candidate or compile budget would help " +"compare search quality. Measuring elapsed time and total cost, including " +"compilation, would tell us whether that quality produces an efficient " +"workflow. Training costs would need their own accounting too." +msgstr "" +"在這些案例上,要統計的是完整、獨立驗證過的匹配成果。給每個模型相同的候選數量或編譯次數預算,可以比較搜尋品質。記錄總耗時和總成本,包括編譯,才能知道這種品質是否轉化成了高效的工作流程。訓練成本也要單獨計算。" + +#: website/public/blog/reverse-engineering-models/index.html:706 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Both views matter because a cheaper proposal can become an expensive " +"workflow if it causes more uninformative compiles. Conversely, a slower " +"model may pay for itself by finding a useful experiment sooner. The " +"comparison should capture those costs." +msgstr "" +"兩種角度都重要。一個便宜的提議,如果導致大量無效編譯,整個工作流程仍可能很貴。反過來,較慢的模型如果更早找到有價值的實驗,也可能更划算。對照實驗應當反映這些成本。" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:712 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"Counting accepted results also keeps intermediate progress in perspective. " +"One TH09 {ph0}replay-save candidate{ph1} looked promising because the " +"compiled function had the expected size, but a full comparison still found " +"substantial differences. Changes in how the compiler arranges instructions " +"can also make a candidate look closer without resolving the underlying " +"problem. Such diagnostics can guide the next experiment, but I would " +"validate any reward for \"getting closer\" against the full acceptance " +"check." +msgstr "" +"只統計透過驗收的結果,也能幫助我們合理看待中間進展。TH09 " +"的一個{ph0}錄影儲存候選方案{ph1}看起來很有希望,因為編譯後函式的大小符合預期,但完整比較仍發現大量差異。編譯器改變指令排列,也可能讓候選原始碼看起來更接近,卻沒有解決根本問題。這些診斷可以指導下一步,但若要獎勵“變得更接近”,我會先對照完整驗收檢查,確認這種獎勵有效。" + +#: website/public/blog/reverse-engineering-models/index.html:727 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"I would be interested in the pattern across these results. If a specialist " +"does well within a familiar compiler family, does that advantage survive a " +"toolchain change? And once both models have good retrieval, how much does " +"training still add? The answers would help us decide where each kind of " +"model is most useful." +msgstr "" +"我想看這些結果呈現出什麼規律。如果專用模型在熟悉的編譯器上表現很好,換了工具鏈後,這種優勢還在嗎?當雙方都擁有良好的檢索能力時,訓練還能增加多少價值?這些答案能幫助我們決定,每類模型最適合用在哪裡。" + +#: website/public/blog/reverse-engineering-models/index.html:736 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "What kind of agent does this suggest?" +msgstr "這意味著怎樣的 agent?" + +#: website/public/blog/reverse-engineering-models/index.html:737 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"If the models turn out to help at different stages, a useful arrangement " +"might be a broad agent with access to a specialist. Once a task becomes a " +"familiar compiler search, it could ask the specialist for candidates. If the" +" results stop making sense, the broader agent could return to the target " +"evidence or investigate the measurement itself." +msgstr "" +"如果不同模型在不同階段各有所長,一種有價值的安排可能是:讓能力全面的 agent " +"可以呼叫專用模型。當任務變成一種熟悉的編譯器搜尋時,它可以請專用模型提出候選方案。如果結果開始說不通,通用 agent " +"可以回頭檢查目標證據,或者調查測量本身。" + +#: website/public/blog/reverse-engineering-models/index.html:745 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"In that arrangement, specialization becomes a tool the agent can choose as " +"the investigation develops. The agent keeps the freedom to change direction " +"that made the earlier breakthroughs possible." +msgstr "" +"在這樣的安排中,專用模型成為 agent 隨著調查進展可以選擇的工具。agent 仍然擁有改變方向的自由,而這正是前面那些突破發生的重要條件。" + +#: website/public/blog/reverse-engineering-models/index.html:750 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"This follows naturally from the industrial shift I described in the previous" +" essay. Being able to run many more experiments changes how we organize the " +"work. It also gives us a reason to ask which decisions need broad judgment " +"and which improve through repeated experience with a particular craft." +msgstr "" +"這自然延續了上一篇文章中談到的工業化轉變。能夠開展更多實驗,會改變我們組織工作的方式,也讓我們有理由追問:哪些決策需要全面的判斷,哪些可以透過反覆積累某項具體技藝來改善?" + +#: website/public/blog/reverse-engineering-models/index.html:757 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"For now, the strongest general model has earned its place in my workflow by " +"opening problems I had left unresolved. Those successes are what make me " +"want to examine the work more closely: how much depended on the model's " +"breadth, and how much could be handled by a model trained in this particular" +" kind of reconstruction?" +msgstr "" +"目前,最強的通用模型解決了我過去沒能解決的問題,因此在我的工作流程中佔據了一席之地。正是這些成功,讓我想更仔細地研究過程:有多少依賴模型能力的廣度?又有多少可以交給專門學習這類重建的模型?" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:765 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}The repositories give us a way to test the question.{ph1} I would like " +"to use that record to build the comparison and see what we learn." +msgstr "{ph0}這些程式碼倉庫讓我們有機會驗證這個問題。{ph1} 我想利用這些記錄建立對照實驗,看看我們能學到什麼。" + +#: website/public/blog/reverse-engineering-models/index.html:775 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Evidence and model attribution" +msgstr "證據與模型歸屬" + +#: website/public/blog/reverse-engineering-models/index.html:783 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The examples describe historical function-level comparisons recorded in the " +"linked commits. Whole-game runtime checks are separate milestones. Source " +"and matching records were reviewed on 13 October 2026." +msgstr "" +"這些例子來自連結中的提交中記錄的歷史函式級比較。整款遊戲的執行時檢查屬於另一個里程碑。相關原始碼與匹配記錄核查於 2026 年 10 月 13 日。" + +#. {ph0}: gpt-dots +#. {ph1}: gpt-web +#: website/public/blog/reverse-engineering-models/index.html:789 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The TH09 label {ph0} identifies GPT-6-Astra. {ph1} changed with the model " +"used at the time, so that label alone does not identify an individual run. " +"Explicit model names in the other cited commits establish their recorded " +"attribution; the histories do not contain uniform per-run cost or effort " +"measurements." +msgstr "" +"TH09 中的標籤 {ph0} 對應 GPT-6-Astra。{ph1} " +"會隨當時使用的模型變化,因此僅憑這個標籤無法確定某次執行的具體模型。其他引用提交中的明確模型名,提供了對應的歸屬記錄;這些歷史記錄並沒有統一的單次執行成本或投入統計。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#: website/public/blog/reverse-engineering-models/index.html:798 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 temporary lifetimes:{ph1} the {ph2}earlier context review{ph3} and" +" {ph4}complete cold reconstruction{ph5}." +msgstr "{ph0}TH09 臨時值生命週期:{ph1}{ph2}早期上下文檢查{ph3}與{ph4}完整的乾淨重建{ph5}。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:810 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}TH09 Front callback:{ph1} {ph2}non-exact diagnostic regions{ph3}, " +"{ph4}Astra's new graph{ph5} and its {ph6}accepted recompile replay{ph7}." +msgstr "" +"{ph0}TH09 Front 回呼:{ph1}{ph2}未匹配的診斷區域{ph3}、{ph4}Astra " +"提出的新控制流圖{ph5}與{ph6}透過驗收的重新編譯檢查{ph7}。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:826 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Compiler knowledge:{ph1} {ph2}TH095 identifier-bucket reuse{ph3}, " +"{ph4}TH04 player source and linker order{ph5} and {ph6}Luna's TH09 enemy-" +"spawn closure{ph7}." +msgstr "" +"{ph0}編譯器知識:{ph1}{ph2}TH095 識別符號分桶規律的複用{ph3}、{ph4}TH04 " +"玩家原始碼與連結順序{ph5}、{ph6}Luna 完成的 TH09 敵人生成函式匹配{ph7}。" + +#. {ph0}: +#. {ph1}: +#. {ph2}: +#. {ph3}: +#. {ph4}: +#. {ph5}: +#. {ph6}: +#. {ph7}: +#: website/public/blog/reverse-engineering-models/index.html:842 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Checks and diagnostics:{ph1} {ph2}TH08 floating-literal audit{ph3}, " +"{ph4}TH08 numerical investigation{ph5} and {ph6}TH09 replay-save " +"reconstruction{ph7}." +msgstr "" +"{ph0}檢查與診斷:{ph1}{ph2}TH08 浮點常量審計{ph3}、{ph4}TH08 數值問題調查{ph5}、{ph6}TH09 " +"錄影儲存重建{ph7}。" + +#: website/public/blog/reverse-engineering-models/index.html:859 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"The proposed specialist and RL comparisons are future experiments. The " +"project histories motivate them; no trained specialist was evaluated for " +"this essay." +msgstr "文中提出的專用模型與 RL 對照,屬於未來實驗。專案歷史為這些想法提供了依據;本文尚未評估經過訓練的專用模型。" + +#: website/public/blog/reverse-engineering-models/index.html:6 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Does reverse engineering need a bigger model? | REA" +msgstr "逆向工程需要更大的模型嗎? | REA" + +#: website/public/blog/reverse-engineering-models/index.html:7 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"What Touhou reconstruction taught me about general models, specialist models" +" and learning to propose better hypotheses from compiler feedback." +msgstr "東方重建讓我如何思考通用模型、專用模型,以及怎樣從編譯器回饋中學會提出更好的假設。" + +#: website/public/blog/reverse-engineering-models/index.html:240 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "On this page" +msgstr "在本頁" + +#: website/public/blog/reverse-engineering-models/index.html:253 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Models and reverse engineering essay" +msgstr "模型與逆向工程隨筆" + +#: website/public/blog/reverse-engineering-models/index.html:491 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Selected reconstruction experiments" +msgstr "重建實驗選例" + +#: website/public/blog/reverse-engineering-models/index.html:883 +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "Back to top" +msgstr "回到頂部" + +#. {ph0}: +#. {ph1}: +#: website/public/blog/reverse-engineering-models/index.html:776 +#, python-brace-format +msgctxt "page:blog/reverse-engineering-models/index.html" +msgid "" +"{ph0}Detailed evidence notes{ph1} include the recorded comparisons and " +"experiment context." +msgstr "{ph0}詳細證據記錄{ph1}包含歷史比較結果與實驗背景。" + +#~ msgctxt "page:blog/index.html" +#~ msgid "By N0zoM1z0 · 13 October 2026 · Draft for review" +#~ msgstr "作者 N0zoM1z0 · 2026 年 10 月 13 日 · 待審草稿" + #~ msgctxt "page:index.html" #~ msgid "Setup guide" #~ msgstr "安裝指南" diff --git a/website/public/blog/index.html b/website/public/blog/index.html index 3ad338db8..a89e27807 100644 --- a/website/public/blog/index.html +++ b/website/public/blog/index.html @@ -196,6 +196,20 @@

Thoughts on reverse engineering.

+
+

Model specialization
Essay

+
+

Does reverse engineering need a bigger model?

+

+ What Touhou reconstruction suggests about general models, + specialist models and learning from compiler feedback. +

+ Read the article +

By N0zoM1z0 · 13 October 2026

+
+

Game reconstruction
Essay

diff --git a/website/public/blog/reverse-engineering-models/index.html b/website/public/blog/reverse-engineering-models/index.html new file mode 100644 index 000000000..a92f9f84f --- /dev/null +++ b/website/public/blog/reverse-engineering-models/index.html @@ -0,0 +1,888 @@ + + + + + + Does reverse engineering need a bigger model? | REA + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+ +

Essay

+

Does reverse engineering need a bigger model?

+

By N0zoM1z0 · 13 October 2026

+
+

Takeaway

+
    +
  • + Model capability changes the hypotheses an agent can + find. + Later sessions have solved problems that earlier work left + unresolved. +
  • +
  • + A familiar compiler gives specialization something concrete to + learn. + Some exact matches depend on narrow, reusable source patterns. +
  • +
  • + Give repository knowledge a fair comparison. + A smaller model with good retrieval may already know enough to + make progress. +
  • +
  • + Train the next useful experiment, and test its oracle. + The value of feedback depends on what the checker actually + verifies. +
  • +
  • + Measure accepted reconstruction against time and cost. + That is the result I want from the whole agent. +
  • +
+
+
+
+ +
+
+

What better models changed

+

+ Since August 2026, I have been exploring how much reverse + engineering an agent can carry out with access to a program and a + way to check its work. Most of my experience comes from + reconstructing games, especially Touhou. As I have changed models, + the repositories have kept a record of what each session managed + to recover. +

+

+ Much of that work aims at exact reconstruction: we compile the + maintained source under a pinned historical toolchain and check + whether it reproduces the target function. When a later model + succeeds where earlier work stalled, the source changes show what + made that progress possible. +

+

The temporary values had to stay alive together

+

+ Much of a TH09 attack function's logic had already been + reconstructed, but the compiler still stored temporary results + differently from the original. The remaining problem was how long + those results needed to stay alive and when their storage could be + reused. +

+

+ A + GPT-6-Sol checkpoint + recorded this mismatch after testing several ordinary source + arrangements. When + GPT-6.1-Sol returned to the problem, it found a useful hypothesis about how long those temporary + values needed to stay alive. +

+

+ The function produced temporary results and copied them into + animation positions. In C++, a temporary result stays alive until + the expression using it finishes. Grouping the real operations + into one expression kept those results alive together, so the + compiler could no longer reuse the same storage for all of them. +

+

+ One result still shared storage with a value used later for + collision checks. Giving that value its own lifetime resolved the + remaining overlap. After several targeted experiments, + a clean rebuild confirmed that the full function + matched. +

+

+ What interests me is how the discovery and the tests worked + together. The lifetime hypothesis gave the agent a new direction; + compiler feedback helped it refine that hypothesis into a source + arrangement that matched. +

+

A different control-flow graph

+

+ A second TH09 function shows why that new direction can matter. + Here, the obstacle was how several branches shared their final + work. + GPT-6.1-Sol's earlier work + got parts of the function to match, but the branches still joined + up differently from the original. +

+

+ GPT-6-Astra changed the surrounding control-flow graph. The main paths now led to a shared final update and return. In + that source structure, the compiler combined the branches in the + same way as the original. The complete function matched, and a + separate rebuild check confirmed it. +

+

+ Changing the representation opened a more productive search + space. + Both cases give me a reason to take general model capability + seriously: an agent can keep trying local variations for a long + time before finding a better way to express the surrounding work. +

+

+ At the same time, the later sessions started with earlier source + and recorded failures. That accumulated work was part of their + advantage. The history lets us inspect the discoveries and their + checks; a controlled comparison would help separate the effect of + the model from the experience it inherited. +

+

+ This is where the question becomes interesting to me. Some of the + progress came from finding a new representation, while much of the + remaining work involved testing source forms against a familiar + compiler. How much needs broad reasoning ability, and how much + could a model learn through repeated experience with that compiler + and code? +

+
+
+

Why might a specialist do well?

+

+ Touhou gives us a concrete reason to expect that experience to + help. Much of the game code comes from the same author, ZUN, so + recurring habits are a reasonable starting assumption. The + repositories back that up with useful examples: related loops and + helper arrangements have supplied hypotheses across games. +

+

+ Those source patterns become more specific when we include the + compiler. Under a particular version and set of options, small + changes in source can produce repeatable differences in machine + code. Some of those differences are surprisingly obscure. +

+

+ In TH095, for example, + even the name used for a local variable could change where + VC7.1 stored it. Moving the declaration alone did not solve the layout, but an + earlier background function had revealed a naming pattern that + affected the compiler's choice. That gave us a more promising + hypothesis for + the reconstruction of Supervisor::LoadDat. +

+

+ Reusing that pattern for the file-size variable produced the + required layout without adding storage. The full function then + matched. +

+

+ This is the sort of narrow detail a general model may not know + when it arrives. A specialist trained on this compiler could + encounter many such cases and learn which source variations are + worth testing first. +

+

+ The opportunity is especially clear in final matching problems + where we already understand the behavior. We are then asking which + ordinary source form this compiler turns into the observed + instructions. Specialization could help the model propose those + forms more efficiently, while the target still determines which + hypotheses survive. +

+

+ The useful patterns would have to retain their context, though. + TH08 uses an earlier Visual C++ version than TH095 and TH09. TH04 + uses a different DOS-era compiler, and its + player reconstruction + needed different hypotheses about source patterns and how the + compiled code was combined into an executable. Shared authorship + gives us a prior, but each target tells us whether it applies. + Distinguishing those contexts is part of what a specialist would + need to learn. +

+
+
+

Where should that experience live?

+

+ Before deciding to teach those patterns through model training, I + would look at the knowledge we already keep in the repository. In + the + previous essay, I wrote + about it becoming the project's working memory. That memory + includes compiler lessons, along with the source and failed + experiments that made them useful. +

+

+ By retrieving that record, an agent can start with a known result + and inspect how we reached it. It can also avoid alternatives that + have already failed in the same context. The starting point + improves without changing the model's weights. +

+

+ When reconstructing TH09's enemy-spawn function, a + GPT-6-Luna-max session + drew a hypothesis from TH08's corresponding source family. That + earlier source suggested how to structure the loop and handle + failures within it. TH09's own evidence supplied the game-specific + details, and clean rebuilds confirmed the complete match. +

+

+ "The model did not know that compiler quirk" and "the model + could not use it" are different problems. + Retrieval may solve the first, but turning that fact into a useful + candidate still takes work. And if the candidate fails, the model + must decide what the result means for its next attempt. Knowing + the relevant pattern is only the beginning of that process. +

+

+ For that reason, I would include a smaller general model with good + retrieval in any comparison. Giving it the same library of + experience would help us identify what training adds: better + candidate generation, better decisions after feedback, or both. +

+
+
+

Learning the next useful experiment

+

+ Those decisions after feedback are what I would most like to + study. In agentic reconstruction, the model proposes an + explanation and tests it against the program. Even when the + candidate fails, the result can narrow the remaining possibilities + and suggest a more useful next experiment. +

+

+ Consider the temporary-results example again. Its final source + shows the arrangement that worked, but the route to that source + contains more information. Each of the following experiments + helped identify what still needed to change. +

+
+ + + + + + + + + + + + + + + + + + + + + + + + + +
ExperimentWhat its result taught us
Try a different way to copy the results + The copy code changes, but the function still differs. +
Use the returned values directly + Copying follows the original pattern, but temporary + storage still differs. +
Keep the temporary results alive together + They get separate storage, but one still overlaps a value + used later for collision checks. +
Keep the collision-check value separateA clean rebuild confirms the complete match.
+
+

+ Selected experiments from the attack-function reconstruction + above, grouped by what they taught us. +

+

+ A training record that keeps this sequence can teach more than the + final source alone. It connects each proposed change to an + observed result and the next decision. Keeping ineffective + variations matters too: they tell a later agent which hypotheses + have already been tested in that context. +

+

+ I would want the model to learn how to use feedback. + A neutral result is a reason to reconsider the variation; a + changed mismatch may point to another cause. When local edits stop + being informative, the agent should be able to revisit the source + representation or inspect the evidence again. +

+

+ Learning to make those choices requires autonomy. Sometimes the + useful next step is inspecting another caller rather than + compiling another edit. A rigid sequence of required rewrites + would remove that choice, along with part of the behavior we want + the model to learn. +

+
+
+

Why RE is interesting for reinforcement learning

+

+ Bounded reconstruction gives us a practical environment for + studying these choices. Once the target and compiler are pinned, + many source experiments produce quick, repeatable feedback. The + agent can compile a candidate, inspect the mismatch and use that + observation to choose its next step. +

+

+ Reinforcement learning could train that decision-making process + through the feedback it receives. A candidate accepted by a + defined check gives us a concrete outcome, while the sequence of + attempts lets us observe how much work it took to reach it. +

+

+ There is already research supporting this direction. + LLM4Decompile + studies models specialized for decompilation. + AutoDecompiler + trains multi-turn refinement using compiler and execution + feedback. Their studies focus on functional recovery. For this + work, I would use them as a starting point for the related task of + finding source that matches a historical compiler's output. +

+

+ To reach such a well-defined task, though, we may first need to + establish what a function owns or devise a new runtime + measurement. In TH08, for example, a + collision difference led back to how the original program + handled floating-point calculations. Before the agent could test a useful correction, it had to work + out what to measure. +

+

+ This suggests another role for a broadly capable model: helping + make the problem checkable in the first place. Once that check + exists, we can investigate how far a learned search policy can get + with it. +

+
+
+

When the oracle is wrong

+

+ Once we train against that check, its coverage becomes part of + what the model learns. I call the reference or checker an oracle. + Even if it gives a repeatable answer, we still need to understand + what that answer covers. +

+

+ TH08 showed how much can hide outside that coverage. Some + reconstructed functions passed their exact comparisons with wrong + floating constants. The comparator adjusted relocated addresses + before checking instructions, but it checked the values behind + those addresses only when they had been explicitly recorded for + verification. Other literal references could pass without their + values being examined. +

+

+ To close that gap, a + wider audit + found incorrect constants in functions that had already been + accepted. In one calculation, the reconstructed source added + 1.0f where the original added 32.0f. +

+

+ A learner rewarded by the old check could learn to produce the + wrong reconstruction. + We had not trained an RL model against this oracle; the observed + failure was in the check itself. But using that check as a reward + would have made the same blind spot part of the learning + environment. +

+

+ A functional test can have a similar blind spot. + Differential-fuzzing research on LLM decompilers + has found behavior differences beyond the supplied tests. Passing + those tests therefore needs to be read in terms of the cases they + exercise, just as an instruction match needs to be read in terms + of the bytes and data it checks. +

+

+ Before using either kind of check as a reward, I would test it + with deliberate failures. I would also keep final acceptance + outside the candidate's editable source, so the model has to + satisfy the check through its reconstruction. Otherwise, something + like a byte array copied from the target could satisfy a naive + score without recovering useful source. +

+

+ Improving the learner and improving its oracle belong to the + same research problem. + As the learner gets better at optimizing the feedback we give it, + understanding that feedback becomes more important too. +

+
+
+

A comparison worth running

+

+ With those checks in place, I would start the comparison on a + fixed VC7.1 task family. First, I would establish baselines for a + strong general model and a smaller one. Adding repository + knowledge to the smaller model would show what retrieval changes. + From there, supervised specialization and an RL-trained version + would let us examine what training adds. +

+

+ To make those comparisons informative, each model would receive + the same target evidence and tools. For the training-specific + comparisons, I would keep the base model and available knowledge + fixed as well. Repeated trials would help distinguish a reliable + search strategy from a lucky candidate. +

+

+ I would then separate two starting conditions: reconstructing a + function from its instructions, and closing the last few + differences in maintained source. A specialist might be excellent + at the second task while a general model does much of the work + needed to reach it. Measuring them separately would reveal that + division of work. +

+

+ Both tasks need genuinely held-out examples. Because related + Touhou games share engine code, holding out a game's name would + still allow nearly identical functions to appear in training. I + would instead hold out source families and keep their accepted + answers out of both training and retrieval. +

+

+ On those examples, the outcome would be complete, independently + validated matches. Giving each model the same candidate or compile + budget would help compare search quality. Measuring elapsed time + and total cost, including compilation, would tell us whether that + quality produces an efficient workflow. Training costs would need + their own accounting too. +

+

+ Both views matter because a cheaper proposal can become an + expensive workflow if it causes more uninformative compiles. + Conversely, a slower model may pay for itself by finding a useful + experiment sooner. The comparison should capture those costs. +

+

+ Counting accepted results also keeps intermediate progress in + perspective. One TH09 + replay-save candidate + looked promising because the compiled function had the expected + size, but a full comparison still found substantial differences. + Changes in how the compiler arranges instructions can also make a + candidate look closer without resolving the underlying problem. + Such diagnostics can guide the next experiment, but I would + validate any reward for "getting closer" against the full + acceptance check. +

+

+ I would be interested in the pattern across these results. If a + specialist does well within a familiar compiler family, does that + advantage survive a toolchain change? And once both models have + good retrieval, how much does training still add? The answers + would help us decide where each kind of model is most useful. +

+
+
+

What kind of agent does this suggest?

+

+ If the models turn out to help at different stages, a useful + arrangement might be a broad agent with access to a specialist. + Once a task becomes a familiar compiler search, it could ask the + specialist for candidates. If the results stop making sense, the + broader agent could return to the target evidence or investigate + the measurement itself. +

+

+ In that arrangement, specialization becomes a tool the agent can + choose as the investigation develops. The agent keeps the freedom + to change direction that made the earlier breakthroughs possible. +

+

+ This follows naturally from the industrial shift I described in + the previous essay. Being able to run many more experiments + changes how we organize the work. It also gives us a reason to ask + which decisions need broad judgment and which improve through + repeated experience with a particular craft. +

+

+ For now, the strongest general model has earned its place in my + workflow by opening problems I had left unresolved. Those + successes are what make me want to examine the work more closely: + how much depended on the model's breadth, and how much could be + handled by a model trained in this particular kind of + reconstruction? +

+

+ The repositories give us a way to test the question. + I would like to use that record to build the comparison and see + what we learn. +

+
+
+
+ Evidence and model attribution +

+ Detailed evidence notes + include the recorded comparisons and experiment context. +

+

+ The examples describe historical function-level comparisons + recorded in the linked commits. Whole-game runtime checks are + separate milestones. Source and matching records were reviewed + on 13 October 2026. +

+

+ The TH09 label gpt-dots identifies GPT-6-Astra. + gpt-web changed with the model used at the time, so + that label alone does not identify an individual run. Explicit + model names in the other cited commits establish their recorded + attribution; the histories do not contain uniform per-run cost + or effort measurements. +

+ +

+ The proposed specialist and RL comparisons are future + experiments. The project histories motivate them; no trained + specialist was evaluated for this essay. +

+
+
+
+
+
+ + + +

+ + Top + + + + diff --git a/website/public/guides/browser/index.html b/website/public/guides/browser/index.html index 6437873c2..60185ddb6 100644 --- a/website/public/guides/browser/index.html +++ b/website/public/guides/browser/index.html @@ -477,6 +477,15 @@ link.click(); application workflow to follow files and process boundaries.

+

+ For HTTP request and response bodies, use + Fiddler Everywhere: SAZ captures, MCP and public skills. Export SAZ or HAR 1.2 for offline REA capture Evidence, or + connect Everywhere's MCP server to inspect captured traffic with + your agent. +

diff --git a/website/templates/header.html.tmpl b/website/templates/header.html.tmpl index 6ac759580..4e8d0caaa 100644 --- a/website/templates/header.html.tmpl +++ b/website/templates/header.html.tmpl @@ -7,7 +7,7 @@ >
- $language_picker +$language_picker