shifathosn
/
rea
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #79 from morluto/morl/roadmap-investigation-core 

Add evidence-backed investigation workflows
This commit is contained in:
morluto 2026-07-13 05:49:52 +08:00 committed by GitHub
parent 057279f820 293cea7723
commit 4045d92623
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
183 changed files with 26047 additions and 1278 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

65
README.md
View File

@ -10,11 +10,11 @@
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![50 MCP tools](https://img.shields.io/badge/MCP_tools-50-5c4ee5?style=flat-square)](#50-tools-for-investigation)
[![68 MCP tools](https://img.shields.io/badge/MCP_tools-68-5c4ee5?style=flat-square)](#68-tools-for-investigation)
[![Node.js 24](https://img.shields.io/badge/Node.js-24.18.x-339933?style=flat-square&logo=nodedotjs&logoColor=white)](https://nodejs.org/)
[![MIT license](https://img.shields.io/badge/license-MIT-f4c430?style=flat-square)](LICENSE)
[Quick start](#quick-start) · [Current status](#current-status) · [Investigation model](#the-investigation-model) · [50 tools](#50-tools-for-investigation) · [Roadmap](#roadmap) · [How it works](#how-it-works)
[Quick start](#quick-start) · [Current status](#current-status) · [Investigation model](#the-investigation-model) · [68 tools](#68-tools-for-investigation) · [Roadmap](#roadmap) · [How it works](#how-it-works)
<br />
@ -104,6 +104,14 @@ If macOS or an installer asks for confirmation, complete the prompt and run the
If process capture reports that its native PTY backend is unavailable, install Xcode command-line tools and run `npm run rebuild:native`. Linux source builds require Python, `make`, and a C++ toolchain. Compatible packaged binaries do not require this rebuild.
Process capture is disabled by default. Enabling it requires
`REA_PROCESS_CAPTURE_ENABLED=true`, approved executable and working roots in
`REA_PROCESS_EXECUTABLE_ROOTS_JSON` and `REA_PROCESS_WORKING_ROOTS_JSON`, and an
environment allowlist in `REA_PROCESS_ALLOWED_ENV_JSON`. The current PTY adapter
uses host networking, so it additionally requires the explicit operator grant
`REA_PROCESS_ALLOW_EXTERNAL_NETWORK=true`. Without that grant the adapter
refuses to launch; loopback replay alone does not constitute network isolation.
You do not need to install the reverse-engineering tools manually. Setup installs Homebrew and [Hopper](https://www.hopperapp.com/) when needed, configures detected Claude Desktop and Cursor installations, and installs the REA skill. Hopper is separate software and requires its own license; setup installs it but does not provide a license.
If something is not working, run:
@ -114,10 +122,29 @@ npx -y rea-agents doctor
### CLI or coding agent?
| If you want to… | Use |
| --------------------------------------------------------- | ------------------------------------------ |
| Ask an agent to investigate an app and build a feature | Install the skill, then talk to your agent |
| Inspect or decompile one part of an app from the Terminal | `rea analyze` or `rea decompile` |
| If you want to… | Use |
| --------------------------------------------------------- | ---------------------------------------------- |
| Ask an agent to investigate an app and build a feature | Install the skill, then talk to your agent |
| Inspect or decompile one part of an app from the Terminal | `rea analyze` or `rea decompile` |
| Validate or canonicalize an Evidence v2 bundle | `rea evidence-import` or `rea evidence-export` |
| Import source as historical reference | `rea import-reference-source` |
Filesystem evidence commands and MCP file tools are disabled until the operator approves absolute roots:
```bash
export REA_EVIDENCE_ROOTS_JSON='["/absolute/path/to/evidence"]'
rea evidence-import /absolute/path/to/evidence/bundle.json
rea evidence-export /absolute/path/to/evidence/bundle.json /absolute/path/to/evidence/canonical.json
```
Historical source import requires a separate allowlist and never treats source as current behavioral authority:
```bash
export REA_REFERENCE_ROOTS_JSON='["/absolute/path/to/source"]'
rea import-reference-source /absolute/path/to/source
```
Exports never replace an existing file unless `--overwrite` is explicit. Imports are size/depth bounded, validate every Evidence v2 ID and manifest, and never execute bundle content.
## One prompt, a full investigation
@ -151,26 +178,38 @@ REA handles the app analysis in steps 1–5. The agent performs step 6 with its
- Analyze Swift and Objective-C metadata without manually untangling every mangled symbol.
- Leave names, comments, and bookmarks in Hopper so human and agent analysis reinforce each other.
## 50 tools for investigation
## 68 tools for investigation
| Tool family | Count | Examples |
| ------------------------- | ----: | ----------------------------------------------------------------------------------------------------------------------- |
| Native inspection | 33 | procedures, pseudocode, assembly, strings, names, segments, callers, callees, xrefs, annotations |
| Investigation workflows | 10 | `binary_overview`, `analyze_function`, `batch_decompile`, `trace_feature`, call graphs, Swift and Objective-C discovery |
| Workspace and observation | 7 | target lifecycle, Evidence v2 bundle import/export, deterministic process capture and comparison |
| Native macOS utilities | 5 | Mach-O metadata, code signatures, plists, architectures, Swift demangling; Hopper-free and provenance-bearing |
| Artifact graph | 2 | deterministic directory, ZIP/APK/IPA, and ASAR inventory; explicitly selected extraction into an absent owned tree |
| Workspace and observation | 18 | target lifecycle, Evidence v2 bundles, process/artifact/function comparison, evidence-linked residual-unknown lifecycle |
The public interface describes what the agent is trying to learn. Providers decide how to answer. Hopper currently implements the native-analysis capabilities; the process harness implements controlled behavioral capture. Future providers can satisfy the same capability without changing the agent's investigation workflow.
The public interface describes what the agent is trying to learn. Providers decide how to answer. macOS utilities handle common semantic inspection without launching Hopper; Hopper handles deeper native analysis; the process harness implements controlled behavioral capture.
## Current status
REA is already useful for native application investigation on macOS:
- Open Mach-O, ELF, PE, `.app`, and Hopper database targets.
- Open Mach-O, ELF, PE, `.app`, ZIP, APK, IPA, ASAR, plist, JavaScript, source-map, and Hopper database targets.
- Traverse content-addressed artifact graphs without extraction; materialize only approved occurrences into absent output roots.
- Build bounded function dossiers with pseudocode, assembly, CFG edges, comments, calls, references, strings, and names.
- Search and trace features across symbols, strings, metadata, references, and call paths.
- Record every successful result as deterministic Evidence v2 with artifact and provider identity, confidence, authority, limitations, and locations.
- Export and import evidence bundles across sessions.
- Capture approved PTY scenarios, child processes, filesystem changes, and loopback HTTP/WebSocket exchanges, then compare normalized captures.
- Compare complete artifact inventories by stable path, content, metadata, and relations; incomplete evidence never implies equivalence.
- Compare explicit function dossiers across text, calls, references, strings, and address-normalized CFG topology with per-facet unknowns.
- Compare canonical Evidence bundles by exact membership, explicit observation pairs, and residual-unknown histories without turning omissions into behavioral absence.
- Aggregate runtime comparisons into observed behavior changes while keeping static artifact/function differences labeled as candidates.
- Build bounded, Evidence-cited direct call paths by exact address without treating missing dossiers as graph leaves.
- Correlate exact static/runtime findings through explicit hypotheses without claiming causality from cochange.
- Verify finite behavioral and structural reconstruction specifications with pass, fail, and unknown kept distinct.
- Track residual unknowns through immutable CAS revisions, evidence-qualified resolution, contradictions, probes, and validated dependency relationships.
- With explicit `unknown_registry_approved: true`, record bounded trace/capture residuals, typed provider unavailability, and capture disagreements automatically.
Hopper is the first provider, not the boundary of the project. Some current workflows still require Hopper and macOS; every evidence record identifies the provider and limitations behind its result.
@ -215,10 +254,14 @@ flowchart LR
REA --> Workspace["Investigation workspace<br/>evidence + artifacts + captures"]
Workspace --> Router["Capability router"]
Router --> Hopper["Hopper provider"]
Router --> Native["Native macOS provider"]
Router --> Artifact["Artifact graph provider"]
Router --> Process["Process capture provider"]
Router -. roadmap .-> More["Artifact, browser, dynamic,<br/>and additional static providers"]
Router -. roadmap .-> More["Browser, dynamic,<br/>and additional static providers"]
Hopper --> Target["Target software"]
Process --> Target
Native --> Target
Artifact --> Target
```
The CLI and MCP server use the same application workflows and evidence contracts. A provider declares which capabilities it supports and the side effects those capabilities may have. Terminal commands are short-lived; an MCP session can retain an active target and evidence ledger across an investigation.

12
README_ar.md
View File

@ -10,11 +10,11 @@
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![50 MCP tools](https://img.shields.io/badge/MCP_tools-50-5c4ee5?style=flat-square)](#منصة-من-50-أداة)
[![68 MCP tools](https://img.shields.io/badge/MCP_tools-68-5c4ee5?style=flat-square)](#منصة-من-68-أداة)
[![Node.js 24](https://img.shields.io/badge/Node.js-24.18.x-339933?style=flat-square&logo=nodedotjs&logoColor=white)](https://nodejs.org/)
[![MIT license](https://img.shields.io/badge/license-MIT-f4c430?style=flat-square)](LICENSE)
[البدء السريع](#البدء-السريع) · [من الملف التنفيذي إلى السلوك](#من-الملف-التنفيذي-إلى-السلوك) · [50 أداة](#منصة-من-50-أداة) · [كيف يعمل](#كيف-يعمل) · [الأسئلة الشائعة](#الأسئلة-الشائعة)
[البدء السريع](#البدء-السريع) · [من الملف التنفيذي إلى السلوك](#من-الملف-التنفيذي-إلى-السلوك) · [68 أداة](#منصة-من-68-أداة) · [كيف يعمل](#كيف-يعمل) · [الأسئلة الشائعة](#الأسئلة-الشائعة)
<br />
@ -145,13 +145,15 @@ npx -y rea-agents doctor
افحص التحقق من الإدخال، والتشفير، والصلاحيات، والتخزين، ومسارات الأخطاء من دون رفع الملف التنفيذي إلى خدمة بعيدة.
## منصة من 50 أداة
## منصة من 68 أداة
| عائلة الأدوات | العدد | أمثلة |
| --------------------- | ----: | ----------------------------------------------------------------------------------- |
| فحص الملفات التنفيذية | 33 | الدوال، والشيفرة شبه المصدرية، والتعليمات، والسلاسل، والأسماء، والمراجع، والتعليقات |
| التحليل المركب | 10 | النظرة العامة، وفك الترجمة الدفعي، ومخططات الاستدعاء، والمراجع، واكتشاف الأنواع |
| جلسة الملف التنفيذي | 7 | فتح الأهداف وحزم الأدلة والتقاط العمليات ومقارنتها |
| أدوات macOS الأصلية | 5 | بيانات Mach-O والتوقيعات وملفات plist والمعماريات وفك رموز Swift دون تشغيل Hopper |
| رسم القطع الأثرية | 2 | جرد حتمي للدلائل وZIP/APK/IPA وASAR واستخراج معاملاتي محدد صراحة |
| جلسة الملف التنفيذي | 18 | فتح الأهداف وحزم الأدلة ومقارنة العمليات والقطع والدوال وسجل المجهولات المتبقية |
## استخدام REA مع وكلاء برمجة آخرين
@ -200,7 +202,7 @@ npx -y rea-agents analyze /Applications/Notes.app
لتجنب مربع اختيار معمارية ملف شامل، حدد معطيات المحمل مسبقًا:
```bash
export HOPPER_LOADER_ARGS_JSON='["-l", "Mach-O", "--aarch64"]'
export HOPPER_LOADER_ARGS_JSON='["-l", "Mach-O", "--aarch68"]'
```
يدير REA دورة حياة Hopper ويحاول إغلاق العمليات التي بدأها عند انتهاء الأمر أو جلسة MCP.

18
README_ja.md
View File

@ -10,11 +10,11 @@
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![50 MCP tools](https://img.shields.io/badge/MCP_tools-50-5c4ee5?style=flat-square)](#50-ツールのワークベンチ)
[![68 MCP tools](https://img.shields.io/badge/MCP_tools-68-5c4ee5?style=flat-square)](#68-ツールのワークベンチ)
[![Node.js 24](https://img.shields.io/badge/Node.js-24.18.x-339933?style=flat-square&logo=nodedotjs&logoColor=white)](https://nodejs.org/)
[![MIT license](https://img.shields.io/badge/license-MIT-f4c430?style=flat-square)](LICENSE)
[クイックスタート](#クイックスタート) · [バイナリから動作へ](#バイナリから動作へ) · [50 ツール](#50-ツールのワークベンチ) · [仕組み](#仕組み) · [FAQ](#faq)
[クイックスタート](#クイックスタート) · [バイナリから動作へ](#バイナリから動作へ) · [68 ツール](#68-ツールのワークベンチ) · [仕組み](#仕組み) · [FAQ](#faq)
<br />
@ -135,13 +135,15 @@ REA は手順 1〜5 のバイナリ解析を処理し、手順 6 はエージェ
- Swift / Objective-C のメタデータを解析する。
- Hopper に名前、コメント、ブックマークを残し、人間とエージェントの調査を共有する。
## 50 ツールのワークベンチ
## 68 ツールのワークベンチ
| ツール群 | 数 | 例 |
| ------------------ | --: | ------------------------------------------------------------------------------------------------------------------- |
| バイナリ調査 | 33 | プロシージャ、疑似コード、アセンブリ、文字列、名前、セグメント、callers、callees、xrefs、注釈 |
| 合成解析 | 10 | `binary_overview`, `analyze_function`, `batch_decompile`, `get_call_graph`, `find_xrefs_to_name`, Swift / ObjC 検出 |
| バイナリセッション | 7 | `open_binary`、`binary_session`、証拠バンドル、プロセス取得と比較 |
| ツール群 | 数 | 例 |
| ---------------------- | --: | ------------------------------------------------------------------------------------------------------------------- |
| バイナリ調査 | 33 | プロシージャ、疑似コード、アセンブリ、文字列、名前、セグメント、callers、callees、xrefs、注釈 |
| 合成解析 | 10 | `binary_overview`, `analyze_function`, `batch_decompile`, `get_call_graph`, `find_xrefs_to_name`, Swift / ObjC 検出 |
| macOS ネイティブ | 5 | Mach-O メタデータ、署名、plist、アーキテクチャ、Swift デマングル。Hopper 起動不要 |
| アーティファクトグラフ | 2 | ディレクトリ、ZIP/APK/IPA、ASAR の決定的一覧と明示選択されたトランザクション抽出 |
| バイナリセッション | 18 | `open_binary`、`binary_session`、証拠バンドル、プロセス／成果物／関数比較、残余未知レジストリ |
## 他のコーディングエージェントで使う

18
README_ko.md
View File

@ -10,11 +10,11 @@
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![50 MCP tools](https://img.shields.io/badge/MCP_tools-50-5c4ee5?style=flat-square)](#50개-도구로-구성된-워크벤치)
[![68 MCP tools](https://img.shields.io/badge/MCP_tools-68-5c4ee5?style=flat-square)](#68개-도구로-구성된-워크벤치)
[![Node.js 24](https://img.shields.io/badge/Node.js-24.18.x-339933?style=flat-square&logo=nodedotjs&logoColor=white)](https://nodejs.org/)
[![MIT license](https://img.shields.io/badge/license-MIT-f4c430?style=flat-square)](LICENSE)
[빠른 시작](#빠른-시작) · [바이너리에서 동작까지](#바이너리에서-동작까지) · [50개 도구](#50개-도구로-구성된-워크벤치) · [작동 방식](#작동-방식) · [FAQ](#faq)
[빠른 시작](#빠른-시작) · [바이너리에서 동작까지](#바이너리에서-동작까지) · [68개 도구](#68개-도구로-구성된-워크벤치) · [작동 방식](#작동-방식) · [FAQ](#faq)
<br />
@ -135,13 +135,15 @@ REA는 1–5단계의 바이너리 분석을 처리합니다. 6단계는 에이
- Swift 및 Objective-C 메타데이터를 분석합니다.
- Hopper에 이름, 주석, 북마크를 남겨 사람과 에이전트의 분석을 연결합니다.
## 50개 도구로 구성된 워크벤치
## 68개 도구로 구성된 워크벤치
| 도구 그룹 | 수 | 예시 |
| ------------- | --: | -------------------------------------------------------------------------------------------------------------------- |
| 바이너리 검사 | 33 | 프로시저, 의사 코드, 어셈블리, 문자열, 이름, 세그먼트, callers, callees, xrefs, 주석 |
| 합성 분석 | 10 | `binary_overview`, `analyze_function`, `batch_decompile`, `get_call_graph`, `find_xrefs_to_name`, Swift 및 ObjC 탐색 |
| 바이너리 세션 | 7 | `open_binary`, `binary_session`, 증거 번들, 프로세스 캡처 및 비교 |
| 도구 그룹 | 수 | 예시 |
| ------------------- | --: | -------------------------------------------------------------------------------------------------------------------- |
| 바이너리 검사 | 33 | 프로시저, 의사 코드, 어셈블리, 문자열, 이름, 세그먼트, callers, callees, xrefs, 주석 |
| 합성 분석 | 10 | `binary_overview`, `analyze_function`, `batch_decompile`, `get_call_graph`, `find_xrefs_to_name`, Swift 및 ObjC 탐색 |
| macOS 네이티브 도구 | 5 | Mach-O 메타데이터, 코드 서명, plist, 아키텍처, Swift 디맹글링. Hopper 실행 불필요 |
| 아티팩트 그래프 | 2 | 디렉터리, ZIP/APK/IPA, ASAR 결정적 인벤토리와 명시적으로 선택한 트랜잭션 추출 |
| 바이너리 세션 | 18 | `open_binary`, `binary_session`, 증거 번들, 프로세스·아티팩트·함수 비교, 잔여 미지수 레지스트리 |
## 다른 코딩 에이전트에서 사용하기

18
README_zh.md
View File

@ -10,11 +10,11 @@
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![50 MCP tools](https://img.shields.io/badge/MCP_tools-50-5c4ee5?style=flat-square)](#50-个工具组成的工作台)
[![68 MCP tools](https://img.shields.io/badge/MCP_tools-68-5c4ee5?style=flat-square)](#68-个工具组成的工作台)
[![Node.js 24](https://img.shields.io/badge/Node.js-24.18.x-339933?style=flat-square&logo=nodedotjs&logoColor=white)](https://nodejs.org/)
[![MIT license](https://img.shields.io/badge/license-MIT-f4c430?style=flat-square)](LICENSE)
[快速开始](#快速开始) · [从二进制到行为](#从二进制到行为) · [50 个工具](#50-个工具组成的工作台) · [工作原理](#工作原理) · [常见问题](#常见问题)
[快速开始](#快速开始) · [从二进制到行为](#从二进制到行为) · [68 个工具](#68-个工具组成的工作台) · [工作原理](#工作原理) · [常见问题](#常见问题)
<br />
@ -135,13 +135,15 @@ REA 负责第 1–5 步中的二进制分析。第 6 步由智能体使用其常
- 分析 Swift 和 Objective-C 元数据。
- 在 Hopper 中留下名称、注释与书签，使人与智能体的分析互相增强。
## 50 个工具组成的工作台
## 68 个工具组成的工作台
| 工具类别 | 数量 | 示例 |
| ---------- | ---: | -------------------------------------------------------------------------------------------------------------------- |
| 二进制检查 | 33 | 过程、伪代码、汇编、字符串、名称、段、调用者、被调用者、交叉引用、注释 |
| 组合分析 | 10 | `binary_overview`, `analyze_function`, `batch_decompile`, `get_call_graph`, `find_xrefs_to_name`、Swift 与 ObjC 发现 |
| 二进制会话 | 7 | `open_binary`、`binary_session`、证据包、进程捕获与比较 |
| 工具类别 | 数量 | 示例 |
| -------------- | ---: | -------------------------------------------------------------------------------------------------------------------- |
| 二进制检查 | 33 | 过程、伪代码、汇编、字符串、名称、段、调用者、被调用者、交叉引用、注释 |
| 组合分析 | 10 | `binary_overview`, `analyze_function`, `batch_decompile`, `get_call_graph`, `find_xrefs_to_name`、Swift 与 ObjC 发现 |
| macOS 原生工具 | 5 | Mach-O 元数据、代码签名、plist、架构与 Swift 符号还原，无需启动 Hopper |
| 制品图谱 | 2 | 确定性目录、ZIP/APK/IPA 与 ASAR 清单，以及显式选择的事务式提取 |
| 二进制会话 | 18 | `open_binary`、`binary_session`、证据包、进程、制品与函数比较、残余未知项注册表 |
## 与其他编程智能体一起使用

154
bridge/hopper_bridge.py
View File

@ -9,11 +9,15 @@ import json
import hmac
import os
import re
import sre_parse
import socket
MAX_LINE_BYTES = 10 * 1024 * 1024
BAD_ADDRESSES = (-1, 0xFFFFFFFFFFFFFFFF, None)
_selected_document = None
_search_inventory_cache = {}
MAX_SEARCH_PATTERN_LENGTH = 256
MAX_SEARCH_VALUE_LENGTH = 4096
def _hex(value):
@ -89,6 +93,17 @@ def _procedure_identity(procedure):
return {"address": _hex(procedure.getEntryPoint()), "name": _procedure_name(procedure)}
def _procedure_locals(procedure):
"""Project opaque Hopper local-variable objects into an exact public shape."""
return [
{
"description": str(local),
"provenance": "hopper-public-python-api",
}
for local in procedure.getLocalVariableList()
]
def _containing_procedure(document, address):
segment = document.getSegmentAtAddress(address)
if segment is None:
@ -182,6 +197,105 @@ def _strings(document):
return result
def _invalidate_search_inventory(document):
"""Discard derived names after analysis metadata changes."""
document_id = id(document)
for key in list(_search_inventory_cache):
if key[0] == document_id:
del _search_inventory_cache[key]
def _search_inventory(document, kind):
"""Cache an immutable, address-sorted inventory for an unchanged document."""
key = (id(document), kind)
inventory = _search_inventory_cache.get(key)
if inventory is None:
values = _procedure_map(document) if kind == "procedure" else _strings(document)
inventory = tuple(sorted(values.items(), key=lambda item: int(item[0], 16)))
_search_inventory_cache[key] = inventory
return inventory
def _validate_regex_node(node, inside_repeat=False):
"""Reject regex structures with disproportionate or non-local evaluation cost."""
forbidden = {
sre_parse.ASSERT,
sre_parse.ASSERT_NOT,
sre_parse.GROUPREF,
sre_parse.GROUPREF_EXISTS,
}
repeat_tokens = {sre_parse.MAX_REPEAT, sre_parse.MIN_REPEAT}
possessive = getattr(sre_parse, "POSSESSIVE_REPEAT", None)
if possessive is not None:
repeat_tokens.add(possessive)
for operation, argument in node:
if operation in forbidden:
raise ValueError("Regex lookarounds and backreferences are not supported")
if operation in repeat_tokens:
if inside_repeat:
raise ValueError("Nested regex repetitions are not supported")
minimum, maximum, child = argument
if maximum == sre_parse.MAXREPEAT or maximum > 1000:
raise ValueError("Unbounded or excessive regex repetitions are not supported")
_validate_regex_node(child, True)
elif operation == sre_parse.SUBPATTERN:
_validate_regex_node(argument[-1], inside_repeat)
elif operation == sre_parse.BRANCH:
for branch in argument[1]:
_validate_regex_node(branch, inside_repeat)
def _search_page(document, kind, params):
pattern = params.get("pattern")
if not isinstance(pattern, str) or not pattern or len(pattern) > MAX_SEARCH_PATTERN_LENGTH:
raise ValueError("pattern must contain between 1 and 256 characters")
mode = params.get("mode", "literal")
if mode not in ("literal", "regex"):
raise ValueError("mode must be literal or regex")
case_sensitive = params.get("case_sensitive", False)
if not isinstance(case_sensitive, bool):
raise ValueError("case_sensitive must be a boolean")
offset = params.get("offset", 0)
limit = params.get("limit", 100)
if not isinstance(offset, int) or isinstance(offset, bool) or offset < 0:
raise ValueError("offset must be a non-negative integer")
if not isinstance(limit, int) or isinstance(limit, bool) or limit < 1 or limit > 100:
raise ValueError("limit must be an integer between 1 and 100")
if mode == "literal":
needle = pattern if case_sensitive else pattern.casefold()
matches = lambda value: needle in (value if case_sensitive else value.casefold())
else:
try:
parsed = sre_parse.parse(pattern)
_validate_regex_node(parsed)
expression = re.compile(pattern, 0 if case_sensitive else re.IGNORECASE)
except re.error as error:
raise ValueError("Invalid regex pattern") from error
matches = lambda value: expression.search(value) is not None
matching = [item for item in _search_inventory(document, kind) if matches(item[1])]
selected = matching[offset:offset + limit]
total = len(matching)
next_offset = offset + len(selected)
has_more = next_offset < total
return {
"items": [
{
"address": address,
"value": value[:MAX_SEARCH_VALUE_LENGTH],
"value_truncated": len(value) > MAX_SEARCH_VALUE_LENGTH,
}
for address, value in selected
],
"offset": offset,
"limit": limit,
"total": total,
"next_offset": next_offset if has_more else None,
"has_more": has_more,
}
def _page(values, offset, limit):
"""Return one deterministically address-sorted page without crossing an unbounded map."""
if not isinstance(offset, int) or isinstance(offset, bool) or offset < 0:
@ -354,7 +468,7 @@ def _analyze_function(document, params):
def collection(name, items, scan_limited=False):
return _bounded(items, _collection_offset(params, name), limit, None, scan_limited)
return {
"procedure": {"address": _hex(procedure.getEntryPoint()), "name": _procedure_name(procedure), "signature": procedure.signatureString(), "locals": _json_safe(procedure.getLocalVariableList())},
"procedure": {"address": _hex(procedure.getEntryPoint()), "name": _procedure_name(procedure), "signature": procedure.signatureString(), "locals": _procedure_locals(procedure)},
"pseudocode": {"text": pseudo_text, "total_chars": len(pseudo), "returned_chars": len(pseudo_text), "truncated": pseudo_next < len(pseudo), "next_offset": pseudo_next if pseudo_next < len(pseudo) else None},
"assembly": _bounded(assembly_lines, assembly_offset, max_instructions),
"comments": collection("comments", comments, instruction_scan_truncated),
@ -372,7 +486,7 @@ def _dispatch(method, params):
"""Dispatch only the closed operation set implemented by REA's public tools."""
global _selected_document
if method == "health":
return {"name": "REA Hopper bridge", "version": "1.0.0"}
return {"name": "REA Hopper bridge", "version": "1.0.0", "run_id": REA_RUN_ID}
if method == "shutdown":
return {"shutdown": True}
if method == "list_documents":
@ -426,18 +540,30 @@ def _dispatch(method, params):
return _hex(result)
if method == "list_segments":
result = []
permission_limitation = _unavailable(
"Hopper's public Python API does not expose segment or section permissions"
)
for segment in document.getSegmentsList():
start = segment.getStartingAddress()
sections = [{"name": section.getName(), "start": _hex(section.getStartingAddress()), "end": _hex(section.getStartingAddress() + section.getLength())} for section in segment.getSectionsList()]
sections = [{
"name": section.getName(),
"start": _hex(section.getStartingAddress()),
"end": _hex(section.getStartingAddress() + section.getLength()),
"readable": None,
"writable": None,
"executable": None,
"permissions": permission_limitation,
"provenance": "hopper-public-python-api",
} for section in segment.getSectionsList()]
result.append({
"name": segment.getName(),
"start": _hex(start),
"end": _hex(start + segment.getLength()),
"readable": None,
"writable": None,
"executable": None,
"permissions": _unavailable(
"Hopper's public Python API does not expose segment permissions"
),
"permissions": permission_limitation,
"provenance": "hopper-public-python-api",
"sections": sections,
})
return result
@ -461,10 +587,8 @@ def _dispatch(method, params):
result = {key: result[key]} if key in result else {}
return _page(result, params.get("offset", 0), params.get("limit", 100))
if method in ("search_procedures", "search_strings"):
flags = 0 if params.get("case_sensitive", False) else re.IGNORECASE
expression = re.compile(params["pattern"], flags)
values = _procedure_map(document) if method == "search_procedures" else _strings(document)
return {key: value for key, value in values.items() if expression.search(value)}
kind = "procedure" if method == "search_procedures" else "string"
return _search_page(document, kind, params)
if method.startswith("procedure_"):
procedure = _procedure(document, params.get("procedure"))
if method == "procedure_address":
@ -480,12 +604,16 @@ def _dispatch(method, params):
if method == "procedure_info":
blocks = list(procedure.basicBlockIterator())
length = sum(max(0, block.getEndingAddress() - block.getStartingAddress()) for block in blocks)
return {"name": _procedure_name(procedure), "entrypoint": _hex(procedure.getEntryPoint()), "basicblock_count": procedure.getBasicBlockCount(), "length": length, "signature": procedure.signatureString(), "locals": procedure.getLocalVariableList()}
return {"name": _procedure_name(procedure), "entrypoint": _hex(procedure.getEntryPoint()), "basicblock_count": procedure.getBasicBlockCount(), "length": length, "signature": procedure.signatureString(), "locals": _procedure_locals(procedure)}
if method == "set_address_name":
address = _address(document, params.get("address"))
return document.setNameAtAddress(address, params["name"])
result = document.setNameAtAddress(address, params["name"])
_invalidate_search_inventory(document)
return result
if method == "set_addresses_names":
return {key: document.setNameAtAddress(_address(document, key), value) for key, value in params["names"].items()}
result = {key: document.setNameAtAddress(_address(document, key), value) for key, value in params["names"].items()}
_invalidate_search_inventory(document)
return result
if method in ("set_comment", "set_inline_comment"):
address = _address(document, params.get("address"))
segment = _segment(document, address)

1202
package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

13
package.json
View File

@ -80,17 +80,30 @@
"*.ts": "oxlint --fix"
},
"dependencies": {
"@babel/parser": "8.0.4",
"@babel/types": "8.0.4",
"@electron/asar": "4.2.0",
"@modelcontextprotocol/server": "2.0.0-beta.3",
"@zip.js/zip.js": "2.8.26",
"canonicalize": "3.0.0",
"diff": "9.0.0",
"graphology": "0.26.0",
"graphology-shortest-path": "2.1.0",
"ignore": "7.0.6",
"incur": "0.4.13",
"isomorphic-git": "1.38.7",
"node-pty": "1.1.0",
"pino": "9.14.0",
"plist": "5.0.0",
"write-file-atomic": "8.0.0",
"ws": "8.21.0",
"zod": "4.4.3"
},
"devDependencies": {
"@fast-check/vitest": "0.4.1",
"@modelcontextprotocol/client": "2.0.0-beta.3",
"@types/node": "24.10.1",
"@types/write-file-atomic": "4.0.3",
"@types/ws": "8.18.1",
"@vitest/coverage-v8": "4.1.10",
"husky": "9.1.7",

2
scripts/rea.mjs
View File

@ -1,7 +1,7 @@
#!/usr/bin/env node
// Route production MCP before importing Incur. Incur owns registration helpers
// such as `mcp add`, while only dist/main.js may serve the 50-tool stdio server.
// such as `mcp add`, while only dist/main.js may serve the 68-tool stdio server.
const args = process.argv.slice(2);
const isMcpMode =
args[0] === "--mcp" || (args.length === 1 && args[0] === "mcp");

228
scripts/verify-package.mjs
View File

@ -13,12 +13,18 @@ import { promisify } from "node:util";
import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { TextReader, Uint8ArrayWriter, ZipWriter } from "@zip.js/zip.js";
import { TOOL_CONTRACTS } from "../dist/contracts/toolContracts.js";
const expectedToolCount = TOOL_CONTRACTS.length;
// Exercise the packed artifact in isolated HOME and prefix directories so the
// verifier cannot mutate real MCP registrations or rely on checkout-only files.
const exec = promisify(execFile);
const root = process.cwd();
const workspace = await mkdtemp(join(tmpdir(), "rea-package-"));
const evidenceRoot = join(workspace, "evidence");
const referenceRoot = join(workspace, "reference-source");
let tarball;
try {
@ -39,6 +45,21 @@ try {
const home = join(workspace, "home");
const fakeBin = join(workspace, "bin");
await mkdir(fakeBin, { recursive: true });
await Promise.all([
mkdir(evidenceRoot, { recursive: true }),
mkdir(join(referenceRoot, "src"), { recursive: true }),
]);
await Promise.all([
writeFile(
join(referenceRoot, "src", "main.ts"),
'import "./dependency";\n',
),
writeFile(
join(referenceRoot, "src", "dependency.ts"),
"export const value = 1;\n",
),
writeFile(join(referenceRoot, ".env"), "PACKAGE_SECRET_SENTINEL=1\n"),
]);
const brew = join(fakeBin, "brew");
const swVers = join(fakeBin, "sw_vers");
const npx = join(fakeBin, "npx");
@ -77,6 +98,9 @@ try {
PATH: `${fakeBin}:${process.env.PATH ?? ""}`,
HOPPER_LAUNCHER_PATH: hopper,
REA_NPX_LOG: npxLog,
REA_EVIDENCE_ROOTS_JSON: JSON.stringify([evidenceRoot]),
REA_REFERENCE_ROOTS_JSON: JSON.stringify([referenceRoot]),
REA_REFERENCE_SECRET_PATTERNS_JSON: JSON.stringify([".env"]),
};
await exec(
"npm",
@ -87,66 +111,165 @@ try {
const help = await run(cli, ["--help"], environment);
const llms = await run(cli, ["--llms"], environment);
const doctor = json(await run(cli, ["doctor", "--json"], environment));
const expectedDoctorHealth = process.platform === "darwin";
if (
!help.includes("setup") ||
!help.includes("inventory-artifact") ||
!help.includes("extract-artifact") ||
!help.includes("import-reference-source") ||
!llms.includes("decompile") ||
doctor.healthy !== true
!llms.includes("inventory-artifact") ||
doctor.healthy !== expectedDoctorHealth ||
doctor.checks?.find(({ name }) => name === "hopper")?.ok !== true
)
throw new Error("packaged CLI discovery or doctor failed");
throw new Error(
`packaged CLI discovery or doctor failed: ${JSON.stringify({ helpSetup: help.includes("setup"), llmsDecompile: llms.includes("decompile"), doctor })}`,
);
const artifactArchive = join(workspace, "artifact.zip");
const artifactWriter = new ZipWriter(new Uint8ArrayWriter());
await artifactWriter.add("app/main.js", new TextReader("main();"));
await writeFile(artifactArchive, await artifactWriter.close());
const artifactInventory = json(
await run(
cli,
["inventory-artifact", artifactArchive, "--limit", "500", "--json"],
environment,
),
);
if (
artifactInventory.operation !== "inventory_artifact" ||
artifactInventory.provider?.id !== "rea-artifact-graph" ||
artifactInventory.normalized_result?.manifest?.root_format !== "zip"
)
throw new Error("packaged artifact inventory CLI failed");
const referenceImport = json(
await run(
cli,
["import-reference-source", referenceRoot, "--json"],
environment,
),
);
if (
referenceImport.authority !== "historical-reference" ||
referenceImport.root_alias !== "$REFERENCE_ROOT" ||
referenceImport.relationships?.[0]?.resolution !== "internal" ||
JSON.stringify(referenceImport).includes("PACKAGE_SECRET_SENTINEL")
)
throw new Error("packaged historical reference import CLI failed");
const { createEvidence } = await import(
new URL("../dist/domain/evidence.js", import.meta.url)
);
const { createEvidenceBundle, serializeEvidenceBundle } = await import(
new URL("../dist/domain/evidenceBundle.js", import.meta.url)
);
const sourceBundle = createEvidenceBundle([
createEvidence(
undefined,
{ id: "package", name: "Package verifier", version: "1" },
{ operation: "health", parameters: {}, result: true },
),
]);
const sourceBundlePath = join(evidenceRoot, "source.json");
const canonicalBundlePath = join(evidenceRoot, "canonical.json");
await writeFile(sourceBundlePath, serializeEvidenceBundle(sourceBundle));
const importedBundle = json(
await run(
cli,
["evidence-import", sourceBundlePath, "--json"],
environment,
),
);
const exportedBundle = json(
await run(
cli,
["evidence-export", sourceBundlePath, canonicalBundlePath, "--json"],
environment,
),
);
if (
importedBundle.imported !== 1 ||
importedBundle.total !== 1 ||
exportedBundle.records !== 1 ||
(await readFile(canonicalBundlePath, "utf8")) !==
serializeEvidenceBundle(sourceBundle)
)
throw new Error("packaged CLI evidence bundle round trip failed");
await run(cli, ["mcp", "add"], environment);
const mcpRegistration = await readFile(npxLog, "utf8");
if (!mcpRegistration.includes("add-mcp npx -y rea-agents mcp --name rea"))
throw new Error("Incur mcp add did not register the floating npx command");
const skillPath = join(home, ".agents/skills/rea-analysis/SKILL.md");
const siblingSkillPath = join(home, ".agents/skills/unrelated/SKILL.md");
if (process.platform === "darwin") {
await mkdir(join(home, ".agents/skills/rea-analysis"), {
recursive: true,
});
await mkdir(join(home, ".agents/skills/unrelated"), { recursive: true });
await writeFile(skillPath, "stale managed skill\n");
await writeFile(siblingSkillPath, "unrelated skill\n");
}
const first = json(await run(cli, ["setup", "--yes", "--json"], environment));
const second = json(
await run(cli, ["setup", "--yes", "--json"], environment),
);
if (
first.status !== "ready" ||
second.status !== "ready" ||
second.actions.length !== 0
)
throw new Error("packaged setup was not ready and idempotent");
for (const configPath of [claudeConfig, cursorConfig]) {
const config = json(await readFile(configPath, "utf8"));
if (process.platform === "darwin") {
if (
config.existing !== true ||
config.mcpServers?.rea?.command !== "npx" ||
JSON.stringify(config.mcpServers?.rea?.args) !==
JSON.stringify(["-y", "rea-agents", "mcp"])
first.status !== "ready" ||
second.status !== "ready" ||
second.actions.length !== 0
)
throw new Error("packaged client readback failed");
throw new Error("packaged setup was not ready and idempotent");
for (const configPath of [claudeConfig, cursorConfig]) {
const config = json(await readFile(configPath, "utf8"));
if (
config.existing !== true ||
config.mcpServers?.rea?.command !== "npx" ||
JSON.stringify(config.mcpServers?.rea?.args) !==
JSON.stringify(["-y", "rea-agents", "mcp"])
)
throw new Error("packaged client readback failed");
if (
!(await readFile(`${configPath}.rea.backup`, "utf8")).includes(
"existing",
)
)
throw new Error("packaged client backup failed");
}
const skill = await readFile(skillPath, "utf8");
const canonicalSkill = await readFile(
join(root, "skills/rea-analysis/SKILL.md"),
"utf8",
);
if (skill !== canonicalSkill)
throw new Error("packaged skill did not match its canonical source");
if (
!(await readFile(`${configPath}.rea.backup`, "utf8")).includes("existing")
(await readFile(`${skillPath}.rea.backup`, "utf8")) !==
"stale managed skill\n" ||
(await readFile(siblingSkillPath, "utf8")) !== "unrelated skill\n"
)
throw new Error("packaged client backup failed");
}
const skill = await readFile(
join(home, ".agents/skills/rea-analysis/SKILL.md"),
"utf8",
);
const canonicalSkill = await readFile(
join(root, "skills/rea-analysis/SKILL.md"),
"utf8",
);
if (skill !== canonicalSkill)
throw new Error("packaged skill did not match its canonical source");
throw new Error("packaged stale-skill upgrade was not isolated");
await writeFile(cursorConfig, "malformed");
const failed = json(
await run(cli, ["setup", "--yes", "--json"], environment),
);
if (
failed.status !== "needs_human" ||
(await readFile(cursorConfig, "utf8")) !== "malformed"
)
throw new Error("packaged setup failure recovery did not preserve input");
await writeFile(cursorConfig, "{}\n");
const recovered = json(
await run(cli, ["setup", "--yes", "--json"], environment),
);
if (recovered.status !== "ready")
throw new Error("packaged setup did not recover");
await writeFile(cursorConfig, "malformed");
const failed = json(
await run(cli, ["setup", "--yes", "--json"], environment),
);
if (
failed.status !== "needs_human" ||
(await readFile(cursorConfig, "utf8")) !== "malformed"
)
throw new Error("packaged setup failure recovery did not preserve input");
await writeFile(cursorConfig, "{}\n");
const recovered = json(
await run(cli, ["setup", "--yes", "--json"], environment),
);
if (recovered.status !== "ready")
throw new Error("packaged setup did not recover");
} else if (
first.status !== "needs_human" ||
second.status !== "needs_human"
) {
throw new Error("packaged setup did not reject an unsupported host");
}
const transport = new StdioClientTransport({
command: cli,
@ -161,14 +284,27 @@ try {
const client = new Client({ name: "package-smoke", version: "1.0.0" });
try {
await client.connect(transport);
if ((await client.listTools()).tools.length !== 50)
throw new Error("packaged MCP did not expose 50 tools");
if ((await client.listTools()).tools.length !== expectedToolCount)
throw new Error("packaged MCP tool inventory diverged from contracts");
const result = await client.callTool({
name: "current_document",
arguments: {},
});
if (result.isError !== true)
throw new Error("packaged target-free MCP omitted no-target error");
const mcpBundlePath = join(evidenceRoot, "mcp.json");
const mcpExport = await client.callTool({
name: "export_evidence_bundle",
arguments: { path: mcpBundlePath },
});
if (mcpExport.isError === true)
throw new Error("packaged MCP evidence export failed");
const mcpImport = await client.callTool({
name: "import_evidence_bundle",
arguments: { path: mcpBundlePath },
});
if (mcpImport.isError === true)
throw new Error("packaged MCP evidence import failed");
} catch (cause) {
throw new Error(`packaged MCP smoke failed: ${mcpStderr}`, { cause });
} finally {
@ -177,7 +313,7 @@ try {
}
process.stdout.write(
`${JSON.stringify({ cli: true, incurMcpCommand: "npx -y rea-agents mcp", doctor: true, setup: "idempotent", clients: 2, backupReadback: true, failureRecovery: true, skill: true, mcpTools: 50, targetFree: true })}\n`,
`${JSON.stringify({ cli: true, artifactCli: true, evidenceCli: true, incurMcpCommand: "npx -y rea-agents mcp", doctor: "platform-appropriate", setup: process.platform === "darwin" ? "idempotent" : "unsupported-host-rejected", clients: process.platform === "darwin" ? 2 : 0, backupReadback: process.platform === "darwin", failureRecovery: process.platform === "darwin", skill: process.platform === "darwin", mcpTools: expectedToolCount, evidenceMcp: true, targetFree: true })}\n`,
);
} finally {
if (tarball) await rm(join(root, tarball), { force: true });

339
scripts/verify-real-hopper.mjs
View File

@ -6,6 +6,12 @@ import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { TOOL_CONTRACTS } from "../dist/contracts/toolContracts.js";
import {
firstProcedureAddress,
requireAddressArray,
requireFunctionDossier,
requirePseudocode,
} from "../dist/application/RealHopperAssertions.js";
const execFileAsync = promisify(execFile);
const timeout = 180_000;
@ -26,90 +32,176 @@ const requireSuccessfulTool = (result, operation) => {
throw new Error(`${operation} failed: ${textValue(result)}`);
}
const value = jsonValue(result);
if (value === null || typeof value !== "object" || !("result" in value)) {
if (
value === null ||
typeof value !== "object" ||
!("normalized_result" in value)
) {
throw new Error(`${operation} omitted its evidence result`);
}
return value.result;
return value.normalized_result;
};
const firstProcedureAddress = (result) => {
const items = result.structuredContent?.result?.items;
if (!Array.isArray(items) || items.length === 0) {
throw new Error("The verifier target contains no analyzed procedure");
}
const address = items[0]?.address;
if (typeof address !== "string" || !/^0x[0-9a-f]+$/iu.test(address)) {
throw new Error("list_procedures returned an invalid procedure address");
}
return address;
};
const requirePseudocode = (value, operation) => {
if (typeof value !== "string" || value.trim().length === 0) {
throw new Error(`${operation} returned empty pseudocode`);
}
if (value === "No output" || value.startsWith("Error:")) {
throw new Error(`${operation} returned an embedded failure: ${value}`);
const requireOverview = (value, operation) => {
if (
value === null ||
typeof value !== "object" ||
typeof value.document !== "string" ||
value.document.length === 0 ||
!Number.isInteger(value.procedure_count) ||
value.procedure_count < 1 ||
!Number.isInteger(value.segment_count) ||
value.segment_count < 1 ||
!Array.isArray(value.segments)
) {
throw new Error(`${operation} returned no analyzed binary overview`);
}
return value;
};
const requireFunctionDossier = (value, expectedAddress) => {
if (value === null || typeof value !== "object" || Array.isArray(value)) {
throw new Error("analyze_function returned no function dossier");
const requireTruthfulMemoryRegions = (segments) => {
if (!Array.isArray(segments) || segments.length === 0) {
throw new Error("list_segments returned no real memory regions");
}
if (value.procedure?.address !== expectedAddress) {
throw new Error("analyze_function returned the wrong procedure");
}
if (
typeof value.procedure?.name !== "string" ||
value.procedure.name.trim().length === 0
) {
throw new Error("analyze_function omitted the procedure name");
}
requirePseudocode(value.pseudocode?.text, "analyze_function");
for (const field of [
"comments",
"callers",
"callees",
"incoming_references",
"outgoing_references",
"referenced_strings",
"referenced_names",
"basic_blocks",
]) {
const collection = value[field];
if (
collection === null ||
typeof collection !== "object" ||
!Array.isArray(collection.items) ||
!Number.isInteger(collection.total)
) {
throw new Error(`analyze_function returned an invalid ${field} result`);
}
}
for (const field of ["callers", "callees"]) {
for (const identity of value[field].items) {
if (
typeof identity?.address !== "string" ||
typeof identity?.name !== "string"
) {
throw new Error(`analyze_function returned an untyped ${field} item`);
const regions = segments.flatMap((segment) => [
segment,
...(Array.isArray(segment.sections) ? segment.sections : []),
]);
for (const region of regions) {
for (const permission of ["readable", "writable", "executable"]) {
if (![true, false, null].includes(region?.[permission])) {
throw new Error(`list_segments omitted tri-state ${permission}`);
}
}
}
for (const block of value.basic_blocks.items) {
if (!Array.isArray(block?.successors)) {
throw new Error("analyze_function omitted CFG successor evidence");
if (
region.provenance !== "hopper-public-python-api" ||
region.permissions?.available !== false ||
typeof region.permissions.reason !== "string"
) {
throw new Error("list_segments omitted permission provenance");
}
}
if (
typeof value.instruction_scan?.scanned !== "number" ||
typeof value.instruction_scan?.truncated !== "boolean"
) {
throw new Error("analyze_function omitted instruction scan limitations");
};
const verifyRelationships = async (client, options, procedure) => {
const related = {};
for (const operation of ["procedure_callers", "procedure_callees"]) {
related[operation] = requireAddressArray(
requireSuccessfulTool(
await client.callTool(
{ name: operation, arguments: { procedure } },
options,
),
operation,
),
operation,
);
}
return value;
related.xrefs = requireAddressArray(
requireSuccessfulTool(
await client.callTool(
{ name: "xrefs", arguments: { address: procedure } },
options,
),
"xrefs",
),
"xrefs",
);
return related;
};
const verifyCurrentTarget = async (client, options) => {
const procedures = requireSuccessfulTool(
await client.callTool({ name: "list_procedures", arguments: {} }, options),
"list_procedures",
);
const firstAddress = firstProcedureAddress(procedures);
const containment = requireSuccessfulTool(
await client.callTool(
{
name: "resolve_containing_procedure",
arguments: { address: firstAddress },
},
options,
),
"resolve_containing_procedure",
);
if (
containment?.found !== true ||
containment.procedure?.address !== firstAddress
) {
throw new Error(
"resolve_containing_procedure returned the wrong procedure",
);
}
const references = requireSuccessfulTool(
await client.callTool(
{
name: "procedure_references",
arguments: {
procedure: firstAddress,
direction: "outgoing",
limit: 10,
max_instructions: 100,
},
},
options,
),
"procedure_references",
);
if (!Array.isArray(references?.references?.items)) {
throw new Error("procedure_references returned an invalid bounded result");
}
const relationships = await verifyRelationships(
client,
options,
firstAddress,
);
const boundedResult = requireSuccessfulTool(
await client.callTool(
{
name: "batch_decompile",
arguments: { addresses: [firstAddress] },
},
options,
),
"batch_decompile",
);
if (
boundedResult === null ||
typeof boundedResult !== "object" ||
Array.isArray(boundedResult) ||
Object.keys(boundedResult).length !== 1
) {
throw new Error("batch_decompile returned an invalid result");
}
const boundedPseudocode = requirePseudocode(
boundedResult[firstAddress],
"batch_decompile",
);
const dossier = requireFunctionDossier(
requireSuccessfulTool(
await client.callTool(
{
name: "analyze_function",
arguments: { procedure: firstAddress },
},
options,
),
"analyze_function",
),
firstAddress,
);
return {
procedure: dossier.procedure,
procedureCount: procedures.total,
boundedPseudocodeChars: boundedPseudocode.length,
analyzedPseudocodeChars: [...dossier.pseudocode.text].length,
callerCount: relationships.procedure_callers.length,
calleeCount: relationships.procedure_callees.length,
xrefCount: relationships.xrefs.length,
outgoingReferenceCount: references.references.items.length,
};
};
const requireSafeDiagnostics = (chunks) => {
@ -176,7 +268,7 @@ try {
const expectedNames = TOOL_CONTRACTS.map(({ name }) => name).sort();
const actualNames = listed.tools.map(({ name }) => name).sort();
if (JSON.stringify(actualNames) !== JSON.stringify(expectedNames)) {
throw new Error("The real server did not expose the intended 50 tools");
throw new Error("The real server did not expose the intended 68 tools");
}
const options = { timeout };
@ -203,77 +295,17 @@ try {
{ name: "list_segments", arguments: {} },
options,
);
const procedures = await client.callTool(
{ name: "list_procedures", arguments: {} },
options,
requireTruthfulMemoryRegions(
requireSuccessfulTool(segments, "list_segments"),
);
requireSuccessfulTool(procedures, "list_procedures");
const firstAddress = firstProcedureAddress(procedures);
const containment = requireSuccessfulTool(
await client.callTool(
{
name: "resolve_containing_procedure",
arguments: { address: firstAddress },
},
options,
),
"resolve_containing_procedure",
);
if (
containment?.found !== true ||
containment.procedure?.address !== firstAddress
) {
throw new Error(
"resolve_containing_procedure returned the wrong procedure",
);
}
const references = requireSuccessfulTool(
await client.callTool(
{
name: "procedure_references",
arguments: {
procedure: firstAddress,
direction: "outgoing",
limit: 10,
max_instructions: 100,
},
},
options,
),
"procedure_references",
);
if (!Array.isArray(references?.references?.items)) {
throw new Error("procedure_references returned an invalid bounded result");
}
const bounded = await client.callTool(
{ name: "batch_decompile", arguments: { addresses: [firstAddress] } },
options,
);
const boundedResult = requireSuccessfulTool(bounded, "batch_decompile");
if (
boundedResult === null ||
typeof boundedResult !== "object" ||
Array.isArray(boundedResult)
) {
throw new Error("batch_decompile returned an invalid result");
}
const boundedPseudocode = requirePseudocode(
boundedResult[firstAddress],
"batch_decompile",
);
const dossier = requireFunctionDossier(
requireSuccessfulTool(
await client.callTool(
{
name: "analyze_function",
arguments: { procedure: firstAddress },
},
options,
),
"analyze_function",
),
firstAddress,
const firstDocuments = requireSuccessfulTool(documents, "list_documents");
if (!Array.isArray(firstDocuments) || firstDocuments.length === 0)
throw new Error("list_documents returned no real Hopper document");
const firstOverview = requireOverview(
requireSuccessfulTool(overview, "binary_overview"),
"binary_overview",
);
const firstAnalysis = await verifyCurrentTarget(client, options);
const switched = await client.callTool(
{ name: "open_binary", arguments: { path: targetB } },
options,
@ -288,8 +320,11 @@ try {
{ name: "binary_overview", arguments: {} },
options,
);
if (secondOverview.isError === true)
throw new Error(textValue(secondOverview));
const verifiedSecondOverview = requireOverview(
requireSuccessfulTool(secondOverview, "binary_overview"),
"binary_overview after target switch",
);
const secondAnalysis = await verifyCurrentTarget(client, options);
const processList = await execFileAsync("ps", ["-ax", "-o", "command="]);
const bundledMcpRunning = processList.stdout
@ -307,25 +342,25 @@ try {
options,
);
if (closed.isError === true) throw new Error(textValue(closed));
const closedSession = jsonValue(
await client.callTool({ name: "binary_session", arguments: {} }, options),
);
if (closedSession.open !== false)
throw new Error("The real session remained open after close_binary");
summary = {
toolCount: actualNames.length,
documentCount: jsonValue(documents).length,
overview: jsonValue(overview),
segmentCount: jsonValue(segments).length,
boundedTool: "batch_decompile",
boundedInputCount: 1,
boundedResultKeys: Object.keys(boundedResult).length,
boundedPseudocodeChars: boundedPseudocode.length,
analyzedProcedure: dossier.procedure,
analyzedPseudocodeChars: dossier.pseudocode.text.length,
documentCount: firstDocuments.length,
overview: firstOverview,
segmentCount: firstOverview.segment_count,
analyses: [firstAnalysis, secondAnalysis],
bundledMcpRunning,
stderrBytes,
diagnosticCount,
dynamicSession: true,
targets: [targetA, targetB],
switched: true,
secondOverview: jsonValue(secondOverview),
secondOverview: verifiedSecondOverview,
};
} finally {
const keepAlive = setInterval(() => undefined, 100);

75
skills/rea-analysis/SKILL.md
View File

@ -1,6 +1,9 @@
---
name: rea-analysis
description: Reverse engineer apps with REA. Explore how features work, then build a version tailored to your project.
metadata:
version: "10"
tool_count: 68
---
# REA
@ -33,8 +36,80 @@ REA accepts a `.app` bundle directly. Do not expose its internal `Contents/MacOS
Briefly tell the user what you will investigate. Open the app with `open_binary`, begin with `binary_overview`, and narrow the investigation around the requested feature. Use decompilation, strings, names, callers, callees, and cross-references as needed.
For applications, ZIP/APK/IPA packages, or Electron ASAR archives, call
`inventory_artifact` before extraction. Follow its deterministic occurrence
pages and cite graph manifest IDs. `extract_artifact` requires explicit user
approval, an absent absolute output root, and selected occurrence IDs; never
extract every entry implicitly. Symlinks and encrypted entries are inventory
facts, not extractable files. Nested containers require a separate deliberate
inventory step.
Explain conclusions in plain language. Point to the relevant decompiled code, strings, names, and connections so the user can see how the explanation was reached. Do not claim to recover original source code or automatically clone an application.
Search uses bounded deterministic pages. Prefer literal mode; use regex mode only
when regex semantics are needed, and continue from `next_offset` while
`has_more` is true. Treat nullable permissions and explicit unavailable metadata
as unknown, never as `false`.
Every successful analysis result is Evidence v2. Cite evidence IDs and preserve
limitations and residual unknowns. Process capture is disabled by default,
requires per-call approval plus operator policy, and uses host networking only
when the operator explicitly permits it. It is behavioral evidence, not a
security sandbox.
Do not let unanswered questions disappear. Use `record_unknown` only with
explicit approval, attach supporting and contradicting evidence IDs, and record
the authority/environment still required. Use `update_unknown` with the current
`expected_revision`; stale updates must be re-read, not retried blindly. A
verified resolution needs qualifying observed evidence. Inference, withdrawn,
and out-of-scope dispositions are never substitutes for observed behavior.
Set `unknown_registry_approved: true` on `trace_feature` or
`capture_process_scenario` only when the user approves durable automatic
recording of their bounded residuals. The same flag on a direct operation
records typed provider unavailability, and on `compare_process_captures`
records an observed disagreement as a contradiction.
Use `compare_artifacts` with one record or bounded arrays of
`inventory_artifact` Evidence pages per side. Pages must share one manifest;
collect all node, occurrence, and edge pages for exhaustive comparison. It
compares stable occurrence paths, content, metadata, and graph relations,
cites both evidence sets on every delta, and paginates changes. Incomplete
inventories are truncated or unknown, never equivalent. Set
`unknown_registry_approved: true` only with approval to preserve the
disagreement or missing evidence as a residual unknown.
Use `compare_functions` with explicit `analyze_function` Evidence page sets;
it does not perform fuzzy whole-binary matching. Collect every pseudocode,
assembly, collection, and CFG page when exhaustive comparison matters.
Absolute addresses are volatile only in CFG topology; pseudocode constants are
never stripped. Omitted assembly, truncated scans, unavailable reference kinds,
and cross-provider text remain unknown rather than equal.
Use `compare_bundles` for canonical Evidence v2 bundle membership and
residual-unknown history changes. Pair cross-version observations explicitly;
the tool never guesses record identity. One-sided records prove only bundle
inclusion or omission, not behavioral absence. Use the returned canonical
bundle digests to anchor paginated reports.
Use `find_changed_behavior` to combine existing comparison Evidence. Runtime
process differences are observed changes; artifact and function differences
remain static candidates, not causal proof. Supply complete comparison pages.
Use `build_call_path` with explicit `analyze_function` Evidence groups from one
artifact and provider. Select endpoints by exact address. Missing dossiers,
incomplete callee pages, or a depth frontier make absence unknown; found paths
remain valid and cite each contributing dossier.
Use `correlate_static_and_runtime` only with explicit mappings between exact
static and runtime comparison findings. A matching pattern is a hypothesis,
never proof of causality. Declare side alignment; unmapped similarities are not
correlated.
Use `verify_reconstruction` with a finite typed specification and canonical
Evidence bundle. Pass means all declared claims passed with comparable
authority, not global implementation equivalence. Missing, limited, active-
unknown, or incompatible evidence stays unknown; observed differences fail.
## Build
When requested, use normal coding tools to build a version suited to the user's project, stack, interface, and requirements. Keep the implementation tied to what the investigation established, and distinguish observed behavior from assumptions or design choices.

78
src/application/AnalysisProvider.ts
View File

@ -1,19 +1,66 @@
import type { BinaryTarget } from "../domain/binaryTarget.js";
import type { OfficialToolName } from "../contracts/toolContracts.js";
import type { EnhancedToolName } from "../contracts/enhancedInputs.js";
import type { NativeToolName } from "../contracts/nativeToolContracts.js";
import type { ArtifactToolName } from "../contracts/artifactToolContracts.js";
import type { AnalysisError } from "../domain/errors.js";
import type { JsonValue } from "../domain/jsonValue.js";
import type { Result } from "../domain/result.js";
import type { EvidenceLocation } from "../domain/evidence.js";
import type { EvidenceSubjectTarget } from "../domain/evidence.js";
import { jsonValueSchema } from "../domain/jsonValue.js";
interface ExecutionOptions {
readonly signal?: AbortSignal;
}
/** Closed operations accepted by provider-backed analysis sessions. */
export type AnalysisOperation =
| OfficialToolName
| EnhancedToolName
| NativeToolName
| ArtifactToolName
| "health";
/** Atomic successful provider observation with exact producing provenance. */
export interface AnalysisExecution {
readonly result: JsonValue;
readonly rawResult: JsonValue | null;
readonly provider: ProviderIdentity;
readonly limitations: readonly string[];
readonly locations: readonly EvidenceLocation[];
readonly subject: EvidenceSubjectTarget | null;
}
/** Build a validated atomic provider observation at an adapter boundary. */
export const createAnalysisExecution = (
result: unknown,
provider: ProviderIdentity,
options: {
readonly rawResult?: unknown;
readonly limitations?: readonly string[];
readonly locations?: readonly EvidenceLocation[];
readonly subject?: EvidenceSubjectTarget;
} = {},
): AnalysisExecution => ({
result: jsonValueSchema.parse(result),
rawResult:
options.rawResult === undefined
? jsonValueSchema.parse(result)
: jsonValueSchema.parse(options.rawResult),
provider,
limitations: [...(options.limitations ?? [])],
locations: [...(options.locations ?? [])],
subject: options.subject ?? null,
});
/** Provider-neutral application capability for one analysis operation. */
export interface AnalysisOperationPort {
execute(
operation: string,
operation: AnalysisOperation,
parameters: Readonly<Record<string, JsonValue>>,
options?: ExecutionOptions,
): Promise<Result<JsonValue, AnalysisError>>;
): Promise<Result<AnalysisExecution, AnalysisError>>;
}
/** Provider session bound to exactly one parsed artifact. */
@ -29,26 +76,37 @@ export interface ProviderIdentity {
readonly version: string | null;
}
type AnalysisOperation = string;
interface CapabilityEffects {
readonly mutatesArtifact: boolean;
readonly launchesProcess: boolean;
readonly mayShowUi: boolean;
readonly mayAccessNetwork: boolean;
readonly mayWriteFilesystem: boolean;
readonly requiresPrivileges: boolean;
readonly changesPermissions: boolean;
readonly requiresRoot: boolean;
}
export interface CapabilityDescriptor {
readonly operation: AnalysisOperation;
readonly version: number;
readonly available: boolean;
type CapabilityAvailability =
| { readonly available: true; readonly reason: null }
| { readonly available: false; readonly reason: string };
interface CapabilityLimits {
readonly maxResults: number | null;
readonly maxPayloadBytes: number | null;
readonly timeoutMs: number | null;
}
export type CapabilityDescriptor = CapabilityAvailability & {
readonly provider: ProviderIdentity;
readonly operation: Exclude<AnalysisOperation, "health">;
readonly inputContractVersion: number;
readonly outputContractVersion: number;
readonly pagination: "none" | "offset" | "cursor";
readonly exhaustive: boolean;
readonly effects: CapabilityEffects;
readonly limits: CapabilityLimits;
readonly limitations: readonly string[];
}
};
/** Factory and capability declaration for an analysis implementation. */
export interface AnalysisProvider {

316
src/application/ArtifactExtraction.ts Normal file
View File

@ -0,0 +1,316 @@
import { lstat, realpath } from "node:fs/promises";
import { AsarArtifactReader } from "../artifacts/AsarArtifactReader.js";
import {
ArtifactPathRegistry,
normalizeArtifactPath,
} from "../artifacts/ArtifactPaths.js";
import {
ArtifactReaderFailure,
type ArtifactEntry,
type ArtifactLimits,
type ArtifactReader,
} from "../artifacts/ArtifactReader.js";
import { DirectoryArtifactReader } from "../artifacts/DirectoryArtifactReader.js";
import { SafeOutputTree } from "../artifacts/SafeOutputTree.js";
import { ZipArtifactReader } from "../artifacts/ZipArtifactReader.js";
import { MachOSliceArtifactReader } from "../artifacts/MachOSliceArtifactReader.js";
import {
artifactExtractionResultSchema,
type ArtifactExtractionResult,
type ArtifactGraphManifest,
type ArtifactNode,
type ArtifactOccurrence,
} from "../domain/artifactGraph.js";
import type { BinaryTarget } from "../domain/binaryTarget.js";
import { scanArtifactInventory } from "./ArtifactInventory.js";
import {
digestCanonical,
pageOf,
toOutputLimits,
} from "./ArtifactGraphConstruction.js";
const PAGE_SIZE = 500;
/** Already-approved extraction request from a caller-owned policy boundary. */
export interface ArtifactExtractionInput {
readonly inputPath: string;
readonly inputFormat: BinaryTarget["format"];
readonly outputRoot: string;
readonly occurrenceIds: readonly string[];
readonly offset: number;
readonly limit: number;
readonly limits: ArtifactLimits;
}
/** Extract selected inventory occurrences into an exclusively owned absent root. */
export const extractArtifact = async (
input: ArtifactExtractionInput,
signal?: AbortSignal,
): Promise<ArtifactExtractionResult> => {
validateSelection(input.occurrenceIds);
const sourcePath = await realpath(input.inputPath);
const selectedIds = new Set(input.occurrenceIds);
const inventory = await loadInventory(
sourcePath,
input.limits,
selectedIds,
signal,
);
const selected = input.occurrenceIds.map((id) => {
const occurrence = inventory.occurrences.get(id);
if (occurrence === undefined)
throw new ArtifactReaderFailure(
"unavailable",
`Selected artifact occurrence was not found: ${id}`,
);
if (
(occurrence.entry_kind !== "file" && occurrence.entry_kind !== "slice") ||
occurrence.artifact_id === null ||
occurrence.encrypted ||
occurrence.logical_path === "."
)
throw new ArtifactReaderFailure(
"format",
`Selected occurrence is not an extractable regular child file: ${id}`,
);
const node = inventory.nodes.get(occurrence.artifact_id);
if (node === undefined)
throw new ArtifactReaderFailure(
"integrity",
`Selected occurrence has no inventory node: ${id}`,
);
return { occurrence, node };
});
return materializeSelection({
input,
sourcePath,
inventory,
selected,
signal,
});
};
interface SelectedOccurrence {
readonly occurrence: ArtifactOccurrence;
readonly node: ArtifactNode;
}
interface ExtractedOccurrence {
readonly artifact_id: string;
readonly relative_path: string;
readonly sha256: string;
readonly bytes_written: number;
readonly created: true;
}
const materializeSelection = async ({
input,
sourcePath,
inventory,
selected,
signal,
}: {
readonly input: ArtifactExtractionInput;
readonly sourcePath: string;
readonly inventory: LoadedInventory;
readonly selected: readonly SelectedOccurrence[];
readonly signal: AbortSignal | undefined;
}): Promise<ArtifactExtractionResult> => {
const byPath = new Map(
selected.map((item) => [item.occurrence.logical_path, item]),
);
const reader = await createReader(sourcePath, input.inputFormat);
const output = await SafeOutputTree.create(input.outputRoot, input.limits);
let readerClosed = false;
const extracted: ExtractedOccurrence[] = [];
try {
const found = new Set<string>();
const registry = new ArtifactPathRegistry();
let entryCount = 0;
for await (const entry of reader.entries(signal)) {
entryCount += 1;
if (entryCount > input.limits.maxEntries)
throw new ArtifactReaderFailure(
"limit",
"Artifact entry limit exceeded during extraction",
);
const path = normalizeArtifactPath(entry.path, input.limits);
registry.add(path, entry.kind);
const selectedItem = byPath.get(path);
if (selectedItem === undefined) continue;
preflight(entry, input.limits);
const stream = await reader.open(entry, signal);
const written = await output.write(
path,
stream,
selectedItem.node.sha256,
signal,
);
extracted.push({
artifact_id: selectedItem.node.artifact_id,
relative_path: written.relativePath,
sha256: written.sha256,
bytes_written: written.bytesWritten,
created: true,
});
found.add(path);
}
if (found.size !== selected.length)
throw new ArtifactReaderFailure(
"integrity",
"Selected inventory occurrence disappeared before extraction",
);
await reader.close();
readerClosed = true;
extracted.sort((left, right) =>
left.relative_path.localeCompare(right.relative_path, "en"),
);
const result = createExtractionResult(
input,
inventory,
selected,
extracted,
);
await output.commit();
return result;
} catch (cause: unknown) {
if (!readerClosed) await reader.close().catch(() => undefined);
await output.rollback();
throw cause;
}
};
const createExtractionResult = (
input: ArtifactExtractionInput,
inventory: LoadedInventory,
selected: readonly SelectedOccurrence[],
extracted: readonly ExtractedOccurrence[],
): ArtifactExtractionResult => {
const extractionSemantic = {
schema_version: 1 as const,
source_manifest_id: inventory.manifest.manifest_id,
selected_occurrence_ids: selected
.map(({ occurrence }) => occurrence.occurrence_id)
.sort((left, right) => left.localeCompare(right)),
files_sha256: digestCanonical(extracted),
output_root_alias: "$OUTPUT_ROOT" as const,
};
return artifactExtractionResultSchema.parse({
manifest: inventory.manifest,
extraction_manifest: {
...extractionSemantic,
extraction_id: `aex_${digestCanonical(extractionSemantic)}`,
},
output_root: "$OUTPUT_ROOT",
artifacts: pageOf(extracted, input.offset, input.limit),
containment_verified: true,
cleanup: { attempted: false, verified: true, residual_paths: [] },
limits: toOutputLimits(input.limits),
provenance: [],
limitations: [
"Only caller-selected regular file occurrences were materialized.",
],
});
};
interface LoadedInventory {
readonly manifest: ArtifactGraphManifest;
readonly occurrences: ReadonlyMap<string, ArtifactOccurrence>;
readonly nodes: ReadonlyMap<string, ArtifactNode>;
}
const loadInventory = async (
path: string,
limits: ArtifactLimits,
selectedIds: ReadonlySet<string>,
signal?: AbortSignal,
): Promise<LoadedInventory> => {
const snapshot = await scanArtifactInventory(path, limits, signal);
const occurrences = new Map<string, ArtifactOccurrence>();
const neededNodes = new Set<string>();
collectOccurrences(
snapshot.occurrences,
selectedIds,
occurrences,
neededNodes,
);
const nodes = new Map<string, ArtifactNode>();
collectNodes(snapshot.nodes, neededNodes, nodes);
return { manifest: snapshot.manifest, occurrences, nodes };
};
const collectOccurrences = (
items: readonly ArtifactOccurrence[],
selected: ReadonlySet<string>,
output: Map<string, ArtifactOccurrence>,
neededNodes: Set<string>,
): void => {
for (const item of items) {
if (!selected.has(item.occurrence_id)) continue;
output.set(item.occurrence_id, item);
if (item.artifact_id !== null) neededNodes.add(item.artifact_id);
}
};
const collectNodes = (
items: readonly ArtifactNode[],
selected: ReadonlySet<string>,
output: Map<string, ArtifactNode>,
): void => {
for (const item of items)
if (selected.has(item.artifact_id)) output.set(item.artifact_id, item);
};
const createReader = async (
path: string,
format: BinaryTarget["format"],
): Promise<ArtifactReader> => {
if ((await lstat(path)).isDirectory())
return new DirectoryArtifactReader(path);
if (format === "asar") return new AsarArtifactReader(path);
if (format === "ipa" || format === "apk" || format === "zip")
return new ZipArtifactReader(path, format);
if (format === "mach-o") return new MachOSliceArtifactReader(path);
throw new ArtifactReaderFailure(
"unavailable",
`Artifact format has no extraction reader: ${format}`,
);
};
const validateSelection = (ids: readonly string[]): void => {
if (ids.length === 0 || ids.length > PAGE_SIZE)
throw new ArtifactReaderFailure(
"limit",
"Extraction requires 1 to 500 explicitly selected occurrences",
);
if (new Set(ids).size !== ids.length)
throw new ArtifactReaderFailure(
"path",
"Extraction occurrence selection contains duplicates",
);
};
const preflight = (entry: ArtifactEntry, limits: ArtifactLimits): void => {
if ((entry.kind !== "file" && entry.kind !== "slice") || entry.encrypted)
throw new ArtifactReaderFailure(
"format",
`Selected artifact entry cannot be read: ${entry.path}`,
);
if (entry.declaredSize !== null && entry.declaredSize > limits.maxEntryBytes)
throw new ArtifactReaderFailure(
"limit",
`Selected artifact exceeds byte limit: ${entry.path}`,
);
if (
entry.declaredSize !== null &&
entry.compressedSize !== null &&
(entry.compressedSize === 0
? entry.declaredSize > 0
: entry.declaredSize / entry.compressedSize > limits.maxCompressionRatio)
)
throw new ArtifactReaderFailure(
"limit",
`Selected artifact exceeds compression ratio limit: ${entry.path}`,
);
};

359
src/application/ArtifactGraphConstruction.ts Normal file
View File

@ -0,0 +1,359 @@
import { createHash } from "node:crypto";
import { extname } from "node:path";
import canonicalize from "canonicalize";
import type {
ArtifactEntry,
ArtifactLimits,
} from "../artifacts/ArtifactReader.js";
import type {
ArtifactCommand,
ArtifactEdge,
ArtifactNode,
ArtifactOccurrence,
} from "../domain/artifactGraph.js";
/** Mutable internal occurrence used until root-bound IDs are known. */
export interface MutableOccurrence {
occurrence_id: string;
artifact_id: string | null;
parent_occurrence_id: string | null;
logical_path: string;
entry_kind: ArtifactOccurrence["entry_kind"];
declared_size: number | null;
compressed_size: number | null;
executable: boolean;
encrypted: boolean;
hash_status: ArtifactOccurrence["hash_status"];
source_location: ArtifactOccurrence["source_location"];
limitations: string[];
}
export const createOccurrence = (
entry: ArtifactEntry,
path: string,
parent: string | null,
): MutableOccurrence => ({
occurrence_id: `occ_${digestCanonical({ path, kind: entry.kind })}`,
artifact_id: null,
parent_occurrence_id: parent,
logical_path: path,
entry_kind: entry.kind,
declared_size: entry.declaredSize,
compressed_size: entry.compressedSize,
executable: entry.executable,
encrypted: entry.encrypted,
hash_status: entry.encrypted ? "unavailable" : "not-hashed-limit",
source_location:
entry.byteOffset === null || entry.declaredSize === null
? null
: { offset: entry.byteOffset, length: entry.declaredSize },
limitations: [...entry.limitations],
});
export const materializeDirectoryNodes = (
occurrences: MutableOccurrence[],
nodes: Map<string, ArtifactNode>,
): void => {
const directories = occurrences
.filter(({ entry_kind: kind }) => kind === "directory")
.sort(
(left, right) => depth(right.logical_path) - depth(left.logical_path),
);
const childrenByParent = new Map<string, MutableOccurrence[]>();
for (const occurrence of occurrences) {
if (occurrence.parent_occurrence_id === null) continue;
const children = childrenByParent.get(occurrence.parent_occurrence_id);
if (children === undefined)
childrenByParent.set(occurrence.parent_occurrence_id, [occurrence]);
else children.push(occurrence);
}
for (const directory of directories) {
const children = (childrenByParent.get(directory.occurrence_id) ?? [])
.map(({ logical_path, artifact_id, entry_kind }) => ({
name: logical_path.split("/").at(-1),
artifact_id,
entry_kind,
}))
.sort((left, right) =>
String(left.name).localeCompare(String(right.name)),
);
const node = createArtifactNode({
sha256: digestCanonical({ kind: "directory", children }),
size: 0,
kind: directory.logical_path.toLowerCase().endsWith(".framework")
? "framework"
: "container",
format: "directory",
executable: false,
contentState: "virtual",
});
const existing = nodes.get(node.artifact_id);
nodes.set(
node.artifact_id,
existing?.kind === "framework" ? existing : node,
);
directory.artifact_id = node.artifact_id;
directory.hash_status = "verified";
}
};
export const createRootNode = (input: {
readonly path: string;
readonly format: ArtifactNode["format"];
readonly directory: boolean;
readonly digest: { readonly sha256: string; readonly bytes: number } | null;
readonly occurrences: readonly MutableOccurrence[];
}): ArtifactNode =>
createArtifactNode({
sha256:
input.digest?.sha256 ??
digestCanonical({
kind: "directory-root",
children: input.occurrences.map(({ logical_path, artifact_id }) => ({
logical_path,
artifact_id,
})),
}),
size: input.digest?.bytes ?? 0,
kind:
input.directory ||
["zip", "ipa", "apk", "asar", "dmg", "pkg"].includes(input.format)
? "container"
: classifyArtifactPath(input.path).kind,
format: input.format,
executable: false,
contentState: "materialized",
});
export const createArtifactNode = (input: {
readonly sha256: string;
readonly size: number;
readonly kind: ArtifactNode["kind"];
readonly format: ArtifactNode["format"];
readonly executable: boolean;
readonly contentState: ArtifactNode["content_state"];
}): ArtifactNode => ({
artifact_id: `art_${digestCanonical({ schema_version: 1, sha256: input.sha256 })}`,
kind: input.kind,
format: input.format,
sha256: input.sha256,
size: input.size,
media_type: null,
architecture: null,
executable: input.executable,
content_state: input.contentState,
limitations: [],
});
export const rootOccurrenceFor = (
node: ArtifactNode,
declaredSize: number,
): MutableOccurrence => ({
occurrence_id: `occ_${digestCanonical({ root: node.artifact_id })}`,
artifact_id: node.artifact_id,
parent_occurrence_id: null,
logical_path: ".",
entry_kind: node.format === "directory" ? "directory" : "file",
declared_size: declaredSize,
compressed_size: null,
executable: node.executable,
encrypted: false,
hash_status: "verified",
source_location: null,
limitations: [],
});
export const rekeyOccurrences = (
rootArtifactId: string,
occurrences: MutableOccurrence[],
): void => {
const replacements = new Map<string, string>();
for (const occurrence of occurrences)
replacements.set(
occurrence.occurrence_id,
`occ_${digestCanonical({
root_artifact_id: rootArtifactId,
logical_path: occurrence.logical_path,
entry_kind: occurrence.entry_kind,
})}`,
);
for (const occurrence of occurrences) {
occurrence.occurrence_id =
replacements.get(occurrence.occurrence_id) ?? occurrence.occurrence_id;
if (occurrence.parent_occurrence_id !== null)
occurrence.parent_occurrence_id =
replacements.get(occurrence.parent_occurrence_id) ??
occurrence.parent_occurrence_id;
}
};
export const createArtifactEdges = (
rootArtifactId: string,
occurrences: MutableOccurrence[],
producer?: ArtifactCommand,
): ArtifactEdge[] => {
const byId = new Map(occurrences.map((item) => [item.occurrence_id, item]));
const edges: ArtifactEdge[] = [];
for (const occurrence of occurrences) {
if (
occurrence.parent_occurrence_id === null ||
occurrence.artifact_id === null
)
continue;
const mappedSource = occurrence.logical_path.endsWith(".map")
? occurrences.find(
({ logical_path: path }) =>
path === occurrence.logical_path.slice(0, -".map".length),
)
: undefined;
const parentArtifactId =
mappedSource?.artifact_id ??
byId.get(occurrence.parent_occurrence_id)?.artifact_id ??
rootArtifactId;
const semantic = {
parent_artifact_id: parentArtifactId,
child_artifact_id: occurrence.artifact_id,
relation:
occurrence.entry_kind === "slice"
? ("slice-of" as const)
: relationFor(occurrence.logical_path),
occurrence_id: occurrence.occurrence_id,
logical_path: occurrence.logical_path,
};
edges.push({
edge_id: `edge_${digestCanonical(semantic)}`,
...semantic,
producer: occurrence.entry_kind === "slice" ? (producer ?? null) : null,
ordinal: edges.length,
});
}
return edges;
};
export const nearestParent = (
path: string,
occurrences: ReadonlyMap<string, MutableOccurrence>,
): MutableOccurrence | undefined => {
const parts = path.split("/");
while (parts.length > 1) {
parts.pop();
const candidate = occurrences.get(parts.join("/"));
if (candidate?.entry_kind === "directory") return candidate;
}
return undefined;
};
export const classifyArtifactPath = (
path: string,
): Pick<ArtifactNode, "kind" | "format"> => {
const lower = path.toLowerCase();
if (lower.endsWith(".map"))
return { kind: "source-map", format: "source-map" };
if (/\.(?:m?js|cjs)$/u.test(lower))
return { kind: "javascript", format: "javascript-bundle" };
if (lower.endsWith(".asar")) return { kind: "container", format: "asar" };
const archiveFormat = archiveFormatForExtension(extname(lower));
if (archiveFormat !== undefined)
return { kind: "container", format: archiveFormat };
if (/\.framework(?:\/|$)/u.test(lower))
return { kind: "framework", format: "file" };
if (/\.(?:node|dylib|so)$/u.test(lower))
return {
kind: lower.endsWith(".node") ? "native-addon" : "dynamic-library",
format: "file",
};
if (lower.endsWith(".plist")) return { kind: "plist", format: "plist" };
if (lower.endsWith(".entitlements"))
return { kind: "entitlements", format: "entitlements" };
return { kind: "resource", format: "file" };
};
/** Classify executable formats from bytes while retaining path-specific roles. */
export const classifyArtifactContent = (
path: string,
prefix: Buffer,
): Pick<ArtifactNode, "kind" | "format"> => {
const byPath = classifyArtifactPath(path);
if (prefix.length >= 4) {
const magic = prefix.readUInt32BE(0);
if ([0xcafebabe, 0xbebafeca, 0xcafebabf, 0xbfbafeca].includes(magic))
return {
kind: byPath.kind === "native-addon" ? "native-addon" : "executable",
format: "mach-o-universal",
};
if ([0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe].includes(magic))
return {
kind: ["native-addon", "dynamic-library"].includes(byPath.kind)
? byPath.kind
: "executable",
format: "mach-o",
};
if (prefix.subarray(0, 4).equals(Buffer.from([0x7f, 0x45, 0x4c, 0x46])))
return {
kind: ["native-addon", "dynamic-library"].includes(byPath.kind)
? byPath.kind
: "executable",
format: "elf",
};
}
if (prefix.length >= 2 && prefix[0] === 0x4d && prefix[1] === 0x5a)
return {
kind: byPath.kind === "native-addon" ? "native-addon" : "executable",
format: "pe",
};
return byPath;
};
export const pageOf = <Value>(
items: readonly Value[],
offset: number,
limit: number,
) => ({
items: items.slice(offset, offset + limit),
offset,
limit,
total: items.length,
next_offset: offset + limit < items.length ? offset + limit : null,
});
export const toOutputLimits = (limits: ArtifactLimits) => ({
max_entries: limits.maxEntries,
max_total_bytes: limits.maxTotalBytes,
max_entry_bytes: limits.maxEntryBytes,
max_compression_ratio: limits.maxCompressionRatio,
max_depth: limits.maxDepth,
max_path_bytes: limits.maxPathBytes,
});
export const digestCanonical = (value: unknown): string => {
const encoded = canonicalize(value);
if (encoded === undefined)
throw new TypeError("Artifact value is not canonical JSON");
return createHash("sha256").update(encoded).digest("hex");
};
const relationFor = (path: string): ArtifactEdge["relation"] => {
const lower = path.toLowerCase();
if (lower.endsWith(".map")) return "maps-source";
if (lower.includes(".framework/")) return "embeds";
return "contains";
};
const archiveFormatForExtension = (
extension: string,
): "zip" | "ipa" | "apk" | undefined => {
switch (extension) {
case ".zip":
return "zip";
case ".ipa":
return "ipa";
case ".apk":
return "apk";
default:
return undefined;
}
};
const depth = (path: string): number => path.split("/").length;

415
src/application/ArtifactInventory.ts Normal file
View File

@ -0,0 +1,415 @@
import { createHash } from "node:crypto";
import { createReadStream } from "node:fs";
import { lstat, open, realpath } from "node:fs/promises";
import type { Readable } from "node:stream";
import { AsarArtifactReader } from "../artifacts/AsarArtifactReader.js";
import {
ArtifactPathRegistry,
normalizeArtifactPath,
} from "../artifacts/ArtifactPaths.js";
import {
ArtifactReaderFailure,
type ArtifactEntry,
type ArtifactLimits,
type ArtifactReader,
} from "../artifacts/ArtifactReader.js";
import { DirectoryArtifactReader } from "../artifacts/DirectoryArtifactReader.js";
import { ZipArtifactReader } from "../artifacts/ZipArtifactReader.js";
import { MachOSliceArtifactReader } from "../artifacts/MachOSliceArtifactReader.js";
import { streamChunkToBuffer } from "../artifacts/StreamBytes.js";
import {
artifactInventoryResultSchema,
type ArtifactInventoryResult,
type ArtifactNode,
} from "../domain/artifactGraph.js";
import {
classifyArtifactPath,
classifyArtifactContent,
createArtifactEdges,
createArtifactNode,
createOccurrence,
createRootNode,
digestCanonical,
materializeDirectoryNodes,
nearestParent,
pageOf,
rekeyOccurrences,
rootOccurrenceFor,
toOutputLimits,
type MutableOccurrence,
} from "./ArtifactGraphConstruction.js";
interface InventoryPageInput {
readonly nodeOffset: number;
readonly nodeLimit: number;
readonly occurrenceOffset: number;
readonly occurrenceLimit: number;
readonly edgeOffset: number;
readonly edgeLimit: number;
}
/** Immutable inventory produced by one complete artifact scan. */
export interface ArtifactInventorySnapshot {
readonly manifest: ArtifactInventoryResult["manifest"];
readonly nodes: readonly ArtifactNode[];
readonly occurrences: ArtifactInventoryResult["occurrences"]["items"];
readonly edges: ArtifactInventoryResult["edges"]["items"];
readonly limits: ArtifactInventoryResult["limits"];
readonly provenance: ReadonlyArray<
ArtifactInventoryResult["provenance"][number]
>;
readonly limitations: readonly string[];
}
/** Inventory one local artifact without extracting or mounting it. */
export const inventoryArtifact = async (
inputPath: string,
limits: ArtifactLimits,
page: InventoryPageInput,
signal?: AbortSignal,
): Promise<ArtifactInventoryResult> => {
const snapshot = await scanArtifactInventory(inputPath, limits, signal);
return paginateArtifactInventory(snapshot, page);
};
/** Scan an artifact once and retain the complete immutable graph for projection. */
export const scanArtifactInventory = async (
inputPath: string,
limits: ArtifactLimits,
signal?: AbortSignal,
): Promise<ArtifactInventorySnapshot> => {
abortIfNeeded(signal);
const path = await realpath(inputPath);
const metadata = await lstat(path);
const rootFormat = await classifyRoot(path, metadata.isDirectory());
const rootDigest = metadata.isDirectory()
? null
: await hashReadable(createReadStream(path), limits.maxTotalBytes, signal);
const reader = await createReader(path, rootFormat);
try {
const { nodes, occurrences } = await scanReader(reader, limits, signal);
materializeDirectoryNodes(occurrences, nodes);
const rootNode = createRootNode({
path,
format: rootFormat,
directory: metadata.isDirectory(),
digest: rootDigest,
occurrences,
});
nodes.set(rootNode.artifact_id, rootNode);
rekeyOccurrences(rootNode.artifact_id, occurrences);
const rootOccurrence = rootOccurrenceFor(rootNode, metadata.size);
for (const occurrence of occurrences)
if (occurrence.parent_occurrence_id === null)
occurrence.parent_occurrence_id = rootOccurrence.occurrence_id;
occurrences.unshift(rootOccurrence);
const edges = createArtifactEdges(
rootNode.artifact_id,
occurrences,
reader?.provenance()[0],
);
const orderedNodes = [...nodes.values()].sort((left, right) =>
left.artifact_id.localeCompare(right.artifact_id),
);
const orderedOccurrences = occurrences.sort((left, right) =>
left.logical_path.localeCompare(right.logical_path, "en"),
);
const orderedEdges = edges.sort((left, right) =>
left.edge_id.localeCompare(right.edge_id),
);
if (rootDigest !== null) {
const verified = await hashReadable(
createReadStream(path),
limits.maxTotalBytes,
signal,
);
if (
verified.sha256 !== rootDigest.sha256 ||
verified.bytes !== rootDigest.bytes
)
throw new ArtifactReaderFailure(
"integrity",
"Root artifact changed during inventory",
);
}
const graphSha256 = digestCanonical({
nodes: orderedNodes,
occurrences: orderedOccurrences,
edges: orderedEdges,
});
const manifestId = `agm_${digestCanonical({
schema_version: 1,
root_artifact_id: rootNode.artifact_id,
graph_sha256: graphSha256,
})}`;
return {
manifest: artifactInventoryResultSchema.shape.manifest.parse({
schema_version: 1,
manifest_id: manifestId,
root_artifact_id: rootNode.artifact_id,
root_sha256: rootNode.sha256,
root_format: rootNode.format,
graph_sha256: graphSha256,
node_count: orderedNodes.length,
occurrence_count: orderedOccurrences.length,
edge_count: orderedEdges.length,
}),
nodes: orderedNodes,
occurrences: orderedOccurrences,
edges: orderedEdges,
limits: toOutputLimits(limits),
provenance: reader?.provenance() ?? [],
limitations: inventoryLimitations(rootFormat, reader),
};
} finally {
await reader?.close();
}
};
/** Project independently paged graph collections from one inventory snapshot. */
export const paginateArtifactInventory = (
snapshot: ArtifactInventorySnapshot,
page: InventoryPageInput,
): ArtifactInventoryResult =>
artifactInventoryResultSchema.parse({
...snapshot,
nodes: pageOf(snapshot.nodes, page.nodeOffset, page.nodeLimit),
occurrences: pageOf(
snapshot.occurrences,
page.occurrenceOffset,
page.occurrenceLimit,
),
edges: pageOf(snapshot.edges, page.edgeOffset, page.edgeLimit),
});
const inventoryLimitations = (
format: ArtifactNode["format"],
reader: ArtifactReader | undefined,
): string[] => {
if (reader !== undefined) return [];
if (format === "dmg" || format === "pkg")
return [
`${format.toUpperCase()} root hash is observed; child inventory requires an approved native macOS adapter.`,
];
if (format === "mach-o-universal" && process.platform !== "darwin")
return ["Universal Mach-O slices require the native macOS lipo adapter."];
return ["Artifact has no child container entries."];
};
const createReader = async (
path: string,
format: ArtifactNode["format"],
): Promise<ArtifactReader | undefined> => {
switch (format) {
case "directory":
return new DirectoryArtifactReader(path);
case "zip":
case "ipa":
case "apk":
return new ZipArtifactReader(path, format);
case "asar":
return new AsarArtifactReader(path);
case "mach-o-universal":
return process.platform === "darwin"
? new MachOSliceArtifactReader(path)
: undefined;
default:
return undefined;
}
};
const scanReader = async (
reader: ArtifactReader | undefined,
limits: ArtifactLimits,
signal?: AbortSignal,
): Promise<{
readonly nodes: Map<string, ArtifactNode>;
readonly occurrences: MutableOccurrence[];
}> => {
const nodes = new Map<string, ArtifactNode>();
const occurrences: MutableOccurrence[] = [];
if (reader === undefined) return { nodes, occurrences };
const occurrenceByPath = new Map<string, MutableOccurrence>();
const registry = new ArtifactPathRegistry();
let totalBytes = 0;
for await (const entry of reader.entries(signal)) {
if (occurrences.length >= limits.maxEntries)
throw new ArtifactReaderFailure("limit", "Artifact entry limit exceeded");
const logicalPath = normalizeArtifactPath(entry.path, limits);
registry.add(logicalPath, entry.kind);
preflightEntry(entry, limits);
const parent = nearestParent(logicalPath, occurrenceByPath);
const occurrence = createOccurrence(
entry,
logicalPath,
parent?.occurrence_id ?? null,
);
if ((entry.kind === "file" || entry.kind === "slice") && !entry.encrypted) {
const remainingBytes = limits.maxTotalBytes - totalBytes;
if (remainingBytes <= 0)
throw new ArtifactReaderFailure(
"limit",
"Artifact total byte limit exceeded",
);
if (entry.declaredSize !== null && entry.declaredSize > remainingBytes)
throw new ArtifactReaderFailure(
"limit",
"Declared artifact bytes exceed remaining cumulative limit",
);
const digest = await hashReadable(
await reader.open(entry, signal),
Math.min(limits.maxEntryBytes, remainingBytes),
signal,
);
totalBytes += digest.bytes;
if (
entry.declaredSha256 !== null &&
entry.declaredSha256 !== digest.sha256
)
throw new ArtifactReaderFailure(
"integrity",
`Artifact integrity metadata disagrees with content: ${logicalPath}`,
);
if (totalBytes > limits.maxTotalBytes)
throw new ArtifactReaderFailure(
"limit",
"Artifact total byte limit exceeded",
);
const classified =
entry.kind === "slice"
? ({ kind: "universal-slice", format: "mach-o" } as const)
: classifyArtifactContent(logicalPath, digest.prefix);
const node = createArtifactNode({
sha256: digest.sha256,
size: digest.bytes,
kind: classified.kind,
format: classified.format,
executable: entry.executable,
contentState: "embedded",
});
nodes.set(node.artifact_id, node);
occurrence.artifact_id = node.artifact_id;
occurrence.hash_status = "verified";
}
occurrences.push(occurrence);
occurrenceByPath.set(logicalPath, occurrence);
}
return { nodes, occurrences };
};
const classifyRoot = async (
path: string,
directory: boolean,
): Promise<ArtifactNode["format"]> => {
if (directory) return "directory";
const extensionFormat = classifyContainerExtension(path);
if (extensionFormat !== undefined) return extensionFormat;
const handle = await open(path, "r");
try {
const magic = Buffer.alloc(4);
const observed = await handle.read(magic, 0, magic.length, 0);
if (
observed.bytesRead === 4 &&
magic[0] === 0x50 &&
magic[1] === 0x4b &&
[0x03, 0x05, 0x07].includes(magic[2] ?? -1) &&
[0x04, 0x06, 0x08].includes(magic[3] ?? -1)
)
return "zip";
if (observed.bytesRead === 4) {
const header = magic.readUInt32BE(0);
if ([0xcafebabe, 0xbebafeca, 0xcafebabf, 0xbfbafeca].includes(header))
return "mach-o-universal";
if ([0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe].includes(header))
return "mach-o";
if (magic.equals(Buffer.from([0x7f, 0x45, 0x4c, 0x46]))) return "elf";
}
if (observed.bytesRead >= 2 && magic[0] === 0x4d && magic[1] === 0x5a)
return "pe";
} finally {
await handle.close();
}
return classifyArtifactPath(path).format;
};
const classifyContainerExtension = (
path: string,
): ArtifactNode["format"] | undefined => {
const lower = path.toLowerCase();
for (const format of ["asar", "ipa", "apk", "zip", "dmg", "pkg"] as const)
if (lower.endsWith(`.${format}`)) return format;
return undefined;
};
const preflightEntry = (entry: ArtifactEntry, limits: ArtifactLimits): void => {
if (entry.declaredSize !== null && entry.declaredSize > limits.maxEntryBytes)
throw new ArtifactReaderFailure(
"limit",
`Entry exceeds byte limit: ${entry.path}`,
);
if (
entry.declaredSize !== null &&
entry.compressedSize !== null &&
entry.compressedSize === 0 &&
entry.declaredSize > 0
)
throw new ArtifactReaderFailure(
"limit",
`Invalid compression ratio: ${entry.path}`,
);
if (
entry.declaredSize !== null &&
entry.compressedSize !== null &&
entry.compressedSize > 0 &&
entry.declaredSize / entry.compressedSize > limits.maxCompressionRatio
)
throw new ArtifactReaderFailure(
"limit",
`Compression ratio exceeds limit: ${entry.path}`,
);
};
const hashReadable = async (
stream: Readable,
maximum: number,
signal?: AbortSignal,
): Promise<{
readonly sha256: string;
readonly bytes: number;
readonly prefix: Buffer;
}> => {
const hash = createHash("sha256");
const prefixes: Buffer[] = [];
let prefixBytes = 0;
let bytes = 0;
for await (const raw of stream) {
abortIfNeeded(signal);
const chunk = streamChunkToBuffer(raw);
bytes += chunk.length;
if (bytes > maximum) {
stream.destroy();
throw new ArtifactReaderFailure(
"limit",
"Observed entry bytes exceed limit",
);
}
hash.update(chunk);
if (prefixBytes < 16) {
const selected = chunk.subarray(0, 16 - prefixBytes);
prefixes.push(selected);
prefixBytes += selected.length;
}
}
return { sha256: hash.digest("hex"), bytes, prefix: Buffer.concat(prefixes) };
};
const abortIfNeeded = (signal?: AbortSignal): void => {
if (signal?.aborted === true)
throw new ArtifactReaderFailure(
"cancelled",
"Artifact inventory cancelled",
);
};

298
src/application/BinarySession.ts
View File

@ -1,22 +1,47 @@
import type { BinaryTarget } from "../domain/binaryTarget.js";
import { parseBinaryTarget } from "../domain/binaryTarget.js";
import {
HopperCancelledError,
AnalysisCapabilityUnavailableError,
EvidenceIntegrityError,
EvidenceLimitError,
AnalysisCancelledError,
NoBinaryOpenError,
type AnalysisError,
} from "../domain/errors.js";
import { err, ok, type Result } from "../domain/result.js";
import type { JsonValue } from "../domain/jsonValue.js";
import type { Evidence } from "../domain/evidence.js";
import { createEvidence } from "../domain/evidence.js";
import type { EvidenceBundle } from "../domain/evidenceBundle.js";
import type {
AnalysisClient,
AnalysisExecution,
AnalysisClientFactory,
AnalysisOperationPort,
AnalysisProvider,
CapabilityDescriptor,
ProviderIdentity,
} from "./AnalysisProvider.js";
import { EvidenceLedger } from "./EvidenceLedger.js";
import type {
RecordUnknownInput,
ResidualUnknown,
UnknownStatus,
UpdateUnknownInput,
} from "../domain/residualUnknown.js";
import { UnknownRegistryError } from "../domain/errors.js";
import type { AnalysisOperation } from "./AnalysisProvider.js";
import { enhancedToolNameSchema } from "../contracts/enhancedInputs.js";
import { OFFICIAL_TOOL_CONTRACTS } from "../contracts/toolContracts.js";
const REGISTRY_PROVIDER: ProviderIdentity = {
id: "rea-unknown-registry",
name: "REA residual unknown registry",
version: "1",
};
const OFFICIAL_OPERATIONS: ReadonlySet<string> = new Set(
OFFICIAL_TOOL_CONTRACTS.map(({ name }) => name),
);
/** Target lifecycle used by CLI and MCP without exposing a concrete provider. */
export interface BinarySessionPort extends AnalysisOperationPort {
@ -30,10 +55,39 @@ export interface BinarySessionPort extends AnalysisOperationPort {
close(): Promise<Result<null, AnalysisError>>;
status(): JsonValue;
activeTarget(): BinaryTarget | undefined;
recordEvidence(evidence: Evidence): void;
recordEvidence(
evidence: Evidence,
): Result<"added" | "duplicate", EvidenceIntegrityError | EvidenceLimitError>;
hasEvidence(evidenceId: string): boolean;
evidenceById(evidenceId: string): Evidence | undefined;
exportEvidenceBundle(): EvidenceBundle;
importEvidenceBundle(bundle: unknown): number;
providerIdentity(): ProviderIdentity;
importEvidenceBundle(
bundle: unknown,
): Result<number, EvidenceIntegrityError | EvidenceLimitError>;
recordUnknown(
input: RecordUnknownInput,
): Result<ResidualUnknown, AnalysisError>;
recordEvidenceWithUnknown(
evidence: Evidence,
input: RecordUnknownInput,
): Result<ResidualUnknown | null, AnalysisError>;
updateUnknown(
input: UpdateUnknownInput,
): Result<ResidualUnknown, AnalysisError>;
listUnknowns(filters?: {
readonly status?: UnknownStatus;
readonly severity?: ResidualUnknown["severity"];
readonly domain?: string;
}): ResidualUnknown[];
verifyUnknownResolution(unknownId: string): Result<
{
readonly valid: boolean;
readonly truthVerified: boolean;
readonly unknown: ResidualUnknown;
},
UnknownRegistryError
>;
providerIdentity(operation?: AnalysisOperation): ProviderIdentity;
}
/**
@ -52,7 +106,11 @@ export class BinarySession implements BinarySessionPort {
readonly #calls = new Set<Promise<unknown>>();
readonly #createClient: AnalysisClientFactory;
readonly #providerIdentity: ProviderIdentity;
readonly #evidence = new EvidenceLedger({ maxRecords: 10_000 });
readonly #capabilities: ReadonlyMap<string, CapabilityDescriptor> | undefined;
readonly #evidence = new EvidenceLedger({
maxRecords: 10_000,
maxBytes: 64 * 1024 * 1024,
});
constructor(readonly provider: AnalysisProvider | AnalysisClientFactory) {
this.#createClient =
@ -63,16 +121,56 @@ export class BinarySession implements BinarySessionPort {
typeof provider === "function"
? { id: "unidentified", name: "Unidentified provider", version: null }
: provider.identity();
this.#capabilities =
typeof provider === "function"
? undefined
: new Map(
provider
.capabilities()
.map((descriptor) => [descriptor.operation, descriptor]),
);
}
/** Identify the provider producing evidence for this session. */
providerIdentity(): ProviderIdentity {
providerIdentity(operation?: AnalysisOperation): ProviderIdentity {
if (operation !== undefined) {
const exact = this.#capabilities?.get(operation)?.provider;
if (exact !== undefined) return exact;
if (enhancedToolNameSchema.safeParse(operation).success) {
const providers = new Map<string, ProviderIdentity>();
for (const descriptor of this.#capabilities?.values() ?? [])
if (
descriptor.available &&
OFFICIAL_OPERATIONS.has(descriptor.operation)
)
providers.set(descriptor.provider.id, descriptor.provider);
if (providers.size === 1) {
const provider = providers.values().next().value;
if (provider !== undefined) return provider;
}
}
}
return this.#providerIdentity;
}
/** Add one successful public observation to the session ledger. */
recordEvidence(evidence: Evidence): void {
this.#evidence.record(evidence);
recordEvidence(
evidence: Evidence,
): Result<
"added" | "duplicate",
EvidenceIntegrityError | EvidenceLimitError
> {
return this.#evidence.record(evidence);
}
/** Check that comparison input Evidence is already session-owned. */
hasEvidence(evidenceId: string): boolean {
return this.#evidence.has(evidenceId);
}
/** Read one detached session-owned Evidence record by semantic ID. */
evidenceById(evidenceId: string): Evidence | undefined {
return this.#evidence.get(evidenceId);
}
/** Return a deterministic snapshot without clearing session evidence. */
@ -81,10 +179,77 @@ export class BinarySession implements BinarySessionPort {
}
/** Atomically merge a validated evidence bundle into this session. */
importEvidenceBundle(bundle: unknown): number {
importEvidenceBundle(
bundle: unknown,
): Result<number, EvidenceIntegrityError | EvidenceLimitError> {
return this.#evidence.import(bundle);
}
/** Create an approved residual unknown and immutable mutation evidence. */
recordUnknown(
input: RecordUnknownInput,
): Result<ResidualUnknown, AnalysisError> {
const evidence = unknownMutationEvidence(this.#active?.target, input);
return this.#evidence.recordUnknown(input, evidence);
}
/** Atomically record derived Evidence and its approved residual unknown. */
recordEvidenceWithUnknown(
evidence: Evidence,
input: RecordUnknownInput,
): Result<ResidualUnknown | null, AnalysisError> {
return this.#evidence.recordWithUnknown(
evidence,
input,
unknownMutationEvidence(undefined, input),
);
}
/** Update one unknown using compare-and-swap revision semantics. */
updateUnknown(
input: UpdateUnknownInput,
): Result<ResidualUnknown, AnalysisError> {
const evidence = createEvidence(this.#active?.target, REGISTRY_PROVIDER, {
predicateType: "rea.residual-unknown-mutation/v1",
operation: "update_unknown",
parameters: {
unknown_id: input.unknown_id,
expected_revision: input.expected_revision,
},
result: { action: "update", status: input.status },
confidence: "derived",
authority: "analyst-inference",
evidenceLinks: unknownEvidenceLinks(input),
limitations: [
"Registry mutation evidence records analyst intent, not proof of the answer.",
],
});
return this.#evidence.updateUnknown(input, evidence);
}
/** Query stable current residual-unknown heads. */
listUnknowns(
filters: {
readonly status?: UnknownStatus;
readonly severity?: ResidualUnknown["severity"];
readonly domain?: string;
} = {},
): ResidualUnknown[] {
return this.#evidence.listUnknowns(filters);
}
/** Check whether current head is a bundle-valid resolved state. */
verifyUnknownResolution(unknownId: string): Result<
{
readonly valid: boolean;
readonly truthVerified: boolean;
readonly unknown: ResidualUnknown;
},
UnknownRegistryError
> {
return this.#evidence.verifyUnknownResolution(unknownId);
}
/**
* Open or switch targets after draining calls against the current target.
* Returns the switch failure even if best-effort restoration also fails.
@ -97,7 +262,8 @@ export class BinarySession implements BinarySessionPort {
} = {},
): Promise<Result<BinaryTarget, AnalysisError>> {
return this.#serialize(async () => {
if (isAborted(options.signal)) return err(new HopperCancelledError());
if (isAborted(options.signal))
return err(new AnalysisCancelledError("open_binary"));
const parsed = await parseBinaryTarget(
path,
process.cwd(),
@ -105,7 +271,8 @@ export class BinarySession implements BinarySessionPort {
options.targetKind,
);
if (!parsed.ok) return parsed;
if (isAborted(options.signal)) return err(new HopperCancelledError());
if (isAborted(options.signal))
return err(new AnalysisCancelledError("open_binary"));
if (this.#active?.target.path === parsed.value.path)
return ok(parsed.value);
await this.#drainCalls();
@ -122,7 +289,7 @@ export class BinarySession implements BinarySessionPort {
if (isAborted(options.signal)) {
await client.close();
await this.#restore(previous);
return err(new HopperCancelledError());
return err(new AnalysisCancelledError("open_binary"));
}
this.#active = { target: parsed.value, client };
return ok(parsed.value);
@ -144,10 +311,48 @@ export class BinarySession implements BinarySessionPort {
/** Describe the active binary session. */
status(): JsonValue {
const target = this.#active?.target;
const provider = {
id: this.#providerIdentity.id,
name: this.#providerIdentity.name,
version: this.#providerIdentity.version,
};
const capabilities =
this.#capabilities === undefined
? []
: [...this.#capabilities.values()]
.sort((left, right) =>
left.operation.localeCompare(right.operation),
)
.map((descriptor) => ({
operation: descriptor.operation,
available: descriptor.available,
reason: descriptor.reason,
input_contract_version: descriptor.inputContractVersion,
output_contract_version: descriptor.outputContractVersion,
pagination: descriptor.pagination,
exhaustive: descriptor.exhaustive,
effects: {
mutates_artifact: descriptor.effects.mutatesArtifact,
launches_process: descriptor.effects.launchesProcess,
may_show_ui: descriptor.effects.mayShowUi,
may_access_network: descriptor.effects.mayAccessNetwork,
may_write_filesystem: descriptor.effects.mayWriteFilesystem,
changes_permissions: descriptor.effects.changesPermissions,
requires_root: descriptor.effects.requiresRoot,
},
limits: {
max_results: descriptor.limits.maxResults,
max_payload_bytes: descriptor.limits.maxPayloadBytes,
timeout_ms: descriptor.limits.timeoutMs,
},
limitations: [...descriptor.limitations],
}));
return target === undefined
? { open: false }
? { open: false, provider, capabilities }
: {
open: true,
provider,
capabilities,
path: target.path,
sha256: target.sha256,
format: target.format,
@ -167,12 +372,21 @@ export class BinarySession implements BinarySessionPort {
* entering until the transition has settled.
*/
async execute(
name: string,
name: Parameters<AnalysisOperationPort["execute"]>[0],
arguments_: Readonly<Record<string, JsonValue>>,
options?: { readonly signal?: AbortSignal },
): Promise<Result<JsonValue, AnalysisError>> {
const transitioned = await this.#waitForTransition(options?.signal);
): Promise<Result<AnalysisExecution, AnalysisError>> {
const transitioned = await this.#waitForTransition(name, options?.signal);
if (!transitioned.ok) return transitioned;
const capability = this.#capabilities?.get(name);
if (this.#capabilities !== undefined && capability?.available !== true)
return err(
new AnalysisCapabilityUnavailableError(
this.#providerIdentity.id,
name,
capability?.reason ?? "operation is not declared by this provider",
),
);
const active = this.#active;
if (active === undefined) return err(new NoBinaryOpenError());
const call = active.client.execute(name, arguments_, options);
@ -210,29 +424,35 @@ export class BinarySession implements BinarySessionPort {
}
async #waitForTransition(
operation: string,
signal?: AbortSignal,
): Promise<Result<undefined, HopperCancelledError>> {
if (signal?.aborted === true) return err(new HopperCancelledError());
): Promise<Result<undefined, AnalysisCancelledError>> {
if (signal?.aborted === true)
return err(new AnalysisCancelledError(operation));
if (signal === undefined) {
await this.#transition;
return ok(undefined);
}
return new Promise((resolve) => {
const onAbort = (): void => {
resolve(err(new HopperCancelledError()));
resolve(err(new AnalysisCancelledError(operation)));
};
signal.addEventListener("abort", onAbort, { once: true });
this.#transition.then(
() => {
signal.removeEventListener("abort", onAbort);
resolve(
signal.aborted ? err(new HopperCancelledError()) : ok(undefined),
signal.aborted
? err(new AnalysisCancelledError(operation))
: ok(undefined),
);
},
() => {
signal.removeEventListener("abort", onAbort);
resolve(
signal.aborted ? err(new HopperCancelledError()) : ok(undefined),
signal.aborted
? err(new AnalysisCancelledError(operation))
: ok(undefined),
);
},
);
@ -240,5 +460,41 @@ export class BinarySession implements BinarySessionPort {
}
}
const unknownMutationEvidence = (
target: BinaryTarget | undefined,
input: RecordUnknownInput,
): Evidence =>
createEvidence(target, REGISTRY_PROVIDER, {
predicateType: "rea.residual-unknown-mutation/v1",
operation: "record_unknown",
parameters: {
domain: input.domain,
severity: input.severity,
},
result: {
action: "record",
question: input.question,
required_authority: input.required_authority,
required_confidence: input.required_confidence,
},
confidence: "derived",
authority: "analyst-inference",
evidenceLinks: unknownEvidenceLinks(input),
limitations: [
"Registry mutation evidence records analyst intent, not proof of the answer.",
],
});
const unknownEvidenceLinks = (
input: RecordUnknownInput | UpdateUnknownInput,
): string[] =>
[
...input.supporting_evidence_ids,
...input.contradicting_evidence_ids,
...("resolution" in input && input.resolution !== null
? input.resolution.evidence_ids
: []),
].filter((id, index, values) => values.indexOf(id) === index);
const isAborted = (signal: AbortSignal | undefined): boolean =>
signal?.aborted === true;

96
src/application/CompositeProvider.ts Normal file
View File

@ -0,0 +1,96 @@
import { ProviderSelectionError } from "../domain/errors.js";
import { err, ok } from "../domain/result.js";
import type { BinaryTarget } from "../domain/binaryTarget.js";
import type {
AnalysisClient,
AnalysisProvider,
CapabilityDescriptor,
ProviderIdentity,
} from "./AnalysisProvider.js";
const compositeIdentity = (
providers: readonly AnalysisProvider[],
): ProviderIdentity => ({
id: `composite:${providers
.map((provider) => provider.identity().id)
.sort()
.join("+")}`,
name: "REA composite analysis provider",
version: null,
});
/** Deterministically route disjoint operations without eagerly starting children. */
export class CompositeProvider implements AnalysisProvider {
readonly #identity: ProviderIdentity;
readonly #capabilities: readonly CapabilityDescriptor[];
readonly #providerByOperation: ReadonlyMap<string, AnalysisProvider>;
constructor(readonly providers: readonly AnalysisProvider[]) {
if (providers.length === 0)
throw new RangeError("CompositeProvider requires at least one provider");
this.#identity = Object.freeze(compositeIdentity(providers));
const routes = new Map<string, AnalysisProvider>();
const capabilities: CapabilityDescriptor[] = [];
for (const provider of providers) {
for (const descriptor of provider.capabilities()) {
if (routes.has(descriptor.operation))
throw new TypeError(
`Multiple providers declare operation ${descriptor.operation}`,
);
routes.set(descriptor.operation, provider);
capabilities.push(descriptor);
}
}
this.#providerByOperation = routes;
this.#capabilities = Object.freeze(
capabilities.sort(
(left, right) =>
left.operation.localeCompare(right.operation) ||
left.provider.id.localeCompare(right.provider.id),
),
);
}
identity(): ProviderIdentity {
return this.#identity;
}
capabilities(): readonly CapabilityDescriptor[] {
return this.#capabilities;
}
createClient(target: BinaryTarget): AnalysisClient {
const clients = new Map<AnalysisProvider, AnalysisClient>();
const clientFor = (provider: AnalysisProvider): AnalysisClient => {
const existing = clients.get(provider);
if (existing !== undefined) return existing;
const created = provider.createClient(target);
clients.set(provider, created);
return created;
};
return {
execute: (operation, parameters, options) => {
if (operation === "health")
return Promise.resolve(
ok({
result: null,
rawResult: null,
provider: this.#identity,
limitations: [],
locations: [],
subject: null,
}),
);
const provider = this.#providerByOperation.get(operation);
return provider === undefined
? Promise.resolve(err(new ProviderSelectionError(operation)))
: clientFor(provider).execute(operation, parameters, options);
},
close: async () => {
await Promise.allSettled(
[...clients.values()].map(async (client) => client.close()),
);
},
};
}
}

64
src/application/DirectAnalysis.ts
View File

@ -4,6 +4,14 @@ import { EnhancedTools } from "./EnhancedTools.js";
import { createBinarySession } from "./runtime.js";
import { silentLogger, type Logger } from "../logger.js";
import { createEvidence } from "../domain/evidence.js";
import type { NativeToolName } from "../contracts/nativeToolContracts.js";
import type { ArtifactToolName } from "../contracts/artifactToolContracts.js";
const WORKFLOW_PROVIDER = {
id: "rea-workflow",
name: "REA composed investigation workflow",
version: "1",
} as const;
/**
* Open one binary, execute one tool, and always release the bridge session.
@ -15,6 +23,25 @@ export const runDirectAnalysis = async (
tool: "binary_overview" | "procedure_pseudo_code",
arguments_: Readonly<Record<string, JsonValue>>,
logger: Logger = silentLogger,
): Promise<JsonValue> => runAnalysis(path, tool, arguments_, logger);
/** Execute one provider-native semantic operation with atomic provenance. */
export const runProviderAnalysis = async (
path: string,
tool: NativeToolName | ArtifactToolName,
arguments_: Readonly<Record<string, JsonValue>>,
logger: Logger = silentLogger,
): Promise<JsonValue> => runAnalysis(path, tool, arguments_, logger);
const runAnalysis = async (
path: string,
tool:
| NativeToolName
| ArtifactToolName
| "binary_overview"
| "procedure_pseudo_code",
arguments_: Readonly<Record<string, JsonValue>>,
logger: Logger,
): Promise<JsonValue> => {
const config = parseConfig(process.env);
if (!config.ok)
@ -24,19 +51,32 @@ export const runDirectAnalysis = async (
const opened = await session.open(path);
if (!opened.ok)
return { error: opened.error._tag, message: opened.error.message };
const result =
tool === "binary_overview"
? await new EnhancedTools(session).execute(
"binary_overview",
arguments_,
)
: await session.execute(tool, arguments_);
if (tool === "binary_overview") {
const result = await new EnhancedTools(session).execute(tool, arguments_);
return result.ok
? createEvidence(opened.value, WORKFLOW_PROVIDER, {
operation: tool,
parameters: arguments_,
result: result.value,
confidence: "derived",
limitations: ["Derived by an REA composed workflow."],
})
: { error: result.error._tag, message: result.error.message };
}
const result = await session.execute(tool, arguments_);
return result.ok
? createEvidence(opened.value, session.providerIdentity(), {
operation: tool,
parameters: arguments_,
result: result.value,
})
? createEvidence(
result.value.subject ?? opened.value,
result.value.provider,
{
operation: tool,
parameters: arguments_,
result: result.value.result,
rawResult: result.value.rawResult,
limitations: result.value.limitations,
locations: result.value.locations,
},
)
: { error: result.error._tag, message: result.error.message };
} finally {
await session.close();

35
src/application/EnhancedTools.ts
View File

@ -1,7 +1,14 @@
import type { AnalysisOperationPort } from "./AnalysisProvider.js";
import type {
AnalysisOperation,
AnalysisOperationPort,
} from "./AnalysisProvider.js";
import type { EnhancedToolName } from "../contracts/enhancedInputs.js";
import { enhancedInputSchemas } from "../contracts/enhancedInputs.js";
import { AnalysisProtocolError, type AnalysisError } from "../domain/errors.js";
import {
AnalysisInputError,
AnalysisOutputError,
type AnalysisError,
} from "../domain/errors.js";
import {
parseDocuments,
parseFunctionDossier,
@ -384,12 +391,18 @@ export class EnhancedTools {
if (!xrefs.ok) return xrefs;
if (!Array.isArray(xrefs.value))
return err(
new AnalysisProtocolError("xrefs returned a non-array result"),
new AnalysisOutputError(
"xrefs",
"provider returned a non-array result",
),
);
for (const source of xrefs.value) {
if (typeof source !== "string")
return err(
new AnalysisProtocolError("xrefs returned a non-address value"),
new AnalysisOutputError(
"xrefs",
"provider returned a non-address value",
),
);
if (operations >= operationBudget) {
residual.add(
@ -444,8 +457,9 @@ export class EnhancedTools {
}
if (page.value.nextOffset <= offset) {
return err(
new AnalysisProtocolError(
`${tool} returned a non-advancing pagination offset`,
new AnalysisOutputError(
tool,
"provider returned a non-advancing pagination offset",
),
);
}
@ -453,23 +467,24 @@ export class EnhancedTools {
}
}
#call(
name: string,
async #call(
name: AnalysisOperation,
arguments_: Readonly<Record<string, JsonValue>>,
signal?: AbortSignal,
): Promise<Result<JsonValue, AnalysisError>> {
return this.analysis.execute(
const execution = await this.analysis.execute(
name,
arguments_,
signal === undefined ? {} : { signal },
);
return execution.ok ? ok(execution.value.result) : execution;
}
}
const invalidInput = (name: EnhancedToolName, cause: Error): EnhancedResult =>
Promise.resolve(
err(
new AnalysisProtocolError(`Invalid ${name} input after MCP validation`, {
new AnalysisInputError(name, {
cause,
}),
),

56
src/application/EvidenceBundleCommands.ts Normal file
View File

@ -0,0 +1,56 @@
import type { AnalysisError } from "../domain/errors.js";
import type {
EvidenceFilePolicy,
EvidenceBundle,
} from "../domain/evidenceBundle.js";
import type { JsonValue } from "../domain/jsonValue.js";
import { err, ok, type Result } from "../domain/result.js";
import { EvidenceLedger } from "./EvidenceLedger.js";
import {
readEvidenceBundle,
writeEvidenceBundle,
} from "./EvidenceBundleFiles.js";
/** Validate and merge one bundle using the same bounded ledger as MCP. */
export const importEvidenceBundleCommand = async (
path: string,
policy: EvidenceFilePolicy,
): Promise<Result<JsonValue, AnalysisError>> => {
const loaded = await readEvidenceBundle(path, policy);
if (!loaded.ok) return loaded;
const ledger = createLedger();
const imported = ledger.import(loaded.value);
return imported.ok
? ok({ imported: imported.value, total: ledger.export().records.length })
: imported;
};
/** Validate a source bundle and atomically export canonical bytes. */
export const exportEvidenceBundleCommand = async (
sourcePath: string,
outputPath: string,
overwrite: boolean,
policy: EvidenceFilePolicy,
): Promise<Result<JsonValue, AnalysisError>> => {
const loaded = await readEvidenceBundle(sourcePath, policy);
if (!loaded.ok) return loaded;
return projectWrite(
loaded.value,
await writeEvidenceBundle(loaded.value, outputPath, overwrite, policy),
);
};
const createLedger = (): EvidenceLedger =>
new EvidenceLedger({ maxRecords: 10_000, maxBytes: 64 * 1024 * 1024 });
const projectWrite = (
bundle: EvidenceBundle,
written: Awaited<ReturnType<typeof writeEvidenceBundle>>,
): Result<JsonValue, AnalysisError> =>
written.ok
? ok({
path: written.value.path,
bytes: written.value.bytes,
records: bundle.records.length,
})
: err(written.error);

159
src/application/EvidenceBundleFiles.ts Normal file
View File

@ -0,0 +1,159 @@
import { lstat, readFile, realpath } from "node:fs/promises";
import { basename, dirname, isAbsolute, relative, resolve } from "node:path";
import writeFileAtomic from "write-file-atomic";
import {
parseEvidenceBundle,
serializeEvidenceBundle,
type EvidenceBundle,
type EvidenceFilePolicy,
} from "../domain/evidenceBundle.js";
import { EvidenceFileError, EvidenceIntegrityError } from "../domain/errors.js";
import { err, ok, type Result } from "../domain/result.js";
type EvidenceReadFailure = EvidenceFileError | EvidenceIntegrityError;
type EvidenceWriteFailure = EvidenceFileError | EvidenceIntegrityError;
/** Read and validate a bounded evidence bundle inside an approved root. */
export const readEvidenceBundle = async (
path: string,
policy: EvidenceFilePolicy,
): Promise<Result<EvidenceBundle, EvidenceReadFailure>> => {
if (policy.roots.length === 0)
return err(new EvidenceFileError("read", "disabled"));
try {
const canonicalPath = await realpath(resolve(path));
if (!(await isApproved(canonicalPath, policy.roots)))
return err(new EvidenceFileError("read", "outside-root"));
const stats = await lstat(canonicalPath);
if (!stats.isFile()) return err(new EvidenceFileError("read", "not-file"));
if (stats.size > policy.maxBytes)
return err(new EvidenceFileError("read", "too-large"));
const encoded = await readFile(canonicalPath);
if (encoded.byteLength > policy.maxBytes)
return err(new EvidenceFileError("read", "too-large"));
let decoded: unknown;
try {
decoded = JSON.parse(encoded.toString("utf8"));
} catch (cause: unknown) {
return err(new EvidenceFileError("read", "invalid-json", { cause }));
}
const limits = validateJsonLimits(decoded, policy);
if (!limits.ok) return limits;
try {
return ok(parseEvidenceBundle(decoded));
} catch (cause: unknown) {
return err(
new EvidenceIntegrityError("Evidence bundle validation failed", {
cause,
}),
);
}
} catch (cause: unknown) {
return err(new EvidenceFileError("read", "io", { cause }));
}
};
/** Atomically write deterministic evidence JSON inside an approved root. */
export const writeEvidenceBundle = async (
bundle: EvidenceBundle,
path: string,
overwrite: boolean,
policy: EvidenceFilePolicy,
): Promise<
Result<
{ readonly path: string; readonly bytes: number },
EvidenceWriteFailure
>
> => {
if (policy.roots.length === 0)
return err(new EvidenceFileError("write", "disabled"));
let encoded: string;
try {
encoded = serializeEvidenceBundle(bundle);
} catch (cause: unknown) {
return err(
new EvidenceIntegrityError("Evidence bundle validation failed", {
cause,
}),
);
}
const bytes = Buffer.byteLength(encoded, "utf8");
if (bytes > policy.maxBytes)
return err(new EvidenceFileError("write", "too-large"));
try {
const canonicalParent = await realpath(dirname(resolve(path)));
const destination = resolve(canonicalParent, basename(resolve(path)));
if (!(await isApproved(destination, policy.roots)))
return err(new EvidenceFileError("write", "outside-root"));
const existing = await lstat(destination).catch((cause: unknown) => {
if (isFileNotFound(cause)) return undefined;
throw cause;
});
if (existing !== undefined) {
if (!overwrite) return err(new EvidenceFileError("write", "exists"));
if (!existing.isFile() || existing.isSymbolicLink())
return err(new EvidenceFileError("write", "not-file"));
}
await writeFileAtomic(destination, encoded, {
encoding: "utf8",
mode: 0o600,
fsync: true,
});
return ok({ path: destination, bytes });
} catch (cause: unknown) {
return err(new EvidenceFileError("write", "io", { cause }));
}
};
const isApproved = async (
candidate: string,
roots: readonly string[],
): Promise<boolean> => {
for (const configuredRoot of roots) {
const root = await realpath(resolve(configuredRoot));
const relation = relative(root, candidate);
if (
relation === "" ||
(!relation.startsWith("..") && !isAbsolute(relation))
)
return true;
}
return false;
};
const validateJsonLimits = (
root: unknown,
policy: EvidenceFilePolicy,
): Result<null, EvidenceFileError> => {
const pending: Array<{ readonly value: unknown; readonly depth: number }> = [
{ value: root, depth: 0 },
];
let nodes = 0;
while (pending.length > 0) {
const current = pending.pop();
if (current === undefined) break;
nodes += 1;
if (nodes > policy.maxNodes || current.depth > policy.maxDepth)
return err(new EvidenceFileError("read", "too-large"));
if (
typeof current.value === "string" &&
current.value.length > policy.maxStringLength
)
return err(new EvidenceFileError("read", "too-large"));
if (typeof current.value !== "object" || current.value === null) continue;
for (const [key, value] of Object.entries(current.value)) {
if (key.length > policy.maxStringLength)
return err(new EvidenceFileError("read", "too-large"));
pending.push({ value, depth: current.depth + 1 });
}
}
return ok(null);
};
const isFileNotFound = (cause: unknown): boolean =>
typeof cause === "object" &&
cause !== null &&
"code" in cause &&
cause.code === "ENOENT";

362
src/application/EvidenceLedger.ts
View File

@ -1,68 +1,392 @@
import {
createEvidenceBundle,
evidenceBundleSchema,
parseEvidenceBundle,
type EvidenceBundle,
} from "../domain/evidenceBundle.js";
import {
EvidenceIntegrityError,
EvidenceLimitError,
UnknownRegistryError,
} from "../domain/errors.js";
import { parseEvidence, type Evidence } from "../domain/evidence.js";
import {
createResidualUnknown,
updateResidualUnknown,
type RecordUnknownInput,
type ResidualUnknown,
type UnknownStatus,
type UpdateUnknownInput,
} from "../domain/residualUnknown.js";
import { err, ok, type Result } from "../domain/result.js";
export interface EvidenceLedgerLimits {
readonly maxRecords: number;
readonly maxBytes: number;
}
type EvidenceLedgerFailure = EvidenceIntegrityError | EvidenceLimitError;
type RecordResult = Result<"added" | "duplicate", EvidenceLedgerFailure>;
type ImportResult = Result<number, EvidenceLedgerFailure>;
/** Bounded, session-owned set of immutable evidence records. */
export class EvidenceLedger {
readonly #records = new Map<string, Evidence>();
readonly #unknownRevisions = new Map<string, ResidualUnknown>();
readonly #unknownHeads = new Map<string, ResidualUnknown>();
#bytes = 0;
constructor(private readonly limits: EvidenceLedgerLimits) {
if (!Number.isSafeInteger(limits.maxRecords) || limits.maxRecords < 1)
throw new RangeError("maxRecords must be a positive safe integer");
if (!Number.isSafeInteger(limits.maxBytes) || limits.maxBytes < 1)
throw new RangeError("maxBytes must be a positive safe integer");
}
/** Record evidence idempotently; conflicting content is rejected. */
record(input: Evidence): "added" | "duplicate" {
const evidence = parseEvidence(input);
record(input: Evidence): RecordResult {
let evidence: Evidence;
try {
evidence = parseEvidence(input);
} catch (cause: unknown) {
return err(
new EvidenceIntegrityError("Evidence record validation failed", {
cause,
}),
);
}
const existing = this.#records.get(evidence.evidence_id);
if (existing !== undefined) {
if (!recordsAgree(existing, evidence))
throw new Error(`Conflicting evidence record: ${evidence.evidence_id}`);
return "duplicate";
return recordsAgree(existing, evidence)
? ok("duplicate")
: err(new EvidenceIntegrityError("Conflicting evidence record"));
}
if (this.#records.size >= this.limits.maxRecords)
throw new RangeError("Evidence ledger record limit exceeded");
return err(new EvidenceLimitError("records", this.limits.maxRecords));
const bytes = serializedBytes(evidence);
if (this.#bytes + bytes > this.limits.maxBytes)
return err(new EvidenceLimitError("bytes", this.limits.maxBytes));
this.#records.set(evidence.evidence_id, evidence);
return "added";
this.#bytes += bytes;
return ok("added");
}
/** Validate an entire bundle before atomically merging its records. */
import(input: unknown): number {
const bundle = evidenceBundleSchema.parse(input);
import(input: unknown): ImportResult {
let bundle: EvidenceBundle;
try {
bundle = parseEvidenceBundle(input);
} catch (cause: unknown) {
return err(
new EvidenceIntegrityError("Evidence bundle validation failed", {
cause,
}),
);
}
const pending = new Map(this.#records);
for (const unparsed of bundle.records) {
const evidence = parseEvidence(unparsed);
for (const evidence of bundle.records) {
const existing = pending.get(evidence.evidence_id);
if (existing !== undefined && !recordsAgree(existing, evidence))
throw new Error(`Conflicting evidence record: ${evidence.evidence_id}`);
return err(new EvidenceIntegrityError("Conflicting evidence record"));
pending.set(evidence.evidence_id, evidence);
}
if (pending.size > this.limits.maxRecords)
throw new RangeError("Evidence ledger record limit exceeded");
const pendingUnknowns = new Map(this.#unknownRevisions);
for (const unknown of bundle.unknowns) {
const key = unknownRevisionKey(unknown);
const existing = pendingUnknowns.get(key);
if (
existing !== undefined &&
existing.revision_digest !== unknown.revision_digest
)
return err(new EvidenceIntegrityError("Conflicting unknown revision"));
pendingUnknowns.set(key, unknown);
}
const checked = this.#validateCandidate(pending, pendingUnknowns);
if (!checked.ok) {
if (checked.error instanceof UnknownRegistryError)
return err(
new EvidenceIntegrityError(
"Evidence bundle unknown registry failed",
{
cause: checked.error,
},
),
);
return err(checked.error);
}
const added = pending.size - this.#records.size;
this.#records.clear();
for (const [id, evidence] of pending) this.#records.set(id, evidence);
return added;
this.#commit(pending, pendingUnknowns, checked.value);
return ok(added);
}
/** Export records in deterministic, semantically irrelevant ID order. */
export(): EvidenceBundle {
return createEvidenceBundle([...this.#records.values()]);
return createEvidenceBundle(
[...this.#records.values()],
[...this.#unknownRevisions.values()],
);
}
/** Check whether one immutable Evidence ID is present in this ledger. */
has(evidenceId: string): boolean {
return this.#records.has(evidenceId);
}
/** Return a detached immutable Evidence value by semantic ID. */
get(evidenceId: string): Evidence | undefined {
const evidence = this.#records.get(evidenceId);
return evidence === undefined ? undefined : structuredClone(evidence);
}
/** Create one approved unknown and atomically append its mutation evidence. */
recordUnknown(
input: RecordUnknownInput,
mutationEvidence: Evidence,
): Result<
ResidualUnknown,
EvidenceIntegrityError | EvidenceLimitError | UnknownRegistryError
> {
let parsedMutation: Evidence;
try {
parsedMutation = parseEvidence(mutationEvidence);
} catch (cause: unknown) {
return err(
new EvidenceIntegrityError("Unknown mutation evidence is invalid", {
cause,
}),
);
}
let unknown: ResidualUnknown;
try {
unknown = createResidualUnknown(
input,
parsedMutation.evidence_id,
parsedMutation.subject?.digest.sha256 ?? null,
);
} catch (cause: unknown) {
return err(new UnknownRegistryError("invalid-transition", { cause }));
}
if (this.#unknownHeads.has(unknown.unknown_id))
return err(new UnknownRegistryError("already-exists"));
return this.#appendUnknown(unknown, parsedMutation);
}
/** Atomically record derived Evidence and an approved residual unknown. */
recordWithUnknown(
evidenceInput: Evidence,
input: RecordUnknownInput,
mutationInput: Evidence,
): Result<
ResidualUnknown | null,
EvidenceIntegrityError | EvidenceLimitError | UnknownRegistryError
> {
let evidence: Evidence;
let mutation: Evidence;
try {
evidence = parseEvidence(evidenceInput);
mutation = parseEvidence(mutationInput);
} catch (cause: unknown) {
return err(
new EvidenceIntegrityError("Atomic Evidence validation failed", {
cause,
}),
);
}
const existingEvidence = this.#records.get(evidence.evidence_id);
if (
existingEvidence !== undefined &&
!recordsAgree(existingEvidence, evidence)
)
return err(new EvidenceIntegrityError("Conflicting evidence record"));
let unknown: ResidualUnknown;
try {
unknown = createResidualUnknown(
input,
mutation.evidence_id,
mutation.subject?.digest.sha256 ?? null,
);
} catch (cause: unknown) {
return err(new UnknownRegistryError("invalid-transition", { cause }));
}
const pendingRecords = new Map(this.#records);
pendingRecords.set(evidence.evidence_id, evidence);
const existingUnknown = this.#unknownHeads.get(unknown.unknown_id);
if (existingUnknown !== undefined) {
if (
existingUnknown.revision !== 1 ||
existingUnknown.revision_digest !== unknown.revision_digest
)
return err(new UnknownRegistryError("already-exists"));
const checked = this.#validateCandidate(
pendingRecords,
this.#unknownRevisions,
);
if (!checked.ok) return checked;
this.#commit(pendingRecords, this.#unknownRevisions, checked.value);
return ok(null);
}
pendingRecords.set(mutation.evidence_id, mutation);
const pendingUnknowns = new Map(this.#unknownRevisions);
pendingUnknowns.set(unknownRevisionKey(unknown), unknown);
const checked = this.#validateCandidate(pendingRecords, pendingUnknowns);
if (!checked.ok) return checked;
this.#commit(pendingRecords, pendingUnknowns, checked.value);
return ok(unknown);
}
/** Apply one approved full-state update using optimistic revision matching. */
updateUnknown(
input: UpdateUnknownInput,
mutationEvidence: Evidence,
): Result<
ResidualUnknown,
EvidenceIntegrityError | EvidenceLimitError | UnknownRegistryError
> {
const current = this.#unknownHeads.get(input.unknown_id);
if (current === undefined)
return err(new UnknownRegistryError("not-found"));
if (current.revision !== input.expected_revision)
return err(new UnknownRegistryError("revision-conflict"));
let unknown: ResidualUnknown;
try {
unknown = updateResidualUnknown(
current,
input,
mutationEvidence.evidence_id,
);
} catch (cause: unknown) {
return err(new UnknownRegistryError("invalid-transition", { cause }));
}
return this.#appendUnknown(unknown, mutationEvidence);
}
/** Query current heads in stable ID order. */
listUnknowns(
filters: {
readonly status?: UnknownStatus;
readonly severity?: ResidualUnknown["severity"];
readonly domain?: string;
} = {},
): ResidualUnknown[] {
return [...this.#unknownHeads.values()]
.filter(
(unknown) =>
(filters.status === undefined || unknown.status === filters.status) &&
(filters.severity === undefined ||
unknown.severity === filters.severity) &&
(filters.domain === undefined || unknown.domain === filters.domain),
)
.sort((left, right) => left.unknown_id.localeCompare(right.unknown_id));
}
/** Return current resolution validity; imported invalid states are rejected. */
verifyUnknownResolution(unknownId: string): Result<
{
readonly valid: boolean;
readonly truthVerified: boolean;
readonly unknown: ResidualUnknown;
},
UnknownRegistryError
> {
const unknown = this.#unknownHeads.get(unknownId);
if (unknown === undefined)
return err(new UnknownRegistryError("not-found"));
return ok({
valid: unknown.status === "resolved",
truthVerified: unknown.resolution?.disposition === "verified",
unknown,
});
}
/** Clear records when the owning session closes. */
clear(): void {
this.#records.clear();
this.#unknownRevisions.clear();
this.#unknownHeads.clear();
this.#bytes = 0;
}
#appendUnknown(
unknown: ResidualUnknown,
mutationEvidenceInput: Evidence,
): Result<
ResidualUnknown,
EvidenceIntegrityError | EvidenceLimitError | UnknownRegistryError
> {
let mutationEvidence: Evidence;
try {
mutationEvidence = parseEvidence(mutationEvidenceInput);
} catch (cause: unknown) {
return err(
new EvidenceIntegrityError("Unknown mutation evidence is invalid", {
cause,
}),
);
}
const pendingRecords = new Map(this.#records);
pendingRecords.set(mutationEvidence.evidence_id, mutationEvidence);
const pendingUnknowns = new Map(this.#unknownRevisions);
pendingUnknowns.set(unknownRevisionKey(unknown), unknown);
const checked = this.#validateCandidate(pendingRecords, pendingUnknowns);
if (!checked.ok) return checked;
this.#commit(pendingRecords, pendingUnknowns, checked.value);
return ok(unknown);
}
#validateCandidate(
records: ReadonlyMap<string, Evidence>,
unknowns: ReadonlyMap<string, ResidualUnknown>,
): Result<
number,
EvidenceIntegrityError | EvidenceLimitError | UnknownRegistryError
> {
if (records.size + unknowns.size > this.limits.maxRecords)
return err(new EvidenceLimitError("records", this.limits.maxRecords));
const bytes =
[...records.values()].reduce(
(total, evidence) => total + serializedBytes(evidence),
0,
) +
[...unknowns.values()].reduce(
(total, unknown) => total + serializedBytes(unknown),
0,
);
if (bytes > this.limits.maxBytes)
return err(new EvidenceLimitError("bytes", this.limits.maxBytes));
try {
parseEvidenceBundle(
createEvidenceBundle([...records.values()], [...unknowns.values()]),
);
} catch (cause: unknown) {
return err(new UnknownRegistryError("integrity", { cause }));
}
return ok(bytes);
}
#commit(
records: ReadonlyMap<string, Evidence>,
unknowns: ReadonlyMap<string, ResidualUnknown>,
bytes: number,
): void {
this.#records.clear();
for (const [id, evidence] of records) this.#records.set(id, evidence);
this.#unknownRevisions.clear();
this.#unknownHeads.clear();
for (const [key, unknown] of unknowns) {
this.#unknownRevisions.set(key, unknown);
const head = this.#unknownHeads.get(unknown.unknown_id);
if (head === undefined || head.revision < unknown.revision)
this.#unknownHeads.set(unknown.unknown_id, unknown);
}
this.#bytes = bytes;
}
}
const serializedBytes = (value: Evidence | ResidualUnknown): number =>
Buffer.byteLength(JSON.stringify(value), "utf8");
const unknownRevisionKey = (unknown: ResidualUnknown): string =>
`${unknown.unknown_id}:${String(unknown.revision)}`;
const recordsAgree = (left: Evidence, right: Evidence): boolean =>
JSON.stringify(withoutLocalPath(left)) ===
JSON.stringify(withoutLocalPath(right));

157
src/application/FilesystemSnapshot.ts Normal file
View File

@ -0,0 +1,157 @@
import { createHash } from "node:crypto";
import { constants as fsConstants } from "node:fs";
import { lstat, open, readdir, readlink } from "node:fs/promises";
import { dirname, join, relative, resolve } from "node:path";
import type { FileState, ProcessScenario } from "../domain/processCapture.js";
import type { Stats } from "node:fs";
export interface SnapshotResult {
readonly files: readonly FileState[];
readonly truncated: boolean;
}
const isWithin = (candidate: string, root: string): boolean =>
candidate === root ||
candidate.startsWith(`${root.endsWith("/") ? root.slice(0, -1) : root}/`);
const hasSameIdentity = (
before: Stats,
after: Stats,
expected: "directory" | "symlink",
): boolean =>
before.dev === after.dev &&
before.ino === after.ino &&
(expected === "directory" ? after.isDirectory() : after.isSymbolicLink());
const lstatIfPresent = async (path: string): Promise<Stats | undefined> => {
try {
return await lstat(path);
} catch (cause: unknown) {
if (cause instanceof Error && "code" in cause && cause.code === "ENOENT")
return undefined;
throw cause;
}
};
const hashFile = async (
path: string,
maxBytes: number,
signal?: AbortSignal,
): Promise<string | null> => {
const handle = await open(
path,
fsConstants.O_RDONLY | (fsConstants.O_NOFOLLOW ?? 0),
);
try {
const stats = await handle.stat();
if (!stats.isFile() || stats.size > maxBytes) return null;
const hash = createHash("sha256");
const buffer = Buffer.allocUnsafe(Math.min(64 * 1024, maxBytes));
let position = 0;
while (position < stats.size) {
signal?.throwIfAborted();
const { bytesRead } = await handle.read(
buffer,
0,
Math.min(buffer.length, stats.size - position),
position,
);
if (bytesRead === 0) return null;
hash.update(buffer.subarray(0, bytesRead));
position += bytesRead;
}
const after = await handle.stat();
if (after.size !== stats.size || after.mtimeMs !== stats.mtimeMs)
return null;
return hash.digest("hex");
} finally {
await handle.close();
}
};
/** Capture bounded, root-aliased filesystem state without following symlinks. */
export const snapshotRoots = async (
scenario: ProcessScenario,
signal?: AbortSignal,
): Promise<SnapshotResult> => {
const entries: FileState[] = [];
let remainingBytes = scenario.limits.file_bytes;
let truncated = false;
const visit = async (
root: string,
rootAlias: string,
path: string,
depth: number,
): Promise<void> => {
signal?.throwIfAborted();
if (
entries.length >= scenario.limits.files ||
depth > scenario.limits.filesystem_depth
) {
truncated = true;
return;
}
const stats = await lstatIfPresent(path);
if (stats === undefined) return;
const relativePath = relative(root, path) || ".";
if (stats.isSymbolicLink()) {
const target = resolve(dirname(path), await readlink(path));
const afterRead = await lstat(path);
if (!hasSameIdentity(stats, afterRead, "symlink")) {
truncated = true;
return;
}
const safeTarget = isWithin(target, root)
? relative(root, target) || "."
: "<outside-declared-root>";
if (safeTarget === "<outside-declared-root>") truncated = true;
entries.push({
path: `${rootAlias}:${relativePath}`,
type: "symlink",
mode: stats.mode,
size: stats.size,
sha256: null,
symlink_target: safeTarget,
});
return;
}
if (stats.isFile()) {
const sha256 =
remainingBytes >= stats.size
? await hashFile(path, remainingBytes, signal)
: null;
remainingBytes -= sha256 === null ? 0 : stats.size;
if (sha256 === null) truncated = true;
entries.push({
path: `${rootAlias}:${relativePath}`,
type: "file",
mode: stats.mode,
size: stats.size,
sha256,
symlink_target: null,
});
return;
}
const type = stats.isDirectory() ? "directory" : "other";
entries.push({
path: `${rootAlias}:${relativePath}`,
type,
mode: stats.mode,
size: stats.size,
sha256: null,
symlink_target: null,
});
if (type !== "directory") return;
const children = await readdir(path);
const afterRead = await lstat(path);
if (!hasSameIdentity(stats, afterRead, "directory")) {
truncated = true;
return;
}
for (const child of children.sort())
await visit(root, rootAlias, join(path, child), depth + 1);
};
for (const [index, root] of scenario.filesystem_roots.entries())
await visit(root, `root_${String(index)}`, root, 0);
return { files: entries, truncated };
};

276
src/application/LoopbackReplay.ts
View File

@ -1,4 +1,9 @@
import { createServer, type Server } from "node:http";
import {
createServer,
type IncomingMessage,
type Server,
type ServerResponse,
} from "node:http";
import { WebSocketServer, type WebSocket } from "ws";
import type {
ProcessScenario,
@ -10,46 +15,203 @@ export interface LoopbackReplay {
readonly httpUrl: string;
readonly websocketUrl: string;
readonly events: ProtocolEvent[];
readonly truncated: boolean;
close(): Promise<void>;
}
const closeServer = async (server: Server): Promise<void> =>
new Promise((resolveClose, rejectClose) => {
server.close((error) =>
error === undefined ? resolveClose() : rejectClose(error),
);
});
class ReplayRecorder {
readonly events: ProtocolEvent[] = [];
readonly started = Date.now();
truncated = false;
/** Start bounded HTTP and WebSocket replay endpoints bound only to IPv4 loopback. */
export const startLoopbackReplay = async (
scenario: ProcessScenario,
): Promise<LoopbackReplay> => {
const events: ProtocolEvent[] = [];
const record = (event: Omit<ProtocolEvent, "sequence">): void => {
if (events.length < 10_000)
events.push({ sequence: events.length, ...event });
};
const server = createServer((request, response) => {
constructor(readonly scenario: ProcessScenario) {}
atMs(): number {
return (
Math.floor(
(Date.now() - this.started) /
this.scenario.normalization.time_bucket_ms,
) * this.scenario.normalization.time_bucket_ms
);
}
record(event: Omit<ProtocolEvent, "sequence">): void {
if (this.events.length < this.scenario.limits.protocol_events)
this.events.push({ sequence: this.events.length, ...event });
else this.truncated = true;
}
}
const readRequestBody = async (
request: IncomingMessage,
limit: number,
): Promise<string | undefined> => {
const chunks: Buffer[] = [];
let bytes = 0;
for await (const chunk of request) {
const value = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk);
bytes += value.byteLength;
if (bytes > limit) return undefined;
chunks.push(value);
}
return Buffer.concat(chunks).toString("utf8");
};
const createHttpHandler = (
recorder: ReplayRecorder,
): ((request: IncomingMessage, response: ServerResponse) => Promise<void>) => {
const calls = new Map<number, number>();
return async (request, response) => {
const method = request.method ?? "GET";
const path = new URL(request.url ?? "/", "http://127.0.0.1").pathname;
record({ protocol: "http", direction: "request", method, path, data: "" });
const route = scenario.replay.http.find(
(candidate) => candidate.method === method && candidate.path === path,
const body = await readRequestBody(
request,
recorder.scenario.limits.protocol_body_bytes,
);
response.statusCode = route?.status ?? 404;
const body = route?.body ?? "";
response.end(body);
record({
if (body === undefined) {
recorder.truncated = true;
response.statusCode = 413;
response.end();
recorder.record({
at_ms: recorder.atMs(),
protocol: "http",
direction: "request",
method,
path,
data: "<body-over-limit>",
outcome: "unmatched",
});
return;
}
const index = recorder.scenario.replay.http.findIndex((route) => {
if (route.method !== method || route.path !== path) return false;
if (route.request_body !== undefined && route.request_body !== body)
return false;
return Object.entries(route.request_headers).every(
([name, value]) => request.headers[name.toLowerCase()] === value,
);
});
const route = recorder.scenario.replay.http[index];
const used = index < 0 ? 0 : (calls.get(index) ?? 0);
const outcome =
route === undefined
? "unmatched"
: used >= route.max_calls
? "script_exhausted"
: "matched";
recorder.record({
at_ms: recorder.atMs(),
protocol: "http",
direction: "request",
method,
path,
data: body.length === 0 ? "" : "<redacted-request-body>",
outcome,
});
if (route === undefined || outcome === "script_exhausted") {
response.statusCode = route === undefined ? 404 : 409;
response.end();
return;
}
calls.set(index, used + 1);
if (route.delay_ms > 0)
await new Promise((resolveDelay) =>
setTimeout(resolveDelay, route.delay_ms),
);
if (route.disconnect) {
response.destroy();
recorder.record({
at_ms: recorder.atMs(),
protocol: "http",
direction: "response",
method,
path,
data: "",
outcome: "disconnected",
});
return;
}
response.statusCode = route.status;
for (const [name, value] of Object.entries(route.response_headers))
response.setHeader(name, value);
response.end(route.body);
recorder.record({
at_ms: recorder.atMs(),
protocol: "http",
direction: "response",
method,
path,
data: body,
data: route.body,
outcome: "matched",
});
});
};
};
const sendWebSocketScript = async (
client: WebSocket,
connection: number,
recorder: ReplayRecorder,
): Promise<void> => {
const scripts = recorder.scenario.replay.websocket_connections;
const script = scripts[connection - 1];
if (scripts.length > 0 && script === undefined) {
recorder.record({
at_ms: recorder.atMs(),
protocol: "websocket",
direction: "sent",
method: null,
path: "/ws",
data: "",
outcome: "script_exhausted",
});
client.terminate();
return;
}
const messages =
script?.messages ??
recorder.scenario.replay.websocket_messages.map((data) => ({
data,
delay_ms: 0,
}));
for (const message of messages) {
if (message.delay_ms > 0)
await new Promise((resolveDelay) =>
setTimeout(resolveDelay, message.delay_ms),
);
if (client.readyState !== 1) return;
client.send(message.data);
recorder.record({
at_ms: recorder.atMs(),
protocol: "websocket",
direction: "sent",
method: null,
path: "/ws",
data: message.data,
outcome: "matched",
});
}
if (script?.disconnect_after === true) {
client.close();
recorder.record({
at_ms: recorder.atMs(),
protocol: "websocket",
direction: "sent",
method: null,
path: "/ws",
data: "",
outcome: "disconnected",
});
}
};
const createWebSocketReplay = (
server: Server,
recorder: ReplayRecorder,
): WebSocketServer => {
let connections = 0;
const websocket = new WebSocketServer({
noServer: true,
maxPayload: 1_000_000,
maxPayload: recorder.scenario.limits.protocol_body_bytes,
clientTracking: true,
});
server.on("upgrade", (request, socket, head) => {
@ -62,27 +224,30 @@ export const startLoopbackReplay = async (
);
});
websocket.on("connection", (client: WebSocket) => {
connections += 1;
if (connections > recorder.scenario.limits.connections) {
recorder.truncated = true;
client.terminate();
return;
}
client.on("error", () => undefined);
client.on("message", (value) =>
record({
recorder.record({
at_ms: recorder.atMs(),
protocol: "websocket",
direction: "received",
method: null,
path: "/ws",
data: value.toString(),
outcome: "matched",
}),
);
for (const message of scenario.replay.websocket_messages) {
client.send(message);
record({
protocol: "websocket",
direction: "sent",
method: null,
path: "/ws",
data: message,
});
}
void sendWebSocketScript(client, connections, recorder);
});
return websocket;
};
const listen = async (server: Server): Promise<number> => {
await new Promise<void>((resolveListen, rejectListen) => {
server.once("error", rejectListen);
server.listen(0, "127.0.0.1", () => {
@ -91,14 +256,39 @@ export const startLoopbackReplay = async (
});
});
const address = server.address();
if (address === null || typeof address === "string") {
await closeServer(server);
if (address === null || typeof address === "string")
throw new Error("loopback replay did not acquire a TCP port");
return address.port;
};
const closeServer = (server: Server): Promise<void> =>
new Promise((resolveClose, rejectClose) => {
server.close((error) =>
error === undefined ? resolveClose() : rejectClose(error),
);
});
/** Start bounded HTTP and WebSocket replay endpoints on IPv4 loopback only. */
export const startLoopbackReplay = async (
scenario: ProcessScenario,
): Promise<LoopbackReplay> => {
const recorder = new ReplayRecorder(scenario);
const server = createServer(createHttpHandler(recorder));
const websocket = createWebSocketReplay(server, recorder);
let port: number;
try {
port = await listen(server);
} catch (cause: unknown) {
await closeServer(server).catch(() => undefined);
throw cause;
}
return {
httpUrl: `http://127.0.0.1:${String(address.port)}`,
websocketUrl: `ws://127.0.0.1:${String(address.port)}/ws`,
events,
httpUrl: `http://127.0.0.1:${String(port)}`,
websocketUrl: `ws://127.0.0.1:${String(port)}/ws`,
events: recorder.events,
get truncated() {
return recorder.truncated;
},
async close() {
for (const client of websocket.clients) client.terminate();
await new Promise<void>((resolveClose) =>

538
src/application/ProcessHarness.ts
View File

@ -1,17 +1,7 @@
import { createHash } from "node:crypto";
import {
lstat,
mkdtemp,
readdir,
readFile,
readlink,
realpath,
rm,
} from "node:fs/promises";
import { randomUUID } from "node:crypto";
import { mkdtemp, realpath, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join, relative } from "node:path";
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import { join } from "node:path";
import { spawn, type IPty } from "node-pty";
import type {
FileState,
@ -25,8 +15,14 @@ import { authorizeProcessScenario } from "../domain/processCapture.js";
import { err, ok, type Result } from "../domain/result.js";
import { AnalysisError } from "../domain/errors.js";
import { startLoopbackReplay, type LoopbackReplay } from "./LoopbackReplay.js";
const execFileAsync = promisify(execFile);
import { cleanupOwnedProcessGroup } from "./ProcessOwnership.js";
import { startProcessSampler } from "./ProcessSampling.js";
import { snapshotRoots, type SnapshotResult } from "./FilesystemSnapshot.js";
import {
normalizeProcessSamples,
normalizeProcessText,
redactProtocolEvents,
} from "./ProcessNormalization.js";
const isWithin = (candidate: string, root: string): boolean =>
candidate === root ||
@ -66,6 +62,11 @@ export class ProcessCaptureError extends AnalysisError {
readonly _tag = "ProcessCaptureError";
}
const assertNotCancelled = (signal: AbortSignal | undefined): void => {
if (signal?.aborted === true)
throw new ProcessCaptureError("process capture was cancelled");
};
/** Runtime availability of the native PTY adapter on this host. */
export type ProcessCaptureCapability =
| { readonly available: true; readonly backend: "node-pty" }
@ -103,129 +104,17 @@ export const probeProcessCaptureCapability =
}
};
interface SnapshotResult {
readonly files: readonly FileState[];
readonly truncated: boolean;
}
const hashFile = async (
path: string,
maxBytes: number,
): Promise<string | null> => {
const value = await readFile(path);
if (value.byteLength > maxBytes) return null;
return createHash("sha256").update(value).digest("hex");
};
const snapshotRoots = async (
scenario: ProcessScenario,
): Promise<SnapshotResult> => {
const entries: FileState[] = [];
let remainingBytes = scenario.limits.file_bytes;
let truncated = false;
const visit = async (root: string, path: string): Promise<void> => {
if (entries.length >= scenario.limits.files) {
truncated = true;
return;
}
let stats;
try {
stats = await lstat(path);
} catch (error: unknown) {
if (error instanceof Error && "code" in error && error.code === "ENOENT")
return;
throw error;
}
const relativePath = relative(root, path) || ".";
if (stats.isSymbolicLink()) {
entries.push({
path: `${root}:${relativePath}`,
type: "symlink",
mode: stats.mode,
size: stats.size,
sha256: null,
symlink_target: await readlink(path),
});
return;
}
if (stats.isFile()) {
const allowedBytes = Math.max(
0,
Math.min(remainingBytes, scenario.limits.file_bytes),
);
const sha256 =
allowedBytes >= stats.size ? await hashFile(path, allowedBytes) : null;
remainingBytes -= sha256 === null ? 0 : stats.size;
if (sha256 === null) truncated = true;
entries.push({
path: `${root}:${relativePath}`,
type: "file",
mode: stats.mode,
size: stats.size,
sha256,
symlink_target: null,
});
return;
}
const type = stats.isDirectory() ? "directory" : "other";
entries.push({
path: `${root}:${relativePath}`,
type,
mode: stats.mode,
size: stats.size,
sha256: null,
symlink_target: null,
});
if (type !== "directory") return;
for (const child of (await readdir(path)).sort())
await visit(root, join(path, child));
};
for (const root of scenario.filesystem_roots) await visit(root, root);
return { files: entries, truncated };
};
const sampleProcesses = async (
rootPid: number,
elapsedMs: number,
limit: number,
): Promise<readonly ProcessSample[]> => {
if (process.platform === "win32") return [];
const { stdout } = await execFileAsync("ps", ["-axo", "pid=,ppid=,command="]);
const rows = stdout
.split("\n")
.map((line) => /\s*(\d+)\s+(\d+)\s+(.*)/.exec(line))
.filter((match): match is RegExpExecArray => match !== null)
.map((match) => ({
pid: Number(match[1]),
parent_pid: Number(match[2]),
command: match[3] ?? "",
}));
const owned = new Set([rootPid]);
let changed = true;
while (changed) {
changed = false;
for (const row of rows) {
if (owned.has(row.parent_pid) && !owned.has(row.pid)) {
owned.add(row.pid);
changed = true;
}
}
}
return rows
.filter((row) => owned.has(row.pid))
.slice(0, limit)
.map((row) => ({ at_ms: elapsedMs, ...row }));
};
const makeEnvironment = (
scenario: ProcessScenario,
home: string,
replay: LoopbackReplay,
runId: string,
): Record<string, string> => {
const environment: Record<string, string> = {
...scenario.environment,
HOME: home,
TERM: "xterm-256color",
...scenario.environment,
REA_PROCESS_RUN_ID: runId,
};
for (const name of scenario.inherit_environment) {
const value = process.env[name];
@ -236,29 +125,6 @@ const makeEnvironment = (
return environment;
};
const normalizeTerminalData = (
value: string,
scenario: ProcessScenario,
temporaryRoot: string,
pid: number,
): string => {
let normalized = value;
for (const alias of scenario.secret_aliases) {
const secret = scenario.environment[alias];
if (secret !== undefined && secret.length > 0)
normalized = normalized.replaceAll(secret, "<redacted>");
}
if (scenario.normalization.paths)
normalized = normalized.replaceAll(temporaryRoot, "<temporary-root>");
if (scenario.normalization.pids)
normalized = normalized.replaceAll(String(pid), "<pid>");
if (scenario.normalization.ports)
normalized = normalized.replaceAll(/(?<=[:=])\d{2,5}\b/g, "<port>");
for (const pattern of scenario.normalization.patterns)
normalized = normalized.replaceAll(pattern.pattern, pattern.replacement);
return normalized;
};
const scheduleScenarioEvents = (
scenario: ProcessScenario,
getTerminal: () => IPty | undefined,
@ -276,6 +142,24 @@ const scheduleScenarioEvents = (
}
};
const startScenarioEvents = (
scenario: ProcessScenario,
getTerminal: () => IPty | undefined,
timers: Set<NodeJS.Timeout>,
): (() => void) => {
let started = false;
return () => {
if (started) return;
started = true;
scheduleScenarioEvents(scenario, getTerminal, timers);
};
};
const eventsRequireReadiness = (scenario: ProcessScenario): boolean =>
scenario.events.some(
(event) => event.type === "input" || event.type === "resize",
);
interface TerminalExitOptions {
readonly terminal: IPty;
readonly scenario: ProcessScenario;
@ -287,7 +171,11 @@ interface TerminalExitOptions {
interface CaptureResultOptions {
readonly frames: readonly TerminalFrame[];
readonly exit: { readonly exitCode: number; readonly signal?: number };
readonly exit: {
readonly exitCode: number;
readonly signal?: number;
readonly reason: "exited" | "timeout" | "idle_timeout";
};
readonly samples: readonly ProcessSample[];
readonly replay: LoopbackReplay;
readonly before: SnapshotResult;
@ -295,58 +183,43 @@ interface CaptureResultOptions {
readonly truncated: boolean;
readonly scenario: ProcessScenario;
readonly rootPid: number;
readonly samplingPartial: boolean;
}
const normalizeSamples = (
samples: readonly ProcessSample[],
scenario: ProcessScenario,
rootPid: number,
): readonly ProcessSample[] => {
const identifiers = [
rootPid,
...samples.flatMap((sample) => [sample.pid, sample.parent_pid]),
];
const mapping = new Map<number, number>();
for (const identifier of identifiers)
if (identifier > 0 && !mapping.has(identifier))
mapping.set(identifier, mapping.size + 1);
return samples.map((sample) => ({
at_ms:
Math.floor(sample.at_ms / scenario.normalization.time_bucket_ms) *
scenario.normalization.time_bucket_ms,
pid: mapping.get(sample.pid) ?? 1,
parent_pid: mapping.get(sample.parent_pid) ?? 0,
command: normalizeTerminalData(
sample.command,
scenario,
"<no-temporary-root>",
rootPid,
),
}));
const classifyFilesystemEffects = (
before: readonly FileState[],
after: readonly FileState[],
): ProcessCapture["filesystem_effects"] => {
const beforeByPath = new Map(before.map((file) => [file.path, file]));
const afterByPath = new Map(after.map((file) => [file.path, file]));
const paths = [
...new Set([...beforeByPath.keys(), ...afterByPath.keys()]),
].sort();
return paths.map((path) => {
const beforeFile = beforeByPath.get(path) ?? null;
const afterFile = afterByPath.get(path) ?? null;
const status =
beforeFile === null
? "created"
: afterFile === null
? "deleted"
: JSON.stringify(beforeFile) === JSON.stringify(afterFile)
? "unchanged"
: "modified";
return { path, status, before: beforeFile, after: afterFile };
});
};
const redactProtocolEvents = (
events: readonly ProcessCapture["protocol_events"][number][],
scenario: ProcessScenario,
): readonly ProcessCapture["protocol_events"][number][] =>
events.map((event) => ({
...event,
data: normalizeTerminalData(
event.data,
scenario,
"<no-temporary-root>",
-1,
),
}));
const buildCaptureResult = (options: CaptureResultOptions): ProcessCapture => ({
schema_version: 1,
schema_version: 2,
normalization: options.scenario.normalization,
frames: options.frames,
exit: {
code: options.exit.exitCode < 0 ? null : options.exit.exitCode,
signal: options.exit.signal ?? null,
reason: options.exit.reason,
},
process_samples: normalizeSamples(
process_samples: normalizeProcessSamples(
options.samples,
options.scenario,
options.rootPid,
@ -357,12 +230,33 @@ const buildCaptureResult = (options: CaptureResultOptions): ProcessCapture => ({
),
files_before: options.before.files,
files_after: options.after.files,
filesystem_effects: classifyFilesystemEffects(
options.before.files,
options.after.files,
),
truncated: options.truncated,
limitations: [
"Process trees are sampled and may omit short-lived descendants.",
...(options.samplingPartial
? ["Process-tree sampling ended with an incomplete observation."]
: []),
"Filesystem observations are before/after snapshots, not syscall traces.",
"The harness does not enforce external network isolation.",
],
residual_unknowns: [
{
scope: "process",
reason: "Process trees are sampled and may omit short-lived descendants.",
},
{
scope: "network",
reason: "External network isolation is not enforced by this adapter.",
},
],
cleanup: {
owned_process_group: "verified",
temporary_root: "removed",
},
});
const awaitTerminalExit = async ({
@ -372,92 +266,199 @@ const awaitTerminalExit = async ({
lastOutput,
signal,
timers,
}: TerminalExitOptions): Promise<{ exitCode: number; signal?: number }> =>
}: TerminalExitOptions): Promise<{
exitCode: number;
signal?: number;
reason: "exited" | "timeout" | "idle_timeout" | "cancelled";
}> =>
new Promise((resolveExit) => {
terminal.onExit(resolveExit);
let reason: "exited" | "timeout" | "idle_timeout" | "cancelled" = "exited";
terminal.onExit((exit) => {
for (const timer of timers) {
clearTimeout(timer);
clearInterval(timer);
}
timers.clear();
resolveExit({ ...exit, reason });
});
const timeout = setInterval(() => {
if (
Date.now() - started >= scenario.timeout_ms ||
Date.now() - lastOutput() >= scenario.idle_timeout_ms ||
signal?.aborted === true
) {
if (signal?.aborted === true) {
reason = "cancelled";
terminal.kill("SIGKILL");
} else if (Date.now() - started >= scenario.timeout_ms) {
reason = "timeout";
terminal.kill("SIGKILL");
} else if (Date.now() - lastOutput() >= scenario.idle_timeout_ms) {
reason = "idle_timeout";
terminal.kill("SIGKILL");
}
}, 20);
timers.add(timeout);
});
const releaseProcessResources = async (options: {
readonly timers: ReadonlySet<NodeJS.Timeout>;
readonly replay: LoopbackReplay | undefined;
readonly terminal: IPty | undefined;
readonly runId: string;
readonly temporaryRoot: string;
}): Promise<string | undefined> => {
for (const timer of options.timers) clearTimeout(timer);
let failure: string | undefined;
try {
await options.replay?.close();
} catch {
failure = "loopback replay cleanup failed";
}
if (options.terminal !== undefined && process.platform !== "win32") {
const cleaned = await cleanupOwnedProcessGroup({
runId: options.runId,
leaderPid: options.terminal.pid,
processGroupId: options.terminal.pid,
});
if (!cleaned.cleaned) failure ??= cleaned.reason;
}
try {
await rm(options.temporaryRoot, { recursive: true, force: true });
} catch {
failure ??= "temporary process root cleanup failed";
}
return failure;
};
const captureTerminalFrames = (options: {
readonly terminal: IPty;
readonly scenario: ProcessScenario;
readonly frames: TerminalFrame[];
readonly started: number;
readonly temporaryRoot: string;
readonly onOutput: () => void;
readonly onFirstOutput: () => void;
}): (() => boolean) => {
let outputBytes = 0;
let truncated = false;
options.terminal.onData((data) => {
options.onFirstOutput();
options.onOutput();
const bytes = Buffer.byteLength(data);
if (
options.frames.length >= options.scenario.limits.frames ||
outputBytes + bytes > options.scenario.limits.output_bytes
) {
truncated = true;
return;
}
outputBytes += bytes;
options.frames.push({
sequence: options.frames.length,
at_ms:
Math.floor(
(Date.now() - options.started) /
options.scenario.normalization.time_bucket_ms,
) * options.scenario.normalization.time_bucket_ms,
data: normalizeProcessText(
data,
options.scenario,
options.temporaryRoot,
options.terminal.pid,
),
});
});
return () => truncated;
};
const resolveProcessResult = (
capture: ProcessCapture | undefined,
executionFailure: unknown,
cleanupFailure: string | undefined,
): ProcessCapture => {
if (executionFailure instanceof ProcessCaptureError) throw executionFailure;
if (executionFailure !== undefined)
throw new ProcessCaptureError("process capture failed", {
cause: executionFailure,
});
if (cleanupFailure !== undefined)
throw new ProcessCaptureError(cleanupFailure);
if (capture === undefined)
throw new ProcessCaptureError("process capture produced no result");
return capture;
};
const prepareProcessCapture = async (
scenario: ProcessScenario,
policy: ProcessExecutionPolicy,
signal: AbortSignal | undefined,
): Promise<{
readonly temporaryRoot: string;
readonly runId: string;
readonly home: string;
readonly before: SnapshotResult;
}> => {
const decision = authorizeProcessScenario(scenario, policy);
if (!decision.allowed) throw new ProcessCaptureError(decision.reason);
await assertRealPathAuthority(scenario, policy);
assertNotCancelled(signal);
const before = await snapshotRoots(scenario, signal);
const temporaryRoot = await mkdtemp(join(tmpdir(), "rea-process-"));
const runId = randomUUID();
const home = join(temporaryRoot, "home");
await import("node:fs/promises").then(({ mkdir }) => mkdir(home));
return { temporaryRoot, runId, home, before };
};
/** Execute one authorized scenario and return bounded observations. */
const runProcessScenario = async (
scenario: ProcessScenario,
policy: ProcessExecutionPolicy,
signal?: AbortSignal,
): Promise<ProcessCapture> => {
const decision = authorizeProcessScenario(scenario, policy);
if (!decision.allowed) throw new ProcessCaptureError(decision.reason);
await assertRealPathAuthority(scenario, policy);
if (signal?.aborted === true)
throw new ProcessCaptureError("process capture was cancelled");
const temporaryRoot = await mkdtemp(join(tmpdir(), "rea-process-"));
const home = join(temporaryRoot, "home");
await import("node:fs/promises").then(({ mkdir }) => mkdir(home));
const before = await snapshotRoots(scenario);
const { temporaryRoot, runId, home, before } = await prepareProcessCapture(
scenario,
policy,
signal,
);
const frames: TerminalFrame[] = [];
const samples: ProcessSample[] = [];
let outputBytes = 0;
let truncated = before.truncated;
let terminal: IPty | undefined;
let replay: LoopbackReplay | undefined;
const started = Date.now();
let lastOutput = started;
let started = 0;
let lastOutput = 0;
const timers = new Set<NodeJS.Timeout>();
let capture: ProcessCapture | undefined;
let executionFailure: unknown;
let framesTruncated = (): boolean => false;
let stopSampler = async () => ({ partial: false });
let samplingPartial = false;
try {
replay = await startLoopbackReplay(scenario);
started = Date.now();
lastOutput = started;
terminal = spawn(scenario.executable, [...scenario.arguments], {
cwd: scenario.working_directory,
env: makeEnvironment(scenario, home, replay),
env: makeEnvironment(scenario, home, replay, runId),
cols: 80,
rows: 24,
name: "xterm-256color",
});
terminal.onData((data) => {
lastOutput = Date.now();
const bytes = Buffer.byteLength(data);
if (
frames.length >= scenario.limits.frames ||
outputBytes + bytes > scenario.limits.output_bytes
) {
truncated = true;
return;
}
outputBytes += bytes;
frames.push({
sequence: frames.length,
at_ms:
Math.floor(
(Date.now() - started) / scenario.normalization.time_bucket_ms,
) * scenario.normalization.time_bucket_ms,
data: normalizeTerminalData(
data,
scenario,
temporaryRoot,
terminal?.pid ?? -1,
),
});
const startEvents = startScenarioEvents(scenario, () => terminal, timers);
if (!eventsRequireReadiness(scenario)) startEvents();
framesTruncated = captureTerminalFrames({
terminal,
scenario,
frames,
started,
temporaryRoot,
onOutput: () => (lastOutput = Date.now()),
onFirstOutput: startEvents,
});
scheduleScenarioEvents(scenario, () => terminal, timers);
const sampler = setInterval(() => {
void sampleProcesses(
terminal?.pid ?? -1,
Date.now() - started,
scenario.limits.processes - samples.length,
)
.then((values) => samples.push(...values))
.catch(() => undefined);
}, 50);
timers.add(sampler);
stopSampler = startProcessSampler(
terminal.pid,
started,
scenario.limits.processes,
samples,
);
const exit = await awaitTerminalExit({
terminal,
scenario,
@ -466,15 +467,24 @@ const runProcessScenario = async (
signal,
timers,
});
const exitReason = exit.reason;
samplingPartial = (await stopSampler()).partial;
if (exitReason === "cancelled")
throw new ProcessCaptureError("process capture was cancelled");
await new Promise((resolveSettle) =>
setTimeout(resolveSettle, scenario.settle_ms),
);
const after = await snapshotRoots(scenario);
assertNotCancelled(signal);
const after = await snapshotRoots(scenario, signal);
truncated ||=
after.truncated || samples.length >= scenario.limits.processes;
return buildCaptureResult({
after.truncated ||
replay.truncated ||
framesTruncated() ||
samples.length >= scenario.limits.processes;
truncated ||= samplingPartial;
capture = buildCaptureResult({
frames,
exit,
exit: { ...exit, reason: exitReason },
samples,
replay,
before,
@ -482,15 +492,21 @@ const runProcessScenario = async (
truncated,
scenario,
rootPid: terminal.pid,
samplingPartial,
});
} catch (cause: unknown) {
terminal?.kill("SIGKILL");
throw new ProcessCaptureError("process capture failed", { cause });
} finally {
for (const timer of timers) clearTimeout(timer);
await replay?.close();
await rm(temporaryRoot, { recursive: true, force: true });
executionFailure = cause;
}
samplingPartial ||= (await stopSampler()).partial;
const cleanupFailure = await releaseProcessResources({
timers,
replay,
terminal,
runId,
temporaryRoot,
});
return resolveProcessResult(capture, executionFailure, cleanupFailure);
};
/** Execute one scenario through a typed expected-failure channel. */

77
src/application/ProcessNormalization.ts Normal file
View File

@ -0,0 +1,77 @@
import type {
ProcessCapture,
ProcessSample,
ProcessScenario,
} from "../domain/processCapture.js";
/** Normalize and redact one terminal/protocol payload under scenario rules. */
export const normalizeProcessText = (
value: string,
scenario: ProcessScenario,
temporaryRoot: string,
pid: number,
): string => {
let normalized = value;
for (const alias of scenario.secret_aliases) {
const secret = scenario.environment[alias];
if (secret !== undefined && secret.length > 0)
normalized = normalized.replaceAll(secret, "<redacted>");
}
if (scenario.normalization.paths) {
normalized = normalized.replaceAll(temporaryRoot, "<temporary-root>");
normalized = normalized.replaceAll(
scenario.working_directory,
"<working-directory>",
);
normalized = normalized.replaceAll(scenario.executable, "<executable>");
for (const [index, root] of scenario.filesystem_roots.entries())
normalized = normalized.replaceAll(
root,
`<filesystem-root-${String(index)}>`,
);
}
if (scenario.normalization.pids)
normalized = normalized.replaceAll(String(pid), "<pid>");
if (scenario.normalization.ports)
normalized = normalized.replaceAll(/(?<=[:=])\d{2,5}\b/g, "<port>");
for (const pattern of scenario.normalization.patterns)
normalized = normalized.replaceAll(pattern.pattern, pattern.replacement);
return normalized;
};
export const normalizeProcessSamples = (
samples: readonly ProcessSample[],
scenario: ProcessScenario,
rootPid: number,
): readonly ProcessSample[] => {
const identifiers = [
rootPid,
...samples.flatMap((sample) => [sample.pid, sample.parent_pid]),
];
const mapping = new Map<number, number>();
for (const identifier of identifiers)
if (identifier > 0 && !mapping.has(identifier))
mapping.set(identifier, mapping.size + 1);
return samples.map((sample) => ({
at_ms:
Math.floor(sample.at_ms / scenario.normalization.time_bucket_ms) *
scenario.normalization.time_bucket_ms,
pid: mapping.get(sample.pid) ?? 1,
parent_pid: mapping.get(sample.parent_pid) ?? 0,
command: normalizeProcessText(
sample.command,
scenario,
"<no-temporary-root>",
rootPid,
),
}));
};
export const redactProtocolEvents = (
events: readonly ProcessCapture["protocol_events"][number][],
scenario: ProcessScenario,
): readonly ProcessCapture["protocol_events"][number][] =>
events.map((event) => ({
...event,
data: normalizeProcessText(event.data, scenario, "<no-temporary-root>", -1),
}));

115
src/application/ProcessOwnership.ts Normal file
View File

@ -0,0 +1,115 @@
import { execFile } from "node:child_process";
import { readFile } from "node:fs/promises";
import { promisify } from "node:util";
const execFileAsync = promisify(execFile);
export interface OwnedProcessGroup {
readonly runId: string;
readonly leaderPid: number;
readonly processGroupId: number;
}
interface ProcessGroupMember {
readonly pid: number;
readonly parentPid: number;
readonly processGroupId: number;
readonly command: string;
}
export interface ProcessOwnershipHost {
listMembers(processGroupId: number): Promise<readonly ProcessGroupMember[]>;
environment(pid: number): Promise<Readonly<Record<string, string>>>;
signalGroup(processGroupId: number, signal: NodeJS.Signals): void;
}
export type ProcessCleanupResult =
| { readonly cleaned: true; readonly signaled: boolean }
| { readonly cleaned: false; readonly reason: string };
const parseEnvironment = (value: string): Readonly<Record<string, string>> =>
Object.fromEntries(
value
.split("\0")
.filter((entry) => entry.includes("="))
.map((entry) => {
const separator = entry.indexOf("=");
return [entry.slice(0, separator), entry.slice(separator + 1)];
}),
);
const systemHost: ProcessOwnershipHost = {
async listMembers(processGroupId) {
if (process.platform === "win32") return [];
const { stdout } = await execFileAsync("ps", [
"-axo",
"pid=,ppid=,pgid=,command=",
]);
return stdout
.split("\n")
.map((line) => /\s*(\d+)\s+(\d+)\s+(\d+)\s+(.*)/u.exec(line))
.filter((match): match is RegExpExecArray => match !== null)
.map((match) => ({
pid: Number(match[1]),
parentPid: Number(match[2]),
processGroupId: Number(match[3]),
command: match[4] ?? "",
}))
.filter((member) => member.processGroupId === processGroupId);
},
async environment(pid) {
if (process.platform === "linux") {
return parseEnvironment(await readFile(`/proc/${pid}/environ`, "utf8"));
}
const { stdout } = await execFileAsync("ps", ["eww", "-p", String(pid)]);
const environment: Record<string, string> = {};
for (const match of stdout.matchAll(
/(?:^|\s)([A-Za-z_][A-Za-z0-9_]*)=([^\s]*)/gu,
))
environment[match[1]!] = match[2] ?? "";
return environment;
},
signalGroup(processGroupId, signal) {
process.kill(-processGroupId, signal);
},
};
/** Verify run-token ownership before signaling a POSIX process group. */
export const cleanupOwnedProcessGroup = async (
ownership: OwnedProcessGroup,
host: ProcessOwnershipHost = systemHost,
): Promise<ProcessCleanupResult> => {
let members: readonly ProcessGroupMember[];
try {
members = await host.listMembers(ownership.processGroupId);
} catch {
return { cleaned: false, reason: "process group could not be inspected" };
}
if (members.length === 0) return { cleaned: true, signaled: false };
for (const member of members) {
let environment: Readonly<Record<string, string>>;
try {
environment = await host.environment(member.pid);
} catch {
return {
cleaned: false,
reason: "process ownership could not be revalidated",
};
}
if (environment.REA_PROCESS_RUN_ID !== ownership.runId) {
return {
cleaned: false,
reason: "process group contains an unowned or PID-reused process",
};
}
}
try {
host.signalGroup(ownership.processGroupId, "SIGKILL");
} catch (cause: unknown) {
const code =
cause instanceof Error && "code" in cause ? cause.code : undefined;
if (code !== "ESRCH")
return { cleaned: false, reason: "owned process group signal failed" };
}
return { cleaned: true, signaled: true };
};

368
src/application/ProcessSampling.ts Normal file
View File

@ -0,0 +1,368 @@
import { execFile } from "node:child_process";
import { readFile } from "node:fs/promises";
import { promisify } from "node:util";
import type { ProcessSample } from "../domain/processCapture.js";
const execFileAsync = promisify(execFile);
const PROC_READ_CONCURRENCY = 64;
interface ProcessRow {
readonly pid: number;
readonly parent_pid: number;
readonly command: string;
readonly startTime: string | undefined;
}
interface ProcessNode extends ProcessRow {
readonly children: readonly number[];
}
interface SampleProcessContext {
readonly rootPid: number;
readonly elapsedMs: number;
readonly limit: number;
readonly sampledPids: Set<number>;
readonly identities: Map<number, string>;
readonly signal: AbortSignal;
}
const isExpectedProcReadFailure = (cause: unknown): boolean =>
cause instanceof Error &&
"code" in cause &&
(cause.code === "ENOENT" ||
cause.code === "ESRCH" ||
cause.code === "EACCES" ||
cause.code === "EPERM");
const handleProcReadFailure = (cause: unknown): undefined => {
if (isExpectedProcReadFailure(cause)) return undefined;
throw cause;
};
const parseProcStat = (
identifier: string,
stat: string,
): ProcessRow | undefined => {
const commandEnd = stat.lastIndexOf(")");
if (commandEnd < 0) return undefined;
const fields = stat
.slice(commandEnd + 2)
.trim()
.split(/\s+/);
const pid = Number(identifier);
const parentPid = Number(fields[1]);
const startTime = fields[19];
if (
!Number.isSafeInteger(pid) ||
pid <= 0 ||
!Number.isSafeInteger(parentPid) ||
parentPid < 0 ||
startTime === undefined ||
!/^\d+$/.test(startTime)
)
return undefined;
return {
pid,
parent_pid: parentPid,
command: "",
startTime,
};
};
const readProcessStat = async (
pid: number,
signal: AbortSignal,
): Promise<ProcessRow | undefined> => {
try {
const stat = await readFile(`/proc/${String(pid)}/stat`, {
encoding: "utf8",
signal,
});
return parseProcStat(String(pid), stat);
} catch (cause: unknown) {
return handleProcReadFailure(cause);
}
};
const readProcessCommand = async (
pid: number,
signal: AbortSignal,
): Promise<Buffer | undefined> => {
try {
return await readFile(`/proc/${String(pid)}/cmdline`, { signal });
} catch (cause: unknown) {
return handleProcReadFailure(cause);
}
};
const readLinuxChildren = async (
pid: number,
signal: AbortSignal,
): Promise<readonly number[] | undefined> => {
let text: string;
try {
text = await readFile(`/proc/${String(pid)}/task/${String(pid)}/children`, {
encoding: "utf8",
signal,
});
} catch (cause: unknown) {
return handleProcReadFailure(cause);
}
return text
.trim()
.split(/\s+/)
.filter((value) => /^\d+$/.test(value))
.map(Number)
.filter((child) => Number.isSafeInteger(child) && child > 0);
};
const inspectProcess = async (
pid: number,
expectedParent: number | undefined,
signal: AbortSignal,
identities: Map<number, string>,
): Promise<ProcessNode | undefined> => {
const before = await readProcessStat(pid, signal);
if (before === undefined || before.startTime === undefined) return undefined;
const existing = identities.get(pid);
if (existing !== undefined && existing !== before.startTime) return undefined;
if (expectedParent !== undefined && before.parent_pid !== expectedParent)
return undefined;
const [commandBuffer, children] = await Promise.all([
readProcessCommand(pid, signal),
readLinuxChildren(pid, signal),
]);
const after = await readProcessStat(pid, signal);
if (after === undefined || after.startTime === undefined) return undefined;
if (
before.startTime !== after.startTime ||
before.parent_pid !== after.parent_pid
)
return undefined;
if (expectedParent !== undefined && after.parent_pid !== expectedParent)
return undefined;
const command =
commandBuffer === undefined
? ""
: commandBuffer.toString().replaceAll("\0", " ").trim();
return {
pid: before.pid,
parent_pid: before.parent_pid,
startTime: before.startTime,
command,
children: children ?? [],
};
};
const sampleLinux = async (
context: SampleProcessContext,
): Promise<readonly ProcessRow[]> => {
const { rootPid, limit, signal, sampledPids, identities } = context;
if (limit <= 0) return [];
const rootStat = await readProcessStat(rootPid, signal);
if (rootStat === undefined) return [];
const rootChildren = await readLinuxChildren(rootPid, signal);
if (rootChildren === undefined) return samplePs(context);
const rows: ProcessRow[] = [];
const visited = new Set<number>([rootPid]);
let level: Array<{
readonly pid: number;
readonly expectedParent: number | undefined;
}> = [{ pid: rootPid, expectedParent: undefined }];
while (level.length > 0 && rows.length < limit) {
const batchSize = Math.min(level.length, PROC_READ_CONCURRENCY);
const batch = level.slice(0, batchSize);
level = level.slice(batchSize);
const batchResults = await Promise.all(
batch.map((item) =>
inspectProcess(item.pid, item.expectedParent, signal, identities),
),
);
const nextLevel: typeof batch = [];
for (const node of batchResults) {
if (node === undefined) continue;
if (!sampledPids.has(node.pid)) {
if (rows.length >= limit) break;
rows.push(node);
}
if (rows.length >= limit) break;
for (const child of node.children) {
if (visited.has(child)) continue;
visited.add(child);
nextLevel.push({ pid: child, expectedParent: node.pid });
}
if (rows.length >= limit) break;
}
level.push(...nextLevel);
}
return rows;
};
const readPsRows = async (
signal: AbortSignal,
): Promise<readonly ProcessRow[]> => {
const { stdout } = await execFileAsync(
"ps",
["-axo", "pid=,ppid=,command="],
{ signal },
);
return stdout
.split("\n")
.map((line) => /\s*(\d+)\s+(\d+)\s+(.*)/u.exec(line))
.filter((match): match is RegExpExecArray => match !== null)
.map((match) => ({
pid: Number(match[1]),
parent_pid: Number(match[2]),
command: match[3] ?? "",
startTime: undefined,
}))
.filter(
(row) =>
Number.isSafeInteger(row.pid) &&
row.pid > 0 &&
Number.isSafeInteger(row.parent_pid) &&
row.parent_pid >= 0,
)
.sort((left, right) => left.pid - right.pid);
};
const samplePs = async (
context: SampleProcessContext,
): Promise<readonly ProcessRow[]> => {
const { rootPid, limit, signal, sampledPids } = context;
if (limit <= 0) return [];
const rows = await readPsRows(signal);
const rowByPid = new Map(rows.map((row) => [row.pid, row]));
const childrenByParent = new Map<number, ProcessRow[]>();
for (const row of rows) {
const siblings = childrenByParent.get(row.parent_pid) ?? [];
siblings.push(row);
childrenByParent.set(row.parent_pid, siblings);
}
const result: ProcessRow[] = [];
const visited = new Set<number>();
const queue: number[] = [rootPid];
let index = 0;
while (index < queue.length && result.length < limit) {
const pid = queue[index];
if (pid === undefined) break;
index += 1;
if (visited.has(pid)) continue;
visited.add(pid);
const row = rowByPid.get(pid);
if (row !== undefined && !sampledPids.has(row.pid)) {
result.push(row);
}
if (result.length >= limit) break;
for (const child of childrenByParent.get(pid) ?? []) {
if (!visited.has(child.pid)) queue.push(child.pid);
}
}
return result;
};
const sampleProcesses = async (
context: SampleProcessContext,
): Promise<readonly ProcessSample[]> => {
if (process.platform === "win32" || context.limit <= 0) return [];
const rows =
process.platform === "linux"
? await sampleLinux(context)
: await samplePs(context);
const { elapsedMs, sampledPids, identities } = context;
const samples: ProcessSample[] = [];
for (const row of rows) {
if (sampledPids.has(row.pid)) continue;
if (row.startTime !== undefined) {
const existing = identities.get(row.pid);
if (existing !== undefined && existing !== row.startTime) continue;
identities.set(row.pid, row.startTime);
}
samples.push({
at_ms: elapsedMs,
pid: row.pid,
parent_pid: row.parent_pid,
command: row.command,
});
}
return samples;
};
/** Start bounded process-tree sampling and expose an awaited stop. */
export const startProcessSampler = (
rootPid: number,
started: number,
limit: number,
samples: ProcessSample[],
): (() => Promise<{ readonly partial: boolean }>) => {
const identities = new Map<number, string>();
const sampledPids = new Set<number>(samples.map(({ pid }) => pid));
let pending: Promise<void> | undefined;
let stopped = false;
let abortCurrent: (() => void) | undefined;
let partial = false;
const sample = (): void => {
if (stopped || pending !== undefined || sampledPids.size >= limit) return;
const controller = new AbortController();
abortCurrent = () => controller.abort();
pending = sampleProcesses({
rootPid,
elapsedMs: Date.now() - started,
limit: limit - sampledPids.size,
sampledPids,
identities,
signal: controller.signal,
})
.then((values) => {
for (const value of values) {
if (sampledPids.size >= limit) break;
if (sampledPids.has(value.pid)) continue;
sampledPids.add(value.pid);
samples.push(value);
}
})
.catch((cause: unknown) => {
if (!(cause instanceof Error && cause.name === "AbortError"))
partial = true;
})
.then(() => {
abortCurrent = undefined;
pending = undefined;
});
};
const timer = setInterval(sample, 50);
sample();
return async () => {
stopped = true;
clearInterval(timer);
abortCurrent?.();
await pending;
return { partial };
};
};

135
src/application/RealHopperAssertions.ts Normal file
View File

@ -0,0 +1,135 @@
import { z } from "zod";
const addressSchema = z.string().regex(/^0x[0-9a-f]+$/iu);
const identitySchema = z.object({ address: addressSchema, name: z.string() });
const collectionSchema = z.object({
items: z.array(z.unknown()),
total: z.number().int().min(0).nullable(),
returned: z.number().int().min(0),
truncated: z.boolean(),
next_offset: z.number().int().min(0).nullable(),
});
/** Extract a real address from the provider's exact procedure page. */
export const firstProcedureAddress = (input: unknown): string => {
const page = z
.object({
items: z.array(z.object({ address: addressSchema })).min(1),
offset: z.number().int().min(0),
limit: z.number().int().min(1),
total: z.number().int().min(1),
next_offset: z.number().int().min(0).nullable(),
has_more: z.boolean(),
})
.parse(input);
const first = page.items[0];
if (first === undefined) throw new TypeError("Procedure page was empty");
return first.address;
};
/** Reject empty and success-shaped embedded decompilation failures. */
export const requirePseudocode = (
input: unknown,
operation: string,
): string => {
if (typeof input !== "string" || input.trim().length === 0)
throw new TypeError(`${operation} returned empty pseudocode`);
if (input === "No output" || input.startsWith("Error:"))
throw new TypeError(`${operation} returned an embedded failure: ${input}`);
return input;
};
/** Validate an exact bounded Hopper function dossier semantically. */
export const requireFunctionDossier = (
input: unknown,
expectedAddress: string,
): Record<string, unknown> => {
const value = objectValue(
input,
"analyze_function returned no function dossier",
);
const procedure = identitySchema.parse(value.procedure);
if (procedure.address !== expectedAddress)
throw new TypeError("analyze_function returned the wrong procedure");
if (procedure.name.trim().length === 0)
throw new TypeError("analyze_function omitted the procedure name");
validatePseudocodeBounds(value.pseudocode);
for (const field of COLLECTION_FIELDS)
validateCollection(value[field], field);
for (const field of ["callers", "callees"] as const)
z.array(identitySchema).parse(collectionSchema.parse(value[field]).items);
validateBlocks(collectionSchema.parse(value.basic_blocks).items);
validateInstructionScan(value.instruction_scan);
return value;
};
/** Require every relationship to be an actual hexadecimal address. */
export const requireAddressArray = (
input: unknown,
operation: string,
): string[] => {
const parsed = z.array(addressSchema).parse(input);
if (parsed.some((address) => address.length === 0))
throw new TypeError(`${operation} returned an invalid address list`);
return parsed;
};
const COLLECTION_FIELDS = [
"comments",
"callers",
"callees",
"incoming_references",
"outgoing_references",
"referenced_strings",
"referenced_names",
"basic_blocks",
] as const;
const validatePseudocodeBounds = (input: unknown): void => {
const pseudocode = z
.object({
text: z.string(),
returned_chars: z.number().int().min(0),
total_chars: z.number().int().min(0),
})
.parse(input);
const text = requirePseudocode(pseudocode.text, "analyze_function");
if (
pseudocode.returned_chars !== [...text].length ||
pseudocode.total_chars < pseudocode.returned_chars
)
throw new TypeError(
"analyze_function returned inconsistent pseudocode bounds",
);
};
const validateCollection = (input: unknown, field: string): void => {
const collection = collectionSchema.parse(input);
if (collection.returned !== collection.items.length)
throw new TypeError(`analyze_function returned an invalid ${field} result`);
};
const validateBlocks = (items: readonly unknown[]): void => {
for (const block of items)
if (!Array.isArray(objectValue(block, "Invalid basic block").successors))
throw new TypeError("analyze_function omitted CFG successor evidence");
};
const validateInstructionScan = (input: unknown): void => {
const scan = z
.object({ scanned: z.number().int().min(0), truncated: z.boolean() })
.parse(input);
if (scan.scanned === 0 && !scan.truncated)
throw new TypeError(
"analyze_function omitted instruction scan limitations",
);
};
const objectValue = (
input: unknown,
message: string,
): Record<string, unknown> => {
const parsed = z.record(z.string(), z.unknown()).safeParse(input);
if (!parsed.success) throw new TypeError(message);
return parsed.data;
};

260
src/application/ReferenceSourceImport.ts Normal file
View File

@ -0,0 +1,260 @@
import { isSecretLikePath } from "../domain/referenceSourceClassification.js";
import {
createHistoricalSourceGraph,
type HistoricalSourceGraph,
type HistoricalSourceGraphInput,
} from "../domain/referenceSourceGraph.js";
import { err, ok, type Result } from "../domain/result.js";
import { readReferenceSource } from "../reference/ReferenceSourceReader.js";
import { parseReferenceSourceEntries } from "./ReferenceSourceImportEntries.js";
import { readReferenceSourceVcs } from "./ReferenceSourceVcsAdapter.js";
import {
type ReferenceSourceImportError,
type ReferenceSourceImportOptions,
} from "./ReferenceSourceImportTypes.js";
import { prepareReferenceSourceImport } from "./ReferenceSourceImportPolicy.js";
export type {
ReferenceSourceImportError,
ReferenceSourceImportOptions,
} from "./ReferenceSourceImportTypes.js";
const compareCodePoints = (left: string, right: string): number => {
const leftPoints = [...left].map((value) => value.codePointAt(0) ?? 0);
const rightPoints = [...right].map((value) => value.codePointAt(0) ?? 0);
const length = Math.min(leftPoints.length, rightPoints.length);
for (let index = 0; index < length; index += 1) {
const difference = (leftPoints[index] ?? 0) - (rightPoints[index] ?? 0);
if (difference !== 0) return difference;
}
return leftPoints.length - rightPoints.length;
};
const byCodePoint = (left: string, right: string): number =>
compareCodePoints(left, right);
const failure = (
code: ReferenceSourceImportError["code"],
message: string,
): ReferenceSourceImportError => ({
tag: "reference-source-import",
code,
message,
});
const cancelled = (): ReferenceSourceImportError =>
failure("cancelled", "Reference source import cancelled");
const isAborted = (signal?: AbortSignal): boolean => signal?.aborted === true;
const relationshipKey = (
relationship: HistoricalSourceGraphInput["relationships"][number],
): string =>
`${relationship.from_path}\u0000${relationship.to}\u0000${relationship.kind}\u0000${relationship.resolution}\u0000${relationship.parse_state}`;
const deduplicateRelationships = (
relationships: HistoricalSourceGraphInput["relationships"],
): HistoricalSourceGraphInput["relationships"] => {
const seen = new Set<string>();
const result: HistoricalSourceGraphInput["relationships"] = [];
for (const relationship of relationships) {
const key = relationshipKey(relationship);
if (seen.has(key)) continue;
seen.add(key);
result.push(relationship);
}
return result;
};
const buildProvenance = (
options: ReferenceSourceImportOptions,
): HistoricalSourceGraphInput["provenance"] => ({
importer: options.importer ?? "rea-reference-source-import",
importer_version: options.importerVersion ?? null,
caller: options.caller,
});
const deriveLanguages = (
entries: HistoricalSourceGraphInput["entries"],
): string[] =>
[
...new Set(
entries.flatMap((entry) =>
entry.kind === "file" && entry.language !== null
? [entry.language]
: [],
),
),
].sort(byCodePoint);
const deriveManifests = (
entries: HistoricalSourceGraphInput["entries"],
): string[] =>
entries
.flatMap((entry) =>
entry.kind === "file" && entry.classifications.includes("manifest")
? [entry.path]
: [],
)
.sort(byCodePoint);
const deriveInventoryState = (
input: Pick<
HistoricalSourceGraphInput,
| "entries"
| "relationships"
| "parse_failures"
| "exclusions"
| "limitations"
>,
): "complete" | "partial" | "unknown" => {
if (input.limitations.length > 0) return "partial";
if (input.exclusions.length > 0) return "partial";
if (input.parse_failures.length > 0) return "partial";
const partialEntry = input.entries.some(
(entry) =>
entry.limitations.length > 0 ||
(entry.kind === "file" && entry.content_state !== "hashed") ||
(entry.kind === "directory" && entry.tree_state !== "enumerated") ||
(entry.kind === "symlink" && entry.target_state !== "internal"),
);
if (partialEntry) return "partial";
const partialRelationship = input.relationships.some(
({ parse_state, resolution }) =>
parse_state !== "parsed" ||
["unresolved", "unknown"].includes(resolution),
);
if (partialRelationship) return "partial";
return "complete";
};
const sortAndLimit = (
exclusions: HistoricalSourceGraphInput["exclusions"],
maxExclusions: number,
): HistoricalSourceGraphInput["exclusions"] => {
const sorted = [...exclusions].sort((left, right) => {
const byPath = byCodePoint(left.path, right.path);
if (byPath !== 0) return byPath;
return byCodePoint(left.reason, right.reason);
});
if (sorted.length <= maxExclusions) return sorted;
return sorted.slice(0, maxExclusions);
};
/**
* Import a reference source directory into a committed historical source graph.
*
* The import is deterministic, parallel-safe, and never executes source, hooks,
* git subprocesses, or network requests. Secret-like paths are redacted before
* they are committed to the graph.
*/
export const importReferenceSource = async (
options: ReferenceSourceImportOptions,
): Promise<Result<HistoricalSourceGraph, ReferenceSourceImportError>> => {
if (isAborted(options.signal)) return err(cancelled());
const prepared = await prepareReferenceSourceImport(options);
if (!prepared.ok) return prepared;
const { ignored, limits, root, secrets } = prepared.value;
if (isAborted(options.signal)) return err(cancelled());
const exclusions: HistoricalSourceGraphInput["exclusions"] = [];
const shouldExclude = (path: string): boolean => {
if (isSecretLikePath(path) || secrets.ignores(path)) {
exclusions.push({ path, reason: "configured-secret" });
return true;
}
if (ignored.ignores(path)) {
exclusions.push({ path, reason: "caller-excluded" });
return true;
}
return false;
};
const [readResult, vcs] = await Promise.all([
readReferenceSource(root, limits, {
...(options.signal === undefined ? {} : { signal: options.signal }),
shouldExclude,
}),
readReferenceSourceVcs(root, options.signal),
]);
if (!readResult.ok) {
const error = readResult.error;
if (error.code === "cancelled") return err(cancelled());
return err(
failure(
error.code === "invalid-limits" ? "invalid-limits" : "io",
error.message,
),
);
}
if (isAborted(options.signal)) return err(cancelled());
const read = readResult.value;
const filePaths = new Set(
read.entries
.filter((entry) => entry.status === "read" && entry.kind === "file")
.map((entry) => entry.path),
);
const { entries, relationships, parseFailures, limitations } =
parseReferenceSourceEntries(read, filePaths, options.signal);
if (isAborted(options.signal)) return err(cancelled());
const uniqueRelationships = deduplicateRelationships(relationships);
const uniqueFailures = [...parseFailures]
.sort((left, right) => {
const byPath = byCodePoint(left.path, right.path);
if (byPath !== 0) return byPath;
return byCodePoint(left.parser, right.parser);
})
.filter((value, index, array) => {
if (index === 0) return true;
const previous = array[index - 1];
if (previous === undefined) return true;
return value.path !== previous.path || value.parser !== previous.parser;
});
const sortedEntries = [...entries].sort((left, right) =>
byCodePoint(left.path, right.path),
);
const sortedExclusions = sortAndLimit(exclusions, limits.maxEntries);
const sortedLimitations = [...limitations].sort(byCodePoint);
const input: HistoricalSourceGraphInput = {
schema: "HistoricalSourceGraph/v1",
authority: "historical-reference",
root_alias: "$REFERENCE_ROOT",
inventory_state: deriveInventoryState({
entries: sortedEntries,
relationships: uniqueRelationships,
parse_failures: uniqueFailures,
exclusions: sortedExclusions,
limitations: sortedLimitations,
}),
entries: sortedEntries,
relationships: uniqueRelationships.sort((left, right) =>
byCodePoint(relationshipKey(left), relationshipKey(right)),
),
parse_failures: uniqueFailures,
exclusions: sortedExclusions,
languages: deriveLanguages(sortedEntries),
manifests: deriveManifests(sortedEntries),
vcs,
provenance: buildProvenance(options),
limitations: sortedLimitations,
};
try {
return ok(createHistoricalSourceGraph(input));
} catch (error) {
return err(
failure(
"parse",
error instanceof Error ? error.message : "Graph failed validation",
),
);
}
};

166
src/application/ReferenceSourceImportEntries.ts Normal file
View File

@ -0,0 +1,166 @@
import { createHash } from "node:crypto";
import { posix } from "node:path";
import {
classifyReferenceSourcePath,
detectReferenceSourceLanguage,
} from "../domain/referenceSourceClassification.js";
import type { HistoricalSourceGraphInput } from "../domain/referenceSourceGraph.js";
import { parseReferenceSourceImports } from "../domain/referenceSourceImportParsing.js";
import type {
ReferenceSourceEntry,
ReferenceSourceRead,
} from "../reference/ReferenceSourceReader.js";
import { PARSEABLE_REFERENCE_SOURCE_LANGUAGES } from "./ReferenceSourceImportTypes.js";
export interface ParsedReferenceSourceEntries {
readonly entries: HistoricalSourceGraphInput["entries"];
readonly relationships: HistoricalSourceGraphInput["relationships"];
readonly parseFailures: HistoricalSourceGraphInput["parse_failures"];
readonly limitations: string[];
}
const hashBytes = (bytes: Uint8Array): string =>
createHash("sha256").update(bytes).digest("hex");
const failedEntry = (
entry: Extract<ReferenceSourceEntry, { status: "failed" }>,
): HistoricalSourceGraphInput["entries"][number] => {
const classifications = classifyReferenceSourcePath(entry.path);
if (entry.kind === "directory")
return {
path: entry.path,
kind: "directory",
classifications,
tree_state: entry.code === "limit" ? "partial" : "unreadable",
limitations: [entry.message],
};
if (entry.kind === "symlink")
return {
path: entry.path,
kind: "symlink",
target: "<unreadable>",
target_state: "unreadable",
classifications,
limitations: [entry.message],
};
return {
path: entry.path,
kind: "file",
sha256: null,
size: entry.size ?? null,
language: null,
classifications: entry.kind === "file" ? classifications : ["unknown"],
content_state: entry.code === "limit" ? "too-large" : "unreadable",
limitations: [entry.message],
};
};
const resolveInternalSpecifier = (
fromPath: string,
specifier: string,
filePaths: ReadonlySet<string>,
): { to: string; resolution: "internal" | "unresolved" } => {
const normalized = posix.normalize(
posix.join(posix.dirname(fromPath), specifier),
);
if (normalized.startsWith("../") || normalized === "..")
return { to: normalized, resolution: "unresolved" };
const suffixes = ["", ".ts", ".js", ".mjs", ".cjs", ".tsx", ".jsx"];
const candidates = suffixes.flatMap((suffix) => [
`${normalized}${suffix}`,
`${normalized}/index${suffix}`,
]);
const match = candidates.find((candidate) => filePaths.has(candidate));
return match === undefined
? { to: normalized, resolution: "unresolved" }
: { to: match, resolution: "internal" };
};
const appendFile = (
entry: Extract<ReferenceSourceEntry, { status: "read"; kind: "file" }>,
filePaths: ReadonlySet<string>,
output: ParsedReferenceSourceEntries,
): void => {
const language = detectReferenceSourceLanguage(entry.path);
output.entries.push({
path: entry.path,
kind: "file",
sha256: hashBytes(entry.bytes),
size: entry.size,
language,
classifications: classifyReferenceSourcePath(entry.path),
content_state: "hashed",
limitations: [],
});
if (language === null || !PARSEABLE_REFERENCE_SOURCE_LANGUAGES.has(language))
return;
const parsed = parseReferenceSourceImports(entry.path, entry.bytes, language);
for (const relationship of parsed.relationships) {
if (relationship.resolution !== "internal") {
output.relationships.push(relationship);
continue;
}
const resolved = resolveInternalSpecifier(
relationship.from_path,
relationship.to,
filePaths,
);
output.relationships.push({
...relationship,
to: resolved.to,
resolution: resolved.resolution,
});
}
output.parseFailures.push(...parsed.parse_failures);
};
const appendReadEntry = (
entry: Extract<ReferenceSourceEntry, { status: "read" }>,
filePaths: ReadonlySet<string>,
output: ParsedReferenceSourceEntries,
): void => {
if (entry.kind === "file") {
appendFile(entry, filePaths, output);
return;
}
const classifications = classifyReferenceSourcePath(entry.path);
if (entry.kind === "directory") {
output.entries.push({
path: entry.path,
kind: "directory",
classifications,
tree_state: "enumerated",
limitations: [],
});
return;
}
output.entries.push({
path: entry.path,
kind: "symlink",
target: entry.target,
target_state: entry.targetState,
classifications,
limitations: [],
});
};
/** Convert bounded reader observations into graph entries and parsed edges. */
export const parseReferenceSourceEntries = (
read: ReferenceSourceRead,
filePaths: ReadonlySet<string>,
signal?: AbortSignal,
): ParsedReferenceSourceEntries => {
const output: ParsedReferenceSourceEntries = {
entries: [],
relationships: [],
parseFailures: [],
limitations: [...read.limitations],
};
for (const entry of read.entries) {
if (signal?.aborted === true) break;
if (entry.status === "failed") output.entries.push(failedEntry(entry));
else appendReadEntry(entry, filePaths, output);
}
return output;
};

122
src/application/ReferenceSourceImportPolicy.ts Normal file
View File

@ -0,0 +1,122 @@
import { readFile, realpath, stat } from "node:fs/promises";
import { join, resolve } from "node:path";
import ignore from "ignore";
import { err, ok, type Result } from "../domain/result.js";
import type { ReferenceSourceLimits } from "../reference/ReferenceSourceReader.js";
import { withinRoot } from "../reference/ReferenceSourceReaderPaths.js";
import {
DEFAULT_REFERENCE_SOURCE_IGNORE_PATTERNS,
DEFAULT_REFERENCE_SOURCE_LIMITS,
type ReferenceSourceImportError,
type ReferenceSourceImportOptions,
} from "./ReferenceSourceImportTypes.js";
/** Validated, authorized inputs ready for filesystem traversal. */
export interface PreparedReferenceSourceImport {
readonly root: string;
readonly limits: ReferenceSourceLimits;
readonly ignored: ReturnType<typeof ignore>;
readonly secrets: ReturnType<typeof ignore>;
}
const failure = (
code: ReferenceSourceImportError["code"],
message: string,
): ReferenceSourceImportError => ({
tag: "reference-source-import",
code,
message,
});
const validLimits = (limits: ReferenceSourceLimits): boolean =>
Object.values(limits).every(
(value) => Number.isSafeInteger(value) && value > 0,
);
const maximumLimits = (
options: ReferenceSourceImportOptions,
): ReferenceSourceLimits => ({
maxBytes: options.policy.maxBytes,
maxEntries: options.policy.maxEntries,
maxDepth: options.policy.maxDepth,
maxPathBytes: options.policy.maxPathBytes,
});
const clampLimits = (
requested: ReferenceSourceLimits,
maximum: ReferenceSourceLimits,
): ReferenceSourceLimits => ({
maxBytes: Math.min(requested.maxBytes, maximum.maxBytes),
maxEntries: Math.min(requested.maxEntries, maximum.maxEntries),
maxDepth: Math.min(requested.maxDepth, maximum.maxDepth),
maxPathBytes: Math.min(requested.maxPathBytes, maximum.maxPathBytes),
});
const authorizeRoot = async (
requestedRoot: string,
approvedRoots: readonly string[],
): Promise<Result<string, ReferenceSourceImportError>> => {
if (approvedRoots.length === 0)
return err(failure("policy", "No approved reference source roots"));
try {
if (!(await stat(requestedRoot)).isDirectory())
return err(
failure("invalid-root", "Reference source root is not a directory"),
);
const canonicalRoot = await realpath(resolve(requestedRoot));
for (const approvedRoot of approvedRoots) {
const canonicalApproved = await realpath(resolve(approvedRoot));
if (withinRoot(canonicalApproved, canonicalRoot))
return ok(canonicalRoot);
}
return err(
failure("policy", "Reference source root is outside approved roots"),
);
} catch {
return err(
failure("invalid-root", "Reference source root could not be resolved"),
);
}
};
const buildIgnored = async (
root: string,
excludePaths: readonly string[],
): Promise<ReturnType<typeof ignore>> => {
const ignored = ignore();
try {
ignored.add(await readFile(join(root, ".gitignore"), "utf8"));
} catch {
// Missing or unreadable ignore file does not authorize broader access.
}
ignored.add([...DEFAULT_REFERENCE_SOURCE_IGNORE_PATTERNS]);
for (const path of excludePaths) {
ignored.add(path);
ignored.add(`${path}/`);
}
return ignored;
};
/** Validate limits, authorize root, and build non-executing path filters. */
export const prepareReferenceSourceImport = async (
options: ReferenceSourceImportOptions,
): Promise<
Result<PreparedReferenceSourceImport, ReferenceSourceImportError>
> => {
const requested = options.limits ?? DEFAULT_REFERENCE_SOURCE_LIMITS;
const maximum = maximumLimits(options);
if (!validLimits(requested) || !validLimits(maximum))
return err(
failure("invalid-limits", "Import limits must be positive integers"),
);
const root = await authorizeRoot(options.root, options.policy.roots);
if (!root.ok) return root;
return ok({
root: root.value,
limits: clampLimits(requested, maximum),
ignored: await buildIgnored(root.value, options.excludePaths ?? []),
secrets: ignore().add([...options.policy.secretPatterns]),
});
};

50
src/application/ReferenceSourceImportTypes.ts Normal file
View File

@ -0,0 +1,50 @@
import type { ReferenceSourcePolicy } from "../domain/referenceSourcePolicy.js";
import type { ReferenceSourceLimits } from "../reference/ReferenceSourceReader.js";
/** Typed expected failure returned by historical-source imports. */
export interface ReferenceSourceImportError {
readonly tag: "reference-source-import";
readonly code:
| "cancelled"
| "invalid-limits"
| "invalid-root"
| "io"
| "parse"
| "policy"
| "limit";
readonly message: string;
}
/** Bounded import request paired with mandatory operator policy. */
export interface ReferenceSourceImportOptions {
readonly root: string;
readonly limits?: ReferenceSourceLimits;
readonly signal?: AbortSignal;
readonly caller: string;
readonly policy: ReferenceSourcePolicy;
readonly importer?: string;
readonly importerVersion?: string | null;
readonly excludePaths?: readonly string[];
}
export const DEFAULT_REFERENCE_SOURCE_IGNORE_PATTERNS = [
".git/",
".git/hooks/",
"node_modules/",
"dist/",
"build/",
"coverage/",
"htmlcov/",
".coverage",
"*.log",
] as const;
export const DEFAULT_REFERENCE_SOURCE_LIMITS: ReferenceSourceLimits = {
maxBytes: 16 * 1024 * 1024,
maxEntries: 10_000,
maxDepth: 32,
maxPathBytes: 4_096,
};
export const PARSEABLE_REFERENCE_SOURCE_LANGUAGES: ReadonlySet<string> =
new Set(["JavaScript", "TypeScript", "JSX", "TSX"]);

47
src/application/ReferenceSourceVcsAdapter.ts Normal file
View File

@ -0,0 +1,47 @@
import fs from "node:fs";
import { lstat } from "node:fs/promises";
import { join } from "node:path";
import { resolveRef } from "isomorphic-git";
export type ReferenceSourceVcsInfo =
| {
readonly kind: "git";
readonly head: string;
readonly dirty: boolean | null;
}
| { readonly kind: "none"; readonly head: null; readonly dirty: null }
| { readonly kind: "unknown"; readonly head: null; readonly dirty: null };
/**
* Read Git metadata for a directory using isomorphic-git.
*
* No git subprocess or network is used; only the local `.git` object store is read.
*/
export const readReferenceSourceVcs = async (
root: string,
signal?: AbortSignal,
): Promise<ReferenceSourceVcsInfo> => {
if (isAborted(signal)) return { kind: "unknown", head: null, dirty: null };
try {
await lstat(join(root, ".git"));
} catch (cause: unknown) {
return errorCode(cause) === "ENOENT"
? { kind: "none", head: null, dirty: null }
: { kind: "unknown", head: null, dirty: null };
}
try {
const head = await resolveRef({ fs, dir: root, ref: "HEAD" });
if (isAborted(signal)) return { kind: "unknown", head: null, dirty: null };
return { kind: "git", head, dirty: null };
} catch {
return { kind: "unknown", head: null, dirty: null };
}
};
const errorCode = (cause: unknown): string | undefined =>
typeof cause === "object" && cause !== null && "code" in cause
? String(cause.code)
: undefined;
const isAborted = (signal?: AbortSignal): boolean => signal?.aborted === true;

38
src/application/Setup.ts
View File

@ -12,6 +12,7 @@ import { dirname, join } from "node:path";
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import { z } from "zod";
import writeFileAtomic from "write-file-atomic";
import { PRODUCT_IDENTITY } from "../identity.js";
import { runDoctor, systemDoctorHost } from "./Doctor.js";
@ -267,7 +268,8 @@ export const configureJsonClient = async (
};
};
const installCanonicalSkill = async (
/** Transactionally install or upgrade the versioned canonical REA skill. */
export const installCanonicalSkill = async (
home: string,
): Promise<"installed" | "unchanged" | "failed"> => {
const destination = join(
@ -276,6 +278,8 @@ const installCanonicalSkill = async (
PRODUCT_IDENTITY.skillName,
"SKILL.md",
);
const backup = `${destination}.rea.backup`;
let original: string | undefined;
try {
const content = await readFile(
new URL(
@ -284,16 +288,32 @@ const installCanonicalSkill = async (
),
"utf8",
);
if (
(await readFile(destination, "utf8").catch(() => undefined)) === content
)
return "unchanged";
original = await readFile(destination, "utf8").catch(() => undefined);
if (original === content) return "unchanged";
await mkdir(dirname(destination), { recursive: true });
await writeFile(destination, content, { encoding: "utf8", mode: 0o600 });
return (await readFile(destination, "utf8")) === content
? "installed"
: "failed";
if (original !== undefined)
await writeFileAtomic(backup, original, {
encoding: "utf8",
mode: 0o600,
});
await writeFileAtomic(destination, content, {
encoding: "utf8",
mode: 0o600,
});
if ((await readFile(destination, "utf8")) !== content)
throw new Error("skill readback mismatch");
return "installed";
} catch {
try {
if (original === undefined) await rm(destination, { force: true });
else
await writeFileAtomic(destination, original, {
encoding: "utf8",
mode: 0o600,
});
} catch {
// The backup remains beside the skill for explicit operator recovery.
}
return "failed";
}
};

11
src/application/runtime.ts
View File

@ -1,6 +1,9 @@
import type { AppConfig } from "../config.js";
import { BinarySession } from "./BinarySession.js";
import { HopperProvider } from "../hopper/HopperProvider.js";
import { NativeMacOSProvider } from "../native/NativeMacOSProvider.js";
import { ArtifactProvider } from "../artifacts/ArtifactProvider.js";
import { CompositeProvider } from "./CompositeProvider.js";
import { silentLogger, type Logger } from "../logger.js";
/**
@ -12,5 +15,11 @@ export const createBinarySession = (
config: AppConfig,
logger: Logger = silentLogger,
): BinarySession => {
return new BinarySession(new HopperProvider(config, logger));
return new BinarySession(
new CompositeProvider([
new ArtifactProvider(),
new NativeMacOSProvider(),
new HopperProvider(config, logger),
]),
);
};

86
src/artifacts/ArtifactPaths.ts Normal file
View File

@ -0,0 +1,86 @@
import { Buffer } from "node:buffer";
import { posix } from "node:path";
import {
ArtifactReaderFailure,
type ArtifactLimits,
} from "./ArtifactReader.js";
const DRIVE_OR_UNC = /^(?:[A-Za-z]:|\\|\/\/)/u;
/** Normalize an untrusted logical archive path without touching filesystem. */
export const normalizeArtifactPath = (
input: string,
limits: Pick<ArtifactLimits, "maxDepth" | "maxPathBytes">,
): string => {
if (
input.includes("\0") ||
input.includes("\\") ||
input.startsWith("/") ||
DRIVE_OR_UNC.test(input)
)
throw new ArtifactReaderFailure(
"path",
"Artifact path is absolute or unsafe",
);
const normalized = input.normalize("NFC").replace(/\/+$/u, "");
const parts = normalized.split("/");
if (
normalized.length === 0 ||
parts.some((part) => part === "" || part === "." || part === "..") ||
parts.length > limits.maxDepth ||
Buffer.byteLength(normalized, "utf8") > limits.maxPathBytes ||
posix.normalize(normalized) !== normalized
)
throw new ArtifactReaderFailure("path", "Artifact path is not normalized");
return normalized;
};
/** Reject exact, Unicode-normalized, case-folded, and prefix collisions. */
export class ArtifactPathRegistry {
readonly #root: PathTrieNode = { kind: undefined, children: new Map() };
readonly #folded = new Set<string>();
add(path: string, kind: "file" | "directory" | "symlink" | "slice"): void {
const folded = path.toLocaleLowerCase("en-US");
if (this.#folded.has(folded))
throw new ArtifactReaderFailure(
"path",
`Artifact path collision: ${path}`,
);
const parts = path.split("/");
let node = this.#root;
for (const [index, part] of parts.entries()) {
if (node.kind !== undefined && node.kind !== "directory")
throw new ArtifactReaderFailure(
"path",
`Artifact prefix conflict: ${path}`,
);
let child = node.children.get(part);
if (child === undefined) {
child = { kind: undefined, children: new Map() };
node.children.set(part, child);
}
node = child;
if (index === parts.length - 1) {
if (node.kind !== undefined)
throw new ArtifactReaderFailure(
"path",
`Artifact path collision: ${path}`,
);
if (kind !== "directory" && node.children.size > 0)
throw new ArtifactReaderFailure(
"path",
`Artifact prefix conflict: ${path}`,
);
}
}
node.kind = kind;
this.#folded.add(folded);
}
}
interface PathTrieNode {
kind: "file" | "directory" | "symlink" | "slice" | undefined;
readonly children: Map<string, PathTrieNode>;
}

212
src/artifacts/ArtifactProvider.ts Normal file
View File

@ -0,0 +1,212 @@
import {
createAnalysisExecution,
type AnalysisClient,
type AnalysisOperation,
type AnalysisProvider,
type CapabilityDescriptor,
type ProviderIdentity,
} from "../application/AnalysisProvider.js";
import { inventoryArtifact } from "../application/ArtifactInventory.js";
import { extractArtifact } from "../application/ArtifactExtraction.js";
import {
ARTIFACT_TOOL_CONTRACTS,
artifactInventoryInputSchema,
artifactExtractionInputSchema,
type ArtifactToolName,
} from "../contracts/artifactToolContracts.js";
import type { BinaryTarget } from "../domain/binaryTarget.js";
import {
AnalysisCapabilityUnavailableError,
ArtifactOperationError,
type AnalysisError,
} from "../domain/errors.js";
import type { JsonValue } from "../domain/jsonValue.js";
import { err, ok } from "../domain/result.js";
import {
ArtifactReaderFailure,
type ArtifactLimits,
} from "./ArtifactReader.js";
const IDENTITY: ProviderIdentity = Object.freeze({
id: "rea-artifact-graph",
name: "REA safe artifact graph provider",
version: "1",
});
/** Read-only inventory and exclusively owned extraction provider. */
export class ArtifactProvider implements AnalysisProvider {
readonly #capabilities: readonly CapabilityDescriptor[] = Object.freeze(
ARTIFACT_TOOL_CONTRACTS.map((contract) =>
Object.freeze({
provider: IDENTITY,
operation: contract.name,
inputContractVersion: 1,
outputContractVersion: 1,
available: true as const,
reason: null,
pagination: "offset" as const,
exhaustive: false,
effects: Object.freeze({
mutatesArtifact: false,
launchesProcess: true,
mayShowUi: false,
mayAccessNetwork: false,
mayWriteFilesystem: contract.name === "extract_artifact",
changesPermissions: false,
requiresRoot: false,
}),
limits: Object.freeze({
maxResults: 500,
maxPayloadBytes: 4 * 1024 * 1024,
timeoutMs: 120_000,
}),
limitations: Object.freeze([
"DMG and PKG child inventory requires a future native read-only adapter.",
"Nested containers are recorded but not recursively expanded implicitly.",
]),
}),
),
);
identity(): ProviderIdentity {
return IDENTITY;
}
capabilities(): readonly CapabilityDescriptor[] {
return this.#capabilities;
}
createClient(target: BinaryTarget): AnalysisClient {
return new ArtifactClient(target);
}
}
class ArtifactClient implements AnalysisClient {
constructor(private readonly target: BinaryTarget) {}
async execute(
operation: AnalysisOperation,
parameters: Readonly<Record<string, JsonValue>>,
options?: { readonly signal?: AbortSignal },
) {
if (operation === "health")
return ok(createAnalysisExecution(null, IDENTITY));
if (!isArtifactOperation(operation))
return err(
new AnalysisCapabilityUnavailableError(
IDENTITY.id,
operation,
"Operation is not implemented by artifact graph provider.",
),
);
try {
if (operation === "extract_artifact") {
const parsed = artifactExtractionInputSchema.parse(parameters);
const result = await extractArtifact(
{
inputPath: this.target.sourcePath ?? this.target.path,
inputFormat: this.target.format,
outputRoot: parsed.output_root,
occurrenceIds: parsed.occurrence_ids,
offset: parsed.offset,
limit: parsed.limit,
limits: limitsFrom(parsed),
},
options?.signal,
);
return ok(
createAnalysisExecution(result, IDENTITY, {
rawResult: null,
limitations: result.limitations,
subject: subjectFor(
this.target.sourcePath ?? this.target.path,
result.manifest,
),
locations: result.artifacts.items.map(
({ relative_path: path }) => ({
kind: "artifact-path" as const,
path,
}),
),
}),
);
}
const parsed = artifactInventoryInputSchema.parse(parameters);
const result = await inventoryArtifact(
this.target.sourcePath ?? this.target.path,
limitsFrom(parsed),
{
nodeOffset: parsed.node_offset,
nodeLimit: parsed.node_limit,
occurrenceOffset: parsed.occurrence_offset,
occurrenceLimit: parsed.occurrence_limit,
edgeOffset: parsed.edge_offset,
edgeLimit: parsed.edge_limit,
},
options?.signal,
);
return ok(
createAnalysisExecution(result, IDENTITY, {
rawResult: null,
limitations: result.limitations,
subject: subjectFor(
this.target.sourcePath ?? this.target.path,
result.manifest,
),
locations: result.occurrences.items.map(({ logical_path: path }) => ({
kind: "artifact-path" as const,
path,
})),
}),
);
} catch (cause: unknown) {
return err(translateFailure(operation, cause));
}
}
close(): Promise<void> {
return Promise.resolve();
}
}
const limitsFrom = (input: {
readonly max_entries: number;
readonly max_total_bytes: number;
readonly max_entry_bytes: number;
readonly max_compression_ratio: number;
readonly max_depth: number;
readonly max_path_bytes: number;
}): ArtifactLimits => ({
maxEntries: input.max_entries,
maxTotalBytes: input.max_total_bytes,
maxEntryBytes: input.max_entry_bytes,
maxCompressionRatio: input.max_compression_ratio,
maxDepth: input.max_depth,
maxPathBytes: input.max_path_bytes,
});
const isArtifactOperation = (
operation: AnalysisOperation,
): operation is ArtifactToolName =>
ARTIFACT_TOOL_CONTRACTS.some(({ name }) => name === operation);
const translateFailure = (
operation: ArtifactToolName,
cause: unknown,
): AnalysisError => {
if (cause instanceof ArtifactReaderFailure)
return new ArtifactOperationError(operation, cause.reason);
return new ArtifactOperationError(operation, "io");
};
const subjectFor = (
path: string,
manifest: {
readonly root_sha256: string;
readonly root_format: import("../domain/artifactGraph.js").ArtifactNode["format"];
},
) => ({
path,
sha256: manifest.root_sha256,
format: manifest.root_format,
});

58
src/artifacts/ArtifactReader.ts Normal file
View File

@ -0,0 +1,58 @@
import type { Readable } from "node:stream";
import type { ArtifactCommand } from "../domain/artifactGraph.js";
/** Archive-neutral entry metadata. Reader adapters never choose output paths. */
export interface ArtifactEntry {
readonly path: string;
readonly kind: "file" | "directory" | "symlink" | "slice";
readonly declaredSize: number | null;
readonly compressedSize: number | null;
readonly executable: boolean;
readonly encrypted: boolean;
readonly byteOffset: number | null;
readonly declaredSha256: string | null;
readonly limitations: readonly string[];
readonly adapterKey: string;
/** Filesystem identity captured during traversal, when supplied by an adapter. */
readonly sourceIdentity?: {
readonly device: number;
readonly inode: number;
};
}
/** Read-only adapter over one directory, archive, or virtual container. */
export interface ArtifactReader {
readonly format: "directory" | "zip" | "ipa" | "apk" | "asar" | "file";
entries(signal?: AbortSignal): AsyncIterable<ArtifactEntry>;
open(entry: ArtifactEntry, signal?: AbortSignal): Promise<Readable>;
provenance(): readonly ArtifactCommand[];
close(): Promise<void>;
}
/** Hard traversal limits shared by every reader and extraction operation. */
export interface ArtifactLimits {
readonly maxEntries: number;
readonly maxTotalBytes: number;
readonly maxEntryBytes: number;
readonly maxCompressionRatio: number;
readonly maxDepth: number;
readonly maxPathBytes: number;
}
/** Typed adapter failure translated at provider boundary. */
export class ArtifactReaderFailure extends Error {
constructor(
readonly reason:
| "cancelled"
| "format"
| "integrity"
| "limit"
| "path"
| "unavailable",
message: string,
options?: ErrorOptions,
) {
super(message, options);
this.name = "ArtifactReaderFailure";
}
}

80
src/artifacts/AsarArtifactReader.ts Normal file
View File

@ -0,0 +1,80 @@
import { Readable } from "node:stream";
import { extractFile, listPackage, statFile } from "@electron/asar";
import {
ArtifactReaderFailure,
type ArtifactEntry,
type ArtifactReader,
} from "./ArtifactReader.js";
/** Official Electron ASAR adapter. Individual files remain caller-bounded. */
export class AsarArtifactReader implements ArtifactReader {
readonly format = "asar" as const;
constructor(private readonly path: string) {}
async *entries(signal?: AbortSignal): AsyncIterable<ArtifactEntry> {
const paths = listPackage(this.path, { isPack: false }).sort(
(left, right) => left.localeCompare(right, "en"),
);
for (const listed of paths) {
abortIfNeeded(signal);
const path = listed.replace(/^\/+|\/+$/gu, "");
if (path.length === 0) continue;
const metadata = statFile(this.path, path, false);
const kind =
"files" in metadata
? "directory"
: "link" in metadata
? "symlink"
: "file";
yield {
path,
kind,
declaredSize: "size" in metadata ? metadata.size : null,
compressedSize: null,
executable: "executable" in metadata && metadata.executable,
encrypted: false,
byteOffset: null,
declaredSha256:
"integrity" in metadata &&
metadata.integrity.algorithm === "SHA256" &&
/^[a-f0-9]{64}$/u.test(metadata.integrity.hash)
? metadata.integrity.hash
: null,
limitations:
kind === "symlink"
? ["ASAR symlink target was not followed or disclosed."]
: [
"Official ASAR extraction buffers each individually bounded file.",
],
adapterKey: path,
};
}
}
open(entry: ArtifactEntry, signal?: AbortSignal): Promise<Readable> {
abortIfNeeded(signal);
if (entry.kind !== "file")
return Promise.reject(
new ArtifactReaderFailure("format", "ASAR entry is not a regular file"),
);
return Promise.resolve(
Readable.from(extractFile(this.path, entry.adapterKey, false)),
);
}
close(): Promise<void> {
return Promise.resolve();
}
provenance(): readonly [] {
return [];
}
}
const abortIfNeeded = (signal?: AbortSignal): void => {
if (signal?.aborted === true)
throw new ArtifactReaderFailure("cancelled", "ASAR operation cancelled");
};

139
src/artifacts/DirectoryArtifactReader.ts Normal file
View File

@ -0,0 +1,139 @@
import { constants } from "node:fs";
import { lstat, open, opendir, realpath } from "node:fs/promises";
import { isAbsolute, join, relative, sep } from "node:path";
import type { Readable } from "node:stream";
import {
ArtifactReaderFailure,
type ArtifactEntry,
type ArtifactReader,
} from "./ArtifactReader.js";
/** Deterministic, non-symlink-following directory reader. */
export class DirectoryArtifactReader implements ArtifactReader {
readonly format = "directory" as const;
readonly #rootPromise: Promise<string>;
constructor(root: string) {
this.#rootPromise = realpath(root);
}
async *entries(signal?: AbortSignal): AsyncIterable<ArtifactEntry> {
const root = await this.#rootPromise;
const pending = [root];
while (pending.length > 0) {
abortIfNeeded(signal);
const directory = pending.pop();
if (directory === undefined) break;
await assertContainedDirectory(root, directory);
const children: string[] = [];
const handle = await opendir(directory);
try {
for await (const child of handle) children.push(child.name);
} finally {
await handle.close().catch(() => undefined);
}
await assertContainedDirectory(root, directory);
children.sort((left, right) => left.localeCompare(right, "en"));
const directories: string[] = [];
for (const name of children) {
abortIfNeeded(signal);
const absolute = join(directory, name);
const metadata = await lstat(absolute);
const path = relative(root, absolute).split(sep).join("/");
const kind = metadata.isSymbolicLink()
? "symlink"
: metadata.isDirectory()
? "directory"
: metadata.isFile()
? "file"
: undefined;
if (kind === undefined) continue;
if (kind === "directory") directories.push(absolute);
yield {
path,
kind,
declaredSize: kind === "file" ? metadata.size : null,
compressedSize: null,
executable: (metadata.mode & 0o111) !== 0,
encrypted: false,
byteOffset: null,
declaredSha256: null,
limitations:
kind === "symlink"
? ["Symlink target was not followed or disclosed."]
: [],
adapterKey: absolute,
...(kind === "file"
? { sourceIdentity: { device: metadata.dev, inode: metadata.ino } }
: {}),
};
}
directories.reverse();
pending.push(...directories);
}
}
async open(entry: ArtifactEntry, signal?: AbortSignal): Promise<Readable> {
abortIfNeeded(signal);
if (entry.kind !== "file")
throw new ArtifactReaderFailure(
"format",
"Only regular files can be opened",
);
const handle = await open(
entry.adapterKey,
constants.O_RDONLY | constants.O_NOFOLLOW,
);
const observed = await handle.stat();
if (
!observed.isFile() ||
entry.sourceIdentity === undefined ||
observed.dev !== entry.sourceIdentity.device ||
observed.ino !== entry.sourceIdentity.inode
) {
await handle.close();
throw new ArtifactReaderFailure(
"integrity",
`Directory entry changed before open: ${entry.path}`,
);
}
return handle.createReadStream({ autoClose: true });
}
close(): Promise<void> {
return Promise.resolve();
}
provenance(): readonly [] {
return [];
}
}
const abortIfNeeded = (signal?: AbortSignal): void => {
if (signal?.aborted === true)
throw new ArtifactReaderFailure(
"cancelled",
"Artifact traversal cancelled",
);
};
const assertContainedDirectory = async (
root: string,
directory: string,
): Promise<void> => {
const canonical = await realpath(directory);
const fromRoot = relative(root, canonical);
const metadata = await lstat(directory);
if (
metadata.isSymbolicLink() ||
!metadata.isDirectory() ||
fromRoot === ".." ||
fromRoot.startsWith(`..${sep}`) ||
isAbsolute(fromRoot)
)
throw new ArtifactReaderFailure(
"path",
"Directory traversal escaped or changed its artifact root",
);
};

102
src/artifacts/MachOSliceArtifactReader.ts Normal file
View File

@ -0,0 +1,102 @@
import { createReadStream } from "node:fs";
import type { Readable } from "node:stream";
import type { ArtifactCommand } from "../domain/artifactGraph.js";
import {
XcrunCommandRunner,
type NativeCommandRunner,
} from "../native/CommandRunner.js";
import { parseLipoArchitectures } from "../native/parsers/lipo.js";
import {
ArtifactReaderFailure,
type ArtifactEntry,
type ArtifactReader,
} from "./ArtifactReader.js";
/** Read-only universal Mach-O slice reader backed by native lipo metadata. */
export class MachOSliceArtifactReader implements ArtifactReader {
readonly format = "file" as const;
#command: ArtifactCommand | undefined;
constructor(
private readonly path: string,
private readonly runner: NativeCommandRunner = new XcrunCommandRunner(),
) {}
async *entries(signal?: AbortSignal): AsyncIterable<ArtifactEntry> {
const captured = await this.runner.run(
"lipo",
["-detailed_info", this.path],
{
...(signal === undefined ? {} : { signal }),
timeoutMs: 30_000,
maxOutputBytes: 4 * 1024 * 1024,
},
);
if (!captured.ok)
throw new ArtifactReaderFailure(
captured.error.reason === "cancelled" ? "cancelled" : "unavailable",
"lipo could not enumerate universal Mach-O slices",
{ cause: captured.error },
);
this.#command = {
tool: captured.value.tool,
arguments: ["-detailed_info", "$ARTIFACT"],
tool_version: captured.value.toolVersion,
executable_sha256: captured.value.executableSha256,
exit_code: captured.value.exitCode,
effects: ["read"],
};
for (const architecture of parseLipoArchitectures(captured.value.stdout)) {
if (architecture.file_offset === null || architecture.size === null)
throw new ArtifactReaderFailure(
"integrity",
"lipo omitted a universal slice byte range",
);
yield {
path: `slices/${architecture.name}`,
kind: "slice",
declaredSize: architecture.size,
compressedSize: null,
executable: true,
encrypted: false,
byteOffset: architecture.file_offset,
declaredSha256: null,
limitations: [],
adapterKey: `${String(architecture.file_offset)}:${String(architecture.size)}`,
};
}
}
open(entry: ArtifactEntry, signal?: AbortSignal): Promise<Readable> {
if (signal?.aborted === true)
return Promise.reject(
new ArtifactReaderFailure("cancelled", "Mach-O slice read cancelled"),
);
const [offsetText, sizeText] = entry.adapterKey.split(":");
const offset = Number(offsetText);
const size = Number(sizeText);
if (
!Number.isSafeInteger(offset) ||
!Number.isSafeInteger(size) ||
size <= 0
)
return Promise.reject(
new ArtifactReaderFailure(
"integrity",
"Invalid Mach-O slice byte range",
),
);
return Promise.resolve(
createReadStream(this.path, { start: offset, end: offset + size - 1 }),
);
}
provenance(): readonly ArtifactCommand[] {
return this.#command === undefined ? [] : [this.#command];
}
close(): Promise<void> {
return Promise.resolve();
}
}

321
src/artifacts/SafeOutputTree.ts Normal file
View File

@ -0,0 +1,321 @@
import { createHash } from "node:crypto";
import { constants } from "node:fs";
import { lstat, mkdir, open, realpath, rm } from "node:fs/promises";
import { basename, dirname, isAbsolute, join, resolve } from "node:path";
import type { Readable } from "node:stream";
import { streamChunkToBuffer } from "./StreamBytes.js";
import {
ArtifactPathRegistry,
normalizeArtifactPath,
} from "./ArtifactPaths.js";
import {
ArtifactReaderFailure,
type ArtifactLimits,
} from "./ArtifactReader.js";
/** One file durably written to an operation-owned output tree. */
export interface SafeOutputFile {
readonly relativePath: string;
readonly sha256: string;
readonly bytesWritten: number;
}
/** Cleanup facts established while rolling back an uncommitted tree. */
export interface SafeOutputCleanup {
readonly attempted: boolean;
readonly verified: boolean;
readonly residualPaths: readonly string[];
}
/** Symlink-resistant materialization in an exclusively owned, initially absent tree. */
export class SafeOutputTree {
readonly #registry = new ArtifactPathRegistry();
readonly #limits: ArtifactLimits;
readonly #outputRoot: string;
readonly #stagingRoot: string;
#totalBytes = 0;
#published = false;
#cleanup: SafeOutputCleanup = {
attempted: false,
verified: false,
residualPaths: [],
};
private constructor(
outputRoot: string,
stagingRoot: string,
limits: ArtifactLimits,
) {
this.#outputRoot = outputRoot;
this.#stagingRoot = stagingRoot;
this.#limits = limits;
}
/** Exclusively create the absent destination as this operation's owned tree. */
static async create(
outputRoot: string,
limits: ArtifactLimits,
): Promise<SafeOutputTree> {
if (!isAbsolute(outputRoot))
throw new ArtifactReaderFailure(
"path",
"Extraction output root must be absolute",
);
const requested = resolve(outputRoot);
const name = basename(requested);
if (name === "." || name === "..")
throw new ArtifactReaderFailure("path", "Invalid extraction output root");
const parent = await realpath(dirname(requested)).catch(
(cause: unknown) => {
throw new ArtifactReaderFailure(
"unavailable",
"Extraction output parent is unavailable",
{ cause },
);
},
);
const canonicalOutput = join(parent, name);
await mkdir(canonicalOutput, { mode: 0o700 }).catch((cause: unknown) => {
if (isAlreadyExists(cause))
throw new ArtifactReaderFailure(
"path",
"Extraction output root already exists",
{ cause },
);
throw cause;
});
try {
const stagingHandle = await open(
canonicalOutput,
constants.O_RDONLY | constants.O_DIRECTORY,
);
try {
await stagingHandle.chmod(0o700);
} finally {
await stagingHandle.close();
}
return new SafeOutputTree(canonicalOutput, canonicalOutput, limits);
} catch (cause: unknown) {
await rm(canonicalOutput, { recursive: true, force: true });
throw cause;
}
}
get outputRoot(): string {
return this.#outputRoot;
}
get cleanup(): SafeOutputCleanup {
return this.#cleanup;
}
/** Stream one selected regular file with exact byte and digest verification. */
async write(
relativePath: string,
source: Readable,
expectedSha256: string,
signal?: AbortSignal,
): Promise<SafeOutputFile> {
this.#assertWritable();
const path = normalizeArtifactPath(relativePath, this.#limits);
this.#registry.add(path, "file");
const destination = await this.#prepareParent(path);
const handle = await open(
destination,
constants.O_CREAT |
constants.O_EXCL |
constants.O_WRONLY |
constants.O_NOFOLLOW,
0o600,
).catch((cause: unknown) => {
throw new ArtifactReaderFailure(
"path",
`Could not exclusively create extraction path: ${path}`,
{ cause },
);
});
const hash = createHash("sha256");
let bytes = 0;
try {
for await (const raw of source) {
abortIfNeeded(signal);
const chunk = streamChunkToBuffer(raw);
bytes += chunk.length;
if (bytes > this.#limits.maxEntryBytes)
throw new ArtifactReaderFailure(
"limit",
`Observed entry bytes exceed limit: ${path}`,
);
if (this.#totalBytes + bytes > this.#limits.maxTotalBytes)
throw new ArtifactReaderFailure(
"limit",
"Observed extraction bytes exceed cumulative limit",
);
hash.update(chunk);
await writeAll(handle, chunk);
}
const sha256 = hash.digest("hex");
if (sha256 !== expectedSha256)
throw new ArtifactReaderFailure(
"integrity",
`Extracted content disagrees with inventory: ${path}`,
);
await handle.sync();
await handle.close();
const readback = await hashFile(destination, bytes, signal);
if (readback.sha256 !== sha256 || readback.bytes !== bytes)
throw new ArtifactReaderFailure(
"integrity",
`Durable readback verification failed: ${path}`,
);
this.#totalBytes += bytes;
return { relativePath: path, sha256, bytesWritten: bytes };
} catch (cause: unknown) {
await handle.close().catch(() => undefined);
throw cause;
}
}
/** Sync the owned output tree and prevent further writes through this instance. */
async commit(): Promise<void> {
this.#assertWritable();
const parent = await open(
dirname(this.#outputRoot),
constants.O_RDONLY | constants.O_DIRECTORY,
);
const staging = await open(
this.#stagingRoot,
constants.O_RDONLY | constants.O_DIRECTORY,
);
try {
await staging.sync();
await parent.sync();
this.#published = true;
} catch (cause: unknown) {
throw new ArtifactReaderFailure(
"path",
"Could not durably sync extraction output",
{ cause },
);
} finally {
await Promise.allSettled([parent.close(), staging.close()]);
}
}
/** Remove only this operation's unsealed tree and verify absence. */
async rollback(): Promise<SafeOutputCleanup> {
if (this.#published) return this.#cleanup;
await rm(this.#stagingRoot, { recursive: true, force: true });
const absent = await isAbsent(this.#stagingRoot);
this.#cleanup = {
attempted: true,
verified: absent,
residualPaths: absent ? [] : [basename(this.#stagingRoot)],
};
if (!absent)
throw new ArtifactReaderFailure(
"integrity",
"Extraction output cleanup could not be verified",
);
return this.#cleanup;
}
async #prepareParent(relativePath: string): Promise<string> {
const parts = relativePath.split("/");
const fileName = parts.pop();
if (fileName === undefined)
throw new ArtifactReaderFailure("path", "Invalid extraction path");
let current = this.#stagingRoot;
for (const part of parts) {
current = join(current, part);
await mkdir(current, { mode: 0o700 }).catch((cause: unknown) => {
if (!isAlreadyExists(cause)) throw cause;
});
const metadata = await lstat(current);
if (!metadata.isDirectory() || metadata.isSymbolicLink())
throw new ArtifactReaderFailure(
"path",
`Unsafe extraction parent: ${relativePath}`,
);
}
return join(current, fileName);
}
#assertWritable(): void {
if (this.#published)
throw new ArtifactReaderFailure(
"integrity",
"Extraction tree is already committed",
);
}
}
const hashFile = async (
path: string,
maximum: number,
signal?: AbortSignal,
): Promise<{ readonly sha256: string; readonly bytes: number }> => {
const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW);
const hash = createHash("sha256");
let bytes = 0;
try {
for await (const raw of handle.createReadStream({ autoClose: false })) {
abortIfNeeded(signal);
const chunk = streamChunkToBuffer(raw);
bytes += chunk.length;
if (bytes > maximum)
throw new ArtifactReaderFailure(
"integrity",
"Readback exceeded the bytes written",
);
hash.update(chunk);
}
} finally {
await handle.close();
}
return { sha256: hash.digest("hex"), bytes };
};
const writeAll = async (
handle: Awaited<ReturnType<typeof open>>,
chunk: Buffer,
): Promise<void> => {
let offset = 0;
while (offset < chunk.length) {
const { bytesWritten } = await handle.write(
chunk,
offset,
chunk.length - offset,
);
if (bytesWritten === 0)
throw new ArtifactReaderFailure(
"unavailable",
"Extraction output stopped accepting bytes",
);
offset += bytesWritten;
}
};
const isAbsent = async (path: string): Promise<boolean> =>
lstat(path).then(
() => false,
(cause: unknown) => {
if (isNotFound(cause)) return true;
throw cause;
},
);
const isNotFound = (cause: unknown): boolean =>
cause instanceof Error && "code" in cause && cause.code === "ENOENT";
const isAlreadyExists = (cause: unknown): boolean =>
cause instanceof Error && "code" in cause && cause.code === "EEXIST";
const abortIfNeeded = (signal?: AbortSignal): void => {
if (signal?.aborted === true)
throw new ArtifactReaderFailure(
"cancelled",
"Artifact extraction cancelled",
);
};

7
src/artifacts/StreamBytes.ts Normal file
View File

@ -0,0 +1,7 @@
/** Parse a Node readable chunk without unchecked coercion. */
export const streamChunkToBuffer = (value: unknown): Buffer => {
if (Buffer.isBuffer(value)) return value;
if (value instanceof Uint8Array) return Buffer.from(value);
if (typeof value === "string") return Buffer.from(value);
throw new TypeError("Readable stream emitted a non-byte chunk");
};

138
src/artifacts/ZipArtifactReader.ts Normal file
View File

@ -0,0 +1,138 @@
import { open, type FileHandle } from "node:fs/promises";
import { PassThrough, type Readable } from "node:stream";
import { once } from "node:events";
import { Reader, ZipReader, type Entry, type FileEntry } from "@zip.js/zip.js";
import {
ArtifactReaderFailure,
type ArtifactEntry,
type ArtifactReader,
} from "./ArtifactReader.js";
class NodeFileReader extends Reader<string> {
#handle: FileHandle | undefined;
constructor(private readonly path: string) {
super(path);
}
override async init(): Promise<void> {
this.#handle = await open(this.path, "r");
this.size = (await this.#handle.stat()).size;
}
override async readUint8Array(
index: number,
length: number,
): Promise<Uint8Array> {
const handle = this.#handle;
if (handle === undefined) throw new Error("ZIP reader is not initialized");
const bytes = Buffer.alloc(Math.min(length, this.size - index));
const read = await handle.read(bytes, 0, bytes.length, index);
return bytes.subarray(0, read.bytesRead);
}
async closeHandle(): Promise<void> {
await this.#handle?.close();
this.#handle = undefined;
}
}
/** Lazy Zip64-capable reader with CRC and overlap verification on every read. */
export class ZipArtifactReader implements ArtifactReader {
readonly format: "zip" | "ipa" | "apk";
readonly #source: NodeFileReader;
readonly #reader: ZipReader<string>;
readonly #entries = new Map<string, Entry>();
constructor(path: string, format: "zip" | "ipa" | "apk") {
this.format = format;
this.#source = new NodeFileReader(path);
this.#reader = new ZipReader(this.#source, {
checkSignature: true,
checkOverlappingEntry: true,
});
}
async *entries(signal?: AbortSignal): AsyncIterable<ArtifactEntry> {
for await (const entry of this.#reader.getEntriesGenerator()) {
abortIfNeeded(signal);
this.#entries.set(entry.filename, entry);
const symlink = isSymlink(entry);
yield {
path: entry.filename,
kind: entry.directory ? "directory" : symlink ? "symlink" : "file",
declaredSize: entry.directory ? null : entry.uncompressedSize,
compressedSize: entry.directory ? null : entry.compressedSize,
executable: entry.executable,
encrypted: entry.encrypted,
byteOffset: null,
declaredSha256: null,
limitations: symlink ? ["Archive symlink target was not read."] : [],
adapterKey: entry.filename,
};
}
}
open(entry: ArtifactEntry, signal?: AbortSignal): Promise<Readable> {
abortIfNeeded(signal);
const stored = this.#entries.get(entry.adapterKey);
if (stored === undefined || stored.directory || isSymlink(stored))
return Promise.reject(
new ArtifactReaderFailure("format", "ZIP entry is not a regular file"),
);
if (stored.encrypted)
return Promise.reject(
new ArtifactReaderFailure(
"unavailable",
"Encrypted ZIP entry is unsupported",
),
);
return Promise.resolve(extractStream(stored, signal));
}
async close(): Promise<void> {
await this.#reader.close().catch(() => undefined);
await this.#source.closeHandle();
}
provenance(): readonly [] {
return [];
}
}
const extractStream = (entry: FileEntry, signal?: AbortSignal): Readable => {
const output = new PassThrough();
const writable = new WritableStream<Uint8Array>({
write: async (chunk) => {
abortIfNeeded(signal);
if (!output.write(Buffer.from(chunk))) await once(output, "drain");
},
close: () => {
output.end();
},
abort: (reason) => {
output.destroy(toError(reason));
},
});
void entry
.getData(writable, {
checkSignature: true,
checkOverlappingEntry: true,
onprogress: () => abortIfNeeded(signal),
})
.catch((cause: unknown) => output.destroy(toError(cause)));
return output;
};
const isSymlink = (entry: Entry): boolean =>
entry.unixMode !== undefined && (entry.unixMode & 0o170000) === 0o120000;
const abortIfNeeded = (signal?: AbortSignal): void => {
if (signal?.aborted === true)
throw new ArtifactReaderFailure("cancelled", "ZIP operation cancelled");
};
const toError = (cause: unknown): Error =>
cause instanceof Error ? cause : new Error("ZIP extraction failed");

205
src/cli.ts
View File

@ -2,10 +2,19 @@ import { Cli, z } from "incur";
import { fileURLToPath } from "node:url";
import { runDoctor } from "./application/Doctor.js";
import { runDirectAnalysis } from "./application/DirectAnalysis.js";
import {
runDirectAnalysis,
runProviderAnalysis,
} from "./application/DirectAnalysis.js";
import { runSetup } from "./application/Setup.js";
import { PRODUCT_IDENTITY } from "./identity.js";
import { createLogger, parseLogLevel, type Logger } from "./logger.js";
import { parseConfig } from "./config.js";
import {
exportEvidenceBundleCommand,
importEvidenceBundleCommand,
} from "./application/EvidenceBundleCommands.js";
import { importReferenceSource } from "./application/ReferenceSourceImport.js";
/**
* Build the one-shot Incur CLI without starting Hopper at import time.
@ -85,9 +94,203 @@ export const createCli = (): ReturnType<typeof Cli.create> => {
),
),
});
registerNativeCommands(cli, logger);
registerArtifactCommands(cli, logger);
registerEvidenceCommands(cli, logger);
registerReferenceSourceCommand(cli, logger);
return cli;
};
const registerReferenceSourceCommand = (
cli: ReturnType<typeof Cli.create>,
logger: Logger,
): void => {
cli.command("import-reference-source", {
description: "Import a bounded source tree as historical reference only",
args: z.object({
root: z
.string()
.describe("Source root allowed by REA_REFERENCE_ROOTS_JSON"),
}),
run: ({ args }) =>
logCliCommand(logger, "import-reference-source", async () => {
const config = parseConfig(process.env);
if (!config.ok)
return { error: config.error._tag, message: config.error.message };
const imported = await importReferenceSource({
root: args.root,
caller: "rea-cli",
policy: config.value.referenceSourcePolicy,
importer: PRODUCT_IDENTITY.packageName,
importerVersion: null,
});
return imported.ok
? imported.value
: { error: imported.error.code, message: imported.error.message };
}),
});
};
const registerArtifactCommands = (
cli: ReturnType<typeof Cli.create>,
logger: Logger,
): void => {
cli.command("inventory-artifact", {
description: "Build a bounded deterministic artifact graph",
args: z.object({
path: z.string().describe("Application or package path"),
}),
options: z.object({
offset: z.number().int().min(0).default(0),
limit: z.number().int().min(1).max(500).default(100),
}),
run: ({ args, options }) =>
logCliCommand(logger, "inventory-artifact", () =>
runProviderAnalysis(
args.path,
"inventory_artifact",
{
node_offset: options.offset,
node_limit: options.limit,
occurrence_offset: options.offset,
occurrence_limit: options.limit,
edge_offset: options.offset,
edge_limit: options.limit,
},
logger,
),
),
});
cli.command("extract-artifact", {
description: "Extract explicitly selected artifact occurrences safely",
args: z.object({
path: z.string().describe("Application or package path"),
outputRoot: z.string().describe("Absent absolute output root"),
occurrenceIds: z
.array(z.string().regex(/^occ_[a-f0-9]{64}$/u))
.min(1)
.max(500),
}),
alias: { outputRoot: "output-root", occurrenceIds: "occurrence-ids" },
run: ({ args }) =>
logCliCommand(logger, "extract-artifact", () =>
runProviderAnalysis(
args.path,
"extract_artifact",
{
approved: true,
output_root: args.outputRoot,
occurrence_ids: args.occurrenceIds,
},
logger,
),
),
});
};
const registerNativeCommands = (
cli: ReturnType<typeof Cli.create>,
logger: Logger,
): void => {
for (const [command, tool] of [
["inspect-macho", "inspect_macho"],
["inspect-signature", "inspect_signature"],
["list-architectures", "list_architectures"],
] as const) {
cli.command(command, {
description: `Run ${tool} without launching Hopper`,
args: z.object({ path: z.string().describe("Mach-O or app path") }),
run: ({ args }) =>
logCliCommand(logger, command, () =>
runProviderAnalysis(args.path, tool, {}, logger),
),
});
}
cli.command("inspect-plist", {
description: "Parse app plist metadata without launching Hopper",
args: z.object({ path: z.string().describe("App or Mach-O path") }),
options: z.object({
relativePath: z.string().default("Contents/Info.plist"),
}),
alias: { relativePath: "relative-path" },
run: ({ args, options }) =>
logCliCommand(logger, "inspect-plist", () =>
runProviderAnalysis(
args.path,
"inspect_plist",
{ relative_path: options.relativePath },
logger,
),
),
});
cli.command("demangle-swift", {
description: "Demangle a bounded Swift symbol batch without Hopper",
args: z.object({
path: z.string().describe("Artifact path used for evidence identity"),
symbols: z.array(z.string().min(1)).min(1).max(500),
}),
run: ({ args }) =>
logCliCommand(logger, "demangle-swift", () =>
runProviderAnalysis(
args.path,
"demangle_swift",
{ symbols: args.symbols },
logger,
),
),
});
};
const registerEvidenceCommands = (
cli: ReturnType<typeof Cli.create>,
logger: Logger,
): void => {
cli.command("evidence-import", {
description: "Validate and import a bounded local Evidence v2 bundle",
args: z.object({
path: z.string().describe("Evidence bundle JSON path"),
}),
run: ({ args }) =>
logCliCommand(logger, "evidence-import", async () => {
const config = parseConfig(process.env);
if (!config.ok)
return { error: config.error._tag, message: config.error.message };
const imported = await importEvidenceBundleCommand(
args.path,
config.value.evidenceFilePolicy,
);
return imported.ok
? imported.value
: { error: imported.error._tag, message: imported.error.message };
}),
});
cli.command("evidence-export", {
description: "Validate and atomically export canonical Evidence v2 JSON",
args: z.object({
source: z.string().describe("Existing evidence bundle JSON path"),
output: z.string().describe("Canonical output JSON path"),
}),
options: z.object({
overwrite: z.boolean().default(false).describe("Replace output file"),
}),
run: ({ args, options }) =>
logCliCommand(logger, "evidence-export", async () => {
const config = parseConfig(process.env);
if (!config.ok)
return { error: config.error._tag, message: config.error.message };
const exported = await exportEvidenceBundleCommand(
args.source,
args.output,
options.overwrite,
config.value.evidenceFilePolicy,
);
return exported.ok
? exported.value
: { error: exported.error._tag, message: exported.error.message };
}),
});
};
const logCliCommand = async <Value>(
logger: Logger,
command: string,

42
src/config.ts
View File

@ -4,6 +4,8 @@ import { ConfigurationError } from "./domain/errors.js";
import { err, ok, type Result } from "./domain/result.js";
import type { LogLevel } from "./logger.js";
import type { ProcessExecutionPolicy } from "./domain/processCapture.js";
import type { EvidenceFilePolicy } from "./domain/evidenceBundle.js";
import type { ReferenceSourcePolicy } from "./domain/referenceSourcePolicy.js";
const DEFAULT_HOPPER_LAUNCHER_PATH =
"/Applications/Hopper Disassembler.app/Contents/MacOS/hopper";
@ -15,6 +17,8 @@ export interface AppConfig {
readonly hopperLoaderArgs: readonly string[];
readonly logLevel: LogLevel;
readonly processExecutionPolicy: ProcessExecutionPolicy;
readonly evidenceFilePolicy: EvidenceFilePolicy;
readonly referenceSourcePolicy: ReferenceSourcePolicy;
}
const environmentSchema = z.object({
@ -26,9 +30,15 @@ const environmentSchema = z.object({
.enum(["trace", "debug", "info", "warn", "error", "fatal", "silent"])
.default("info"),
REA_PROCESS_CAPTURE_ENABLED: z.enum(["true", "false"]).default("false"),
REA_PROCESS_ALLOW_EXTERNAL_NETWORK: z
.enum(["true", "false"])
.default("false"),
REA_PROCESS_EXECUTABLE_ROOTS_JSON: z.string().default("[]"),
REA_PROCESS_WORKING_ROOTS_JSON: z.string().default("[]"),
REA_PROCESS_ALLOWED_ENV_JSON: z.string().default("[]"),
REA_EVIDENCE_ROOTS_JSON: z.string().default("[]"),
REA_REFERENCE_ROOTS_JSON: z.string().default("[]"),
REA_REFERENCE_SECRET_PATTERNS_JSON: z.string().default("[]"),
});
const parseStringArray = (
@ -106,6 +116,21 @@ export const parseConfig = (
"REA_PROCESS_ALLOWED_ENV_JSON",
);
if (!allowedEnvironment.ok) return allowedEnvironment;
const evidenceRoots = parseStringArray(
parsedEnvironment.data.REA_EVIDENCE_ROOTS_JSON,
"REA_EVIDENCE_ROOTS_JSON",
);
if (!evidenceRoots.ok) return evidenceRoots;
const referenceRoots = parseStringArray(
parsedEnvironment.data.REA_REFERENCE_ROOTS_JSON,
"REA_REFERENCE_ROOTS_JSON",
);
if (!referenceRoots.ok) return referenceRoots;
const secretPatterns = parseStringArray(
parsedEnvironment.data.REA_REFERENCE_SECRET_PATTERNS_JSON,
"REA_REFERENCE_SECRET_PATTERNS_JSON",
);
if (!secretPatterns.ok) return secretPatterns;
return ok({
hopperLauncherPath:
parsedEnvironment.data.HOPPER_LAUNCHER_PATH ??
@ -119,6 +144,23 @@ export const parseConfig = (
executableRoots: executableRoots.value,
workingRoots: workingRoots.value,
allowedEnvironment: allowedEnvironment.value,
allowExternalNetwork:
parsedEnvironment.data.REA_PROCESS_ALLOW_EXTERNAL_NETWORK === "true",
},
evidenceFilePolicy: {
roots: evidenceRoots.value,
maxBytes: 64 * 1024 * 1024,
maxDepth: 64,
maxStringLength: 1024 * 1024,
maxNodes: 1_000_000,
},
referenceSourcePolicy: {
roots: referenceRoots.value,
secretPatterns: secretPatterns.value,
maxBytes: 16 * 1024 * 1024,
maxEntries: 10_000,
maxDepth: 32,
maxPathBytes: 4_096,
},
});
};

98
src/contracts/artifactComparisonExample.ts Normal file
View File

@ -0,0 +1,98 @@
import { createEvidence } from "../domain/evidence.js";
import { jsonValueSchema } from "../domain/jsonValue.js";
const inventory = (digit: string) => {
const sha = digit.repeat(64);
const artifactId = `art_${sha}`;
const occurrenceId = `occ_${sha}`;
return jsonValueSchema.parse({
manifest: {
schema_version: 1,
manifest_id: `agm_${sha}`,
root_artifact_id: artifactId,
root_sha256: sha,
root_format: "file",
graph_sha256: sha,
node_count: 1,
occurrence_count: 1,
edge_count: 0,
},
nodes: {
items: [
{
artifact_id: artifactId,
kind: "resource",
format: "file",
sha256: sha,
size: 1,
media_type: null,
architecture: null,
executable: false,
content_state: "materialized",
limitations: [],
},
],
offset: 0,
limit: 100,
total: 1,
next_offset: null,
},
occurrences: {
items: [
{
occurrence_id: occurrenceId,
artifact_id: artifactId,
parent_occurrence_id: null,
logical_path: ".",
entry_kind: "file",
declared_size: 1,
compressed_size: null,
executable: false,
encrypted: false,
hash_status: "verified",
source_location: null,
limitations: [],
},
],
offset: 0,
limit: 100,
total: 1,
next_offset: null,
},
edges: { items: [], offset: 0, limit: 100, total: 0, next_offset: null },
limits: {
max_entries: 100,
max_total_bytes: 1024,
max_entry_bytes: 1024,
max_compression_ratio: 10,
max_depth: 10,
max_path_bytes: 256,
},
provenance: [],
limitations: [],
});
};
const provider = {
id: "rea-artifact",
name: "REA artifact graph",
version: "1",
} as const;
/** Canonical complete inputs used to advertise artifact comparison. */
export const ARTIFACT_COMPARISON_EXAMPLE = {
left: createEvidence(undefined, provider, {
operation: "inventory_artifact",
parameters: {},
result: inventory("0"),
confidence: "observed",
authority: "shipped-artifact",
}),
right: createEvidence(undefined, provider, {
operation: "inventory_artifact",
parameters: {},
result: inventory("1"),
confidence: "observed",
authority: "shipped-artifact",
}),
} as const;

111
src/contracts/artifactToolContracts.ts Normal file
View File

@ -0,0 +1,111 @@
import { z } from "zod";
import type { ToolContract } from "./toolContracts.js";
import { artifactOutputSchemas } from "./toolOutputSchemas.js";
import { jsonValueSchema } from "../domain/jsonValue.js";
const pageInput = {
node_offset: z.number().int().min(0).default(0),
node_limit: z.number().int().min(1).max(500).default(100),
occurrence_offset: z.number().int().min(0).default(0),
occurrence_limit: z.number().int().min(1).max(500).default(100),
edge_offset: z.number().int().min(0).default(0),
edge_limit: z.number().int().min(1).max(500).default(100),
};
const traversalLimitsInput = {
max_entries: z.number().int().min(1).max(1_000_000).default(10_000),
max_total_bytes: z
.number()
.int()
.min(1)
.max(Number.MAX_SAFE_INTEGER)
.default(1_073_741_824),
max_entry_bytes: z
.number()
.int()
.min(1)
.max(Number.MAX_SAFE_INTEGER)
.default(268_435_456),
max_compression_ratio: z.number().min(1).max(100_000).default(1_000),
max_depth: z.number().int().min(0).max(100).default(20),
max_path_bytes: z.number().int().min(1).max(65_535).default(4_096),
};
/** Exact caller boundary for deterministic artifact inventory. */
export const artifactInventoryInputSchema = z.object({
...pageInput,
...traversalLimitsInput,
});
/** Exact caller boundary for approved artifact extraction. */
export const artifactExtractionInputSchema = z.object({
approved: z.literal(true),
output_root: z.string().min(1).max(4_096),
occurrence_ids: z
.array(z.string().regex(/^occ_[a-f0-9]{64}$/u))
.min(1)
.max(500),
offset: z.number().int().min(0).default(0),
limit: z.number().int().min(1).max(500).default(100),
...traversalLimitsInput,
});
const exampleInputSchema = z.record(z.string(), jsonValueSchema);
const examples: Readonly<Record<string, Readonly<Record<string, unknown>>>> = {
inventory_artifact: {},
extract_artifact: {
approved: true,
output_root: "/tmp/rea-extracted",
occurrence_ids: [`occ_${"0".repeat(64)}`],
},
};
const artifact = <Name extends string>(
name: Name,
description: string,
inputSchema: z.ZodObject,
mutatesFilesystem: boolean,
): ToolContract<Name> => {
const outputSchema = artifactOutputSchemas[name];
if (outputSchema === undefined)
throw new Error(`Missing artifact output schema for ${name}`);
return {
name,
description,
kind: "artifact-provider",
inputSchema,
outputSchema,
annotations: {
readOnlyHint: !mutatesFilesystem,
destructiveHint: false,
idempotentHint: true,
openWorldHint: false,
},
examples: [
{
title: `Example ${name.replaceAll("_", " ")} request`,
input: exampleInputSchema.parse(examples[name] ?? {}),
},
],
};
};
/** Artifact-graph inventory and explicitly approved safe extraction contracts. */
export const ARTIFACT_TOOL_CONTRACTS = [
artifact(
"inventory_artifact",
"Inventory the active application or package as a deterministic, content-addressed artifact graph. Returns bounded node and edge pages without extracting or mounting by default.",
artifactInventoryInputSchema,
false,
),
artifact(
"extract_artifact",
"Extract selected graph artifacts beneath an explicit output root. Requires approval, rejects traversal and symlink escapes, never overwrites, enforces bomb limits, and verifies cleanup.",
artifactExtractionInputSchema,
true,
),
] as const satisfies readonly ToolContract[];
/** Names of provider-neutral artifact graph operations. */
export type ArtifactToolName = (typeof ARTIFACT_TOOL_CONTRACTS)[number]["name"];

4
src/contracts/enhancedInputs.ts
View File

@ -57,6 +57,10 @@ export const enhancedInputSchemas = {
case_sensitive: z.boolean().default(false),
limit: z.number().int().min(1).max(100).default(20),
max_operations: z.number().int().min(1).max(100).default(20),
unknown_registry_approved: z
.literal(true)
.optional()
.describe("Explicit approval to record bounded residuals durably"),
}),
} as const;

67
src/contracts/functionComparisonExample.ts Normal file
View File

@ -0,0 +1,67 @@
import { enhancedInputSchemas } from "./enhancedInputs.js";
import { createEvidence } from "../domain/evidence.js";
import { jsonValueSchema } from "../domain/jsonValue.js";
const bounded = <Item>(items: readonly Item[]) => ({
items,
total: items.length,
returned: items.length,
truncated: false,
next_offset: null,
});
const dossier = (text: string) =>
jsonValueSchema.parse({
procedure: {
address: "0x1000",
name: "main",
signature: "int main(void)",
locals: [],
},
pseudocode: {
text,
total_chars: [...text].length,
returned_chars: [...text].length,
truncated: false,
next_offset: null,
},
assembly: bounded([]),
comments: bounded([]),
callers: bounded([]),
callees: bounded([]),
incoming_references: bounded([]),
outgoing_references: bounded([]),
referenced_strings: bounded([]),
referenced_names: bounded([]),
basic_blocks: bounded([{ start: "0x1000", end: "0x1001", successors: [] }]),
instruction_scan: { scanned: 1, truncated: false },
});
const observe = (digit: string, text: string) =>
createEvidence(
{
path: `/tmp/function-${digit}`,
sha256: digit.repeat(64),
format: "mach-o",
},
{
id: "rea-workflow",
name: "REA composed investigation workflow",
version: "1",
},
{
operation: "analyze_function",
parameters: enhancedInputSchemas.analyze_function.parse({
procedure: "main",
}),
result: dossier(text),
confidence: "derived",
authority: "shipped-artifact",
},
);
/** Canonical explicit function pair used in public contract examples. */
export const FUNCTION_COMPARISON_EXAMPLE = {
left: observe("0", "return 0;"),
right: observe("1", "return 1;"),
} as const;

139
src/contracts/investigationExamples.ts Normal file
View File

@ -0,0 +1,139 @@
import { compareFunctions } from "../domain/functionComparison.js";
import { createEvidence } from "../domain/evidence.js";
import { jsonValueSchema } from "../domain/jsonValue.js";
import { createEvidenceBundle } from "../domain/evidenceBundle.js";
import {
compareProcessCaptures,
processCaptureSchema,
} from "../domain/processCapture.js";
import { FUNCTION_COMPARISON_EXAMPLE } from "./functionComparisonExample.js";
import { EMPTY_PROCESS_CAPTURE_EXAMPLE } from "./processCaptureExample.js";
const comparison = compareFunctions(
FUNCTION_COMPARISON_EXAMPLE.left,
FUNCTION_COMPARISON_EXAMPLE.right,
0,
100,
);
export const FUNCTION_COMPARISON_EVIDENCE = createEvidence(
undefined,
{
id: "rea-function-comparison",
name: "REA function comparison",
version: "1",
},
{
predicateType: "rea.function-comparison/v1",
operation: "compare_functions",
parameters: {
left_evidence_ids: [FUNCTION_COMPARISON_EXAMPLE.left.evidence_id],
right_evidence_ids: [FUNCTION_COMPARISON_EXAMPLE.right.evidence_id],
offset: 0,
limit: 100,
},
result: jsonValueSchema.parse(comparison),
confidence: "derived",
authority: "analyst-inference",
evidenceLinks: [
FUNCTION_COMPARISON_EXAMPLE.left.evidence_id,
FUNCTION_COMPARISON_EXAMPLE.right.evidence_id,
],
},
);
const PROCESS_PROVIDER = {
id: "rea-process",
name: "REA deterministic process harness",
version: "1",
} as const;
const capture = processCaptureSchema.parse(EMPTY_PROCESS_CAPTURE_EXAMPLE);
const captureEvidence = (scenario: string) =>
createEvidence(undefined, PROCESS_PROVIDER, {
predicateType: "rea.process-capture/v2",
operation: "capture_process_scenario",
parameters: { scenario },
result: jsonValueSchema.parse(capture),
confidence: "observed",
authority: "controlled-replay",
environment: {
id: "fixture-linux",
platform: "linux",
architecture: "x86_64",
isolation: "container",
},
});
export const PROCESS_CAPTURE_REFERENCE = captureEvidence("reference");
export const PROCESS_CAPTURE_RECONSTRUCTION = captureEvidence("reconstruction");
export const PROCESS_COMPARISON_EVIDENCE = createEvidence(
undefined,
PROCESS_PROVIDER,
{
predicateType: "rea.process-comparison/v1",
operation: "compare_process_captures",
parameters: {
left_evidence_id: PROCESS_CAPTURE_REFERENCE.evidence_id,
right_evidence_id: PROCESS_CAPTURE_RECONSTRUCTION.evidence_id,
left_normalization: capture.normalization,
right_normalization: capture.normalization,
},
result: jsonValueSchema.parse(compareProcessCaptures(capture, capture)),
confidence: "derived",
authority: "analyst-inference",
evidenceLinks: [
PROCESS_CAPTURE_REFERENCE.evidence_id,
PROCESS_CAPTURE_RECONSTRUCTION.evidence_id,
],
},
);
/** Canonical inputs for comparison-composed investigation contracts. */
export const INVESTIGATION_EXAMPLES = {
find_changed_behavior: { comparisons: [FUNCTION_COMPARISON_EVIDENCE] },
build_call_path: {
functions: [FUNCTION_COMPARISON_EXAMPLE.left],
start: { address: "0x1000" },
goal: { address: "0x1000" },
},
correlate_static_and_runtime: {
static_comparisons: [FUNCTION_COMPARISON_EVIDENCE],
runtime_comparisons: [PROCESS_COMPARISON_EVIDENCE],
mappings: [
{
static: {
comparison_evidence_id: FUNCTION_COMPARISON_EVIDENCE.evidence_id,
selector: { kind: "function", dimension: "pseudocode" },
},
runtime: {
comparison_evidence_id: PROCESS_COMPARISON_EVIDENCE.evidence_id,
dimension: "terminal",
},
side_alignment: "left_to_left",
hypothesis: {
statement: "Static implementation changed without terminal change.",
expected_pattern: "static_only",
},
},
],
},
verify_reconstruction: {
specification: {
schema_version: 1,
name: "Terminal compatibility",
claims: [
{
kind: "behavioral",
claim_id: "terminal-output",
title: "Terminal output remains equivalent",
comparison_evidence_id: PROCESS_COMPARISON_EVIDENCE.evidence_id,
dimension: "terminal",
},
],
},
evidence_bundle: createEvidenceBundle([
PROCESS_CAPTURE_REFERENCE,
PROCESS_CAPTURE_RECONSTRUCTION,
PROCESS_COMPARISON_EVIDENCE,
]),
},
};

83
src/contracts/nativeToolContracts.ts Normal file
View File

@ -0,0 +1,83 @@
import { z } from "zod";
import type { ToolContract } from "./toolContracts.js";
import { nativeOutputSchemas } from "./toolOutputSchemas.js";
import { jsonValueSchema } from "../domain/jsonValue.js";
const examples: Readonly<Record<string, Readonly<Record<string, unknown>>>> = {
inspect_macho: {},
inspect_signature: {},
inspect_plist: { relative_path: "Contents/Info.plist" },
list_architectures: {},
demangle_swift: { symbols: ["$s4Test3fooyyF"] },
};
const native = <Name extends string>(
name: Name,
description: string,
inputSchema: z.ZodObject,
): ToolContract<Name> => {
const outputSchema = nativeOutputSchemas[name];
if (outputSchema === undefined)
throw new Error(`Missing native output schema for ${name}`);
return {
name,
description,
kind: "native-provider",
inputSchema,
outputSchema,
annotations: {
readOnlyHint: true,
destructiveHint: false,
idempotentHint: true,
openWorldHint: false,
},
examples: [
{
title: `Example ${name.replaceAll("_", " ")} request`,
input: z
.record(z.string(), jsonValueSchema)
.parse(examples[name] ?? {}),
},
],
};
};
/** Provider-neutral semantic operations backed initially by macOS utilities. */
export const NATIVE_TOOL_CONTRACTS = [
native(
"inspect_macho",
"Inspect Mach-O slices, load commands, imports, exports, dependencies, build metadata, segments, sections, permissions, and exact command provenance without launching Hopper.",
z.object({}),
),
native(
"inspect_signature",
"Inspect the active artifact's code-signing identity, hashes, authorities, requirements, entitlements, hardened-runtime state, and bounded command provenance.",
z.object({}),
),
native(
"inspect_plist",
"Parse a plist at a bounded relative path beneath the active artifact container. Symlink and traversal escapes are rejected; output is normalized JSON rather than plutil text.",
z.object({
relative_path: z
.string()
.min(1)
.max(1_024)
.default("Contents/Info.plist"),
}),
),
native(
"list_architectures",
"List thin or universal Mach-O slices with offsets, sizes, alignment, explicit coverage, and bounded native-tool provenance.",
z.object({}),
),
native(
"demangle_swift",
"Demangle an ordered bounded batch of Swift symbols without requiring Hopper. Each input returns demangled, unchanged, or invalid status.",
z.object({
symbols: z.array(z.string().min(1).max(4_096)).min(1).max(500),
}),
),
] as const satisfies readonly ToolContract[];
export type NativeToolName = (typeof NATIVE_TOOL_CONTRACTS)[number]["name"];

27
src/contracts/processCaptureExample.ts Normal file
View File

@ -0,0 +1,27 @@
import type { JsonValue } from "../domain/jsonValue.js";
/** Minimal valid process capture used only in public contract examples. */
export const EMPTY_PROCESS_CAPTURE_EXAMPLE = {
schema_version: 2,
normalization: {
paths: true,
pids: true,
ports: true,
time_bucket_ms: 10,
patterns: [],
},
frames: [],
exit: { code: 0, signal: null, reason: "exited" },
process_samples: [],
protocol_events: [],
files_before: [],
files_after: [],
filesystem_effects: [],
truncated: false,
limitations: [],
residual_unknowns: [],
cleanup: {
owned_process_group: "verified",
temporary_root: "removed",
},
} satisfies JsonValue;

73
src/contracts/toolContractExamples.ts Normal file
View File

@ -0,0 +1,73 @@
import type { JsonValue } from "../domain/jsonValue.js";
import { EMPTY_PROCESS_CAPTURE_EXAMPLE } from "./processCaptureExample.js";
import { UNKNOWN_CONTRACT_EXAMPLES } from "./unknownContractExamples.js";
import { ARTIFACT_COMPARISON_EXAMPLE } from "./artifactComparisonExample.js";
import { FUNCTION_COMPARISON_EXAMPLE } from "./functionComparisonExample.js";
import { INVESTIGATION_EXAMPLES } from "./investigationExamples.js";
/** Canonical examples for contracts whose required inputs have no defaults. */
export const TOOL_EXAMPLE_OVERRIDES: Readonly<
Record<string, Readonly<Record<string, JsonValue>>>
> = {
...UNKNOWN_CONTRACT_EXAMPLES,
...INVESTIGATION_EXAMPLES,
goto_address: { address: "0x1000" },
procedure_address: { procedure: "main" },
procedure_assembly: { procedure: "main" },
procedure_callees: { procedure: "main" },
procedure_callers: { procedure: "main" },
procedure_info: { procedure: "main" },
procedure_references: { procedure: "main" },
procedure_pseudo_code: { procedure: "main" },
resolve_containing_procedure: { address: "0x1000" },
search_procedures: { pattern: "main" },
search_strings: { pattern: "authorization failed" },
set_address_name: { address: "0x1000", name: "entry" },
set_addresses_names: { names: { "0x1000": "entry" } },
set_bookmark: { address: "0x1000" },
set_comment: { address: "0x1000", comment: "validated entry point" },
set_current_document: { document: "fixture" },
set_inline_comment: { address: "0x1000", comment: "calls parser" },
unset_bookmark: { address: "0x1000" },
get_call_graph: { address: "0x1000" },
find_xrefs_to_name: { name: "malloc" },
analyze_function: { procedure: "main" },
trace_feature: { query: "license" },
open_binary: { path: "/tmp/fixture" },
import_evidence_bundle: { path: "evidence.json" },
capture_process_scenario: {
approved: true,
executable: "/usr/bin/true",
working_directory: "/tmp",
},
compare_process_captures: {
left_evidence_id: `ev_${"0".repeat(64)}`,
left: EMPTY_PROCESS_CAPTURE_EXAMPLE,
right_evidence_id: `ev_${"1".repeat(64)}`,
right: EMPTY_PROCESS_CAPTURE_EXAMPLE,
},
compare_artifacts: ARTIFACT_COMPARISON_EXAMPLE,
compare_functions: FUNCTION_COMPARISON_EXAMPLE,
compare_bundles: {
left: {
bundle_version: 2,
artifacts: [],
providers: [],
environments: [],
scenarios: [],
captures: [],
unknowns: [],
records: [],
},
right: {
bundle_version: 2,
artifacts: [],
providers: [],
environments: [],
scenarios: [],
captures: [],
unknowns: [],
records: [],
},
},
};

237
src/contracts/toolContracts.ts
View File

@ -1,22 +1,41 @@
import { z } from "zod";
import { evidenceBundleSchema } from "../domain/evidenceBundle.js";
import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js";
import {
processCaptureSchema,
processScenarioSchema,
} from "../domain/processCapture.js";
import {
recordUnknownInputSchema,
updateUnknownInputSchema,
} from "../domain/residualUnknown.js";
import { artifactComparisonInputSchema } from "../domain/artifactComparison.js";
import { functionComparisonInputSchema } from "../domain/functionComparison.js";
import { bundleComparisonInputSchema } from "../domain/bundleComparison.js";
import { changedBehaviorInputSchema } from "../domain/changedBehavior.js";
import { callPathInputSchema } from "../domain/callPath.js";
import { staticRuntimeCorrelationInputSchema } from "../domain/staticRuntimeCorrelation.js";
import { reconstructionVerificationInputSchema } from "../domain/reconstructionVerification.js";
import { enhancedInputSchemas } from "./enhancedInputs.js";
import {
enhancedOutputSchemas,
officialOutputSchemas,
requireOutputSchema,
sessionOutputSchemas,
} from "./toolOutputSchemas.js";
import { TOOL_EXAMPLE_OVERRIDES } from "./toolContractExamples.js";
import { NATIVE_TOOL_CONTRACTS } from "./nativeToolContracts.js";
import { ARTIFACT_TOOL_CONTRACTS } from "./artifactToolContracts.js";
const document = z.string().optional().describe("The document name");
const address = z.string().describe("A Hopper address");
const optionalAddress = address.optional();
const procedure = z.string().describe("The procedure name or address");
const pattern = z.string().describe("The regex pattern to search for");
const searchPattern = z
.string()
.min(1)
.max(256)
.describe("The literal text or bounded regex pattern to search for");
const caseSensitive = z
.boolean()
.default(false)
@ -25,94 +44,129 @@ const pagination = {
offset: z.number().int().min(0).default(0),
limit: z.number().int().min(1).max(500).default(100),
};
const searchInput = {
pattern: searchPattern,
mode: z.enum(["literal", "regex"]).default("literal"),
case_sensitive: caseSensitive,
offset: z.number().int().min(0).default(0),
limit: z.number().int().min(1).max(100).default(100),
document,
};
export interface ToolAnnotations {
readonly readOnlyHint: boolean;
readonly destructiveHint: boolean;
readonly idempotentHint: true;
readonly openWorldHint: false;
readonly idempotentHint: boolean;
readonly openWorldHint: boolean;
}
export interface ToolExample {
readonly title: string;
readonly input: Readonly<Record<string, JsonValue>>;
}
/** Adapter family responsible for implementing a public MCP tool. */
type ToolKind = "official-proxy" | "enhanced" | "session";
type ToolKind =
| "official-proxy"
| "enhanced"
| "native-provider"
| "artifact-provider"
| "session";
/** Single source of truth for a public tool's name, description, and input. */
export interface ToolContract {
readonly name: string;
export interface ToolContract<Name extends string = string> {
readonly name: Name;
readonly description: string;
readonly kind: ToolKind;
readonly inputSchema: z.ZodObject;
readonly outputSchema: z.ZodObject;
readonly annotations: ToolAnnotations;
readonly examples: readonly ToolExample[];
}
const exampleInputSchema = z.record(z.string(), jsonValueSchema);
const examplesFor = (
name: string,
inputSchema: z.ZodObject,
): readonly ToolExample[] => {
const parsed = inputSchema.parse(TOOL_EXAMPLE_OVERRIDES[name] ?? {});
return [
{
title: `Example ${name.replaceAll("_", " ")} request`,
input: exampleInputSchema.parse(parsed),
},
];
};
const annotations = (name: string, kind: ToolKind): ToolAnnotations => ({
readOnlyHint:
kind === "enhanced" ||
name === "export_evidence_bundle" ||
(!name.startsWith("set_") &&
name !== "unset_bookmark" &&
name !== "goto_address" &&
kind !== "session"),
(kind === "enhanced" && name !== "trace_feature") ||
name === "binary_session" ||
name === "list_unknowns" ||
name === "verify_unknown_resolution",
destructiveHint:
name === "export_evidence_bundle" ||
name === "unset_bookmark" ||
name === "set_address_name" ||
name === "set_addresses_names" ||
name === "set_comment" ||
name === "set_inline_comment",
idempotentHint: true,
openWorldHint: false,
idempotentHint: name !== "record_unknown" && name !== "update_unknown",
openWorldHint: name === "capture_process_scenario",
});
const official = (
name: string,
const official = <Name extends string>(
name: Name,
description: string,
inputSchema: z.ZodObject,
): ToolContract => ({
name,
description,
kind: "official-proxy",
inputSchema,
outputSchema: requireOutputSchema(officialOutputSchemas, name),
annotations: annotations(name, "official-proxy"),
});
): ToolContract<Name> => {
const trackedInputSchema = inputSchema.extend({
unknown_registry_approved: z
.literal(true)
.optional()
.describe(
"Explicit approval to record typed capability unavailability as a residual unknown",
),
});
return {
name,
description,
kind: "official-proxy",
inputSchema: trackedInputSchema,
outputSchema: requireOutputSchema(officialOutputSchemas, name),
annotations: annotations(name, "official-proxy"),
examples: examplesFor(name, trackedInputSchema),
};
};
const enhanced = (
name: string,
const enhanced = <Name extends string>(
name: Name,
description: string,
inputSchema: z.ZodObject,
): ToolContract => ({
): ToolContract<Name> => ({
name,
description,
kind: "enhanced",
inputSchema,
outputSchema: requireOutputSchema(enhancedOutputSchemas, name),
annotations: annotations(name, "enhanced"),
examples: examplesFor(name, inputSchema),
});
const session = (
name: string,
const session = <Name extends string>(
name: Name,
description: string,
inputSchema: z.ZodObject,
): ToolContract => ({
): ToolContract<Name> => ({
name,
description,
kind: "session",
inputSchema,
outputSchema: requireOutputSchema(sessionOutputSchemas, name),
annotations: annotations(name, "session"),
examples: examplesFor(name, inputSchema),
});
const requireOutputSchema = (
schemas: Readonly<Record<string, z.ZodObject>>,
name: string,
): z.ZodObject => {
const schema = schemas[name];
if (schema === undefined)
throw new Error(`Missing output schema for ${name}`);
return schema;
};
/** Bridge operations exposed without additional application composition. */
export const OFFICIAL_TOOL_CONTRACTS = [
official(
@ -239,13 +293,13 @@ export const OFFICIAL_TOOL_CONTRACTS = [
),
official(
"search_procedures",
"Regex-search all analyzed procedure names with optional case sensitivity. The current bridge returns an unpaginated address/name map, so constrain patterns and use list_procedures for controlled exhaustive traversal.",
z.object({ pattern, case_sensitive: caseSensitive, document }),
"Search analyzed procedure names using literal matching by default or a structurally bounded regex. Returns a deterministic, offset-paginated page; continue at next_offset while has_more is true.",
z.object(searchInput),
),
official(
"search_strings",
"Regex-search all analyzed strings with optional case sensitivity. The current bridge is unpaginated and evaluates Python regex, so use narrow patterns and follow matches with xrefs.",
z.object({ pattern, case_sensitive: caseSensitive, document }),
"Search analyzed strings using literal matching by default or a structurally bounded regex. Returns a deterministic, offset-paginated page with explicit value truncation; follow matches with xrefs.",
z.object(searchInput),
),
official(
"set_address_name",
@ -289,6 +343,9 @@ export const OFFICIAL_TOOL_CONTRACTS = [
),
] as const satisfies readonly ToolContract[];
/** Closed provider operation names exposed by direct analysis adapters. */
export type OfficialToolName = (typeof OFFICIAL_TOOL_CONTRACTS)[number]["name"];
/** Bounded workflows composed from one or more bridge operations. */
export const ENHANCED_TOOL_CONTRACTS = [
enhanced(
@ -338,7 +395,7 @@ export const ENHANCED_TOOL_CONTRACTS = [
),
enhanced(
"trace_feature",
"Trace a bounded literal feature query through matching strings and procedures, xrefs, and truthful containing-procedure resolution. Returns the operation budget, truncation, and residual unknowns; it does not infer reference kinds.",
"Trace a bounded literal feature query through matching strings and procedures, xrefs, and truthful containing-procedure resolution. Returns the operation budget, truncation, and residual unknowns; unknown_registry_approved: true records them durably without inferring reference kinds.",
enhancedInputSchemas.trace_feature,
),
] as const satisfies readonly ToolContract[];
@ -347,32 +404,35 @@ export const ENHANCED_TOOL_CONTRACTS = [
export const SESSION_TOOL_CONTRACTS = [
session(
"open_binary",
"Open a local executable, application bundle, or Hopper database, replacing the active target only after validation. This launches Hopper and may show UI; call binary_overview after success.",
"Open a local executable, application bundle, archive, JavaScript, source map, plist, or Hopper database after validation. Providers start lazily: inventory_artifact does not launch Hopper; deep native operations may show Hopper UI.",
z.object({ path: z.string().min(1) }),
),
session(
"close_binary",
"Close the active Hopper-backed target and release its provider process. The operation is idempotent; call binary_session to verify the session is closed.",
"Close the active target and every provider resource started for it. The operation is idempotent; call binary_session to verify the session is closed.",
z.object({}),
),
session(
"binary_session",
"Report whether a target is open and, when open, its canonical path, format, and kind. Use before analysis calls or target switches; this performs no analysis.",
"Report provider identity, deterministic capability descriptors, and whether a target is open; open targets include canonical path, format, and kind. Use availability, effects, limits, and limitations before selecting analysis operations.",
z.object({}),
),
session(
"export_evidence_bundle",
"Return the session's deterministic Evidence v2 bundle without clearing it. Records are sorted by evidence ID, and array order carries no investigative meaning.",
z.object({}),
"Return the session's deterministic Evidence v2 bundle, or atomically write it beneath an operator-approved root. Existing files require overwrite: true; records and manifests use canonical byte-stable ordering.",
z.object({
path: z.string().min(1).optional(),
overwrite: z.boolean().default(false),
}),
),
session(
"import_evidence_bundle",
"Validate and atomically merge a local Evidence v2 bundle supplied as data. Semantic IDs are recomputed; tampering, unsupported versions, conflicts, and ledger overflow reject the entire import.",
z.object({ bundle: evidenceBundleSchema }),
"Read a bounded local JSON bundle beneath an operator-approved root, validate every Evidence v2 ID and canonical manifest, then atomically merge it. Imported content is data only and is never executed.",
z.object({ path: z.string().min(1) }),
),
session(
"capture_process_scenario",
"Run one bounded process under a PTY using operator-approved executable and working roots. Requires approved: true on every call. Captures normalized terminal frames, sampled descendants, filesystem snapshots, and loopback HTTP/WebSocket replay. This launches a process and is disabled unless operator policy enables it; it is not a security sandbox.",
"Run one bounded process under a PTY using operator-approved executable and working roots. Requires approved: true; unknown_registry_approved: true separately records capture residuals. Captures normalized terminal frames, descendants, filesystem snapshots, and loopback replay. Disabled unless operator policy enables it; not a security sandbox.",
processScenarioSchema,
),
session(
@ -383,17 +443,84 @@ export const SESSION_TOOL_CONTRACTS = [
left: processCaptureSchema,
right_evidence_id: z.string().regex(/^ev_[a-f0-9]{64}$/u),
right: processCaptureSchema,
unknown_registry_approved: z
.literal(true)
.optional()
.describe("Explicit approval to record capture disagreement durably"),
}),
),
session(
"compare_artifacts",
"Compare two bounded sets of inventory_artifact Evidence pages by logical occurrence path, content identity, metadata, and graph relations. Pages must share and satisfy their graph commitment; every delta cites both sets, and gaps yield truncated or unknown, never equivalence.",
artifactComparisonInputSchema,
),
session(
"compare_functions",
"Compare two explicit bounded sets of analyze_function Evidence pages across identity, exact provider text, calls, references, strings, and address-normalized CFG topology. Missing or provider-incompatible facets remain truncated or unknown; every conclusion cites both Evidence sets.",
functionComparisonInputSchema,
),
session(
"compare_bundles",
"Compare two canonical Evidence v2 bundles by exact record membership, explicit one-to-one observation pairs, and complete residual-unknown revision histories. Missing bundle members describe omission only, never behavioral equivalence; output is digest-anchored and deterministically paginated.",
bundleComparisonInputSchema,
),
session(
"find_changed_behavior",
"Aggregate validated process, artifact, and function comparison Evidence into a deterministic change report. Runtime observations remain distinct from static behavior candidates; missing or incomplete comparisons produce unresolved findings, never causal claims.",
changedBehaviorInputSchema,
),
session(
"build_call_path",
"Build bounded shortest-first direct-callee paths from explicit analyze_function Evidence groups using exact canonical addresses. Missing dossiers, incomplete callee pages, provider mixing, and depth frontiers remain unknown; every node and edge cites source Evidence.",
callPathInputSchema,
),
session(
"correlate_static_and_runtime",
"Evaluate explicit caller-declared hypotheses between exact static comparison findings and runtime comparison dimensions. Similar names or paths are never auto-matched, consistent cochange never proves causality, and unknown or truncated inputs remain unresolved.",
staticRuntimeCorrelationInputSchema,
),
session(
"verify_reconstruction",
"Verify a finite typed behavioral and structural specification against a canonical Evidence bundle. Pass means every declared claim has complete comparable authority—not global source equivalence; changed claims fail and missing, limited, or unresolved evidence stays unknown.",
reconstructionVerificationInputSchema,
),
session(
"list_unknowns",
"List current residual-unknown heads in deterministic ID order, with optional exact status, severity, and domain filters. This is read-only; unresolved, contradicted, and non-truth dispositions remain distinct.",
z.object({
status: z
.enum(["open", "investigating", "blocked", "contradicted", "resolved"])
.optional(),
severity: z.enum(["low", "medium", "high", "critical"]).optional(),
domain: z.string().trim().min(1).max(100).optional(),
}),
),
session(
"record_unknown",
"Create one deterministic residual unknown and immutable mutation evidence. Requires approved: true, validates all evidence and relationship references, and rejects duplicate stable identity.",
recordUnknownInputSchema,
),
session(
"update_unknown",
"Append one immutable full-state revision and mutation evidence. Requires approved: true and exact expected_revision; stale concurrent writers fail instead of overwriting newer analysis.",
updateUnknownInputSchema,
),
session(
"verify_unknown_resolution",
"Revalidate the current residual-unknown head against live bundled evidence, exact authority/confidence/environment requirements, and revision integrity. Withdrawn and out-of-scope dispositions are not truth claims.",
z.object({ unknown_id: z.string().regex(/^unk_[a-f0-9]{64}$/u) }),
),
] as const satisfies readonly ToolContract[];
/**
* Complete ordered public inventory used by registration and verification.
* Keep this collection at 50 tools unless a deliberate contract change updates
* Keep this collection at 68 tools unless a deliberate contract change updates
* snapshots, package verification, and real-Hopper verification together.
*/
export const TOOL_CONTRACTS = [
...OFFICIAL_TOOL_CONTRACTS,
...ENHANCED_TOOL_CONTRACTS,
...NATIVE_TOOL_CONTRACTS,
...ARTIFACT_TOOL_CONTRACTS,
...SESSION_TOOL_CONTRACTS,
] as const;

292
src/contracts/toolOutputSchemas.ts
View File

@ -1,23 +1,108 @@
import { z } from "zod";
import { evidenceSchema } from "../domain/evidence.js";
import { processCaptureSchema } from "../domain/processCapture.js";
import {
processCaptureComparisonSchema,
processCaptureSchema,
} from "../domain/processCapture.js";
import { evidenceBundleSchema } from "../domain/evidenceBundle.js";
import { residualUnknownSchema } from "../domain/residualUnknown.js";
import { functionDossierSchema } from "../domain/hopperValues.js";
import {
demangleSwiftSchema,
inspectMachoSchema,
inspectPlistSchema,
inspectSignatureSchema,
listArchitecturesSchema,
} from "../domain/nativeInspection.js";
import {
artifactExtractionResultSchema,
artifactInventoryResultSchema,
} from "../domain/artifactGraph.js";
import { artifactComparisonResultSchema } from "../domain/artifactComparison.js";
import { functionComparisonResultSchema } from "../domain/functionComparison.js";
import { bundleComparisonResultSchema } from "../domain/bundleComparison.js";
import { changedBehaviorResultSchema } from "../domain/changedBehavior.js";
import { callPathResultSchema } from "../domain/callPath.js";
import { staticRuntimeCorrelationResultSchema } from "../domain/staticRuntimeCorrelation.js";
import { reconstructionVerificationResultSchema } from "../domain/reconstructionVerification.js";
const resultOf = (schema: z.ZodType) =>
evidenceSchema.omit({ result: true }).extend({ result: schema });
evidenceSchema
.omit({ normalized_result: true })
.extend({ normalized_result: schema });
const lifecycleResultOf = (schema: z.ZodType) => z.object({ result: schema });
const comparisonStatus = z.enum([
"unchanged",
"added",
"removed",
"changed",
"truncated",
"unknown",
/** Resolve a required named output schema or reject contract drift. */
export const requireOutputSchema = (
schemas: Readonly<Record<string, z.ZodObject>>,
name: string,
): z.ZodObject => {
const schema = schemas[name];
if (schema === undefined)
throw new Error(`Missing output schema for ${name}`);
return schema;
};
const targetFormatSchema = z.enum([
"hopper",
"mach-o",
"elf",
"pe",
"zip",
"ipa",
"apk",
"asar",
"dmg",
"pkg",
"plist",
"javascript",
"source-map",
]);
const targetKindSchema = z.enum([
"executable",
"database",
"archive",
"artifact",
]);
const providerCapability = z.object({
operation: z.string(),
available: z.boolean(),
reason: z.string().nullable(),
input_contract_version: z.number().int().min(1),
output_contract_version: z.number().int().min(1),
pagination: z.enum(["none", "offset", "cursor"]),
exhaustive: z.boolean(),
effects: z.object({
mutates_artifact: z.boolean(),
launches_process: z.boolean(),
may_show_ui: z.boolean(),
may_access_network: z.boolean(),
may_write_filesystem: z.boolean(),
changes_permissions: z.boolean(),
requires_root: z.boolean(),
}),
limits: z.object({
max_results: z.number().int().min(0).nullable(),
max_payload_bytes: z.number().int().min(0).nullable(),
timeout_ms: z.number().int().min(0).nullable(),
}),
limitations: z.array(z.string()),
});
const sessionProvider = z.object({
provider: z.object({
id: z.string(),
name: z.string(),
version: z.string().nullable(),
}),
capabilities: z.array(providerCapability),
});
const nullableText = z.string().nullable();
const addressList = z.array(z.string());
const addressedEntry = z.object({ address: z.string(), name: z.string() });
const procedureIdentity = z.object({ address: z.string(), name: z.string() });
const localVariable = z.object({
description: z.string(),
provenance: z.literal("hopper-public-python-api"),
});
const containingProcedureResolution = z.discriminatedUnion("found", [
z.object({
query_address: z.string(),
@ -51,20 +136,27 @@ const pageOutput = z.object({
next_offset: z.number().int().min(0).nullable(),
has_more: z.boolean(),
});
const segmentOutput = resultOf(
z.array(
const searchPageOutput = pageOutput.extend({
items: z.array(
z.object({
name: z.string(),
start: z.string(),
end: z.string(),
writable: z.boolean().nullable(),
executable: z.boolean().nullable(),
permissions: unavailable,
sections: z.array(
z.object({ name: z.string(), start: z.string(), end: z.string() }),
),
address: z.string(),
value: z.string(),
value_truncated: z.boolean(),
}),
),
});
const memoryRegionOutput = z.object({
name: z.string(),
start: z.string(),
end: z.string(),
readable: z.boolean().nullable(),
writable: z.boolean().nullable(),
executable: z.boolean().nullable(),
permissions: unavailable,
provenance: z.literal("hopper-public-python-api"),
});
const segmentOutput = resultOf(
z.array(memoryRegionOutput.extend({ sections: z.array(memoryRegionOutput) })),
);
const procedureInfoOutput = resultOf(
z.object({
@ -73,7 +165,7 @@ const procedureInfoOutput = resultOf(
basicblock_count: z.number().int().min(0),
length: z.number().min(0),
signature: nullableText,
locals: z.array(z.json()),
locals: z.array(localVariable),
}),
);
const symbolDiscoveryOutput = (property: "classes" | "protocols") =>
@ -87,60 +179,7 @@ const graphNode = z.union([
z.object({ address: z.string(), calls: z.array(z.string()) }),
z.object({ address: z.string(), error: z.string() }),
]);
const referenceEdge = z.object({
source_address: z.string(),
target_address: z.string(),
source_procedure: procedureIdentity.nullable(),
target_procedure: procedureIdentity.nullable(),
kind: unavailable,
});
const referencedValue = z.object({
address: z.string(),
value: z.string(),
source_address: z.string(),
});
const functionDossierOutput = resultOf(
z.object({
procedure: z.object({
address: z.string(),
name: z.string(),
signature: nullableText,
locals: z.array(z.json()),
}),
pseudocode: z.object({
text: z.string(),
total_chars: z.number().int().min(0),
returned_chars: z.number().int().min(0),
truncated: z.boolean(),
next_offset: z.number().int().min(0).nullable(),
}),
assembly: bounded(z.string()),
comments: bounded(
z.object({
address: z.string(),
kind: z.enum(["comment", "inline"]),
text: z.string(),
}),
),
callers: bounded(procedureIdentity),
callees: bounded(procedureIdentity),
incoming_references: bounded(referenceEdge),
outgoing_references: bounded(referenceEdge),
referenced_strings: bounded(referencedValue),
referenced_names: bounded(referencedValue),
basic_blocks: bounded(
z.object({
start: z.string(),
end: z.string(),
successors: z.array(z.string()),
}),
),
instruction_scan: z.object({
scanned: z.number().int().min(0),
truncated: z.boolean(),
}),
}),
);
const functionDossierOutput = resultOf(functionDossierSchema);
/** Exact structured-content schemas for the direct Hopper operations. */
export const officialOutputSchemas: Readonly<Record<string, z.ZodObject>> = {
@ -183,8 +222,8 @@ export const officialOutputSchemas: Readonly<Record<string, z.ZodObject>> = {
),
procedure_pseudo_code: resultOf(nullableText),
resolve_containing_procedure: resultOf(containingProcedureResolution),
search_procedures: resultOf(z.record(z.string(), z.string())),
search_strings: resultOf(z.record(z.string(), z.string())),
search_procedures: resultOf(searchPageOutput),
search_strings: resultOf(searchPageOutput),
set_address_name: resultOf(z.boolean()),
set_addresses_names: resultOf(z.record(z.string(), z.boolean())),
set_bookmark: resultOf(z.boolean()),
@ -269,13 +308,28 @@ export const enhancedOutputSchemas: Readonly<Record<string, z.ZodObject>> = {
),
};
/** Exact Evidence v2 schemas for provider-neutral native inspection. */
export const nativeOutputSchemas: Readonly<Record<string, z.ZodObject>> = {
inspect_macho: resultOf(inspectMachoSchema),
inspect_signature: resultOf(inspectSignatureSchema),
inspect_plist: resultOf(inspectPlistSchema),
list_architectures: resultOf(listArchitecturesSchema),
demangle_swift: resultOf(demangleSwiftSchema),
};
/** Exact Evidence v2 schemas for provider-neutral artifact graph operations. */
export const artifactOutputSchemas: Readonly<Record<string, z.ZodObject>> = {
inventory_artifact: resultOf(artifactInventoryResultSchema),
extract_artifact: resultOf(artifactExtractionResultSchema),
};
/** Exact structured-content schemas for target lifecycle operations. */
export const sessionOutputSchemas: Readonly<Record<string, z.ZodObject>> = {
open_binary: lifecycleResultOf(
z.object({
path: z.string(),
format: z.enum(["hopper", "mach-o", "elf", "pe"]),
kind: z.enum(["executable", "database"]),
format: targetFormatSchema,
kind: targetKindSchema,
loaderArgs: z.array(z.string()),
sha256: z.string().regex(/^[a-f0-9]{64}$/u),
architecture: z.enum(["x86", "x86_64", "arm", "arm64"]).nullable(),
@ -284,18 +338,27 @@ export const sessionOutputSchemas: Readonly<Record<string, z.ZodObject>> = {
close_binary: lifecycleResultOf(z.null()),
binary_session: lifecycleResultOf(
z.union([
z.object({ open: z.literal(false) }),
z.object({
sessionProvider.extend({ open: z.literal(false) }),
sessionProvider.extend({
open: z.literal(true),
path: z.string(),
format: z.enum(["hopper", "mach-o", "elf", "pe"]),
kind: z.enum(["executable", "database"]),
format: targetFormatSchema,
kind: targetKindSchema,
sha256: z.string().regex(/^[a-f0-9]{64}$/u),
architecture: z.enum(["x86", "x86_64", "arm", "arm64"]).nullable(),
}),
]),
),
export_evidence_bundle: lifecycleResultOf(evidenceBundleSchema),
export_evidence_bundle: lifecycleResultOf(
z.union([
evidenceBundleSchema,
z.object({
path: z.string(),
bytes: z.number().int().min(0),
records: z.number().int().min(0),
}),
]),
),
import_evidence_bundle: lifecycleResultOf(
z.object({
imported: z.number().int().min(0),
@ -304,20 +367,57 @@ export const sessionOutputSchemas: Readonly<Record<string, z.ZodObject>> = {
),
capture_process_scenario: lifecycleResultOf(
evidenceSchema
.omit({ result: true })
.extend({ result: processCaptureSchema }),
.omit({ normalized_result: true })
.extend({ normalized_result: processCaptureSchema }),
),
compare_process_captures: lifecycleResultOf(
evidenceSchema.omit({ result: true }).extend({
result: z.object({
status: comparisonStatus,
terminal: comparisonStatus,
exit: comparisonStatus,
filesystem: comparisonStatus,
protocol: comparisonStatus,
process: comparisonStatus,
limitations: z.array(z.string()),
}),
evidenceSchema.omit({ normalized_result: true }).extend({
normalized_result: processCaptureComparisonSchema,
}),
),
compare_artifacts: lifecycleResultOf(
evidenceSchema.omit({ normalized_result: true }).extend({
normalized_result: artifactComparisonResultSchema,
}),
),
compare_functions: lifecycleResultOf(
evidenceSchema.omit({ normalized_result: true }).extend({
normalized_result: functionComparisonResultSchema,
}),
),
compare_bundles: lifecycleResultOf(
evidenceSchema.omit({ normalized_result: true }).extend({
normalized_result: bundleComparisonResultSchema,
}),
),
find_changed_behavior: lifecycleResultOf(
evidenceSchema.omit({ normalized_result: true }).extend({
normalized_result: changedBehaviorResultSchema,
}),
),
build_call_path: lifecycleResultOf(
evidenceSchema.omit({ normalized_result: true }).extend({
normalized_result: callPathResultSchema,
}),
),
correlate_static_and_runtime: lifecycleResultOf(
evidenceSchema.omit({ normalized_result: true }).extend({
normalized_result: staticRuntimeCorrelationResultSchema,
}),
),
verify_reconstruction: lifecycleResultOf(
evidenceSchema.omit({ normalized_result: true }).extend({
normalized_result: reconstructionVerificationResultSchema,
}),
),
list_unknowns: lifecycleResultOf(z.array(residualUnknownSchema)),
record_unknown: lifecycleResultOf(residualUnknownSchema),
update_unknown: lifecycleResultOf(residualUnknownSchema),
verify_unknown_resolution: lifecycleResultOf(
z.object({
valid: z.boolean(),
truthVerified: z.boolean(),
unknown: residualUnknownSchema,
}),
),
};

35
src/contracts/unknownContractExamples.ts Normal file
View File

@ -0,0 +1,35 @@
import type { JsonValue } from "../domain/jsonValue.js";
/** Exact example inputs for residual-unknown lifecycle contracts. */
export const UNKNOWN_CONTRACT_EXAMPLES = {
list_unknowns: {},
record_unknown: {
approved: true,
question: "Does this branch require an unavailable external service?",
severity: "medium",
domain: "protocol",
required_authority: "controlled-replay",
required_confidence: "observed",
required_environment: null,
recommended_probes: [
{ operation: "capture_process_scenario", rationale: "Replay branch." },
],
relationships: [],
},
update_unknown: {
approved: true,
unknown_id: `unk_${"0".repeat(64)}`,
expected_revision: 1,
status: "investigating",
severity: "medium",
supporting_evidence_ids: [],
contradicting_evidence_ids: [],
required_authority: "controlled-replay",
required_confidence: "observed",
required_environment: null,
recommended_probes: [],
relationships: [],
resolution: null,
},
verify_unknown_resolution: { unknown_id: `unk_${"0".repeat(64)}` },
} satisfies Readonly<Record<string, Readonly<Record<string, JsonValue>>>>;

390
src/domain/artifactComparison.ts Normal file
View File

@ -0,0 +1,390 @@
import canonicalize from "canonicalize";
import { z } from "zod";
import {
type ArtifactInventoryResult,
type ArtifactNode,
type ArtifactOccurrence,
} from "./artifactGraph.js";
import { evidenceSchema } from "./evidence.js";
import {
parseArtifactInventoryEvidence,
type InventorySet,
} from "./artifactInventoryEvidence.js";
const evidenceIdSchema = z.string().regex(/^ev_[a-f0-9]{64}$/u);
const comparisonStatusSchema = z.enum([
"unchanged",
"added",
"removed",
"changed",
"truncated",
"unknown",
]);
const changeKindSchema = z.enum(["added", "removed", "changed", "unknown"]);
const comparisonDimensionSchema = z.enum([
"content",
"kind",
"format",
"size",
"executable",
"relations",
"metadata",
"availability",
]);
/** Strict Evidence-backed input for deterministic artifact comparison. */
export const artifactComparisonInputSchema = z.object({
left: z.union([evidenceSchema, z.array(evidenceSchema).min(1).max(100)]),
right: z.union([evidenceSchema, z.array(evidenceSchema).min(1).max(100)]),
offset: z.number().int().min(0).default(0),
limit: z.number().int().min(1).max(500).default(100),
unknown_registry_approved: z
.literal(true)
.optional()
.describe("Explicit approval to record incomplete or differing evidence"),
});
/** One path-classified artifact change with citations to both observations. */
const artifactChangeSchema = z.object({
classification: changeKindSchema,
logical_path: z.string().min(1).max(4_096),
dimensions: z.array(comparisonDimensionSchema).min(1).max(8),
left_occurrence_id: z
.string()
.regex(/^occ_[a-f0-9]{64}$/u)
.nullable(),
right_occurrence_id: z
.string()
.regex(/^occ_[a-f0-9]{64}$/u)
.nullable(),
left_artifact_id: z
.string()
.regex(/^art_[a-f0-9]{64}$/u)
.nullable(),
right_artifact_id: z
.string()
.regex(/^art_[a-f0-9]{64}$/u)
.nullable(),
evidence_links: z.array(evidenceIdSchema).min(2).max(200),
});
/** Bounded deterministic artifact comparison with an independently paged delta. */
export const artifactComparisonResultSchema = z.object({
status: comparisonStatusSchema,
left_manifest_id: z.string().regex(/^agm_[a-f0-9]{64}$/u),
right_manifest_id: z.string().regex(/^agm_[a-f0-9]{64}$/u),
summary: z.object({
unchanged: z.number().int().min(0),
added: z.number().int().min(0),
removed: z.number().int().min(0),
changed: z.number().int().min(0),
unknown: z.number().int().min(0),
}),
changes: z.object({
items: z.array(artifactChangeSchema).max(500),
offset: z.number().int().min(0),
limit: z.number().int().min(1).max(500),
total: z.number().int().min(0),
next_offset: z.number().int().min(0).nullable(),
}),
limitations: z.array(z.string()),
});
export type ArtifactComparisonResult = z.infer<
typeof artifactComparisonResultSchema
>;
type ArtifactChange = z.infer<typeof artifactChangeSchema>;
/** Compare complete inventory pages without treating missing evidence as equality. */
export const compareArtifacts = (
leftEvidence: unknown,
rightEvidence: unknown,
offset: number,
limit: number,
): ArtifactComparisonResult => {
const left = parseArtifactInventoryEvidence(leftEvidence);
const right = parseArtifactInventoryEvidence(rightEvidence);
const leftComplete = left.inventory.complete;
const rightComplete = right.inventory.complete;
const leftCovered = leftComplete && left.inventory.limitations.length === 0;
const rightCovered =
rightComplete && right.inventory.limitations.length === 0;
const limitations = [
...left.inventory.limitations.map((item) => `Left: ${item}`),
...right.inventory.limitations.map((item) => `Right: ${item}`),
...(leftComplete ? [] : ["Left artifact inventory is incomplete."]),
...(rightComplete ? [] : ["Right artifact inventory is incomplete."]),
];
const changes = compareOccurrences(
left.inventory,
right.inventory,
[
...left.evidence.map(({ evidence_id: id }) => id),
...right.evidence.map(({ evidence_id: id }) => id),
],
leftCovered && rightCovered,
);
const summary = summarize(
left.inventory,
right.inventory,
changes,
leftCovered && rightCovered,
);
const page = changes.slice(offset, offset + limit);
return artifactComparisonResultSchema.parse({
status:
!leftComplete || !rightComplete
? "truncated"
: !leftCovered || !rightCovered
? "unknown"
: changes.some(({ classification }) => classification === "unknown")
? "unknown"
: changes.length === 0
? "unchanged"
: "changed",
left_manifest_id: left.inventory.manifest.manifest_id,
right_manifest_id: right.inventory.manifest.manifest_id,
summary,
changes: {
items: page,
offset,
limit,
total: changes.length,
next_offset:
offset + page.length < changes.length ? offset + page.length : null,
},
limitations: [...new Set(limitations)].sort((a, b) => a.localeCompare(b)),
});
};
const compareOccurrences = (
left: InventorySet,
right: InventorySet,
evidenceLinks: readonly string[],
complete: boolean,
): ArtifactChange[] => {
const leftByPath = new Map(
left.occurrences.map((item) => [item.logical_path, item]),
);
const rightByPath = new Map(
right.occurrences.map((item) => [item.logical_path, item]),
);
const leftNodes = new Map(left.nodes.map((item) => [item.artifact_id, item]));
const rightNodes = new Map(
right.nodes.map((item) => [item.artifact_id, item]),
);
const leftRelations = relationsByPath(left);
const rightRelations = relationsByPath(right);
const paths = [
...new Set([...leftByPath.keys(), ...rightByPath.keys()]),
].sort((a, b) => a.localeCompare(b, "en"));
const output: ArtifactChange[] = [];
for (const path of paths) {
const leftOccurrence = leftByPath.get(path);
const rightOccurrence = rightByPath.get(path);
const change = classifyPath({
path,
leftOccurrence,
rightOccurrence,
leftNode: nodeFor(leftOccurrence, leftNodes),
rightNode: nodeFor(rightOccurrence, rightNodes),
leftRelations: leftRelations.get(path) ?? [],
rightRelations: rightRelations.get(path) ?? [],
complete,
evidenceLinks,
});
if (change !== null) output.push(change);
}
return output;
};
const classifyPath = (input: {
readonly path: string;
readonly leftOccurrence: ArtifactOccurrence | undefined;
readonly rightOccurrence: ArtifactOccurrence | undefined;
readonly leftNode: ArtifactNode | undefined;
readonly rightNode: ArtifactNode | undefined;
readonly leftRelations: readonly RelationProjection[];
readonly rightRelations: readonly RelationProjection[];
readonly complete: boolean;
readonly evidenceLinks: readonly string[];
}): ArtifactChange | null => {
const base = {
logical_path: input.path,
left_occurrence_id: input.leftOccurrence?.occurrence_id ?? null,
right_occurrence_id: input.rightOccurrence?.occurrence_id ?? null,
left_artifact_id: input.leftOccurrence?.artifact_id ?? null,
right_artifact_id: input.rightOccurrence?.artifact_id ?? null,
evidence_links: [...input.evidenceLinks],
};
if (input.leftOccurrence === undefined)
return {
...base,
classification: input.complete ? "added" : "unknown",
dimensions: [input.complete ? "content" : "availability"],
};
if (input.rightOccurrence === undefined)
return {
...base,
classification: input.complete ? "removed" : "unknown",
dimensions: [input.complete ? "content" : "availability"],
};
if (
input.leftOccurrence.hash_status !== "verified" ||
input.rightOccurrence.hash_status !== "verified" ||
input.leftNode === undefined ||
input.rightNode === undefined
)
return { ...base, classification: "unknown", dimensions: ["availability"] };
const dimensions = changedDimensions(input);
return dimensions.length === 0
? null
: { ...base, classification: "changed", dimensions };
};
const changedDimensions = (input: {
readonly leftNode: ArtifactNode | undefined;
readonly rightNode: ArtifactNode | undefined;
readonly leftOccurrence: ArtifactOccurrence | undefined;
readonly rightOccurrence: ArtifactOccurrence | undefined;
readonly leftRelations: readonly RelationProjection[];
readonly rightRelations: readonly RelationProjection[];
}): z.infer<typeof comparisonDimensionSchema>[] => {
const left = input.leftNode;
const right = input.rightNode;
if (left === undefined || right === undefined) return ["availability"];
const checks: readonly [
z.infer<typeof comparisonDimensionSchema>,
boolean,
][] = [
["content", left.sha256 !== right.sha256],
["kind", left.kind !== right.kind],
["format", left.format !== right.format],
["size", left.size !== right.size],
[
"executable",
input.leftOccurrence?.executable !== input.rightOccurrence?.executable,
],
[
"relations",
JSON.stringify(input.leftRelations) !==
JSON.stringify(input.rightRelations),
],
["metadata", metadataChanged(input, left, right)],
];
return checks
.filter(([, changed]) => changed)
.map(([dimension]) => dimension);
};
const metadataChanged = (
input: {
readonly leftOccurrence: ArtifactOccurrence | undefined;
readonly rightOccurrence: ArtifactOccurrence | undefined;
},
left: ArtifactNode,
right: ArtifactNode,
): boolean =>
left.media_type !== right.media_type ||
left.architecture !== right.architecture ||
left.executable !== right.executable ||
left.content_state !== right.content_state ||
input.leftOccurrence?.entry_kind !== input.rightOccurrence?.entry_kind ||
input.leftOccurrence?.encrypted !== input.rightOccurrence?.encrypted ||
JSON.stringify(left.limitations) !== JSON.stringify(right.limitations) ||
JSON.stringify(input.leftOccurrence?.limitations) !==
JSON.stringify(input.rightOccurrence?.limitations);
const canonicalJson = (value: unknown): string => {
const encoded = canonicalize(value);
if (encoded === undefined)
throw new TypeError(
"Artifact comparison could not canonicalize graph data",
);
return encoded;
};
const nodeFor = (
occurrence: ArtifactOccurrence | undefined,
nodes: ReadonlyMap<string, ArtifactNode>,
): ArtifactNode | undefined =>
occurrence?.artifact_id === null || occurrence?.artifact_id === undefined
? undefined
: nodes.get(occurrence.artifact_id);
const relationsByPath = (
inventory: InventorySet,
): ReadonlyMap<string, readonly RelationProjection[]> => {
const output = new Map<string, RelationProjection[]>();
const occurrences = new Map(
inventory.occurrences.map((item) => [item.occurrence_id, item]),
);
for (const edge of inventory.edges) {
const path = edge.logical_path ?? ".";
const values = output.get(path);
const occurrence = occurrences.get(edge.occurrence_id);
const projection: RelationProjection = {
parent_logical_path:
occurrence?.parent_occurrence_id === null ||
occurrence?.parent_occurrence_id === undefined
? null
: (occurrences.get(occurrence.parent_occurrence_id)?.logical_path ??
null),
child_artifact_id: edge.child_artifact_id,
relation: edge.relation,
logical_path: edge.logical_path,
producer: edge.producer,
};
if (values === undefined) output.set(path, [projection]);
else values.push(projection);
}
for (const values of output.values())
values.sort((left, right) =>
canonicalJson(left).localeCompare(canonicalJson(right)),
);
return output;
};
interface RelationProjection {
/** Logical parent is stable across graph-root ID changes. */
readonly parent_logical_path: string | null;
readonly child_artifact_id: string;
readonly relation: ArtifactInventoryResult["edges"]["items"][number]["relation"];
readonly logical_path: string | null;
/** Producer is semantic provenance; edge ordinal is presentation order. */
readonly producer: ArtifactInventoryResult["edges"]["items"][number]["producer"];
}
const summarize = (
left: InventorySet,
right: InventorySet,
changes: readonly ArtifactChange[],
covered: boolean,
) => {
const counts = { added: 0, removed: 0, changed: 0, unknown: 0 };
for (const change of changes) counts[change.classification] += 1;
return {
unchanged: covered
? Math.max(
0,
Math.min(left.occurrences.length, right.occurrences.length) -
counts.changed -
counts.unknown,
)
: 0,
...counts,
unknown: covered
? counts.unknown
: Math.max(
counts.unknown,
Math.max(
left.manifest.occurrence_count,
right.manifest.occurrence_count,
) -
counts.added -
counts.removed -
counts.changed,
),
};
};

239
src/domain/artifactGraph.ts Normal file
View File

@ -0,0 +1,239 @@
import { z } from "zod";
const sha256Schema = z.string().regex(/^[a-f0-9]{64}$/u);
const artifactIdSchema = z.string().regex(/^art_[a-f0-9]{64}$/u);
const occurrenceIdSchema = z.string().regex(/^occ_[a-f0-9]{64}$/u);
const edgeIdSchema = z.string().regex(/^edge_[a-f0-9]{64}$/u);
const manifestIdSchema = z.string().regex(/^agm_[a-f0-9]{64}$/u);
const extractionIdSchema = z.string().regex(/^aex_[a-f0-9]{64}$/u);
const boundedRelativePathSchema = z
.string()
.min(1)
.max(4_096)
.refine(
(path) =>
!path.startsWith("/") &&
!path.includes("\\") &&
path.split("/").every((part) => part !== "" && part !== ".."),
"Expected a normalized relative POSIX path without traversal",
);
/** Closed artifact categories used to traverse application containers. */
const artifactKindSchema = z.enum([
"container",
"executable",
"dynamic-library",
"framework",
"native-addon",
"javascript",
"source-map",
"plist",
"entitlements",
"resource",
"universal-slice",
"package-metadata",
"unknown",
]);
/** Recognized storage or application formats; unknown bytes remain explicit. */
const artifactFormatSchema = z.enum([
"dmg",
"zip",
"pkg",
"asar",
"javascript-bundle",
"source-map",
"mach-o-universal",
"mach-o",
"elf",
"pe",
"hopper",
"ipa",
"apk",
"plist",
"entitlements",
"directory",
"file",
"unknown",
]);
/** Exact bounded policy applied while discovering or extracting artifacts. */
const artifactTraversalLimitsSchema = z.object({
max_entries: z.number().int().min(1).max(1_000_000),
max_total_bytes: z.number().int().min(1).max(Number.MAX_SAFE_INTEGER),
max_entry_bytes: z.number().int().min(1).max(Number.MAX_SAFE_INTEGER),
max_compression_ratio: z.number().min(1).max(100_000),
max_depth: z.number().int().min(0).max(100),
max_path_bytes: z.number().int().min(1).max(65_535),
});
/** File byte interval for an artifact derived without writing it to disk. */
const artifactByteRangeSchema = z.object({
offset: z.number().int().min(0),
length: z.number().int().min(0),
});
/** Bounded external command provenance, including observable side effects. */
const artifactCommandSchema = z.object({
tool: z.string().min(1),
arguments: z.array(z.string().max(4_096)).max(1_024),
tool_version: z.string().nullable(),
executable_sha256: sha256Schema.nullable(),
exit_code: z.number().int().nullable(),
effects: z.array(z.enum(["read", "write", "mount"])).max(3),
});
/** Parsed bounded artifact-producing command provenance. */
export type ArtifactCommand = z.infer<typeof artifactCommandSchema>;
/** One content-addressed application artifact in deterministic manifest order. */
const artifactNodeSchema = z.object({
artifact_id: artifactIdSchema,
kind: artifactKindSchema,
format: artifactFormatSchema,
sha256: sha256Schema,
size: z.number().int().min(0).max(Number.MAX_SAFE_INTEGER),
media_type: z.string().min(1).nullable(),
architecture: z.string().min(1).nullable(),
executable: z.boolean(),
content_state: z.enum(["materialized", "embedded", "virtual"]),
limitations: z.array(z.string()),
});
/** One path or virtual location where an artifact occurs in its parent. */
const artifactOccurrenceSchema = z.object({
occurrence_id: occurrenceIdSchema,
artifact_id: artifactIdSchema.nullable(),
parent_occurrence_id: occurrenceIdSchema.nullable(),
logical_path: boundedRelativePathSchema,
entry_kind: z.enum(["file", "directory", "symlink", "slice"]),
declared_size: z
.number()
.int()
.min(0)
.max(Number.MAX_SAFE_INTEGER)
.nullable(),
compressed_size: z
.number()
.int()
.min(0)
.max(Number.MAX_SAFE_INTEGER)
.nullable(),
executable: z.boolean(),
encrypted: z.boolean(),
hash_status: z.enum(["verified", "not-hashed-limit", "unavailable"]),
source_location: artifactByteRangeSchema.nullable(),
limitations: z.array(z.string()),
});
/** Parent-child derivation relation between two content-addressed artifacts. */
const artifactEdgeSchema = z.object({
edge_id: edgeIdSchema,
ordinal: z.number().int().min(0),
parent_artifact_id: artifactIdSchema,
child_artifact_id: artifactIdSchema,
relation: z.enum([
"contains",
"extracts",
"slice-of",
"embeds",
"loads",
"maps-source",
"derived-from",
]),
occurrence_id: occurrenceIdSchema,
logical_path: boundedRelativePathSchema.nullable(),
producer: artifactCommandSchema.nullable(),
});
/** Stable cryptographic commitment and counts for a complete artifact graph. */
const artifactGraphManifestSchema = z.object({
schema_version: z.literal(1),
manifest_id: manifestIdSchema,
root_artifact_id: artifactIdSchema,
root_sha256: sha256Schema,
root_format: artifactFormatSchema,
graph_sha256: sha256Schema,
node_count: z.number().int().min(1),
occurrence_count: z.number().int().min(1),
edge_count: z.number().int().min(0),
});
/** Independently bounded offset page used for deterministic graph traversal. */
const artifactPageSchema = <Item extends z.ZodType>(item: Item) =>
z.object({
items: z.array(item).max(500),
offset: z.number().int().min(0),
limit: z.number().int().min(1).max(500),
total: z.number().int().min(0),
next_offset: z.number().int().min(0).nullable(),
});
/** Inventory observation containing stable manifest identity and bounded pages. */
export const artifactInventoryResultSchema = z.object({
manifest: artifactGraphManifestSchema,
nodes: artifactPageSchema(artifactNodeSchema),
occurrences: artifactPageSchema(artifactOccurrenceSchema),
edges: artifactPageSchema(artifactEdgeSchema),
limits: artifactTraversalLimitsSchema,
provenance: z.array(artifactCommandSchema).max(256),
limitations: z.array(z.string()),
});
/** One safely materialized artifact beneath caller-approved output root. */
const extractedArtifactSchema = z.object({
artifact_id: artifactIdSchema,
relative_path: boundedRelativePathSchema,
sha256: sha256Schema,
bytes_written: z.number().int().min(0).max(Number.MAX_SAFE_INTEGER),
created: z.boolean(),
});
/** Path-independent commitment to one selected extraction operation. */
const artifactExtractionManifestSchema = z.object({
schema_version: z.literal(1),
extraction_id: extractionIdSchema,
source_manifest_id: manifestIdSchema,
selected_occurrence_ids: z.array(occurrenceIdSchema).min(1).max(500),
files_sha256: sha256Schema,
output_root_alias: z.literal("$OUTPUT_ROOT"),
});
/** Safe extraction observation with containment and cleanup verification. */
export const artifactExtractionResultSchema = z.object({
manifest: artifactGraphManifestSchema,
extraction_manifest: artifactExtractionManifestSchema,
output_root: z.string().min(1),
artifacts: artifactPageSchema(extractedArtifactSchema),
containment_verified: z.boolean(),
cleanup: z.object({
attempted: z.boolean(),
verified: z.boolean(),
residual_paths: z.array(boundedRelativePathSchema).max(500),
}),
limits: artifactTraversalLimitsSchema,
provenance: z.array(artifactCommandSchema).max(256),
limitations: z.array(z.string()),
});
/** Parsed artifact graph node. */
export type ArtifactNode = z.infer<typeof artifactNodeSchema>;
/** Parsed artifact derivation edge. */
export type ArtifactEdge = z.infer<typeof artifactEdgeSchema>;
/** Parsed artifact occurrence. */
export type ArtifactOccurrence = z.infer<typeof artifactOccurrenceSchema>;
/** Parsed deterministic artifact graph manifest. */
export type ArtifactGraphManifest = z.infer<typeof artifactGraphManifestSchema>;
/** Parsed paginated inventory result. */
export type ArtifactInventoryResult = z.infer<
typeof artifactInventoryResultSchema
>;
/** Parsed paginated extraction result. */
export type ArtifactExtractionResult = z.infer<
typeof artifactExtractionResultSchema
>;

268
src/domain/artifactInventoryEvidence.ts Normal file
View File

@ -0,0 +1,268 @@
import { createHash } from "node:crypto";
import canonicalize from "canonicalize";
import {
artifactInventoryResultSchema,
type ArtifactInventoryResult,
type ArtifactNode,
type ArtifactOccurrence,
} from "./artifactGraph.js";
import { parseEvidence, type Evidence } from "./evidence.js";
/** Complete or partially assembled inventory used by comparison workflows. */
export interface InventorySet {
readonly manifest: ArtifactInventoryResult["manifest"];
readonly nodes: readonly ArtifactNode[];
readonly occurrences: readonly ArtifactOccurrence[];
readonly edges: readonly ArtifactInventoryResult["edges"]["items"][number][];
readonly limitations: readonly string[];
readonly complete: boolean;
}
interface ParsedInventorySet {
readonly evidence: readonly Evidence[];
readonly inventory: InventorySet;
}
/** Parse, merge, and verify a bounded set of inventory Evidence pages. */
export const parseArtifactInventoryEvidence = (
input: unknown,
): ParsedInventorySet =>
assembleInventorySet(
(Array.isArray(input) ? input : [input]).map((value) => {
const evidence = parseEvidence(value);
if (evidence.operation !== "inventory_artifact")
throw new TypeError(
"Artifact comparison requires inventory_artifact Evidence",
);
const inventory = artifactInventoryResultSchema.parse(
evidence.normalized_result,
);
if (
evidence.subject === null ||
evidence.subject.digest.sha256 !== inventory.manifest.root_sha256 ||
evidence.subject.format !== inventory.manifest.root_format
)
throw new TypeError(
"Artifact inventory Evidence subject does not match its root digest",
);
validateInventoryPage(inventory);
return { evidence, inventory };
}),
);
const validateInventoryPage = (inventory: ArtifactInventoryResult): void => {
assertUnique(
inventory.nodes.items.map(({ artifact_id: id }) => id),
"node ID",
);
assertUnique(
inventory.occurrences.items.map(({ occurrence_id: id }) => id),
"occurrence ID",
);
assertUnique(
inventory.occurrences.items.map(({ logical_path: path }) => path),
"logical path",
);
assertUnique(
inventory.edges.items.map(({ edge_id: id }) => id),
"edge ID",
);
for (const node of inventory.nodes.items)
if (
node.artifact_id !==
`art_${digestCanonical({ schema_version: 1, sha256: node.sha256 })}`
)
throw new TypeError("Artifact node ID is not content-addressed");
};
const assembleInventorySet = (
pages: readonly {
readonly evidence: Evidence;
readonly inventory: ArtifactInventoryResult;
}[],
): ParsedInventorySet => {
if (pages.length === 0 || pages.length > 100)
throw new TypeError("Artifact comparison requires 1 to 100 Evidence pages");
const first = pages[0];
if (first === undefined)
throw new TypeError("Artifact comparison requires Evidence pages");
const manifestJson = canonicalJson(first.inventory.manifest);
const nodes = new Map<string, ArtifactNode>();
const occurrences = new Map<string, ArtifactOccurrence>();
const paths = new Map<string, ArtifactOccurrence>();
const edges = new Map<
string,
ArtifactInventoryResult["edges"]["items"][number]
>();
for (const page of pages) {
if (canonicalJson(page.inventory.manifest) !== manifestJson)
throw new TypeError("Artifact inventory pages do not share one manifest");
for (const node of page.inventory.nodes.items)
mergeExact(nodes, node.artifact_id, node, "node");
for (const occurrence of page.inventory.occurrences.items) {
mergeExact(
occurrences,
occurrence.occurrence_id,
occurrence,
"occurrence",
);
mergeExact(paths, occurrence.logical_path, occurrence, "logical path");
}
for (const edge of page.inventory.edges.items)
mergeExact(edges, edge.edge_id, edge, "edge");
}
const inventory = orderedInventory(first.inventory, pages, {
nodes,
occurrences,
edges,
});
if (inventory.complete) validateCompleteInventory(inventory);
return { evidence: pages.map(({ evidence }) => evidence), inventory };
};
const orderedInventory = (
first: ArtifactInventoryResult,
pages: readonly { readonly inventory: ArtifactInventoryResult }[],
values: {
readonly nodes: ReadonlyMap<string, ArtifactNode>;
readonly occurrences: ReadonlyMap<string, ArtifactOccurrence>;
readonly edges: ReadonlyMap<
string,
ArtifactInventoryResult["edges"]["items"][number]
>;
},
): InventorySet => {
const nodes = [...values.nodes.values()].sort((left, right) =>
left.artifact_id.localeCompare(right.artifact_id),
);
const occurrences = [...values.occurrences.values()].sort((left, right) =>
left.logical_path.localeCompare(right.logical_path, "en"),
);
const edges = [...values.edges.values()].sort((left, right) =>
left.edge_id.localeCompare(right.edge_id),
);
return {
manifest: first.manifest,
nodes,
occurrences,
edges,
limitations: [
...new Set(pages.flatMap(({ inventory }) => inventory.limitations)),
].sort((left, right) => left.localeCompare(right)),
complete:
nodes.length === first.manifest.node_count &&
occurrences.length === first.manifest.occurrence_count &&
edges.length === first.manifest.edge_count,
};
};
const validateCompleteInventory = (inventory: InventorySet): void => {
const nodeIds = new Set(inventory.nodes.map(({ artifact_id: id }) => id));
const occurrenceIds = new Set(
inventory.occurrences.map(({ occurrence_id: id }) => id),
);
const rootNode = inventory.nodes.find(
({ artifact_id: id }) => id === inventory.manifest.root_artifact_id,
);
if (rootNode?.sha256 !== inventory.manifest.root_sha256)
throw new TypeError(
"Artifact root node does not match its manifest digest",
);
for (const occurrence of inventory.occurrences)
validateOccurrence(occurrence, inventory, nodeIds, occurrenceIds);
for (const edge of inventory.edges)
validateEdge(edge, nodeIds, occurrenceIds);
const graphSha256 = digestCanonical({
nodes: inventory.nodes,
occurrences: inventory.occurrences,
edges: inventory.edges,
});
if (graphSha256 !== inventory.manifest.graph_sha256)
throw new TypeError("Artifact graph commitment does not match its members");
const manifestId = `agm_${digestCanonical({
schema_version: 1,
root_artifact_id: inventory.manifest.root_artifact_id,
graph_sha256: graphSha256,
})}`;
if (manifestId !== inventory.manifest.manifest_id)
throw new TypeError("Artifact manifest ID does not match its commitment");
};
const validateOccurrence = (
occurrence: ArtifactOccurrence,
inventory: InventorySet,
nodeIds: ReadonlySet<string>,
occurrenceIds: ReadonlySet<string>,
): void => {
if (occurrence.artifact_id !== null && !nodeIds.has(occurrence.artifact_id))
throw new TypeError("Artifact occurrence references a missing node");
if (
occurrence.parent_occurrence_id !== null &&
!occurrenceIds.has(occurrence.parent_occurrence_id)
)
throw new TypeError("Artifact occurrence references a missing parent");
const expectedId =
occurrence.logical_path === "."
? `occ_${digestCanonical({ root: inventory.manifest.root_artifact_id })}`
: `occ_${digestCanonical({
root_artifact_id: inventory.manifest.root_artifact_id,
logical_path: occurrence.logical_path,
entry_kind: occurrence.entry_kind,
})}`;
if (occurrence.occurrence_id !== expectedId)
throw new TypeError("Artifact occurrence ID does not match its identity");
};
const validateEdge = (
edge: ArtifactInventoryResult["edges"]["items"][number],
nodeIds: ReadonlySet<string>,
occurrenceIds: ReadonlySet<string>,
): void => {
if (
!nodeIds.has(edge.parent_artifact_id) ||
!nodeIds.has(edge.child_artifact_id) ||
!occurrenceIds.has(edge.occurrence_id)
)
throw new TypeError("Artifact edge references a missing graph member");
const semantic = {
parent_artifact_id: edge.parent_artifact_id,
child_artifact_id: edge.child_artifact_id,
relation: edge.relation,
occurrence_id: edge.occurrence_id,
logical_path: edge.logical_path,
};
if (edge.edge_id !== `edge_${digestCanonical(semantic)}`)
throw new TypeError("Artifact edge ID does not match its identity");
};
const mergeExact = <Value>(
output: Map<string, Value>,
key: string,
value: Value,
label: string,
): void => {
const previous = output.get(key);
if (
previous !== undefined &&
canonicalJson(previous) !== canonicalJson(value)
)
throw new TypeError(`Artifact inventory has conflicting ${label} pages`);
output.set(key, value);
};
const assertUnique = (values: readonly string[], label: string): void => {
if (new Set(values).size !== values.length)
throw new TypeError(`Artifact inventory contains a duplicate ${label}`);
};
const digestCanonical = (value: unknown): string =>
createHash("sha256").update(canonicalJson(value)).digest("hex");
const canonicalJson = (value: unknown): string => {
const encoded = canonicalize(value);
if (encoded === undefined)
throw new TypeError("Artifact inventory could not canonicalize graph data");
return encoded;
};

85
src/domain/binaryTarget.ts
View File

@ -27,9 +27,23 @@ type BinaryArchitecture = "x86" | "x86_64" | "arm" | "arm64";
*/
export interface BinaryTarget {
readonly path: string;
readonly sourcePath?: string;
readonly sha256: string;
readonly kind: "executable" | "database";
readonly format: "hopper" | "mach-o" | "elf" | "pe";
readonly kind: "executable" | "database" | "archive" | "artifact";
readonly format:
| "hopper"
| "mach-o"
| "elf"
| "pe"
| "zip"
| "ipa"
| "apk"
| "asar"
| "dmg"
| "pkg"
| "plist"
| "javascript"
| "source-map";
readonly architecture?: BinaryArchitecture;
readonly availableArchitectures?: readonly BinaryArchitecture[];
readonly loaderArgs: readonly string[];
@ -61,6 +75,7 @@ export const parseBinaryTarget = async (
)
return ok({
path,
sourcePath: canonical,
sha256: await sha256File(path),
kind: "database",
format: "hopper",
@ -69,6 +84,16 @@ export const parseBinaryTarget = async (
const handle = await open(path, "r");
let detected: Result<ExecutableMetadata, string>;
try {
const artifactFormat = await detectArtifactFormat(path, handle);
if (artifactFormat !== undefined)
return ok({
path,
sourcePath: canonical,
sha256: await sha256File(path),
kind: isArchiveFormat(artifactFormat) ? "archive" : "artifact",
format: artifactFormat,
loaderArgs: [],
});
detected = await readExecutableMetadata(handle, hostArchitecture);
} finally {
await handle.close();
@ -76,6 +101,7 @@ export const parseBinaryTarget = async (
if (!detected.ok) return err(new BinaryTargetError(path, detected.error));
return ok({
path,
sourcePath: canonical,
sha256: await sha256File(path),
kind: "executable",
...detected.value,
@ -87,6 +113,61 @@ export const parseBinaryTarget = async (
}
};
const detectArtifactFormat = async (
path: string,
handle: FileHandle,
): Promise<
| Exclude<BinaryTarget["format"], "hopper" | "mach-o" | "elf" | "pe">
| undefined
> => {
const lower = path.toLowerCase();
const magic = Buffer.alloc(8);
const observed = await handle.read(magic, 0, magic.length, 0);
if (
observed.bytesRead >= 4 &&
magic[0] === 0x50 &&
magic[1] === 0x4b &&
[0x03, 0x05, 0x07].includes(magic[2] ?? -1) &&
[0x04, 0x06, 0x08].includes(magic[3] ?? -1)
) {
return lower.endsWith(".ipa")
? "ipa"
: lower.endsWith(".apk")
? "apk"
: "zip";
}
const named = namedArtifactFormat(lower);
if (named !== undefined) return named;
if (
lower.endsWith(".pkg") &&
observed.bytesRead >= 4 &&
magic.subarray(0, 4).toString("ascii") === "xar!"
)
return "pkg";
if (lower.endsWith(".dmg")) {
const size = (await handle.stat()).size;
if (size >= 512) {
const trailer = Buffer.alloc(4);
const read = await handle.read(trailer, 0, trailer.length, size - 512);
if (read.bytesRead === 4 && trailer.toString("ascii") === "koly")
return "dmg";
}
}
return undefined;
};
const namedArtifactFormat = (
lowerPath: string,
): "asar" | "plist" | "source-map" | "javascript" | undefined => {
if (lowerPath.endsWith(".asar")) return "asar";
if (lowerPath.endsWith(".plist")) return "plist";
if (lowerPath.endsWith(".map")) return "source-map";
return /\.(?:m?js|cjs)$/u.test(lowerPath) ? "javascript" : undefined;
};
const isArchiveFormat = (format: BinaryTarget["format"]): boolean =>
["zip", "ipa", "apk", "asar", "dmg", "pkg"].includes(format);
const sha256File = async (path: string): Promise<string> => {
const hash = createHash("sha256");
for await (const chunk of createReadStream(path)) hash.update(chunk);

371
src/domain/bundleComparison.ts Normal file
View File

@ -0,0 +1,371 @@
import { createHash } from "node:crypto";
import { z } from "zod";
import {
evidenceBundleSchema,
parseEvidenceBundle,
serializeEvidenceBundle,
type EvidenceBundle,
} from "./evidenceBundle.js";
import type { ResidualUnknown } from "./residualUnknown.js";
const evidenceIdSchema = z.string().regex(/^ev_[a-f0-9]{64}$/u);
const digestSchema = z.string().regex(/^[a-f0-9]{64}$/u);
const classificationSchema = z.enum([
"added",
"removed",
"changed",
"unknown",
"history_advanced",
"history_diverged",
]);
/** Strict bounded input for canonical Evidence bundle comparison. */
export const bundleComparisonInputSchema = z.object({
left: evidenceBundleSchema,
right: evidenceBundleSchema,
record_pairs: z
.array(
z.object({
left_evidence_id: evidenceIdSchema,
right_evidence_id: evidenceIdSchema,
}),
)
.max(500)
.default([]),
offset: z.number().int().min(0).default(0),
limit: z.number().int().min(1).max(500).default(100),
});
const bundleChangeSchema = z.object({
entity: z.enum(["evidence", "residual_unknown"]),
key: z.string().min(1),
classification: classificationSchema,
conclusion_kind: z.enum([
"observed_change",
"contradiction",
"unresolved_branch",
]),
left_evidence_ids: z.array(evidenceIdSchema),
right_evidence_ids: z.array(evidenceIdSchema),
left_digest: digestSchema.nullable(),
right_digest: digestSchema.nullable(),
limitations: z.array(z.string()),
});
/** Deterministic classified delta between two canonical Evidence bundles. */
export const bundleComparisonResultSchema = z.object({
status: z.enum(["unchanged", "changed", "unknown"]),
left_bundle_sha256: digestSchema,
right_bundle_sha256: digestSchema,
summary: z.object({
records_unchanged: z.number().int().min(0),
records_added: z.number().int().min(0),
records_removed: z.number().int().min(0),
records_changed: z.number().int().min(0),
unknowns_unchanged: z.number().int().min(0),
unknowns_added: z.number().int().min(0),
unknowns_removed: z.number().int().min(0),
unknowns_advanced: z.number().int().min(0),
unknowns_diverged: z.number().int().min(0),
unresolved: z.number().int().min(0),
}),
changes: z.object({
items: z.array(bundleChangeSchema).max(500),
offset: z.number().int().min(0),
limit: z.number().int().min(1).max(500),
total: z.number().int().min(0),
next_offset: z.number().int().min(0).nullable(),
}),
limitations: z.array(z.string()),
});
export type BundleComparisonResult = z.infer<
typeof bundleComparisonResultSchema
>;
type BundleChange = z.infer<typeof bundleChangeSchema>;
type RecordPair = z.infer<
typeof bundleComparisonInputSchema
>["record_pairs"][number];
const MAX_ENTITIES = 100_000;
/** Compare canonical bundle membership and only explicitly paired observations. */
export const compareBundles = (
...[leftInput, rightInput, recordPairs, offset, limit]: readonly [
unknown,
unknown,
readonly RecordPair[],
number,
number,
]
): BundleComparisonResult => {
const left = parseEvidenceBundle(leftInput);
const right = parseEvidenceBundle(rightInput);
enforceBounds(left, right);
const leftDigest = bundleDigest(left);
const rightDigest = bundleDigest(right);
const recordResult = compareRecords(left, right, recordPairs);
const unknownResult = compareUnknowns(left, right);
const changes = [...recordResult.changes, ...unknownResult.changes].sort(
compareChanges,
);
const page = changes.slice(offset, offset + limit);
const unresolved = changes.filter(
({ conclusion_kind: kind }) => kind !== "observed_change",
).length;
return bundleComparisonResultSchema.parse({
status:
leftDigest === rightDigest && changes.length === 0
? "unchanged"
: unresolved > 0
? "unknown"
: "changed",
left_bundle_sha256: leftDigest,
right_bundle_sha256: rightDigest,
summary: {
...recordResult.summary,
...unknownResult.summary,
unresolved,
},
changes: {
items: page,
offset,
limit,
total: changes.length,
next_offset:
offset + page.length < changes.length ? offset + page.length : null,
},
limitations: changes.some(({ limitations }) => limitations.length > 0)
? [
"One-sided membership proves only bundle inclusion or omission, not behavioral absence.",
]
: [],
});
};
const compareRecords = (
left: EvidenceBundle,
right: EvidenceBundle,
pairs: readonly RecordPair[],
) => {
const leftIds = new Set(left.records.map(({ evidence_id: id }) => id));
const rightIds = new Set(right.records.map(({ evidence_id: id }) => id));
const pairedLeft = new Set<string>();
const pairedRight = new Set<string>();
const changes: BundleChange[] = [];
let changed = 0;
for (const pair of pairs) {
if (!leftIds.has(pair.left_evidence_id))
throw new TypeError(
`Explicit pair references missing left evidence ${pair.left_evidence_id}`,
);
if (!rightIds.has(pair.right_evidence_id))
throw new TypeError(
`Explicit pair references missing right evidence ${pair.right_evidence_id}`,
);
if (
pairedLeft.has(pair.left_evidence_id) ||
pairedRight.has(pair.right_evidence_id)
)
throw new TypeError("Explicit evidence pairs must be one-to-one");
pairedLeft.add(pair.left_evidence_id);
pairedRight.add(pair.right_evidence_id);
if (pair.left_evidence_id === pair.right_evidence_id) continue;
changed += 1;
changes.push({
entity: "evidence",
key: `pair:${pair.left_evidence_id}:${pair.right_evidence_id}`,
classification: "changed",
conclusion_kind: "observed_change",
left_evidence_ids: [pair.left_evidence_id],
right_evidence_ids: [pair.right_evidence_id],
left_digest: pair.left_evidence_id.slice(3),
right_digest: pair.right_evidence_id.slice(3),
limitations: [],
});
}
const common = [...leftIds].filter(
(id) => rightIds.has(id) && !pairedLeft.has(id) && !pairedRight.has(id),
);
const removed = [...leftIds].filter(
(id) => !rightIds.has(id) && !pairedLeft.has(id),
);
const added = [...rightIds].filter(
(id) => !leftIds.has(id) && !pairedRight.has(id),
);
for (const [classification, ids, side] of [
["removed", removed, "left"],
["added", added, "right"],
] as const)
for (const id of ids)
changes.push({
entity: "evidence",
key: id,
classification,
conclusion_kind: "observed_change",
left_evidence_ids: side === "left" ? [id] : [],
right_evidence_ids: side === "right" ? [id] : [],
left_digest: side === "left" ? id.slice(3) : null,
right_digest: side === "right" ? id.slice(3) : null,
limitations: [
"One-sided membership does not establish behavioral absence.",
],
});
return {
changes,
summary: {
records_unchanged: common.length + pairs.length - changed,
records_added: added.length,
records_removed: removed.length,
records_changed: changed,
},
};
};
const compareUnknowns = (left: EvidenceBundle, right: EvidenceBundle) => {
const leftHistories = histories(left.unknowns);
const rightHistories = histories(right.unknowns);
const ids = [...new Set([...leftHistories.keys(), ...rightHistories.keys()])];
const changes: BundleChange[] = [];
const summary = {
unknowns_unchanged: 0,
unknowns_added: 0,
unknowns_removed: 0,
unknowns_advanced: 0,
unknowns_diverged: 0,
};
for (const id of ids) {
const leftHistory = leftHistories.get(id);
const rightHistory = rightHistories.get(id);
if (leftHistory === undefined || rightHistory === undefined) {
const added = leftHistory === undefined;
summary[added ? "unknowns_added" : "unknowns_removed"] += 1;
changes.push(
unknownChange(
id,
added ? "added" : "removed",
leftHistory,
rightHistory,
),
);
continue;
}
const leftDigests = leftHistory.map(
({ revision_digest: digest }) => digest,
);
const rightDigests = rightHistory.map(
({ revision_digest: digest }) => digest,
);
if (same(leftDigests, rightDigests)) {
summary.unknowns_unchanged += 1;
continue;
}
const advanced =
isPrefix(leftDigests, rightDigests) ||
isPrefix(rightDigests, leftDigests);
summary[advanced ? "unknowns_advanced" : "unknowns_diverged"] += 1;
changes.push(
unknownChange(
id,
advanced ? "history_advanced" : "history_diverged",
leftHistory,
rightHistory,
),
);
}
return { changes, summary };
};
const unknownChange = (
id: string,
classification: BundleChange["classification"],
left: readonly ResidualUnknown[] | undefined,
right: readonly ResidualUnknown[] | undefined,
): BundleChange => ({
entity: "residual_unknown",
key: id,
classification,
conclusion_kind:
classification === "history_diverged"
? "contradiction"
: classification === "history_advanced"
? "observed_change"
: "unresolved_branch",
left_evidence_ids: historyEvidenceIds(left ?? []),
right_evidence_ids: historyEvidenceIds(right ?? []),
left_digest: left?.at(-1)?.revision_digest ?? null,
right_digest: right?.at(-1)?.revision_digest ?? null,
limitations:
left === undefined || right === undefined
? ["A missing unknown history does not establish resolution."]
: [],
});
const histories = (unknowns: readonly ResidualUnknown[]) => {
const output = new Map<string, ResidualUnknown[]>();
for (const unknown of unknowns) {
const history = output.get(unknown.unknown_id) ?? [];
history.push(unknown);
output.set(unknown.unknown_id, history);
}
return output;
};
const historyEvidenceIds = (history: readonly ResidualUnknown[]): string[] =>
[
...new Set(
history.flatMap((item) => [
...item.supporting_evidence_ids,
...item.contradicting_evidence_ids,
...item.mutation_evidence_ids,
...(item.resolution?.evidence_ids ?? []),
]),
),
].sort(ascii);
const enforceBounds = (left: EvidenceBundle, right: EvidenceBundle): void => {
const total =
left.records.length +
right.records.length +
left.unknowns.length +
right.unknowns.length;
if (total > MAX_ENTITIES)
throw new TypeError(`Bundle comparison exceeds ${MAX_ENTITIES} entities`);
};
const bundleDigest = (bundle: EvidenceBundle): string =>
createHash("sha256").update(serializeEvidenceBundle(bundle)).digest("hex");
const same = (left: readonly string[], right: readonly string[]): boolean =>
left.length === right.length &&
left.every((value, index) => value === right[index]);
const isPrefix = (
shorter: readonly string[],
longer: readonly string[],
): boolean =>
shorter.length < longer.length &&
shorter.every((value, index) => value === longer[index]);
const ascii = (left: string, right: string): number =>
left < right ? -1 : left > right ? 1 : 0;
const compareChanges = (left: BundleChange, right: BundleChange): number =>
ascii(
[
left.entity,
left.key,
left.classification,
left.left_digest ?? "",
left.right_digest ?? "",
].join("\u0000"),
[
right.entity,
right.key,
right.classification,
right.left_digest ?? "",
right.right_digest ?? "",
].join("\u0000"),
);

466
src/domain/callPath.ts Normal file
View File

@ -0,0 +1,466 @@
import { DirectedGraph } from "graphology";
import { bidirectional } from "graphology-shortest-path/unweighted.js";
import { z } from "zod";
import { evidenceSchema } from "./evidence.js";
import {
parseFunctionEvidence,
type FunctionSnapshot,
} from "./functionDossierEvidence.js";
const addressSchema = z.string().regex(/^0x(?:0|[1-9a-f][0-9a-f]*)$/u);
const inputAddressSchema = z
.string()
.regex(/^0[xX][0-9a-fA-F]+$/u)
.transform((address) => `0x${BigInt(address).toString(16)}`);
const evidenceIdSchema = z.string().regex(/^ev_[a-f0-9]{64}$/u);
const evidenceGroupSchema = z.union([
evidenceSchema,
z.array(evidenceSchema).min(1).max(100),
]);
const MAX_EVIDENCE_PAGES = 2_000;
const MAX_GRAPH_EDGES = 10_000;
const MAX_PATH_EXPANSIONS = 100_000;
/** Strict, bounded input for explicit call-path reconstruction. */
export const callPathInputSchema = z
.object({
functions: z.array(evidenceGroupSchema).min(1).max(500),
start: z.object({ address: inputAddressSchema }).strict(),
goal: z.object({ address: inputAddressSchema }).strict(),
max_depth: z.number().int().min(0).max(32).default(8),
max_paths: z.number().int().min(1).max(100).default(10),
offset: z.number().int().min(0).default(0),
limit: z.number().int().min(1).max(100).default(100),
unknown_registry_approved: z.literal(true).optional(),
})
.superRefine(({ functions }, context) => {
const pages = functions.reduce(
(total, group) => total + (Array.isArray(group) ? group.length : 1),
0,
);
if (pages > MAX_EVIDENCE_PAGES)
context.addIssue({
code: "custom",
message: `Call-path Evidence exceeds ${MAX_EVIDENCE_PAGES} pages`,
path: ["functions"],
});
});
const citedNodeSchema = z.object({
address: addressSchema,
name: z.string().nullable(),
evidence_links: z.array(evidenceIdSchema).min(1).max(100),
});
const citedEdgeSchema = z.object({
source: addressSchema,
target: addressSchema,
evidence_links: z.array(evidenceIdSchema).min(1).max(100),
});
const pathSchema = z.object({
hops: z.number().int().min(0),
nodes: z.array(citedNodeSchema).min(1).max(33),
edges: z.array(citedEdgeSchema).max(32),
evidence_links: z.array(evidenceIdSchema).min(1).max(3_300),
});
/** Evidence-cited, bounded directed call-path result. */
export const callPathResultSchema = z.object({
status: z.enum(["found", "not_found", "unknown", "truncated"]),
start: addressSchema,
goal: addressSchema,
shortest_hops: z.number().int().min(0).nullable(),
search_scope: z.object({
max_depth: z.number().int().min(0).max(32),
max_paths: z.number().int().min(1).max(100),
exhaustive: z.boolean(),
}),
explored: z.object({
nodes: z.number().int().min(0),
edges: z.number().int().min(0),
depth_reached: z.number().int().min(0),
}),
paths: z.object({
items: z.array(pathSchema).max(100),
offset: z.number().int().min(0),
limit: z.number().int().min(1).max(100),
total: z.number().int().min(0).nullable(),
returned: z.number().int().min(0).max(100),
truncated: z.boolean(),
lower_bound: z.number().int().min(0),
next_offset: z.number().int().min(0).nullable(),
}),
evidence_links: z.array(evidenceIdSchema).min(1).max(50_000),
limitations: z.array(z.string()),
});
export type CallPathInput = z.infer<typeof callPathInputSchema>;
export type CallPathResult = z.infer<typeof callPathResultSchema>;
type OutputPath = z.infer<typeof pathSchema>;
interface SearchState {
readonly graph: DirectedGraph;
readonly snapshots: ReadonlyMap<string, FunctionSnapshot>;
readonly reached: ReadonlyMap<string, number>;
readonly exhaustive: boolean;
readonly limitations: readonly string[];
}
/** Reconstruct bounded direct-callee paths from explicit analyze_function Evidence. */
export const buildCallPath = (input: CallPathInput): CallPathResult => {
const parsed = callPathInputSchema.parse(input);
const snapshots = parseSnapshots(parsed.functions);
assertCompatible(snapshots);
if (!snapshots.has(parsed.start.address))
throw new TypeError(
`No analyze_function Evidence was supplied for start ${parsed.start.address}`,
);
const graph = createGraph(snapshots);
const search = inspectSearch({
graph,
snapshots,
start: parsed.start.address,
goal: parsed.goal.address,
maxDepth: parsed.max_depth,
});
const shortest =
graph.hasNode(parsed.start.address) && graph.hasNode(parsed.goal.address)
? bidirectional(graph, parsed.start.address, parsed.goal.address)
: null;
const enumeration =
shortest === null || shortest.length - 1 > parsed.max_depth
? { paths: [], budgetExhausted: false }
: enumeratePaths({
graph,
start: parsed.start.address,
goal: parsed.goal.address,
shortestDepth: shortest.length - 1,
maxDepth: parsed.max_depth,
limit: parsed.max_paths + 1,
});
const hasKnownExtraPath = enumeration.paths.length > parsed.max_paths;
const capped = hasKnownExtraPath || enumeration.budgetExhausted;
const retained = enumeration.paths.slice(0, parsed.max_paths);
const paths = retained.map((path) => citePath(path, snapshots));
const total = paths.length;
const items = paths.slice(parsed.offset, parsed.offset + parsed.limit);
const found = paths.length > 0;
const exhaustive = search.exhaustive && !capped;
const status = callPathStatus({ capped, found, exhaustive });
const limitations = [...search.limitations];
if (capped)
limitations.push(
enumeration.budgetExhausted
? `Path enumeration stopped at ${MAX_PATH_EXPANSIONS} expansions`
: `Path enumeration stopped at max_paths=${parsed.max_paths}`,
);
if (!found && shortest !== null && shortest.length - 1 > parsed.max_depth)
limitations.push(
`The shortest observed path exceeds max_depth=${parsed.max_depth}`,
);
return callPathResultSchema.parse({
status,
start: parsed.start.address,
goal: parsed.goal.address,
shortest_hops: found ? (paths[0]?.hops ?? null) : null,
search_scope: {
max_depth: parsed.max_depth,
max_paths: parsed.max_paths,
exhaustive,
},
explored: {
nodes: search.reached.size,
edges: [...search.reached.keys()].reduce(
(count, node) =>
count + (graph.hasNode(node) ? graph.outDegree(node) : 0),
0,
),
depth_reached: Math.max(0, ...search.reached.values()),
},
paths: {
items,
offset: parsed.offset,
limit: parsed.limit,
total: capped ? null : total,
returned: items.length,
truncated: capped,
lower_bound: total + (hasKnownExtraPath ? 1 : 0),
next_offset:
parsed.offset + items.length < total
? parsed.offset + items.length
: null,
},
evidence_links: uniqueEvidence(snapshots.values()),
limitations: [...new Set(limitations)].sort((left, right) =>
left.localeCompare(right),
),
});
};
const callPathStatus = ({
capped,
found,
exhaustive,
}: {
readonly capped: boolean;
readonly found: boolean;
readonly exhaustive: boolean;
}): CallPathResult["status"] => {
if (capped) return "truncated";
if (found) return "found";
return exhaustive ? "not_found" : "unknown";
};
const parseSnapshots = (
groups: readonly z.infer<typeof evidenceGroupSchema>[],
): Map<string, FunctionSnapshot> => {
const snapshots = new Map<string, FunctionSnapshot>();
for (const group of groups) {
const snapshot = parseFunctionEvidence(group);
const address = normalizeAddress(snapshot.procedure.address);
for (const callee of snapshot.collections.callees.items)
normalizeAddress(callee.address);
if (snapshots.has(address))
throw new TypeError(`Duplicate function Evidence for ${address}`);
snapshots.set(address, snapshot);
}
return snapshots;
};
const assertCompatible = (
snapshots: ReadonlyMap<string, FunctionSnapshot>,
): void => {
const first = snapshots.values().next().value as FunctionSnapshot | undefined;
if (first === undefined) return;
for (const snapshot of snapshots.values()) {
if (
snapshot.subject.digest.sha256 !== first.subject.digest.sha256 ||
snapshot.subject.format !== first.subject.format ||
snapshot.subject.architecture !== first.subject.architecture
)
throw new TypeError("Call-path Evidence mixes artifact subjects");
if (
snapshot.provider.id !== first.provider.id ||
snapshot.provider.name !== first.provider.name ||
snapshot.provider.version !== first.provider.version
)
throw new TypeError("Call-path Evidence mixes providers");
}
};
const createGraph = (
snapshots: ReadonlyMap<string, FunctionSnapshot>,
): DirectedGraph => {
const graph = new DirectedGraph({ allowSelfLoops: true, multi: false });
for (const [address, snapshot] of snapshots) {
graph.mergeNode(address);
for (const callee of snapshot.collections.callees.items) {
const calleeAddress = normalizeAddress(callee.address);
graph.mergeNode(calleeAddress);
graph.mergeDirectedEdge(address, calleeAddress);
if (graph.size > MAX_GRAPH_EDGES)
throw new TypeError(
`Call graph exceeds ${MAX_GRAPH_EDGES} directed edges`,
);
}
}
return graph;
};
interface SearchInput {
readonly graph: DirectedGraph;
readonly snapshots: ReadonlyMap<string, FunctionSnapshot>;
readonly start: string;
readonly goal: string;
readonly maxDepth: number;
}
const inspectSearch = ({
graph,
snapshots,
start,
goal,
maxDepth,
}: SearchInput): SearchState => {
if (!graph.hasNode(start))
return {
graph,
snapshots,
reached: new Map(),
exhaustive: false,
limitations: [
`No analyze_function Evidence was supplied for start ${start}`,
],
};
const reached = new Map<string, number>([[start, 0]]);
const queue = [start];
const limitations: string[] = [];
for (let index = 0; index < queue.length; index += 1) {
const node = queue[index];
if (node === undefined) continue;
const depth = reached.get(node) ?? 0;
if (node === goal) continue;
const snapshot = snapshots.get(node);
if (snapshot === undefined) {
limitations.push(
`No analyze_function Evidence covers reachable function ${node}`,
);
continue;
}
if (!snapshot.collections.callees.complete)
limitations.push(
`Callee coverage is incomplete for reachable function ${node}`,
);
const neighbors = graph
.outNeighbors(node)
.sort((left, right) => left.localeCompare(right));
if (depth === maxDepth) {
if (neighbors.length > 0)
limitations.push(`Search reached max_depth=${maxDepth} at ${node}`);
continue;
}
for (const neighbor of neighbors)
if (!reached.has(neighbor)) {
reached.set(neighbor, depth + 1);
queue.push(neighbor);
}
}
return {
graph,
snapshots,
reached,
exhaustive: limitations.length === 0,
limitations,
};
};
interface EnumerationInput {
readonly graph: DirectedGraph;
readonly start: string;
readonly goal: string;
readonly shortestDepth: number;
readonly maxDepth: number;
readonly limit: number;
}
const enumeratePaths = ({
graph,
start,
goal,
shortestDepth,
maxDepth,
limit,
}: EnumerationInput): {
readonly paths: string[][];
readonly budgetExhausted: boolean;
} => {
const output: string[][] = [];
const state: EnumerationState = {
graph,
goal,
path: [start],
visited: new Set([start]),
output,
limit,
expansions: 0,
budgetExhausted: false,
};
for (
let depth = shortestDepth;
depth <= maxDepth && output.length < limit;
depth += 1
)
enumerateAtDepth(state, depth);
return { paths: output, budgetExhausted: state.budgetExhausted };
};
interface EnumerationState {
readonly graph: DirectedGraph;
readonly goal: string;
readonly path: string[];
readonly visited: Set<string>;
readonly output: string[][];
readonly limit: number;
expansions: number;
budgetExhausted: boolean;
}
const enumerateAtDepth = (state: EnumerationState, remaining: number): void => {
const { graph, goal, path, visited, output, limit } = state;
state.expansions += 1;
if (state.expansions > MAX_PATH_EXPANSIONS) {
state.budgetExhausted = true;
return;
}
const current = path.at(-1);
if (current === undefined || output.length >= limit) return;
if (remaining === 0) {
if (current === goal) output.push([...path]);
return;
}
if (current === goal) return;
for (const neighbor of graph
.outNeighbors(current)
.sort((left, right) => left.localeCompare(right))) {
if (visited.has(neighbor)) continue;
visited.add(neighbor);
path.push(neighbor);
enumerateAtDepth(state, remaining - 1);
path.pop();
visited.delete(neighbor);
if (output.length >= limit || state.budgetExhausted) return;
}
};
const citePath = (
addresses: readonly string[],
snapshots: ReadonlyMap<string, FunctionSnapshot>,
): OutputPath => {
const edges = addresses.slice(0, -1).map((source, index) => {
const target = addresses[index + 1];
if (target === undefined)
throw new TypeError("Call path has an invalid edge");
return {
source,
target,
evidence_links: snapshotLinks(snapshots.get(source)),
};
});
const nodes = addresses.map((address, index) => {
const snapshot = snapshots.get(address);
const supporting =
snapshot ??
(index > 0 ? snapshots.get(addresses[index - 1] ?? "") : undefined);
return {
address,
name: snapshot?.procedure.name ?? null,
evidence_links: snapshotLinks(supporting),
};
});
return {
hops: edges.length,
nodes,
edges,
evidence_links: [
...new Set(
edges
.flatMap(({ evidence_links: links }) => links)
.concat(nodes.flatMap(({ evidence_links: links }) => links)),
),
],
};
};
const snapshotLinks = (snapshot: FunctionSnapshot | undefined): string[] => {
if (snapshot === undefined)
throw new TypeError("Every call-path claim requires supporting Evidence");
return snapshot.evidence.map(({ evidence_id }) => evidence_id);
};
const uniqueEvidence = (snapshots: Iterable<FunctionSnapshot>): string[] =>
[
...new Set([...snapshots].flatMap((snapshot) => snapshotLinks(snapshot))),
].sort((left, right) => left.localeCompare(right));
const normalizeAddress = (input: string): string =>
inputAddressSchema.parse(input);

404
src/domain/changedBehavior.ts Normal file
View File

@ -0,0 +1,404 @@
import { z } from "zod";
import { artifactComparisonResultSchema } from "./artifactComparison.js";
import { evidenceSchema, parseEvidence, type Evidence } from "./evidence.js";
import { functionComparisonResultSchema } from "./functionComparison.js";
import {
comparisonStatusSchema,
processCaptureComparisonSchema,
} from "./processCapture.js";
const evidenceIdSchema = z.string().regex(/^ev_[a-f0-9]{64}$/u);
/** Strict bounded input for aggregating existing comparison Evidence. */
export const changedBehaviorInputSchema = z.object({
comparisons: z.array(evidenceSchema).min(1).max(100),
offset: z.number().int().min(0).default(0),
limit: z.number().int().min(1).max(100).default(100),
unknown_registry_approved: z.literal(true).optional(),
});
const findingSchema = z.object({
scope: z.enum(["runtime", "protocol", "resource", "static_candidate"]),
dimension: z.string().min(1),
classification: z.enum([
"observed_change",
"derived_relationship",
"contradiction",
"unresolved_branch",
]),
status: comparisonStatusSchema,
source_comparison_id: evidenceIdSchema,
evidence_links: z.array(evidenceIdSchema).min(3).max(201),
limitations: z.array(z.string()),
});
/** Truthful aggregate: runtime observations remain separate from static candidates. */
export const changedBehaviorResultSchema = z.object({
behavior_status: z.enum([
"observed_changed",
"observed_unchanged",
"unknown",
"truncated",
]),
summary: z.object({
observed_changes: z.number().int().min(0),
static_candidates: z.number().int().min(0),
contradictions: z.number().int().min(0),
unresolved: z.number().int().min(0),
}),
findings: z.object({
items: z.array(findingSchema).max(100),
offset: z.number().int().min(0),
limit: z.number().int().min(1).max(100),
total: z.number().int().min(0),
next_offset: z.number().int().min(0).nullable(),
}),
evidence_links: z.array(evidenceIdSchema).min(3).max(20_100),
limitations: z.array(z.string()),
});
export type ChangedBehaviorResult = z.infer<typeof changedBehaviorResultSchema>;
type Finding = z.infer<typeof findingSchema>;
type RuntimeStatus = z.infer<typeof processCaptureComparisonSchema>["status"];
const EXPECTED_PROVIDERS = {
compare_process_captures: {
predicate: "rea.process-comparison/v1",
id: "rea-process",
name: "REA deterministic process harness",
version: "1",
},
compare_artifacts: {
predicate: "rea.artifact-comparison/v1",
id: "rea-artifact-comparison",
name: "REA artifact comparison",
version: "1",
},
compare_functions: {
predicate: "rea.function-comparison/v1",
id: "rea-function-comparison",
name: "REA function comparison",
version: "1",
},
} as const;
/** Find observed runtime changes and separately report static change candidates. */
export const findChangedBehavior = (
comparisonsInput: unknown,
offset: number,
limit: number,
): ChangedBehaviorResult => {
const parsedInput = changedBehaviorInputSchema.parse({
comparisons: comparisonsInput,
offset,
limit,
});
const comparisons = parsedInput.comparisons;
const evidence = comparisons.map(parseComparisonEvidence);
assertUnique(
evidence.map(({ evidence_id: id }) => id),
"comparison Evidence",
);
const findings = evidence.flatMap(findingsFor).sort(compareFindings);
const runtimeStatuses = evidence.flatMap(runtimeStatus);
const links = uniqueSorted(
evidence.flatMap((item) => [item.evidence_id, ...item.evidence_links]),
);
if (links.length > 20_100)
throw new TypeError("Changed-behavior Evidence closure exceeds limit");
const page = findings.slice(
parsedInput.offset,
parsedInput.offset + parsedInput.limit,
);
return changedBehaviorResultSchema.parse({
behavior_status: behaviorStatus(runtimeStatuses),
summary: {
observed_changes: count(findings, "observed_change"),
static_candidates: count(findings, "derived_relationship"),
contradictions: count(findings, "contradiction"),
unresolved: count(findings, "unresolved_branch"),
},
findings: {
items: page,
offset: parsedInput.offset,
limit: parsedInput.limit,
total: findings.length,
next_offset:
parsedInput.offset + page.length < findings.length
? parsedInput.offset + page.length
: null,
},
evidence_links: links,
limitations: uniqueSorted([
...evidence.flatMap(({ limitations }) => limitations),
...(evidence.some(
({ operation }) => operation !== "compare_process_captures",
)
? [
"Static differences are behavior candidates, not runtime observations.",
]
: []),
...(runtimeStatuses.length === 0
? ["No process comparison Evidence was supplied."]
: []),
]),
});
};
const parseComparisonEvidence = (input: unknown): Evidence => {
const evidence = parseEvidence(input);
if (!(evidence.operation in EXPECTED_PROVIDERS))
throw new TypeError(
"Changed-behavior analysis requires process, artifact, or function comparison Evidence",
);
const operation = z
.enum([
"compare_process_captures",
"compare_artifacts",
"compare_functions",
])
.parse(evidence.operation);
const expected = EXPECTED_PROVIDERS[operation];
if (
evidence.predicate_type !== expected.predicate ||
evidence.provider.id !== expected.id ||
evidence.provider.name !== expected.name ||
evidence.provider.version !== expected.version ||
evidence.confidence !== "derived" ||
evidence.authority !== "analyst-inference" ||
evidence.subject !== null
)
throw new TypeError(
"Comparison Evidence operation, predicate, and provider disagree",
);
if (evidence.evidence_links.length < 2)
throw new TypeError(
"Comparison Evidence must cite both source observations",
);
assertUnique(evidence.evidence_links, "source Evidence links");
parseResult(evidence);
return evidence;
};
const parseResult = (evidence: Evidence): void => {
if (evidence.operation === "compare_process_captures") {
const result = processCaptureComparisonSchema.parse(
evidence.normalized_result,
);
const dimensions = [
result.terminal,
result.exit,
result.filesystem,
result.protocol,
result.process,
];
const expected = dimensions.includes("truncated")
? "truncated"
: dimensions.some(isChange)
? "changed"
: dimensions.includes("unknown")
? "unknown"
: "unchanged";
if (result.status !== expected)
throw new TypeError(
"Process comparison status contradicts its dimensions",
);
return;
}
if (evidence.operation === "compare_functions") {
const result = functionComparisonResultSchema.parse(
evidence.normalized_result,
);
assertNestedLinks(
evidence,
result.dimensions.flatMap(({ evidence_links: links }) => links),
);
return;
}
const result = artifactComparisonResultSchema.parse(
evidence.normalized_result,
);
if (
result.changes.offset !== 0 ||
result.changes.next_offset !== null ||
result.changes.items.length !== result.changes.total
)
throw new TypeError(
"Changed-behavior analysis requires complete artifact comparison pagination",
);
assertNestedLinks(
evidence,
result.changes.items.flatMap(({ evidence_links: links }) => links),
);
};
const assertNestedLinks = (
evidence: Evidence,
nestedLinks: readonly string[],
): void => {
const topLevel = new Set(evidence.evidence_links);
if (nestedLinks.some((evidenceId) => !topLevel.has(evidenceId)))
throw new TypeError(
"Comparison finding cites Evidence outside its top-level closure",
);
};
const findingsFor = (evidence: Evidence): Finding[] => {
if (evidence.operation === "compare_process_captures")
return processFindings(evidence);
if (evidence.operation === "compare_functions")
return functionFindings(evidence);
return artifactFindings(evidence);
};
const processFindings = (evidence: Evidence): Finding[] => {
const result = processCaptureComparisonSchema.parse(
evidence.normalized_result,
);
const dimensions = [
["terminal", result.terminal, "runtime"],
["exit", result.exit, "runtime"],
["filesystem", result.filesystem, "resource"],
["protocol", result.protocol, "protocol"],
["process", result.process, "runtime"],
] as const;
return dimensions.flatMap(([dimension, status, scope]) =>
status === "unchanged"
? []
: [
makeFinding({
evidence,
dimension,
scope,
status,
classification: isChange(status)
? "observed_change"
: "unresolved_branch",
limitations: result.limitations,
}),
],
);
};
const functionFindings = (evidence: Evidence): Finding[] => {
const result = functionComparisonResultSchema.parse(
evidence.normalized_result,
);
return result.dimensions.flatMap((dimension) =>
dimension.status === "unchanged"
? []
: [
makeFinding({
evidence,
dimension: dimension.dimension,
scope: "static_candidate",
status: dimension.status,
classification:
dimension.status === "unknown" || dimension.status === "truncated"
? "unresolved_branch"
: dimension.conclusion_kind,
limitations: dimension.limitations,
links: dimension.evidence_links,
}),
],
);
};
const artifactFindings = (evidence: Evidence): Finding[] => {
const result = artifactComparisonResultSchema.parse(
evidence.normalized_result,
);
const changes = result.changes.items.map((change) =>
makeFinding({
evidence,
dimension: `artifact:${change.logical_path}`,
scope: "static_candidate",
status: change.classification,
classification:
change.classification === "unknown"
? "unresolved_branch"
: "derived_relationship",
limitations: result.limitations,
links: change.evidence_links,
}),
);
return (result.status === "unknown" || result.status === "truncated") &&
!changes.some(
({ classification }) => classification === "unresolved_branch",
)
? [
...changes,
makeFinding({
evidence,
dimension: "artifact:coverage",
scope: "static_candidate",
status: result.status,
classification: "unresolved_branch",
limitations: result.limitations,
}),
]
: changes;
};
const makeFinding = (input: {
readonly evidence: Evidence;
readonly dimension: string;
readonly scope: Finding["scope"];
readonly status: Finding["status"];
readonly classification: Finding["classification"];
readonly limitations: readonly string[];
readonly links?: readonly string[];
}): Finding =>
findingSchema.parse({
scope: input.scope,
dimension: input.dimension,
classification: input.classification,
status: input.status,
source_comparison_id: input.evidence.evidence_id,
evidence_links: uniqueSorted([
input.evidence.evidence_id,
...(input.links ?? input.evidence.evidence_links),
]),
limitations: uniqueSorted(input.limitations),
});
const runtimeStatus = (evidence: Evidence): RuntimeStatus[] =>
evidence.operation === "compare_process_captures"
? [processCaptureComparisonSchema.parse(evidence.normalized_result).status]
: [];
const behaviorStatus = (
statuses: readonly RuntimeStatus[],
): ChangedBehaviorResult["behavior_status"] => {
if (statuses.length === 0) return "unknown";
if (statuses.includes("truncated")) return "truncated";
if (statuses.includes("unknown")) return "unknown";
return statuses.some(isChange) ? "observed_changed" : "observed_unchanged";
};
const isChange = (status: RuntimeStatus): boolean =>
status === "added" || status === "removed" || status === "changed";
const count = (
findings: readonly Finding[],
classification: Finding["classification"],
): number =>
findings.filter((finding) => finding.classification === classification)
.length;
const compareFindings = (left: Finding, right: Finding): number =>
[left.scope, left.dimension, left.source_comparison_id]
.join("\0")
.localeCompare(
[right.scope, right.dimension, right.source_comparison_id].join("\0"),
"en",
);
const uniqueSorted = (values: readonly string[]): string[] =>
[...new Set(values)].sort((left, right) => left.localeCompare(right, "en"));
const assertUnique = (values: readonly string[], label: string): void => {
if (new Set(values).size !== values.length)
throw new TypeError(`Changed-behavior analysis rejects duplicate ${label}`);
};

272
src/domain/errors.ts
View File

@ -1,6 +1,31 @@
export type AnalysisErrorTag =
| "AnalysisProtocolError"
| "AnalysisInputError"
| "AnalysisOutputError"
| "AnalysisCapabilityUnavailableError"
| "AnalysisCancelledError"
| "AnalysisTimeoutError"
| "ProviderSelectionError"
| "ProviderAdapterError"
| "ArtifactOperationError"
| "ProcessCaptureError"
| "EvidenceIntegrityError"
| "EvidenceLimitError"
| "EvidenceFileError"
| "UnknownRegistryError"
| "HopperTimeoutError"
| "HopperCancelledError"
| "HopperProtocolError"
| "HopperRemoteError"
| "HopperProcessError"
| "HopperStartError"
| "ConfigurationError"
| "NoBinaryOpenError"
| "BinaryTargetError";
/** Base class for expected analysis, provider, and session failures. */
export abstract class AnalysisError extends Error {
abstract readonly _tag: string;
abstract readonly _tag: AnalysisErrorTag;
}
/** Base class for failures produced specifically by the Hopper provider. */
@ -11,9 +36,161 @@ export class AnalysisProtocolError extends AnalysisError {
readonly _tag = "AnalysisProtocolError";
}
/** An evidence bundle or bounded session ledger rejected caller-controlled data. */
export class EvidenceLedgerError extends AnalysisError {
readonly _tag = "EvidenceLedgerError";
/** Caller input failed provider-neutral application parsing. */
export class AnalysisInputError extends AnalysisError {
readonly _tag = "AnalysisInputError";
constructor(
readonly operation: string,
options?: ErrorOptions,
) {
super(`Invalid analysis input for ${operation}`, options);
}
}
/** Provider output failed provider-neutral application parsing. */
export class AnalysisOutputError extends AnalysisError {
readonly _tag = "AnalysisOutputError";
constructor(
readonly operation: string,
readonly reason: string,
options?: ErrorOptions,
) {
super(`Invalid analysis output for ${operation}: ${reason}`, options);
}
}
/** Selected provider cannot execute a declared analysis operation. */
export class AnalysisCapabilityUnavailableError extends AnalysisError {
readonly _tag = "AnalysisCapabilityUnavailableError";
constructor(
readonly providerId: string,
readonly operation: string,
readonly reason: string,
) {
super(`Provider ${providerId} cannot execute ${operation}: ${reason}`);
}
}
/** Caller cancellation won before provider-neutral work completed. */
export class AnalysisCancelledError extends AnalysisError {
readonly _tag = "AnalysisCancelledError";
constructor(readonly operation: string) {
super(`Analysis operation was cancelled: ${operation}`);
}
}
/** Provider-neutral operation exceeded its declared execution deadline. */
export class AnalysisTimeoutError extends AnalysisError {
readonly _tag = "AnalysisTimeoutError";
constructor(
readonly operation: string,
readonly timeoutMs: number,
) {
super(
`Analysis operation timed out after ${String(timeoutMs)}ms: ${operation}`,
);
}
}
/** No configured provider can satisfy a requested operation. */
export class ProviderSelectionError extends AnalysisError {
readonly _tag = "ProviderSelectionError";
constructor(readonly operation: string) {
super(`No configured provider can execute ${operation}`);
}
}
/** A provider adapter failed outside its more precise typed variants. */
export class ProviderAdapterError extends AnalysisError {
readonly _tag = "ProviderAdapterError";
constructor(
readonly providerId: string,
readonly operation: string,
options?: ErrorOptions,
) {
super(`Provider ${providerId} adapter failed during ${operation}`, options);
}
}
/** Artifact inventory or extraction failed a typed safety boundary. */
export class ArtifactOperationError extends AnalysisError {
readonly _tag = "ArtifactOperationError";
constructor(
readonly operation: "inventory_artifact" | "extract_artifact",
readonly reason:
| "cancelled"
| "format"
| "integrity"
| "limit"
| "path"
| "unavailable"
| "io",
) {
super(`Artifact ${operation} failed: ${reason}`);
}
}
/** Evidence identity, schema, or bundle manifests failed integrity checks. */
export class EvidenceIntegrityError extends AnalysisError {
readonly _tag = "EvidenceIntegrityError";
}
/** A bounded evidence ledger cannot accept more records or serialized bytes. */
export class EvidenceLimitError extends AnalysisError {
readonly _tag = "EvidenceLimitError";
constructor(
readonly limit: "records" | "bytes",
readonly maximum: number,
) {
super(`Evidence ledger ${limit} limit exceeded (${String(maximum)})`);
}
}
/** Evidence bundle filesystem access failed within the configured policy. */
export class EvidenceFileError extends AnalysisError {
readonly _tag = "EvidenceFileError";
constructor(
readonly operation: "read" | "write",
readonly reason:
| "disabled"
| "outside-root"
| "not-file"
| "too-large"
| "exists"
| "invalid-json"
| "io",
options?: ErrorOptions,
) {
super(`Evidence bundle ${operation} failed: ${reason}`, options);
}
}
/** Residual-unknown mutation failed a lifecycle, reference, or CAS invariant. */
export class UnknownRegistryError extends AnalysisError {
readonly _tag = "UnknownRegistryError";
constructor(
readonly reason:
| "not-found"
| "already-exists"
| "revision-conflict"
| "invalid-transition"
| "integrity"
| "limit",
options?: ErrorOptions,
) {
super(`Residual unknown registry mutation failed: ${reason}`, options);
}
}
/** Hopper did not respond within the configured operation deadline. */
@ -92,6 +269,91 @@ export class BinaryTargetError extends AnalysisError {
reason: string,
options?: ErrorOptions,
) {
super(`Cannot open app: ${reason}: ${path}`, options);
super(`Cannot open artifact: ${reason}`, options);
}
}
export interface AnalysisErrorProjection {
readonly tag: AnalysisErrorTag;
readonly message: string;
readonly details: Readonly<Record<string, string | number | null>>;
}
/** Project expected failures into exhaustive, secret-safe caller fields. */
export const projectAnalysisError = (
error: AnalysisError,
): AnalysisErrorProjection => {
assertKnownTag(error._tag);
return {
tag: error._tag,
message: error.message,
details: safeDetails(error),
};
};
const safeDetails = (
error: AnalysisError,
): Readonly<Record<string, string | number | null>> => {
if (error instanceof AnalysisInputError)
return { operation: error.operation };
if (error instanceof AnalysisOutputError)
return { operation: error.operation, reason: error.reason };
if (error instanceof AnalysisCapabilityUnavailableError)
return {
providerId: error.providerId,
operation: error.operation,
reason: error.reason,
};
if (error instanceof AnalysisCancelledError)
return { operation: error.operation };
if (error instanceof AnalysisTimeoutError)
return { operation: error.operation, timeoutMs: error.timeoutMs };
if (error instanceof ProviderSelectionError)
return { operation: error.operation };
if (error instanceof ProviderAdapterError)
return { providerId: error.providerId, operation: error.operation };
if (error instanceof ArtifactOperationError)
return { operation: error.operation, reason: error.reason };
if (error instanceof EvidenceLimitError)
return { limit: error.limit, maximum: error.maximum };
if (error instanceof EvidenceFileError)
return { operation: error.operation, reason: error.reason };
if (error instanceof UnknownRegistryError) return { reason: error.reason };
if (error instanceof HopperTimeoutError)
return { timeoutMs: error.timeoutMs };
if (error instanceof HopperRemoteError)
return { code: error.code, safeMessage: error.safeMessage };
if (error instanceof HopperProcessError) return { exitCode: error.exitCode };
return {};
};
const KNOWN_ERROR_TAGS = {
AnalysisProtocolError: true,
AnalysisInputError: true,
AnalysisOutputError: true,
AnalysisCapabilityUnavailableError: true,
AnalysisCancelledError: true,
AnalysisTimeoutError: true,
ProviderSelectionError: true,
ProviderAdapterError: true,
ArtifactOperationError: true,
ProcessCaptureError: true,
EvidenceIntegrityError: true,
EvidenceLimitError: true,
EvidenceFileError: true,
UnknownRegistryError: true,
HopperTimeoutError: true,
HopperCancelledError: true,
HopperProtocolError: true,
HopperRemoteError: true,
HopperProcessError: true,
HopperStartError: true,
ConfigurationError: true,
NoBinaryOpenError: true,
BinaryTargetError: true,
} as const satisfies Readonly<Record<AnalysisErrorTag, true>>;
const assertKnownTag = (tag: AnalysisErrorTag): void => {
if (KNOWN_ERROR_TAGS[tag] !== true)
throw new TypeError("Unknown analysis error tag");
};

70
src/domain/evidence.ts
View File

@ -15,11 +15,32 @@ const providerSchema = z.object({
const subjectSchema = z.object({
name: z.string().min(1),
digest: z.object({ sha256: digestSchema }),
format: z.enum(["hopper", "mach-o", "elf", "pe"]),
format: z.enum([
"hopper",
"mach-o",
"elf",
"pe",
"zip",
"ipa",
"apk",
"asar",
"dmg",
"pkg",
"plist",
"javascript",
"source-map",
"directory",
"file",
"unknown",
"mach-o-universal",
"javascript-bundle",
"entitlements",
]),
architecture: z.enum(["x86", "x86_64", "arm", "arm64"]).nullable(),
local_path: z.string(),
});
const locationSchema = z.discriminatedUnion("kind", [
/** Source location attached to an evidence observation. */
const evidenceLocationSchema = z.discriminatedUnion("kind", [
z.object({ kind: z.literal("address"), address: z.string().min(1) }),
z.object({
kind: z.literal("address-range"),
@ -27,6 +48,12 @@ const locationSchema = z.discriminatedUnion("kind", [
end: z.string().min(1),
}),
z.object({ kind: z.literal("artifact-path"), path: z.string().min(1) }),
z.object({ kind: z.literal("file-offset"), offset: z.number().int().min(0) }),
z.object({
kind: z.literal("file-offset-range"),
start: z.number().int().min(0),
end: z.number().int().min(0),
}),
]);
const evidenceAuthoritySchema = z.enum([
@ -53,18 +80,26 @@ export const evidenceSchema = z.object({
predicate_type: z.string().min(1),
operation: z.string().min(1),
parameters: z.record(z.string(), jsonValueSchema),
result: jsonValueSchema,
raw_payload_sha256: digestSchema.nullable(),
raw_result: jsonValueSchema.nullable(),
normalized_result: jsonValueSchema,
confidence: z.enum(["observed", "derived", "inferred"]),
authority: evidenceAuthoritySchema,
environment: executionEnvironmentSchema.nullable(),
limitations: z.array(z.string()),
locations: z.array(locationSchema),
locations: z.array(evidenceLocationSchema),
evidence_links: z.array(z.string().regex(/^ev_[a-f0-9]{64}$/u)),
});
export type Evidence = z.infer<typeof evidenceSchema>;
type EvidenceLocation = z.infer<typeof locationSchema>;
export type EvidenceLocation = z.infer<typeof evidenceLocationSchema>;
/** Minimal immutable local artifact identity accepted by Evidence v2. */
export interface EvidenceSubjectTarget {
readonly path: string;
readonly sha256: string;
readonly format: z.infer<typeof subjectSchema>["format"];
readonly architecture?: "x86" | "x86_64" | "arm" | "arm64";
}
type EvidenceAuthority = z.infer<typeof evidenceAuthoritySchema>;
type ExecutionEnvironment = z.infer<typeof executionEnvironmentSchema>;
@ -79,7 +114,7 @@ export interface EvidenceObservation {
readonly operation: string;
readonly parameters: Readonly<Record<string, JsonValue>>;
readonly result: JsonValue;
readonly redactedRawPayload?: JsonValue;
readonly rawResult?: JsonValue;
readonly confidence?: "observed" | "derived" | "inferred";
readonly authority?: EvidenceAuthority;
readonly environment?: ExecutionEnvironment | null;
@ -115,7 +150,8 @@ const semanticProjection = (
predicate_type: evidence.predicate_type,
operation: evidence.operation,
parameters: evidence.parameters,
result: evidence.result,
raw_result: evidence.raw_result,
normalized_result: evidence.normalized_result,
confidence: evidence.confidence,
authority: evidence.authority,
environment: evidence.environment,
@ -130,6 +166,15 @@ const computeEvidenceId = (evidence: Omit<Evidence, "evidence_id">): string =>
/** Parse evidence and reject a syntactically valid but tampered semantic ID. */
export const parseEvidence = (input: unknown): Evidence => {
if (
typeof input === "object" &&
input !== null &&
"schema_version" in input &&
input.schema_version !== 2
)
throw new TypeError(
`Unsupported evidence schema_version ${String(input.schema_version)}; Evidence v1 is not accepted. Produce Evidence v2.`,
);
const evidence = evidenceSchema.parse(input);
const { evidence_id: evidenceId, ...withoutId } = evidence;
if (computeEvidenceId(withoutId) !== evidenceId)
@ -141,7 +186,7 @@ export const parseEvidence = (input: unknown): Evidence => {
/** Build deterministic Evidence v2 from an immutable artifact subject. */
export const createEvidence = (
target: BinaryTarget | undefined,
target: EvidenceSubjectTarget | BinaryTarget | undefined,
provider: EvidenceProvider,
observation: EvidenceObservation,
): Evidence => {
@ -173,7 +218,8 @@ export const createEvidence = (
predicate_type: observation.predicateType ?? "rea.analysis/v2",
operation: observation.operation,
parameters: observation.parameters,
result: observation.result,
raw_result: observation.rawResult ?? null,
normalized_result: observation.result,
confidence: observation.confidence ?? "observed",
authority: observation.authority ?? "shipped-artifact",
environment: observation.environment ?? null,
@ -190,9 +236,5 @@ export const createEvidence = (
...semantic,
evidence_id: `ev_${sha256(canonicalJson(semantic))}`,
subject,
raw_payload_sha256:
observation.redactedRawPayload === undefined
? null
: sha256(canonicalJson(observation.redactedRawPayload)),
});
};

262
src/domain/evidenceBundle.ts
View File

@ -1,20 +1,268 @@
import { z } from "zod";
import canonicalize from "canonicalize";
import { evidenceSchema, type Evidence } from "./evidence.js";
import { evidenceSchema, parseEvidence, type Evidence } from "./evidence.js";
import {
residualUnknownSchema,
type ResidualUnknown,
} from "./residualUnknown.js";
const artifactManifestSchema = evidenceSchema.shape.subject.unwrap().pick({
digest: true,
format: true,
architecture: true,
});
const providerManifestSchema = evidenceSchema.shape.provider;
const environmentManifestSchema = evidenceSchema.shape.environment.unwrap();
const scenarioManifestSchema = z.object({
evidence_id: evidenceSchema.shape.evidence_id,
operation: z.string().min(1),
authority: z.literal("controlled-replay"),
});
const captureManifestSchema = z.object({
evidence_id: evidenceSchema.shape.evidence_id,
predicate_type: z.string().min(1),
});
export const evidenceBundleSchema = z.object({
bundle_version: z.literal(1),
bundle_version: z.literal(2),
artifacts: z.array(artifactManifestSchema),
providers: z.array(providerManifestSchema),
environments: z.array(environmentManifestSchema),
scenarios: z.array(scenarioManifestSchema),
captures: z.array(captureManifestSchema),
unknowns: z.array(residualUnknownSchema),
records: z.array(evidenceSchema),
});
export type EvidenceBundle = z.infer<typeof evidenceBundleSchema>;
export interface EvidenceFilePolicy {
readonly roots: readonly string[];
readonly maxBytes: number;
readonly maxDepth: number;
readonly maxStringLength: number;
readonly maxNodes: number;
}
/** Project records into a deterministic bundle whose order has no semantics. */
export const createEvidenceBundle = (
records: readonly Evidence[],
): EvidenceBundle => ({
bundle_version: 1,
records: [...records].sort((left, right) =>
unknowns: readonly ResidualUnknown[] = [],
): EvidenceBundle => {
const sortedRecords = [...records].sort((left, right) =>
left.evidence_id.localeCompare(right.evidence_id),
),
});
);
return {
bundle_version: 2,
artifacts: uniqueSorted(
sortedRecords.flatMap(({ subject }) =>
subject === null
? []
: [
{
digest: subject.digest,
format: subject.format,
architecture: subject.architecture,
},
],
),
),
providers: uniqueSorted(sortedRecords.map(({ provider }) => provider)),
environments: uniqueSorted(
sortedRecords.flatMap(({ environment }) =>
environment === null ? [] : [environment],
),
),
scenarios: sortedRecords.flatMap((evidence) =>
evidence.authority === "controlled-replay"
? [
{
evidence_id: evidence.evidence_id,
operation: evidence.operation,
authority: evidence.authority,
},
]
: [],
),
captures: sortedRecords.flatMap((evidence) =>
evidence.predicate_type.startsWith("rea.process-capture/")
? [
{
evidence_id: evidence.evidence_id,
predicate_type: evidence.predicate_type,
},
]
: [],
),
unknowns: [...unknowns].sort(
(left, right) =>
left.unknown_id.localeCompare(right.unknown_id) ||
left.revision - right.revision,
),
records: sortedRecords,
};
};
/** Parse records, verify semantic IDs, and reject inconsistent manifests. */
export const parseEvidenceBundle = (input: unknown): EvidenceBundle => {
const parsed = evidenceBundleSchema.parse(input);
const recordIds = parsed.records.map(({ evidence_id: id }) => id);
if (new Set(recordIds).size !== recordIds.length)
throw new TypeError("Evidence bundle contains duplicate record IDs");
validateUnknownGraph(parsed.unknowns, parsed.records);
const canonical = createEvidenceBundle(
parsed.records.map(parseEvidence),
parsed.unknowns,
);
if (JSON.stringify(parsed) !== JSON.stringify(canonical))
throw new TypeError("Evidence bundle manifests are not canonical");
return canonical;
};
const validateUnknownGraph = (
unknowns: readonly ResidualUnknown[],
records: readonly Evidence[],
): void => {
const evidenceById = new Map(
records.map((record) => [record.evidence_id, record]),
);
const histories = new Map<string, ResidualUnknown[]>();
for (const unknown of unknowns) {
const history = histories.get(unknown.unknown_id) ?? [];
history.push(unknown);
histories.set(unknown.unknown_id, history);
}
const unknownById = new Map<string, ResidualUnknown>();
for (const [id, unordered] of histories) {
const history = [...unordered].sort(
(left, right) => left.revision - right.revision,
);
for (const [index, revision] of history.entries()) {
if (revision.revision !== index + 1)
throw new TypeError(
`Residual unknown ${id} revision history has a gap`,
);
if (
index > 0 &&
revision.previous_revision_digest !==
history[index - 1]?.revision_digest
)
throw new TypeError(`Residual unknown ${id} revision chain is broken`);
}
const head = history.at(-1);
if (head === undefined)
throw new TypeError("Residual unknown revision history is empty");
unknownById.set(id, head);
}
for (const unknown of unknowns) {
const referencedEvidence = [
...unknown.supporting_evidence_ids,
...unknown.contradicting_evidence_ids,
...unknown.mutation_evidence_ids,
...(unknown.resolution?.evidence_ids ?? []),
];
for (const evidenceId of referencedEvidence)
if (!evidenceById.has(evidenceId))
throw new TypeError(
`Residual unknown references missing evidence ${evidenceId}`,
);
for (const relationship of unknown.relationships)
if (!unknownById.has(relationship.unknown_id))
throw new TypeError(
`Residual unknown references missing unknown ${relationship.unknown_id}`,
);
if (unknown.resolution?.disposition === "verified")
validateVerifiedResolution(unknown, evidenceById);
}
rejectDependencyCycles(unknownById);
};
const validateVerifiedResolution = (
unknown: ResidualUnknown,
evidenceById: ReadonlyMap<string, Evidence>,
): void => {
if (unknown.contradicting_evidence_ids.length > 0)
throw new TypeError(
`Residual unknown ${unknown.unknown_id} retains contradicting evidence`,
);
const qualifies = unknown.resolution?.evidence_ids.some((evidenceId) => {
const evidence = evidenceById.get(evidenceId);
return (
unknown.supporting_evidence_ids.includes(evidenceId) &&
evidence !== undefined &&
evidence.predicate_type !== "rea.residual-unknown-mutation/v1" &&
evidenceQualifies(unknown, evidence)
);
});
if (qualifies !== true)
throw new TypeError(
`Residual unknown ${unknown.unknown_id} has no qualifying resolution evidence`,
);
};
const evidenceQualifies = (
unknown: ResidualUnknown,
evidence: Evidence,
): boolean => {
if (
unknown.required_authority !== null &&
evidence.authority !== unknown.required_authority
)
return false;
const confidenceRank = { inferred: 1, derived: 2, observed: 3 } as const;
if (
confidenceRank[evidence.confidence] <
confidenceRank[unknown.required_confidence]
)
return false;
const requirement = unknown.required_environment;
if (requirement === null) return true;
if (evidence.environment === null) return false;
return (
(requirement.id === null || requirement.id === evidence.environment.id) &&
(requirement.platform === null ||
requirement.platform === evidence.environment.platform) &&
(requirement.architecture === null ||
requirement.architecture === evidence.environment.architecture) &&
(requirement.isolation === null ||
requirement.isolation === evidence.environment.isolation)
);
};
const rejectDependencyCycles = (
unknownById: ReadonlyMap<string, ResidualUnknown>,
): void => {
const visited = new Set<string>();
const active = new Set<string>();
const visit = (id: string): void => {
if (active.has(id))
throw new TypeError("Residual unknown dependency graph contains a cycle");
if (visited.has(id)) return;
active.add(id);
const unknown = unknownById.get(id);
if (unknown === undefined)
throw new TypeError("Residual unknown dependency graph is inconsistent");
for (const relationship of unknown.relationships)
if (relationship.type === "depends-on") visit(relationship.unknown_id);
active.delete(id);
visited.add(id);
};
for (const id of unknownById.keys()) visit(id);
};
/** Encode a validated bundle as byte-stable RFC 8785 canonical JSON. */
export const serializeEvidenceBundle = (bundle: EvidenceBundle): string => {
const serialized = canonicalize(parseEvidenceBundle(bundle));
if (serialized === undefined)
throw new TypeError("Evidence bundle canonicalization failed");
return serialized;
};
const uniqueSorted = <Value>(values: readonly Value[]): Value[] => {
const unique = new Map<string, Value>();
for (const value of values) unique.set(JSON.stringify(value), value);
return [...unique.entries()]
.sort(([left], [right]) => left.localeCompare(right))
.map(([, value]) => value);
};

509
src/domain/functionComparison.ts Normal file
View File

@ -0,0 +1,509 @@
import { diffLines } from "diff";
import {
parseFunctionEvidence,
type FunctionCollection,
type FunctionSnapshot,
} from "./functionDossierEvidence.js";
import {
commentProjection,
combineCoverage,
functionMatch,
identityProjection,
isAutoName,
normalizeCfg,
project,
referenceProjection,
sorted,
stringAndNameProjection,
} from "./functionComparisonNormalization.js";
import {
functionComparisonResultSchema,
type DimensionName,
type FunctionComparisonResult,
type FunctionDimension,
} from "./functionComparisonSchemas.js";
import {
canonicalJson,
dimensionResult,
overallStatus,
summarize,
unresolvedDimension as buildUnresolvedDimension,
} from "./functionComparisonResults.js";
export {
functionComparisonInputSchema,
functionComparisonResultSchema,
} from "./functionComparisonSchemas.js";
const unresolvedDimension = (
...[dimension, status, links, leftCount, rightCount, limitations]: readonly [
DimensionName,
"truncated" | "unknown",
readonly string[],
number | null,
number | null,
readonly string[],
]
): FunctionDimension =>
buildUnresolvedDimension({
dimension,
status,
links,
leftCount,
rightCount,
limitations,
});
/** Compare two explicit function Evidence page sets without fuzzy matching. */
export const compareFunctions = (
leftInput: unknown,
rightInput: unknown,
offset: number,
limit: number,
): FunctionComparisonResult => {
const left = parseFunctionEvidence(leftInput);
const right = parseFunctionEvidence(rightInput);
const links = [
...left.evidence.map(({ evidence_id: id }) => id),
...right.evidence.map(({ evidence_id: id }) => id),
];
const providersDiffer =
canonicalJson(left.provider) !== canonicalJson(right.provider);
const dimensions = compareDimensions(left, right, links, providersDiffer);
const match = functionMatch(left, right);
const changes = dimensions.filter(({ status }) => status !== "unchanged");
const page = changes.slice(offset, offset + limit);
return functionComparisonResultSchema.parse({
status: overallStatus(dimensions, match.status),
function_match: match,
left_subject_sha256: left.subject.digest.sha256,
right_subject_sha256: right.subject.digest.sha256,
summary: summarize(dimensions),
dimensions,
changes: {
items: page,
offset,
limit,
total: changes.length,
next_offset:
offset + page.length < changes.length ? offset + page.length : null,
},
limitations: [
...new Set([
...left.limitations.map((item) => `Left: ${item}`),
...right.limitations.map((item) => `Right: ${item}`),
...(providersDiffer
? [
"Provider-specific pseudocode and assembly representations were not equated.",
]
: []),
]),
].sort((a, b) => a.localeCompare(b)),
});
};
const compareDimensions = (
left: FunctionSnapshot,
right: FunctionSnapshot,
links: readonly string[],
providersDiffer: boolean,
): FunctionDimension[] => [
compareIdentity(left, right, links, providersDiffer),
compareText(
"pseudocode",
left.pseudocode.text,
right.pseudocode.text,
left.pseudocode.complete,
right.pseudocode.complete,
links,
providersDiffer,
),
compareText(
"assembly",
left.collections.assembly.items.join("\n"),
right.collections.assembly.items.join("\n"),
left.collections.assembly.complete,
right.collections.assembly.complete,
links,
providersDiffer,
left.collections.assembly.truncated || right.collections.assembly.truncated,
"Assembly is opaque provider text; relocation normalization is unavailable.",
),
compareComments(left, right, links, providersDiffer),
compareCalls(left, right, links, providersDiffer),
compareReferences(left, right, links, providersDiffer),
compareStringsAndNames(left, right, links, providersDiffer),
compareCfg(left, right, links, providersDiffer),
];
const compareText = (
...[
dimension,
left,
right,
leftComplete,
rightComplete,
links,
providersDiffer,
explicitTruncation,
limitation,
]: readonly [
"pseudocode" | "assembly",
string,
string,
boolean,
boolean,
readonly string[],
boolean,
boolean?,
string?,
]
): FunctionDimension => {
const truncated = explicitTruncation ?? (!leftComplete || !rightComplete);
if (!leftComplete || !rightComplete || providersDiffer)
return unresolvedDimension(
dimension,
truncated ? "truncated" : "unknown",
links,
left.length,
right.length,
[
...(limitation === undefined ? [] : [limitation]),
...(providersDiffer
? ["Exact text comparison requires one provider identity."]
: []),
],
);
const changes = diffLines(left, right, {
timeout: 100,
maxEditLength: 10_000,
});
if (changes === undefined)
return unresolvedDimension(
dimension,
"unknown",
links,
left.length,
right.length,
["Bounded line diff exceeded its time or edit-distance limit."],
);
const delta = changes.reduce(
(summary, change) => ({
added_lines:
summary.added_lines + (change.added ? (change.count ?? 0) : 0),
removed_lines:
summary.removed_lines + (change.removed ? (change.count ?? 0) : 0),
hunks: summary.hunks + (change.added || change.removed ? 1 : 0),
}),
{ added_lines: 0, removed_lines: 0, hunks: 0 },
);
return dimensionResult({
dimension,
status: delta.hunks === 0 ? "unchanged" : "changed",
left,
right,
links,
providersDiffer,
leftCount: [...left].length,
rightCount: [...right].length,
textDelta: delta,
limitations: limitation === undefined ? [] : [limitation],
});
};
const compareCollections = (
...[
dimension,
left,
right,
leftCoverage,
rightCoverage,
links,
providersDiffer,
]: readonly [
DimensionName,
readonly unknown[],
readonly unknown[],
Pick<FunctionCollection, "complete" | "truncated">,
Pick<FunctionCollection, "complete" | "truncated">,
readonly string[],
boolean,
]
): FunctionDimension => {
if (!leftCoverage.complete || !rightCoverage.complete)
return unresolvedDimension(
dimension,
leftCoverage.truncated || rightCoverage.truncated
? "truncated"
: "unknown",
links,
left.length,
right.length,
[],
);
return compareValues(
dimension,
sorted(left),
sorted(right),
links,
true,
true,
providersDiffer,
);
};
const compareComments = (
left: FunctionSnapshot,
right: FunctionSnapshot,
links: readonly string[],
providersDiffer: boolean,
): FunctionDimension => {
const leftValues = commentProjection(left);
const rightValues = commentProjection(right);
if (leftValues === null || rightValues === null)
return unresolvedDimension(
"comments",
"unknown",
links,
left.collections.comments.items.length,
right.collections.comments.items.length,
["Comment locations could not be normalized relative to the function."],
);
return compareCollections(
"comments",
leftValues,
rightValues,
left.collections.comments,
right.collections.comments,
links,
providersDiffer,
);
};
const compareStringsAndNames = (
left: FunctionSnapshot,
right: FunctionSnapshot,
links: readonly string[],
providersDiffer: boolean,
): FunctionDimension => {
const leftValues = stringAndNameProjection(left);
const rightValues = stringAndNameProjection(right);
if (leftValues === null || rightValues === null)
return unresolvedDimension("strings_names", "unknown", links, null, null, [
"Reference source locations could not be normalized to function offsets.",
]);
return compareCollections(
"strings_names",
leftValues,
rightValues,
combineCoverage(
left.collections.referenced_strings,
left.collections.referenced_names,
),
combineCoverage(
right.collections.referenced_strings,
right.collections.referenced_names,
),
links,
providersDiffer,
);
};
const compareReferences = (
left: FunctionSnapshot,
right: FunctionSnapshot,
links: readonly string[],
providersDiffer: boolean,
): FunctionDimension => {
const leftCoverage = combineCoverage(
left.collections.incoming_references,
left.collections.outgoing_references,
);
const rightCoverage = combineCoverage(
right.collections.incoming_references,
right.collections.outgoing_references,
);
if (!leftCoverage.complete || !rightCoverage.complete)
return unresolvedDimension(
"references",
leftCoverage.truncated || rightCoverage.truncated
? "truncated"
: "unknown",
links,
leftCoverage.items.length,
rightCoverage.items.length,
[],
);
const leftProjection = referenceProjection(left);
const rightProjection = referenceProjection(right);
if (leftProjection.length === 0 && rightProjection.length === 0)
return compareValues(
"references",
leftProjection,
rightProjection,
links,
true,
true,
providersDiffer,
);
if (canonicalJson(leftProjection) !== canonicalJson(rightProjection))
return dimensionResult({
dimension: "references",
status: "changed",
left: leftProjection,
right: rightProjection,
links,
providersDiffer,
leftCount: leftProjection.length,
rightCount: rightProjection.length,
textDelta: null,
limitations: [
"Reference endpoints differ; exact reference kinds remain unavailable.",
],
});
return unresolvedDimension(
"references",
"unknown",
links,
leftProjection.length,
rightProjection.length,
["Reference kinds are unavailable in the current dossier contract."],
);
};
const compareIdentity = (
left: FunctionSnapshot,
right: FunctionSnapshot,
links: readonly string[],
providersDiffer: boolean,
): FunctionDimension => {
if (isAutoName(left.procedure.name) || isAutoName(right.procedure.name))
return unresolvedDimension("identity", "unknown", links, null, null, [
"Address-derived function names are not stable cross-version identity.",
]);
return compareValues(
"identity",
identityProjection(left),
identityProjection(right),
links,
true,
true,
providersDiffer,
);
};
const compareCalls = (
left: FunctionSnapshot,
right: FunctionSnapshot,
links: readonly string[],
providersDiffer: boolean,
): FunctionDimension => {
const leftValues = callProjection(left);
const rightValues = callProjection(right);
if ([...leftValues, ...rightValues].some(({ name }) => isAutoName(name)))
return unresolvedDimension(
"calls",
"unknown",
links,
leftValues.length,
rightValues.length,
["Address-derived callee or caller names cannot be matched safely."],
);
return compareCollections(
"calls",
leftValues,
rightValues,
combineCoverage(left.collections.callers, left.collections.callees),
combineCoverage(right.collections.callers, right.collections.callees),
links,
providersDiffer,
);
};
const callProjection = (snapshot: FunctionSnapshot) => [
...project(snapshot.collections.callers, ({ name }) => ({
direction: "in" as const,
name,
})),
...project(snapshot.collections.callees, ({ name }) => ({
direction: "out" as const,
name,
})),
];
const compareCfg = (
left: FunctionSnapshot,
right: FunctionSnapshot,
links: readonly string[],
providersDiffer: boolean,
): FunctionDimension => {
const leftBlocks = left.collections.basic_blocks;
const rightBlocks = right.collections.basic_blocks;
if (!leftBlocks.complete || !rightBlocks.complete)
return unresolvedDimension(
"cfg",
leftBlocks.truncated || rightBlocks.truncated ? "truncated" : "unknown",
links,
leftBlocks.items.length,
rightBlocks.items.length,
[],
);
const leftGraph = normalizeCfg(leftBlocks.items);
const rightGraph = normalizeCfg(rightBlocks.items);
if (leftGraph === null || rightGraph === null)
return unresolvedDimension(
"cfg",
"unknown",
links,
leftBlocks.items.length,
rightBlocks.items.length,
["CFG addresses could not be normalized to local block indices."],
);
return compareValues(
"cfg",
leftGraph,
rightGraph,
links,
true,
true,
providersDiffer,
);
};
const compareValues = (
...[
dimension,
left,
right,
links,
leftComplete,
rightComplete,
providersDiffer,
]: readonly [
DimensionName,
unknown,
unknown,
readonly string[],
boolean,
boolean,
boolean,
]
): FunctionDimension => {
if (!leftComplete || !rightComplete)
return unresolvedDimension(dimension, "unknown", links, null, null, []);
const leftJson = canonicalJson(left);
const rightJson = canonicalJson(right);
return dimensionResult({
dimension,
status: leftJson === rightJson ? "unchanged" : "changed",
left,
right,
links,
providersDiffer,
leftCount: Array.isArray(left) ? left.length : null,
rightCount: Array.isArray(right) ? right.length : null,
textDelta: null,
limitations: [],
});
};

171
src/domain/functionComparisonNormalization.ts Normal file
View File

@ -0,0 +1,171 @@
import canonicalize from "canonicalize";
import type {
FunctionCollection,
FunctionSnapshot,
} from "./functionDossierEvidence.js";
import type { FunctionComparisonResult } from "./functionComparisonSchemas.js";
export const identityProjection = (snapshot: FunctionSnapshot) => ({
name: snapshot.procedure.name,
signature: snapshot.procedure.signature,
locals: snapshot.procedure.locals
.map(({ description }) => description)
.sort(),
});
export const functionMatch = (
left: FunctionSnapshot,
right: FunctionSnapshot,
): FunctionComparisonResult["function_match"] => {
if (
!isAutoName(left.procedure.name) &&
left.procedure.name === right.procedure.name
)
return {
status: "matched",
method: "symbol",
left_name: left.procedure.name,
right_name: right.procedure.name,
};
return {
status:
left.procedure.name === right.procedure.name ? "ambiguous" : "mismatched",
method: "explicit",
left_name: left.procedure.name,
right_name: right.procedure.name,
};
};
export const normalizeCfg = (
blocks: FunctionSnapshot["collections"]["basic_blocks"]["items"],
): readonly unknown[] | null => {
const parsed = blocks.map((block) => ({
block,
start: parseAddress(block.start),
end: parseAddress(block.end),
}));
if (parsed.some(({ start, end }) => start === null || end === null))
return null;
const ordered = parsed.sort((left, right) =>
(left.start ?? 0n) < (right.start ?? 0n) ? -1 : 1,
);
if (new Set(ordered.map(({ block }) => block.start)).size !== ordered.length)
return null;
if (ordered.some(({ start, end }) => (end ?? 0n) <= (start ?? 0n)))
return null;
const indices = new Map(
ordered.map(({ block }, index) => [block.start, index]),
);
const graph = [];
for (const { block, start, end } of ordered) {
const successors = block.successors.map((address) => indices.get(address));
if (successors.some((index) => index === undefined)) return null;
graph.push({
size: String((end ?? 0n) - (start ?? 0n)),
successors: successors.sort((left, right) => (left ?? 0) - (right ?? 0)),
});
}
return graph;
};
export const referenceProjection = (snapshot: FunctionSnapshot) =>
sorted([
...snapshot.collections.incoming_references.items.map((item) => ({
direction: "in",
source: item.source_procedure?.name ?? null,
target: item.target_procedure?.name ?? null,
})),
...snapshot.collections.outgoing_references.items.map((item) => ({
direction: "out",
source: item.source_procedure?.name ?? null,
target: item.target_procedure?.name ?? null,
})),
]);
export const commentProjection = (
snapshot: FunctionSnapshot,
): readonly unknown[] | null => {
const values = snapshot.collections.comments.items.map(
({ address, kind, text }) => ({
offset: relativeAddress(address, snapshot.procedure.address),
kind,
text,
}),
);
return values.some(({ offset }) => offset === null) ? null : sorted(values);
};
export const stringAndNameProjection = (
snapshot: FunctionSnapshot,
): readonly unknown[] | null => {
const values = [
...snapshot.collections.referenced_strings.items.map(
({ source_address: sourceAddress, value }) => ({
kind: "string",
source_offset: relativeAddress(
sourceAddress,
snapshot.procedure.address,
),
value,
}),
),
...snapshot.collections.referenced_names.items.map(
({ source_address: sourceAddress, value }) => ({
kind: "name",
source_offset: relativeAddress(
sourceAddress,
snapshot.procedure.address,
),
value,
}),
),
];
return values.some(({ source_offset: offset }) => offset === null)
? null
: sorted(values);
};
export const sorted = (values: readonly unknown[]): readonly unknown[] =>
[...values].sort((left, right) =>
canonicalJson(left).localeCompare(canonicalJson(right)),
);
export const combineCoverage = <Item>(
left: FunctionCollection<Item>,
right: FunctionCollection<Item>,
): FunctionCollection<Item> => ({
items: [...left.items, ...right.items],
total:
left.total === null || right.total === null
? null
: left.total + right.total,
complete: left.complete && right.complete,
truncated: left.truncated || right.truncated,
});
export const project = <Input, Output>(
collection: FunctionCollection<Input>,
mapper: (item: Input) => Output,
): Output[] => collection.items.map(mapper);
export const isAutoName = (name: string): boolean =>
/^(?:sub_[0-9a-f]+|fcn\.)/iu.test(name);
const parseAddress = (value: string): bigint | null =>
/^0x[0-9a-f]+$/iu.test(value) ? BigInt(value) : null;
const relativeAddress = (value: string, base: string): string | null => {
const parsed = parseAddress(value);
const parsedBase = parseAddress(base);
return parsed === null || parsedBase === null || parsed < parsedBase
? null
: String(parsed - parsedBase);
};
const canonicalJson = (value: unknown): string => {
const encoded = canonicalize(value);
if (encoded === undefined)
throw new TypeError("Function normalization could not canonicalize data");
return encoded;
};

95
src/domain/functionComparisonResults.ts Normal file
View File

@ -0,0 +1,95 @@
import { createHash } from "node:crypto";
import canonicalize from "canonicalize";
import type {
DimensionName,
FunctionComparisonResult,
FunctionDimension,
} from "./functionComparisonSchemas.js";
export interface DimensionResultInput {
readonly dimension: DimensionName;
readonly status: "unchanged" | "changed";
readonly left: unknown;
readonly right: unknown;
readonly links: readonly string[];
readonly providersDiffer: boolean;
readonly leftCount: number | null;
readonly rightCount: number | null;
readonly textDelta: FunctionDimension["text_delta"];
readonly limitations: readonly string[];
}
export const dimensionResult = (
input: DimensionResultInput,
): FunctionDimension => ({
dimension: input.dimension,
status: input.status,
left_digest: digest(input.left),
right_digest: digest(input.right),
left_count: input.leftCount,
right_count: input.rightCount,
text_delta: input.textDelta,
conclusion_kind:
input.status === "changed" && input.providersDiffer
? "contradiction"
: "derived_relationship",
evidence_links: [...input.links],
limitations: [...input.limitations],
});
export const unresolvedDimension = (input: {
readonly dimension: DimensionName;
readonly status: "truncated" | "unknown";
readonly links: readonly string[];
readonly leftCount: number | null;
readonly rightCount: number | null;
readonly limitations: readonly string[];
}): FunctionDimension => ({
dimension: input.dimension,
status: input.status,
left_digest: null,
right_digest: null,
left_count: input.leftCount,
right_count: input.rightCount,
text_delta: null,
conclusion_kind: "unresolved_branch",
evidence_links: [...input.links],
limitations: [...input.limitations],
});
export const overallStatus = (
dimensions: readonly FunctionDimension[],
match: FunctionComparisonResult["function_match"]["status"],
): FunctionComparisonResult["status"] => {
if (dimensions.some(({ status }) => status === "truncated"))
return "truncated";
if (
match === "ambiguous" ||
match === "unknown" ||
dimensions.some(({ status }) => status === "unknown")
)
return "unknown";
if (match === "mismatched") return "changed";
return dimensions.some(({ status }) => status === "changed")
? "changed"
: "unchanged";
};
export const summarize = (dimensions: readonly FunctionDimension[]) => ({
unchanged: dimensions.filter(({ status }) => status === "unchanged").length,
changed: dimensions.filter(({ status }) => status === "changed").length,
truncated: dimensions.filter(({ status }) => status === "truncated").length,
unknown: dimensions.filter(({ status }) => status === "unknown").length,
});
export const canonicalJson = (value: unknown): string => {
const encoded = canonicalize(value);
if (encoded === undefined)
throw new TypeError("Function comparison could not canonicalize data");
return encoded;
};
const digest = (value: unknown): string =>
createHash("sha256").update(canonicalJson(value)).digest("hex");

94
src/domain/functionComparisonSchemas.ts Normal file
View File

@ -0,0 +1,94 @@
import { z } from "zod";
import { evidenceSchema } from "./evidence.js";
const evidenceInput = z.union([
evidenceSchema,
z.array(evidenceSchema).min(1).max(100),
]);
const comparisonStatusSchema = z.enum([
"unchanged",
"added",
"removed",
"changed",
"truncated",
"unknown",
]);
const dimensionNameSchema = z.enum([
"identity",
"pseudocode",
"assembly",
"comments",
"calls",
"references",
"strings_names",
"cfg",
]);
const evidenceIdSchema = z.string().regex(/^ev_[a-f0-9]{64}$/u);
const digestSchema = z.string().regex(/^[a-f0-9]{64}$/u);
/** Strict bounded input for explicit function-to-function comparison. */
export const functionComparisonInputSchema = z.object({
left: evidenceInput,
right: evidenceInput,
offset: z.number().int().min(0).default(0),
limit: z.number().int().min(1).max(100).default(100),
unknown_registry_approved: z.literal(true).optional(),
});
const textDeltaSchema = z.object({
added_lines: z.number().int().min(0),
removed_lines: z.number().int().min(0),
hunks: z.number().int().min(0),
});
const functionDimensionSchema = z.object({
dimension: dimensionNameSchema,
status: comparisonStatusSchema,
left_digest: digestSchema.nullable(),
right_digest: digestSchema.nullable(),
left_count: z.number().int().min(0).nullable(),
right_count: z.number().int().min(0).nullable(),
text_delta: textDeltaSchema.nullable(),
conclusion_kind: z.enum([
"derived_relationship",
"contradiction",
"unresolved_branch",
]),
evidence_links: z.array(evidenceIdSchema).min(2).max(200),
limitations: z.array(z.string()),
});
/** Deterministic dimension-classified function comparison Evidence payload. */
export const functionComparisonResultSchema = z.object({
status: comparisonStatusSchema,
function_match: z.object({
status: z.enum(["matched", "mismatched", "ambiguous", "unknown"]),
method: z.enum(["explicit", "symbol"]),
left_name: z.string(),
right_name: z.string(),
}),
left_subject_sha256: digestSchema,
right_subject_sha256: digestSchema,
summary: z.object({
unchanged: z.number().int().min(0),
changed: z.number().int().min(0),
truncated: z.number().int().min(0),
unknown: z.number().int().min(0),
}),
dimensions: z.array(functionDimensionSchema).length(8),
changes: z.object({
items: z.array(functionDimensionSchema).max(100),
offset: z.number().int().min(0),
limit: z.number().int().min(1).max(100),
total: z.number().int().min(0),
next_offset: z.number().int().min(0).nullable(),
}),
limitations: z.array(z.string()),
});
export type FunctionComparisonResult = z.infer<
typeof functionComparisonResultSchema
>;
export type FunctionDimension = z.infer<typeof functionDimensionSchema>;
export type DimensionName = z.infer<typeof dimensionNameSchema>;

261
src/domain/functionDossierEvidence.ts Normal file
View File

@ -0,0 +1,261 @@
import canonicalize from "canonicalize";
import { enhancedInputSchemas } from "../contracts/enhancedInputs.js";
import { parseEvidence, type Evidence } from "./evidence.js";
import { functionDossierSchema, type FunctionDossier } from "./hopperValues.js";
const MAX_PAGES = 100;
const MAX_ITEMS = 50_000;
const MAX_PSEUDOCODE_CHARS = 2_000_000;
const DOSSIER_COLLECTION_FIELDS = [
"assembly",
"comments",
"callers",
"callees",
"incoming_references",
"outgoing_references",
"referenced_strings",
"referenced_names",
"basic_blocks",
] as const;
type DossierCollectionField = (typeof DOSSIER_COLLECTION_FIELDS)[number];
type DossierItem<Field extends DossierCollectionField> =
FunctionDossier[Field]["items"][number];
export interface FunctionCollection<Item = unknown> {
readonly items: readonly Item[];
readonly total: number | null;
readonly complete: boolean;
readonly truncated: boolean;
}
type FunctionCollections = {
readonly [Field in DossierCollectionField]: FunctionCollection<
DossierItem<Field>
>;
};
/** Verified, bounded function observation assembled from Evidence pages. */
export interface FunctionSnapshot {
readonly evidence: readonly Evidence[];
readonly procedure: FunctionDossier["procedure"];
readonly provider: Evidence["provider"];
readonly subject: NonNullable<Evidence["subject"]>;
readonly pseudocode: {
readonly text: string;
readonly total: number;
readonly complete: boolean;
readonly truncated: boolean;
};
readonly collections: FunctionCollections;
readonly instructionScan: FunctionDossier["instruction_scan"];
readonly limitations: readonly string[];
}
interface DossierPage {
readonly evidence: Omit<Evidence, "subject"> & {
readonly subject: NonNullable<Evidence["subject"]>;
};
readonly dossier: FunctionDossier;
readonly parameters: ReturnType<
typeof enhancedInputSchemas.analyze_function.parse
>;
}
/** Parse and assemble up to 100 mutually consistent analyze_function pages. */
export const parseFunctionEvidence = (input: unknown): FunctionSnapshot => {
const values = Array.isArray(input) ? input : [input];
if (values.length === 0 || values.length > MAX_PAGES)
throw new TypeError("Function comparison requires 1 to 100 Evidence pages");
const pages = values.map(parsePage);
assertPageGroup(pages);
const first = pages[0];
if (first === undefined)
throw new TypeError("Function comparison requires Evidence pages");
const pseudocode = mergePseudocode(pages);
const collections: FunctionCollections = {
assembly: mergeCollection(pages, "assembly"),
comments: mergeCollection(pages, "comments"),
callers: mergeCollection(pages, "callers"),
callees: mergeCollection(pages, "callees"),
incoming_references: mergeCollection(pages, "incoming_references"),
outgoing_references: mergeCollection(pages, "outgoing_references"),
referenced_strings: mergeCollection(pages, "referenced_strings"),
referenced_names: mergeCollection(pages, "referenced_names"),
basic_blocks: mergeCollection(pages, "basic_blocks"),
};
return {
evidence: pages.map(({ evidence }) => evidence),
procedure: first.dossier.procedure,
provider: first.evidence.provider,
subject: first.evidence.subject,
pseudocode,
collections,
instructionScan: first.dossier.instruction_scan,
limitations: [
...new Set(pages.flatMap(({ evidence }) => evidence.limitations)),
].sort((left, right) => left.localeCompare(right)),
};
};
const parsePage = (input: unknown): DossierPage => {
const evidence = parseEvidence(input);
if (evidence.operation !== "analyze_function")
throw new TypeError(
"Function comparison requires analyze_function Evidence",
);
if (evidence.subject === null)
throw new TypeError("Function comparison requires artifact-bound Evidence");
return {
evidence: { ...evidence, subject: evidence.subject },
dossier: functionDossierSchema.parse(evidence.normalized_result),
parameters: enhancedInputSchemas.analyze_function.parse(
evidence.parameters,
),
};
};
const assertPageGroup = (pages: readonly DossierPage[]): void => {
const first = pages[0];
if (first === undefined) return;
const invariant = pageInvariant(first);
for (const page of pages)
if (canonicalJson(pageInvariant(page)) !== canonicalJson(invariant))
throw new TypeError(
"Function Evidence pages mix subjects, providers, or immutable analysis controls",
);
};
const pageInvariant = (page: DossierPage) => ({
subject: {
digest: page.evidence.subject?.digest,
format: page.evidence.subject?.format,
architecture: page.evidence.subject?.architecture,
},
provider: page.evidence.provider,
procedure: page.dossier.procedure,
requested_procedure: page.parameters.procedure,
include_assembly: page.parameters.include_assembly,
max_instructions: page.parameters.max_instructions,
instruction_scan: page.dossier.instruction_scan,
});
const mergePseudocode = (
pages: readonly DossierPage[],
): FunctionSnapshot["pseudocode"] => {
const totals = new Set(
pages.map(({ dossier }) => dossier.pseudocode.total_chars),
);
if (totals.size !== 1)
throw new TypeError("Function pseudocode pages disagree on total length");
const total = pages[0]?.dossier.pseudocode.total_chars ?? 0;
if (total > MAX_PSEUDOCODE_CHARS)
throw new TypeError("Function pseudocode exceeds comparison limit");
const characters = new Map<number, string>();
for (const page of pages) {
const offset = page.parameters.pseudocode_offset;
for (const [index, character] of [
...page.dossier.pseudocode.text,
].entries())
mergeIndexed(characters, offset + index, character, "pseudocode");
if (characters.size > MAX_PSEUDOCODE_CHARS)
throw new TypeError("Function pseudocode exceeds comparison limit");
}
const complete = hasCompleteRange(characters, total);
return {
text: orderedValues(characters).join(""),
total,
complete,
truncated: !complete,
};
};
const mergeCollection = <Field extends DossierCollectionField>(
pages: readonly DossierPage[],
field: Field,
): FunctionCollection<DossierItem<Field>> => {
const totals = new Set(pages.map(({ dossier }) => dossier[field].total));
if (totals.size !== 1)
throw new TypeError(`Function ${field} pages disagree on total count`);
const total = pages[0]?.dossier[field].total ?? null;
if (total !== null && total > MAX_ITEMS)
throw new TypeError(`Function ${field} exceeds comparison item limit`);
const items = new Map<number, DossierItem<Field>>();
for (const page of pages) {
const offset = collectionOffset(page, field);
for (const [index, item] of page.dossier[field].items.entries())
mergeIndexed(items, offset + index, item, field);
if (items.size > MAX_ITEMS)
throw new TypeError(`Function ${field} exceeds comparison item limit`);
}
const scanLimited =
field !== "assembly" && pageDependsOnScan(field)
? pages.some(({ dossier }) => dossier.instruction_scan.truncated)
: false;
const unavailable =
field === "assembly" &&
pages.every(({ parameters }) => !parameters.include_assembly);
const complete =
!scanLimited &&
!unavailable &&
total !== null &&
hasCompleteRange(items, total);
return {
items: orderedValues(items),
total,
complete,
truncated: scanLimited || (!unavailable && !complete),
};
};
const collectionOffset = (
page: DossierPage,
field: DossierCollectionField,
): number =>
field === "assembly"
? page.parameters.assembly_offset
: page.parameters.collection_offset[field];
const pageDependsOnScan = (field: DossierCollectionField): boolean =>
field !== "callers" && field !== "callees";
const mergeIndexed = <Value>(
output: Map<number, Value>,
index: number,
value: Value,
field: string,
): void => {
const previous = output.get(index);
if (
previous !== undefined &&
canonicalJson(previous) !== canonicalJson(value)
)
throw new TypeError(
`Function ${field} pages overlap with conflicting data`,
);
output.set(index, value);
};
const hasCompleteRange = <Value>(
values: ReadonlyMap<number, Value>,
total: number,
): boolean => {
if (values.size !== total) return false;
for (let index = 0; index < total; index += 1)
if (!values.has(index)) return false;
return true;
};
const orderedValues = <Value>(values: ReadonlyMap<number, Value>): Value[] =>
[...values.entries()]
.sort(([left], [right]) => left - right)
.map(([, value]) => value);
const canonicalJson = (value: unknown): string => {
const encoded = canonicalize(value);
if (encoded === undefined)
throw new TypeError("Function Evidence could not be canonicalized");
return encoded;
};

33
src/domain/hopperValues.ts
View File

@ -1,7 +1,7 @@
import { z } from "zod";
import type { JsonValue } from "./jsonValue.js";
import { AnalysisProtocolError, HopperProtocolError } from "./errors.js";
import { AnalysisOutputError, HopperProtocolError } from "./errors.js";
import { err, ok, type Result } from "./result.js";
export interface AddressedName {
@ -13,6 +13,9 @@ export interface SegmentSummary {
readonly name: string;
readonly start: string;
readonly end: string;
readonly readable: boolean | null;
readonly writable: boolean | null;
readonly executable: boolean | null;
}
export interface AddressedPage {
@ -30,6 +33,9 @@ const segmentSchema = z.object({
name: z.string().default(""),
start: z.string().default(""),
end: z.string().default(""),
readable: z.boolean().nullable().default(null),
writable: z.boolean().nullable().default(null),
executable: z.boolean().nullable().default(null),
});
const addressedPageSchema = z.object({
items: z.array(z.object({ address: z.string(), value: z.string() })),
@ -42,6 +48,12 @@ const unavailableSchema = z
const procedureIdentitySchema = z
.object({ address: z.string(), name: z.string() })
.strict();
const localVariableSchema = z
.object({
description: z.string(),
provenance: z.literal("hopper-public-python-api"),
})
.strict();
const boundedSchema = <T extends z.ZodType>(item: T) =>
z
.object({
@ -84,14 +96,14 @@ const referenceEdgeSchema = z
kind: unavailableSchema,
})
.strict();
const functionDossierSchema = z
export const functionDossierSchema = z
.object({
procedure: z
.object({
address: z.string(),
name: z.string(),
signature: z.string().nullable(),
locals: z.array(z.json()),
locals: z.array(localVariableSchema),
})
.strict(),
pseudocode: z
@ -104,7 +116,7 @@ const functionDossierSchema = z
})
.strict()
.superRefine((value, context) => {
if (value.returned_chars !== value.text.length) {
if (value.returned_chars !== [...value.text].length) {
context.addIssue({
code: "custom",
message: "returned_chars must equal the text length",
@ -173,17 +185,22 @@ const functionDossierSchema = z
})
.strict();
/** Strict analyzed-function dossier shared by provider and comparison boundaries. */
export type FunctionDossier = z.infer<typeof functionDossierSchema>;
/** Strictly parse a complete Hopper function dossier at the provider boundary. */
export const parseFunctionDossier = (
value: JsonValue,
): Result<JsonValue, AnalysisProtocolError> => {
): Result<JsonValue, AnalysisOutputError> => {
const parsed = functionDossierSchema.safeParse(value);
return parsed.success
? ok(parsed.data)
: err(
new AnalysisProtocolError("Invalid analyze_function provider output", {
cause: parsed.error,
}),
new AnalysisOutputError(
"analyze_function",
"provider output did not match the dossier contract",
{ cause: parsed.error },
),
);
};

158
src/domain/nativeInspection.ts Normal file
View File

@ -0,0 +1,158 @@
import { z } from "zod";
const hexAddress = z.string().regex(/^0x[a-fA-F0-9]+$/u);
const coverage = <Schema extends z.ZodType>(item: Schema) =>
z.object({
items: z.array(item),
total: z.number().int().min(0).nullable(),
exhaustive: z.boolean(),
limitations: z.array(z.string()),
});
const nativeCommandInvocationSchema = z.object({
tool: z.string().min(1),
command: z.array(z.string()),
tool_version: z.string().nullable(),
version_reason: z.string().nullable(),
executable_sha256: z.string().regex(/^[a-f0-9]{64}$/u),
exit: z.object({
code: z.number().int().nullable(),
signal: z.string().nullable(),
}),
stdout_bytes: z.number().int().min(0),
stderr_bytes: z.number().int().min(0),
stdout_truncated: z.boolean(),
stderr_truncated: z.boolean(),
});
const architectureSchema = z.object({
name: z.string(),
cpu_type: z.string().nullable(),
cpu_subtype: z.string().nullable(),
file_offset: z.number().int().min(0).nullable(),
size: z.number().int().min(0).nullable(),
alignment: z.number().int().min(0).nullable(),
});
const permissionSchema = z.object({
read: z.boolean().nullable(),
write: z.boolean().nullable(),
execute: z.boolean().nullable(),
raw: z.string().nullable(),
});
const sectionSchema = z.object({
segment: z.string(),
name: z.string(),
address: hexAddress.nullable(),
size: z.number().int().min(0).nullable(),
file_offset: z.number().int().min(0).nullable(),
alignment: z.number().int().min(0).nullable(),
flags: z.array(z.string()),
});
const segmentSchema = z.object({
name: z.string(),
vm_address: hexAddress.nullable(),
vm_size: z.number().int().min(0).nullable(),
file_offset: z.number().int().min(0).nullable(),
file_size: z.number().int().min(0).nullable(),
maximum_permissions: permissionSchema,
initial_permissions: permissionSchema,
sections: coverage(sectionSchema),
});
const symbolSchema = z.object({
name: z.string(),
address: hexAddress.nullable(),
weak: z.boolean().nullable(),
reexport: z.boolean().nullable(),
source: z.string().nullable(),
});
const dependencySchema = z.object({
path: z.string(),
kind: z.string(),
current_version: z.string().nullable(),
compatibility_version: z.string().nullable(),
});
const loadCommandSchema = z.object({
index: z.number().int().min(0),
kind: z.string(),
file_offset: z.number().int().min(0).nullable(),
fields: z.record(z.string(), z.union([z.string(), z.number(), z.null()])),
});
const buildMetadataSchema = z.object({
platform: z.string().nullable(),
minimum_os: z.string().nullable(),
sdk: z.string().nullable(),
tools: z.array(z.object({ name: z.string(), version: z.string() })),
});
export const inspectMachoSchema = z.object({
format: z.literal("mach-o"),
endian: z.enum(["little", "big"]).nullable(),
word_size: z.union([z.literal(32), z.literal(64)]).nullable(),
file_type: z.string().nullable(),
flags: z.array(z.string()),
uuid: z.string().nullable(),
entrypoints: coverage(z.object({ file_offset: z.number().int().min(0) })),
architectures: coverage(architectureSchema),
build_metadata: coverage(buildMetadataSchema),
load_commands: coverage(loadCommandSchema),
dependencies: coverage(dependencySchema),
imports: coverage(symbolSchema),
exports: coverage(symbolSchema),
segments: coverage(segmentSchema),
provenance: z.array(nativeCommandInvocationSchema),
limitations: z.array(z.string()),
});
export const inspectSignatureSchema = z.object({
signed: z.boolean(),
identifier: z.string().nullable(),
team_identifier: z.string().nullable(),
format: z.string().nullable(),
cdhashes: z.array(z.string()),
hash_algorithms: z.array(z.string()),
authorities: z.array(z.string()),
designated_requirement: z.string().nullable(),
entitlements: z.unknown().nullable(),
timestamp: z.string().nullable(),
hardened_runtime: z.boolean().nullable(),
provenance: z.array(nativeCommandInvocationSchema),
limitations: z.array(z.string()),
});
export const inspectPlistSchema = z.object({
format: z.enum(["xml", "binary", "json", "unknown"]),
value: z.unknown(),
bundle: z.object({
identifier: z.string().nullable(),
executable: z.string().nullable(),
name: z.string().nullable(),
version: z.string().nullable(),
short_version: z.string().nullable(),
}),
source_path: z.string(),
provenance: z.array(nativeCommandInvocationSchema),
limitations: z.array(z.string()),
});
export const listArchitecturesSchema = z.object({
architectures: coverage(architectureSchema),
provenance: z.array(nativeCommandInvocationSchema),
limitations: z.array(z.string()),
});
export const demangleSwiftSchema = z.object({
symbols: z.array(
z.object({
input: z.string(),
output: z.string(),
status: z.enum(["demangled", "unchanged", "invalid"]),
}),
),
provenance: z.array(nativeCommandInvocationSchema),
limitations: z.array(z.string()),
});
export type NativeCommandInvocation = z.infer<
typeof nativeCommandInvocationSchema
>;
export type InspectSignature = z.infer<typeof inspectSignatureSchema>;

287
src/domain/processCapture.ts
View File

@ -3,6 +3,26 @@ import { z } from "zod";
const positiveBudget = z.number().int().positive();
const timedEventBase = { at_ms: z.number().int().nonnegative() };
const environmentName = z.string().regex(/^[A-Za-z_][A-Za-z0-9_]*$/u);
const reservedEnvironment = new Set([
"HOME",
"TERM",
"REA_PROCESS_RUN_ID",
"REA_REPLAY_HTTP_URL",
"REA_REPLAY_WEBSOCKET_URL",
]);
const normalizationSchema = z.object({
paths: z.boolean(),
pids: z.boolean(),
ports: z.boolean(),
time_bucket_ms: positiveBudget.max(60_000),
patterns: z.array(
z.object({
pattern: z.string().max(500),
replacement: z.string().max(100),
}),
),
});
/** Exact boundary schema for bounded dynamic process scenarios. */
export const processScenarioSchema = z
@ -12,12 +32,17 @@ export const processScenarioSchema = z
.describe(
"Explicit per-call acknowledgement that this operation launches the target",
),
unknown_registry_approved: z
.literal(true)
.optional()
.describe("Explicit approval to record capture residuals durably"),
executable: z.string().startsWith("/"),
arguments: z.array(z.string()).max(256).default([]),
working_directory: z.string().startsWith("/"),
environment: z.record(z.string(), z.string()).default({}),
inherit_environment: z.array(z.string()).max(64).default([]),
secret_aliases: z.array(z.string()).max(64).default([]),
environment: z.record(environmentName, z.string()).default({}),
inherit_environment: z.array(environmentName).max(64).default([]),
secret_aliases: z.array(environmentName).max(64).default([]),
network_access: z.literal("host").default("host"),
filesystem_roots: z.array(z.string().startsWith("/")).max(16).default([]),
events: z
.array(
@ -52,6 +77,10 @@ export const processScenarioSchema = z
files: positiveBudget.max(100_000).default(10_000),
file_bytes: positiveBudget.max(100_000_000).default(10_000_000),
processes: positiveBudget.max(10_000).default(1_000),
protocol_events: positiveBudget.max(100_000).default(10_000),
protocol_body_bytes: positiveBudget.max(10_000_000).default(1_000_000),
connections: positiveBudget.max(1_000).default(100),
filesystem_depth: z.number().int().min(1).max(64).default(16),
})
.default({
output_bytes: 1_000_000,
@ -59,6 +88,10 @@ export const processScenarioSchema = z
files: 10_000,
file_bytes: 10_000_000,
processes: 1_000,
protocol_events: 10_000,
protocol_body_bytes: 1_000_000,
connections: 100,
filesystem_depth: 16,
}),
normalization: z
.object({
@ -92,6 +125,12 @@ export const processScenarioSchema = z
path: z.string().startsWith("/"),
status: z.number().int().min(100).max(599),
body: z.string().max(1_000_000),
request_headers: z.record(z.string(), z.string()).default({}),
request_body: z.string().max(1_000_000).optional(),
response_headers: z.record(z.string(), z.string()).default({}),
delay_ms: z.number().int().nonnegative().max(30_000).default(0),
disconnect: z.boolean().default(false),
max_calls: z.number().int().min(1).max(100).default(1),
}),
)
.max(100)
@ -100,8 +139,33 @@ export const processScenarioSchema = z
.array(z.string().max(1_000_000))
.max(100)
.default([]),
websocket_connections: z
.array(
z.object({
messages: z
.array(
z.object({
data: z.string().max(1_000_000),
delay_ms: z
.number()
.int()
.nonnegative()
.max(30_000)
.default(0),
}),
)
.max(100),
disconnect_after: z.boolean().default(false),
}),
)
.max(100)
.default([]),
})
.default({ http: [], websocket_messages: [] }),
.default({
http: [],
websocket_messages: [],
websocket_connections: [],
}),
})
.strict()
.superRefine((scenario, context) => {
@ -124,6 +188,34 @@ export const processScenarioSchema = z
});
}
}
const explicit = new Set(Object.keys(scenario.environment));
for (const name of [...explicit, ...scenario.inherit_environment]) {
if (reservedEnvironment.has(name)) {
context.addIssue({
code: "custom",
message: `${name} is reserved by the process adapter`,
path: ["environment", name],
});
}
}
for (const name of scenario.inherit_environment) {
if (explicit.has(name)) {
context.addIssue({
code: "custom",
message: "an environment name cannot be explicit and inherited",
path: ["inherit_environment"],
});
}
}
for (const event of scenario.events) {
if (event.at_ms > scenario.timeout_ms) {
context.addIssue({
code: "custom",
message: "event occurs after the scenario timeout",
path: ["events"],
});
}
}
});
/** A parsed, bounded dynamic process observation scenario. */
@ -139,6 +231,7 @@ export interface ProcessExecutionPolicy {
readonly executableRoots: readonly string[];
readonly workingRoots: readonly string[];
readonly allowedEnvironment: readonly string[];
readonly allowExternalNetwork: boolean;
}
/** A safe, caller-visible process-policy decision. */
@ -159,6 +252,11 @@ export const authorizeProcessScenario = (
): ProcessPolicyDecision => {
if (!policy.enabled)
return { allowed: false, reason: "process capture is disabled" };
if (scenario.network_access === "host" && !policy.allowExternalNetwork)
return {
allowed: false,
reason: "host network access is not approved by operator policy",
};
if (
!policy.executableRoots.some((root) => isWithin(scenario.executable, root))
) {
@ -216,6 +314,13 @@ export interface FileState {
readonly symlink_target: string | null;
}
interface FileEffect {
readonly path: string;
readonly status: "created" | "modified" | "deleted" | "unchanged";
readonly before: FileState | null;
readonly after: FileState | null;
}
/** A sampled owned-process observation; sampling cannot prove syscall completeness. */
export interface ProcessSample {
readonly at_ms: number;
@ -227,32 +332,57 @@ export interface ProcessSample {
/** A bounded loopback replay observation. */
export interface ProtocolEvent {
readonly sequence: number;
readonly at_ms: number;
readonly protocol: "http" | "websocket";
readonly direction: "request" | "response" | "received" | "sent";
readonly method: string | null;
readonly path: string | null;
readonly data: string;
readonly outcome:
| "matched"
| "unmatched"
| "script_exhausted"
| "disconnected";
}
/** A bounded process observation with explicit incompleteness metadata. */
export interface ProcessCapture {
readonly schema_version: 1;
readonly schema_version: 2;
readonly normalization: z.infer<typeof normalizationSchema>;
readonly frames: readonly TerminalFrame[];
readonly exit: {
readonly code: number | null;
readonly signal: number | null;
readonly reason: "exited" | "timeout" | "idle_timeout";
};
readonly process_samples: readonly ProcessSample[];
readonly protocol_events: readonly ProtocolEvent[];
readonly files_before: readonly FileState[];
readonly files_after: readonly FileState[];
readonly filesystem_effects: readonly FileEffect[];
readonly truncated: boolean;
readonly limitations: readonly string[];
readonly residual_unknowns: readonly {
readonly scope:
| "terminal"
| "exit"
| "process"
| "filesystem"
| "protocol"
| "cleanup"
| "network";
readonly reason: string;
}[];
readonly cleanup: {
readonly owned_process_group: "verified";
readonly temporary_root: "removed";
};
}
/** Exact serialized shape of a bounded process capture. */
export const processCaptureSchema: z.ZodType<ProcessCapture> = z.object({
schema_version: z.literal(1),
schema_version: z.literal(2),
normalization: normalizationSchema,
frames: z.array(
z.object({
sequence: z.number().int().nonnegative(),
@ -263,6 +393,7 @@ export const processCaptureSchema: z.ZodType<ProcessCapture> = z.object({
exit: z.object({
code: z.number().int().nullable(),
signal: z.number().int().nullable(),
reason: z.enum(["exited", "timeout", "idle_timeout"]),
}),
process_samples: z.array(
z.object({
@ -275,11 +406,18 @@ export const processCaptureSchema: z.ZodType<ProcessCapture> = z.object({
protocol_events: z.array(
z.object({
sequence: z.number().int().nonnegative(),
at_ms: z.number().int().nonnegative(),
protocol: z.enum(["http", "websocket"]),
direction: z.enum(["request", "response", "received", "sent"]),
method: z.string().nullable(),
path: z.string().nullable(),
data: z.string(),
outcome: z.enum([
"matched",
"unmatched",
"script_exhausted",
"disconnected",
]),
}),
),
files_before: z.array(
@ -308,89 +446,62 @@ export const processCaptureSchema: z.ZodType<ProcessCapture> = z.object({
symlink_target: z.string().nullable(),
}),
),
filesystem_effects: z.array(
z.object({
path: z.string(),
status: z.enum(["created", "modified", "deleted", "unchanged"]),
before: z
.object({
path: z.string(),
type: z.enum(["file", "directory", "symlink", "other"]),
mode: z.number().int().nonnegative(),
size: z.number().int().nonnegative(),
sha256: z
.string()
.regex(/^[a-f0-9]{64}$/u)
.nullable(),
symlink_target: z.string().nullable(),
})
.nullable(),
after: z
.object({
path: z.string(),
type: z.enum(["file", "directory", "symlink", "other"]),
mode: z.number().int().nonnegative(),
size: z.number().int().nonnegative(),
sha256: z
.string()
.regex(/^[a-f0-9]{64}$/u)
.nullable(),
symlink_target: z.string().nullable(),
})
.nullable(),
}),
),
truncated: z.boolean(),
limitations: z.array(z.string()),
residual_unknowns: z.array(
z.object({
scope: z.enum([
"terminal",
"exit",
"process",
"filesystem",
"protocol",
"cleanup",
"network",
]),
reason: z.string(),
}),
),
cleanup: z.object({
owned_process_group: z.literal("verified"),
temporary_root: z.literal("removed"),
}),
});
/** Comparison classification that never equates incomplete evidence. */
type ComparisonStatus =
| "unchanged"
| "added"
| "removed"
| "changed"
| "truncated"
| "unknown";
/** Pure normalized comparison between two captures. */
export interface ProcessCaptureComparison {
readonly status: ComparisonStatus;
readonly terminal: ComparisonStatus;
readonly exit: ComparisonStatus;
readonly filesystem: ComparisonStatus;
readonly protocol: ComparisonStatus;
readonly process: ComparisonStatus;
readonly limitations: readonly string[];
}
const stableFiles = (files: readonly FileState[]): string =>
JSON.stringify(
[...files].sort((left, right) => left.path.localeCompare(right.path)),
);
/** Compare bounded captures without claiming equality for incomplete observations. */
export const compareProcessCaptures = (
left: ProcessCapture,
right: ProcessCapture,
): ProcessCaptureComparison => {
if (left.truncated || right.truncated) {
return {
status: "truncated",
terminal: "truncated",
exit: "truncated",
filesystem: "truncated",
protocol: "truncated",
process: "truncated",
limitations: ["At least one capture is truncated."],
};
}
const terminal =
left.frames.map(({ data }) => data).join("") ===
right.frames.map(({ data }) => data).join("")
? "unchanged"
: "changed";
const exit =
JSON.stringify(left.exit) === JSON.stringify(right.exit)
? "unchanged"
: "changed";
const filesystem =
stableFiles(left.files_after) === stableFiles(right.files_after)
? "unchanged"
: "changed";
const protocol =
JSON.stringify(left.protocol_events) ===
JSON.stringify(right.protocol_events)
? "unchanged"
: "changed";
const process =
JSON.stringify(left.process_samples.map(({ command }) => command)) ===
JSON.stringify(right.process_samples.map(({ command }) => command))
? "unchanged"
: "changed";
const status =
terminal === "unchanged" &&
exit === "unchanged" &&
filesystem === "unchanged" &&
protocol === "unchanged" &&
process === "unchanged"
? "unchanged"
: "changed";
return {
status,
terminal,
exit,
filesystem,
protocol,
process,
limitations: [...left.limitations, ...right.limitations],
};
};
export {
compareProcessCaptures,
comparisonStatusSchema,
processCaptureComparisonSchema,
} from "./processComparison.js";

160
src/domain/processComparison.ts Normal file
View File

@ -0,0 +1,160 @@
import { z } from "zod";
import type { ProcessCapture } from "./processCapture.js";
/** Comparison classification that never equates incomplete evidence. */
export const comparisonStatusSchema = z.enum([
"unchanged",
"added",
"removed",
"changed",
"truncated",
"unknown",
]);
type ComparisonStatus = z.infer<typeof comparisonStatusSchema>;
/** Pure normalized comparison between two captures. */
export const processCaptureComparisonSchema = z
.object({
status: comparisonStatusSchema,
terminal: comparisonStatusSchema,
exit: comparisonStatusSchema,
filesystem: comparisonStatusSchema,
protocol: comparisonStatusSchema,
process: comparisonStatusSchema,
limitations: z.array(z.string()),
})
.superRefine((comparison, context) => {
const dimensions = [
comparison.terminal,
comparison.exit,
comparison.filesystem,
comparison.protocol,
comparison.process,
];
const expected = dimensions.includes("truncated")
? "truncated"
: dimensions.some((status) =>
["added", "removed", "changed"].includes(status),
)
? "changed"
: dimensions.includes("unknown")
? "unknown"
: "unchanged";
if (comparison.status !== expected)
context.addIssue({
code: "custom",
message: "Process comparison status contradicts its dimensions",
path: ["status"],
});
});
type ProcessCaptureComparison = z.infer<typeof processCaptureComparisonSchema>;
const stableFiles = (capture: ProcessCapture): string =>
JSON.stringify(
[...capture.files_after].sort((left, right) =>
left.path.localeCompare(right.path),
),
);
const classifyCollection = (
left: readonly unknown[],
right: readonly unknown[],
): ComparisonStatus => {
if (JSON.stringify(left) === JSON.stringify(right)) return "unchanged";
if (left.length === 0) return "added";
if (right.length === 0) return "removed";
return "changed";
};
const hasUnknown = (
capture: ProcessCapture,
scope: ProcessCapture["residual_unknowns"][number]["scope"],
): boolean => capture.residual_unknowns.some((item) => item.scope === scope);
const comparisonStatus = (
dimensions: readonly ComparisonStatus[],
): ComparisonStatus => {
if (
dimensions.some((status) =>
["changed", "added", "removed"].includes(status),
)
)
return "changed";
return dimensions.includes("unknown") ? "unknown" : "unchanged";
};
/** Compare bounded captures without equating missing or incompatible evidence. */
export const compareProcessCaptures = (
left: ProcessCapture,
right: ProcessCapture,
): ProcessCaptureComparison => {
if (left.truncated || right.truncated) {
return {
status: "truncated",
terminal: "truncated",
exit: "truncated",
filesystem: "truncated",
protocol: "truncated",
process: "truncated",
limitations: ["At least one capture is truncated."],
};
}
const sameNormalization =
JSON.stringify(left.normalization) === JSON.stringify(right.normalization);
const classify = (
scope: ProcessCapture["residual_unknowns"][number]["scope"],
leftValues: readonly unknown[],
rightValues: readonly unknown[],
): ComparisonStatus =>
!sameNormalization || hasUnknown(left, scope) || hasUnknown(right, scope)
? "unknown"
: classifyCollection(leftValues, rightValues);
const terminal = classify(
"terminal",
left.frames.map(({ data }) => data),
right.frames.map(({ data }) => data),
);
const exit =
hasUnknown(left, "exit") || hasUnknown(right, "exit")
? "unknown"
: JSON.stringify(left.exit) === JSON.stringify(right.exit)
? "unchanged"
: "changed";
const filesystem =
!hasUnknown(left, "filesystem") &&
!hasUnknown(right, "filesystem") &&
sameNormalization &&
stableFiles(left) === stableFiles(right)
? "unchanged"
: classify(
"filesystem",
left.filesystem_effects,
right.filesystem_effects,
);
const protocol = classify(
"protocol",
left.protocol_events,
right.protocol_events,
);
const process = classify(
"process",
left.process_samples,
right.process_samples,
);
return {
status: comparisonStatus([terminal, exit, filesystem, protocol, process]),
terminal,
exit,
filesystem,
protocol,
process,
limitations: [
...left.limitations,
...right.limitations,
...left.residual_unknowns.map(({ reason }) => reason),
...right.residual_unknowns.map(({ reason }) => reason),
...(sameNormalization ? [] : ["Capture normalization rules differ."]),
],
};
};

137
src/domain/reconstructionUnknowns.ts Normal file
View File

@ -0,0 +1,137 @@
import type {
ReconstructionClaim,
ReconstructionClaimResult,
ReconstructionVerificationResult,
} from "./reconstructionVerificationSchemas.js";
import type { ResidualUnknown } from "./residualUnknown.js";
/** Select the latest revision of each residual unknown. */
export const reconstructionUnknownHeads = (
revisions: readonly ResidualUnknown[],
): ReadonlyMap<string, ResidualUnknown> => {
const heads = new Map<string, ResidualUnknown>();
for (const item of revisions) {
const current = heads.get(item.unknown_id);
if (current === undefined || current.revision < item.revision)
heads.set(item.unknown_id, item);
}
return heads;
};
/** Find active unknowns that affect a comparison or one of its sources. */
export const reconstructionClaimUnknowns = (
heads: ReadonlyMap<string, ResidualUnknown>,
comparisonId: string,
sourceIds: readonly string[],
): ResidualUnknown[] => {
const evidenceIds = new Set([comparisonId, ...sourceIds]);
const relevantIds = new Set(
[...heads.values()]
.filter(
(item) =>
item.status !== "resolved" &&
[
...item.supporting_evidence_ids,
...item.contradicting_evidence_ids,
].some((id) => evidenceIds.has(id)),
)
.map(({ unknown_id: id }) => id),
);
const pending = [...relevantIds];
while (pending.length > 0) {
const currentId = pending.pop();
if (currentId === undefined) continue;
const current = heads.get(currentId);
if (current === undefined) continue;
for (const relationship of current.relationships) {
if (relationship.type === "related-to") continue;
const dependency = heads.get(relationship.unknown_id);
if (
dependency === undefined ||
dependency.status === "resolved" ||
relevantIds.has(dependency.unknown_id)
)
continue;
relevantIds.add(dependency.unknown_id);
pending.push(dependency.unknown_id);
}
}
return [...relevantIds]
.flatMap((id) => {
const unknown = heads.get(id);
return unknown === undefined ? [] : [unknown];
})
.sort((left, right) =>
left.unknown_id.localeCompare(right.unknown_id, "en"),
);
};
/** Aggregate and bound deterministic probes for unresolved claims. */
export const reconstructionProbes = (
results: readonly ReconstructionClaimResult[],
heads: ReadonlyMap<string, ResidualUnknown>,
): {
readonly items: ReconstructionVerificationResult["recommended_probes"];
readonly truncated: boolean;
} => {
const output = new Map<
string,
{
operation: string;
rationale: string;
claim_ids: string[];
unknown_ids: string[];
}
>();
for (const result of results.filter(({ status }) => status === "unknown")) {
const unknowns = result.unknown_ids.flatMap((id) => {
const item = heads.get(id);
return item === undefined ? [] : [item];
});
const probes = unknowns.flatMap(({ recommended_probes: items }) => items);
const candidates = probes.length > 0 ? probes : [defaultProbe(result.kind)];
for (const probe of candidates) {
const key = `${probe.operation}\0${probe.rationale}`;
const current = output.get(key) ?? {
...probe,
claim_ids: [],
unknown_ids: [],
};
current.claim_ids = uniqueSorted([...current.claim_ids, result.claim_id]);
current.unknown_ids = uniqueSorted([
...current.unknown_ids,
...result.unknown_ids,
]);
output.set(key, current);
}
}
const sorted = [...output.values()].sort((left, right) =>
`${left.operation}\0${left.rationale}`.localeCompare(
`${right.operation}\0${right.rationale}`,
"en",
),
);
return { items: sorted.slice(0, 2_000), truncated: sorted.length > 2_000 };
};
const defaultProbe = (
kind: ReconstructionClaim["kind"],
): { operation: string; rationale: string } =>
kind === "behavioral"
? {
operation: "capture_process_scenario",
rationale: "Repeat both scenarios under one controlled environment.",
}
: kind === "structural-function"
? {
operation: "analyze_function",
rationale: "Capture complete function dossiers under equal limits.",
}
: {
operation: "inventory_artifact",
rationale:
"Capture complete artifact inventories under equal limits.",
};
const uniqueSorted = (values: readonly string[]): string[] =>
[...new Set(values)].sort((left, right) => left.localeCompare(right, "en"));

422
src/domain/reconstructionVerification.ts Normal file
View File

@ -0,0 +1,422 @@
import { createHash } from "node:crypto";
import canonicalize from "canonicalize";
import { z } from "zod";
import { artifactComparisonResultSchema } from "./artifactComparison.js";
import { artifactInventoryResultSchema } from "./artifactGraph.js";
import { parseEvidence, type Evidence } from "./evidence.js";
import { parseEvidenceBundle } from "./evidenceBundle.js";
import { functionComparisonResultSchema } from "./functionComparison.js";
import { functionDossierSchema } from "./hopperValues.js";
import {
processCaptureComparisonSchema,
processCaptureSchema,
} from "./processCapture.js";
import {
reconstructionClaimResultSchema,
reconstructionSpecificationSchema,
reconstructionVerificationResultSchema,
type ReconstructionClaim as Claim,
type ReconstructionClaimResult as ClaimResult,
type ReconstructionObservedStatus as ObservedStatus,
type ReconstructionVerificationResult,
} from "./reconstructionVerificationSchemas.js";
import {
reconstructionClaimUnknowns,
reconstructionProbes,
reconstructionUnknownHeads,
} from "./reconstructionUnknowns.js";
import type { ResidualUnknown } from "./residualUnknown.js";
const evidenceIdSchema = z.string().regex(/^ev_[a-f0-9]{64}$/u);
export {
reconstructionVerificationInputSchema,
reconstructionVerificationResultSchema,
} from "./reconstructionVerificationSchemas.js";
export type { ReconstructionVerificationResult } from "./reconstructionVerificationSchemas.js";
const MAX_BUNDLE_RECORDS = 20_100;
const MAX_UNKNOWN_REVISIONS = 10_000;
const MAX_CANONICAL_BYTES = 32 * 1024 * 1024;
const providers = {
behavioral: {
operation: "compare_process_captures",
predicate: "rea.process-comparison/v1",
id: "rea-process",
name: "REA deterministic process harness",
},
"structural-function": {
operation: "compare_functions",
predicate: "rea.function-comparison/v1",
id: "rea-function-comparison",
name: "REA function comparison",
},
"structural-artifact": {
operation: "compare_artifacts",
predicate: "rea.artifact-comparison/v1",
id: "rea-artifact-comparison",
name: "REA artifact comparison",
},
} as const;
/** Verify only explicitly declared claims; no global equivalence is inferred. */
export const verifyReconstruction = (
specificationInput: unknown,
bundleInput: unknown,
offset: number,
limit: number,
): ReconstructionVerificationResult => {
const specification =
reconstructionSpecificationSchema.parse(specificationInput);
enforceInputSize(bundleInput);
const bundle = parseEvidenceBundle(bundleInput);
if (bundle.records.length > MAX_BUNDLE_RECORDS)
throw new TypeError("Reconstruction Evidence record limit exceeded");
if (bundle.unknowns.length > MAX_UNKNOWN_REVISIONS)
throw new TypeError("Reconstruction unknown revision limit exceeded");
const records = new Map(
bundle.records.map((item) => [item.evidence_id, item]),
);
const heads = reconstructionUnknownHeads(bundle.unknowns);
const results = specification.claims
.map((claim) => evaluateClaim(claim, records, heads))
.sort((left, right) => left.claim_id.localeCompare(right.claim_id, "en"));
const page = results.slice(offset, offset + limit);
const failed = count(results, "fail");
const unresolved = count(results, "unknown");
const probes = reconstructionProbes(results, heads);
const evidenceLinks = uniqueSorted(
results.flatMap(({ evidence_links: links }) => links),
);
if (evidenceLinks.length > 20_100)
throw new TypeError("Reconstruction Evidence closure exceeds limit");
const output = {
status: failed > 0 ? "fail" : unresolved > 0 ? "unknown" : "pass",
specification_sha256: digest({
...specification,
claims: [...specification.claims].sort((left, right) =>
left.claim_id.localeCompare(right.claim_id, "en"),
),
}),
summary: {
total: results.length,
passed: count(results, "pass"),
failed,
unknown: unresolved,
behavioral: results.filter(({ kind }) => kind === "behavioral").length,
structural: results.filter(({ kind }) => kind !== "behavioral").length,
},
claims: {
items: page,
offset,
limit,
total: results.length,
next_offset:
offset + page.length < results.length ? offset + page.length : null,
},
recommended_probes: probes.items,
evidence_links: evidenceLinks,
limitations: [
"Pass means every declared claim passed; it does not establish global implementation equivalence.",
...(probes.truncated
? ["Recommended probes were truncated at the 2,000-item limit."]
: []),
],
};
return reconstructionVerificationResultSchema.parse(output);
};
const evaluateClaim = (
claim: Claim,
records: ReadonlyMap<string, Evidence>,
heads: ReadonlyMap<string, ResidualUnknown>,
): ClaimResult => {
const comparison = records.get(claim.comparison_evidence_id);
if (comparison === undefined)
throw new TypeError(
`Missing comparison Evidence ${claim.comparison_evidence_id}`,
);
validateComparisonIdentity(claim, comparison);
const sides = sourceSides(claim, comparison);
const allSourceIds = [...sides.left, ...sides.right];
if (new Set(allSourceIds).size !== allSourceIds.length)
throw new TypeError(
"Comparison source Evidence must be unique and two-sided",
);
if (!sameSet(comparison.evidence_links, allSourceIds))
throw new TypeError(
"Comparison Evidence closure disagrees with its parameters",
);
const sources = allSourceIds.map((id) => {
const source = records.get(id);
if (source === undefined)
throw new TypeError(`Missing source Evidence ${id}`);
return source;
});
validateSourceKinds(claim, sources);
const observed = observedStatus(claim, comparison);
const limitations = authorityLimitations(claim, sources, sides);
const relevantUnknowns = reconstructionClaimUnknowns(
heads,
comparison.evidence_id,
allSourceIds,
);
const status = classify(observed, limitations, relevantUnknowns);
const displayedUnknowns = relevantUnknowns.slice(0, 100);
const hiddenUnknownCount = relevantUnknowns.length - displayedUnknowns.length;
return reconstructionClaimResultSchema.parse({
claim_id: claim.claim_id,
kind: claim.kind,
dimension: claim.dimension,
status,
observed_status: observed,
comparison_evidence_id: comparison.evidence_id,
left_evidence_ids: sides.left,
right_evidence_ids: sides.right,
evidence_links: uniqueSorted([comparison.evidence_id, ...allSourceIds]),
unknown_ids: displayedUnknowns.map(({ unknown_id: id }) => id),
limitations: uniqueSorted([
...comparison.limitations,
...limitations,
...displayedUnknowns.map(
({ question }) => `Residual unknown: ${question}`,
),
...(hiddenUnknownCount > 0
? [
`${hiddenUnknownCount} additional active residual unknowns affect this claim but are omitted by the 100-item display limit.`,
]
: []),
]),
});
};
const validateSourceKinds = (
claim: Claim,
sources: readonly Evidence[],
): void => {
const expected =
claim.kind === "behavioral"
? ["capture_process_scenario", "rea.process-capture/v2"]
: claim.kind === "structural-function"
? ["analyze_function", "rea.analysis/v2"]
: ["inventory_artifact", "rea.analysis/v2"];
if (
sources.some(
(source) =>
source.operation !== expected[0] ||
source.predicate_type !== expected[1],
)
)
throw new TypeError(
"Comparison source Evidence has an unexpected observation type",
);
for (const source of sources) {
if (claim.kind === "behavioral")
processCaptureSchema.parse(source.normalized_result);
else if (claim.kind === "structural-function")
functionDossierSchema.parse(source.normalized_result);
else artifactInventoryResultSchema.parse(source.normalized_result);
}
};
const validateComparisonIdentity = (claim: Claim, evidence: Evidence): void => {
parseEvidence(evidence);
const expected = providers[claim.kind];
if (
evidence.operation !== expected.operation ||
evidence.predicate_type !== expected.predicate ||
evidence.provider.id !== expected.id ||
evidence.provider.name !== expected.name ||
evidence.provider.version !== "1" ||
evidence.confidence !== "derived" ||
evidence.authority !== "analyst-inference" ||
evidence.subject !== null
)
throw new TypeError("Comparison Evidence identity or authority disagrees");
};
const sourceSides = (
claim: Claim,
evidence: Evidence,
): { left: string[]; right: string[] } => {
const parameters = evidence.parameters;
if (claim.kind === "behavioral") {
const parsed = z
.object({
left_evidence_id: evidenceIdSchema,
right_evidence_id: evidenceIdSchema,
})
.passthrough()
.parse(parameters);
return {
left: [parsed.left_evidence_id],
right: [parsed.right_evidence_id],
};
}
const parsed = z
.object({
left_evidence_ids: z.array(evidenceIdSchema).min(1).max(100),
right_evidence_ids: z.array(evidenceIdSchema).min(1).max(100),
})
.passthrough()
.parse(parameters);
return { left: parsed.left_evidence_ids, right: parsed.right_evidence_ids };
};
const observedStatus = (claim: Claim, evidence: Evidence): ObservedStatus => {
if (claim.kind === "behavioral") {
const result = processCaptureComparisonSchema.parse(
evidence.normalized_result,
);
validateProcessStatus(result);
return claim.dimension === "overall"
? result.status
: result[claim.dimension];
}
if (claim.kind === "structural-function") {
const result = functionComparisonResultSchema.parse(
evidence.normalized_result,
);
if (claim.dimension === "overall") return result.status;
const dimension = result.dimensions.find(
(item) => item.dimension === claim.dimension,
);
if (dimension === undefined)
throw new TypeError("Function comparison dimension missing");
if (
dimension.evidence_links.some(
(id) => !evidence.evidence_links.includes(id),
)
)
throw new TypeError(
"Function dimension cites Evidence outside comparison closure",
);
return dimension.status;
}
const result = artifactComparisonResultSchema.parse(
evidence.normalized_result,
);
if (
result.changes.offset !== 0 ||
result.changes.next_offset !== null ||
result.changes.items.length !== result.changes.total
)
return "unknown";
if (
result.changes.items.some((item) =>
item.evidence_links.some((id) => !evidence.evidence_links.includes(id)),
)
)
throw new TypeError(
"Artifact change cites Evidence outside comparison closure",
);
return result.status;
};
const validateProcessStatus = (
result: z.infer<typeof processCaptureComparisonSchema>,
): void => {
const dimensions = [
result.terminal,
result.exit,
result.filesystem,
result.protocol,
result.process,
];
const expected = dimensions.includes("truncated")
? "truncated"
: dimensions.some((status) =>
["added", "removed", "changed"].includes(status),
)
? "changed"
: dimensions.includes("unknown")
? "unknown"
: "unchanged";
if (result.status !== expected)
throw new TypeError("Process comparison status contradicts its dimensions");
};
const authorityLimitations = (
claim: Claim,
sources: readonly Evidence[],
sides: {
readonly left: readonly string[];
readonly right: readonly string[];
},
): string[] => {
const requiredAuthority =
claim.kind === "behavioral" ? "controlled-replay" : "shipped-artifact";
const limitations: string[] = [];
if (
sources.some(
(source) =>
source.authority !== requiredAuthority ||
(claim.kind === "structural-function"
? source.confidence === "inferred"
: source.confidence !== "observed"),
)
)
limitations.push(
`Source Evidence lacks qualifying ${requiredAuthority} authority.`,
);
if (
claim.kind !== "behavioral" &&
sources.some(({ subject }) => subject === null)
)
limitations.push("Structural source artifact identity is unavailable.");
if (claim.kind === "behavioral") {
const byId = new Map(sources.map((source) => [source.evidence_id, source]));
const left = byId.get(sides.left[0] ?? "");
const right = byId.get(sides.right[0] ?? "");
if (
left?.environment === null ||
right?.environment === null ||
JSON.stringify(left?.environment) !== JSON.stringify(right?.environment)
)
limitations.push(
"Controlled replay environments are unavailable or incompatible.",
);
}
return limitations;
};
const classify = (
observed: ObservedStatus,
limitations: readonly string[],
unknowns: readonly ResidualUnknown[],
): ClaimResult["status"] => {
if (limitations.length > 0) return "unknown";
if (["changed", "added", "removed"].includes(observed)) return "fail";
if (observed === "unknown" || observed === "truncated" || unknowns.length > 0)
return "unknown";
return "pass";
};
const digest = (value: unknown): string => {
const encoded = canonicalize(value);
if (encoded === undefined)
throw new TypeError("Specification canonicalization failed");
return createHash("sha256").update(encoded).digest("hex");
};
const enforceInputSize = (value: unknown): void => {
const encoded = canonicalize(value);
if (
encoded === undefined ||
Buffer.byteLength(encoded, "utf8") > MAX_CANONICAL_BYTES
)
throw new TypeError("Reconstruction Evidence bundle exceeds byte limit");
};
const count = (
items: readonly ClaimResult[],
status: ClaimResult["status"],
): number => items.filter((item) => item.status === status).length;
const sameSet = (left: readonly string[], right: readonly string[]): boolean =>
left.length === right.length &&
uniqueSorted(left).every((id, index) => id === uniqueSorted(right)[index]);
const uniqueSorted = (values: readonly string[]): string[] =>
[...new Set(values)].sort((left, right) => left.localeCompare(right, "en"));

149
src/domain/reconstructionVerificationSchemas.ts Normal file
View File

@ -0,0 +1,149 @@
import { z } from "zod";
import { evidenceBundleSchema } from "./evidenceBundle.js";
const evidenceIdSchema = z.string().regex(/^ev_[a-f0-9]{64}$/u);
const unknownIdSchema = z.string().regex(/^unk_[a-f0-9]{64}$/u);
const digestSchema = z.string().regex(/^[a-f0-9]{64}$/u);
const claimIdSchema = z.string().regex(/^[A-Za-z][A-Za-z0-9._-]{0,99}$/u);
const titleSchema = z.string().trim().min(1).max(500);
const verificationStatusSchema = z.enum(["pass", "fail", "unknown"]);
const commonClaim = {
claim_id: claimIdSchema,
title: titleSchema,
comparison_evidence_id: evidenceIdSchema,
};
const behavioralClaimSchema = z.object({
...commonClaim,
kind: z.literal("behavioral"),
dimension: z.enum([
"overall",
"terminal",
"exit",
"filesystem",
"protocol",
"process",
]),
});
const functionClaimSchema = z.object({
...commonClaim,
kind: z.literal("structural-function"),
dimension: z.enum([
"overall",
"identity",
"pseudocode",
"assembly",
"comments",
"calls",
"references",
"strings_names",
"cfg",
]),
});
const artifactClaimSchema = z.object({
...commonClaim,
kind: z.literal("structural-artifact"),
dimension: z.literal("overall"),
});
const reconstructionClaimSchema = z.discriminatedUnion("kind", [
behavioralClaimSchema,
functionClaimSchema,
artifactClaimSchema,
]);
/** Finite, typed behavioral and structural specification. */
export const reconstructionSpecificationSchema = z
.object({
schema_version: z.literal(1),
name: z.string().trim().min(1).max(200),
claims: z.array(reconstructionClaimSchema).min(1).max(100),
})
.superRefine((value, context) => {
const ids = value.claims.map(({ claim_id: id }) => id);
if (new Set(ids).size !== ids.length)
context.addIssue({ code: "custom", message: "Claim IDs must be unique" });
const selectors = value.claims.map(
({ kind, comparison_evidence_id: id, dimension }) =>
`${kind}:${id}:${dimension}`,
);
if (new Set(selectors).size !== selectors.length)
context.addIssue({
code: "custom",
message: "Claim comparison selectors must be unique",
});
});
/** Bounded reconstruction-verification input. */
export const reconstructionVerificationInputSchema = z.object({
specification: reconstructionSpecificationSchema,
evidence_bundle: evidenceBundleSchema,
offset: z.number().int().min(0).default(0),
limit: z.number().int().min(1).max(100).default(100),
unknown_registry_approved: z.literal(true).optional(),
});
export const reconstructionClaimResultSchema = z.object({
claim_id: claimIdSchema,
kind: z.enum(["behavioral", "structural-function", "structural-artifact"]),
dimension: z.string().min(1),
status: verificationStatusSchema,
observed_status: z.enum([
"unchanged",
"added",
"removed",
"changed",
"truncated",
"unknown",
]),
comparison_evidence_id: evidenceIdSchema,
left_evidence_ids: z.array(evidenceIdSchema).min(1).max(100),
right_evidence_ids: z.array(evidenceIdSchema).min(1).max(100),
evidence_links: z.array(evidenceIdSchema).min(3).max(201),
unknown_ids: z.array(unknownIdSchema).max(100),
limitations: z.array(z.string()),
});
/** Evidence-cited result over every declared claim, independently paged. */
export const reconstructionVerificationResultSchema = z.object({
status: verificationStatusSchema,
specification_sha256: digestSchema,
summary: z.object({
total: z.number().int().min(1).max(100),
passed: z.number().int().min(0),
failed: z.number().int().min(0),
unknown: z.number().int().min(0),
behavioral: z.number().int().min(0),
structural: z.number().int().min(0),
}),
claims: z.object({
items: z.array(reconstructionClaimResultSchema).max(100),
offset: z.number().int().min(0),
limit: z.number().int().min(1).max(100),
total: z.number().int().min(1).max(100),
next_offset: z.number().int().min(0).nullable(),
}),
recommended_probes: z
.array(
z.object({
operation: z.string().min(1).max(200),
rationale: z.string().min(1).max(1_000),
claim_ids: z.array(claimIdSchema).min(1).max(100),
unknown_ids: z.array(unknownIdSchema).max(100),
}),
)
.max(2_000),
evidence_links: z.array(evidenceIdSchema).min(3).max(20_100),
limitations: z.array(z.string()),
});
export type ReconstructionClaim = z.infer<typeof reconstructionClaimSchema>;
export type ReconstructionClaimResult = z.infer<
typeof reconstructionClaimResultSchema
>;
export type ReconstructionVerificationResult = z.infer<
typeof reconstructionVerificationResultSchema
>;
export type ReconstructionObservedStatus =
ReconstructionClaimResult["observed_status"];

516
src/domain/referenceSourceClassification.ts Normal file
View File

@ -0,0 +1,516 @@
import { posix } from "node:path";
const SOURCE_CLASSIFICATIONS = [
"source",
"test",
"config",
"manifest",
"generated",
"vendor",
"documentation",
"unknown",
] as const;
export type ReferenceSourceClassification =
(typeof SOURCE_CLASSIFICATIONS)[number];
const CODE_EXTENSIONS = new Set([
"js",
"mjs",
"cjs",
"jsx",
"ts",
"tsx",
"mts",
"cts",
"py",
"rs",
"go",
"java",
"c",
"h",
"cpp",
"hpp",
"cc",
"cxx",
"swift",
"m",
"mm",
"sh",
]);
const LANGUAGE_BY_EXTENSION = new Map<string, string>([
["js", "JavaScript"],
["mjs", "JavaScript"],
["cjs", "JavaScript"],
["jsx", "JSX"],
["ts", "TypeScript"],
["tsx", "TSX"],
["mts", "TypeScript"],
["cts", "TypeScript"],
["py", "Python"],
["rs", "Rust"],
["go", "Go"],
["java", "Java"],
["c", "C"],
["h", "C"],
["cpp", "C++"],
["hpp", "C++"],
["cc", "C++"],
["cxx", "C++"],
["swift", "Swift"],
["m", "Objective-C"],
["mm", "Objective-C++"],
["sh", "Shell"],
["json", "JSON"],
["jsonc", "JSON"],
["yaml", "YAML"],
["yml", "YAML"],
["toml", "TOML"],
["ini", "INI"],
["cfg", "INI"],
["conf", "INI"],
["md", "Markdown"],
["markdown", "Markdown"],
["txt", "Text"],
["html", "HTML"],
["htm", "HTML"],
["css", "CSS"],
["scss", "SCSS"],
["sass", "Sass"],
["less", "Less"],
["xml", "XML"],
["svg", "SVG"],
]);
const MANIFEST_FILENAMES = new Set([
"package.json",
"package-lock.json",
"yarn.lock",
"pnpm-lock.yaml",
"npm-shrinkwrap.json",
"composer.json",
"composer.lock",
"gemfile",
"gemfile.lock",
"cargo.toml",
"cargo.lock",
"go.mod",
"go.sum",
"pom.xml",
"build.gradle",
"build.gradle.kts",
"settings.gradle",
"requirements.txt",
"setup.py",
"pyproject.toml",
"pipfile",
"pipfile.lock",
"setup.cfg",
"manifest.in",
"CMakeLists.txt",
"meson.build",
"configure",
"configure.ac",
"makefile",
"makefile.am",
"dockerfile",
"docker-compose.yml",
"docker-compose.yaml",
"kubernetes.yml",
"kubernetes.yaml",
"kustomization.yml",
"kustomization.yaml",
".releaserc",
".releaserc.json",
".releaserc.yml",
".releaserc.yaml",
"release-please-config.json",
"release-please-manifest.json",
]);
const CONFIG_FILENAMES = new Set([
".gitignore",
".gitattributes",
".gitmodules",
".editorconfig",
".npmrc",
".nvmrc",
".yarnrc",
".yarnrc.yml",
"eslint.config.js",
"eslint.config.mjs",
"eslint.config.ts",
".prettierignore",
".prettierrc",
".prettierrc.json",
".oxlintrc.json",
"tsconfig.json",
"tsconfig.build.json",
"jsconfig.json",
"vitest.config.ts",
"wrangler.json",
"wrangler.jsonc",
"dependabot.yml",
".github",
".husky",
]);
const DOCUMENTATION_EXTENSIONS = new Set([
"md",
"markdown",
"mdx",
"rst",
"txt",
"adoc",
"asciidoc",
]);
const DOCUMENTATION_FILENAMES = new Set([
"readme",
"readme.md",
"readme.markdown",
"readme.txt",
"changelog",
"changelog.md",
"changes.md",
"history.md",
"news.md",
"license",
"license.md",
"license.txt",
"copying",
"copying.md",
"authors",
"authors.md",
"contributing",
"contributing.md",
"code_of_conduct",
"code_of_conduct.md",
"security",
"security.md",
"support",
"support.md",
"faq",
"faq.md",
]);
const GENERATED_DIRECTORY_NAMES = new Set([
"dist",
"build",
"out",
"target",
"coverage",
"htmlcov",
"site",
"public",
".next",
".nuxt",
".svelte-kit",
".astro",
"storybook-static",
"docs/api",
".codex",
".devin",
".factory",
"__pycache__",
".pytest_cache",
".mypy_cache",
".ruff_cache",
".tox",
".nyc_output",
]);
const VENDOR_DIRECTORY_NAMES = new Set([
"node_modules",
"vendor",
"bower_components",
"third_party",
"third-party",
"thirdparty",
"libs",
"external",
]);
const SOURCE_DIRECTORY_NAMES = new Set([
"src",
"source",
"sources",
"lib",
"libs",
"packages",
"pkg",
"app",
"apps",
"bin",
"scripts",
"bridge",
"components",
"domain",
"application",
"server",
"contracts",
"hopper",
"tests",
"test",
"__tests__",
"spec",
"specs",
"testing",
"fixtures",
]);
const TEST_DIRECTORY_NAMES = new Set([
"tests",
"test",
"__tests__",
"spec",
"specs",
"testing",
]);
const DOCUMENTATION_DIRECTORY_NAMES = new Set([
"docs",
"doc",
"documentation",
"guides",
"examples",
"tutorials",
"wiki",
"man",
]);
const SECRET_BASENAMES = new Set([
"id_rsa",
"id_ed25519",
"pgpass",
".netrc",
".npmrc",
".pypirc",
".envrc",
".env.local",
".env.production",
".env.development",
".env.test",
".env.staging",
".env.example",
"auth.json",
"auth.yaml",
"auth.yml",
"token.json",
"tokens.json",
"credentials.json",
"secrets.json",
"google-services.json",
"service-account.json",
".htpasswd",
]);
const SECRET_BASENAME_PREFIXES = [
".env",
".env.",
"private",
"secret",
"secrets",
"credential",
"credentials",
"token",
"tokens",
"password",
"passwords",
"auth",
"apikey",
"api-key",
"api_key",
"service-account",
"kubeconfig",
];
const SECRET_BASENAME_SUFFIXES = [
".pem",
".key",
".p12",
".gpg",
".pgp",
".keystore",
];
const SECRET_PATH_SEGMENTS = new Set([
".ssh",
".aws",
".gnupg",
".pki",
".docker",
".secrets",
"secrets",
"secret",
"private",
"credentials",
"tokens",
"passwords",
"passphrase",
"kubernetes",
"k8s",
]);
const filenameIsTest = (filename: string): boolean => {
const lower = filename.toLowerCase();
if (lower.includes(".test.") || lower.includes(".spec.")) return true;
if (lower.startsWith("test_") || lower.startsWith("spec_")) return true;
if (lower.endsWith("_test") || lower.endsWith("_spec")) return true;
return false;
};
const filenameIsGenerated = (filename: string): boolean => {
const lower = filename.toLowerCase();
if (lower.endsWith(".min.js") || lower.endsWith(".min.css")) return true;
if (lower.endsWith(".map")) return true;
if (lower.endsWith(".lock")) return true;
if (lower.endsWith(".d.ts") || lower.endsWith(".d.ts.map")) return true;
if (lower.endsWith(".d.mts") || lower.endsWith(".d.mts.map")) return true;
if (lower.endsWith(".d.cts") || lower.endsWith(".d.cts.map")) return true;
return false;
};
const basenameParts = (filename: string): { base: string; ext: string } => {
const lastDot = filename.lastIndexOf(".");
if (lastDot <= 0) return { base: filename, ext: "" };
return { base: filename.slice(0, lastDot), ext: filename.slice(lastDot + 1) };
};
const basenameMatch = (filename: string, set: ReadonlySet<string>): boolean => {
const lower = filename.toLowerCase();
return set.has(lower);
};
const isPathUnderAny = (path: string, names: ReadonlySet<string>): boolean => {
for (const segment of path.split("/")) {
if (segment === "") continue;
if (names.has(segment.toLowerCase())) return true;
}
return false;
};
const secretPrefixMatches = (filename: string): boolean => {
const lower = filename.toLowerCase();
for (const prefix of SECRET_BASENAME_PREFIXES) {
const prefixLower = prefix.toLowerCase();
if (!lower.startsWith(prefixLower)) continue;
const nextIndex = prefixLower.length;
if (nextIndex < lower.length) {
const next = lower[nextIndex];
if (next !== "." && next !== "_" && next !== "-") continue;
}
const extension = basenameParts(filename).ext.toLowerCase();
if (CODE_EXTENSIONS.has(extension)) continue;
return true;
}
return false;
};
/** Return true when a path or any parent segment matches common secret patterns. */
export const isSecretLikePath = (path: string): boolean => {
const parts = path.split("/").filter((segment) => segment !== "");
for (const segment of parts) {
const lower = segment.toLowerCase();
if (SECRET_PATH_SEGMENTS.has(lower)) return true;
}
const filename = parts[parts.length - 1];
if (filename === undefined) return false;
const lower = filename.toLowerCase();
if (SECRET_BASENAMES.has(lower)) return true;
if (secretPrefixMatches(filename)) return true;
for (const suffix of SECRET_BASENAME_SUFFIXES) {
if (lower.endsWith(suffix.toLowerCase())) return true;
}
return false;
};
/** Detect a source language from a POSIX path. */
export const detectReferenceSourceLanguage = (path: string): string | null => {
const filename = posix.basename(path);
if (filename === "Dockerfile" || filename.startsWith("Dockerfile."))
return "Dockerfile";
if (filename.toLowerCase().startsWith("dockerfile")) return "Dockerfile";
const lower = filename.toLowerCase();
if (lower.endsWith(".d.ts")) return "TypeScript";
if (lower.endsWith(".d.mts")) return "TypeScript";
if (lower.endsWith(".d.cts")) return "TypeScript";
const parts = basenameParts(filename);
const extension = parts.ext.toLowerCase();
return LANGUAGE_BY_EXTENSION.get(extension) ?? null;
};
/** Classify a reference source path deterministically. */
export const classifyReferenceSourcePath = (
path: string,
): ReferenceSourceClassification[] => {
const classifications: Set<ReferenceSourceClassification> = new Set();
const parts = path.split("/").filter((segment) => segment !== "");
const filename = parts[parts.length - 1] ?? "";
const lower = filename.toLowerCase();
if (parts.length === 0) {
classifications.add("source");
return [...classifications].sort();
}
addDirectoryClassifications(path, classifications);
addFilenameClassifications(filename, classifications);
if (
lower.endsWith(".json") ||
lower.endsWith(".yaml") ||
lower.endsWith(".yml")
) {
if (!classifications.has("manifest") && !classifications.has("config"))
classifications.add("config");
}
if (isPathUnderAny(path, SOURCE_DIRECTORY_NAMES)) {
if (!classifications.has("vendor") && !classifications.has("generated"))
classifications.add("source");
}
if (classifications.size === 0) {
if (lower === "license" || lower === "copying")
classifications.add("documentation");
else classifications.add("unknown");
}
return [...classifications].sort();
};
const addDirectoryClassifications = (
path: string,
classifications: Set<ReferenceSourceClassification>,
): void => {
if (isPathUnderAny(path, TEST_DIRECTORY_NAMES)) classifications.add("test");
if (isPathUnderAny(path, DOCUMENTATION_DIRECTORY_NAMES))
classifications.add("documentation");
if (isPathUnderAny(path, VENDOR_DIRECTORY_NAMES))
classifications.add("vendor");
if (isPathUnderAny(path, GENERATED_DIRECTORY_NAMES))
classifications.add("generated");
};
const addFilenameClassifications = (
filename: string,
classifications: Set<ReferenceSourceClassification>,
): void => {
const extension = basenameParts(filename).ext.toLowerCase();
if (filenameIsTest(filename)) classifications.add("test");
if (filenameIsGenerated(filename)) classifications.add("generated");
if (basenameMatch(filename, MANIFEST_FILENAMES))
classifications.add("manifest");
if (basenameMatch(filename, CONFIG_FILENAMES)) classifications.add("config");
if (DOCUMENTATION_EXTENSIONS.has(extension))
classifications.add("documentation");
if (DOCUMENTATION_FILENAMES.has(filename.toLowerCase()))
classifications.add("documentation");
if (CODE_EXTENSIONS.has(extension)) classifications.add("source");
};

524
src/domain/referenceSourceGraph.ts Normal file
View File

@ -0,0 +1,524 @@
import { createHash } from "node:crypto";
import { posix } from "node:path";
import canonicalize from "canonicalize";
import { z } from "zod";
const MAX_PATH = 4_096;
const digestSchema = z.string().regex(/^[a-f0-9]{64}$/u);
const boundedTextSchema = z.string().min(1).max(1_024);
const relativePathSchema = z
.string()
.min(1)
.max(MAX_PATH)
.refine(
(path) =>
!path.startsWith("/") &&
!path.includes("\\") &&
path
.split("/")
.every((part) => part !== "" && part !== "." && part !== ".."),
"Expected a normalized relative POSIX path without traversal",
);
const classificationSchema = z.enum([
"source",
"test",
"config",
"manifest",
"generated",
"vendor",
"documentation",
"unknown",
]);
const entryBaseShape = {
path: relativePathSchema,
classifications: z.array(classificationSchema).min(1).max(8),
limitations: z.array(boundedTextSchema).max(100),
};
const sourceFileSchema = z
.strictObject({
...entryBaseShape,
kind: z.literal("file"),
sha256: digestSchema.nullable(),
size: z.number().int().min(0).max(Number.MAX_SAFE_INTEGER).nullable(),
language: z.string().min(1).max(100).nullable(),
content_state: z.enum([
"hashed",
"redacted-secret",
"excluded",
"unreadable",
"too-large",
"unknown",
]),
})
.superRefine((file, context) => {
checkSortedUnique(file.classifications, "classifications", context);
if ((file.content_state === "hashed") !== (file.sha256 !== null))
context.addIssue({
code: "custom",
message: "Only hashed files have a content digest",
path: ["sha256"],
});
if (file.content_state === "hashed" && file.size === null)
context.addIssue({
code: "custom",
message: "Hashed files require a byte size",
path: ["size"],
});
});
const sourceDirectorySchema = z.strictObject({
...entryBaseShape,
kind: z.literal("directory"),
tree_state: z.enum([
"enumerated",
"partial",
"excluded",
"unreadable",
"unknown",
]),
});
const symlinkTargetSchema = z
.string()
.min(1)
.max(MAX_PATH)
.refine(
(target) =>
target === "<outside-root>" ||
(!target.startsWith("/") &&
!target.includes("\\") &&
posix.normalize(target) === target),
"Expected a normalized POSIX symlink target",
);
const sourceSymlinkSchema = z.strictObject({
...entryBaseShape,
kind: z.literal("symlink"),
target: symlinkTargetSchema,
target_state: z.enum([
"internal",
"external",
"missing",
"unreadable",
"unknown",
]),
});
const sourceEntrySchema = z.discriminatedUnion("kind", [
sourceFileSchema,
sourceDirectorySchema,
sourceSymlinkSchema,
]);
const sourceRelationshipSchema = z.strictObject({
from_path: relativePathSchema,
to: z.string().min(1).max(MAX_PATH),
kind: z.enum(["imports", "requires", "references", "declares-module"]),
resolution: z.enum(["internal", "external", "unresolved", "unknown"]),
parse_state: z.enum(["parsed", "partial", "unknown"]),
});
const parseFailureSchema = z.strictObject({
path: relativePathSchema,
parser: z.string().min(1).max(100),
reason: boundedTextSchema,
});
const exclusionSchema = z.strictObject({
path: relativePathSchema,
reason: z.enum([
"configured-secret",
"symlink-escape",
"size-limit",
"inventory-limit",
"unreadable",
"caller-excluded",
]),
});
const vcsSchema = z
.discriminatedUnion("kind", [
z.strictObject({
kind: z.literal("git"),
head: z.string().regex(/^(?:[a-f0-9]{40}|[a-f0-9]{64})$/u),
dirty: z.boolean().nullable(),
}),
z.strictObject({
kind: z.literal("none"),
head: z.null(),
dirty: z.null(),
}),
z.strictObject({
kind: z.literal("unknown"),
head: z.null(),
dirty: z.null(),
}),
])
.nullable();
const graphShape = {
schema: z.literal("HistoricalSourceGraph/v1"),
authority: z.literal("historical-reference"),
root_alias: z.literal("$REFERENCE_ROOT"),
root_sha256: digestSchema,
inventory_state: z.enum(["complete", "partial", "unknown"]),
entries: z.array(sourceEntrySchema).max(100_000),
relationships: z.array(sourceRelationshipSchema).max(200_000),
parse_failures: z.array(parseFailureSchema).max(10_000),
exclusions: z.array(exclusionSchema).max(100_000),
languages: z.array(z.string().min(1).max(100)).max(1_000),
manifests: z.array(relativePathSchema).max(10_000),
vcs: vcsSchema,
provenance: z.strictObject({
importer: z.string().min(1).max(100),
importer_version: z.string().min(1).max(100).nullable(),
caller: z
.string()
.min(1)
.max(200)
.regex(/^[\w .:@/+-]+$/u),
}),
limitations: z.array(boundedTextSchema).max(1_000),
};
const historicalSourceGraphStructureSchema = z
.strictObject(graphShape)
.superRefine(checkGraphInvariants);
export const historicalSourceGraphSchema =
historicalSourceGraphStructureSchema.superRefine((graph, context) => {
if (
computeRootSha256(graph.entries, graph.exclusions) !== graph.root_sha256
)
context.addIssue({
code: "custom",
message: "Historical source root commitment does not match entries",
path: ["root_sha256"],
});
});
const historicalSourceGraphInputSchema = z.strictObject({
...graphShape,
root_sha256: z.never().optional(),
});
const historicalSourceManifestBaseSchema = z.strictObject({
schema: z.literal("HistoricalSourceManifest/v1"),
authority: z.literal("historical-reference"),
manifest_id: z.string().regex(/^hsm_[a-f0-9]{64}$/u),
graph_sha256: digestSchema,
root_sha256: digestSchema,
inventory_state: z.enum(["complete", "partial", "unknown"]),
entry_count: z.number().int().min(0).max(100_000),
relationship_count: z.number().int().min(0).max(200_000),
});
export type HistoricalSourceGraph = z.infer<typeof historicalSourceGraphSchema>;
export type HistoricalSourceGraphInput = z.infer<
typeof historicalSourceGraphInputSchema
>;
export type HistoricalSourceManifest = z.infer<
typeof historicalSourceManifestBaseSchema
>;
const canonicalJson = (value: unknown): string => {
const serialized = canonicalize(value);
if (serialized === undefined)
throw new TypeError("RFC 8785 canonicalization rejected a graph value");
return serialized;
};
const digestCanonical = (value: unknown): string =>
createHash("sha256").update(canonicalJson(value)).digest("hex");
const relationshipKey = (
relationship: z.infer<typeof sourceRelationshipSchema>,
): string =>
`${relationship.from_path}\u0000${relationship.to}\u0000${relationship.kind}\u0000${relationship.resolution}\u0000${relationship.parse_state}`;
const compareCodePoints = (left: string, right: string): number =>
left < right ? -1 : left > right ? 1 : 0;
const checkSortedUnique = (
values: readonly string[],
field: string,
context: z.RefinementCtx,
): void => {
for (let index = 1; index < values.length; index += 1)
if (compareCodePoints(values[index - 1] ?? "", values[index] ?? "") >= 0) {
context.addIssue({
code: "custom",
message: `${field} must be sorted and unique by Unicode code point`,
path: [field, index],
});
return;
}
};
const expectedLanguages = (
entries: readonly z.infer<typeof sourceEntrySchema>[],
): string[] =>
[
...new Set(
entries.flatMap((entry) =>
entry.kind === "file" && entry.language !== null
? [entry.language]
: [],
),
),
].sort(compareCodePoints);
const expectedManifests = (
entries: readonly z.infer<typeof sourceEntrySchema>[],
): string[] =>
entries
.flatMap((entry) =>
entry.kind === "file" && entry.classifications.includes("manifest")
? [entry.path]
: [],
)
.sort(compareCodePoints);
type GraphStructure = z.infer<typeof historicalSourceGraphStructureSchema>;
const checkRelationships = (
graph: GraphStructure,
paths: ReadonlySet<string>,
context: z.RefinementCtx,
): void => {
const files = new Set(
graph.entries.flatMap((entry) =>
entry.kind === "file" ? [entry.path] : [],
),
);
for (const [index, relationship] of graph.relationships.entries()) {
if (!files.has(relationship.from_path))
context.addIssue({
code: "custom",
message: "Relationship source must name an inventoried file",
path: ["relationships", index, "from_path"],
});
if (relationship.resolution !== "internal") continue;
const parsed = relativePathSchema.safeParse(relationship.to);
if (!parsed.success || !paths.has(parsed.data))
context.addIssue({
code: "custom",
message:
"Internal relationship target must be normalized and inventoried",
path: ["relationships", index, "to"],
});
}
};
const checkSymlinks = (
graph: GraphStructure,
paths: ReadonlySet<string>,
context: z.RefinementCtx,
): void => {
for (const [index, entry] of graph.entries.entries()) {
if (entry.kind !== "symlink") continue;
if (
(entry.target_state === "external") !==
(entry.target === "<outside-root>")
)
context.addIssue({
code: "custom",
message:
"External symlink targets must use the sanitized <outside-root> sentinel",
path: ["entries", index, "target"],
});
if (entry.target_state !== "internal") continue;
const resolved = posix.normalize(
posix.join(posix.dirname(entry.path), entry.target),
);
if (resolved.startsWith("../") || resolved === ".." || !paths.has(resolved))
context.addIssue({
code: "custom",
message: "Internal symlink target must resolve to an inventoried entry",
path: ["entries", index, "target"],
});
}
};
const checkCompleteInventory = (
graph: GraphStructure,
context: z.RefinementCtx,
): void => {
if (graph.inventory_state !== "complete") return;
const partialEntry = graph.entries.some(
(entry) =>
entry.limitations.length > 0 ||
(entry.kind === "file" && entry.content_state !== "hashed") ||
(entry.kind === "directory" && entry.tree_state !== "enumerated") ||
(entry.kind === "symlink" && entry.target_state !== "internal"),
);
const partialRelationship = graph.relationships.some(
({ parse_state, resolution }) =>
parse_state !== "parsed" ||
["unresolved", "unknown"].includes(resolution),
);
if (
!partialEntry &&
!partialRelationship &&
graph.parse_failures.length === 0 &&
graph.exclusions.length === 0 &&
graph.limitations.length === 0
)
return;
context.addIssue({
code: "custom",
message:
"Complete inventory cannot contain partial states, exclusions, failures, or limitations",
path: ["inventory_state"],
});
};
function checkGraphInvariants(
graph: GraphStructure,
context: z.RefinementCtx,
): void {
checkSortedUnique(
graph.entries.map(({ path }) => path),
"entries",
context,
);
for (const [index, entry] of graph.entries.entries())
checkSortedUnique(
entry.classifications,
`entries.${index}.classifications`,
context,
);
checkSortedUnique(graph.languages, "languages", context);
checkSortedUnique(graph.manifests, "manifests", context);
checkSortedUnique(
graph.relationships.map(relationshipKey),
"relationships",
context,
);
checkSortedUnique(
graph.parse_failures.map(({ path, parser }) => `${path}\u0000${parser}`),
"parse_failures",
context,
);
checkSortedUnique(
graph.exclusions.map(({ path, reason }) => `${path}\u0000${reason}`),
"exclusions",
context,
);
if (
canonicalJson(graph.languages) !==
canonicalJson(expectedLanguages(graph.entries))
)
context.addIssue({
code: "custom",
message: "languages must be derived from file entries",
path: ["languages"],
});
if (
canonicalJson(graph.manifests) !==
canonicalJson(expectedManifests(graph.entries))
)
context.addIssue({
code: "custom",
message: "manifests must be derived from classified file entries",
path: ["manifests"],
});
const paths = new Set(graph.entries.map(({ path }) => path));
checkRelationships(graph, paths, context);
checkSymlinks(graph, paths, context);
checkCompleteInventory(graph, context);
}
const rootCommitment = (
entries: readonly z.infer<typeof sourceEntrySchema>[],
exclusions: readonly z.infer<typeof exclusionSchema>[],
): unknown => ({
entries: entries.map((entry) => {
switch (entry.kind) {
case "file":
return {
kind: entry.kind,
path: entry.path,
sha256: entry.sha256,
size: entry.size,
content_state: entry.content_state,
};
case "directory":
return {
kind: entry.kind,
path: entry.path,
tree_state: entry.tree_state,
};
case "symlink":
return {
kind: entry.kind,
path: entry.path,
target: entry.target,
target_state: entry.target_state,
};
}
}),
exclusions,
});
const computeRootSha256 = (
entries: readonly z.infer<typeof sourceEntrySchema>[],
exclusions: readonly z.infer<typeof exclusionSchema>[],
): string => digestCanonical(rootCommitment(entries, exclusions));
/** Build and internally commit a normalized historical source graph. */
export const createHistoricalSourceGraph = (
input: unknown,
): HistoricalSourceGraph => {
const parsed = historicalSourceGraphInputSchema.parse(input);
return historicalSourceGraphSchema.parse({
...parsed,
root_sha256: computeRootSha256(parsed.entries, parsed.exclusions),
});
};
/** Parse a graph and reject stale or caller-invented root commitments. */
export const parseHistoricalSourceGraph = (
input: unknown,
): HistoricalSourceGraph => historicalSourceGraphSchema.parse(input);
/** Compute a deterministic graph commitment containing no absolute root path. */
export const computeHistoricalSourceGraphSha256 = (input: unknown): string =>
digestCanonical(parseHistoricalSourceGraph(input));
/** Build a deterministic, relocation-independent manifest for a source graph. */
export const createHistoricalSourceManifest = (
input: unknown,
): HistoricalSourceManifest => {
const graph = parseHistoricalSourceGraph(input);
const semantic = {
schema: "HistoricalSourceManifest/v1" as const,
authority: "historical-reference" as const,
graph_sha256: computeHistoricalSourceGraphSha256(graph),
root_sha256: graph.root_sha256,
inventory_state: graph.inventory_state,
entry_count: graph.entries.length,
relationship_count: graph.relationships.length,
};
return historicalSourceManifestBaseSchema.parse({
...semantic,
manifest_id: `hsm_${digestCanonical(semantic)}`,
});
};
/** Parse and verify a historical source manifest's semantic identifier. */
export const parseHistoricalSourceManifest = (
input: unknown,
): HistoricalSourceManifest => {
const manifest = historicalSourceManifestBaseSchema.parse(input);
const { manifest_id: manifestId, ...semantic } = manifest;
if (`hsm_${digestCanonical(semantic)}` !== manifestId)
throw new TypeError("Historical source manifest identifier does not match");
return manifest;
};

350
src/domain/referenceSourceImportParsing.ts Normal file
View File

@ -0,0 +1,350 @@
import { parse, type ParserPlugin } from "@babel/parser";
import type { CallExpression, File, Node, StringLiteral } from "@babel/types";
import {
isCallExpression,
isExportAllDeclaration,
isExportNamedDeclaration,
isIdentifier,
isImport,
isImportDeclaration,
isMemberExpression,
isNode,
isStringLiteral,
isTSExternalModuleReference,
isTSImportEqualsDeclaration,
isTSModuleDeclaration,
} from "@babel/types";
type ReferenceSourceImportKind =
| "imports"
| "requires"
| "references"
| "declares-module";
type ReferenceSourceImportResolution =
| "internal"
| "external"
| "unresolved"
| "unknown";
type ReferenceSourceImportParseState = "parsed" | "partial" | "unknown";
interface ReferenceSourceImportRelationship {
readonly from_path: string;
readonly to: string;
readonly kind: ReferenceSourceImportKind;
readonly resolution: ReferenceSourceImportResolution;
readonly parse_state: ReferenceSourceImportParseState;
}
interface ReferenceSourceParseFailure {
readonly path: string;
readonly parser: string;
readonly reason: string;
}
export interface ReferenceSourceImportParseResult {
readonly relationships: readonly ReferenceSourceImportRelationship[];
readonly parse_failures: readonly ReferenceSourceParseFailure[];
}
const MAX_REASON_LENGTH = 1_024;
const codeSourceLanguages = new Set(["JavaScript", "TypeScript", "JSX", "TSX"]);
const truncateReason = (reason: string): string =>
reason.length > MAX_REASON_LENGTH
? `${reason.slice(0, MAX_REASON_LENGTH - 3)}...`
: reason;
const safeModuleName = (node: Node | null | undefined): string | undefined => {
if (isStringLiteral(node)) return node.value;
return undefined;
};
const resolveSpecifier = (
specifier: string,
): ReferenceSourceImportResolution => {
if (specifier.startsWith(".") || specifier.startsWith("/")) return "internal";
if (specifier.startsWith("node:") || specifier.startsWith("data:"))
return "external";
return "external";
};
const isModuleExpression = (
expression: Node | null | undefined,
): expression is StringLiteral | CallExpression =>
isStringLiteral(expression) || isCallExpression(expression);
const moduleSpecifierFromExpression = (
expression: Node | null | undefined,
):
| { specifier: string; parseState: ReferenceSourceImportParseState }
| undefined => {
if (isStringLiteral(expression))
return { specifier: expression.value, parseState: "parsed" };
if (
isCallExpression(expression) &&
isImport(expression.callee) &&
expression.arguments.length > 0
) {
const first = expression.arguments[0];
if (isStringLiteral(first)) {
return { specifier: first.value, parseState: "parsed" };
}
return { specifier: "<dynamic-import>", parseState: "partial" };
}
return undefined;
};
const appendRelationship = (
relationships: ReferenceSourceImportRelationship[],
input: {
readonly fromPath: string;
readonly to: string | undefined;
readonly kind: ReferenceSourceImportKind;
readonly parseState: ReferenceSourceImportParseState;
readonly resolution?: ReferenceSourceImportResolution;
},
): void => {
if (input.to === undefined) return;
relationships.push({
from_path: input.fromPath,
to: input.to,
kind: input.kind,
resolution: input.resolution ?? resolveSpecifier(input.to),
parse_state: input.parseState,
});
};
const extractImportDeclarations = (
body: readonly Node[],
from_path: string,
relationships: ReferenceSourceImportRelationship[],
): void => {
for (const statement of body) {
if (isImportDeclaration(statement)) {
appendRelationship(relationships, {
fromPath: from_path,
to: safeModuleName(statement.source),
kind: "imports",
parseState: "parsed",
});
continue;
}
if (isExportNamedDeclaration(statement)) {
if (statement.source !== null && statement.source !== undefined) {
appendRelationship(relationships, {
fromPath: from_path,
to: safeModuleName(statement.source),
kind: "imports",
parseState: "parsed",
});
}
continue;
}
if (isExportAllDeclaration(statement)) {
appendRelationship(relationships, {
fromPath: from_path,
to: safeModuleName(statement.source),
kind: "imports",
parseState: "parsed",
});
continue;
}
if (isTSImportEqualsDeclaration(statement)) {
const reference = statement.moduleReference;
if (isTSExternalModuleReference(reference)) {
appendRelationship(relationships, {
fromPath: from_path,
to: safeModuleName(reference.expression),
kind: "requires",
parseState: "parsed",
});
}
continue;
}
if (isTSModuleDeclaration(statement)) {
if (statement.declare === true && isStringLiteral(statement.id)) {
appendRelationship(relationships, {
fromPath: from_path,
to: statement.id.value,
kind: "declares-module",
parseState: "parsed",
resolution: "unknown",
});
}
}
}
};
const isRequireCallee = (callee: Node | null | undefined): boolean => {
if (isIdentifier(callee) && callee.name === "require") return true;
if (
isMemberExpression(callee) &&
isIdentifier(callee.object) &&
callee.object.name === "require" &&
(isIdentifier(callee.property) || isStringLiteral(callee.property))
) {
const propertyName = isIdentifier(callee.property)
? callee.property.name
: isStringLiteral(callee.property)
? callee.property.value
: undefined;
return propertyName === "resolve" || propertyName === "main";
}
return false;
};
const isImportCallee = (callee: Node | null | undefined): boolean =>
isImport(callee);
const collectCallExpressions = (
node: unknown,
targets: CallExpression[],
): void => {
if (node === null || node === undefined) return;
if (typeof node !== "object") return;
if (Array.isArray(node)) {
for (const item of node) collectCallExpressions(item, targets);
return;
}
if (!isNode(node)) return;
if (isCallExpression(node)) {
targets.push(node);
}
for (const value of Object.values(node)) {
if (
value !== null &&
value !== undefined &&
typeof value === "object" &&
!isSourceLocation(value)
) {
collectCallExpressions(value, targets);
}
}
};
const isSourceLocation = (value: unknown): boolean =>
typeof value === "object" &&
value !== null &&
"start" in value &&
"end" in value &&
!("type" in value);
const extractRequireAndDynamicImports = (
body: readonly Node[],
from_path: string,
relationships: ReferenceSourceImportRelationship[],
): void => {
const calls: CallExpression[] = [];
for (const statement of body) collectCallExpressions(statement, calls);
for (const call of calls) {
const first = call.arguments[0];
if (isRequireCallee(call.callee) && isModuleExpression(first)) {
const result = moduleSpecifierFromExpression(first);
if (result !== undefined) {
appendRelationship(relationships, {
fromPath: from_path,
to: result.specifier,
kind: "requires",
parseState: result.parseState,
});
}
continue;
}
if (isImportCallee(call.callee) && isModuleExpression(first)) {
const result = moduleSpecifierFromExpression(first);
if (result !== undefined) {
appendRelationship(relationships, {
fromPath: from_path,
to: result.specifier,
kind: "imports",
parseState: result.parseState,
});
}
}
}
};
const parseWithBabel = (
path: string,
source: string,
language: string | null,
): {
ast: File | undefined;
parseState: ReferenceSourceImportParseState;
reason?: string;
} => {
const plugins: ParserPlugin[] = ["jsx"];
if (language === "TypeScript" || language === "TSX") {
plugins.push([
"typescript",
{
dts:
path.endsWith(".d.ts") ||
path.endsWith(".d.mts") ||
path.endsWith(".d.cts"),
},
]);
}
try {
const ast = parse(source, {
sourceType: "unambiguous",
allowImportExportEverywhere: false,
allowReturnOutsideFunction: true,
allowUndeclaredExports: true,
errorRecovery: true,
plugins,
});
return { ast, parseState: "parsed" };
} catch (error) {
return {
ast: undefined,
parseState: "unknown",
reason: error instanceof Error ? error.message : "Parse failed",
};
}
};
/** Parse ESM/CommonJS imports and dynamic imports from source bytes. */
export const parseReferenceSourceImports = (
path: string,
bytes: Uint8Array,
language: string | null,
): ReferenceSourceImportParseResult => {
if (language !== null && !codeSourceLanguages.has(language))
return { relationships: [], parse_failures: [] };
const source = new TextDecoder("utf-8", { fatal: false }).decode(bytes);
const { ast, parseState, reason } = parseWithBabel(path, source, language);
if (ast === undefined || parseState === "unknown") {
return {
relationships: [],
parse_failures: [
{
path,
parser: "babel",
reason: truncateReason(reason ?? "Unknown parse failure"),
},
],
};
}
const relationships: ReferenceSourceImportRelationship[] = [];
extractImportDeclarations(ast.program.body, path, relationships);
extractRequireAndDynamicImports(ast.program.body, path, relationships);
return {
relationships,
parse_failures: [],
};
};

9
src/domain/referenceSourcePolicy.ts Normal file
View File

@ -0,0 +1,9 @@
/** Operator-approved roots, exclusions, and hard caps for source imports. */
export interface ReferenceSourcePolicy {
readonly roots: readonly string[];
readonly secretPatterns: readonly string[];
readonly maxBytes: number;
readonly maxEntries: number;
readonly maxDepth: number;
readonly maxPathBytes: number;
}

304
src/domain/residualUnknown.ts Normal file
View File

@ -0,0 +1,304 @@
import { createHash } from "node:crypto";
import canonicalize from "canonicalize";
import { z } from "zod";
const evidenceIdSchema = z.string().regex(/^ev_[a-f0-9]{64}$/u);
const unknownIdSchema = z.string().regex(/^unk_[a-f0-9]{64}$/u);
const boundedText = z.string().trim().min(1).max(1_000);
const environmentRequirementSchema = z.object({
id: z.string().min(1).max(200).nullable(),
platform: z.string().min(1).max(100).nullable(),
architecture: z.string().min(1).max(100).nullable(),
isolation: z
.enum(["none", "process", "container", "virtual-machine"])
.nullable(),
});
const probeSchema = z.object({
operation: z.string().min(1).max(200),
rationale: boundedText,
});
const relationshipSchema = z.object({
type: z.enum(["depends-on", "related-to", "contradicts"]),
unknown_id: unknownIdSchema,
});
const resolutionSchema = z.discriminatedUnion("disposition", [
z.object({
disposition: z.literal("verified"),
rationale: boundedText,
evidence_ids: z.array(evidenceIdSchema).min(1).max(100),
}),
z.object({
disposition: z.enum(["withdrawn", "out-of-scope"]),
rationale: boundedText,
evidence_ids: z.array(evidenceIdSchema).max(100),
}),
]);
/** Strict durable record for one unresolved or evidence-qualified question. */
const residualUnknownObjectSchema = z.object({
registry_version: z.literal(1),
unknown_id: unknownIdSchema,
revision: z.number().int().min(1),
previous_revision_digest: z
.string()
.regex(/^[a-f0-9]{64}$/u)
.nullable(),
revision_digest: z.string().regex(/^[a-f0-9]{64}$/u),
scope_digest: z
.string()
.regex(/^[a-f0-9]{64}$/u)
.nullable(),
question: boundedText,
status: z.enum([
"open",
"investigating",
"blocked",
"contradicted",
"resolved",
]),
severity: z.enum(["low", "medium", "high", "critical"]),
domain: z.string().trim().min(1).max(100),
supporting_evidence_ids: z.array(evidenceIdSchema).max(100),
contradicting_evidence_ids: z.array(evidenceIdSchema).max(100),
required_authority: z
.enum([
"shipped-artifact",
"controlled-replay",
"historical-reference",
"external-service",
"analyst-inference",
])
.nullable(),
required_confidence: z.enum(["observed", "derived", "inferred"]),
required_environment: environmentRequirementSchema.nullable(),
recommended_probes: z.array(probeSchema).max(20),
relationships: z.array(relationshipSchema).max(100),
resolution: resolutionSchema.nullable(),
mutation_evidence_ids: z.array(evidenceIdSchema).min(1).max(100),
});
export const residualUnknownSchema = residualUnknownObjectSchema.superRefine(
(value, context) => {
if (computeRevisionDigest(value) !== value.revision_digest)
context.addIssue({
code: "custom",
message: "Residual unknown revision digest does not match",
path: ["revision_digest"],
});
if ((value.revision === 1) !== (value.previous_revision_digest === null))
context.addIssue({
code: "custom",
message: "Only the first revision has no predecessor",
path: ["previous_revision_digest"],
});
if ((value.status === "resolved") !== (value.resolution !== null))
context.addIssue({
code: "custom",
message: "Only resolved unknowns carry a resolution",
path: ["resolution"],
});
if (
value.status === "contradicted" &&
value.contradicting_evidence_ids.length === 0
)
context.addIssue({
code: "custom",
message: "Contradicted status requires contradicting evidence",
path: ["contradicting_evidence_ids"],
});
if (
value.relationships.some(
(relationship) => relationship.unknown_id === value.unknown_id,
)
)
context.addIssue({
code: "custom",
message: "An unknown cannot relate to itself",
path: ["relationships"],
});
for (const ids of [
value.supporting_evidence_ids,
value.contradicting_evidence_ids,
value.mutation_evidence_ids,
])
if (new Set(ids).size !== ids.length)
context.addIssue({
code: "custom",
message: "Evidence IDs must be unique",
});
if (
value.supporting_evidence_ids.some((id) =>
value.contradicting_evidence_ids.includes(id),
)
)
context.addIssue({
code: "custom",
message: "Evidence cannot simultaneously support and contradict",
});
const relationships = value.relationships.map(
(relationship) => `${relationship.type}:${relationship.unknown_id}`,
);
if (new Set(relationships).size !== relationships.length)
context.addIssue({
code: "custom",
message: "Relationships must be unique",
path: ["relationships"],
});
},
);
export type ResidualUnknown = z.infer<typeof residualUnknownSchema>;
export type UnknownStatus = ResidualUnknown["status"];
/** Input for deterministic creation of one residual unknown. */
export const recordUnknownInputSchema = residualUnknownObjectSchema
.omit({
registry_version: true,
unknown_id: true,
revision: true,
previous_revision_digest: true,
revision_digest: true,
scope_digest: true,
status: true,
supporting_evidence_ids: true,
contradicting_evidence_ids: true,
resolution: true,
mutation_evidence_ids: true,
})
.extend({
supporting_evidence_ids: z.array(evidenceIdSchema).max(100).default([]),
contradicting_evidence_ids: z.array(evidenceIdSchema).max(100).default([]),
approved: z.literal(true),
});
export type RecordUnknownInput = z.infer<typeof recordUnknownInputSchema>;
/** Explicit optimistic-concurrency update command for one unknown. */
export const updateUnknownInputSchema = z.object({
unknown_id: unknownIdSchema,
expected_revision: z.number().int().min(1),
approved: z.literal(true),
status: residualUnknownObjectSchema.shape.status,
severity: residualUnknownObjectSchema.shape.severity,
supporting_evidence_ids: z.array(evidenceIdSchema).max(100),
contradicting_evidence_ids: z.array(evidenceIdSchema).max(100),
required_authority: residualUnknownObjectSchema.shape.required_authority,
required_confidence: residualUnknownObjectSchema.shape.required_confidence,
required_environment: environmentRequirementSchema.nullable(),
recommended_probes: z.array(probeSchema).max(20),
relationships: z.array(relationshipSchema).max(100),
resolution: resolutionSchema.nullable(),
});
export type UpdateUnknownInput = z.infer<typeof updateUnknownInputSchema>;
/** Create stable identity from immutable question and domain fields. */
export const createResidualUnknown = (
input: RecordUnknownInput,
mutationEvidenceId: string,
scopeDigest: string | null,
): ResidualUnknown => {
const question = input.question.replace(/\s+/gu, " ");
const identity = canonicalJson({
domain: input.domain,
question,
scope_digest: scopeDigest,
});
const revision = {
registry_version: 1,
unknown_id: `unk_${createHash("sha256").update(identity).digest("hex")}`,
revision: 1,
previous_revision_digest: null,
scope_digest: scopeDigest,
question,
status:
input.contradicting_evidence_ids.length > 0 ? "contradicted" : "open",
domain: input.domain,
...projectRevisionDetails(input),
resolution: null,
mutation_evidence_ids: [mutationEvidenceId],
} satisfies Omit<ResidualUnknown, "revision_digest">;
return residualUnknownSchema.parse({
...revision,
revision_digest: computeRevisionDigest(revision),
});
};
/** Apply a complete, revision-guarded replacement while preserving identity. */
export const updateResidualUnknown = (
current: ResidualUnknown,
input: UpdateUnknownInput,
mutationEvidenceId: string,
): ResidualUnknown => {
const revision = {
registry_version: 1,
unknown_id: current.unknown_id,
revision: current.revision + 1,
previous_revision_digest: current.revision_digest,
scope_digest: current.scope_digest,
question: current.question,
status: input.status,
domain: current.domain,
...projectRevisionDetails(input),
resolution: input.resolution,
mutation_evidence_ids: sortedUnique([
...current.mutation_evidence_ids,
mutationEvidenceId,
]),
} satisfies Omit<ResidualUnknown, "revision_digest">;
return residualUnknownSchema.parse({
...revision,
revision_digest: computeRevisionDigest(revision),
});
};
type RevisionDetailsInput = Pick<
ResidualUnknown,
| "severity"
| "supporting_evidence_ids"
| "contradicting_evidence_ids"
| "required_authority"
| "required_confidence"
| "required_environment"
| "recommended_probes"
| "relationships"
>;
const projectRevisionDetails = (input: RevisionDetailsInput) => ({
severity: input.severity,
supporting_evidence_ids: sortedUnique(input.supporting_evidence_ids),
contradicting_evidence_ids: sortedUnique(input.contradicting_evidence_ids),
required_authority: input.required_authority,
required_confidence: input.required_confidence,
required_environment: input.required_environment,
recommended_probes: [...input.recommended_probes],
relationships: sortedRelationships(input.relationships),
});
const computeRevisionDigest = (
value: Omit<ResidualUnknown, "revision_digest"> | ResidualUnknown,
): string => {
const { revision_digest: _revisionDigest, ...semantic } =
"revision_digest" in value ? value : { ...value, revision_digest: "" };
void _revisionDigest;
return createHash("sha256").update(canonicalJson(semantic)).digest("hex");
};
const canonicalJson = (value: unknown): string => {
const serialized = canonicalize(value);
if (serialized === undefined) throw new TypeError("Unknown identity failed");
return serialized;
};
const sortedUnique = (values: readonly string[]): string[] =>
[...new Set(values)].sort((left, right) => left.localeCompare(right));
const sortedRelationships = (
values: readonly z.infer<typeof relationshipSchema>[],
): z.infer<typeof relationshipSchema>[] =>
[...values].sort((left, right) =>
`${left.type}:${left.unknown_id}`.localeCompare(
`${right.type}:${right.unknown_id}`,
),
);

513
src/domain/staticRuntimeCorrelation.ts Normal file
View File

@ -0,0 +1,513 @@
import { createHash } from "node:crypto";
import canonicalize from "canonicalize";
import { z } from "zod";
import { artifactComparisonResultSchema } from "./artifactComparison.js";
import { evidenceSchema, parseEvidence, type Evidence } from "./evidence.js";
import { functionComparisonResultSchema } from "./functionComparison.js";
import type { JsonValue } from "./jsonValue.js";
import {
comparisonStatusSchema,
processCaptureComparisonSchema,
} from "./processCapture.js";
const evidenceIdSchema = z.string().regex(/^ev_[a-f0-9]{64}$/u);
const functionDimensionSchema = z.enum([
"identity",
"pseudocode",
"assembly",
"comments",
"calls",
"references",
"strings_names",
"cfg",
]);
const runtimeDimensionSchema = z.enum([
"terminal",
"exit",
"filesystem",
"protocol",
"process",
]);
const expectedPatternSchema = z.enum([
"cochanged",
"static_only",
"runtime_only",
"both_unchanged",
]);
const observedPatternSchema = z.enum([
...expectedPatternSchema.options,
"unknown",
"truncated",
]);
const artifactSelectorSchema = z
.object({
kind: z.literal("artifact"),
logical_path: z.string().min(1).max(4_096),
})
.strict();
const functionSelectorSchema = z
.object({ kind: z.literal("function"), dimension: functionDimensionSchema })
.strict();
const staticSelectorSchema = z.discriminatedUnion("kind", [
artifactSelectorSchema,
functionSelectorSchema,
]);
const mappingSchema = z
.object({
static: z
.object({
comparison_evidence_id: evidenceIdSchema,
selector: staticSelectorSchema,
})
.strict(),
runtime: z
.object({
comparison_evidence_id: evidenceIdSchema,
dimension: runtimeDimensionSchema,
})
.strict(),
side_alignment: z.enum(["left_to_left", "left_to_right"]),
hypothesis: z
.object({
statement: z.string().trim().min(1).max(500),
expected_pattern: expectedPatternSchema,
})
.strict(),
})
.strict();
/** Strict bounded input for explicit static/runtime hypothesis correlation. */
export const staticRuntimeCorrelationInputSchema = z
.object({
static_comparisons: z.array(evidenceSchema).min(1).max(100),
runtime_comparisons: z.array(evidenceSchema).min(1).max(100),
mappings: z.array(mappingSchema).min(1).max(500),
offset: z.number().int().min(0).default(0),
limit: z.number().int().min(1).max(100).default(100),
unknown_registry_approved: z.literal(true).optional(),
})
.strict();
const correlationItemSchema = z.object({
correlation_id: z.string().regex(/^cor_[a-f0-9]{64}$/u),
static: z.object({
comparison_evidence_id: evidenceIdSchema,
selector: staticSelectorSchema,
status: comparisonStatusSchema,
conclusion_kind: z.enum([
"derived_relationship",
"contradiction",
"unresolved_branch",
]),
}),
runtime: z.object({
comparison_evidence_id: evidenceIdSchema,
dimension: runtimeDimensionSchema,
status: comparisonStatusSchema,
}),
side_alignment: z.enum(["left_to_left", "left_to_right"]),
hypothesis: z.object({
statement: z.string().min(1).max(500),
expected_pattern: expectedPatternSchema,
}),
observed_pattern: observedPatternSchema,
classification: z.enum(["hypothesis", "contradiction", "unresolved_branch"]),
evidence_links: z.array(evidenceIdSchema).min(2).max(20_100),
limitations: z.array(z.string()),
});
/** Deterministic, paginated static/runtime correlation payload. */
export const staticRuntimeCorrelationResultSchema = z.object({
status: z.enum(["correlated", "contradicted", "unknown", "truncated"]),
summary: z.object({
hypotheses: z.number().int().min(0),
contradictions: z.number().int().min(0),
unresolved: z.number().int().min(0),
}),
correlations: z.object({
items: z.array(correlationItemSchema).max(100),
offset: z.number().int().min(0),
limit: z.number().int().min(1).max(100),
total: z.number().int().min(0),
next_offset: z.number().int().min(0).nullable(),
}),
evidence_links: z.array(evidenceIdSchema).min(2).max(20_100),
limitations: z.array(z.string()),
});
export type StaticRuntimeCorrelationResult = z.infer<
typeof staticRuntimeCorrelationResultSchema
>;
type Mapping = z.infer<typeof mappingSchema>;
type CorrelationItem = z.infer<typeof correlationItemSchema>;
type ComparisonStatus = z.infer<typeof comparisonStatusSchema>;
const EXPECTED = {
compare_artifacts: [
"rea.artifact-comparison/v1",
"rea-artifact-comparison",
"REA artifact comparison",
],
compare_functions: [
"rea.function-comparison/v1",
"rea-function-comparison",
"REA function comparison",
],
compare_process_captures: [
"rea.process-comparison/v1",
"rea-process",
"REA deterministic process harness",
],
} as const;
/** Correlate only caller-declared static/runtime hypotheses. */
export const correlateStaticAndRuntime = (
input: unknown,
): StaticRuntimeCorrelationResult => {
const parsed = staticRuntimeCorrelationInputSchema.parse(input);
const staticEvidence = parsed.static_comparisons.map(parseStatic);
const runtimeEvidence = parsed.runtime_comparisons.map(parseRuntime);
assertUnique(staticEvidence, "static comparison Evidence");
assertUnique(runtimeEvidence, "runtime comparison Evidence");
const staticById = byId(staticEvidence);
const runtimeById = byId(runtimeEvidence);
const correlations = parsed.mappings
.map((mapping) => correlate(mapping, staticById, runtimeById))
.sort((left, right) =>
left.correlation_id.localeCompare(right.correlation_id, "en"),
);
if (
new Set(correlations.map(({ correlation_id: id }) => id)).size !==
correlations.length
)
throw new TypeError(
"Static/runtime correlation rejects duplicate mappings",
);
const page = correlations.slice(parsed.offset, parsed.offset + parsed.limit);
const links = uniqueSorted(
correlations.flatMap(({ evidence_links: evidenceLinks }) => evidenceLinks),
);
if (links.length > 20_100)
throw new TypeError("Static/runtime Evidence closure exceeds limit");
const unresolved = correlations.filter(
({ classification }) => classification === "unresolved_branch",
);
const contradictions = correlations.filter(
({ classification }) => classification === "contradiction",
);
return staticRuntimeCorrelationResultSchema.parse({
status: correlations.some(
({ observed_pattern: pattern }) => pattern === "truncated",
)
? "truncated"
: unresolved.length > 0
? "unknown"
: contradictions.length > 0
? "contradicted"
: "correlated",
summary: {
hypotheses:
correlations.length - contradictions.length - unresolved.length,
contradictions: contradictions.length,
unresolved: unresolved.length,
},
correlations: {
items: page,
offset: parsed.offset,
limit: parsed.limit,
total: correlations.length,
next_offset:
parsed.offset + page.length < correlations.length
? parsed.offset + page.length
: null,
},
evidence_links: links,
limitations: [
"Mappings are caller-declared hypotheses; consistency does not establish causality.",
"Only explicit mappings were evaluated; similar unmapped observations were not correlated.",
],
});
};
const correlate = (
mapping: Mapping,
staticById: ReadonlyMap<string, Evidence>,
runtimeById: ReadonlyMap<string, Evidence>,
): CorrelationItem => {
const staticEvidence = required(
staticById,
mapping.static.comparison_evidence_id,
"static comparison",
);
const runtimeEvidence = required(
runtimeById,
mapping.runtime.comparison_evidence_id,
"runtime comparison",
);
const selectedStatic = selectStatic(staticEvidence, mapping.static.selector);
const process = processCaptureComparisonSchema.parse(
runtimeEvidence.normalized_result,
);
const runtimeStatus = process[mapping.runtime.dimension];
const pattern = observedPattern(selectedStatic.status, runtimeStatus);
const classification =
pattern === "unknown" || pattern === "truncated"
? "unresolved_branch"
: pattern === mapping.hypothesis.expected_pattern
? "hypothesis"
: "contradiction";
const projection = {
static: mapping.static,
runtime: mapping.runtime,
side_alignment: mapping.side_alignment,
hypothesis: mapping.hypothesis,
} satisfies JsonValue;
return correlationItemSchema.parse({
correlation_id: `cor_${sha256(canonicalJson(projection))}`,
static: { ...mapping.static, ...selectedStatic },
runtime: { ...mapping.runtime, status: runtimeStatus },
side_alignment: mapping.side_alignment,
hypothesis: mapping.hypothesis,
observed_pattern: pattern,
classification,
evidence_links: uniqueSorted([
staticEvidence.evidence_id,
...staticEvidence.evidence_links,
runtimeEvidence.evidence_id,
...runtimeEvidence.evidence_links,
]),
limitations: uniqueSorted([
...staticEvidence.limitations,
...runtimeEvidence.limitations,
...process.limitations,
"This correlation tests an explicit hypothesis and does not establish causality.",
"Side alignment is caller-declared and was not independently validated against source subjects or environments.",
]),
});
};
const selectStatic = (
evidence: Evidence,
selector: z.infer<typeof staticSelectorSchema>,
): Pick<CorrelationItem["static"], "status" | "conclusion_kind"> => {
if (selector.kind === "function") {
if (evidence.operation !== "compare_functions")
throw new TypeError(
"Function selector requires function comparison Evidence",
);
const result = functionComparisonResultSchema.parse(
evidence.normalized_result,
);
const matches = result.dimensions.filter(
({ dimension }) => dimension === selector.dimension,
);
if (matches.length !== 1)
throw new TypeError("Function selector must resolve exactly once");
return {
status: matches[0]!.status,
conclusion_kind: matches[0]!.conclusion_kind,
};
}
if (evidence.operation !== "compare_artifacts")
throw new TypeError(
"Artifact selector requires artifact comparison Evidence",
);
const result = artifactComparisonResultSchema.parse(
evidence.normalized_result,
);
if (
result.changes.offset !== 0 ||
result.changes.next_offset !== null ||
result.changes.items.length !== result.changes.total
)
throw new TypeError(
"Static/runtime correlation requires complete artifact comparison pagination",
);
const matches = result.changes.items.filter(
({ logical_path: path }) => path === selector.logical_path,
);
if (matches.length !== 1)
throw new TypeError(
"Artifact selector must resolve exactly one reported delta",
);
const status = matches[0]!.classification;
return {
status,
conclusion_kind:
status === "unknown" ? "unresolved_branch" : "derived_relationship",
};
};
const parseStatic = (input: unknown): Evidence => {
const evidence = parseEvidence(input);
if (
evidence.operation !== "compare_artifacts" &&
evidence.operation !== "compare_functions"
)
throw new TypeError(
"Static/runtime correlation requires artifact or function comparison Evidence",
);
assertComparisonIdentity(evidence);
if (evidence.operation === "compare_artifacts") {
const result = artifactComparisonResultSchema.parse(
evidence.normalized_result,
);
assertCompleteArtifactResult(result);
assertNestedLinks(
evidence,
result.changes.items.flatMap(({ evidence_links: links }) => links),
);
} else {
const result = functionComparisonResultSchema.parse(
evidence.normalized_result,
);
assertNestedLinks(
evidence,
result.dimensions.flatMap(({ evidence_links: links }) => links),
);
}
return evidence;
};
const assertNestedLinks = (
evidence: Evidence,
nestedLinks: readonly string[],
): void => {
const closure = new Set(evidence.evidence_links);
if (nestedLinks.some((evidenceId) => !closure.has(evidenceId)))
throw new TypeError(
"Comparison finding cites Evidence outside its top-level closure",
);
};
const parseRuntime = (input: unknown): Evidence => {
const evidence = parseEvidence(input);
if (evidence.operation !== "compare_process_captures")
throw new TypeError(
"Static/runtime correlation requires process comparison Evidence",
);
assertComparisonIdentity(evidence);
processCaptureComparisonSchema.parse(evidence.normalized_result);
return evidence;
};
const assertComparisonIdentity = (evidence: Evidence): void => {
const expected = EXPECTED[evidence.operation as keyof typeof EXPECTED];
if (
expected === undefined ||
evidence.predicate_type !== expected[0] ||
evidence.provider.id !== expected[1] ||
evidence.provider.name !== expected[2] ||
evidence.provider.version !== "1" ||
evidence.confidence !== "derived" ||
evidence.authority !== "analyst-inference" ||
evidence.subject !== null ||
evidence.evidence_links.length < 2 ||
new Set(evidence.evidence_links).size !== evidence.evidence_links.length
)
throw new TypeError(
"Comparison Evidence operation, predicate, provider, or citations disagree",
);
const parameterLinks = comparisonParameterLinks(evidence);
if (!sameSet(parameterLinks, evidence.evidence_links))
throw new TypeError(
"Comparison Evidence closure disagrees with its source parameters",
);
};
const comparisonParameterLinks = (evidence: Evidence): string[] => {
if (evidence.operation === "compare_process_captures") {
const parameters = z
.object({
left_evidence_id: evidenceIdSchema,
right_evidence_id: evidenceIdSchema,
})
.passthrough()
.parse(evidence.parameters);
return [parameters.left_evidence_id, parameters.right_evidence_id];
}
const parameters = z
.object({
left_evidence_ids: z.array(evidenceIdSchema).min(1).max(100),
right_evidence_ids: z.array(evidenceIdSchema).min(1).max(100),
})
.passthrough()
.parse(evidence.parameters);
return [...parameters.left_evidence_ids, ...parameters.right_evidence_ids];
};
const sameSet = (left: readonly string[], right: readonly string[]): boolean =>
left.length === right.length &&
new Set(left).size === left.length &&
left.every((item) => right.includes(item));
const assertCompleteArtifactResult = (
result: z.infer<typeof artifactComparisonResultSchema>,
): void => {
if (
result.changes.offset !== 0 ||
result.changes.next_offset !== null ||
result.changes.items.length !== result.changes.total
)
throw new TypeError(
"Static/runtime correlation requires complete artifact comparison pagination",
);
};
const observedPattern = (
staticStatus: ComparisonStatus,
runtimeStatus: ComparisonStatus,
): z.infer<typeof observedPatternSchema> => {
if (staticStatus === "truncated" || runtimeStatus === "truncated")
return "truncated";
if (staticStatus === "unknown" || runtimeStatus === "unknown")
return "unknown";
const staticChanged = staticStatus !== "unchanged";
const runtimeChanged = runtimeStatus !== "unchanged";
if (staticChanged && runtimeChanged) return "cochanged";
if (staticChanged) return "static_only";
if (runtimeChanged) return "runtime_only";
return "both_unchanged";
};
const required = (
values: ReadonlyMap<string, Evidence>,
id: string,
label: string,
): Evidence => {
const value = values.get(id);
if (value === undefined)
throw new TypeError(`Mapping references absent ${label} Evidence`);
return value;
};
const byId = (evidence: readonly Evidence[]): Map<string, Evidence> =>
new Map(evidence.map((item) => [item.evidence_id, item]));
const assertUnique = (evidence: readonly Evidence[], label: string): void => {
if (
new Set(evidence.map(({ evidence_id: id }) => id)).size !== evidence.length
)
throw new TypeError(
`Static/runtime correlation rejects duplicate ${label}`,
);
};
const uniqueSorted = (values: readonly string[]): string[] =>
[...new Set(values)].sort((left, right) => left.localeCompare(right, "en"));
const sha256 = (value: string): string =>
createHash("sha256").update(value).digest("hex");
const canonicalJson = (value: JsonValue): string => {
const serialized = canonicalize(value);
if (serialized === undefined)
throw new TypeError(
"RFC 8785 canonicalization rejected correlation mapping",
);
return serialized;
};

54
src/hopper/BridgeLauncher.ts
View File

@ -6,6 +6,11 @@ import { promisify } from "node:util";
import { HopperCancelledError, HopperStartError } from "../domain/errors.js";
import { err, ok, type Result } from "../domain/result.js";
import {
cleanupOwnedProcessGroup,
type ProcessCleanupResult,
} from "../application/ProcessOwnership.js";
import writeFileAtomic from "write-file-atomic";
const execFileAsync = promisify(execFile);
const HOPPER_BACKGROUND_STARTUP_MS = 5_000;
@ -15,12 +20,14 @@ export interface BridgeSession {
readonly directory: string;
readonly socketPath: string;
readonly token: string;
readonly runId: string;
}
/** Process handle returned by a bridge launcher. */
export interface BridgeLaunch {
readonly process: ChildProcess;
readonly ownsProcessLifetime: boolean;
readonly cleanup?: () => Promise<ProcessCleanupResult>;
}
/** Application-owned capability that starts the in-Hopper bridge. */
@ -49,8 +56,9 @@ export interface HopperApplicationLauncherOptions {
* supported launcher; callers must treat possible foreground UI as an upstream
* Hopper constraint, not as evidence that analysis failed.
*
* The returned process is the short-lived launcher helper, not the Hopper app,
* so REA does not claim ownership of the application's lifetime.
* REA owns only the short-lived launcher helper's run-token-authenticated
* process group. It does not claim ownership of the Hopper GUI process that
* macOS LaunchServices may create or reuse.
*/
export class HopperApplicationLauncher implements BridgeLauncher {
constructor(readonly options: HopperApplicationLauncherOptions) {}
@ -63,6 +71,7 @@ export class HopperApplicationLauncher implements BridgeLauncher {
const source = [
`REA_SOCKET = ${JSON.stringify(session.socketPath)}`,
`REA_TOKEN = ${JSON.stringify(session.token)}`,
`REA_RUN_ID = ${JSON.stringify(session.runId)}`,
`exec(compile(open(${JSON.stringify(this.options.bridgeScriptPath)}, 'rb').read(), ${JSON.stringify(this.options.bridgeScriptPath)}, 'exec'))`,
"",
].join("\n");
@ -93,6 +102,8 @@ export class HopperApplicationLauncher implements BridgeLauncher {
if (!prepared) return err(new HopperCancelledError());
const child = spawn(this.options.launcherPath, args, {
stdio: ["ignore", "ignore", "pipe"],
detached: process.platform !== "win32",
env: { ...process.env, REA_PROCESS_RUN_ID: session.runId },
});
const started = await new Promise<Result<undefined, HopperStartError>>(
(resolve) => {
@ -109,7 +120,44 @@ export class HopperApplicationLauncher implements BridgeLauncher {
},
);
if (!started.ok) return started;
return ok({ process: child, ownsProcessLifetime: false });
const pid = child.pid;
if (pid === undefined)
return err(
new HopperStartError({ cause: new Error("missing launcher PID") }),
);
const ownership = {
schema_version: 1,
run_id: session.runId,
pid,
process_group_id: pid,
parent_pid: process.pid,
launcher: this.options.launcherPath,
created_at: new Date().toISOString(),
};
try {
await writeFileAtomic(
join(session.directory, "ownership.json"),
`${JSON.stringify(ownership)}\n`,
{ encoding: "utf8", mode: 0o600 },
);
} catch (cause: unknown) {
await cleanupOwnedProcessGroup({
runId: session.runId,
leaderPid: pid,
processGroupId: pid,
});
return err(new HopperStartError({ cause }));
}
return ok({
process: child,
ownsProcessLifetime: true,
cleanup: () =>
cleanupOwnedProcessGroup({
runId: session.runId,
leaderPid: pid,
processGroupId: pid,
}),
});
} catch (cause: unknown) {
return err(new HopperStartError({ cause }));
}

19
src/hopper/HopperClient.ts
View File

@ -1,4 +1,4 @@
import { randomBytes } from "node:crypto";
import { randomBytes, randomUUID } from "node:crypto";
import { access, mkdtemp, rm } from "node:fs/promises";
import { createConnection, type Socket } from "node:net";
import { tmpdir } from "node:os";
@ -101,11 +101,13 @@ export class HopperClient {
}
const socketPath = join(this.#directory, "bridge.sock");
this.#token = randomBytes(32).toString("hex");
const runId = randomUUID();
const launched = await this.#options.launcher.launch(
{
directory: this.#directory,
socketPath,
token: this.#token,
runId,
},
signal === undefined ? {} : { signal },
);
@ -132,7 +134,7 @@ export class HopperClient {
await this.close();
return health;
}
const parsed = parseServerInfo(health.value);
const parsed = parseServerInfo(health.value, runId);
if (!parsed.ok) await this.close();
return parsed;
}
@ -170,7 +172,14 @@ export class HopperClient {
socket?.destroy();
this.#socket = undefined;
if (this.#launch?.ownsProcessLifetime === true) {
this.#launch.process.kill("SIGTERM");
if (this.#launch.cleanup !== undefined) {
const cleanup = await this.#launch.cleanup();
if (!cleanup.cleaned)
this.#logger.warn(
{ reason: cleanup.reason },
"Owned launcher cleanup failed closed",
);
} else this.#launch.process.kill("SIGTERM");
}
this.#launch = undefined;
if (this.#directory !== undefined) {
@ -388,13 +397,15 @@ const connectOnce = async (
const parseServerInfo = (
value: JsonValue,
expectedRunId: string,
): Result<HopperServerInfo, HopperProtocolError> => {
if (
typeof value === "object" &&
value !== null &&
!Array.isArray(value) &&
value.name === "REA Hopper bridge" &&
typeof value.version === "string"
typeof value.version === "string" &&
value.run_id === expectedRunId
) {
return ok({ name: value.name, version: value.version });
}

110
src/hopper/HopperProvider.ts
View File

@ -1,5 +1,6 @@
import { fileURLToPath } from "node:url";
import { createAnalysisExecution } from "../application/AnalysisProvider.js";
import type {
AnalysisClient,
AnalysisProvider,
@ -9,42 +10,66 @@ import type {
import type { AppConfig } from "../config.js";
import type { BinaryTarget } from "../domain/binaryTarget.js";
import type { Logger } from "../logger.js";
import { AnalysisCapabilityUnavailableError } from "../domain/errors.js";
import { err } from "../domain/result.js";
import {
ENHANCED_TOOL_CONTRACTS,
OFFICIAL_TOOL_CONTRACTS,
} from "../contracts/toolContracts.js";
import { HopperApplicationLauncher } from "./BridgeLauncher.js";
import { HopperClient } from "./HopperClient.js";
const IDENTITY: ProviderIdentity = {
const IDENTITY: ProviderIdentity = Object.freeze({
id: "hopper",
name: "Hopper Disassembler",
version: null,
};
});
const MUTATING_OPERATIONS = new Set([
"set_address_name",
"set_addresses_names",
"set_bookmark",
"set_comment",
"set_inline_comment",
"unset_bookmark",
]);
const OPERATIONS = [
"direct-analysis",
"decompilation",
"disassembly",
"cross-references",
"containing-procedure-resolution",
"procedure-references",
"analysis-metadata-mutation",
] as const;
const PROVIDER_TOOL_CONTRACTS = [
...OFFICIAL_TOOL_CONTRACTS,
...ENHANCED_TOOL_CONTRACTS.filter(
(contract) => contract.name === "analyze_function",
),
];
const CAPABILITIES: readonly CapabilityDescriptor[] = OPERATIONS.map(
(operation) => ({
operation,
version: 1,
available: true,
pagination: "none",
exhaustive: false,
effects: {
mutatesArtifact: operation === "analysis-metadata-mutation",
launchesProcess: true,
mayShowUi: true,
mayAccessNetwork: false,
mayWriteFilesystem: operation === "analysis-metadata-mutation",
requiresPrivileges: false,
},
limitations: ["Results depend on Hopper's completed static analysis."],
}),
const CAPABILITIES: readonly CapabilityDescriptor[] = Object.freeze(
PROVIDER_TOOL_CONTRACTS.map((contract) =>
Object.freeze({
provider: IDENTITY,
operation: contract.name,
inputContractVersion: 1,
outputContractVersion: 1,
available: true,
reason: null,
pagination: "offset" in contract.inputSchema.shape ? "offset" : "none",
exhaustive: false,
effects: Object.freeze({
mutatesArtifact: MUTATING_OPERATIONS.has(contract.name),
launchesProcess: true,
mayShowUi: true,
mayAccessNetwork: false,
mayWriteFilesystem: MUTATING_OPERATIONS.has(contract.name),
changesPermissions: false,
requiresRoot: false,
}),
limits: Object.freeze({
maxResults: null,
maxPayloadBytes: null,
timeoutMs: null,
}),
limitations: Object.freeze([
"Results depend on Hopper's completed static analysis.",
]),
}),
),
);
/** Concrete analysis provider backed by REA's private Hopper bridge. */
@ -63,6 +88,20 @@ export class HopperProvider implements AnalysisProvider {
}
createClient(target: BinaryTarget): AnalysisClient {
if (target.kind !== "executable" && target.kind !== "database")
return {
execute: (operation) =>
Promise.resolve(
err(
new AnalysisCapabilityUnavailableError(
IDENTITY.id,
operation,
`Hopper cannot open ${target.kind} targets directly. Inventory or extract the artifact first.`,
),
),
),
close: () => Promise.resolve(),
};
const client = new HopperClient({
launcher: new HopperApplicationLauncher({
launcherPath: this.config.hopperLauncherPath,
@ -79,8 +118,19 @@ export class HopperProvider implements AnalysisProvider {
logger: this.logger.child({ layer: "bridge" }),
});
return {
execute: (operation, parameters, options) =>
client.callTool(operation, parameters, options),
execute: async (operation, parameters, options) => {
const result = await client.callTool(operation, parameters, options);
return result.ok
? {
ok: true,
value: createAnalysisExecution(result.value, IDENTITY, {
limitations: [
"Results depend on Hopper's completed static analysis.",
],
}),
}
: result;
},
close: () => client.close(),
};
}

1
src/identity.ts
View File

@ -10,5 +10,6 @@ export const PRODUCT_IDENTITY = {
mcpCommand: "npx -y rea-agents mcp",
mcpServerKey: "rea",
skillName: "rea-analysis",
skillVersion: "10",
configFileName: "rea.json",
} as const;

9
src/main.ts
View File

@ -46,12 +46,11 @@ export const run = async (): Promise<number> => {
try {
handle = serveStdio(
() =>
createServer(
session,
session,
createServer(session, session, {
logger,
config.value.processExecutionPolicy,
),
processPolicy: config.value.processExecutionPolicy,
evidenceFilePolicy: config.value.evidenceFilePolicy,
}),
{
onerror: () => {
serverLogger.error("MCP stdio transport error");

254
src/native/CommandRunner.ts Normal file
View File

@ -0,0 +1,254 @@
import { createHash, randomUUID } from "node:crypto";
import { createReadStream } from "node:fs";
import { realpath } from "node:fs/promises";
import { spawn } from "node:child_process";
import { err, ok, type Result } from "../domain/result.js";
export interface NativeCommandCapture {
readonly tool: string;
readonly executable: string;
readonly executableSha256: string;
readonly toolVersion: string | null;
readonly versionReason: string | null;
readonly arguments: readonly string[];
readonly stdout: string;
readonly stderr: string;
readonly stdoutBytes: number;
readonly stderrBytes: number;
readonly stdoutTruncated: boolean;
readonly stderrTruncated: boolean;
readonly exitCode: number | null;
readonly signal: string | null;
}
export class NativeCommandFailure extends Error {
constructor(
readonly tool: string,
readonly reason:
| "unavailable"
| "cancelled"
| "timeout"
| "output-limit"
| "nonzero-exit"
| "io",
readonly exitCode: number | null = null,
options?: ErrorOptions,
) {
super(`Native command ${tool} failed: ${reason}`, options);
}
}
export interface NativeCommandRunner {
run(
tool: string,
arguments_: readonly string[],
options: {
readonly signal?: AbortSignal;
readonly timeoutMs: number;
readonly maxOutputBytes: number;
readonly acceptNonZero?: boolean;
},
): Promise<Result<NativeCommandCapture, NativeCommandFailure>>;
}
/** Run allowlisted Xcode tools directly with bounded output and no shell. */
export class XcrunCommandRunner implements NativeCommandRunner {
readonly #resolved = new Map<
string,
Promise<Result<ResolvedTool, NativeCommandFailure>>
>();
run(
tool: string,
arguments_: readonly string[],
options: {
readonly signal?: AbortSignal;
readonly timeoutMs: number;
readonly maxOutputBytes: number;
readonly acceptNonZero?: boolean;
},
): Promise<Result<NativeCommandCapture, NativeCommandFailure>> {
if (!ALLOWED_TOOLS.has(tool))
return Promise.resolve(
err(new NativeCommandFailure(tool, "unavailable")),
);
return this.#resolve(tool).then(async (resolved) => {
if (!resolved.ok) return resolved;
const captured = await captureProcess(
resolved.value.path,
arguments_,
tool,
options,
);
return captured.ok
? ok({
...captured.value,
tool,
executable: resolved.value.path,
executableSha256: resolved.value.sha256,
toolVersion: null,
versionReason:
"Tool exposes no uniform stable version flag; executable digest identifies it.",
arguments: [...arguments_],
})
: captured;
});
}
#resolve(tool: string): Promise<Result<ResolvedTool, NativeCommandFailure>> {
const existing = this.#resolved.get(tool);
if (existing !== undefined) return existing;
const pending = resolveTool(tool);
this.#resolved.set(tool, pending);
return pending;
}
}
interface ResolvedTool {
readonly path: string;
readonly sha256: string;
}
const ALLOWED_TOOLS = new Set([
"codesign",
"dyld_info",
"dwarfdump",
"file",
"lipo",
"nm",
"otool",
"plutil",
"strings",
"swift-demangle",
"vtool",
]);
const resolveTool = async (
tool: string,
): Promise<Result<ResolvedTool, NativeCommandFailure>> => {
const found = await captureProcess(
"/usr/bin/xcrun",
["--find", tool],
"xcrun",
{ timeoutMs: 5_000, maxOutputBytes: 64 * 1024 },
);
if (!found.ok || found.value.exitCode !== 0)
return err(new NativeCommandFailure(tool, "unavailable"));
const candidate = found.value.stdout.trim();
if (!candidate.startsWith("/"))
return err(new NativeCommandFailure(tool, "unavailable"));
try {
const path = await realpath(candidate);
return ok({ path, sha256: await hashFile(path) });
} catch (cause: unknown) {
return err(new NativeCommandFailure(tool, "io", null, { cause }));
}
};
type ProcessCapture = Omit<
NativeCommandCapture,
| "tool"
| "executable"
| "executableSha256"
| "toolVersion"
| "versionReason"
| "arguments"
>;
const captureProcess = (
executable: string,
arguments_: readonly string[],
tool: string,
options: {
readonly signal?: AbortSignal;
readonly timeoutMs: number;
readonly maxOutputBytes: number;
readonly acceptNonZero?: boolean;
},
): Promise<Result<ProcessCapture, NativeCommandFailure>> =>
new Promise((resolve) => {
if (options.signal?.aborted === true) {
resolve(err(new NativeCommandFailure(tool, "cancelled")));
return;
}
const runToken = randomUUID();
const child = spawn(executable, [...arguments_], {
shell: false,
stdio: ["ignore", "pipe", "pipe"],
env: {
PATH: "/usr/bin:/bin:/usr/sbin:/sbin",
LC_ALL: "C",
LANG: "C",
REA_NATIVE_RUN_TOKEN: runToken,
},
});
const stdout: Buffer[] = [];
const stderr: Buffer[] = [];
let stdoutBytes = 0;
let stderrBytes = 0;
let settled = false;
let reason: NativeCommandFailure["reason"] | undefined;
const finish = (
result: Result<ProcessCapture, NativeCommandFailure>,
): void => {
if (settled) return;
settled = true;
clearTimeout(timeout);
options.signal?.removeEventListener("abort", onAbort);
resolve(result);
};
const stop = (failure: NativeCommandFailure["reason"]): void => {
reason ??= failure;
child.kill("SIGKILL");
};
const onAbort = (): void => stop("cancelled");
options.signal?.addEventListener("abort", onAbort, { once: true });
const timeout = setTimeout(() => stop("timeout"), options.timeoutMs);
child.stdout.on("data", (chunk: Buffer) => {
stdoutBytes += chunk.length;
if (stdoutBytes + stderrBytes > options.maxOutputBytes)
stop("output-limit");
else stdout.push(chunk);
});
child.stderr.on("data", (chunk: Buffer) => {
stderrBytes += chunk.length;
if (stdoutBytes + stderrBytes > options.maxOutputBytes)
stop("output-limit");
else stderr.push(chunk);
});
child.once("error", (cause: unknown) =>
finish(err(new NativeCommandFailure(tool, "io", null, { cause }))),
);
child.once("close", (code, signal) => {
if (reason !== undefined) {
finish(err(new NativeCommandFailure(tool, reason, code)));
return;
}
if (code !== 0 && options.acceptNonZero !== true) {
finish(err(new NativeCommandFailure(tool, "nonzero-exit", code)));
return;
}
finish(
ok({
stdout: Buffer.concat(stdout).toString("utf8"),
stderr: Buffer.concat(stderr).toString("utf8"),
stdoutBytes,
stderrBytes,
stdoutTruncated: false,
stderrTruncated: false,
exitCode: code,
signal,
}),
);
});
});
const hashFile = (path: string): Promise<string> =>
new Promise((resolve, reject) => {
const hash = createHash("sha256");
const stream = createReadStream(path);
stream.on("data", (chunk) => hash.update(chunk));
stream.once("error", reject);
stream.once("end", () => resolve(hash.digest("hex")));
});

490
src/native/NativeMacOSProvider.ts Normal file
View File

@ -0,0 +1,490 @@
import { dirname, isAbsolute, relative, resolve, sep } from "node:path";
import { realpath, stat } from "node:fs/promises";
import { parse as parseXmlPlist } from "plist";
import { z } from "zod";
import {
createAnalysisExecution,
type AnalysisClient,
type AnalysisOperation,
type AnalysisProvider,
type CapabilityDescriptor,
type ProviderIdentity,
} from "../application/AnalysisProvider.js";
import {
NATIVE_TOOL_CONTRACTS,
type NativeToolName,
} from "../contracts/nativeToolContracts.js";
import type { BinaryTarget } from "../domain/binaryTarget.js";
import type { EvidenceLocation } from "../domain/evidence.js";
import {
AnalysisCancelledError,
AnalysisCapabilityUnavailableError,
AnalysisOutputError,
AnalysisTimeoutError,
ProviderAdapterError,
type AnalysisError,
} from "../domain/errors.js";
import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js";
import {
demangleSwiftSchema,
inspectPlistSchema,
inspectSignatureSchema,
listArchitecturesSchema,
type NativeCommandInvocation,
} from "../domain/nativeInspection.js";
import { err, ok, type Result } from "../domain/result.js";
import {
NativeCommandFailure,
XcrunCommandRunner,
type NativeCommandCapture,
type NativeCommandRunner,
} from "./CommandRunner.js";
import { parseCodeSignature } from "./parsers/codesign.js";
import { parseDemangledSymbols } from "./parsers/demangle.js";
import { parseLipoArchitectures } from "./parsers/lipo.js";
import { parsePlistJson } from "./parsers/plist.js";
import {
architectureLocations,
inspectNativeMacho,
} from "./NativeMachoInspection.js";
const IDENTITY: ProviderIdentity = Object.freeze({
id: "native-macos",
name: "macOS native inspection utilities",
version: null,
});
const TIMEOUT_MS = 30_000;
const OUTPUT_LIMIT = 4 * 1024 * 1024;
/** Read-only semantic provider composed from Xcode command-line utilities. */
export class NativeMacOSProvider implements AnalysisProvider {
readonly #capabilities: readonly CapabilityDescriptor[];
constructor(
private readonly runner: NativeCommandRunner = new XcrunCommandRunner(),
platform: NodeJS.Platform = process.platform,
) {
const available = platform === "darwin";
this.#capabilities = Object.freeze(
NATIVE_TOOL_CONTRACTS.map((contract): CapabilityDescriptor => {
const availability = available
? ({ available: true, reason: null } as const)
: ({
available: false,
reason: "Native macOS utilities require macOS.",
} as const);
return Object.freeze({
provider: IDENTITY,
operation: contract.name,
inputContractVersion: 1,
outputContractVersion: 1,
...availability,
pagination: "none" as const,
exhaustive: contract.name !== "inspect_macho",
effects: Object.freeze({
mutatesArtifact: false,
launchesProcess: true,
mayShowUi: false,
mayAccessNetwork: false,
mayWriteFilesystem: false,
changesPermissions: false,
requiresRoot: false,
}),
limits: Object.freeze({
maxResults: contract.name === "demangle_swift" ? 500 : null,
maxPayloadBytes: OUTPUT_LIMIT,
timeoutMs: TIMEOUT_MS,
}),
limitations: Object.freeze([
"Availability and textual formats depend on the installed macOS/Xcode toolchain.",
]),
});
}),
);
}
identity(): ProviderIdentity {
return IDENTITY;
}
capabilities(): readonly CapabilityDescriptor[] {
return this.#capabilities;
}
createClient(target: BinaryTarget): AnalysisClient {
return new NativeMacOSClient(target, this.runner);
}
}
class NativeMacOSClient implements AnalysisClient {
constructor(
private readonly target: BinaryTarget,
private readonly runner: NativeCommandRunner,
) {}
async execute(
operation: AnalysisOperation,
parameters: Readonly<Record<string, JsonValue>>,
options?: { readonly signal?: AbortSignal },
) {
if (operation === "health")
return ok(createAnalysisExecution(null, IDENTITY));
if (!isNativeOperation(operation))
return err(
new AnalysisCapabilityUnavailableError(
IDENTITY.id,
operation,
"Operation is not implemented by native macOS tools.",
),
);
try {
const observation = await this.#dispatch(
operation,
parameters,
options?.signal,
);
return observation.ok
? ok(
createAnalysisExecution(observation.value.result, IDENTITY, {
rawResult: { provenance: observation.value.provenance },
limitations: observation.value.limitations,
locations: observation.value.locations,
}),
)
: observation;
} catch (cause: unknown) {
return err(
new AnalysisOutputError(operation, "Native output parsing failed", {
cause,
}),
);
}
}
close(): Promise<void> {
return Promise.resolve();
}
#dispatch(
operation: NativeToolName,
parameters: Readonly<Record<string, JsonValue>>,
signal?: AbortSignal,
): Promise<Result<NativeObservation, AnalysisError>> {
switch (operation) {
case "inspect_macho":
return this.#inspectMacho(signal);
case "inspect_signature":
return this.#inspectSignature(signal);
case "inspect_plist":
return this.#inspectPlist(parameters, signal);
case "list_architectures":
return this.#listArchitectures(signal);
case "demangle_swift":
return this.#demangle(parameters, signal);
}
}
async #listArchitectures(
signal?: AbortSignal,
): Promise<Result<NativeObservation, AnalysisError>> {
const capture = await this.#run(
"list_architectures",
"lipo",
["-detailed_info", this.target.path],
{ signal },
);
if (!capture.ok) return capture;
const provenance = [invocation(capture.value, this.target.path)];
const result = listArchitecturesSchema.parse({
architectures: {
items: parseLipoArchitectures(capture.value.stdout),
total: parseLipoArchitectures(capture.value.stdout).length,
exhaustive: true,
limitations: [],
},
provenance,
limitations: [],
});
return ok({
result: jsonValueSchema.parse(result),
provenance,
limitations: [],
locations: architectureLocations(result.architectures.items),
});
}
async #demangle(
parameters: Readonly<Record<string, JsonValue>>,
signal?: AbortSignal,
): Promise<Result<NativeObservation, AnalysisError>> {
const symbols = z.array(z.string()).safeParse(parameters.symbols);
if (!symbols.success)
return err(
new AnalysisOutputError(
"demangle_swift",
"symbols were not parsed strings",
),
);
const capture = await this.#run(
"demangle_swift",
"swift-demangle",
["--compact", ...symbols.data],
{ signal },
);
if (!capture.ok) return capture;
const provenance = [invocation(capture.value, this.target.path)];
const result = demangleSwiftSchema.parse({
symbols: parseDemangledSymbols(symbols.data, capture.value.stdout),
provenance,
limitations: [],
});
return ok({
result: jsonValueSchema.parse(result),
provenance,
limitations: [],
locations: [],
});
}
async #inspectSignature(
signal?: AbortSignal,
): Promise<Result<NativeObservation, AnalysisError>> {
const display = await this.#run(
"inspect_signature",
"codesign",
["-d", "--verbose=4", this.target.path],
{ signal, acceptNonZero: true },
);
if (!display.ok) return display;
const requirements = await this.#run(
"inspect_signature",
"codesign",
["-d", "-r-", this.target.path],
{ signal, acceptNonZero: true },
);
if (!requirements.ok) return requirements;
const entitlements = await this.#run(
"inspect_signature",
"codesign",
["-d", "--entitlements", ":-", this.target.path],
{ signal, acceptNonZero: true },
);
if (!entitlements.ok) return entitlements;
const unsigned = /not signed at all|code object is not signed/iu.test(
`${display.value.stdout}\n${display.value.stderr}`,
);
const parsed = parseCodeSignature(
`${display.value.stdout}\n${display.value.stderr}`,
unsigned,
);
const requirementText =
/designated\s*=>\s*(.+)$/mu.exec(
`${requirements.value.stdout}\n${requirements.value.stderr}`,
)?.[1] ?? null;
const entitlementValue = parseEntitlements(
`${entitlements.value.stdout}\n${entitlements.value.stderr}`,
);
const provenance = [
display.value,
requirements.value,
entitlements.value,
].map((capture) => invocation(capture, this.target.path));
const result = inspectSignatureSchema.parse({
...parsed,
designated_requirement: requirementText,
entitlements: entitlementValue,
provenance,
});
return ok({
result: jsonValueSchema.parse(result),
provenance,
limitations: result.limitations,
locations: [],
});
}
async #inspectPlist(
parameters: Readonly<Record<string, JsonValue>>,
signal?: AbortSignal,
): Promise<Result<NativeObservation, AnalysisError>> {
const requested = parameters.relative_path;
if (typeof requested !== "string")
return err(
new AnalysisOutputError(
"inspect_plist",
"relative_path was not a string",
),
);
const plist = await resolvePlistPath(this.target, requested);
if (!plist.ok) return plist;
const classified = await this.#run(
"inspect_plist",
"file",
["-b", plist.value],
{ signal },
);
if (!classified.ok) return classified;
const capture = await this.#run(
"inspect_plist",
"plutil",
["-convert", "json", "-o", "-", "--", plist.value],
{ signal },
);
if (!capture.ok) return capture;
const parsed = parsePlistJson(capture.value.stdout);
const provenance = [classified.value, capture.value].map((item) =>
invocation(item, plist.value, "$PLIST"),
);
const result = inspectPlistSchema.parse({
format: /binary property list/iu.test(classified.value.stdout)
? "binary"
: /XML|text/iu.test(classified.value.stdout)
? "xml"
: "unknown",
...parsed,
source_path: requested,
provenance,
limitations: [],
});
return ok({
result: jsonValueSchema.parse(result),
provenance,
limitations: [],
locations: [{ kind: "artifact-path", path: requested }],
});
}
async #inspectMacho(
signal?: AbortSignal,
): Promise<Result<NativeObservation, AnalysisError>> {
if (this.target.format !== "mach-o")
return err(
new AnalysisCapabilityUnavailableError(
IDENTITY.id,
"inspect_macho",
"Active artifact is not Mach-O.",
),
);
return inspectNativeMacho({
target: this.target,
...(signal === undefined ? {} : { signal }),
run: (tool, arguments_, commandSignal) =>
this.#run("inspect_macho", tool, arguments_, {
signal: commandSignal,
}),
invocation: (capture) => invocation(capture, this.target.path),
});
}
async #run(
operation: NativeToolName,
tool: string,
arguments_: readonly string[],
options: {
readonly signal?: AbortSignal | undefined;
readonly acceptNonZero?: boolean;
} = {},
): Promise<Result<NativeCommandCapture, AnalysisError>> {
const captured = await this.runner.run(tool, arguments_, {
...(options.signal === undefined ? {} : { signal: options.signal }),
timeoutMs: TIMEOUT_MS,
maxOutputBytes: OUTPUT_LIMIT,
acceptNonZero: options.acceptNonZero ?? false,
});
if (captured.ok) return captured;
return err(translateCommandFailure(operation, captured.error));
}
}
interface NativeObservation {
readonly result: JsonValue;
readonly provenance: readonly NativeCommandInvocation[];
readonly limitations: readonly string[];
readonly locations: readonly EvidenceLocation[];
}
const isNativeOperation = (
operation: AnalysisOperation,
): operation is NativeToolName =>
NATIVE_TOOL_CONTRACTS.some(({ name }) => name === operation);
const translateCommandFailure = (
operation: NativeToolName,
failure: NativeCommandFailure,
): AnalysisError => {
if (failure.reason === "unavailable")
return new AnalysisCapabilityUnavailableError(
IDENTITY.id,
operation,
`${failure.tool} is unavailable through xcrun.`,
);
if (failure.reason === "cancelled")
return new AnalysisCancelledError(operation);
if (failure.reason === "timeout")
return new AnalysisTimeoutError(operation, TIMEOUT_MS);
return new ProviderAdapterError(IDENTITY.id, operation, { cause: failure });
};
const invocation = (
capture: NativeCommandCapture,
artifactPath: string,
alias = "$ARTIFACT",
): NativeCommandInvocation => ({
tool: capture.tool,
command: [
capture.executable,
...capture.arguments.map((argument) =>
argument === artifactPath ? alias : argument,
),
],
tool_version: capture.toolVersion,
version_reason: capture.versionReason,
executable_sha256: capture.executableSha256,
exit: { code: capture.exitCode, signal: capture.signal },
stdout_bytes: capture.stdoutBytes,
stderr_bytes: capture.stderrBytes,
stdout_truncated: capture.stdoutTruncated,
stderr_truncated: capture.stderrTruncated,
});
const parseEntitlements = (output: string): JsonValue | null => {
const start = output.indexOf("<?xml");
const end = output.lastIndexOf("</plist>");
if (start < 0 || end < start) return null;
return jsonValueSchema.parse(
parseXmlPlist(output.slice(start, end + "</plist>".length)),
);
};
const resolvePlistPath = async (
target: BinaryTarget,
requested: string,
): Promise<Result<string, AnalysisError>> => {
try {
const source = target.sourcePath ?? target.path;
const sourceMetadata = await stat(source);
const root = sourceMetadata.isDirectory() ? source : dirname(target.path);
const candidate = resolve(root, requested);
if (!within(root, candidate))
return err(new ProviderAdapterError(IDENTITY.id, "inspect_plist"));
const canonicalRoot = await realpath(root);
const canonical = await realpath(candidate);
if (!within(canonicalRoot, canonical))
return err(new ProviderAdapterError(IDENTITY.id, "inspect_plist"));
return ok(canonical);
} catch (cause: unknown) {
return err(
new ProviderAdapterError(IDENTITY.id, "inspect_plist", { cause }),
);
}
};
const within = (root: string, candidate: string): boolean => {
const child = relative(root, candidate);
return (
child === "" ||
(!child.startsWith(`..${sep}`) && child !== ".." && !isAbsolute(child))
);
};

206
src/native/NativeMachoInspection.ts Normal file
View File

@ -0,0 +1,206 @@
import type { BinaryTarget } from "../domain/binaryTarget.js";
import type { EvidenceLocation } from "../domain/evidence.js";
import type { AnalysisError } from "../domain/errors.js";
import {
inspectMachoSchema,
type NativeCommandInvocation,
} from "../domain/nativeInspection.js";
import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js";
import { err, ok, type Result } from "../domain/result.js";
import type { NativeCommandCapture } from "./CommandRunner.js";
import { parseDyldSymbols } from "./parsers/dyldInfo.js";
import { parseLipoArchitectures } from "./parsers/lipo.js";
import { parseOtoolLoadCommands } from "./parsers/otool.js";
interface NativeMachoObservation {
readonly result: JsonValue;
readonly provenance: readonly NativeCommandInvocation[];
readonly limitations: readonly string[];
readonly locations: readonly EvidenceLocation[];
}
interface NativeMachoInspectionContext {
readonly target: BinaryTarget;
readonly signal?: AbortSignal;
readonly run: (
tool: string,
arguments_: readonly string[],
signal?: AbortSignal,
) => Promise<Result<NativeCommandCapture, AnalysisError>>;
readonly invocation: (
capture: NativeCommandCapture,
) => NativeCommandInvocation;
}
const COMMANDS = [
["file", ["-b"]],
["lipo", ["-detailed_info"]],
["otool", ["-l"]],
["nm", ["-gjU"]],
["dyld_info", ["-imports"]],
["dyld_info", ["-exports"]],
["dwarfdump", ["--uuid"]],
["vtool", ["-show-build"]],
] as const;
/** Inspect one Mach-O with bounded native commands and normalized output. */
export const inspectNativeMacho = async (
context: NativeMachoInspectionContext,
): Promise<Result<NativeMachoObservation, AnalysisError>> => {
const captures: NativeCommandCapture[] = [];
for (const [tool, prefix] of COMMANDS) {
const captured = await context.run(
tool,
[...prefix, context.target.path],
context.signal,
);
if (!captured.ok) return err(captured.error);
captures.push(captured.value);
}
const result = normalizeMacho(captures, context.invocation);
return ok({
result: jsonValueSchema.parse(result),
provenance: result.provenance,
limitations: result.limitations,
locations: fileOffsetLocations(result),
});
};
const normalizeMacho = (
captures: readonly NativeCommandCapture[],
toInvocation: (capture: NativeCommandCapture) => NativeCommandInvocation,
) => {
const byTool = captureLookup(captures);
const architectures = parseLipoArchitectures(byTool("lipo").stdout);
const load = parseOtoolLoadCommands(byTool("otool").stdout);
const imports = parseDyldSymbols(byTool("dyld_info", 0).stdout, "imports");
const dyldExports = parseDyldSymbols(
byTool("dyld_info", 1).stdout,
"exports",
);
const exports = uniqueSymbols([
...dyldExports,
...parseNmExports(byTool("nm").stdout),
]);
const uuid =
/UUID:\s*([A-Fa-f0-9-]+)/u.exec(byTool("dwarfdump").stdout)?.[1] ??
load.uuid;
const provenance = captures.map(toInvocation);
const limitations = [
"Imports and exports combine dyld_info and nm; stripped or toolchain-hidden symbols may be absent.",
"vtool output is retained as provenance but only otool build metadata is normalized.",
];
return inspectMachoSchema.parse({
format: "mach-o",
endian: parseEndian(byTool("file").stdout),
word_size: parseWordSize(byTool("file").stdout),
file_type: load.fileType,
flags: load.flags,
uuid: uuid ?? null,
entrypoints: covered(load.entrypoints, true),
architectures: covered(architectures, true),
build_metadata: covered(load.builds, true),
load_commands: covered(load.commands, true),
dependencies: covered(load.dependencies, true),
imports: covered(imports, false, [
"dyld_info textual imports may omit chained or toolchain-unsupported metadata.",
]),
exports: covered(exports, false, [
"Merged nm/dyld_info results may be incomplete for stripped binaries.",
]),
segments: covered(
load.segments.map((segment) => ({
...segment,
sections: covered(segment.sections, true),
})),
true,
),
provenance,
limitations,
});
};
const captureLookup =
(captures: readonly NativeCommandCapture[]) =>
(tool: string, occurrence = 0): NativeCommandCapture => {
const capture = captures.filter((item) => item.tool === tool)[occurrence];
if (capture === undefined) throw new TypeError(`Missing ${tool} capture`);
return capture;
};
const parseNmExports = (output: string) =>
output
.split(/\r?\n/u)
.filter((name) => name.length > 0)
.map((name) => ({
name,
address: null,
weak: null,
reexport: null,
source: "nm",
}));
const parseEndian = (output: string): "little" | "big" | null =>
/little-endian/iu.test(output)
? "little"
: /big-endian/iu.test(output)
? "big"
: null;
const parseWordSize = (output: string): 32 | 64 | null =>
/64-bit/iu.test(output) ? 64 : /32-bit/iu.test(output) ? 32 : null;
const covered = <Value>(
items: readonly Value[],
exhaustive: boolean,
limitations: readonly string[] = [],
) => ({
items: [...items],
total: exhaustive ? items.length : null,
exhaustive,
limitations: [...limitations],
});
const uniqueSymbols = <Value extends { readonly name: string }>(
items: readonly Value[],
): Value[] => {
const unique = new Map<string, Value>();
for (const item of items) unique.set(item.name, item);
return [...unique.values()].sort((left, right) =>
left.name.localeCompare(right.name),
);
};
const fileOffsetLocations = (
macho: ReturnType<typeof inspectMachoSchema.parse>,
): EvidenceLocation[] => {
const locations = architectureLocations(macho.architectures.items);
for (const segment of macho.segments.items) {
if (segment.file_offset === null || segment.file_size === null) continue;
locations.push({
kind: "file-offset-range",
start: segment.file_offset,
end: segment.file_offset + segment.file_size,
});
}
return locations;
};
/** Project architecture slices into evidence file-offset locations. */
export const architectureLocations = (
architectures: readonly {
readonly file_offset: number | null;
readonly size: number | null;
}[],
): EvidenceLocation[] => {
const locations: EvidenceLocation[] = [];
for (const { file_offset: offset, size } of architectures) {
if (offset === null) continue;
locations.push(
size === null
? { kind: "file-offset", offset }
: { kind: "file-offset-range", start: offset, end: offset + size },
);
}
return locations;
};

62
src/native/parsers/codesign.ts Normal file
View File

@ -0,0 +1,62 @@
import {
inspectSignatureSchema,
type InspectSignature,
} from "../../domain/nativeInspection.js";
/** Parse bounded `codesign -d` diagnostics, which Apple emits on stderr. */
export const parseCodeSignature = (
output: string,
unsigned: boolean,
): Omit<InspectSignature, "provenance"> => {
const values = new Map<string, string>();
const authorities: string[] = [];
const cdhashes: string[] = [];
for (const line of output.split(/\r?\n/u)) {
if (line.startsWith("CodeDirectory ")) {
values.set("CodeDirectory", line.slice("CodeDirectory ".length));
continue;
}
const separator = line.indexOf("=");
if (separator < 1) continue;
const key = line.slice(0, separator).trim();
const value = line.slice(separator + 1).trim();
if (key === "Authority") authorities.push(value);
else if (key === "CDHash") cdhashes.push(value);
else values.set(key, value);
}
const parsed = inspectSignatureSchema.omit({ provenance: true }).parse({
signed: !unsigned,
identifier: values.get("Identifier") ?? null,
team_identifier: nullableCodeSignValue(values.get("TeamIdentifier")),
format: values.get("Format") ?? null,
cdhashes,
hash_algorithms: splitAlgorithms(values.get("Hash choices")),
authorities,
designated_requirement: values.get("designated") ?? null,
entitlements: null,
timestamp: values.get("Timestamp") ?? null,
hardened_runtime: parseRuntime(values.get("CodeDirectory")),
limitations: unsigned
? ["Artifact is not signed."]
: [
"Entitlements and designated requirements require separate bounded commands.",
],
});
return parsed;
};
const nullableCodeSignValue = (value: string | undefined): string | null =>
value === undefined || value === "not set" ? null : value;
const splitAlgorithms = (value: string | undefined): string[] =>
value === undefined
? []
: value
.split(/[,+\s]+/u)
.map((part) => part.trim())
.filter((part) => part.length > 0);
const parseRuntime = (value: string | undefined): boolean | null => {
if (value === undefined) return null;
return /\bruntime\b/u.test(value);
};

32
src/native/parsers/demangle.ts Normal file
View File

@ -0,0 +1,32 @@
import { z } from "zod";
const resultSchema = z.object({
input: z.string(),
output: z.string(),
status: z.enum(["demangled", "unchanged", "invalid"]),
});
/** Preserve input ordering while parsing `swift-demangle --compact` lines. */
export const parseDemangledSymbols = (
inputs: readonly string[],
output: string,
) => {
const lines = output.trimEnd().split(/\r?\n/u);
if (lines.length !== inputs.length)
throw new TypeError("swift-demangle output count does not match input");
return inputs.map((input, index) => {
const value = lines[index];
if (value === undefined)
throw new TypeError("swift-demangle omitted an output line");
return resultSchema.parse({
input,
output: value,
status:
value === input
? input.startsWith("$s") || input.startsWith("_$s")
? "invalid"
: "unchanged"
: "demangled",
});
});
};

29
src/native/parsers/dyldInfo.ts Normal file
View File

@ -0,0 +1,29 @@
/** Parse bounded imports/exports from dyld_info's line-oriented tables. */
export const parseDyldSymbols = (output: string, mode: "imports" | "exports") =>
output.split(/\r?\n/u).flatMap((rawLine) => {
const line = rawLine.trim();
if (
line.length === 0 ||
/^(?:imports|exports|binding|address|segment|ordinal)\b/iu.test(line)
)
return [];
const tokens = line.split(/\s+/u);
const name = tokens.at(-1);
if (name === undefined || !/^(?:_|\$s|objc_|swift_)/u.test(name)) return [];
const addressToken = tokens.find((token) =>
/^0x[a-fA-F0-9]+$/u.test(token),
);
return [
{
name,
address: addressToken ?? null,
weak: /\bweak\b/iu.test(line) ? true : null,
reexport: /\bre-?export\b/iu.test(line)
? true
: mode === "exports"
? false
: null,
source: mode === "imports" ? (tokens.at(-2) ?? null) : null,
},
];
});

71
src/native/parsers/lipo.ts Normal file
View File

@ -0,0 +1,71 @@
import { z } from "zod";
const architectureSchema = z.object({
name: z.string().min(1),
cpu_type: z.string().nullable(),
cpu_subtype: z.string().nullable(),
file_offset: z.number().int().min(0).nullable(),
size: z.number().int().min(0).nullable(),
alignment: z.number().int().min(0).nullable(),
});
export type LipoArchitecture = z.infer<typeof architectureSchema>;
/** Parse `lipo -detailed_info` into deterministic slice metadata. */
export const parseLipoArchitectures = (output: string): LipoArchitecture[] => {
const architectures: LipoArchitecture[] = [];
let current: Record<string, string> | undefined;
const flush = (): void => {
if (current === undefined) return;
const name = current.architecture ?? current["Non-fat file"];
if (name !== undefined)
architectures.push(
architectureSchema.parse({
name: name.includes(" is architecture: ")
? (name.split(" is architecture: ").at(-1) ?? name)
: name,
cpu_type: current.cputype ?? null,
cpu_subtype: current.cpusubtype ?? null,
file_offset: integer(current.offset),
size: integer(current.size),
alignment: alignment(current.align),
}),
);
current = undefined;
};
for (const rawLine of output.split(/\r?\n/u)) {
const line = rawLine.trim();
if (line.startsWith("Non-fat file:")) {
flush();
current = { "Non-fat file": line };
continue;
}
if (line.startsWith("architecture ")) {
flush();
current = { architecture: line.slice("architecture ".length).trim() };
continue;
}
const match = /^(cputype|cpusubtype|offset|size|align)\s+(.+)$/u.exec(line);
if (match?.[1] !== undefined && match[2] !== undefined) {
current ??= {};
current[match[1]] = match[2].trim();
}
}
flush();
if (architectures.length === 0)
throw new TypeError("lipo output contained no architectures");
return architectures;
};
const integer = (value: string | undefined): number | null => {
if (value === undefined) return null;
const parsed = Number.parseInt(value, 10);
return Number.isSafeInteger(parsed) && parsed >= 0 ? parsed : null;
};
const alignment = (value: string | undefined): number | null => {
if (value === undefined) return null;
const exponent = /2\^(\d+)/u.exec(value)?.[1];
if (exponent !== undefined) return 2 ** Number.parseInt(exponent, 10);
return integer(value);
};

296
src/native/parsers/otool.ts Normal file
View File

@ -0,0 +1,296 @@
interface ParsedSection {
readonly segment: string;
readonly name: string;
readonly address: string | null;
readonly size: number | null;
readonly file_offset: number | null;
readonly alignment: number | null;
readonly flags: string[];
}
interface ParsedSegment {
readonly name: string;
readonly vm_address: string | null;
readonly vm_size: number | null;
readonly file_offset: number | null;
readonly file_size: number | null;
readonly maximum_permissions: ReturnType<typeof permissions>;
readonly initial_permissions: ReturnType<typeof permissions>;
readonly sections: ParsedSection[];
}
interface ParsedLoadCommands {
readonly commands: Array<{
index: number;
kind: string;
file_offset: number | null;
fields: Record<string, string | number | null>;
}>;
readonly segments: ParsedSegment[];
readonly dependencies: Array<{
path: string;
kind: string;
current_version: string | null;
compatibility_version: string | null;
}>;
readonly entrypoints: Array<{ file_offset: number }>;
readonly builds: Array<{
platform: string | null;
minimum_os: string | null;
sdk: string | null;
tools: Array<{ name: string; version: string }>;
}>;
uuid: string | null;
readonly flags: Set<string>;
}
/** Parse stable fields from `otool -l`, preserving unknown command fields. */
export const parseOtoolLoadCommands = (output: string) => {
const headerTokens = parseHeaderTokens(output);
const state = createLoadCommandState(headerTokens);
for (const block of output.split(/(?=Load command \d+)/u))
parseLoadCommand(block, state);
return {
fileType: headerTokens?.[4] ?? null,
flags: [...state.flags].sort(),
uuid: state.uuid,
commands: state.commands,
segments: state.segments,
dependencies: state.dependencies,
entrypoints: state.entrypoints,
builds: state.builds,
};
};
const parseHeaderTokens = (output: string): string[] | undefined =>
output
.split(/\r?\n/u)
.map((line) => line.trim())
.find((line) => /^0x[a-fA-F0-9]+\s/u.test(line))
?.split(/\s+/u);
const createLoadCommandState = (
headerTokens: readonly string[] | undefined,
): ParsedLoadCommands => ({
commands: [],
segments: [],
dependencies: [],
entrypoints: [],
builds: [],
uuid: null,
flags: new Set(headerTokens?.slice(7) ?? []),
});
const parseLoadCommand = (block: string, state: ParsedLoadCommands): void => {
const indexText = /^Load command (\d+)/u.exec(block)?.[1];
if (indexText === undefined) return;
const header = block.split(/\n\s*Section\n/u)[0] ?? block;
const fields = parseFields(header);
const kind = stringField(fields, "cmd") ?? "unknown";
state.commands.push({
index: Number.parseInt(indexText, 10),
kind,
file_offset: null,
fields,
});
collectUuid(kind, fields, state);
collectEntrypoint(kind, fields, state);
collectBuild(kind, block, fields, state);
collectDependency(kind, fields, state);
collectSegment(kind, block, fields, state);
collectFlags(fields, state.flags);
};
const collectUuid = (
kind: string,
fields: Readonly<Record<string, string | number | null>>,
state: ParsedLoadCommands,
): void => {
if (kind === "LC_UUID")
state.uuid = stringField(fields, "uuid") ?? state.uuid;
};
const collectEntrypoint = (
kind: string,
fields: Readonly<Record<string, string | number | null>>,
state: ParsedLoadCommands,
): void => {
if (kind !== "LC_MAIN") return;
const entry = numberField(fields, "entryoff");
if (entry !== null) state.entrypoints.push({ file_offset: entry });
};
const collectBuild = (
kind: string,
block: string,
fields: Readonly<Record<string, string | number | null>>,
state: ParsedLoadCommands,
): void => {
if (kind !== "LC_BUILD_VERSION") return;
state.builds.push({
platform: stringField(fields, "platform"),
minimum_os: stringField(fields, "minos"),
sdk: stringField(fields, "sdk"),
tools: parseBuildTools(block),
});
};
const collectDependency = (
kind: string,
fields: Readonly<Record<string, string | number | null>>,
state: ParsedLoadCommands,
): void => {
if (!kind.startsWith("LC_LOAD_") && kind !== "LC_ID_DYLIB") return;
state.dependencies.push({
path: stripOffsetSuffix(stringField(fields, "name")),
kind,
current_version: stringField(fields, "current version"),
compatibility_version: stringField(fields, "compatibility version"),
});
};
const collectSegment = (
kind: string,
block: string,
fields: Readonly<Record<string, string | number | null>>,
state: ParsedLoadCommands,
): void => {
if (kind === "LC_SEGMENT" || kind === "LC_SEGMENT_64")
state.segments.push(parseSegment(block, fields));
};
const collectFlags = (
fields: Readonly<Record<string, string | number | null>>,
flags: Set<string>,
): void => {
const rawFlags = stringField(fields, "flags");
if (rawFlags === null) return;
for (const flag of rawFlags.split(/\s+/u))
if (flag.length > 0) flags.add(flag);
};
const parseFields = (block: string): Record<string, string | number | null> => {
const fields: Record<string, string | number | null> = {};
for (const rawLine of block.split(/\r?\n/u).slice(1)) {
const line = rawLine.trim();
const match = /^(\S+(?:\s+version)?)\s+(.+)$/u.exec(line);
if (match?.[1] === undefined || match[2] === undefined) continue;
const value = match[2].trim();
fields[match[1]] = numeric(value) ?? value;
}
return fields;
};
const parseSegment = (
block: string,
fields: Readonly<Record<string, string | number | null>>,
): ParsedSegment => ({
name: stringField(fields, "segname") ?? "unknown",
vm_address: hexField(fields, "vmaddr"),
vm_size: numberField(fields, "vmsize"),
file_offset: numberField(fields, "fileoff"),
file_size: numberField(fields, "filesize"),
maximum_permissions: permissions(stringField(fields, "maxprot")),
initial_permissions: permissions(stringField(fields, "initprot")),
sections: parseSections(block),
});
const parseSections = (block: string): ParsedSection[] =>
block
.split(/(?=\n\s*Section\n)/u)
.slice(1)
.map((sectionBlock) => {
const fields = parseFields(`Section${sectionBlock}`);
return {
segment: stringField(fields, "segname") ?? "unknown",
name: stringField(fields, "sectname") ?? "unknown",
address: hexField(fields, "addr"),
size: numberField(fields, "size"),
file_offset: numberField(fields, "offset"),
alignment: sectionAlignment(numberField(fields, "align")),
flags: (stringField(fields, "flags") ?? "")
.split(/\s+/u)
.filter((flag) => flag.length > 0),
};
});
const parseBuildTools = (
block: string,
): Array<{ name: string; version: string }> => {
const tools: Array<{ name: string; version: string }> = [];
const lines = block.split(/\r?\n/u).map((line) => line.trim());
for (let index = 0; index < lines.length - 1; index += 1) {
const tool = /^tool\s+(.+)$/u.exec(lines[index] ?? "")?.[1];
const version = /^version\s+(.+)$/u.exec(lines[index + 1] ?? "")?.[1];
if (tool !== undefined && version !== undefined)
tools.push({ name: tool, version });
}
return tools;
};
const permissions = (raw: string | null) => {
if (raw === null)
return { read: null, write: null, execute: null, raw: null };
const numericValue = numeric(raw);
if (numericValue !== null)
return {
read: (numericValue & 4) !== 0,
write: (numericValue & 2) !== 0,
execute: (numericValue & 1) !== 0,
raw,
};
if (/^[r-][w-][x-]$/u.test(raw))
return {
read: raw[0] === "r",
write: raw[1] === "w",
execute: raw[2] === "x",
raw,
};
return { read: null, write: null, execute: null, raw };
};
const numeric = (value: string): number | null => {
const token = value.split(/\s+/u)[0];
if (token === undefined || !/^(?:0x[a-fA-F0-9]+|\d+)$/u.test(token))
return null;
const parsed = Number.parseInt(token, token.startsWith("0x") ? 16 : 10);
return Number.isSafeInteger(parsed) && parsed >= 0 ? parsed : null;
};
const stringField = (
fields: Readonly<Record<string, string | number | null>>,
name: string,
): string | null => {
const value = fields[name];
return typeof value === "string"
? value
: value === undefined || value === null
? null
: String(value);
};
const numberField = (
fields: Readonly<Record<string, string | number | null>>,
name: string,
): number | null => {
const value = fields[name];
return typeof value === "number"
? value
: typeof value === "string"
? numeric(value)
: null;
};
const hexField = (
fields: Readonly<Record<string, string | number | null>>,
name: string,
): string | null => {
const value = numberField(fields, name);
return value === null ? null : `0x${value.toString(16)}`;
};
const stripOffsetSuffix = (value: string | null): string =>
value?.replace(/\s+\(offset\s+\d+\)$/u, "") ?? "unknown";
const sectionAlignment = (exponent: number | null): number | null =>
exponent === null || exponent > 52 ? null : 2 ** exponent;

Some files were not shown because too many files have changed in this diff Show More