diff --git a/AGENTS.md b/AGENTS.md index 6e27eee99..09580a6d7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -41,7 +41,7 @@ See [docs/architecture.mermaid](docs/architecture.mermaid) for a visual architec - `tests/`: Vitest suite. `tests/fixtures/` holds reusable provider-process fixtures, source-owned synthetic JavaScript graph and Electron/Webpack/Rspack artifact trees, and the fake launcher, Hopper bridge, and CDP seams. - `scripts/verify-real-hopper.mjs`: real-Hopper end-to-end verifier. - `scripts/verify-real-ghidra.mjs`: real Linux Ghidra verifier for x86-64 debug/stripped ELF, AArch64 ELF, PE, and Mach-O function semantics plus complete cleanup. -- `scripts/verify-real-ghidra-windows.mjs`: real Windows x64 Ghidra P0 verifier for the source-owned native PE fixture, all 18 admitted operations, digest linkage, loopback transport, and cleanup. +- `scripts/verify-real-ghidra-windows.mjs`: real Windows x64 Ghidra P0 verifier for the source-owned native PE fixture, all 19 admitted operations, digest linkage, loopback transport, and cleanup. - `scripts/verify-real-browser.mjs`: real Chrome end-to-end verifier for the passive CDP provider. - `scripts/print-mcp-config.mjs`: prints an MCP server config with absolute paths filled in (`npm run config:print -- /path/to/binary`). diff --git a/README.md b/README.md index b0823f4b0..13d10fdb4 100644 --- a/README.md +++ b/README.md @@ -237,13 +237,13 @@ On Windows, set the same variables in PowerShell and run `rea doctor --json`; au REA loads its packaged Java `HeadlessScript` with `-scriptPath`, copies and digest-verifies the target in an ephemeral runtime, enables `-readOnly` and `-deleteProject`, caps auto-analysis at 300 seconds with two CPUs and a 2 GiB Java heap, and authenticates every request. Linux uses a mode-0600 Unix socket. Windows P0 uses token-authenticated IPv4 loopback and a token-free endpoint record because Node path-based IPC does not connect to Java AF_UNIX sockets on Windows. The bridge verifies Ghidra's imported-byte SHA-256 before serving any operation. -The Ghidra adapter declares 18 direct and enhanced operations. Its ten inventory operations are `list_documents`, `list_procedures`, `list_strings`, `list_names`, `list_segments`, `address_name`, `procedure_address`, `resolve_containing_procedure`, `search_procedures`, and `search_strings`. It also admits `procedure_info`, `procedure_pseudo_code`, `procedure_assembly`, `procedure_callers`, `procedure_callees`, `procedure_references`, `xrefs`, and `analyze_function`. These capabilities enable the shared Swift/Objective-C inventory workflows, `binary_overview`, `batch_decompile`, `get_call_graph`, `find_xrefs_to_name`, `trace_feature`, and complete function dossiers. Default-space addresses are lowercase `0x` hexadecimal. Other spaces, including `EXTERNAL`, use `:0x`. Symbol results identify primary, dynamic, external, type, and source facts; procedures distinguish external functions and thunks; strings identify charset, missing-terminator state, byte length, and value truncation; memory-block ends are exclusive and permissions come directly from Ghidra. +The Ghidra adapter declares 19 direct and enhanced operations. Its ten inventory operations are `list_documents`, `list_procedures`, `list_strings`, `list_names`, `list_segments`, `address_name`, `procedure_address`, `resolve_containing_procedure`, `search_procedures`, and `search_strings`. It also admits `procedure_info`, `procedure_pseudo_code`, `procedure_assembly`, `read_function_instructions`, `procedure_callers`, `procedure_callees`, `procedure_references`, `xrefs`, and `analyze_function`. `read_function_instructions` is the offset-paginated fast path for raw instruction windows: it does not invoke the decompiler or whole-program name/string inventories, and is also exposed as `rea instructions`. These capabilities enable the shared Swift/Objective-C inventory workflows, `binary_overview`, `batch_decompile`, `get_call_graph`, `find_xrefs_to_name`, `trace_feature`, and complete function dossiers. Default-space addresses are lowercase `0x` hexadecimal. Other spaces, including `EXTERNAL`, use `:0x`. Symbol results identify primary, dynamic, external, type, and source facts; procedures distinguish external functions and thunks; strings identify charset, missing-terminator state, byte length, and value truncation; memory-block ends are exclusive and permissions come directly from Ghidra. The bridge serves operations only after auto-analysis completes. Each Program owns one persistent `DecompInterface`; a bounded 32-request FIFO serializes Ghidra API access, and every decompile has a 30-second native deadline. Reference results preserve Ghidra's call/jump/data/read/write/indirect/computed/external facts, while unresolved targetless flows remain explicitly unknown. Synthetic entry-point references without actionable memory sources are omitted. Pseudocode and assembly are provider-specific observations, not original source or Hopper-equivalent text. An analysis timeout, scan or inventory safety limit, request timeout, or oversized response fails explicitly instead of returning a partial result labeled complete. `npm run verify:ghidra` compiles source-owned x86-64 debug and stripped ELF, AArch64 ELF, x86-64 PE, and x86-64 Mach-O fixtures. Against real Ghidra 12.1.2 it validates every admitted operation, direct and indirect calls, imports/exports/thunks, typed references, strings/xrefs, multi-block CFG, cancellation, deadlines, concurrency, malformed inputs, and complete process/project cleanup. Set `REA_CC`, `REA_CLANG`, or `REA_LLD_LINK` only when the corresponding compiler command is not on `PATH`. -`npm run verify:ghidra:windows` uses a deterministic source-owned native x86-64 PE application and requires all 18 operations, target/snapshot/import digest linkage, authenticated loopback transport, and cleanup on a controlled Windows x64 Ghidra 12.1.2 runner. This proof does not establish Job Object ownership, private DACLs, or reparse-point-safe authority. +`npm run verify:ghidra:windows` uses a deterministic source-owned native x86-64 PE application and requires all 19 operations, target/snapshot/import digest linkage, authenticated loopback transport, and cleanup on a controlled Windows x64 Ghidra 12.1.2 runner. This proof does not establish Job Object ownership, private DACLs, or reparse-point-safe authority. To remove only REA-owned MCP registrations and the managed skill: @@ -373,7 +373,7 @@ REA handles the app analysis in steps 1–5. The agent performs step 6 with its | Tool family | Count | Examples | | ------------------------- | ----: | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Native inspection | 33 | procedures, pseudocode, assembly, strings, names, segments, callers, callees, xrefs, annotations | +| Native inspection | 34 | procedures, pseudocode, assembly, strings, names, segments, callers, callees, xrefs, annotations | | Investigation workflows | 10 | `binary_overview`, `analyze_function`, `batch_decompile`, `trace_feature`, call graphs, Swift and Objective-C discovery | | Native macOS utilities | 5 | Mach-O metadata, code signatures, plists, architectures, Swift demangling; Hopper-free and provenance-bearing | | Artifact graph | 2 | deterministic directory, ZIP/APK/IPA/MSIX/AppX, and ASAR inventory; explicitly selected extraction into an absent owned tree | @@ -592,10 +592,17 @@ and retains one provider-attributed, token-verified observation per started provider. Each observation carries `observed_at` and remains `unavailable` or `verified`; a verified empty descendant list is distinct from both. Snapshots describe bounded observations, not current live state or historical absence. +For providers with serial work, `analysis_activity` distinguishes `idle`, +`busy`, and `timed_out_busy`, and reports the active operation, elapsed time, +caller state, timeout, and queued-request count. A caller timeout therefore +does not falsely imply that Hopper's Python thread is available. `close_binary` +clears the REA session but returns `cleanup_incomplete` when authenticated +document shutdown, owned process cleanup, or private runtime removal cannot be +verified. Reopening same target without a selector keeps its binding; runtime failure never selects another provider silently. Ghidra can appear as an available, target-compatible candidate after doctor -validates its exact installation. Its capability list contains the 18 admitted +validates its exact installation. Its capability list contains the 19 admitted read-only inventory and function-analysis operations; selecting it still does not make Hopper-only GUI or mutation operations available and never triggers a silent fallback. @@ -627,7 +634,11 @@ REA starts Hopper when needed; Hopper does not need to be running first. Hopper' REA derives explicit format and architecture arguments to prevent common FAT and ARM selection dialogs. Other Hopper or macOS dialogs may still require a person. REA reports timeouts and remediation through CLI or MCP results instead of attempting to answer UI prompts. -Closing a REA session shuts down its bridge and removes its private socket directory. It does not quit a Hopper application the user may be using. +Hopper bridge calls pass through a bounded serial FIFO because Hopper's Python API runs on one dedicated thread. Queue wait counts against the caller deadline; a timeout settles the caller but does not release the wire slot until Hopper actually replies. `binary_session.analysis_activity` exposes that detached work and MCP progress reports elapsed time. Successful decompilation text is cached per document and procedure, shared by pseudocode and dossier requests, and invalidated after rename or comment mutation. + +Hopper's synchronous public Python calls cannot stream a partial response after the caller deadline. Use `read_function_instructions` or `rea instructions` when raw instruction orientation is enough: this path does not decompile or scan whole-program name/string inventories. If a dossier has already timed out, inspect `analysis_activity`; do not assume a raw request can enter the serial bridge until that detached Hopper call actually returns. + +Closing a REA session shuts down its bridge and removes its private socket directory. It does not quit a Hopper application the user may be using. If shutdown or cleanup cannot be verified, `close_binary` returns `cleanup_incomplete` with the affected local resources. ## Advanced process-capture setup diff --git a/README_ar.md b/README_ar.md index 37359d5ed..007a83ba2 100644 --- a/README_ar.md +++ b/README_ar.md @@ -161,7 +161,7 @@ rea uninstall --purge-data | عائلة الأدوات | العدد | أمثلة | | --------------------- | ----: | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| فحص الملفات التنفيذية | 33 | الدوال، والشيفرة شبه المصدرية، والتعليمات، والسلاسل، والأسماء، والمراجع، والتعليقات | +| فحص الملفات التنفيذية | 34 | الدوال، والشيفرة شبه المصدرية، والتعليمات، والسلاسل، والأسماء، والمراجع، والتعليقات | | التحليل المركب | 10 | النظرة العامة، وفك الترجمة الدفعي، ومخططات الاستدعاء، والمراجع، واكتشاف الأنواع | | أدوات macOS الأصلية | 5 | بيانات Mach-O والتوقيعات وملفات plist والمعماريات وفك رموز Swift دون تشغيل Hopper | | رسم القطع الأثرية | 2 | جرد حتمي للدلائل وZIP/APK/IPA وASAR واستخراج معاملاتي محدد صراحة | diff --git a/README_ja.md b/README_ja.md index 86afdcdfb..5c68b6053 100644 --- a/README_ja.md +++ b/README_ja.md @@ -151,7 +151,7 @@ REA は手順 1〜5 のバイナリ解析を処理し、手順 6 はエージェ | ツール群 | 数 | 例 | | ---------------------- | --: | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| バイナリ調査 | 33 | プロシージャ、疑似コード、アセンブリ、文字列、名前、セグメント、callers、callees、xrefs、注釈 | +| バイナリ調査 | 34 | プロシージャ、疑似コード、アセンブリ、文字列、名前、セグメント、callers、callees、xrefs、注釈 | | 合成解析 | 10 | `binary_overview`, `analyze_function`, `batch_decompile`, `get_call_graph`, `find_xrefs_to_name`, Swift / ObjC 検出 | | macOS ネイティブ | 5 | Mach-O メタデータ、署名、plist、アーキテクチャ、Swift デマングル。Hopper 起動不要 | | アーティファクトグラフ | 2 | ディレクトリ、ZIP/APK/IPA、ASAR の決定的一覧と明示選択されたトランザクション抽出 | diff --git a/README_ko.md b/README_ko.md index f64b885a2..6709261d3 100644 --- a/README_ko.md +++ b/README_ko.md @@ -151,7 +151,7 @@ REA는 1–5단계의 바이너리 분석을 처리합니다. 6단계는 에이 | 도구 그룹 | 수 | 예시 | | ------------------- | --: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| 바이너리 검사 | 33 | 프로시저, 의사 코드, 어셈블리, 문자열, 이름, 세그먼트, callers, callees, xrefs, 주석 | +| 바이너리 검사 | 34 | 프로시저, 의사 코드, 어셈블리, 문자열, 이름, 세그먼트, callers, callees, xrefs, 주석 | | 합성 분석 | 10 | `binary_overview`, `analyze_function`, `batch_decompile`, `get_call_graph`, `find_xrefs_to_name`, Swift 및 ObjC 탐색 | | macOS 네이티브 도구 | 5 | Mach-O 메타데이터, 코드 서명, plist, 아키텍처, Swift 디맹글링. Hopper 실행 불필요 | | 아티팩트 그래프 | 2 | 디렉터리, ZIP/APK/IPA, ASAR 결정적 인벤토리와 명시적으로 선택한 트랜잭션 추출 | diff --git a/README_zh.md b/README_zh.md index 2d43d0d4c..c621bf066 100644 --- a/README_zh.md +++ b/README_zh.md @@ -151,7 +151,7 @@ REA 负责第 1–5 步中的二进制分析。第 6 步由智能体使用其常 | 工具类别 | 数量 | 示例 | | -------------- | ---: | --------------------------------------------------------------------------------------------------------------------------------------------- | -| 二进制检查 | 33 | 过程、伪代码、汇编、字符串、名称、段、调用者、被调用者、交叉引用、注释 | +| 二进制检查 | 34 | 过程、伪代码、汇编、字符串、名称、段、调用者、被调用者、交叉引用、注释 | | 组合分析 | 10 | `binary_overview`, `analyze_function`, `batch_decompile`, `get_call_graph`, `find_xrefs_to_name`、Swift 与 ObjC 发现 | | macOS 原生工具 | 5 | Mach-O 元数据、代码签名、plist、架构与 Swift 符号还原,无需启动 Hopper | | 制品图谱 | 2 | 确定性目录、ZIP/APK/IPA 与 ASAR 清单,以及显式选择的事务式提取 | diff --git a/bridge/ghidra/ReaGhidraBridge.java b/bridge/ghidra/ReaGhidraBridge.java index 57b65dac4..285750182 100644 --- a/bridge/ghidra/ReaGhidraBridge.java +++ b/bridge/ghidra/ReaGhidraBridge.java @@ -72,7 +72,7 @@ import ghidra.program.model.symbol.Symbol; import ghidra.program.model.symbol.SymbolIterator; public final class ReaGhidraBridge extends HeadlessScript { - private static final int BRIDGE_VERSION = 4; + private static final int BRIDGE_VERSION = 5; private static final int MAX_DESCRIPTOR_BYTES = 16 * 1024; private static final int MAX_REQUEST_CHARACTERS = 256 * 1024; private static final int MAX_RESPONSE_BYTES = 1024 * 1024; @@ -123,6 +123,7 @@ public final class ReaGhidraBridge extends HeadlessScript { "procedure_callers", "procedure_info", "procedure_pseudo_code", + "read_function_instructions", "procedure_references", "xrefs" }; @@ -311,6 +312,7 @@ public final class ReaGhidraBridge extends HeadlessScript { case "procedure_callers" -> procedureCalls(request.params, true); case "procedure_info" -> procedureInfo(request.params); case "procedure_pseudo_code" -> procedurePseudocode(request.params); + case "read_function_instructions" -> readFunctionInstructions(request.params); case "procedure_references" -> procedureReferences(request.params); case "resolve_containing_procedure" -> containingProcedure(request.params); case "search_procedures" -> search(request.params, true); @@ -390,6 +392,51 @@ public final class ReaGhidraBridge extends HeadlessScript { return GSON.toJsonTree(renderAssembly(scan.instructions)); } + private JsonObject readFunctionInstructions(JsonObject params) throws Exception { + requireKeys(params, Set.of("document", "procedure", "offset", "limit")); + requireDocument(params); + Function function = resolveProcedure(requireString(params, "procedure")); + int offset = requireBoundedInteger(params, "offset", 0, 100_000); + int limit = requireBoundedInteger(params, "limit", 1, 500); + InstructionScan scan = scanInstructions(function, offset + limit + 1); + int start = Math.min(offset, scan.instructions.size()); + int end = Math.min(scan.instructions.size(), start + limit); + JsonArray items = new JsonArray(); + for (String line : renderAssemblyLines(scan.instructions.subList(start, end))) { + items.add(line); + } + boolean hasMore = scan.truncated || end < scan.instructions.size(); + JsonObject instructions = new JsonObject(); + instructions.add("items", items); + if (scan.truncated) { + instructions.add("total", JsonNull.INSTANCE); + } + else { + instructions.addProperty("total", scan.instructions.size()); + } + instructions.addProperty("returned", items.size()); + instructions.addProperty("truncated", hasMore); + if (hasMore) { + instructions.addProperty("next_offset", end); + } + else { + instructions.add("next_offset", JsonNull.INSTANCE); + } + + JsonArray limitations = new JsonArray(); + limitations.add("Instruction text and ordering are Ghidra-specific representations."); + limitations.add( + "The fast path does not invoke the decompiler or scan whole-program names and strings." + ); + JsonObject result = new JsonObject(); + result.add("procedure", procedureIdentity(function)); + result.add("instructions", instructions); + result.addProperty("instructions_scanned", scan.instructions.size()); + result.addProperty("instruction_scan_truncated", scan.truncated); + result.add("limitations", limitations); + return result; + } + private JsonArray procedureCalls(JsonObject params, boolean callers) throws Exception { requireKeys(params, Set.of("document", "procedure")); requireDocument(params); diff --git a/bridge/hopper_bridge.py b/bridge/hopper_bridge.py index 74070ad59..1a9d03c00 100644 --- a/bridge/hopper_bridge.py +++ b/bridge/hopper_bridge.py @@ -16,6 +16,7 @@ MAX_LINE_BYTES = 10 * 1024 * 1024 BAD_ADDRESSES = (-1, 0xFFFFFFFFFFFFFFFF, None) _selected_document = None _search_inventory_cache = {} +_pseudocode_cache = {} MAX_SEARCH_PATTERN_LENGTH = 256 MAX_SEARCH_VALUE_LENGTH = 4096 @@ -225,12 +226,35 @@ def _invalidate_search_inventory(document): del _search_inventory_cache[key] +def _invalidate_pseudocode(document): + """Discard cached decompilation after an analysis annotation changes.""" + document_id = id(document) + for key in list(_pseudocode_cache): + if key[0] == document_id: + del _pseudocode_cache[key] + + +def _pseudocode(document, procedure): + """Reuse one provider decompilation for repeated bounded projections.""" + key = (id(document), procedure.getEntryPoint()) + if key not in _pseudocode_cache: + _pseudocode_cache[key] = procedure.decompile() + return _pseudocode_cache[key] + + def _search_inventory(document, kind): """Cache an immutable, address-sorted inventory for an unchanged document.""" key = (id(document), kind) inventory = _search_inventory_cache.get(key) if inventory is None: - values = _procedure_map(document) if kind == "procedure" else _strings(document) + if kind == "procedure": + values = _procedure_map(document) + elif kind == "string": + values = _strings(document) + elif kind == "name": + values = {_hex(address): value for address, value in _name_map(document).items()} + else: + raise ValueError("Unknown inventory kind") inventory = tuple(sorted(values.items(), key=lambda item: int(item[0], 16))) _search_inventory_cache[key] = inventory return inventory @@ -507,6 +531,22 @@ def _name_map(document): return result +def _render_instruction(segment, address): + instruction = segment.getInstructionAtAddress(address) + if instruction is None: + return None + arguments = [ + instruction.getFormattedArgument(index) + for index in range(instruction.getArgumentCount()) + ] + suffix = ", ".join(value for value in arguments if value is not None) + return "%s: %s%s" % ( + _hex(address), + instruction.getInstructionString(), + (" " + suffix) if suffix else "", + ) + + def _assembly(procedure, limit=None): """Render bounded assembly while guarding against malformed instruction cycles.""" lines = [] @@ -520,9 +560,9 @@ def _assembly(procedure, limit=None): instruction = segment.getInstructionAtAddress(address) if instruction is None: break - arguments = [instruction.getFormattedArgument(index) for index in range(instruction.getArgumentCount())] - suffix = ", ".join(value for value in arguments if value is not None) - lines.append("%s: %s%s" % (_hex(address), instruction.getInstructionString(), (" " + suffix) if suffix else "")) + line = _render_instruction(segment, address) + if line is not None: + lines.append(line) if limit is not None and len(lines) >= limit: return "\n".join(lines) length = instruction.getInstructionLength() @@ -532,6 +572,45 @@ def _assembly(procedure, limit=None): return "\n".join(lines) +def _read_function_instructions(document, params): + """Read one bounded raw-instruction window without invoking decompilation.""" + procedure = _procedure(document, params.get("procedure")) + offset = _offset(params, "offset") + limit = params.get("limit", 64) + if offset > 100000: + raise ValueError("offset must not exceed 100000") + if not isinstance(limit, int) or isinstance(limit, bool) or limit < 1 or limit > 500: + raise ValueError("limit must be an integer between 1 and 500") + addresses, scan_truncated = _instruction_addresses( + procedure, offset + limit + 1 + ) + start = min(offset, len(addresses)) + end = min(len(addresses), start + limit) + segment = procedure.getSegment() + items = [] + for address in addresses[start:end]: + line = _render_instruction(segment, address) + if line is not None: + items.append(line) + has_more = scan_truncated or end < len(addresses) + return { + "procedure": _procedure_identity(procedure), + "instructions": { + "items": items, + "total": None if scan_truncated else len(addresses), + "returned": len(items), + "truncated": has_more, + "next_offset": end if has_more else None, + }, + "instructions_scanned": len(addresses), + "instruction_scan_truncated": scan_truncated, + "limitations": [ + "Instruction text and ordering are Hopper-specific representations.", + "The fast path does not decompile the procedure or scan whole-program names and strings.", + ], + } + + def _analyze_function(document, params): """Collect a bounded, single-pass function dossier for agent callers.""" procedure = _procedure(document, params.get("procedure")) @@ -560,8 +639,8 @@ def _analyze_function(document, params): "end": _hex(block.getEndingAddress()), "successors": sorted(set(successors), key=lambda value: int(value, 16)), }) - pseudo = procedure.decompile() or "" - assembly_lines = _assembly(procedure).splitlines() if params.get("include_assembly", False) else [] + pseudo = _pseudocode(document, procedure) or "" + assembly_lines = _assembly(procedure, max_instructions).splitlines() if params.get("include_assembly", False) else [] callers = sorted((_procedure_identity(item) for item in procedure.getAllCallerProcedures()), key=lambda item: int(item["address"], 16)) callees = sorted((_procedure_identity(item) for item in procedure.getAllCalleeProcedures()), key=lambda item: int(item["address"], 16)) comments = [] @@ -595,8 +674,14 @@ def _analyze_function(document, params): incoming.append(item) if source in procedure_addresses: outgoing.append(item) - string_map = {int(address, 16): value for address, value in _strings(document).items()} - name_map = _name_map(document) + string_map = { + int(address, 16): value + for address, value in _search_inventory(document, "string") + } + name_map = { + int(address, 16): value + for address, value in _search_inventory(document, "name") + } referenced_strings = [] referenced_names = [] for edge in outgoing: @@ -615,7 +700,13 @@ def _analyze_function(document, params): return { "procedure": {"address": _hex(procedure.getEntryPoint()), "name": _procedure_name(procedure), "classification": None, "signature": procedure.signatureString(), "locals": _procedure_locals(procedure)}, "pseudocode": {"text": pseudo_text, "total_chars": len(pseudo), "returned_chars": len(pseudo_text), "truncated": pseudo_next < len(pseudo), "next_offset": pseudo_next if pseudo_next < len(pseudo) else None}, - "assembly": _bounded(assembly_lines, assembly_offset, max_instructions), + "assembly": _bounded( + assembly_lines, + assembly_offset, + max_instructions, + None, + instruction_scan_truncated if params.get("include_assembly", False) else False, + ), "comments": collection("comments", comments, instruction_scan_truncated), "callers": collection("callers", callers), "callees": collection("callees", callees), "incoming_references": collection("incoming_references", incoming, instruction_scan_truncated), @@ -674,6 +765,8 @@ def _dispatch(method, params): if method == "analyze_function": return _analyze_function(document, params) + if method == "read_function_instructions": + return _read_function_instructions(document, params) if method == "resolve_containing_procedure": address = _address(document, params.get("address")) procedure, reason = _containing_procedure(document, address) @@ -740,19 +833,16 @@ def _dispatch(method, params): }) return result if method == "list_procedures": - return _page(_procedure_map(document), params.get("offset", 0), params.get("limit", 100)) + return _page(dict(_search_inventory(document, "procedure")), params.get("offset", 0), params.get("limit", 100)) if method == "list_strings": - values = _strings(document) + values = dict(_search_inventory(document, "string")) requested = params.get("address") if requested is not None: key = _hex(_address(document, requested)) values = {key: values[key]} if key in values else {} return _page(values, params.get("offset", 0), params.get("limit", 100)) if method == "list_names": - result = {} - for segment in document.getSegmentsList(): - for item in segment.getNamedAddresses(): - result[_hex(item)] = segment.getNameAtAddress(item) + result = dict(_search_inventory(document, "name")) requested = params.get("address") if requested is not None: key = _hex(_address(document, requested)) @@ -768,7 +858,7 @@ def _dispatch(method, params): if method == "procedure_assembly": return _assembly(procedure) if method == "procedure_pseudo_code": - return procedure.decompile() + return _pseudocode(document, procedure) if method == "procedure_callers": return sorted((_hex(item.getEntryPoint()) for item in procedure.getAllCallerProcedures()), key=lambda value: int(value, 16)) if method == "procedure_callees": @@ -779,12 +869,18 @@ def _dispatch(method, params): return {"name": _procedure_name(procedure), "entrypoint": _hex(procedure.getEntryPoint()), "basicblock_count": procedure.getBasicBlockCount(), "length": length, "signature": procedure.signatureString(), "locals": _procedure_locals(procedure), "classification": None} if method == "set_address_name": address = _address(document, params.get("address")) - result = document.setNameAtAddress(address, params["name"]) - _invalidate_search_inventory(document) + try: + result = document.setNameAtAddress(address, params["name"]) + finally: + _invalidate_search_inventory(document) + _invalidate_pseudocode(document) return result if method == "set_addresses_names": - result = {key: document.setNameAtAddress(_address(document, key), value) for key, value in params["names"].items()} - _invalidate_search_inventory(document) + try: + result = {key: document.setNameAtAddress(_address(document, key), value) for key, value in params["names"].items()} + finally: + _invalidate_search_inventory(document) + _invalidate_pseudocode(document) return result if method in ("set_comment", "set_inline_comment"): address = _address(document, params.get("address")) @@ -792,6 +888,7 @@ def _dispatch(method, params): setter = segment.setCommentAtAddress if method == "set_comment" else segment.setInlineCommentAtAddress getter = segment.getCommentAtAddress if method == "set_comment" else segment.getInlineCommentAtAddress setter(address, params["comment"]) + _invalidate_pseudocode(document) return getter(address) == params["comment"] if method == "list_bookmarks": return [{"address": _hex(item), "name": document.getBookmarkName(item)} for item in document.getBookmarks()] diff --git a/docs/adr/0001-provider-selection-and-analysis-profiles.md b/docs/adr/0001-provider-selection-and-analysis-profiles.md index 4c653007c..20848a71f 100644 --- a/docs/adr/0001-provider-selection-and-analysis-profiles.md +++ b/docs/adr/0001-provider-selection-and-analysis-profiles.md @@ -6,7 +6,7 @@ target binding, analysis-profile commitment, and snapshot/Evidence migration are implemented. Ghidra discovery, target/profile resolution, doctor checks, the private headless-session foundation, ten read-only inventory capabilities, - and eight function-analysis capabilities are implemented with real + and nine function-analysis capabilities are implemented with real cross-format conformance. ## Context diff --git a/docs/installation.md b/docs/installation.md index 3d074e50c..7cabc57d0 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -151,7 +151,7 @@ REA's Ghidra provider is bring-your-own and supports Linux x64 with the exact official Ghidra 12.1.2 release and a 64-bit full JDK 21. An experimental Windows x64 P0 supports approved native x86-64 PE applications. It supplies discovery, analysis-profile commitment, an isolated read-only headless session, -ten bounded inventory/name/search operations, and eight function-analysis +ten bounded inventory/name/search operations, and nine function-analysis operations covering metadata, decompilation, assembly, resolved calls, typed references, xrefs, CFG, and dossiers. GUI state and analysis mutations remain unavailable through Ghidra. diff --git a/docs/product-catalog.json b/docs/product-catalog.json index 5e5390236..598b4628d 100644 --- a/docs/product-catalog.json +++ b/docs/product-catalog.json @@ -11,12 +11,12 @@ "skill_version": "23" }, "tools": { - "total": 98, + "total": 99, "families": [ { "id": "direct", "surface": "official-proxy", - "count": 33, + "count": 34, "tools": [ "address_name", "comment", @@ -40,6 +40,7 @@ "procedure_info", "procedure_pseudo_code", "procedure_references", + "read_function_instructions", "resolve_containing_procedure", "search_procedures", "search_strings", @@ -193,6 +194,7 @@ "procedure_info", "procedure_pseudo_code", "procedure_references", + "read_function_instructions", "resolve_containing_procedure", "search_procedures", "search_strings", @@ -227,6 +229,7 @@ "procedure_info", "procedure_pseudo_code", "procedure_references", + "read_function_instructions", "resolve_containing_procedure", "search_procedures", "search_strings", @@ -517,7 +520,7 @@ } ], "cli": { - "primary_count": 58, + "primary_count": 59, "commands": [ "analyze", "analyze-javascript-application", @@ -553,6 +556,7 @@ "inspect-plist", "inspect-signature", "inspect-web-page", + "instructions", "inventory-artifact", "investigate-versions", "list-architectures", @@ -587,17 +591,17 @@ }, "runtime_catalog": { "counts": { - "cli_commands": 58, - "mcp_tools": 98, + "cli_commands": 59, + "mcp_tools": 99, "mcp_prompts": 6, "mcp_resources": 2, "mcp_resource_templates": 9 }, "digests": { - "tools_sha256": "15544cf838c730784a082d0ff9e0feccbe35acd1675d7dfbe99d63296b72b3e3", + "tools_sha256": "9159861cda3cc76f876dde306efae886dd310419d4e972f89ab7b85ff5789485", "prompts_sha256": "f6da3ad38a1000c11364d1710f122ac4990551f04efd8c53ffb2e819b5cf5707", "resources_sha256": "94bebbab1a1b3cf47644c38d3de6469de568caa7bd7b842f529ead8ad320a912", - "combined_sha256": "aa413204625f3aeb62732d0f732b9b8b24832eea4520214b1e6bd204e599bcd6" + "combined_sha256": "e787c57af8337f17b9da72745fcb328634af3b39be1d9ca8bfe1836274351f92" } } } diff --git a/docs/provider-evaluation.md b/docs/provider-evaluation.md index 369bff7ac..46d2264c7 100644 --- a/docs/provider-evaluation.md +++ b/docs/provider-evaluation.md @@ -4,7 +4,7 @@ Status: the Ghidra read-only analysis provider is shipped on Linux x64 and has an experimental Windows x64 P0 for approved native PE applications. It validates an exact bring-your-own Ghidra 12.1.2/JDK 21 environment, resolves a provider/version/profile commitment, runs one isolated read-only headless -import, and publishes 18 operation-level capabilities after the authenticated +import, and publishes 19 operation-level capabilities after the authenticated post-analysis handshake. The provider-neutral target, provider registry, deterministic target binding, @@ -69,14 +69,14 @@ Unix socket. Windows uses authenticated IPv4 loopback with a strict token-free endpoint record. Both transports report actual Ghidra/language/compiler/analysis/import-digest metadata, accept only the exact authenticated `ping`, `shutdown`, and ten -inventory plus eight function-analysis methods, and deletes the socket. Close, +inventory plus nine function-analysis methods, and deletes the socket. Close, cancellation, timeout, malformed protocol, or process exit stops the owned process resources and removes the entire runtime root. Windows P0 uses bounded `taskkill` tree termination and explicitly does not claim Job Object, private-DACL, or reparse-point authority; see [Windows Ghidra P0](windows-ghidra-p0.md). -The provider catalog lists only the 18 proved Ghidra operations. GUI cursor, +The provider catalog lists only the 19 proved Ghidra operations. GUI cursor, navigation, and mutation operations remain absent; the router therefore reports them unavailable instead of borrowing Hopper semantics or inferring capability from a successful import. @@ -103,13 +103,14 @@ capability from a successful import. | Serialization | A bounded FIFO admits at most 32 active-plus-queued requests and sends one Program request at a time. Queue wait counts against the caller's deadline, and queued cancellation is prompt. This is an adapter safety commitment, not a claim that every Ghidra API is thread-safe. | | Function identity | Every function result carries the entry address and Ghidra FunctionManager classification for external, thunk, and resolved thunk target. These are observations, not proof that unresolved targetless calls have been recovered. | | Assembly and pseudocode | Assembly is bounded Ghidra Listing text; pseudocode is bounded Ghidra decompiler output. Neither is original source, and cross-provider comparison never treats Hopper and Ghidra text as equal or unequal semantic facts. | +| Instruction fast path | `read_function_instructions` returns one offset-paginated raw Listing window without invoking the decompiler or whole-program name/string inventories. It reports scan work, exact-or-unknown totals, truncation, and continuation metadata. | | Calls and references | Callers/callees contain only resolved functions. Reference edges preserve exact ReferenceManager type and call/jump/data/read/write/indirect/computed/conditional/terminal/external facts. Targetless computed flow remains unknown. Synthetic entry-point references without actionable memory sources are omitted explicitly. | | CFG | Dossiers use `BasicBlockModel` and retain only non-call successors inside the function body. CFG topology is address-normalized for comparison; provider-specific block construction remains a declared difference. | | Bounds | Direct assembly fails above 100,000 instructions or the wire ceiling. Dossiers scan at most 5,000 requested instructions, return bounded independent pages, mark unknown totals after an incomplete scan, and paginate pseudocode by Unicode code points. | `npm run verify:ghidra` compiles the versioned C oracles into x86-64 debug and stripped ELF, AArch64 ELF, x86-64 PE, and x86-64 Mach-O targets. It proves all -18 operations, external functions, resolved thunks, exports, stripped-name +19 operations, external functions, resolved thunks, exports, stripped-name behavior, direct and targetless indirect calls, typed references, strings/xrefs, multi-block CFG, semantic enhanced workflows, cancellation, deadlines, serialized concurrency, malformed-target rejection, profile identity, and @@ -118,7 +119,7 @@ cover analysis/decompile timeouts, process exit, queue saturation, and malformed wire output. `npm run verify:ghidra:windows` generates a deterministic source-owned native -x86-64 PE application, proves all 18 operations through the production Windows +x86-64 PE application, proves all 19 operations through the production Windows launcher and loopback transport, checks target/snapshot/import SHA-256 linkage, and requires project, endpoint, process, and runtime cleanup. Passing that lane proves only the documented P0 boundary, not the remaining Windows security @@ -146,7 +147,7 @@ only the generic process mechanisms. | Provider | License / automation surface | What it brings | REA fit and blockers | | ------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [Ghidra](https://github.com/NationalSecurityAgency/ghidra) | Apache-2.0 source license; `analyzeHeadless`, Java APIs, and PyGhidra | Broad static analysis, many processors and formats, scripting, and project/database workflows | Read-only analysis shipped: exact BYO checks, isolated state, bounded process and queue, packaged bridge, 18 admitted inventory/function operations, and real ELF/PE/Mach-O x86-64 plus AArch64 conformance. GUI and mutation semantics remain intentionally unavailable. | +| [Ghidra](https://github.com/NationalSecurityAgency/ghidra) | Apache-2.0 source license; `analyzeHeadless`, Java APIs, and PyGhidra | Broad static analysis, many processors and formats, scripting, and project/database workflows | Read-only analysis shipped: exact BYO checks, isolated state, bounded process and queue, packaged bridge, 19 admitted inventory/function operations, and real ELF/PE/Mach-O x86-64 plus AArch64 conformance. GUI and mutation semantics remain intentionally unavailable. | | [Rizin](https://github.com/rizinorg/rizin) / [rz-pipe](https://github.com/rizinorg/rz-pipe) | Rizin repository contains LGPL-3.0 and GPL-3.0 components; `rizin`, `rz-bin`, and language bridges through `rzpipe` | Portable CLI analysis, disassembly/debugging, many architectures and file formats, JSON command output | Good candidate for a process-backed Linux provider and fast metadata fallback. License/component inventory must be preserved; command output needs version-pinned parsers and semantic conformance before evidence is trusted. | | [LIEF](https://github.com/lief-project/LIEF) | Apache-2.0; C++, Python, and other bindings | Deterministic parsing and modification of ELF, PE, Mach-O, COFF, and related executable formats; headers, sections, symbols, relocations, and functions | Best near-term complement, not a decompiler replacement. It can cover format metadata and artifact evidence without a long-lived analysis process; function semantics, pseudocode, CFG, and cross-reference parity remain out of scope unless separately demonstrated. | | [Binary Ninja](https://docs.binary.ninja/dev/index.html) | API/documentation components are MIT, while the analysis product is licensed by edition; commercial, Ultimate, or Headless license is required for headless automation | Python/Core/C++/Rust APIs, headless loading, IL layers, function analysis, plugins, and configurable analysis | Strong technical fit for a native provider, especially function dossiers. Commercial licensing, license-secret handling, native runtime packaging, and multithreaded lifecycle rules are material deployment blockers. | diff --git a/docs/roadmap.md b/docs/roadmap.md index e519f21d4..a68507ebc 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -11,11 +11,11 @@ additively; Devin is reported but left unchanged. The Ghidra foundation supports Linux x64 with exact Ghidra 12.1.2 and a 64-bit full JDK 21. An experimental Windows x64 P0 admits approved native x86-64 PE -applications through the same 18-operation Java bridge. Doctor validates those coordinates; approved Linux/macOS setup propagates them +applications through the same 19-operation Java bridge. Doctor validates those coordinates; approved Linux/macOS setup propagates them to MCP registrations without installing or modifying either dependency. REA's packaged Java bridge then proves an isolated read-only headless import, post-analysis handshake, complete cleanup, ten admitted read-only inventory -operations, and eight function-analysis operations. Real source-owned x86-64 +operations, and nine function-analysis operations. Real source-owned x86-64 debug/stripped ELF, AArch64 ELF, PE, and Mach-O fixtures cover program, procedure, string, symbol, external/thunk, memory, resolution, search, decompilation, assembly, calls, typed references, xrefs, and CFG semantics. diff --git a/docs/verification/managed-conformance-manifest.json b/docs/verification/managed-conformance-manifest.json index c423bcc63..83468ca49 100644 --- a/docs/verification/managed-conformance-manifest.json +++ b/docs/verification/managed-conformance-manifest.json @@ -1,6 +1,6 @@ { "schema_version": 1, - "manifest_id": "ecm_e2ed4ec5ff3bf3b3c25678a0c1e69971eb03d0fe6bbaebb1bf26a13b0151b17b", + "manifest_id": "ecm_0b704bd3b1a8e8f6840d113cf064de935393db1936ca5eba1018a49ccd566cf2", "verifier": { "id": "rea-managed-conformance", "version": "1" @@ -14,7 +14,7 @@ "skill_digests": [ { "skill_id": "reverse-engineer-anything", - "sha256": "f330e6b677d1d68cdf369d852e61c390eb3369503fead732be560e2a1f0fbdc7" + "sha256": "fdfbbc813a935b93dbd814c509997024da4a98500b7d12887b0ee6ecf69a9fcb" } ], "claims": [ diff --git a/docs/windows-ghidra-p0.md b/docs/windows-ghidra-p0.md index 9d777eaef..b93537e68 100644 --- a/docs/windows-ghidra-p0.md +++ b/docs/windows-ghidra-p0.md @@ -13,7 +13,7 @@ The P0 accepts exactly: - an operator-installed official Ghidra 12.1.2 distribution; - a 64-bit full JDK 21; - an explicit native, non-managed, non-DLL x86-64 PE application; and -- the existing 18 read-only Ghidra inventory and function-analysis operations. +- the existing 19 read-only Ghidra inventory and function-analysis operations. Hopper, Ghidra GUI state, mutation, controlled JavaScript replay, process capture, browser/Electron observation, artifact extraction, and general @@ -146,7 +146,7 @@ npm run verify:ghidra:windows The verifier creates an ignored deterministic x86-64 PE fixture, checks its fixed SHA-256, opens Ghidra 12.1.2 through the production Java bridge, exercises -all 18 admitted operations, verifies the target/snapshot/import digest chain, +all 19 admitted operations, verifies the target/snapshot/import digest chain, and checks endpoint, project, process, and runtime cleanup. The GitHub workflow accepts only the fixed `real-ghidra-windows` repository-dispatch event against default-branch `main`, a protected `real-ghidra-windows` environment, and a diff --git a/scripts/lib/real-hopper-function-basics.mjs b/scripts/lib/real-hopper-function-basics.mjs new file mode 100644 index 000000000..ca2f38d3f --- /dev/null +++ b/scripts/lib/real-hopper-function-basics.mjs @@ -0,0 +1,71 @@ +import { requireMcpResult } from "./mcp-verifier-results.mjs"; + +export const verifyHopperFunctionBasics = async ( + client, + options, + procedure, +) => { + const containment = requireMcpResult( + await client.callTool( + { + name: "resolve_containing_procedure", + arguments: { address: procedure }, + }, + options, + ), + "resolve_containing_procedure", + ); + if ( + containment?.found !== true || + containment.procedure?.address !== procedure + ) { + throw new Error( + "resolve_containing_procedure returned the wrong procedure", + ); + } + + const references = requireMcpResult( + await client.callTool( + { + name: "procedure_references", + arguments: { + procedure, + direction: "outgoing", + limit: 10, + max_instructions: 100, + }, + }, + options, + ), + "procedure_references", + ); + if (!Array.isArray(references?.references?.items)) { + throw new Error("procedure_references returned an invalid bounded result"); + } + + const instructions = requireMcpResult( + await client.callTool( + { + name: "read_function_instructions", + arguments: { procedure, limit: 2 }, + }, + options, + ), + "read_function_instructions", + ); + if ( + instructions?.procedure?.address !== procedure || + !Array.isArray(instructions.instructions?.items) || + instructions.instructions.items.length === 0 || + instructions.instructions.items.length > 2 + ) { + throw new Error( + "read_function_instructions returned an invalid bounded window", + ); + } + + return { + outgoingReferenceCount: references.references.items.length, + instructionWindowCount: instructions.instructions.items.length, + }; +}; diff --git a/scripts/verify-real-ghidra-function.mjs b/scripts/verify-real-ghidra-function.mjs index fd294b990..4572e6c43 100644 --- a/scripts/verify-real-ghidra-function.mjs +++ b/scripts/verify-real-ghidra-function.mjs @@ -44,10 +44,27 @@ export async function verifyDebugFunctionOperations( if (entryString === undefined) throw new Error("Ghidra debug string function probe is unavailable"); - const [info, assembly, pseudocode, callees, callers, outgoing, entryXrefs] = - await collectEntryFunctionData(client, entry); + const [ + info, + assembly, + instructionWindow, + pseudocode, + callees, + callers, + outgoing, + entryXrefs, + ] = await collectEntryFunctionData(client, entry); assertLocalProcedureInfo(info, entry.address); assertProviderText(assembly, pseudocode, entry.address); + if ( + instructionWindow.procedure?.address !== entry.address || + instructionWindow.instructions?.items?.length !== 1 || + instructionWindow.instructions.returned !== 1 || + instructionWindow.instructions_scanned < 1 + ) + throw new Error( + "Ghidra read_function_instructions failed its bounded fast-path contract", + ); assertEntryCallGraph({ branch, indirect, main, callees, callers }); const { xrefOwners, xrefOwnerFailures } = await resolveXrefOwners( client, @@ -83,6 +100,10 @@ export async function verifyDebugFunctionOperations( entry: dossierSummary(entryDossier), branch: dossierSummary(branchDossier), indirect: dossierSummary(indirectDossier), + instruction_window: { + returned: instructionWindow.instructions.returned, + truncated: instructionWindow.instructions.truncated, + }, cancellation, timeout, concurrency: concurrent, @@ -99,6 +120,12 @@ async function collectEntryFunctionData(client, entry) { document: null, procedure: entry.value, }), + functionCall(client, "read_function_instructions", { + document: null, + procedure: entry.value, + offset: 0, + limit: 1, + }), functionCall(client, "procedure_pseudo_code", { document: null, procedure: entry.value, diff --git a/scripts/verify-real-ghidra-windows.mjs b/scripts/verify-real-ghidra-windows.mjs index cd95bfa4a..de8b03960 100644 --- a/scripts/verify-real-ghidra-windows.mjs +++ b/scripts/verify-real-ghidra-windows.mjs @@ -146,6 +146,7 @@ try { "procedure_callers", "procedure_info", "procedure_pseudo_code", + "read_function_instructions", "procedure_references", ]) await functionOperation(operation, { procedure: procedure.value }); diff --git a/scripts/verify-real-hopper.mjs b/scripts/verify-real-hopper.mjs index 5f9c40216..34de6507c 100644 --- a/scripts/verify-real-hopper.mjs +++ b/scripts/verify-real-hopper.mjs @@ -26,6 +26,7 @@ import { requireMcpResult, requireWorkflowEvidenceProvider, } from "./lib/mcp-verifier-results.mjs"; +import { verifyHopperFunctionBasics } from "./lib/real-hopper-function-basics.mjs"; import { openAndVerifyLargeFixture } from "./lib/real-hopper-pagination.mjs"; import { completeVerifierRun, createVerifierRun } from "./lib/verifier-run.mjs"; import { startUnrelatedHopperSentinel } from "./lib/unrelated-hopper-sentinel.mjs"; @@ -155,42 +156,11 @@ const verifyCurrentTarget = async (client, options) => { "list_procedures", ); const firstAddress = firstProcedureAddress(procedures); - const containment = requireSuccessfulTool( - await client.callTool( - { - name: "resolve_containing_procedure", - arguments: { address: firstAddress }, - }, - options, - ), - "resolve_containing_procedure", + const basics = await verifyHopperFunctionBasics( + client, + options, + firstAddress, ); - if ( - containment?.found !== true || - containment.procedure?.address !== firstAddress - ) { - throw new Error( - "resolve_containing_procedure returned the wrong procedure", - ); - } - const references = requireSuccessfulTool( - await client.callTool( - { - name: "procedure_references", - arguments: { - procedure: firstAddress, - direction: "outgoing", - limit: 10, - max_instructions: 100, - }, - }, - options, - ), - "procedure_references", - ); - if (!Array.isArray(references?.references?.items)) { - throw new Error("procedure_references returned an invalid bounded result"); - } const relationships = await verifyRelationships( client, options, @@ -245,7 +215,8 @@ const verifyCurrentTarget = async (client, options) => { callerCount: relationships.procedure_callers.length, calleeCount: relationships.procedure_callees.length, xrefCount: relationships.xrefs.length, - outgoingReferenceCount: references.references.items.length, + outgoingReferenceCount: basics.outgoingReferenceCount, + instructionWindowCount: basics.instructionWindowCount, }; }; diff --git a/skills/reverse-engineer-anything/SKILL.md b/skills/reverse-engineer-anything/SKILL.md index 92573911d..ffb62f967 100644 --- a/skills/reverse-engineer-anything/SKILL.md +++ b/skills/reverse-engineer-anything/SKILL.md @@ -3,8 +3,8 @@ name: reverse-engineer-anything description: Reverse engineer native, managed, Electron/JavaScript, packaged, and browser applications with REA. Use shipped-artifact or approved runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. metadata: version: "23" - tool_count: 98 - catalog_digest: "aa413204625f3aeb62732d0f732b9b8b24832eea4520214b1e6bd204e599bcd6" + tool_count: 99 + catalog_digest: "e787c57af8337f17b9da72745fcb328634af3b39be1d9ca8bfe1836274351f92" --- # REA diff --git a/src/application/AnalysisClientCleanup.ts b/src/application/AnalysisClientCleanup.ts new file mode 100644 index 000000000..de5f06a49 --- /dev/null +++ b/src/application/AnalysisClientCleanup.ts @@ -0,0 +1,27 @@ +import { ProviderCleanupError } from "../domain/providerCleanupError.js"; +import { err, ok, type Result } from "../domain/result.js"; +import type { AnalysisError } from "../domain/errors.js"; +import type { AnalysisClient, ExecutionOptions } from "./AnalysisProvider.js"; + +/** Close one provider client and normalize unexpected adapter rejection. */ +export const closeAnalysisClient = async ( + client: AnalysisClient, + providerId: string, + options: Pick = {}, +): Promise> => { + try { + if (client.closeWithOutcome !== undefined) + return await client.closeWithOutcome(options); + await client.close(); + return ok(null); + } catch (cause: unknown) { + return err( + new ProviderCleanupError( + providerId, + ["provider-client"], + { reason: "provider client close rejected unexpectedly" }, + { cause }, + ), + ); + } +}; diff --git a/src/application/AnalysisProvider.ts b/src/application/AnalysisProvider.ts index 3f3ac6328..6ef2b1265 100644 --- a/src/application/AnalysisProvider.ts +++ b/src/application/AnalysisProvider.ts @@ -81,10 +81,29 @@ export interface ProviderRuntimeLineageSnapshot { readonly observation: ProcessLineageObservation; } +/** One provider's current serial-request activity, including detached callers. */ +export interface ProviderRequestActivitySnapshot { + readonly provider: ProviderIdentity; + readonly active: { + readonly requestId: number; + readonly operation: string; + readonly elapsedMs: number; + readonly timeoutMs: number; + readonly callerState: "waiting" | "timed_out" | "cancelled"; + } | null; + readonly queuedRequests: number; +} + /** Provider session bound to exactly one parsed artifact. */ export interface AnalysisClient extends AnalysisOperationPort { /** Retained token-verified process snapshots for dynamic providers this client started. */ runtimeLineageSnapshots?(): readonly ProviderRuntimeLineageSnapshot[]; + /** Current provider work that may outlive a timed-out or cancelled caller. */ + requestActivitySnapshots?(): readonly ProviderRequestActivitySnapshot[]; + /** Close with a typed result when the provider can verify cleanup. */ + closeWithOutcome?( + options?: Pick, + ): Promise>; close(): Promise; } diff --git a/src/application/BinarySession.ts b/src/application/BinarySession.ts index 8aa123326..0f2751b77 100644 --- a/src/application/BinarySession.ts +++ b/src/application/BinarySession.ts @@ -16,6 +16,7 @@ import type { AnalysisClient, AnalysisExecution, AnalysisClientFactory, + ExecutionOptions, AnalysisProfileResolution, AnalysisProfileResolutionOptions, AnalysisOperationPort, @@ -37,6 +38,7 @@ import { type BinarySessionOpenOptions, } from "./BinarySessionOpen.js"; import { prepareSessionExecution } from "./BinarySessionExecution.js"; +import { closeAnalysisClient } from "./AnalysisClientCleanup.js"; export type { BinarySessionPort } from "./BinarySessionPort.js"; const OFFICIAL_OPERATIONS: ReadonlySet = new Set( OFFICIAL_TOOL_CONTRACTS.map(({ name }) => name), @@ -186,7 +188,17 @@ export class BinarySession await this.#drainCalls(); const previous = this.#active; this.#active = undefined; - await previous?.client.close(); + if (previous !== undefined) { + const closed = await closeAnalysisClient( + previous.client, + previous.route.identity.id, + ); + if (!closed.ok) { + this.clearSessionRecords(); + this.#clearRuntimeAvailability(); + return closed; + } + } const runId = randomUUID(); const client = route.createClient(target, { runId }); const started = await client.execute("health", {}, options); @@ -226,15 +238,24 @@ export class BinarySession } /** Close the active target, if any. */ - close(): Promise> { + close( + options: Pick = {}, + ): Promise> { return this.#serialize(async () => { const previous = this.#active; this.#active = undefined; await this.#drainCalls(); - await previous?.client.close(); + const closed = + previous === undefined + ? ok(null) + : await closeAnalysisClient( + previous.client, + previous.route.identity.id, + options, + ); this.clearSessionRecords(); this.#clearRuntimeAvailability(); - return ok(null); + return closed; }); } @@ -248,6 +269,8 @@ export class BinarySession runId: this.#active?.runId, runtimeLineageSnapshots: this.#active?.client.runtimeLineageSnapshots?.() ?? [], + requestActivitySnapshots: + this.#active?.client.requestActivitySnapshots?.() ?? [], }); } diff --git a/src/application/BinarySessionPort.ts b/src/application/BinarySessionPort.ts index e24c6a6f5..66d391aae 100644 --- a/src/application/BinarySessionPort.ts +++ b/src/application/BinarySessionPort.ts @@ -23,6 +23,7 @@ import type { ReconstructionCoverageWorkspace } from "../domain/reconstructionCo import type { AnalysisOperation, AnalysisOperationPort, + ExecutionOptions, ProviderIdentity, } from "./AnalysisProvider.js"; @@ -37,7 +38,9 @@ export interface BinarySessionPort extends AnalysisOperationPort { readonly providerId?: AnalysisProviderSelector; }, ): Promise>; - close(): Promise>; + close( + options?: Pick, + ): Promise>; status(): JsonValue; activeTarget(): BinaryTarget | undefined; recordEvidence( diff --git a/src/application/BinarySessionStatus.ts b/src/application/BinarySessionStatus.ts index b386e6818..07ac5f390 100644 --- a/src/application/BinarySessionStatus.ts +++ b/src/application/BinarySessionStatus.ts @@ -2,6 +2,7 @@ import type { BinaryTarget } from "../domain/binaryTarget.js"; import type { JsonValue } from "../domain/jsonValue.js"; import type { CapabilityDescriptor, + ProviderRequestActivitySnapshot, ProviderRuntimeLineageSnapshot, } from "./AnalysisProvider.js"; import type { @@ -19,6 +20,7 @@ interface BinarySessionStatusInput { >; readonly runId: string | undefined; readonly runtimeLineageSnapshots: readonly ProviderRuntimeLineageSnapshot[]; + readonly requestActivitySnapshots: readonly ProviderRequestActivitySnapshot[]; } /** Project internal provider routing state into the caller-visible session status. */ @@ -29,6 +31,7 @@ export const binarySessionStatus = ({ runtimeAvailability, runId, runtimeLineageSnapshots, + requestActivitySnapshots, }: BinarySessionStatusInput): JsonValue => { const configuredProvider = router.configuredIdentity(); const provider = providerSummary(configuredProvider); @@ -73,6 +76,7 @@ export const binarySessionStatus = ({ run_id: runId, process_lineage: processLineageStatus(runtimeLineageSnapshots), }, + analysis_activity: requestActivityStatus(requestActivitySnapshots), }; return target === undefined ? { open: false, ...common } @@ -87,6 +91,42 @@ export const binarySessionStatus = ({ }; }; +const requestActivityStatus = ( + snapshots: readonly ProviderRequestActivitySnapshot[], +) => { + const providers = snapshots + .slice() + .sort((left, right) => left.provider.id.localeCompare(right.provider.id)) + .map(({ provider, active, queuedRequests }) => ({ + provider: providerSummary(provider), + active: + active === null + ? null + : { + request_id: active.requestId, + operation: active.operation, + elapsed_ms: active.elapsedMs, + timeout_ms: active.timeoutMs, + caller_state: active.callerState, + }, + queued_requests: queuedRequests, + })); + const active = snapshots.flatMap((snapshot) => + snapshot.active === null ? [] : [snapshot.active], + ); + return { + status: + snapshots.length === 0 + ? "not_observed" + : active.some(({ callerState }) => callerState === "timed_out") + ? "timed_out_busy" + : active.length > 0 + ? "busy" + : "idle", + providers, + }; +}; + const providerSummary = ({ id, name, diff --git a/src/application/CompositeProvider.ts b/src/application/CompositeProvider.ts index 9b88b9a1f..28db5671b 100644 --- a/src/application/CompositeProvider.ts +++ b/src/application/CompositeProvider.ts @@ -10,6 +10,7 @@ import type { CapabilityDescriptor, ProviderIdentity, } from "./AnalysisProvider.js"; +import { closeAnalysisClient } from "./AnalysisClientCleanup.js"; /** Synthetic compatibility identity for a deterministic provider set. */ export const compositeProviderIdentity = ( @@ -103,6 +104,16 @@ export class CompositeProvider implements AnalysisProvider { clients.set(provider, created); return created; }; + const closeWithOutcome: NonNullable< + AnalysisClient["closeWithOutcome"] + > = async (options) => { + const outcomes = await Promise.all( + [...clients.entries()].map(([provider, client]) => + closeAnalysisClient(client, provider.identity().id, options), + ), + ); + return outcomes.find((outcome) => !outcome.ok) ?? ok(null); + }; return { execute: (operation, parameters, options) => { if (operation === "health") @@ -135,10 +146,15 @@ export class CompositeProvider implements AnalysisProvider { .sort((left, right) => left.provider.id.localeCompare(right.provider.id), ), + requestActivitySnapshots: () => + [...clients.values()] + .flatMap((client) => client.requestActivitySnapshots?.() ?? []) + .sort((left, right) => + left.provider.id.localeCompare(right.provider.id), + ), + closeWithOutcome, close: async () => { - await Promise.allSettled( - [...clients.values()].map(async (client) => client.close()), - ); + await closeWithOutcome(); }, }; } diff --git a/src/application/DirectAnalysis.ts b/src/application/DirectAnalysis.ts index 808b5e3f4..b3d65d031 100644 --- a/src/application/DirectAnalysis.ts +++ b/src/application/DirectAnalysis.ts @@ -49,6 +49,7 @@ export { type DirectAnalysisTool = | "binary_overview" | "procedure_pseudo_code" + | "read_function_instructions" | "analyze_function" | "search_strings" | "search_procedures" diff --git a/src/application/EnhancedTools.ts b/src/application/EnhancedTools.ts index 62973b2f6..12c4539ff 100644 --- a/src/application/EnhancedTools.ts +++ b/src/application/EnhancedTools.ts @@ -309,22 +309,24 @@ export class EnhancedTools { input: { detail: "concise" | "detailed"; limit: number }, signal?: AbortSignal, ): EnhancedResult { - const [segmentsResult, documentsResult, procedures, stringsResult] = + const [segmentsResult, documentsResult, proceduresResult, stringsResult] = await Promise.all([ this.#call("list_segments", {}, signal), this.#call("list_documents", {}, signal), - this.#allAddressed("list_procedures", signal), - this.#call("list_strings", {}, signal), + this.#call("list_procedures", { offset: 0, limit: 1 }, signal), + this.#call("list_strings", { offset: 0, limit: 1 }, signal), ]); if (!segmentsResult.ok) return segmentsResult; if (!documentsResult.ok) return documentsResult; - if (!procedures.ok) return procedures; + if (!proceduresResult.ok) return proceduresResult; if (!stringsResult.ok) return stringsResult; const segments = parseSegments(segmentsResult.value); if (!segments.ok) return segments; const documents = parseDocuments(documentsResult.value); if (!documents.ok) return documents; + const procedureCount = parseListCount(proceduresResult.value, "procedures"); + if (!procedureCount.ok) return procedureCount; const stringCount = parseListCount(stringsResult.value, "strings"); if (!stringCount.ok) return stringCount; @@ -339,7 +341,7 @@ export class EnhancedTools { : { name, start, end }, ), segment_count: segments.value.length, - procedure_count: procedures.value.length, + procedure_count: procedureCount.value, string_count: stringCount.value, }); } diff --git a/src/cli/coreAnalysisCommands.ts b/src/cli/coreAnalysisCommands.ts index b7a14e3c4..d6a0faeec 100644 --- a/src/cli/coreAnalysisCommands.ts +++ b/src/cli/coreAnalysisCommands.ts @@ -16,6 +16,7 @@ export const registerCoreAnalysisCommands = ( ): void => { registerCoreCommands(cli, logger); registerFunctionCommand(cli, logger); + registerInstructionsCommand(cli, logger); registerSearchCommand(cli, logger); registerXrefsCommand(cli, logger); registerTraceCommand(cli, logger); @@ -285,6 +286,54 @@ const registerFunctionCommand = (cli: CliInstance, logger: Logger): void => { }); }; +const registerInstructionsCommand = ( + cli: CliInstance, + logger: Logger, +): void => { + cli.command(CLI_COMMANDS.instructions, { + description: "Read one bounded raw-instruction window without decompiling", + args: z.object({ + path: z.string().describe("App or program path"), + address: z.string().describe("Procedure name or address"), + }), + options: z.object({ + offset: z + .number() + .int() + .min(0) + .max(100_000) + .default(0) + .describe("Zero-based instruction offset"), + limit: z + .number() + .int() + .min(1) + .max(500) + .default(64) + .describe("Maximum raw instructions to return"), + snapshot: z + .string() + .min(1) + .optional() + .describe("Load and update a local analysis snapshot"), + provider: providerSelectionOption, + }), + run: ({ args, options }) => + logCliCommand(logger, CLI_COMMANDS.instructions, () => + runDirectAnalysis( + args.path, + "read_function_instructions", + { + procedure: args.address, + offset: options.offset, + limit: options.limit, + }, + directAnalysisOptions(logger, options.snapshot, options.provider), + ), + ), + }); +}; + const registerSearchCommand = (cli: CliInstance, logger: Logger): void => { cli.command(CLI_COMMANDS.search, { description: "Search bounded analyzed strings or procedure names", diff --git a/src/cliCommandNames.ts b/src/cliCommandNames.ts index fca7d94bc..1f776ab79 100644 --- a/src/cliCommandNames.ts +++ b/src/cliCommandNames.ts @@ -12,6 +12,7 @@ export const CLI_COMMANDS = Object.freeze({ capabilities: "capabilities", providers: "providers", function: "function", + instructions: "instructions", search: "search", importReferenceSource: "import-reference-source", inventoryArtifact: "inventory-artifact", diff --git a/src/contracts/functionInstructionContract.ts b/src/contracts/functionInstructionContract.ts new file mode 100644 index 000000000..0e38d3709 --- /dev/null +++ b/src/contracts/functionInstructionContract.ts @@ -0,0 +1,9 @@ +import { z } from "zod"; + +/** Provider-neutral input for one bounded raw function-instruction window. */ +export const functionInstructionInputSchema = z.object({ + procedure: z.string().describe("The procedure name or address"), + offset: z.number().int().min(0).max(100_000).default(0), + limit: z.number().int().min(1).max(500).default(64), + document: z.string().optional().describe("The document name"), +}); diff --git a/src/contracts/toolContractExamples.ts b/src/contracts/toolContractExamples.ts index f2b388273..c8ccd642c 100644 --- a/src/contracts/toolContractExamples.ts +++ b/src/contracts/toolContractExamples.ts @@ -17,6 +17,7 @@ export const TOOL_EXAMPLE_OVERRIDES: Readonly< procedure_callees: { procedure: "main" }, procedure_callers: { procedure: "main" }, procedure_info: { procedure: "main" }, + read_function_instructions: { procedure: "main" }, procedure_references: { procedure: "main" }, procedure_pseudo_code: { procedure: "main" }, resolve_containing_procedure: { address: "0x1000" }, diff --git a/src/contracts/toolContracts.ts b/src/contracts/toolContracts.ts index 90a7300ac..02843d0df 100644 --- a/src/contracts/toolContracts.ts +++ b/src/contracts/toolContracts.ts @@ -37,6 +37,7 @@ import { import type { ToolContract, ToolExample } from "./toolContractTypes.js"; import { toolContractMetadata } from "./toolEffects.js"; import { binarySessionInputSchema } from "./sessionStatusContract.js"; +import { functionInstructionInputSchema } from "./functionInstructionContract.js"; export { binarySessionInputSchema } from "./sessionStatusContract.js"; @@ -240,6 +241,11 @@ export const OFFICIAL_TOOL_CONTRACTS = [ "Return bounded metadata for one procedure identified by symbol or address: entrypoint, signature, locals, size, and block count. Follow with decompilation or assembly for behavior.", z.object({ procedure, document }), ), + official( + "read_function_instructions", + "Read one offset-paginated window of raw instructions for an analyzed procedure without decompilation, caller discovery, or whole-program string/name scans. Use this fast path for instruction-level orientation; follow next_offset when truncated.", + functionInstructionInputSchema, + ), official( "procedure_references", "Return a bounded set of raw incoming or outgoing reference edges for one procedure. Endpoint procedures are resolved only from provider containment; Ghidra preserves observed reference kinds while providers without kind authority mark them unavailable.", diff --git a/src/contracts/toolEffects.ts b/src/contracts/toolEffects.ts index d9bca24be..d4c31330f 100644 --- a/src/contracts/toolEffects.ts +++ b/src/contracts/toolEffects.ts @@ -60,6 +60,7 @@ export const TOOL_EFFECTS: Readonly> = { procedure_callees: evidence, procedure_callers: evidence, procedure_info: evidence, + read_function_instructions: evidence, procedure_references: evidence, procedure_pseudo_code: evidence, resolve_containing_procedure: evidence, diff --git a/src/contracts/toolOutputSchemaGroups.ts b/src/contracts/toolOutputSchemaGroups.ts index 00c081e43..5c405fd2d 100644 --- a/src/contracts/toolOutputSchemaGroups.ts +++ b/src/contracts/toolOutputSchemaGroups.ts @@ -6,7 +6,10 @@ import { } from "../domain/processCapture.js"; import { evidenceBundleSchema } from "../domain/evidenceBundle.js"; import { residualUnknownSchema } from "../domain/residualUnknown.js"; -import { referenceKindSchema } from "../domain/hopperValues.js"; +import { + functionInstructionWindowSchema, + referenceKindSchema, +} from "../domain/hopperValues.js"; import { demangleSwiftSchema, inspectMachoSchema, @@ -39,6 +42,7 @@ import { replayMachineRunOutputSchema } from "../domain/replayMachineRun.js"; import { analysisErrorProjectionSchema } from "./errorSchemas.js"; import { addressList, + analysisActivity, addressedEntry, bounded, containingProcedureResolution, @@ -82,6 +86,7 @@ export const officialOutputSchemas: Readonly> = { procedure_callees: resultOf(addressList), procedure_callers: resultOf(addressList), procedure_info: procedureInfoOutput, + read_function_instructions: resultOf(functionInstructionWindowSchema), procedure_references: resultOf( z.object({ procedure: procedureIdentity, @@ -297,6 +302,7 @@ export const sessionOutputSchemas: Readonly> = { reasons: z.array(z.string()), remediation: z.string().nullable(), }), + analysis_activity: analysisActivity, recommended_actions: z.array(z.string()), }), z.object({ diff --git a/src/contracts/toolOutputSchemaPrimitives.ts b/src/contracts/toolOutputSchemaPrimitives.ts index 5a0d30f72..a1165c6ae 100644 --- a/src/contracts/toolOutputSchemaPrimitives.ts +++ b/src/contracts/toolOutputSchemaPrimitives.ts @@ -140,6 +140,25 @@ const analysisRun = z }) .nullable(); +export const analysisActivity = z.object({ + status: z.enum(["not_observed", "idle", "busy", "timed_out_busy"]), + providers: z.array( + z.object({ + provider: providerIdentity, + active: z + .object({ + request_id: z.number().int().min(1), + operation: z.string().min(1), + elapsed_ms: z.number().int().min(0), + timeout_ms: z.number().int().min(0), + caller_state: z.enum(["waiting", "timed_out", "cancelled"]), + }) + .nullable(), + queued_requests: z.number().int().min(0), + }), + ), +}); + const providerRejectionCode: z.ZodType = z.enum( PROVIDER_REJECTION_CODES, ); @@ -225,6 +244,7 @@ export const sessionProvider = z.object({ providers: z.array(providerIdentity), capabilities: z.array(providerCapability), analysis_run: analysisRun, + analysis_activity: analysisActivity, analysis_provider_binding: z .object({ provider: providerIdentity, diff --git a/src/domain/analysisErrorProjection.ts b/src/domain/analysisErrorProjection.ts index 9fa0542f4..a84cb6a8f 100644 --- a/src/domain/analysisErrorProjection.ts +++ b/src/domain/analysisErrorProjection.ts @@ -60,6 +60,7 @@ export const projectAnalysisError = ( }; const errorCode = (error: AnalysisError): AnalysisErrorProjection["code"] => { + if (error.cleanupIncomplete) return "cleanup_incomplete"; if (error instanceof ReplayPlanStaleError) return "plan_stale"; if (error instanceof PermissionRequiredError) return "permission_required"; if (error instanceof ProviderSelectionError) @@ -295,6 +296,12 @@ const providerErrorDetails = ( return { provider_id: error.providerId, operation: error.operation, + ...(error.cleanupIncomplete + ? { + cleanup: "incomplete", + resources: [...error.cleanupResources], + } + : {}), ...(error.diagnostics === undefined ? {} : { diagnostics: error.diagnostics }), diff --git a/src/domain/hopperValues.ts b/src/domain/hopperValues.ts index 6ac088ec2..7328f4115 100644 --- a/src/domain/hopperValues.ts +++ b/src/domain/hopperValues.ts @@ -143,6 +143,18 @@ const boundedSchema = (item: T) => }); } }); + +/** Provider-neutral bounded raw-instruction window for one analyzed function. */ +export const functionInstructionWindowSchema = z + .object({ + procedure: procedureIdentitySchema, + instructions: boundedSchema(z.string()), + instructions_scanned: z.number().int().min(0), + instruction_scan_truncated: z.boolean(), + limitations: z.array(z.string()), + }) + .strict(); + const availableReferenceKindSchema = z .object({ available: z.literal(true), diff --git a/src/domain/providerCleanupError.ts b/src/domain/providerCleanupError.ts new file mode 100644 index 000000000..71e51d698 --- /dev/null +++ b/src/domain/providerCleanupError.ts @@ -0,0 +1,20 @@ +import { ProviderAdapterError } from "./errors.js"; +import type { JsonValue } from "./jsonValue.js"; + +/** Provider resources could not be proven closed after bounded cleanup. */ +export class ProviderCleanupError extends ProviderAdapterError { + override readonly cleanupIncomplete = true; + override readonly cleanupResources: readonly string[]; + override readonly userMessage = + "Provider cleanup could not be fully confirmed. Review the reported local resources before opening another target."; + + constructor( + providerId: string, + resources: readonly string[], + diagnostics: Readonly>, + options?: ErrorOptions, + ) { + super(providerId, "close_binary", { ...options, diagnostics }); + this.cleanupResources = [...resources]; + } +} diff --git a/src/ghidra/GhidraDefaults.ts b/src/ghidra/GhidraDefaults.ts index ea98b3b24..197877f34 100644 --- a/src/ghidra/GhidraDefaults.ts +++ b/src/ghidra/GhidraDefaults.ts @@ -17,4 +17,4 @@ export const GHIDRA_MAX_QUEUED_REQUESTS = 32; /** Maximum encoded response line accepted from the Java bridge. */ export const GHIDRA_MAX_LINE_BYTES = 1024 * 1024; /** Version of REA's Java bridge handshake and request protocol. */ -export const GHIDRA_BRIDGE_VERSION = 4; +export const GHIDRA_BRIDGE_VERSION = 5; diff --git a/src/ghidra/GhidraFunctionValues.ts b/src/ghidra/GhidraFunctionValues.ts index 3a0114718..b8d42f068 100644 --- a/src/ghidra/GhidraFunctionValues.ts +++ b/src/ghidra/GhidraFunctionValues.ts @@ -2,6 +2,7 @@ import { z } from "zod"; import { AnalysisInputError, AnalysisOutputError } from "../domain/errors.js"; import { + functionInstructionWindowSchema, functionDossierSchema, type FunctionDossier, } from "../domain/hopperValues.js"; @@ -24,6 +25,7 @@ export const GHIDRA_FUNCTION_OPERATIONS = [ "procedure_callers", "procedure_info", "procedure_pseudo_code", + "read_function_instructions", "procedure_references", "xrefs", ] as const; @@ -88,6 +90,13 @@ const inputSchemas = { procedure_callers: z.object(directProcedure).strict(), procedure_info: z.object(directProcedure).strict(), procedure_pseudo_code: z.object(directProcedure).strict(), + read_function_instructions: z + .object({ + ...directProcedure, + offset: z.number().int().min(0).max(100_000).default(0), + limit: z.number().int().min(1).max(500).default(64), + }) + .strict(), procedure_references: z .object({ ...directProcedure, @@ -281,6 +290,9 @@ const ghidraFunctionDossier = functionDossierSchema.superRefine( }); }, ); +const ghidraFunctionInstructionWindow = functionInstructionWindowSchema + .extend({ procedure: procedureIdentity }) + .strict(); const resultSchemas = { analyze_function: ghidraFunctionDossier, @@ -289,6 +301,7 @@ const resultSchemas = { procedure_callers: z.array(ghidraCanonicalAddressSchema), procedure_info: procedureInfo, procedure_pseudo_code: z.string().nullable(), + read_function_instructions: ghidraFunctionInstructionWindow, procedure_references: procedureReferences, xrefs: z.array(ghidraCanonicalAddressSchema), } satisfies Readonly>; diff --git a/src/ghidra/GhidraProviderCapabilities.ts b/src/ghidra/GhidraProviderCapabilities.ts index ad69a7ba4..8b3b882bf 100644 --- a/src/ghidra/GhidraProviderCapabilities.ts +++ b/src/ghidra/GhidraProviderCapabilities.ts @@ -49,6 +49,7 @@ const PAGINATED_OPERATIONS: ReadonlySet = new Set([ "search_procedures", "search_strings", "procedure_references", + "read_function_instructions", "analyze_function", ]); const SEARCH_OPERATIONS: ReadonlySet = new Set([ @@ -119,6 +120,12 @@ export const limitationsFor = (operation: string): readonly string[] => { "Pseudocode is Ghidra decompiler output, not original source and not text-equivalent to Hopper output; each decompile has a 30-second native deadline.", "External functions and functions without an analyzable body return null; other decompiler failures remain explicit.", ]; + case "read_function_instructions": + return [ + ...common, + "This bounded fast path reads only the requested function instruction window and does not invoke the decompiler or whole-program string/name inventories.", + "Instruction text is Ghidra-specific and does not claim textual equivalence with Hopper output.", + ]; case "procedure_assembly": return [ ...common, diff --git a/src/hopper/HopperCleanup.ts b/src/hopper/HopperCleanup.ts new file mode 100644 index 000000000..09dac9317 --- /dev/null +++ b/src/hopper/HopperCleanup.ts @@ -0,0 +1,265 @@ +import type { Socket } from "node:net"; + +import type { ProgressReporter } from "../application/ProgressReporter.js"; +import type { AnalysisError, HopperError } from "../domain/errors.js"; +import { HopperProcessError } from "../domain/errors.js"; +import type { JsonValue } from "../domain/jsonValue.js"; +import { ProviderCleanupError } from "../domain/providerCleanupError.js"; +import { err, ok, type Result } from "../domain/result.js"; +import type { Logger } from "../logger.js"; +import type { PrivateRuntimeRoot } from "../process/PrivateRuntimeRoot.js"; +import type { ProcessCleanupResult } from "../process/ProcessOwnership.js"; +import type { ProviderProcessSupervisor } from "../process/ProviderProcess.js"; +import type { BridgeLaunch } from "./BridgeLauncher.js"; +import { + createOwnedHopperShutdownDiagnostic, + type HopperDiagnostic, + providerCleanupFailure, +} from "./HopperDiagnostics.js"; +import type { HopperRequestActivity } from "./HopperRequestQueue.js"; +import { + isHopperCleanupRequired, + isHopperShutdownAcknowledgement, +} from "./HopperSessionValues.js"; + +interface CleanupIssue { + readonly resource: string; + readonly reason: string; +} + +interface CleanupState { + readonly issues: CleanupIssue[]; + readonly resources: Set; + cleanupResult: ProcessCleanupResult | undefined; + shutdownConfirmed: boolean; + ownedProcessStopped: boolean; +} + +export interface HopperCleanupInput { + readonly socket: Socket | undefined; + readonly launch: BridgeLaunch | undefined; + readonly processSupervisor: ProviderProcessSupervisor | undefined; + readonly runtimeRoot: PrivateRuntimeRoot | undefined; + readonly activeRequest: HopperRequestActivity | null; + readonly progress: ProgressReporter | undefined; + readonly logger: Logger; + readonly onDiagnostic: ((event: HopperDiagnostic) => void) | undefined; + request( + method: "shutdown" | "shutdown_document", + ): Promise>; + releaseTransport(socket: Socket | undefined): void; +} + +/** Close Hopper phases and retain every resource whose cleanup is unverified. */ +export const cleanupHopperSession = async ( + input: HopperCleanupInput, +): Promise> => { + const state: CleanupState = { + issues: [], + resources: new Set(), + cleanupResult: undefined, + shutdownConfirmed: false, + ownedProcessStopped: false, + }; + await report(input.progress, 0, "requesting Hopper document shutdown"); + await requestShutdown(input, state); + await report(input.progress, 0.35, "releasing Hopper bridge transport"); + input.releaseTransport(input.socket); + await stopProcess(input, state); + recordUnconfirmedDocument(input, state); + await report(input.progress, 0.75, "removing Hopper private runtime files"); + await closeRuntimeRoot(input.runtimeRoot, state); + return cleanupOutcome(input, state); +}; + +const requestShutdown = async ( + input: HopperCleanupInput, + state: CleanupState, +): Promise => { + if ( + input.socket === undefined || + input.socket.destroyed || + input.activeRequest !== null + ) { + if (input.activeRequest !== null) + input.logger.warn( + { + method: input.activeRequest.operation, + callerState: input.activeRequest.callerState, + }, + "Hopper shutdown skipped because a bridge operation remained active", + ); + return; + } + const shutdown = await input + .request("shutdown") + .catch(() => err(new HopperProcessError(null))); + if ( + shutdown.ok && + isHopperCleanupRequired(shutdown.value) && + input.launch?.shutdownByCleanup === true && + input.launch.cleanup !== undefined + ) { + state.cleanupResult = await input.launch + .cleanup() + .catch(providerCleanupFailure); + if (!state.cleanupResult.cleaned) { + state.shutdownConfirmed = await fallbackDocumentShutdown(input); + state.cleanupResult = await input.launch + .cleanup() + .catch(providerCleanupFailure); + } + } else if (shutdown.ok) { + state.shutdownConfirmed = isHopperShutdownAcknowledgement(shutdown.value); + } + if ( + !shutdown.ok || + (!state.shutdownConfirmed && state.cleanupResult === undefined) + ) + input.logger.warn( + { status: shutdown.ok ? "invalid-acknowledgement" : "failed" }, + "Hopper document shutdown was not confirmed", + ); +}; + +const fallbackDocumentShutdown = async ( + input: HopperCleanupInput, +): Promise => { + const fallback = await input + .request("shutdown_document") + .catch(() => err(new HopperProcessError(null))); + const confirmed = + fallback.ok && isHopperShutdownAcknowledgement(fallback.value); + if (!confirmed) + input.logger.warn( + { status: fallback.ok ? "invalid-acknowledgement" : "failed" }, + "Hopper document shutdown fallback was not confirmed", + ); + return confirmed; +}; + +const stopProcess = async ( + input: HopperCleanupInput, + state: CleanupState, +): Promise => { + const supervisor = input.processSupervisor; + if (supervisor === undefined) return; + const stopped = await supervisor.stop( + state.cleanupResult === undefined + ? {} + : { cleanupResult: state.cleanupResult }, + ); + const diagnostic = createOwnedHopperShutdownDiagnostic( + supervisor.launch, + stopped, + state.cleanupResult, + ); + try { + input.onDiagnostic?.(diagnostic); + } catch { + // Diagnostic consumers cannot change the already-observed cleanup result. + } + input.logger.info(diagnostic, "Owned Hopper launcher shutdown completed"); + if (stopped.status !== "incomplete") { + state.ownedProcessStopped = stopped.status !== "not-owned"; + return; + } + const processGroupId = supervisor.launch.ownership?.processGroupId; + const resource = + processGroupId === undefined + ? "hopper-process" + : `process-group:${String(processGroupId)}`; + state.resources.add(resource); + state.issues.push({ resource, reason: stopped.reason }); + input.logger.warn( + { reason: stopped.reason }, + "Owned launcher cleanup failed closed", + ); +}; + +const recordUnconfirmedDocument = ( + input: HopperCleanupInput, + state: CleanupState, +): void => { + if ( + (input.socket === undefined && input.launch === undefined) || + state.shutdownConfirmed || + state.ownedProcessStopped + ) + return; + state.resources.add("hopper-document"); + state.issues.push({ + resource: "hopper-document", + reason: + input.activeRequest === null + ? "authenticated shutdown acknowledgement was not observed" + : `bridge operation ${input.activeRequest.operation} remained active after caller ${input.activeRequest.callerState}`, + }); +}; + +const closeRuntimeRoot = async ( + runtimeRoot: PrivateRuntimeRoot | undefined, + state: CleanupState, +): Promise => { + try { + await runtimeRoot?.close(); + } catch (cause: unknown) { + const resource = runtimeRoot?.path ?? "hopper-runtime-root"; + state.resources.add(resource); + state.issues.push({ + resource, + reason: + cause instanceof Error + ? cause.message + : "private runtime root cleanup failed", + }); + } +}; + +const cleanupOutcome = ( + input: HopperCleanupInput, + state: CleanupState, +): Result => { + if (state.issues.length === 0) return ok(null); + return err( + new ProviderCleanupError("hopper", [...state.resources], { + issues: cleanupIssueDetails(state.issues), + ...(input.runtimeRoot === undefined + ? {} + : { runtime_root: input.runtimeRoot.path }), + ...(input.activeRequest === null + ? {} + : { active_request: activityDetails(input.activeRequest) }), + }), + ); +}; + +const cleanupIssueDetails = (issues: readonly CleanupIssue[]): JsonValue => + issues.map((issue) => ({ + resource: issue.resource, + reason: issue.reason, + })); + +const activityDetails = (activity: HopperRequestActivity): JsonValue => ({ + request_id: activity.requestId, + operation: activity.operation, + elapsed_ms: activity.elapsedMs, + timeout_ms: activity.timeoutMs, + caller_state: activity.callerState, + queued_requests: activity.queuedRequests, +}); + +const report = async ( + progress: ProgressReporter | undefined, + completed: number, + message: string, +): Promise => { + await progress + ?.report({ + phase: "hopper_cleanup", + completed, + total: 1, + message, + }) + .catch(() => undefined); +}; diff --git a/src/hopper/HopperClient.ts b/src/hopper/HopperClient.ts index c14c47d11..699ab8415 100644 --- a/src/hopper/HopperClient.ts +++ b/src/hopper/HopperClient.ts @@ -6,6 +6,7 @@ import { join } from "node:path"; import { HopperCancelledError, + type AnalysisError, type HopperError, HopperProcessError, HopperProtocolError, @@ -16,36 +17,35 @@ import { } from "../domain/errors.js"; import { err, ok, type Result } from "../domain/result.js"; import type { JsonValue } from "../domain/jsonValue.js"; +import type { ProgressReporter } from "../application/ProgressReporter.js"; import { silentLogger, type Logger } from "../logger.js"; -import { PendingOperations } from "../process/PendingOperations.js"; import { PrivateRuntimeRoot } from "../process/PrivateRuntimeRoot.js"; import { ProviderStartupDeadline } from "../process/ProviderDeadline.js"; -import type { ProcessCleanupResult } from "../process/ProcessOwnership.js"; import { ProviderRunLineage } from "../process/ProviderRunLineage.js"; import { type ProviderProcessDiagnostic, ProviderProcessSupervisor, } from "../process/ProviderProcess.js"; import type { BridgeLaunch, BridgeLauncher } from "./BridgeLauncher.js"; +import type { HopperDiagnostic } from "./HopperDiagnostics.js"; +import { cleanupHopperSession } from "./HopperCleanup.js"; import { - createOwnedHopperShutdownDiagnostic, - type HopperDiagnostic, - providerCleanupFailure, -} from "./HopperDiagnostics.js"; -import { - isHopperCleanupRequired, - isHopperShutdownAcknowledgement, parseHopperServerInfo, type HopperServerInfo, } from "./HopperSessionValues.js"; import { connectHopperSocketOnce } from "./HopperSocketConnection.js"; import { hopperLauncherFailureDiagnostic } from "./HopperProcessDiagnostic.js"; -import { parseResponseLine, responseResult } from "./protocol.js"; +import { + HopperRequestQueue, + type HopperRequestActivity, +} from "./HopperRequestQueue.js"; +import { HopperResponseStream } from "./HopperResponseStream.js"; +import { responseResult } from "./protocol.js"; export type { HopperServerInfo } from "./HopperSessionValues.js"; -const MAX_LINE_BYTES = 10 * 1024 * 1024; const SHUTDOWN_TIMEOUT_MS = 30_000; +const MAX_QUEUED_REQUESTS = 64; const SESSION_ROOT = process.platform === "darwin" ? "/tmp" : tmpdir(); /** Dependencies, deadlines, and redacted diagnostics for one bridge client. */ @@ -72,10 +72,8 @@ export class HopperClient { Pick > & HopperClientOptions; - readonly #pending = new PendingOperations< - number, - Result - >(); + readonly #requests: HopperRequestQueue; + readonly #responses: HopperResponseStream; readonly #logger: Logger; #socket: Socket | undefined; #launch: BridgeLaunch | undefined; @@ -84,15 +82,14 @@ export class HopperClient { #token: string | undefined; readonly #lineage = new ProviderRunLineage(); #nextId = 1; - #buffer = ""; #closing = false; #launcherExitCode: number | null | undefined; #launcherFailureDiagnostic: HopperStartupDiagnostic | undefined; #startupController: AbortController | undefined; #startPromise: Promise> | undefined; - #closePromise: Promise | undefined; + #closePromise: Promise> | undefined; readonly #onSocketData = (chunk: string): void => { - this.#onData(chunk); + this.#responses.push(chunk); }; readonly #onSocketError = (): void => { this.#failAll(new HopperProcessError(null)); @@ -108,6 +105,30 @@ export class HopperClient { requestTimeoutMs: options.requestTimeoutMs ?? 30_000, startupTimeoutMs: options.startupTimeoutMs ?? 120_000, }; + this.#requests = new HopperRequestQueue( + MAX_QUEUED_REQUESTS, + ({ id, method, params }, failed) => { + const socket = this.#socket; + const token = this.#token; + if (socket === undefined || socket.destroyed || token === undefined) { + failed(); + return; + } + socket.write( + `${JSON.stringify({ id, token, method, params })}\n`, + (cause) => { + if (cause !== undefined && cause !== null) failed(); + }, + ); + }, + ); + this.#responses = new HopperResponseStream({ + accept: (response) => + this.#requests.accept(response.id, responseResult(response)), + hasQueued: (id) => this.#requests.hasQueued(id), + nextRequestId: () => this.#nextId, + abort: (message, cause) => this.#abortProtocol(message, cause), + }); } /** Latest token-verified launcher lineage for the active run. */ @@ -115,6 +136,11 @@ export class HopperClient { return this.#lineage.snapshot(); } + /** Observe a request that still occupies Hopper after caller timeout/cancel. */ + requestActivity(): HopperRequestActivity | null { + return this.#requests.activity(); + } + /** Launch the bridge once and complete its authenticated health handshake. */ start(signal?: AbortSignal): Promise> { if (this.#closePromise !== undefined) { @@ -251,6 +277,7 @@ export class HopperClient { options: { readonly signal?: AbortSignal; readonly timeoutMs?: number; + readonly progress?: ProgressReporter; } = {}, ): Promise> { const started = await this.start(options.signal); @@ -260,11 +287,18 @@ export class HopperClient { /** Stop the bridge, settle outstanding requests, and remove session artifacts. */ close(): Promise { + return this.closeWithOutcome().then(() => undefined); + } + + /** Stop the bridge and report whether every owned resource was verified clean. */ + closeWithOutcome( + options: { readonly progress?: ProgressReporter } = {}, + ): Promise> { const starting = this.#startPromise; const controller = this.#startupController; controller?.abort(); this.#closePromise ??= Promise.resolve().then(() => - this.#close(starting, controller), + this.#close(starting, controller, options), ); return this.#closePromise; } @@ -272,10 +306,11 @@ export class HopperClient { async #close( starting: Promise> | undefined, controller: AbortController | undefined, - ): Promise { + options: { readonly progress?: ProgressReporter }, + ): Promise> { try { await starting?.catch(() => undefined); - await this.#cleanup(); + return await this.#cleanup(options); } finally { if (this.#startPromise === starting) this.#startPromise = undefined; if (this.#startupController === controller) { @@ -285,92 +320,41 @@ export class HopperClient { } } - async #cleanup(): Promise { + async #cleanup( + options: { readonly progress?: ProgressReporter } = {}, + ): Promise> { this.#closing = true; try { - const socket = this.#socket; - let cleanupResult: ProcessCleanupResult | undefined; - if (socket !== undefined && !socket.destroyed) { - const shutdown = await this.#request( - "shutdown", - {}, - { timeoutMs: SHUTDOWN_TIMEOUT_MS }, - ).catch(() => err(new HopperProcessError(null))); - if ( - shutdown.ok && - isHopperCleanupRequired(shutdown.value) && - this.#launch?.shutdownByCleanup === true && - this.#launch.cleanup !== undefined - ) { - cleanupResult = await this.#launch - .cleanup() - .catch(providerCleanupFailure); - if (!cleanupResult.cleaned) { - await this.#fallbackDocumentShutdown(); - cleanupResult = await this.#launch - .cleanup() - .catch(providerCleanupFailure); - } - } else if ( - !shutdown.ok || - !isHopperShutdownAcknowledgement(shutdown.value) - ) - this.#logger.warn( - { status: shutdown.ok ? "invalid-acknowledgement" : "failed" }, - "Hopper document shutdown was not confirmed", - ); - } - this.#failAll(new HopperProcessError(null)); - if (socket !== undefined) this.#detachSocket(socket); - socket?.destroy(); + return await cleanupHopperSession({ + socket: this.#socket, + launch: this.#launch, + processSupervisor: this.#process, + runtimeRoot: this.#runtimeRoot, + activeRequest: this.#requests.activity(), + progress: options.progress, + logger: this.#logger, + onDiagnostic: this.#options.onDiagnostic, + request: (method) => + this.#request(method, {}, { timeoutMs: SHUTDOWN_TIMEOUT_MS }), + releaseTransport: (socket) => { + this.#failAll(new HopperProcessError(null)); + if (socket !== undefined) this.#detachSocket(socket); + socket?.destroy(); + }, + }); + } finally { this.#socket = undefined; - if (this.#process !== undefined) { - const stopped = await this.#process.stop( - cleanupResult === undefined ? {} : { cleanupResult }, - ); - const diagnostic = createOwnedHopperShutdownDiagnostic( - this.#process.launch, - stopped, - cleanupResult, - ); - this.#options.onDiagnostic?.(diagnostic); - this.#logger.info( - diagnostic, - "Owned Hopper launcher shutdown completed", - ); - if (stopped.status === "incomplete") - this.#logger.warn( - { reason: stopped.reason }, - "Owned launcher cleanup failed closed", - ); - } this.#process = undefined; this.#launch = undefined; - const runtimeRoot = this.#runtimeRoot; this.#runtimeRoot = undefined; - await runtimeRoot?.close(); this.#token = undefined; - this.#buffer = ""; + this.#responses.reset(); this.#launcherExitCode = undefined; this.#launcherFailureDiagnostic = undefined; - } finally { this.#closing = false; } } - async #fallbackDocumentShutdown(): Promise { - const fallback = await this.#request( - "shutdown_document", - {}, - { timeoutMs: SHUTDOWN_TIMEOUT_MS }, - ).catch(() => err(new HopperProcessError(null))); - if (!fallback.ok || !isHopperShutdownAcknowledgement(fallback.value)) - this.#logger.warn( - { status: fallback.ok ? "invalid-acknowledgement" : "failed" }, - "Hopper document shutdown fallback was not confirmed", - ); - } - async #connect( socketPath: string, deadline: ProviderStartupDeadline, @@ -413,7 +397,11 @@ export class HopperClient { async #request( method: string, params: JsonValue, - options: { readonly signal?: AbortSignal; readonly timeoutMs?: number }, + options: { + readonly signal?: AbortSignal; + readonly timeoutMs?: number; + readonly progress?: ProgressReporter; + }, ): Promise> { const socket = this.#socket; const token = this.#token; @@ -425,20 +413,11 @@ export class HopperClient { const id = this.#nextId++; const startedAt = performance.now(); const timeoutMs = options.timeoutMs ?? this.#options.requestTimeoutMs; - const response = this.#pending.wait(id, { + const result = await this.#requests.run(id, method, params, { timeoutMs, ...(options.signal === undefined ? {} : { signal: options.signal }), - timeoutValue: () => err(new HopperTimeoutError(timeoutMs)), - cancelledValue: () => err(new HopperCancelledError()), + ...(options.progress !== undefined ? { progress: options.progress } : {}), }); - socket.write( - `${JSON.stringify({ id, token, method, params })}\n`, - (cause) => { - if (cause !== undefined && cause !== null) - this.#pending.settle(id, err(new HopperProcessError(null))); - }, - ); - const result = await response; this.#logger[result.ok ? "debug" : "warn"]( { method, @@ -501,46 +480,13 @@ export class HopperClient { ); } - #onData(chunk: string): void { - this.#buffer += chunk; - let newline = this.#buffer.indexOf("\n"); - while (newline >= 0) { - const line = this.#buffer.slice(0, newline).trim(); - this.#buffer = this.#buffer.slice(newline + 1); - if (Buffer.byteLength(line) > MAX_LINE_BYTES) { - this.#abortProtocol("Hopper response exceeded the maximum line size"); - return; - } - if (line.length > 0) this.#onLine(line); - newline = this.#buffer.indexOf("\n"); - } - if (Buffer.byteLength(this.#buffer) > MAX_LINE_BYTES) { - this.#abortProtocol("Hopper response exceeded the maximum line size"); - } - } - - #onLine(line: string): void { - const parsed = parseResponseLine(line); - if (!parsed.ok) { - this.#abortProtocol(parsed.error.message, parsed.error); - return; - } - if (!this.#pending.has(parsed.value.id)) { - if (parsed.value.id >= this.#nextId) { - this.#abortProtocol("Hopper returned an unknown response id"); - } - return; - } - this.#pending.settle(parsed.value.id, responseResult(parsed.value)); - } - #abortProtocol(message: string, cause?: Error): void { this.#failAll(new HopperProtocolError(message, { cause })); this.#socket?.destroy(); } #failAll(error: HopperError): void { - this.#pending.failAll(() => err(error)); + this.#requests.failAll(error); } } diff --git a/src/hopper/HopperProvider.ts b/src/hopper/HopperProvider.ts index ef23e1fd9..fabbaa7f5 100644 --- a/src/hopper/HopperProvider.ts +++ b/src/hopper/HopperProvider.ts @@ -255,6 +255,26 @@ export class HopperProvider implements AnalysisProviderCandidate { ? [] : [{ provider: executionProvider, observation }]; }, + requestActivitySnapshots: () => { + const activity = client.requestActivity(); + return [ + { + provider: executionProvider, + active: + activity === null + ? null + : { + requestId: activity.requestId, + operation: activity.operation, + elapsedMs: activity.elapsedMs, + timeoutMs: activity.timeoutMs, + callerState: activity.callerState, + }, + queuedRequests: activity?.queuedRequests ?? 0, + }, + ]; + }, + closeWithOutcome: (options) => client.closeWithOutcome(options), close: () => client.close(), }; } diff --git a/src/hopper/HopperRequestQueue.ts b/src/hopper/HopperRequestQueue.ts new file mode 100644 index 000000000..7c3376966 --- /dev/null +++ b/src/hopper/HopperRequestQueue.ts @@ -0,0 +1,279 @@ +import { performance } from "node:perf_hooks"; + +import type { ProgressReporter } from "../application/ProgressReporter.js"; +import { + HopperCancelledError, + type HopperError, + HopperProcessError, + HopperProtocolError, + HopperTimeoutError, +} from "../domain/errors.js"; +import type { JsonValue } from "../domain/jsonValue.js"; +import { err, type Result } from "../domain/result.js"; + +export type HopperRequestResult = Result; + +export interface HopperRequestQueueOptions { + readonly signal?: AbortSignal; + readonly timeoutMs: number; + readonly progress?: ProgressReporter; +} + +export interface HopperRequestActivity { + readonly requestId: number; + readonly operation: string; + readonly elapsedMs: number; + readonly timeoutMs: number; + readonly callerState: "waiting" | "timed_out" | "cancelled"; + readonly queuedRequests: number; +} + +type RequestSender = ( + request: { + readonly id: number; + readonly method: string; + readonly params: JsonValue; + }, + failed: () => void, +) => void; + +interface QueuedRequest { + readonly id: number; + readonly method: string; + readonly params: JsonValue; + readonly signal: AbortSignal | undefined; + readonly timeoutMs: number; + readonly deadline: number; + readonly progress: ProgressReporter | undefined; + readonly resolve: (result: HopperRequestResult) => void; + readonly timer: NodeJS.Timeout; + readonly onAbort: (() => void) | undefined; + callerState: "waiting" | "timed_out" | "cancelled"; + callerSettled: boolean; + startedAt: number | undefined; + heartbeat: NodeJS.Timeout | undefined; +} + +/** Bounded FIFO that keeps the wire serialized until Hopper actually replies. */ +export class HopperRequestQueue { + readonly #queue: QueuedRequest[] = []; + #active: QueuedRequest | undefined; + + constructor( + private readonly maximum: number, + private readonly send: RequestSender, + ) {} + + /** Queue one request, counting queue wait against its caller deadline. */ + run( + id: number, + method: string, + params: JsonValue, + options: HopperRequestQueueOptions, + ): Promise { + if (options.signal?.aborted === true) + return Promise.resolve(err(new HopperCancelledError())); + if (options.timeoutMs <= 0) + return Promise.resolve(err(new HopperTimeoutError(options.timeoutMs))); + if (this.#size() >= this.maximum) + return Promise.resolve( + err( + new HopperProtocolError( + `Hopper serial request queue reached its ${String(this.maximum)}-request limit`, + ), + ), + ); + return new Promise((resolve) => { + let entry: QueuedRequest; + const onAbort = + options.signal === undefined ? undefined : () => this.#cancel(entry); + const timer = setTimeout(() => this.#timeout(entry), options.timeoutMs); + entry = { + id, + method, + params, + signal: options.signal, + timeoutMs: options.timeoutMs, + deadline: performance.now() + options.timeoutMs, + progress: options.progress, + resolve, + timer, + onAbort, + callerState: "waiting", + callerSettled: false, + startedAt: undefined, + heartbeat: undefined, + }; + this.#queue.push(entry); + if (onAbort !== undefined) + options.signal?.addEventListener("abort", onAbort, { once: true }); + if (this.#active !== undefined || this.#queue.length > 1) + this.#report( + entry, + `${method} queued behind ${String(this.#size() - 1)} Hopper request(s)`, + ); + this.#drain(); + }); + } + + /** Settle the active wire request. False identifies an unexpected response id. */ + accept(id: number, result: HopperRequestResult): boolean { + const entry = this.#active; + if (entry === undefined || entry.id !== id) return false; + this.#releaseWire(entry); + if (!entry.callerSettled) this.#settleCaller(entry, result, "waiting"); + this.#active = undefined; + this.#drain(); + return true; + } + + /** Fail the active wire request and every request still waiting in the FIFO. */ + failAll(error: HopperError): void { + const active = this.#active; + this.#active = undefined; + if (active !== undefined) { + this.#releaseWire(active); + if (!active.callerSettled) + this.#settleCaller(active, err(error), "waiting"); + } + for (const entry of this.#queue.splice(0)) + this.#settleCaller(entry, err(error), "waiting"); + } + + /** Snapshot the one request still occupying Hopper's serial Python thread. */ + activity(): HopperRequestActivity | null { + const active = this.#active; + if (active === undefined || active.startedAt === undefined) return null; + return { + requestId: active.id, + operation: active.method, + elapsedMs: Math.max(0, Math.round(performance.now() - active.startedAt)), + timeoutMs: active.timeoutMs, + callerState: active.callerState, + queuedRequests: this.#queue.length, + }; + } + + /** Return whether an id belongs to a request that has not crossed the wire. */ + hasQueued(id: number): boolean { + return this.#queue.some((entry) => entry.id === id); + } + + /** Number of caller requests waiting behind the active bridge operation. */ + queuedCount(): number { + return this.#queue.length; + } + + #size(): number { + return this.#queue.length + (this.#active === undefined ? 0 : 1); + } + + #drain(): void { + if (this.#active !== undefined) return; + const entry = this.#queue.shift(); + if (entry === undefined) return; + const remaining = entry.deadline - performance.now(); + if (entry.signal?.aborted === true || remaining <= 0) { + this.#settleCaller( + entry, + entry.signal?.aborted === true + ? err(new HopperCancelledError()) + : err(new HopperTimeoutError(entry.timeoutMs)), + entry.signal?.aborted === true ? "cancelled" : "timed_out", + ); + this.#drain(); + return; + } + this.#active = entry; + entry.startedAt = performance.now(); + this.#report(entry, `${entry.method} started on Hopper's serial bridge`); + entry.heartbeat = setInterval(() => { + if (entry.callerSettled || entry.startedAt === undefined) return; + const elapsed = Math.round(performance.now() - entry.startedAt); + this.#report( + entry, + `${entry.method} is still running in Hopper (${String(elapsed)} ms elapsed)`, + ); + }, 1_000); + try { + this.send( + { id: entry.id, method: entry.method, params: entry.params }, + () => this.accept(entry.id, err(new HopperProcessError(null))), + ); + } catch (cause: unknown) { + this.accept( + entry.id, + err(new HopperProtocolError("Hopper socket write failed", { cause })), + ); + } + } + + #timeout(entry: QueuedRequest): void { + if (entry.callerSettled) return; + if (this.#active === entry) { + this.#settleCaller( + entry, + err(new HopperTimeoutError(entry.timeoutMs)), + "timed_out", + ); + return; + } + const index = this.#queue.indexOf(entry); + if (index < 0) return; + this.#queue.splice(index, 1); + this.#settleCaller( + entry, + err(new HopperTimeoutError(entry.timeoutMs)), + "timed_out", + ); + } + + #cancel(entry: QueuedRequest): void { + if (entry.callerSettled) return; + if (this.#active === entry) { + this.#settleCaller(entry, err(new HopperCancelledError()), "cancelled"); + return; + } + const index = this.#queue.indexOf(entry); + if (index < 0) return; + this.#queue.splice(index, 1); + this.#settleCaller(entry, err(new HopperCancelledError()), "cancelled"); + } + + #settleCaller( + entry: QueuedRequest, + result: HopperRequestResult, + callerState: QueuedRequest["callerState"], + ): void { + if (entry.callerSettled) return; + entry.callerSettled = true; + entry.callerState = callerState; + if (this.#active === entry && entry.heartbeat !== undefined) { + clearInterval(entry.heartbeat); + entry.heartbeat = undefined; + } + clearTimeout(entry.timer); + if (entry.signal !== undefined && entry.onAbort !== undefined) + entry.signal.removeEventListener("abort", entry.onAbort); + entry.resolve(result); + } + + #releaseWire(entry: QueuedRequest): void { + if (entry.heartbeat !== undefined) clearInterval(entry.heartbeat); + entry.heartbeat = undefined; + clearTimeout(entry.timer); + if (entry.signal !== undefined && entry.onAbort !== undefined) + entry.signal.removeEventListener("abort", entry.onAbort); + } + + #report(entry: QueuedRequest, message: string): void { + void entry.progress + ?.report({ + phase: "hopper_request", + completed: 0, + total: 1, + message, + }) + .catch(() => undefined); + } +} diff --git a/src/hopper/HopperResponseStream.ts b/src/hopper/HopperResponseStream.ts new file mode 100644 index 000000000..e683afc49 --- /dev/null +++ b/src/hopper/HopperResponseStream.ts @@ -0,0 +1,56 @@ +import { parseResponseLine, type HopperResponse } from "./protocol.js"; + +const MAX_LINE_BYTES = 10 * 1024 * 1024; + +export interface HopperResponseStreamOptions { + readonly accept: (response: HopperResponse) => boolean; + readonly hasQueued: (id: number) => boolean; + readonly nextRequestId: () => number; + readonly abort: (message: string, cause?: Error) => void; +} + +/** Incrementally validates bounded Hopper NDJSON response frames. */ +export class HopperResponseStream { + #buffer = ""; + + constructor(readonly options: HopperResponseStreamOptions) {} + + push(chunk: string): void { + this.#buffer += chunk; + let newline = this.#buffer.indexOf("\n"); + while (newline >= 0) { + const line = this.#buffer.slice(0, newline).trim(); + this.#buffer = this.#buffer.slice(newline + 1); + if (Buffer.byteLength(line) > MAX_LINE_BYTES) { + this.options.abort("Hopper response exceeded the maximum line size"); + return; + } + if (line.length > 0 && !this.#acceptLine(line)) return; + newline = this.#buffer.indexOf("\n"); + } + if (Buffer.byteLength(this.#buffer) > MAX_LINE_BYTES) { + this.options.abort("Hopper response exceeded the maximum line size"); + } + } + + reset(): void { + this.#buffer = ""; + } + + #acceptLine(line: string): boolean { + const parsed = parseResponseLine(line); + if (!parsed.ok) { + this.options.abort(parsed.error.message, parsed.error); + return false; + } + if (this.options.accept(parsed.value)) return true; + if ( + parsed.value.id >= this.options.nextRequestId() || + this.options.hasQueued(parsed.value.id) + ) { + this.options.abort("Hopper returned an unknown response id"); + return false; + } + return true; + } +} diff --git a/src/server/lifecycleProgress.ts b/src/server/lifecycleProgress.ts new file mode 100644 index 000000000..49df9ddae --- /dev/null +++ b/src/server/lifecycleProgress.ts @@ -0,0 +1,22 @@ +import type { ProgressReporter } from "../application/ProgressReporter.js"; + +/** Report one lifecycle operation start through the shared monotonic boundary. */ +export const reportLifecycleStart = ( + progress: ProgressReporter, + phase: string, + message = "started", +): Promise => progress.report({ phase, completed: 0, total: 1, message }); + +/** Report one lifecycle operation's terminal disposition. */ +export const reportLifecycleEnd = ( + progress: ProgressReporter, + phase: string, + succeeded: boolean, +): Promise => + progress.report({ + phase, + completed: 1, + total: 1, + message: succeeded ? "completed" : "failed", + terminal: true, + }); diff --git a/src/server/permissionFailure.ts b/src/server/permissionFailure.ts new file mode 100644 index 000000000..061abd14d --- /dev/null +++ b/src/server/permissionFailure.ts @@ -0,0 +1,22 @@ +import type { PermissionAuthority } from "../application/PermissionAuthority.js"; +import { + AnalysisProtocolError, + PermissionRequiredError, + type AnalysisError, +} from "../domain/errors.js"; +import { err, type Result } from "../domain/result.js"; + +/** Preserve a typed permission denial or contain an unexpected authority error. */ +export const permissionFailure = ( + failure: Awaited>, +): Result => { + if (failure.ok) + return err(new AnalysisProtocolError("Expected a denied permission")); + return err( + failure.error instanceof PermissionRequiredError + ? failure.error + : new AnalysisProtocolError(failure.error.message, { + cause: failure.error, + }), + ); +}; diff --git a/src/server/registerCloseLifecycleTool.ts b/src/server/registerCloseLifecycleTool.ts new file mode 100644 index 000000000..8ff2c9688 --- /dev/null +++ b/src/server/registerCloseLifecycleTool.ts @@ -0,0 +1,93 @@ +import { writeAnalysisSnapshot } from "../application/AnalysisSnapshotFiles.js"; +import { closeBinaryInputSchema } from "../contracts/sessionLifecycleInputs.js"; +import { ok } from "../domain/result.js"; +import { + reportLifecycleEnd, + reportLifecycleStart, +} from "./lifecycleProgress.js"; +import { logToolExecution } from "./toolLogging.js"; +import { toCallToolResult } from "./toolResult.js"; +import { mcpProgressReporter } from "./mcpProgress.js"; +import { permissionFailure } from "./permissionFailure.js"; +import type { LifecycleToolRegistration } from "./registerSessionTools.js"; +import { safeParseToolInput } from "./toolInputValidation.js"; +import { toolRegistrationOptions } from "./toolRegistrationOptions.js"; + +/** Register provider cleanup and optional pre-close snapshot persistence. */ +export const registerCloseLifecycleTool = ({ + server, + session, + logger, + contracts: [, closeContract], + snapshotFilePolicy, + permissionAuthority, +}: LifecycleToolRegistration): void => { + server.registerTool( + closeContract.name, + toolRegistrationOptions(closeContract), + async (input, context) => { + const parsedInput = safeParseToolInput( + closeBinaryInputSchema, + input, + closeContract.name, + ); + if (!parsedInput.ok) return toCallToolResult(parsedInput, closeContract); + const parsed = parsedInput.value; + const progress = mcpProgressReporter(context); + if (parsed.snapshot_path === undefined) { + await reportLifecycleStart(progress, closeContract.name); + const closed = await logToolExecution(logger, closeContract.name, () => + session.close({ progress }), + ); + await reportLifecycleEnd(progress, closeContract.name, closed.ok); + server.sendToolListChanged(); + return toCallToolResult(closed, closeContract); + } + if (permissionAuthority !== undefined) { + const authorized = await permissionAuthority.authorize( + { + capability: "snapshot_write", + roots: [parsed.snapshot_path], + executables: [], + environment_names: [], + network: "none", + mount: false, + operation_identity: `close_binary:snapshot:${parsed.snapshot_path}`, + }, + "write", + ); + if (!authorized.ok) + return toCallToolResult(permissionFailure(authorized), closeContract); + } + const snapshot = session.exportAnalysisSnapshot(); + if (!snapshot.ok) return toCallToolResult(snapshot, closeContract); + const written = await writeAnalysisSnapshot( + snapshot.value, + parsed.snapshot_path, + parsed.overwrite, + snapshotFilePolicy, + ); + if (!written.ok) return toCallToolResult(written, closeContract); + await reportLifecycleStart( + progress, + closeContract.name, + "snapshot written; closing provider", + ); + const closed = await logToolExecution(logger, closeContract.name, () => + session.close({ progress }), + ); + await reportLifecycleEnd(progress, closeContract.name, closed.ok); + server.sendToolListChanged(); + return closed.ok + ? toCallToolResult( + ok({ + path: written.value.path, + bytes: written.value.bytes, + entries: snapshot.value.entries.length, + }), + closeContract, + ) + : toCallToolResult(closed, closeContract); + }, + ); +}; diff --git a/src/server/registerEnhancedTools.ts b/src/server/registerEnhancedTools.ts index 14907981f..7f51ef21d 100644 --- a/src/server/registerEnhancedTools.ts +++ b/src/server/registerEnhancedTools.ts @@ -49,9 +49,8 @@ export const registerEnhancedTools = ( analysis: AnalysisOperationPort, options: EnhancedToolRegistration, ): void => { - const services = new EnhancedTools(analysis); for (const contract of ENHANCED_TOOL_CONTRACTS) { - registerEnhancedTool(server, services, contract, { + registerEnhancedTool(server, analysis, contract, { logger: options.logger, activeTarget: options.activeTarget, analysisProfile: options.analysisProfile, @@ -63,7 +62,7 @@ export const registerEnhancedTools = ( const registerEnhancedTool = ( server: McpServer, - services: EnhancedTools, + analysis: AnalysisOperationPort, contract: ToolContract, registration: { readonly logger: Logger; @@ -91,6 +90,13 @@ const registerEnhancedTool = ( if (!validatedCall.ok) return toCallToolResult(validatedCall, contract); const parsedInput = validatedCall.value.input; const parameters = jsonParameters(parsedInput); + const services = new EnhancedTools({ + execute: (operation, operationParameters, executionOptions) => + analysis.execute(operation, operationParameters, { + ...executionOptions, + progress, + }), + }); const result = await logToolExecution(registration.logger, name, () => services.executeValidated(validatedCall.value, context.mcpReq.signal), ); diff --git a/src/server/registerSessionStatusTool.ts b/src/server/registerSessionStatusTool.ts index 6914434cc..3b451c612 100644 --- a/src/server/registerSessionStatusTool.ts +++ b/src/server/registerSessionStatusTool.ts @@ -148,6 +148,7 @@ const projectSessionStatus = (options: { active_provider: activeProvider, target: open ? targetFrom(options.status) : null, alignment: options.serverIdentity.alignment, + analysis_activity: options.status.analysis_activity, recommended_actions: recommendedActions( open, options.serverIdentity.alignment.state, diff --git a/src/server/registerSessionTools.ts b/src/server/registerSessionTools.ts index 2f81a483b..2c4231cfc 100644 --- a/src/server/registerSessionTools.ts +++ b/src/server/registerSessionTools.ts @@ -6,7 +6,7 @@ import { SESSION_TOOL_CONTRACTS } from "../contracts/toolContracts.js"; import { toCallToolResult } from "./toolResult.js"; import type { Logger } from "../logger.js"; import { logToolExecution } from "./toolLogging.js"; -import { err, ok, type Result } from "../domain/result.js"; +import { ok, type Result } from "../domain/result.js"; import type { ProcessCapture, ProcessExecutionPolicy, @@ -18,16 +18,8 @@ import { processCapturePermissionRequest } from "../application/ProcessCapturePe import type { Evidence } from "../domain/evidence.js"; import type { AnalysisSnapshot } from "../domain/analysisSnapshot.js"; import type { EvidenceFilePolicy } from "../domain/evidenceBundle.js"; -import { - readAnalysisSnapshot, - writeAnalysisSnapshot, -} from "../application/AnalysisSnapshotFiles.js"; -import { - AnalysisProtocolError, - PermissionRequiredError, - UnknownRegistryError, - type AnalysisError, -} from "../domain/errors.js"; +import { readAnalysisSnapshot } from "../application/AnalysisSnapshotFiles.js"; +import { UnknownRegistryError, type AnalysisError } from "../domain/errors.js"; import { DENY_EVIDENCE_FILE_POLICY, DENY_PROCESS_POLICY, @@ -40,10 +32,7 @@ import { toolRegistrationOptions } from "./toolRegistrationOptions.js"; import { safeParseToolInput } from "./toolInputValidation.js"; import { registerBundleComparisonTool } from "./registerBundleComparisonTool.js"; import { registerInvestigationTools } from "./registerInvestigationTools.js"; -import { - closeBinaryInputSchema, - openBinaryInputSchema, -} from "../contracts/sessionLifecycleInputs.js"; +import { openBinaryInputSchema } from "../contracts/sessionLifecycleInputs.js"; import { registerSessionStatusTool } from "./registerSessionStatusTool.js"; import type { PermissionAuthority } from "../application/PermissionAuthority.js"; import { mcpProgressReporter } from "./mcpProgress.js"; @@ -61,20 +50,8 @@ import { sessionAvailabilityPolicy, type SessionAvailability, } from "./sessionAvailabilityPolicy.js"; - -const permissionFailure = ( - failure: Awaited>, -): Result => { - if (failure.ok) - return err(new AnalysisProtocolError("Expected a denied permission")); - return err( - failure.error instanceof PermissionRequiredError - ? failure.error - : new AnalysisProtocolError(failure.error.message, { - cause: failure.error, - }), - ); -}; +import { permissionFailure } from "./permissionFailure.js"; +import { registerCloseLifecycleTool } from "./registerCloseLifecycleTool.js"; const recordProcessResidualUnknowns = ( session: BinarySessionPort, @@ -204,7 +181,7 @@ const registerProcessTools = ({ ); }; -interface LifecycleToolRegistration { +export interface LifecycleToolRegistration { readonly server: McpServer; readonly session: BinarySessionPort; readonly logger: Logger; @@ -314,75 +291,6 @@ const registerOpenLifecycleTool = ({ ); }; -const registerCloseLifecycleTool = ({ - server, - session, - logger, - contracts: [, closeContract], - snapshotFilePolicy, - permissionAuthority, -}: LifecycleToolRegistration): void => { - server.registerTool( - closeContract.name, - toolRegistrationOptions(closeContract), - async (input) => { - const parsedInput = safeParseToolInput( - closeBinaryInputSchema, - input, - closeContract.name, - ); - if (!parsedInput.ok) return toCallToolResult(parsedInput, closeContract); - const parsed = parsedInput.value; - if (parsed.snapshot_path === undefined) { - const closed = await logToolExecution(logger, closeContract.name, () => - session.close(), - ); - if (closed.ok) server.sendToolListChanged(); - return toCallToolResult(closed, closeContract); - } - if (permissionAuthority !== undefined) { - const authorized = await permissionAuthority.authorize( - { - capability: "snapshot_write", - roots: [parsed.snapshot_path], - executables: [], - environment_names: [], - network: "none", - mount: false, - operation_identity: `close_binary:snapshot:${parsed.snapshot_path}`, - }, - "write", - ); - if (!authorized.ok) - return toCallToolResult(permissionFailure(authorized), closeContract); - } - const snapshot = session.exportAnalysisSnapshot(); - if (!snapshot.ok) return toCallToolResult(snapshot, closeContract); - const written = await writeAnalysisSnapshot( - snapshot.value, - parsed.snapshot_path, - parsed.overwrite, - snapshotFilePolicy, - ); - if (!written.ok) return toCallToolResult(written, closeContract); - const closed = await logToolExecution(logger, closeContract.name, () => - session.close(), - ); - if (closed.ok) server.sendToolListChanged(); - return closed.ok - ? toCallToolResult( - ok({ - path: written.value.path, - bytes: written.value.bytes, - entries: snapshot.value.entries.length, - }), - closeContract, - ) - : toCallToolResult(closed, closeContract); - }, - ); -}; - /** Register MCP-only target lifecycle operations on a long-lived session. */ export interface SessionToolOptions { readonly processPolicy?: ProcessExecutionPolicy; diff --git a/tests/binarySession.test.ts b/tests/binarySession.test.ts index 35876b94d..42ffa52a0 100644 --- a/tests/binarySession.test.ts +++ b/tests/binarySession.test.ts @@ -14,6 +14,7 @@ import { HopperStartError, ProviderAdapterError, } from "../src/domain/errors.js"; +import { ProviderCleanupError } from "../src/domain/providerCleanupError.js"; import { err, ok as resultOk } from "../src/domain/result.js"; import { createEvidenceBundle } from "../src/domain/evidenceBundle.js"; import { createAnalysisProfile } from "../src/domain/analysisProfile.js"; @@ -123,6 +124,19 @@ describe("binary session", () => { }, }, ], + requestActivitySnapshots: () => [ + { + provider: provider.identity(), + active: { + requestId: 7, + operation: "analyze_function", + elapsedMs: 31_000, + timeoutMs: 30_000, + callerState: "timed_out", + }, + queuedRequests: 2, + }, + ], close: () => Promise.resolve(), }; }, @@ -184,6 +198,22 @@ describe("binary session", () => { ], }, }, + analysis_activity: { + status: "timed_out_busy", + providers: [ + { + provider: { id: "fixture", name: "Fixture", version: "1" }, + active: { + request_id: 7, + operation: "analyze_function", + elapsed_ms: 31_000, + timeout_ms: 30_000, + caller_state: "timed_out", + }, + queued_requests: 2, + }, + ], + }, }); expect(operations).toEqual(["health", "address_name"]); await session.close(); @@ -724,6 +754,50 @@ describe("binary session", () => { expect(overlapped).toBe(false); await session.close(); }); + + it("clears session state while preserving a typed provider cleanup failure", async () => { + const [first] = await targets(); + const cleanupError = new ProviderCleanupError( + "fixture", + ["fixture-document"], + { reason: "shutdown acknowledgement missing" }, + ); + const session = new BinarySession(() => ({ + execute: () => Promise.resolve(ok(null)), + closeWithOutcome: () => Promise.resolve(err(cleanupError)), + close: () => Promise.resolve(), + })); + expect((await session.open(first)).ok).toBe(true); + + expect(await session.close()).toEqual(err(cleanupError)); + expect(session.status()).toMatchObject({ + open: false, + analysis_activity: { status: "not_observed", providers: [] }, + }); + }); + + it("does not start a replacement after the active provider cleanup is unconfirmed", async () => { + const [first, second] = await targets(); + const cleanupError = new ProviderCleanupError( + "fixture", + ["fixture-document"], + { reason: "shutdown acknowledgement missing" }, + ); + let created = 0; + const session = new BinarySession(() => { + created += 1; + return { + execute: () => Promise.resolve(ok(null)), + closeWithOutcome: () => Promise.resolve(err(cleanupError)), + close: () => Promise.resolve(), + }; + }); + expect((await session.open(first)).ok).toBe(true); + + expect(await session.open(second)).toEqual(err(cleanupError)); + expect(created).toBe(1); + expect(session.status()).toMatchObject({ open: false }); + }); }); const client = (fail = false): AnalysisClient => ({ diff --git a/tests/bridgeTruthfulness.test.ts b/tests/bridgeTruthfulness.test.ts index 23d537ba7..da6d7a3e7 100644 --- a/tests/bridgeTruthfulness.test.ts +++ b/tests/bridgeTruthfulness.test.ts @@ -55,6 +55,22 @@ describe("Hopper bridge truthfulness", () => { expect(bridgeSource).toContain('params.get("collection_offset", {})'); }); + it("provides a raw-instruction fast path without decompilation or global scans", () => { + const start = bridgeSource.indexOf("def _read_function_instructions"); + const end = bridgeSource.indexOf("\ndef _analyze_function", start); + const implementation = bridgeSource.slice(start, end); + expect(implementation).toContain("_instruction_addresses("); + expect(implementation).toContain("offset + limit + 1"); + expect(implementation).not.toContain("decompile("); + expect(implementation).not.toContain("_search_inventory("); + expect(implementation).not.toContain("_strings("); + expect(implementation).not.toContain("_name_map("); + expect(bridgeSource).toContain( + "pseudo = _pseudocode(document, procedure) or", + ); + expect(bridgeSource).toContain("return _pseudocode(document, procedure)"); + }); + it("projects opaque Hopper locals into an exact provenance-bearing shape", () => { expect(bridgeSource).toContain("def _procedure_locals(procedure):"); expect(bridgeSource).toContain('"description": str(local)'); diff --git a/tests/conformance/README.md b/tests/conformance/README.md index c09178888..8805402bf 100644 --- a/tests/conformance/README.md +++ b/tests/conformance/README.md @@ -18,7 +18,7 @@ the read-only Ghidra inventory contracts without committing a binary. `scripts/create-ghidra-windows-fixture.mjs` separately emits a deterministic, ignored native x86-64 PE application for the controlled Windows Ghidra P0 lane. The generator and fixed SHA-256 are versioned; the `.exe` is never committed or -executed. `npm run verify:ghidra:windows` uses it to prove all 18 operations, +executed. `npm run verify:ghidra:windows` uses it to prove all 19 operations, digest linkage, transport, and cleanup on the self-hosted real-Ghidra runner. The C fixture fixes a known call chain and strings. The version pair fixes an diff --git a/tests/enhancedTools.test.ts b/tests/enhancedTools.test.ts index 58d923164..a36346775 100644 --- a/tests/enhancedTools.test.ts +++ b/tests/enhancedTools.test.ts @@ -546,7 +546,7 @@ describe("enhanced MCP tools", () => { arguments: { detail: "detailed", limit: 1 }, }), ); - expect(procedureOffsets).toEqual([0, 500]); + expect(procedureOffsets).toEqual([0]); expect(result).toEqual({ document: "fixture", detail: "detailed", diff --git a/tests/fixtures/fakeGhidra.mjs b/tests/fixtures/fakeGhidra.mjs index 66d8e34cc..ba21325d3 100644 --- a/tests/fixtures/fakeGhidra.mjs +++ b/tests/fixtures/fakeGhidra.mjs @@ -176,7 +176,7 @@ const sessionInfo = ({ timedOut, }) => ({ name: "REA Ghidra bridge", - bridge_version: 4, + bridge_version: 5, run_id: sessionRunId, profile_digest: digest, provider: { id: "ghidra", version }, @@ -202,6 +202,7 @@ const sessionInfo = ({ "procedure_callers", "procedure_info", "procedure_pseudo_code", + "read_function_instructions", "procedure_references", "xrefs", ], diff --git a/tests/fixtures/fakeHopper.mjs b/tests/fixtures/fakeHopper.mjs index 3505fbb78..bf091a59c 100644 --- a/tests/fixtures/fakeHopper.mjs +++ b/tests/fixtures/fakeHopper.mjs @@ -5,7 +5,7 @@ const [socketPath, token, runId, shutdownMode = "acknowledge"] = const shutdownResult = () => ({ shutdown: true, analysis_stopped: true, - document_closed: true, + document_closed: shutdownMode !== "unconfirmed", ...(shutdownMode === "cleanup-required" ? { cleanup_required: true } : {}), }); const enhancedFixtureResult = (method) => { diff --git a/tests/ghidraClient.test.ts b/tests/ghidraClient.test.ts index 98a6e0a31..f0d3aecc7 100644 --- a/tests/ghidraClient.test.ts +++ b/tests/ghidraClient.test.ts @@ -156,7 +156,7 @@ describe("GhidraClient", () => { await expect(client.start()).resolves.toMatchObject({ ok: true, value: { - bridge_version: 4, + bridge_version: 5, target: { sha256: TARGET_SHA256 }, }, }); diff --git a/tests/ghidraFunctionValues.test.ts b/tests/ghidraFunctionValues.test.ts index cf1157db7..e0e0e33da 100644 --- a/tests/ghidraFunctionValues.test.ts +++ b/tests/ghidraFunctionValues.test.ts @@ -52,6 +52,19 @@ describe("Ghidra function-analysis boundary values", () => { collection_offset: { basic_blocks: 0, outgoing_references: 0 }, }, }); + expect( + parseGhidraFunctionInput("read_function_instructions", { + procedure: "main", + }), + ).toEqual({ + ok: true, + value: { + document: null, + procedure: "main", + offset: 0, + limit: 64, + }, + }); expect(parseGhidraFunctionInput("xrefs", {})).toMatchObject({ ok: false, error: { _tag: "AnalysisInputError" }, @@ -65,14 +78,39 @@ describe("Ghidra function-analysis boundary values", () => { }); it("keeps the admitted operation set closed", () => { - expect(GHIDRA_FUNCTION_OPERATIONS).toHaveLength(8); + expect(GHIDRA_FUNCTION_OPERATIONS).toHaveLength(9); expect(isGhidraFunctionOperation("analyze_function")).toBe(true); + expect(isGhidraFunctionOperation("read_function_instructions")).toBe(true); expect(isGhidraFunctionOperation("procedure_pseudo_code")).toBe(true); expect(isGhidraFunctionOperation("list_procedures")).toBe(false); expect(isGhidraFunctionOperation("set_comment")).toBe(false); }); it("parses provider-classified function facts and reference kinds", () => { + expect( + parseGhidraFunctionResult("read_function_instructions", { + procedure: ghidraFunctionIdentity(), + instructions: ghidraBounded(["0x401000: push rbp"]), + instructions_scanned: 1, + instruction_scan_truncated: false, + limitations: ["Ghidra-specific instruction text."], + }), + ).toMatchObject({ ok: true }); + expect( + parseGhidraFunctionResult("read_function_instructions", { + procedure: { + ...ghidraFunctionIdentity(), + address: "0X401000", + }, + instructions: ghidraBounded(["0x401000: push rbp"]), + instructions_scanned: 1, + instruction_scan_truncated: false, + limitations: ["Ghidra-specific instruction text."], + }), + ).toMatchObject({ + ok: false, + error: { _tag: "AnalysisOutputError" }, + }); expect( parseGhidraFunctionResult("procedure_info", { name: "fixture_main", diff --git a/tests/ghidraMcp.test.ts b/tests/ghidraMcp.test.ts index 8701ecad4..a221e0af4 100644 --- a/tests/ghidraMcp.test.ts +++ b/tests/ghidraMcp.test.ts @@ -306,7 +306,7 @@ const installationHost = (): GhidraInstallationHost => ({ const sessionInfo = (profileDigest: string, targetSha256: string) => ({ name: "REA Ghidra bridge" as const, - bridge_version: 4 as const, + bridge_version: 5 as const, run_id: "11111111-1111-4111-8111-111111111111", profile_digest: profileDigest, provider: { id: "ghidra" as const, version: "12.1.2" }, @@ -430,6 +430,16 @@ const resultBuilders = new Map([ }), ], ["procedure_pseudo_code", () => "int fixture_main(void) { return 42; }"], + [ + "read_function_instructions", + () => ({ + procedure: ghidraFunctionIdentity(), + instructions: ghidraBounded(["0x401000: CALL 0x401020", "0x401005: RET"]), + instructions_scanned: 2, + instruction_scan_truncated: false, + limitations: ["Ghidra-specific instruction text."], + }), + ], [ "procedure_references", (input, _limit) => ({ @@ -459,12 +469,12 @@ const resultFor = ( }; const page = (items: readonly JsonValue[], limit: number): JsonValue => ({ - items: [...items], + items: items.slice(0, limit), offset: 0, limit, total: items.length, - next_offset: null, - has_more: false, + next_offset: items.length > limit ? limit : null, + has_more: items.length > limit, }); const stringItem = (address: string, value: string): JsonValue => ({ diff --git a/tests/ghidraProvider.test.ts b/tests/ghidraProvider.test.ts index d6c5d0f25..d871faa38 100644 --- a/tests/ghidraProvider.test.ts +++ b/tests/ghidraProvider.test.ts @@ -48,7 +48,7 @@ describe("Ghidra provider", () => { expect(ghidra.capabilities().map(({ operation }) => operation)).toEqual( GHIDRA_PROVIDER_TOOL_CONTRACTS.map(({ name }) => name), ); - expect(GHIDRA_PROVIDER_TOOL_CONTRACTS).toHaveLength(18); + expect(GHIDRA_PROVIDER_TOOL_CONTRACTS).toHaveLength(19); expect(Object.isFrozen(ghidra.capabilities())).toBe(true); expect(ghidra.capabilities()).toEqual( expect.arrayContaining([ @@ -389,7 +389,7 @@ describe("Ghidra provider", () => { const sessionInfo = () => ({ name: "REA Ghidra bridge" as const, - bridge_version: 4 as const, + bridge_version: 5 as const, run_id: "11111111-1111-4111-8111-111111111111", profile_digest: "a".repeat(64), provider: { id: "ghidra" as const, version: "12.1.2" }, diff --git a/tests/hopperClient.test.ts b/tests/hopperClient.test.ts index 407b8a5e2..c7af32efa 100644 --- a/tests/hopperClient.test.ts +++ b/tests/hopperClient.test.ts @@ -7,6 +7,7 @@ import { afterEach, describe, expect, it } from "vitest"; import { OFFICIAL_TOOL_CONTRACTS } from "../src/contracts/toolContracts.js"; import type { HopperStartupDiagnostic } from "../src/domain/hopperStartupFailure.js"; +import { projectAnalysisError } from "../src/domain/errors.js"; import { ok } from "../src/domain/result.js"; import type { BridgeLauncher, @@ -106,6 +107,27 @@ class OwnedFixtureLauncher implements BridgeLauncher { } } +class UnconfirmedFixtureLauncher implements BridgeLauncher { + launch(session: BridgeSession) { + return Promise.resolve( + ok({ + process: spawn( + process.execPath, + [ + fixturePath, + session.socketPath, + session.token, + session.runId, + "unconfirmed", + ], + { stdio: ["ignore", "ignore", "pipe"] }, + ), + ownsProcessLifetime: false, + }), + ); + } +} + class CancelThenFixtureLauncher implements BridgeLauncher { #launches = 0; @@ -235,10 +257,16 @@ describe("HopperClient", () => { expect(Buffer.byteLength(launcher.socketPaths[0] ?? "")).toBeLessThan(104); }); - it("parses fragmented health and correlates concurrent responses", async () => { + it("serializes concurrent calls until the active Hopper reply arrives", async () => { const client = await startClient(); const slow = client.callTool("echo", { label: "slow", delay: 30 }); const fast = client.callTool("echo", { label: "fast", delay: 1 }); + let fastSettled = false; + void fast.then(() => { + fastSettled = true; + }); + await new Promise((resolve) => setTimeout(resolve, 10)); + expect(fastSettled).toBe(false); await expect(Promise.all([slow, fast])).resolves.toEqual([ { ok: true, value: { label: "slow", delay: 30 } }, { ok: true, value: { label: "fast", delay: 1 } }, @@ -250,7 +278,7 @@ describe("HopperClient", () => { const results = await Promise.all( OFFICIAL_TOOL_CONTRACTS.map(({ name }) => client.callTool(name, {})), ); - expect(results).toHaveLength(33); + expect(results).toHaveLength(34); expect(results.every((result) => result.ok)).toBe(true); }); @@ -312,6 +340,45 @@ describe("HopperClient", () => { }); }); + it("retains timed-out bridge activity until the late reply releases the queue", async () => { + const client = await startClient(); + const progress: string[] = []; + const result = await client.callTool( + "echo", + { delay: 60 }, + { + timeoutMs: 5, + progress: { + report: (update) => { + progress.push(update.message); + return Promise.resolve(); + }, + }, + }, + ); + + expect(result).toMatchObject({ + ok: false, + error: { _tag: "HopperTimeoutError" }, + }); + expect(client.requestActivity()).toMatchObject({ + operation: "echo", + callerState: "timed_out", + timeoutMs: 5, + }); + expect(progress).toEqual( + expect.arrayContaining([ + expect.stringContaining("started on Hopper's serial bridge"), + ]), + ); + await new Promise((resolve) => setTimeout(resolve, 70)); + expect(client.requestActivity()).toBeNull(); + await expect(client.callTool("echo", { alive: true })).resolves.toEqual({ + ok: true, + value: { alive: true }, + }); + }); + it("cancels bridge startup without waiting for the startup timeout", async () => { const client = new HopperClient({ launcher: new CancelThenFixtureLauncher(), @@ -441,6 +508,36 @@ describe("HopperClient", () => { await first; }); + it("reports unconfirmed unowned document cleanup as a typed close failure", async () => { + const client = new HopperClient({ + launcher: new UnconfirmedFixtureLauncher(), + startupTimeoutMs: 1_000, + }); + clients.push(client); + expect((await client.start()).ok).toBe(true); + + const closed = await client.closeWithOutcome(); + + expect(closed).toMatchObject({ + ok: false, + error: { + _tag: "ProviderAdapterError", + cleanupIncomplete: true, + cleanupResources: ["hopper-document"], + }, + }); + if (!closed.ok) + expect(projectAnalysisError(closed.error)).toMatchObject({ + code: "cleanup_incomplete", + details: { + provider_id: "hopper", + operation: "close_binary", + cleanup: "incomplete", + resources: ["hopper-document"], + }, + }); + }); + it("kills only its owned Hopper group and emits sanitized shutdown coordinates", async () => { const unrelated = spawn( process.execPath, diff --git a/tests/officialTools.test.ts b/tests/officialTools.test.ts index bcd8dcada..f6bc450df 100644 --- a/tests/officialTools.test.ts +++ b/tests/officialTools.test.ts @@ -32,6 +32,7 @@ const VALID_INPUTS: Readonly< procedure_callees: { procedure: "main" }, procedure_callers: { procedure: "main" }, procedure_info: { procedure: "main" }, + read_function_instructions: { procedure: "main" }, procedure_references: { procedure: "main" }, procedure_pseudo_code: { procedure: "main" }, resolve_containing_procedure: { address: "0x1000" }, @@ -115,6 +116,20 @@ const outputFor = (name: string): JsonValue => { found: true, procedure: { address: "0x1000", name: "main" }, }; + if (name === "read_function_instructions") + return { + procedure: { address: "0x1000", name: "main" }, + instructions: { + items: ["0x1000: ret"], + total: 1, + returned: 1, + truncated: false, + next_offset: null, + }, + instructions_scanned: 1, + instruction_scan_truncated: false, + limitations: ["Provider-specific instruction text."], + }; if (name === "procedure_references") return { procedure: { address: "0x1000", name: "main" }, diff --git a/tests/providerArchitecture.test.ts b/tests/providerArchitecture.test.ts index 5f269ed1f..be1cbe89a 100644 --- a/tests/providerArchitecture.test.ts +++ b/tests/providerArchitecture.test.ts @@ -78,7 +78,7 @@ describe("provider-neutral architecture", () => { if (!config.ok) return; const provider = new HopperProvider(config.value, silentLogger); const capabilities = provider.capabilities(); - expect(capabilities).toHaveLength(34); + expect(capabilities).toHaveLength(35); expect(new Set(capabilities.map(({ operation }) => operation)).size).toBe( capabilities.length, ); diff --git a/tests/serverIdentity.test.ts b/tests/serverIdentity.test.ts index c6508183b..6d44a1a41 100644 --- a/tests/serverIdentity.test.ts +++ b/tests/serverIdentity.test.ts @@ -26,10 +26,10 @@ describe("server and catalog identity", () => { }); expect(PRODUCT_IDENTITY.packageVersion).toBe(packageJson.version); expect(SDK_IDENTITY.server).toBe("2.0.0-beta.4"); - expect(CLI_COMMAND_NAMES).toHaveLength(58); - expect(new Set(CLI_COMMAND_NAMES).size).toBe(58); + expect(CLI_COMMAND_NAMES).toHaveLength(59); + expect(new Set(CLI_COMMAND_NAMES).size).toBe(59); expect(CATALOG_IDENTITY.counts).toEqual({ - cli_commands: 58, + cli_commands: 59, mcp_tools: TOOL_CONTRACTS.length, mcp_prompts: 6, mcp_resources: 2, diff --git a/tests/toolContracts.test.ts b/tests/toolContracts.test.ts index 9ce8bb6b9..9c1163009 100644 --- a/tests/toolContracts.test.ts +++ b/tests/toolContracts.test.ts @@ -16,7 +16,7 @@ import { APPLICATION_TOOL_CONTRACTS } from "../src/contracts/applicationToolCont describe("tool contract inventory", () => { it("publishes the canonical analysis and session tool inventory", () => { - expect(OFFICIAL_TOOL_CONTRACTS).toHaveLength(33); + expect(OFFICIAL_TOOL_CONTRACTS).toHaveLength(34); expect(ENHANCED_TOOL_CONTRACTS).toHaveLength(10); expect(NATIVE_TOOL_CONTRACTS).toHaveLength(5); expect(ARTIFACT_TOOL_CONTRACTS).toHaveLength(2);