diff --git a/docs/mcp-contracts.md b/docs/mcp-contracts.md index 557c98ab4..02218bf51 100644 --- a/docs/mcp-contracts.md +++ b/docs/mcp-contracts.md @@ -193,7 +193,8 @@ after target content was read. ## Retained application Evidence inputs `analyze_javascript_application`, `inspect_binary_layout`, `inspect_artifact`, -`inspect_wasm_artifact`, and `inspect_macho` accept `detail`: +`inspect_wasm_artifact`, `inspect_macho`, and `decode_interface_builder` +accept `detail`: `complete` (default) returns the complete analysis Evidence; `summary` records that complete Evidence in the session and returns the derived `inspect_analysis_view` summary Evidence instead. Its @@ -276,6 +277,17 @@ symbols (Electron Framework: 2.5 MB of imports and exports, 3.3 KB as a summary), so select `detail: "summary"` on the first call. Delivery detail never reaches the native tools or changes the source Evidence ID. +Completed `decode_interface_builder` Evidence supports a summary and +`view: {kind: "interface-builder", collection: "nodes", offset: 0, limit: 64}`. +Collections are `documents`, `nodes`, `edges`, and `coverage`; for nodes and +edges an optional `document` keeps rows whose evidence names that exact decoded +`relative_path`, and an undecoded path is refused. The summary keeps every +decoded document, the provider, node-kind, edge-relation and coverage-status +counts, and whether the decoded graph is incomplete. Each page repeats +`source_truncated`. A complete app graph often exceeds the default MCP response +budget (XLD.app: 6.7 MB), so select `detail: "summary"` on the first call. +Delivery detail never reaches the decoder or changes the source Evidence ID. + `trace_application_feature`, `trace_javascript_semantics`, `compare_application_versions`, `compare_source_to_bundle`, and `compare_javascript_export_shapes` accept complete inline application Evidence @@ -671,7 +683,13 @@ advertised input schema within a 13,312-byte budget, measured to sit below Moonshot's observed cap (community measurements: a 13.7 KB schema is accepted while a ~15 KB one is rejected). The setting is read at startup; restart the server to apply changes, as with `REA_MCP_MAX_RESPONSE_BYTES`. The default -value `full` keeps the complete advertisement unchanged. +value `full` keeps the complete advertisement unchanged. Compact presentation +also reduces schemas whose expanded JSON tree exceeds ten levels, the limit +reported by strict-depth MCP clients in [#2813](https://github.com/morluto/rea/issues/2813). +WorkBuddy/CodeBuddy gateways have also rejected full REA catalogs with an +opaque `11133` error while accepting the compact profile +([#2999](https://github.com/morluto/rea/issues/2999)); compatibility depends on +the selected gateway and model. The compact profile renders the same tool set, output schemas, annotations, and canonical server-side validation. It changes only the advertised input @@ -696,6 +714,9 @@ JSON Schema: canonical schema still rejects malformed calls server-side, so callers following the reduced advertisement receive the canonical typed error instead of silent acceptance. +- Schemas deeper than ten nodes use the same reduced projection, with + root-property names, types, guidance, and required fields retained. The + server continues to enforce the full canonical input contract. Boundary tests pin every advertised compact schema to the budget, keep them valid JSON Schema that still accepts every canonical example. Client verification @@ -711,6 +732,30 @@ identity. Explicit schema IDs and relative external reference bases are preserved. This keeps every tool and validation rule in discovery and does not change the SDK's catalog invalidation or availability checks. +Some stdio hosts reject a complete `tools/list` frame before they load any +tools. The opt-in `REA_MCP_OUTPUT_SCHEMA_PROFILE=off` setting omits `outputSchema` +from the advertised catalog for hosts with a small frame limit (the measured +pi-web-ui failure and catalog breakdown are in [#2512](https://github.com/morluto/rea/issues/2512)). +The default `full` profile is unchanged. With `off`, REA validates successful +structured results against the same named canonical tool contract before +delivery; it still returns complete `structuredContent` and text results. +Only client-side schema discovery and validation are omitted, so hosts using +this profile should rely on REA's validation and retain their usual response +size limits. + +### Function comparison limitations + +`compare_functions` compares the `pseudocode` and `assembly` dimensions as +exact provider text. A `changed` status establishes a textual difference, not a +behavior change. Addresses and generated names are not normalized, so relocated +labels can change these dimensions while `cfg` remains unchanged. Both text +dimensions carry this limitation inline. Numeric constants remain part of the +exact comparison. Reconstruction verification and static/runtime correlation +retain limitations from the selected dimension; an overall function claim +includes all dimension limitations. Equal CFGs establish only equal observed +block sizes and successor relationships. Text from different provider +identities remains unknown. + ### Comparison input shapes `compare_web_captures` accepts exactly one of two input shapes: diff --git a/src/application/AnalysisContextQueries.test.ts b/src/application/AnalysisContextQueries.test.ts index 99288d7a3..ef384e935 100644 --- a/src/application/AnalysisContextQueries.test.ts +++ b/src/application/AnalysisContextQueries.test.ts @@ -10,6 +10,7 @@ import { } from "./AnalysisProvider.js"; import { AnalysisCancelledError, + AnalysisOutputError, AnalysisProtocolError, } from "../domain/analysisErrorCore.js"; import { err, ok } from "../domain/result.js"; @@ -17,6 +18,35 @@ import { err, ok } from "../domain/result.js"; const provider = { id: "fixture", name: "Fixture", version: "1" }; describe("analysis context queries: navigation", () => { + it.each([ + [null, "0x401000", "current_document"], + ["fixture", null, "current_address"], + ] as const)( + "rejects malformed %s / %s provider selection before returning success", + async (document, address, operation) => { + const analysis: AnalysisOperationPort = { + execute: async (name) => + ok( + createAnalysisExecution( + name === "current_document" ? document : address, + provider, + ), + ), + }; + const result = await getNavigationContext(analysis, {}); + expect(result).toEqual( + err( + new AnalysisOutputError( + operation, + operation === "current_document" + ? "expected a document identity string" + : "expected an address string", + ), + ), + ); + }, + ); + it("aggregates volatile navigation and treats a missing procedure as null", async () => { const calls: { operation: string; @@ -70,6 +100,23 @@ describe("analysis context queries: navigation", () => { }); }); + it("preserves empty string selections accepted by the output contract", async () => { + const analysis: AnalysisOperationPort = { + execute: async (operation) => + ok( + createAnalysisExecution( + operation === "resolve_containing_procedure" + ? { found: false, procedure: null, reason: "not_in_procedure" } + : "", + provider, + ), + ), + }; + await expect(getNavigationContext(analysis, {})).resolves.toEqual( + ok({ document: "", address: "", procedure: null }), + ); + }); + it("uses an explicit document without passing it to current_document and preserves procedure failures", async () => { const calls: string[] = []; const failure = new AnalysisProtocolError("provider response malformed"); diff --git a/src/application/AnalysisContextQueries.ts b/src/application/AnalysisContextQueries.ts index 379ebc322..bd6846233 100644 --- a/src/application/AnalysisContextQueries.ts +++ b/src/application/AnalysisContextQueries.ts @@ -4,10 +4,11 @@ import type { AnalysisOperationPort, } from "./AnalysisProvider.js"; import type { AnalysisError } from "../domain/analysisErrorBase.js"; +import { AnalysisOutputError } from "../domain/analysisErrorCore.js"; import type { JsonValue } from "../domain/jsonValue.js"; import { analysisBookmarkSchema } from "../domain/hopperValues.js"; import { resolveAnalysisDocument } from "./AnalysisDocument.js"; -import { ok, type Result } from "../domain/result.js"; +import { err, ok, type Result } from "../domain/result.js"; type Facet = | { readonly state: "available"; readonly value: JsonValue } @@ -42,8 +43,14 @@ export const getNavigationContext = async ( if (typeof document !== "string" && !document.ok) return document; const documentName = typeof document === "string" ? document : document.value.result; - const documentArgs = - typeof documentName === "string" ? parameters(documentName) : {}; + if (typeof documentName !== "string") + return err( + new AnalysisOutputError( + "current_document", + "expected a document identity string", + ), + ); + const documentArgs = parameters(documentName); const address = await analysis.execute( "current_address", documentArgs, @@ -51,17 +58,12 @@ export const getNavigationContext = async ( ); if (!address.ok) return address; if (typeof address.value.result !== "string") - return ok({ - document: documentName, - address: address.value.result, - procedure: null, - }); + return err( + new AnalysisOutputError("current_address", "expected an address string"), + ); const procedure = await analysis.execute( "resolve_containing_procedure", - parameters( - typeof documentName === "string" ? documentName : undefined, - address.value.result, - ), + parameters(documentName, address.value.result), executionOptions(signal), ); if (!procedure.ok) { diff --git a/src/application/DirectAnalysis.ts b/src/application/DirectAnalysis.ts index 8a7874598..5e3799b0d 100644 --- a/src/application/DirectAnalysis.ts +++ b/src/application/DirectAnalysis.ts @@ -42,6 +42,7 @@ import { workflowAnalysisProfile, } from "./InvestigationProviders.js"; import { + executeComposedWorkflow, createWorkflowEvidence, workflowSnapshotRecord, recordWorkflowUnknowns, @@ -401,18 +402,17 @@ const executeAnalysisTool = async (input: { tool === "trace_native_ui_action" ) { const { EnhancedTools } = await import("./EnhancedTools.js"); - const result = await new EnhancedTools(session).execute( - tool, - input.arguments, - signal, + const result = await executeComposedWorkflow(session, (observed) => + new EnhancedTools(observed).execute(tool, input.arguments, signal), ); if (!result.ok) return { output: cliError(result.error) }; const evidence = createWorkflowEvidence({ target: input.openedTarget, operation: tool, parameters: input.arguments, - result: result.value, + result: result.value.result, upstreamProfile: evidenceProfile, + upstreamLimitations: result.value.limitations, }); return { output: evidence, evidence }; } diff --git a/src/application/WorkflowEvidence.test.ts b/src/application/WorkflowEvidence.test.ts index dab61a79c..b374efa01 100644 --- a/src/application/WorkflowEvidence.test.ts +++ b/src/application/WorkflowEvidence.test.ts @@ -3,15 +3,145 @@ import { describe, expect, it } from "vitest"; import { recordWorkflowUnknowns, createWorkflowEvidence, + executeComposedWorkflow, } from "./WorkflowEvidence.js"; import { InvestigationRecords } from "./investigation/InvestigationRecords.js"; import { UnknownRegistryError } from "../domain/unknownRegistryError.js"; -import { err } from "../domain/result.js"; +import { err, ok } from "../domain/result.js"; +import { + createAnalysisExecution, + type AnalysisOperationPort, +} from "./AnalysisProvider.js"; +import { AnalysisCancelledError } from "../domain/analysisErrorCore.js"; import { nativeApiBoundarySchema } from "../domain/native/nativeApiBoundary.js"; import { projectNativeApiInspection } from "./native/NativeApiInspection.js"; import { functionDossierSchema } from "../domain/hopperValues.js"; import { ghidraFunctionDossier } from "../domain/ghidraValues.fixture.js"; +const provider = { id: "fixture", name: "Fixture", version: "1" }; + +it("keeps Evidence identity stable when dependent branches finish in opposite orders", async () => { + const run = async (order: readonly number[]) => { + const gate = () => { + let resolve: () => void = () => { + throw new Error("gate not initialized"); + }; + const promise = new Promise((release) => { + resolve = release; + }); + return { promise, resolve }; + }; + const parents = [gate(), gate()]; + const children = [gate(), gate()]; + const analysis: AnalysisOperationPort = { + execute: async (operation, parameters) => { + const index = Number(parameters.index); + if (operation === "procedure_address") { + await parents[index]!.promise; + return ok(createAnalysisExecution(String(index), provider)); + } + children[index]!.resolve(); + return ok( + createAnalysisExecution(String(index), provider, { + limitations: [ + index === 0 ? "first child caveat" : "second child caveat", + ], + }), + ); + }, + }; + const composed = await executeComposedWorkflow( + analysis, + async (observed) => { + const branches = Promise.all( + [0, 1].map(async (index) => { + await observed.execute("procedure_address", { index }); + const child = await observed.execute("address_name", { index }); + if (!child.ok) throw child.error; + return child.value.result; + }), + ); + for (const index of order) { + parents[index]!.resolve(); + await children[index]!.promise; + } + return ok(await branches); + }, + ); + if (!composed.ok) throw composed.error; + return createWorkflowEvidence({ + target: undefined, + operation: "batch_decompile", + parameters: {}, + result: composed.value.result, + upstreamProfile: undefined, + upstreamLimitations: composed.value.limitations, + }); + }; + const left = await run([0, 1]); + const right = await run([1, 0]); + expect(left.normalized_result).toEqual(right.normalized_result); + expect(left.limitations).toEqual(right.limitations); + expect(left.evidence_id).toBe(right.evidence_id); +}); + +it("retains unique upstream limits in canonical order across concurrent observations", async () => { + let release: () => void = () => { + throw new Error("gate not initialized"); + }; + const gate = new Promise((resolve) => { + release = resolve; + }); + let calls = 0; + const options = { signal: new AbortController().signal }; + const analysis: AnalysisOperationPort = { + execute: async (_, parameters, forwarded) => { + expect(parameters).toEqual({ marker: true }); + expect(forwarded).toBe(options); + const call = calls++; + if (call === 0) await gate; + else release(); + return ok( + createAnalysisExecution([], provider, { + limitations: call === 0 ? ["first", "shared"] : ["second", "shared"], + }), + ); + }, + }; + const result = await executeComposedWorkflow(analysis, async (observed) => { + await Promise.all([ + observed.execute("list_strings", { marker: true }, options), + observed.execute("list_procedures", { marker: true }, options), + ]); + return ok({ count: 0 }); + }); + expect(calls).toBe(2); + expect(result).toEqual( + ok({ result: { count: 0 }, limitations: ["first", "second", "shared"] }), + ); +}); + +it("preserves a failed composed operation without an extra provider read", async () => { + const failure = err(new AnalysisCancelledError("list_strings")); + let calls = 0; + const result = await executeComposedWorkflow( + { + execute: async () => { + calls++; + return failure; + }, + }, + async (observed) => { + const observedResult = await observed.execute("list_strings", {}); + return observedResult.ok + ? ok(observedResult.value.result) + : observedResult; + }, + ); + expect(result).toBe(failure); + expect(calls).toBe(1); +}); + describe("workflow residual question retention", () => { it("keeps questions linked to workflow Evidence and records repeats idempotently", () => { const records = new InvestigationRecords(); @@ -24,6 +154,7 @@ describe("workflow residual question retention", () => { parameters: { procedure: "0x1000" }, result, upstreamProfile: undefined, + upstreamLimitations: [], }); expect(records.recordEvidence(evidence).ok).toBe(true); const input = { @@ -61,6 +192,7 @@ describe("workflow residual question retention", () => { parameters: {}, result, upstreamProfile: undefined, + upstreamLimitations: [], }); expect(records.recordEvidence(evidence).ok).toBe(true); @@ -88,6 +220,7 @@ describe("workflow residual question retention", () => { parameters: {}, result, upstreamProfile: undefined, + upstreamLimitations: [], }); expect(records.recordEvidence(evidence).ok).toBe(true); @@ -139,6 +272,7 @@ it.each([ parameters: { procedure: "0x1000" }, result, upstreamProfile: undefined, + upstreamLimitations: [], }); const records = new InvestigationRecords(); expect(records.recordEvidence(evidence).ok).toBe(true); diff --git a/src/application/WorkflowEvidence.ts b/src/application/WorkflowEvidence.ts index bf2530080..5bbd75455 100644 --- a/src/application/WorkflowEvidence.ts +++ b/src/application/WorkflowEvidence.ts @@ -2,7 +2,12 @@ import { REA_WORKFLOW_PROVIDER, workflowAnalysisProfile, } from "./InvestigationProviders.js"; -import type { AnalysisOperation } from "./AnalysisProvider.js"; +import type { + AnalysisOperation, + AnalysisOperationPort, +} from "./AnalysisProvider.js"; +import type { AnalysisError } from "../domain/analysisErrorBase.js"; +import { ok, type Result } from "../domain/result.js"; import type { AnalysisProfileCommitment } from "../domain/analysisProfile.js"; import type { WorkflowSnapshotRecordInput } from "./binary/BinarySessionRecords.js"; import { @@ -18,6 +23,37 @@ import { import type { JsonValue } from "../domain/jsonValue.js"; import type { UnknownRegistryPort } from "./investigation/InvestigationRecordPort.js"; import { UnknownRegistryError } from "../domain/unknownRegistryError.js"; +import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js"; + +/** Retain the limits of exactly the observations used by one composed call. */ +export const executeComposedWorkflow = async ( + analysis: AnalysisOperationPort, + execute: ( + observed: AnalysisOperationPort, + ) => Promise>, +): Promise< + Result< + { readonly result: JsonValue; readonly limitations: readonly string[] }, + AnalysisError + > +> => { + const limitations = new Set(); + const result = await execute({ + execute: async (operation, parameters, options) => { + const execution = await analysis.execute(operation, parameters, options); + if (execution.ok) + execution.value.limitations.forEach((limitation) => + limitations.add(limitation), + ); + return execution; + }, + }); + if (!result.ok) return result; + return ok({ + result: result.value, + limitations: [...limitations].sort(compareUnicodeCodePoints), + }); +}; /** Build the shared Evidence record for a successful composed binary workflow. */ export const createWorkflowEvidence = (input: { @@ -26,6 +62,7 @@ export const createWorkflowEvidence = (input: { readonly parameters: Readonly>; readonly result: JsonValue; readonly upstreamProfile: AnalysisProfileCommitment | undefined; + readonly upstreamLimitations: readonly string[]; }): Evidence => createEvidence(input.target, REA_WORKFLOW_PROVIDER, { operation: input.operation, @@ -40,7 +77,10 @@ export const createWorkflowEvidence = (input: { ), }), confidence: "derived", - limitations: ["Derived by an REA composed workflow."], + limitations: [ + "Derived by an REA composed workflow.", + ...input.upstreamLimitations, + ], ...(input.operation === "batch_decompile" ? { locations: batchProcedureLocations(input.result) } : {}), diff --git a/src/catalogIdentity.ts b/src/catalogIdentity.ts index 250cef4d5..774cfc00f 100644 --- a/src/catalogIdentity.ts +++ b/src/catalogIdentity.ts @@ -8,6 +8,7 @@ import { TOOL_CONTRACTS } from "./contracts/toolContracts.js"; import type { ToolContract } from "./contracts/toolContractTypes.js"; import { CLI_COMMAND_NAMES } from "./cliCommandNames.js"; import { compareUnicodeCodePoints } from "./domain/unicodeCodePointOrder.js"; +import { advertisedInputJsonSchema } from "./contracts/toolSchemaMetadata.js"; const sortByName = ( contracts: readonly ToolContract[], @@ -33,12 +34,7 @@ const createToolCatalog = (contracts: readonly ToolContract[]) => effects: { ...contract.effects }, returns_target_content: contract.returnsTargetContent, annotations: contract.annotations, - input_schema: { - type: "object", - ...z.toJSONSchema(contract.inputSchema, { - unrepresentable: "any", - }), - }, + input_schema: advertisedInputJsonSchema(contract), output_schema: { type: "object", ...z.toJSONSchema(contract.outputSchema, { diff --git a/src/config.test.ts b/src/config.test.ts index efa1b48fc..0230d552b 100644 --- a/src/config.test.ts +++ b/src/config.test.ts @@ -1,6 +1,9 @@ import { describe, expect, it } from "vitest"; -import { parseConfig } from "./config/parseConfig.js"; +import { + defaultHopperLauncherPath, + parseConfig, +} from "./config/parseConfig.js"; import { projectAnalysisError } from "./domain/analysisErrorProjection.js"; it("reports a configuration failure for a NUL in Hopper loader arguments", () => { @@ -324,3 +327,44 @@ describe("configuration failure diagnostics", () => { }); }); }); + +describe("default Linux Hopper launcher", () => { + const home = "/fixture/home"; + const withExecutables = + (paths: readonly string[]) => + (path: string): boolean => + paths.includes(path); + + it.each([["/opt/hopper-v4/bin/hopper"], ["/usr/bin/hopper"]])( + "selects an existing system launcher %s", + (path) => { + expect( + defaultHopperLauncherPath("linux", home, withExecutables([path])), + ).toBe(path); + }, + ); + + it("prefers REA-managed and user-local launchers over system paths", () => { + expect( + defaultHopperLauncherPath( + "linux", + home, + withExecutables([ + "/opt/hopper/bin/Hopper", + "/opt/hopper-v4/bin/hopper", + "/usr/bin/hopper", + ]), + ), + ).toBe("/opt/hopper/bin/Hopper"); + expect( + defaultHopperLauncherPath( + "linux", + home, + withExecutables([ + `${home}/.local/share/rea/hopper/bin/Hopper`, + "/usr/bin/hopper", + ]), + ), + ).toBe(`${home}/.local/share/rea/hopper/bin/Hopper`); + }); +}); diff --git a/src/config/environment.ts b/src/config/environment.ts index 121d46614..f5cc0373d 100644 --- a/src/config/environment.ts +++ b/src/config/environment.ts @@ -17,6 +17,7 @@ import { } from "./providerStartupTimeout.js"; import { mcpInputSchemaProfileSchema } from "./mcpInputSchemaProfile.js"; import { mcpResponseBudgetSchema } from "./mcpResponseBudget.js"; +import { parseLoaderArgs, parseStringArray } from "./parsers.js"; const environmentSchema = z.object({ REA_ANALYSIS_PROVIDER: analysisProviderSelectorSchema.default("auto"), @@ -74,9 +75,16 @@ const environmentSchema = z.object({ REA_REFERENCE_SECRET_PATTERNS_JSON: z.string().default("[]"), }); -export type Environment = z.infer; +type Environment = Omit< + z.infer, + "HOPPER_LOADER_ARGS_JSON" | "REA_REFERENCE_SECRET_PATTERNS_JSON" +> & { + readonly loaderArgs: readonly string[]; + readonly referenceSecretPatterns: readonly string[]; +}; -export const parseEnvironment = ( +/** Parse individual settings before layered registration fragments are merged. */ +export const parseEnvironmentValues = ( environment: Readonly>, ): Result => { const parsed = environmentSchema.safeParse(environment); @@ -93,5 +101,49 @@ export const parseEnvironment = ( }), ); } - return ok(parsed.data); + const { + HOPPER_LOADER_ARGS_JSON: encodedLoaderArgs, + REA_REFERENCE_SECRET_PATTERNS_JSON: encodedSecretPatterns, + ...env + } = parsed.data; + const loaderArgs = parseLoaderArgs(encodedLoaderArgs); + if (!loaderArgs.ok) return loaderArgs; + const secretPatterns = parseStringArray( + encodedSecretPatterns, + "REA_REFERENCE_SECRET_PATTERNS_JSON", + ); + if (!secretPatterns.ok) return secretPatterns; + return ok({ + ...env, + loaderArgs: loaderArgs.value, + referenceSecretPatterns: secretPatterns.value, + }); +}; + +/** Validate complete runtime semantics without probing the host or persisting defaults. */ +export const parseEnvironment = ( + environment: Readonly>, + // Registration expressions are not evaluated; their absence is not proven. + unresolvedSettings: ReadonlySet = new Set(), +): Result => { + const parsed = parseEnvironmentValues(environment); + if (!parsed.ok) return parsed; + const env = parsed.value; + // Ghidra's analyzeHeadless accepts -cspec only together with -processor. + if ( + env.REA_GHIDRA_COMPILER_SPEC_ID !== undefined && + env.REA_GHIDRA_LANGUAGE_ID === undefined && + !unresolvedSettings.has("REA_GHIDRA_LANGUAGE_ID") + ) + return err( + new ConfigurationError("Invalid REA environment configuration", { + settings: [ + { + setting: "REA_GHIDRA_COMPILER_SPEC_ID", + constraint: "requires REA_GHIDRA_LANGUAGE_ID", + }, + ], + }), + ); + return parsed; }; diff --git a/src/config/mcpOutputSchemaProfile.ts b/src/config/mcpOutputSchemaProfile.ts new file mode 100644 index 000000000..e981ac941 --- /dev/null +++ b/src/config/mcpOutputSchemaProfile.ts @@ -0,0 +1,34 @@ +import { z } from "zod"; + +import { ConfigurationError } from "../domain/configurationErrors.js"; +import { err, ok, type Result } from "../domain/result.js"; + +/** Select whether tools/list carries the large output contracts. */ +export const mcpOutputSchemaProfileSchema = z.enum(["full", "off"]); + +export type McpOutputSchemaProfile = z.infer< + typeof mcpOutputSchemaProfileSchema +>; + +const profileConstraint = + 'REA_MCP_OUTPUT_SCHEMA_PROFILE must be "full" (default) or "off". The off profile omits output schemas from tools/list while REA validates every successful result against its canonical tool contract and returns complete structured content.'; + +/** Parse the output schema advertisement selection at server composition. */ +export const parseMcpOutputSchemaProfile = ( + value: string | undefined, +): Result => { + if (value === undefined) return ok(undefined); + const parsed = mcpOutputSchemaProfileSchema.safeParse(value); + return parsed.success + ? ok(parsed.data) + : err( + new ConfigurationError(profileConstraint, { + settings: [ + { + setting: "REA_MCP_OUTPUT_SCHEMA_PROFILE", + constraint: profileConstraint, + }, + ], + }), + ); +}; diff --git a/src/config/parseConfig.ts b/src/config/parseConfig.ts index 757e02af0..415af6b3d 100644 --- a/src/config/parseConfig.ts +++ b/src/config/parseConfig.ts @@ -1,17 +1,17 @@ import { accessSync, constants } from "node:fs"; -import { join } from "node:path"; -import { ConfigurationError } from "../domain/configurationErrors.js"; -import { err, ok, type Result } from "../domain/result.js"; +import type { ConfigurationError } from "../domain/configurationErrors.js"; +import { ok, type Result } from "../domain/result.js"; import { parseEnvironment } from "./environment.js"; import { homeDirectoryFromEnvironment } from "./homeDirectory.js"; -import { parseStringArray, parseLoaderArgs } from "./parsers.js"; import type { AppConfig } from "./types.js"; +import { + linuxHopperLauncherCandidates, + REA_MANAGED_LINUX_HOPPER_LAUNCHER, +} from "../application/LinuxHopper.js"; const DEFAULT_HOPPER_LAUNCHER_PATH = "/Applications/Hopper Disassembler.app/Contents/MacOS/hopper"; -const SYSTEM_LINUX_HOPPER = "/opt/hopper/bin/Hopper"; - const executableAvailable = (path: string): boolean => { try { accessSync(path, constants.X_OK); @@ -21,15 +21,16 @@ const executableAvailable = (path: string): boolean => { } }; -const defaultHopperLauncherPath = ( +export const defaultHopperLauncherPath = ( platform: NodeJS.Platform, homeDirectory: string, executable: (path: string) => boolean = executableAvailable, ): string => { if (platform !== "linux") return DEFAULT_HOPPER_LAUNCHER_PATH; - if (executable(SYSTEM_LINUX_HOPPER)) return SYSTEM_LINUX_HOPPER; - const userLocal = join(homeDirectory, ".local/share/rea/hopper/bin/Hopper"); - return executable(userLocal) ? userLocal : SYSTEM_LINUX_HOPPER; + return ( + linuxHopperLauncherCandidates(homeDirectory).find(executable) ?? + REA_MANAGED_LINUX_HOPPER_LAUNCHER + ); }; /** Parse provider configuration and runtime prerequisites at the composition root. */ @@ -39,28 +40,6 @@ export const parseConfig = ( const parsed = parseEnvironment(environment); if (!parsed.ok) return parsed; const env = parsed.value; - const loaderArgs = parseLoaderArgs(env.HOPPER_LOADER_ARGS_JSON); - if (!loaderArgs.ok) return loaderArgs; - const secretPatterns = parseStringArray( - env.REA_REFERENCE_SECRET_PATTERNS_JSON, - "REA_REFERENCE_SECRET_PATTERNS_JSON", - ); - if (!secretPatterns.ok) return secretPatterns; - // Ghidra's analyzeHeadless accepts -cspec only together with -processor. - if ( - env.REA_GHIDRA_COMPILER_SPEC_ID !== undefined && - env.REA_GHIDRA_LANGUAGE_ID === undefined - ) - return err( - new ConfigurationError("Invalid REA environment configuration", { - settings: [ - { - setting: "REA_GHIDRA_COMPILER_SPEC_ID", - constraint: "requires REA_GHIDRA_LANGUAGE_ID", - }, - ], - }), - ); return ok({ ...(env.REA_MCP_MAX_RESPONSE_BYTES === undefined ? {} @@ -104,8 +83,10 @@ export const parseConfig = ( hopperStartupTimeoutMs: env.REA_HOPPER_STARTUP_TIMEOUT_MS, hopperTargetPath: env.HOPPER_TARGET_PATH, hopperTargetKind: env.HOPPER_TARGET_KIND, - hopperLoaderArgs: loaderArgs.value, + hopperLoaderArgs: env.loaderArgs, logLevel: env.REA_LOG_LEVEL, - referenceSourcePolicy: { secretPatterns: secretPatterns.value }, + referenceSourcePolicy: { + secretPatterns: env.referenceSecretPatterns, + }, }); }; diff --git a/src/contracts/adb/adbToolContracts.ts b/src/contracts/adb/adbToolContracts.ts index e72b603d7..dd5859617 100644 --- a/src/contracts/adb/adbToolContracts.ts +++ b/src/contracts/adb/adbToolContracts.ts @@ -127,7 +127,7 @@ export const ADB_TOOL_CONTRACTS = [ name: "push_adb_file", ...toolContractMetadata("push_adb_file"), description: - "Push one local file to an absolute device path. A device-mutating tool: existing device files are refused unless overwrite is explicitly requested, and the transfer is verified by comparing the local digest with the device's own sha256sum when the device provides one.", + "Push one local file to an absolute device path. A device-mutating tool: checks for an existing device file before transfer and refuses it unless overwrite is explicitly requested; this check is not an atomic create and another process can create the path before adb writes. When the device provides sha256sum, the transfer is verified by comparing it with the local digest.", kind: "adb-provider", inputSchema: adbInputSchemas.push_adb_file, outputSchema: evidenceResultOf(adbResultSchemas.push_adb_file), @@ -224,7 +224,7 @@ export const ADB_TOOL_CONTRACTS = [ name: "inspect_adb_window", ...toolContractMetadata("inspect_adb_window"), description: - "Report the window manager's current focus declarations (`dumpsys window windows`: mCurrentFocus and mFocusedApp) — what is on screen at read time without any interaction. Read-only.", + "Report the window manager's current focus declarations (mCurrentFocus and mFocusedApp from `dumpsys window windows`, or from `dumpsys window displays` when the windows section declares neither, as on Android 16) — what is on screen at read time without any interaction. Read-only.", kind: "adb-provider", inputSchema: adbInputSchemas.inspect_adb_window, outputSchema: evidenceResultOf(adbResultSchemas.inspect_adb_window), diff --git a/src/contracts/compactInputSchema.test.ts b/src/contracts/compactInputSchema.test.ts index 3fcfc4265..212d3dbb4 100644 --- a/src/contracts/compactInputSchema.test.ts +++ b/src/contracts/compactInputSchema.test.ts @@ -162,6 +162,24 @@ describe("compact advertised input schema", () => { expect(compact(input)).toBe(canonical(input)); }); + it("does not reintroduce unsupported dialect keywords in reduced schemas", () => { + let nested: Record = { type: "string" }; + for (let level = 0; level < 12; level += 1) + nested = { type: "object", properties: { child: nested } }; + const { schema, presentation } = compactAdvertisedInputSchema( + { + $schema: "https://json-schema.org/draft/2020-12/schema", + type: "object", + properties: { deep: nested }, + dependentRequired: { detail: ["target"] }, + }, + 1_500, + ); + expect(presentation).toBe("reduced"); + expect(schema.$schema).toBe("https://json-schema.org/draft/2020-12/schema"); + expect(schema).not.toHaveProperty("dependentRequired"); + }); + it("keeps unconstrained JSON values explicit for client type inference", () => { const { schema } = compactAdvertisedInputSchema( { diff --git a/src/contracts/compactInputSchema.ts b/src/contracts/compactInputSchema.ts index 321ea8323..f756fdda4 100644 --- a/src/contracts/compactInputSchema.ts +++ b/src/contracts/compactInputSchema.ts @@ -37,6 +37,22 @@ const isSchemaObject = (value: unknown): value is Record => const serializedBytes = (value: unknown): number => Buffer.byteLength(JSON.stringify(value), "utf8"); +const schemaDepth = (value: unknown, depth = 0): number => { + if (Array.isArray(value)) + return Math.max( + depth, + ...value.map((child) => schemaDepth(child, depth + 1)), + ); + if (!isSchemaObject(value)) return depth; + return Math.max( + depth, + ...Object.values(value).map((child) => schemaDepth(child, depth + 1)), + ); +}; + +/** Strict clients commonly reject advertised schemas deeper than ten nodes. */ +const COMPACT_SCHEMA_MAX_DEPTH = 10; + /** Rewrite every schema-valued position; literal data stays opaque. */ const rebuildSchemaPositions = ( node: unknown, @@ -240,12 +256,24 @@ const reduceToShallowProperties = ( const properties = isSchemaObject(schema.properties) ? schema.properties : {}; const description = [ typeof schema.description === "string" ? schema.description : undefined, + ...Object.entries( + isSchemaObject(schema.dependentRequired) + ? schema.dependentRequired + : isSchemaObject(schema.dependencies) + ? schema.dependencies + : {}, + ).flatMap(([property, partners]) => + Array.isArray(partners) && + partners.every((partner) => typeof partner === "string") + ? [ + `When ${property} is provided, also provide ${partners.join(" and ")}.`, + ] + : [], + ), REDUCED_ANNOTATION, ] .filter((text) => text !== undefined) .join(" "); - const $schema = - typeof schema.$schema === "string" ? { $schema: schema.$schema } : {}; const shallow = Object.fromEntries( Object.entries(properties).map(([name, value]) => { const types = detectedTypes(value, schema); @@ -261,7 +289,7 @@ const reduceToShallowProperties = ( }), ); return { - ...$schema, + ...(typeof schema.$schema === "string" ? { $schema: schema.$schema } : {}), type: "object", description, properties: shallow, @@ -271,9 +299,6 @@ const reduceToShallowProperties = ( ...(typeof schema.minProperties === "number" ? { minProperties: schema.minProperties } : {}), - ...(isSchemaObject(schema.dependentRequired) - ? { dependentRequired: { ...schema.dependentRequired } } - : {}), ...(schema.additionalProperties === false ? { additionalProperties: false } : {}), @@ -292,7 +317,8 @@ export const compactAdvertisedInputSchema = ( budgetBytes: number, ): CompactedInputSchema => { const inline = inlineWithinBudget(stripAnnotations(schema), budgetBytes); - if (inline !== undefined) return { schema: inline, presentation: "inline" }; + if (inline !== undefined && schemaDepth(inline) <= COMPACT_SCHEMA_MAX_DEPTH) + return { schema: inline, presentation: "inline" }; const reduced = reduceToShallowProperties(schema); if (serializedBytes(reduced) <= budgetBytes) return { schema: reduced, presentation: "reduced" }; diff --git a/src/contracts/enhancedToolContracts.ts b/src/contracts/enhancedToolContracts.ts index be8c7dac9..9b124f818 100644 --- a/src/contracts/enhancedToolContracts.ts +++ b/src/contracts/enhancedToolContracts.ts @@ -62,7 +62,7 @@ export const ENHANCED_TOOL_CONTRACTS = [ ), enhanced( "binary_overview", - "Return native-binary metadata, every segment with its length, and exhaustive procedure/string counts. When every provider-reported segment agrees on an analysis image base, return it as image_base so callers can relate provider addresses to their loader coordinates. Use search_procedures, list_procedures, or analyze_function directly when you already know which procedure to inspect.", + "Return the bound document and exhaustive procedure/string counts, plus every segment and segment_count when the provider supports segment enumeration. If segment fields are omitted, segment coverage is unavailable rather than empty. When every observed segment agrees on an analysis image base, return it as image_base so callers can relate provider addresses to their loader coordinates. Use search_procedures, list_procedures, or analyze_function directly when you already know which procedure to inspect.", enhancedInputSchemas.binary_overview, ), ...FUNCTION_WORKFLOW_TOOL_CONTRACTS, diff --git a/src/contracts/firmware/firmwareToolContracts.ts b/src/contracts/firmware/firmwareToolContracts.ts index c93c8f903..00b1e373b 100644 --- a/src/contracts/firmware/firmwareToolContracts.ts +++ b/src/contracts/firmware/firmwareToolContracts.ts @@ -30,7 +30,7 @@ export const FIRMWARE_TOOL_CONTRACTS = [ ...toolContractMetadata("extract_firmware"), kind: "firmware-provider", description: - "Extract caller-selected firmware bytes with Unblob 26.6.x (verified with 26.6.4) into an absent output directory. Returns verified child paths/digests, task derivations, unknown chunks and extraction diagnostics inline. Preserves the default sandbox; publishes regular files only. Linux, caller-supplied tool, format-specific extractors and util-linux prlimit required. No mounts or extracted-code execution. Select a returned native file with open_binary for Ghidra analysis; decompressed offsets are not original-file addresses.", + "Extract caller-selected firmware bytes with Unblob 26.6.x (verified with 26.6.4) into an absent output directory. Returns verified child paths/digests, task derivations, unknown chunks, extraction diagnostics and explicit extraction_gaps rows naming unextracted regions, missing extractor dependencies and a continuation_id referencing shared extraction_continuations recovery instructions. Preserves the default sandbox; publishes regular files only. Linux, caller-supplied tool, format-specific extractors and util-linux prlimit required. No mounts or extracted-code execution. Select a returned native file with open_binary for Ghidra analysis; decompressed offsets are not original-file addresses.", inputSchema: firmwareInputSchemas.extract_firmware, outputSchema: evidenceResultOf(firmwareResultSchemas.extract_firmware), examples: [ diff --git a/src/contracts/inputSchemaPresentation.ts b/src/contracts/inputSchemaPresentation.ts index 22ad0ae30..3ed83907a 100644 --- a/src/contracts/inputSchemaPresentation.ts +++ b/src/contracts/inputSchemaPresentation.ts @@ -174,17 +174,6 @@ const flattenRootObjectUnion = ( const required = sharedRequiredProperties(branches); // A field unique to a group still needs that group's required partners. // Counting properties alone would let optional fields pad an incomplete pair. - const dependencies = Object.fromEntries( - [...variants.keys()].flatMap((name) => { - const owners = branches.filter((branch) => - Object.hasOwn(branch.properties ?? {}, name), - ); - const partners = sharedRequiredProperties(owners).filter( - (partner) => partner !== name && !required.includes(partner), - ); - return partners.length > 0 ? [[name, partners]] : []; - }), - ); const fewestRequired = Math.min( ...branches.map((branch) => requiredProperties(branch).length), ); @@ -196,8 +185,13 @@ const flattenRootObjectUnion = ( ? `Provide a complete input group: ${groups.join("; ")}.` : undefined, ].filter((text) => text !== undefined); + const { + dependencies: _dependencies, + dependentRequired: _dependentRequired, + ...presentation + } = rest; return { - ...rest, + ...presentation, type: "object", ...(description.length > 0 ? { description: description.join(" ") } : {}), properties: Object.fromEntries( @@ -208,13 +202,6 @@ const flattenRootObjectUnion = ( ), ...(required.length > 0 ? { required } : {}), ...(fewestRequired > 0 ? { minProperties: fewestRequired } : {}), - ...(Object.keys(dependencies).length > 0 - ? { - [root.$schema === "http://json-schema.org/draft-07/schema#" - ? "dependencies" - : "dependentRequired"]: dependencies, - } - : {}), ...(branches.every((branch) => branch.additionalProperties === false) ? { additionalProperties: false } : {}), @@ -228,7 +215,36 @@ export const presentInputJsonSchema = ( ): Record => { const flattened = flattenRootObjectUnion(root); const described = describeSchema(flattened, flattened, describeProperty); - return isObject(described) ? described : flattened; + if (!isObject(described)) return flattened; + const { + dependencies: _dependencies, + dependentRequired: _dependentRequired, + ...presentation + } = described; + const rootDependencies = isObject(described.dependentRequired) + ? described.dependentRequired + : isObject(described.dependencies) + ? described.dependencies + : undefined; + const dependencyGuidance = + rootDependencies === undefined + ? [] + : Object.entries(rootDependencies).flatMap(([property, partners]) => + Array.isArray(partners) && + partners.every((partner) => typeof partner === "string") + ? [ + `When ${property} is provided, also provide ${partners.join(" and ")}.`, + ] + : [], + ); + if (dependencyGuidance.length === 0) return presentation; + const description = [ + typeof presentation.description === "string" + ? presentation.description + : undefined, + ...dependencyGuidance, + ].filter((text) => text !== undefined); + return { ...presentation, description: description.join(" ") }; }; /** diff --git a/src/contracts/providerSelection.ts b/src/contracts/providerSelection.ts index d43ee5852..2a9e1bc0f 100644 --- a/src/contracts/providerSelection.ts +++ b/src/contracts/providerSelection.ts @@ -20,6 +20,7 @@ export const PROVIDER_REJECTION_CODES = [ "not_configured", "executable_missing", "runtime_missing", + "runtime_unavailable", "unsupported_host", "unsupported_version", "target_kind_unsupported", diff --git a/src/contracts/toolEffects.test.ts b/src/contracts/toolEffects.test.ts index 7d23db92d..fbb92e2ab 100644 --- a/src/contracts/toolEffects.test.ts +++ b/src/contracts/toolEffects.test.ts @@ -1,7 +1,11 @@ import { describe, expect, it } from "vitest"; import { TOOL_CONTRACTS } from "./toolContracts.js"; -import { TOOL_EFFECTS, annotationsFromEffects } from "./toolEffects.js"; +import { + TOOL_EFFECTS, + annotationsFromEffects, + type ToolEffects, +} from "./toolEffects.js"; /** * Descriptions that name running, launching, executing, starting, installing, @@ -159,8 +163,19 @@ describe("tool effects", () => { }); it("derives destructiveHint from each destructive effect alone", () => { - const base = TOOL_EFFECTS["binary_session"]; - if (base === undefined) throw new Error("binary_session is unaudited"); + // Keep annotation algebra independent of any operation's audited effects. + const base: ToolEffects = { + mutatesTarget: false, + mutatesSession: false, + writesFilesystem: false, + launchesProcess: false, + accessesNetwork: false, + changesUiState: false, + mayDiscardData: false, + executesForeignCode: false, + mutatesDevice: false, + idempotent: true, + }; expect(annotationsFromEffects(base)).toEqual({ readOnlyHint: true, destructiveHint: false, diff --git a/src/contracts/toolEffects.ts b/src/contracts/toolEffects.ts index c64edefca..a010467be 100644 --- a/src/contracts/toolEffects.ts +++ b/src/contracts/toolEffects.ts @@ -677,7 +677,7 @@ export const TOOL_EFFECTS: Readonly> = { mayDiscardData: true, idempotent: false, }), - binary_session: effects(), + binary_session: effects({ launchesProcess: true }), export_evidence_bundle: effects({ writesFilesystem: true, mayDiscardData: true, diff --git a/src/contracts/toolInputReferences.test.ts b/src/contracts/toolInputReferences.test.ts index f75a06531..24985ed21 100644 --- a/src/contracts/toolInputReferences.test.ts +++ b/src/contracts/toolInputReferences.test.ts @@ -40,7 +40,41 @@ it.each(["draft-2020-12", "draft-07"] as const)( }, ); -describe("advertised input references", () => { +describe("advertised object dependencies", () => { + it.each([ + ["draft-07", "dependencies"], + ["draft-2020-12", "dependentRequired"], + ] as const)( + "presents object-group dependencies as guidance without unsupported dialect keywords (%s)", + (dialect, dependencyKeyword) => { + const advertised = presentInputJsonSchema( + { + $schema: + dialect === "draft-07" + ? "http://json-schema.org/draft-07/schema#" + : "https://json-schema.org/draft/2020-12/schema", + type: "object", + properties: { url: { type: "string" }, body: { type: "string" } }, + required: ["body"], + [dependencyKeyword]: { body: ["url"] }, + }, + () => undefined, + ); + expect(advertised.$schema).toBe( + dialect === "draft-07" + ? "http://json-schema.org/draft-07/schema#" + : "https://json-schema.org/draft/2020-12/schema", + ); + expect(advertised).not.toHaveProperty("dependencies"); + expect(advertised).not.toHaveProperty("dependentRequired"); + expect(advertised.description).toContain( + "When body is provided, also provide url.", + ); + }, + ); +}); + +describe("advertised production selector contracts", () => { it.each(["draft-2020-12", "draft-07"] as const)( "preserves production native-selector constraints in %s", (target) => { @@ -89,7 +123,9 @@ describe("advertised input references", () => { expect(validate(unexpected)).toBe(false); }, ); +}); +describe("advertised schema metadata and validation", () => { it.each(["draft-2020-12", "draft-07"] as const)( "retains specialized descriptions, literal metadata and strict validation in %s", (target) => { @@ -339,12 +375,20 @@ describe("root input presentation", () => { { before: "first" }, { unexpected: true }, { before_scenario: 1, after_scenario: "wrong" }, - { before_scenario: 1, normalization: true }, - { after_scenario: 2, normalization: true }, ]) { expect(schema.safeParse(input).success).toBe(false); expect(validate(input)).toBe(false); } + // The vendor-safe advertisement uses group guidance instead of + // dependentRequired; the canonical runtime parser keeps exact unions. + expect( + schema.safeParse({ before_scenario: 1, normalization: true }).success, + ).toBe(false); + expect(validate({ before_scenario: 1, normalization: true })).toBe(true); + expect( + schema.safeParse({ after_scenario: 2, normalization: true }).success, + ).toBe(false); + expect(validate({ after_scenario: 2, normalization: true })).toBe(true); // Cross-group exclusion is enforced only by the canonical runtime parser. const mixed = { before: "first", diff --git a/src/contracts/toolSchemaMetadata.ts b/src/contracts/toolSchemaMetadata.ts index 3034b8354..093f4845c 100644 --- a/src/contracts/toolSchemaMetadata.ts +++ b/src/contracts/toolSchemaMetadata.ts @@ -94,6 +94,14 @@ export const toolInputSchemaWithMetadata = ( }, ); +/** Return the same complete input JSON Schema projection registered on MCP. */ +export const advertisedInputJsonSchema = ( + contract: Contract, +): Record => + toolInputSchemaWithMetadata(contract)["~standard"].jsonSchema.input({ + target: "draft-2020-12", + }) as Record; + /** Share output definitions and content-bound validator identities on the wire. */ export const toolOutputSchemaWithMetadata = ( contract: Contract, diff --git a/src/domain/analysisErrorCore.ts b/src/domain/analysisErrorCore.ts index 380b440a7..71a77a647 100644 --- a/src/domain/analysisErrorCore.ts +++ b/src/domain/analysisErrorCore.ts @@ -83,17 +83,29 @@ export interface AnalysisInputIssue { readonly maximum?: number; } +/** Refusal-specific context a producer projects with an output failure. */ +export interface AnalysisOutputErrorOptions extends AnalysisErrorOptions { + /** Caller-facing explanation for a deterministic, diagnosable refusal. */ + readonly userMessage?: string; + readonly remediationAction?: string; + readonly cleanupOnly?: boolean; +} + /** Provider output failed provider-neutral application parsing. */ export class AnalysisOutputError extends AnalysisError { readonly _tag = "AnalysisOutputError"; + override readonly userMessage: string | undefined; + readonly remediationAction: string | undefined; readonly cleanupOnly: boolean; constructor( readonly operation: string, readonly reason: string, - options?: AnalysisErrorOptions & { readonly cleanupOnly?: boolean }, + options?: AnalysisOutputErrorOptions, ) { super(`Invalid analysis output for ${operation}: ${reason}`, options); + this.userMessage = options?.userMessage; + this.remediationAction = options?.remediationAction; this.cleanupOnly = options?.cleanupOnly ?? false; } } diff --git a/src/domain/analysisErrorPresentation.ts b/src/domain/analysisErrorPresentation.ts index 19c3a9002..54ad586c5 100644 --- a/src/domain/analysisErrorPresentation.ts +++ b/src/domain/analysisErrorPresentation.ts @@ -3,6 +3,7 @@ import { AnalysisArtifactChangedError, AnalysisCapabilityUnavailableError, AnalysisInputError, + AnalysisOutputError, AnalysisUnsupportedTargetError, AnalysisResourceConstraintError, } from "./analysisErrorCore.js"; @@ -107,6 +108,11 @@ export const analysisErrorRemediationAction = ( return `Correct ${[...new Set(error.settings.map(({ setting }) => setting))].join(", ")} in the environment that starts REA (your shell or the MCP client's registration), then rerun.`; if (error instanceof AnalysisInputError) return "Correct the listed arguments and retry."; + if ( + error instanceof AnalysisOutputError && + error.remediationAction !== undefined + ) + return error.remediationAction; if (error instanceof UnknownRegistryError && error.reason === "not-found") return "Check that the unknown_id belongs to this session, then retry."; return analysisErrorUserMessage(error); diff --git a/src/domain/analysisErrorProjection.ts b/src/domain/analysisErrorProjection.ts index fbd1b5386..2884bcfcb 100644 --- a/src/domain/analysisErrorProjection.ts +++ b/src/domain/analysisErrorProjection.ts @@ -273,7 +273,13 @@ const requestErrorDetails = ( boundary: "filesystem-read", }; if (error instanceof AnalysisOutputError) - return { operation: error.operation, reason: error.reason }; + return { + operation: error.operation, + reason: error.reason, + ...(error.diagnostics === undefined + ? {} + : { diagnostics: error.diagnostics }), + }; if (error instanceof AnalysisInputError && error.issues.length > 0) return { operation: error.operation, diff --git a/src/domain/analysisView/analysisView.ts b/src/domain/analysisView/analysisView.ts index b08001e35..c15d15a12 100644 --- a/src/domain/analysisView/analysisView.ts +++ b/src/domain/analysisView/analysisView.ts @@ -62,6 +62,13 @@ import { machoViewRequestSchema, projectMachoView, } from "./machoView.js"; +import { + interfaceBuilderSummarySchema, + interfaceBuilderViewContextSchema, + interfaceBuilderViewItemsSchema, + interfaceBuilderViewRequestSchema, + projectInterfaceBuilderView, +} from "./interfaceBuilderView.js"; import { projectWebNetworkCaptureView, captureViewRequestSchema, @@ -206,6 +213,7 @@ export const analysisViewRequestSchema = z.discriminatedUnion("kind", [ inventoryViewSchema, wasmViewRequestSchema, machoViewRequestSchema, + interfaceBuilderViewRequestSchema, evmViewRequestSchema, managedViewRequestSchema, androidViewRequestSchema, @@ -326,6 +334,7 @@ const viewResultBase = { "inventory_artifact", "inspect_wasm_artifact", "inspect_macho", + "decode_interface_builder", "inspect_evm_interface", "inspect_managed_artifact", "inspect_web_network_capture", @@ -382,6 +391,13 @@ export const machoSummaryViewResultSchema = z.strictObject({ summary: machoSummarySchema, }); +/** Decoded documents and graph counts for Interface Builder's selected delivery. */ +export const interfaceBuilderSummaryViewResultSchema = z.strictObject({ + ...summaryViewResultBase, + parent_operation: z.literal("decode_interface_builder"), + summary: interfaceBuilderSummarySchema, +}); + /** Exact inferred-interface summary for EVM's caller-selected delivery. */ export const evmInterfaceSummaryViewResultSchema = z.strictObject({ ...summaryViewResultBase, @@ -444,6 +460,7 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [ inventorySummarySchema, wasmSummarySchema, machoSummarySchema, + interfaceBuilderSummarySchema, evmSummarySchema, managedSummarySchema, captureSummarySchema, @@ -514,6 +531,14 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [ ...machoViewContextSchema.shape, parent_operation: z.literal("inspect_macho"), }), + z.strictObject({ + kind: z.literal("interface-builder"), + view: interfaceBuilderViewRequestSchema, + items: interfaceBuilderViewItemsSchema, + ...viewResultBase, + ...interfaceBuilderViewContextSchema.shape, + parent_operation: z.literal("decode_interface_builder"), + }), z.strictObject({ kind: z.literal("evm"), view: evmViewRequestSchema, @@ -611,6 +636,10 @@ export type UnsignedAnalysisView = > | Omit, "view_digest"> | Omit, "view_digest"> + | Omit< + Extract, + "view_digest" + > | Omit, "view_digest"> | Omit, "view_digest"> | Omit, "view_digest"> @@ -674,7 +703,7 @@ const unsupportedParent = ( new AnalysisUnsupportedTargetError( INSPECT_ANALYSIS_VIEW_OPERATION, parentArtifactPath(parent), - `Parent Evidence operation ${JSON.stringify(parent.operation)} does not support analysis views. Use completed binary layout, JavaScript application, native function, artifact inventory/inspection, WASM, Mach-O, EVM, managed artifact, Android package, Dart AOT, or web network capture Evidence.`, + `Parent Evidence operation ${JSON.stringify(parent.operation)} does not support analysis views. Use completed binary layout, JavaScript application, native function, artifact inventory/inspection, WASM, Mach-O, Interface Builder, EVM, managed artifact, Android package, Dart AOT, or web network capture Evidence.`, ); const parentArtifactPath = (parent: AnalysisViewParent): string => { @@ -702,6 +731,7 @@ const parentArtifactPath = (parent: AnalysisViewParent): string => { const retainedViewProjectors = new Map([ ["inspect_wasm_artifact", projectWasmArtifactView], ["inspect_macho", projectMachoView], + ["decode_interface_builder", projectInterfaceBuilderView], ["inspect_evm_interface", projectEvmInterfaceView], ["inspect_managed_artifact", projectManagedArtifactView], ["inspect_android_package", projectAndroidPackageView], diff --git a/src/domain/analysisView/analysisViewParent.ts b/src/domain/analysisView/analysisViewParent.ts new file mode 100644 index 000000000..0a2b57180 --- /dev/null +++ b/src/domain/analysisView/analysisViewParent.ts @@ -0,0 +1,25 @@ +import type { z } from "zod"; +import type { AnalysisError } from "../analysisErrorBase.js"; +import { analysisInputErrorFromIssues } from "../inputIssueProjection.js"; +import { err, ok, type Result } from "../result.js"; +import type { AnalysisViewParent } from "./analysisView.js"; + +/** Parse a parent's result with its owning schema; issues point into the view source. */ +export const parseAnalysisViewResult = ( + schema: Schema, + parent: AnalysisViewParent, +): Result, AnalysisError> => { + const parsed = schema.safeParse(parent.normalizedResult); + if (parsed.success) return ok(parsed.data); + return err( + analysisInputErrorFromIssues( + "inspect_analysis_view", + parsed.error.issues.map((issue) => ({ + ...issue, + path: ["source", "normalized_result", ...issue.path], + })), + { source: { normalized_result: parent.normalizedResult } }, + { cause: parsed.error }, + ), + ); +}; diff --git a/src/mcpDoctor.ts b/src/mcpDoctor.ts index d7bf1e366..ee2529ff1 100644 --- a/src/mcpDoctor.ts +++ b/src/mcpDoctor.ts @@ -280,6 +280,7 @@ export const runProductionMcpDoctorCli = async ( arguments_: readonly string[], input: { readonly dispatcherPath: string; readonly packageRoot: string }, ): Promise<{ readonly output: string; readonly exitCode: number }> => { + const started = performance.now(); const parsed = parseOutputArguments(arguments_); if (!parsed.ok) { if (parsed.help) @@ -298,8 +299,18 @@ export const runProductionMcpDoctorCli = async ( cwd: input.packageRoot, environment: process.env, }); + const outputValue = parsed.fullOutput + ? { + ok: result.healthy, + ...(result.healthy ? { data: result } : { error: result }), + meta: { + command: MCP_DOCTOR_COMMAND, + duration: `${Math.round(performance.now() - started)}ms`, + }, + } + : result; return { - output: `${Formatter.format(result, parsed.format)}\n`, + output: `${Formatter.format(outputValue, parsed.format)}\n`, exitCode: result.healthy ? 0 : 1, }; }; @@ -369,7 +380,11 @@ const parseIdentityToolResult = ( * flag is an output selector declared in `MCP_DOCTOR_OPTIONS`. */ type McpDoctorArguments = - | { readonly ok: true; readonly format: OutputFormat } + | { + readonly ok: true; + readonly format: OutputFormat; + readonly fullOutput: boolean; + } | { readonly ok: false; readonly help: true } | { readonly ok: false; @@ -387,6 +402,7 @@ const parseOutputArguments = ( // masking the typo with usage. if (hasHelpInOptionPosition(arguments_)) return { ok: false, help: true }; let format: OutputFormat = "toon"; + let fullOutput = false; for (let index = 0; index < arguments_.length; index += 1) { const argument = arguments_[index]; if (argument === "--json") { @@ -409,7 +425,10 @@ const parseOutputArguments = ( format = candidate; continue; } - if (argument === "--full-output") continue; + if (argument === "--full-output") { + fullOutput = true; + continue; + } if (HELP_FLAGS.has(argument ?? "")) return { ok: false, help: true }; return { ok: false, @@ -418,7 +437,7 @@ const parseOutputArguments = ( message: `Unknown mcp doctor option: ${argument ?? "missing"}`, }; } - return { ok: true, format }; + return { ok: true, format, fullOutput }; }; /** diff --git a/src/server/EvidenceMcpServer.test.ts b/src/server/EvidenceMcpServer.test.ts new file mode 100644 index 000000000..dedbfe5ce --- /dev/null +++ b/src/server/EvidenceMcpServer.test.ts @@ -0,0 +1,24 @@ +import { describe, expect, it } from "vitest"; + +import { forgetPendingMcpRequest } from "./EvidenceMcpServer.js"; + +describe("MCP request bookkeeping", () => { + it("releases list, tool, and completion state when cancellation arrives", () => { + const completion = new AbortController(); + const pendingTools = new Map([[7, "inspect_artifact"]]); + const pendingToolLists = new Set([7]); + const pendingCompletions = new Map([[7, completion]]); + + forgetPendingMcpRequest( + pendingTools, + pendingToolLists, + pendingCompletions, + 7, + ); + + expect(pendingTools.has(7)).toBe(false); + expect(pendingToolLists.has(7)).toBe(false); + expect(completion.signal.aborted).toBe(true); + expect(pendingCompletions.has(7)).toBe(false); + }); +}); diff --git a/src/server/EvidenceMcpServer.ts b/src/server/EvidenceMcpServer.ts index f42fe9de6..28618adce 100644 --- a/src/server/EvidenceMcpServer.ts +++ b/src/server/EvidenceMcpServer.ts @@ -60,6 +60,47 @@ const hasAdvertisedInputJsonSchema = (value: unknown): boolean => { ); }; +const isRecord = (value: unknown): value is Record => + typeof value === "object" && value !== null && !Array.isArray(value); + +/** Remove only client-facing output schemas; SDK registrations stay canonical. */ +const omitOutputSchemasFromToolsList = ( + message: JSONRPCMessage, + requestId: RequestId, +): JSONRPCMessage => { + if ( + !isJSONRPCResultResponse(message) || + message.id !== requestId || + !isRecord(message.result) || + !Array.isArray(message.result.tools) + ) + return message; + return { + ...message, + result: { + ...message.result, + tools: message.result.tools.map((tool: unknown) => { + if (!isRecord(tool)) return tool; + const { outputSchema: _outputSchema, ...advertisedTool } = tool; + return advertisedTool; + }), + }, + } as JSONRPCMessage; +}; + +/** Release every transport-owned reference for a cancelled/completed request. */ +export const forgetPendingMcpRequest = ( + pendingTools: Map, + pendingToolLists: Set, + pendingCompletions: Map, + id: RequestId, +): void => { + pendingTools.delete(id); + pendingToolLists.delete(id); + pendingCompletions.get(id)?.abort(); + pendingCompletions.delete(id); +}; + /** Bind oversized MCP error recovery to this server's Evidence ledger. */ export class EvidenceMcpServer extends McpServer { constructor( @@ -72,15 +113,15 @@ export class EvidenceMcpServer extends McpServer { private readonly compactPresentation: | CompactInputSchemaPresentation | undefined = undefined, + private readonly omitOutputSchemas = false, ) { super(info, options); } /** - * Compact the advertised input schema of every registered tool when the - * compact profile is active. The registered canonical schema, its parser, - * and output schemas are untouched; only the advertised JSON Schema - * projection is rendered within the provider's per-schema size budget. + * Compact advertised input schemas and optionally omit output-schema + * advertisements. Canonical parsers and contracts remain registered; the + * output-schema-off path validates successful structured results locally. * The signatures mirror the SDK's registrations so call sites keep their * inferred handler argument types. */ @@ -114,11 +155,11 @@ export class EvidenceMcpServer extends McpServer { compactAdvertisedInputSchema(projected, budget).schema, ) : undefined; + const registeredConfig = + inputSchema === undefined ? config : { ...config, inputSchema }; return super.registerTool( name, - (inputSchema === undefined - ? config - : { ...config, inputSchema }) as never, + registeredConfig as never, callback as never, ); } @@ -126,7 +167,12 @@ export class EvidenceMcpServer extends McpServer { /** Preserve the SDK transport lifecycle while retaining oversized errors. */ override connect(transport: Transport): Promise { return super.connect( - new ErrorEvidenceTransport(transport, this.recordEvidence, this.delivery), + new ErrorEvidenceTransport( + transport, + this.recordEvidence, + this.delivery, + this.omitOutputSchemas, + ), ); } } @@ -137,6 +183,7 @@ class ErrorEvidenceTransport implements Transport { onmessage: Transport["onmessage"]; readonly hasPerRequestStream?: boolean; private readonly pendingTools = new Map(); + private readonly pendingToolLists = new Set(); private readonly pendingCompletions = new Map(); constructor( @@ -145,6 +192,7 @@ class ErrorEvidenceTransport implements Transport { | EvidenceWriter["recordEvidence"] | undefined, private readonly delivery: ToolResultDelivery, + private readonly omitOutputSchemas: boolean, ) { if (transport.hasPerRequestStream !== undefined) this.hasPerRequestStream = transport.hasPerRequestStream; @@ -155,6 +203,8 @@ class ErrorEvidenceTransport implements Transport { typeof message.params?.["name"] === "string" ) this.pendingTools.set(message.id, message.params["name"]); + if (isJSONRPCRequest(message) && message.method === "tools/list") + this.pendingToolLists.add(message.id); if (isJSONRPCRequest(message) && message.method === "completion/complete") this.pendingCompletions.set(message.id, new AbortController()); // The SDK suppresses replies after cancellation, so send() cannot release @@ -164,11 +214,13 @@ class ErrorEvidenceTransport implements Transport { message.method === "notifications/cancelled" ) { const id = message.params?.["requestId"]; - if (typeof id === "string" || typeof id === "number") { - this.pendingTools.delete(id); - this.pendingCompletions.get(id)?.abort(); - this.pendingCompletions.delete(id); - } + if (typeof id === "string" || typeof id === "number") + forgetPendingMcpRequest( + this.pendingTools, + this.pendingToolLists, + this.pendingCompletions, + id, + ); } const completion = isJSONRPCRequest(message) && message.method === "completion/complete" @@ -183,6 +235,7 @@ class ErrorEvidenceTransport implements Transport { transport.onerror = (error) => this.onerror?.(error); transport.onclose = () => { this.pendingTools.clear(); + this.pendingToolLists.clear(); for (const completion of this.pendingCompletions.values()) completion.abort(); this.pendingCompletions.clear(); @@ -206,6 +259,7 @@ class ErrorEvidenceTransport implements Transport { await this.transport.close(); } finally { this.pendingTools.clear(); + this.pendingToolLists.clear(); for (const completion of this.pendingCompletions.values()) completion.abort(); this.pendingCompletions.clear(); @@ -230,17 +284,24 @@ class ErrorEvidenceTransport implements Transport { ? message.id : undefined; const tool = id === undefined ? undefined : this.pendingTools.get(id); + const omitToolSchemas = + this.omitOutputSchemas && + id !== undefined && + this.pendingToolLists.has(id); try { const recovered = this.recoverError(message, tool); + const advertised = omitToolSchemas + ? omitOutputSchemasFromToolsList(recovered, id) + : recovered; if ( - isJSONRPCResultResponse(recovered) && - isCallToolResult(recovered.result) && - recovered.result.isError === true + isJSONRPCResultResponse(advertised) && + isCallToolResult(advertised.result) && + advertised.result.isError === true ) { // Keep the complete typed JSON in content. Error projections are not // success outputSchema data; oversized recovery must consume their // private structured carrier before it is removed from the wire. - const { structuredContent: _diagnostic, ...result } = recovered.result; + const { structuredContent: _diagnostic, ...result } = advertised.result; // Errors the SDK builds itself (input validation, a thrown handler) // bypass ToolResultDelivery; mark them for target-content tools too. const marked = @@ -252,12 +313,16 @@ class ErrorEvidenceTransport implements Transport { _meta: { ...result._meta, ...TARGET_CONTENT_RESULT_META }, } : result; - await this.transport.send({ ...recovered, result: marked }, options); - } else await this.transport.send(recovered, options); + await this.transport.send({ ...advertised, result: marked }, options); + } else await this.transport.send(advertised, options); } finally { if (!isJSONRPCRequest(message) && id !== undefined) { - this.pendingTools.delete(id); - this.pendingCompletions.delete(id); + forgetPendingMcpRequest( + this.pendingTools, + this.pendingToolLists, + this.pendingCompletions, + id, + ); } } } diff --git a/src/server/createServer.ts b/src/server/createServer.ts index ad1c10677..e637f8d0d 100644 --- a/src/server/createServer.ts +++ b/src/server/createServer.ts @@ -39,6 +39,7 @@ import { type ToolResultDelivery, } from "./toolResult.js"; import { parseMcpResponseBudget } from "../config/mcpResponseBudget.js"; +import { parseMcpOutputSchemaProfile } from "../config/mcpOutputSchemaProfile.js"; import { COMPACT_INPUT_SCHEMA_BUDGET_BYTES, parseMcpInputSchemaProfile, @@ -250,6 +251,7 @@ const createMcpServer = ( session: BinarySessionPort | undefined, delivery: ToolResultDelivery, compactPresentation: CompactInputSchemaPresentation | undefined, + omitOutputSchemas: boolean, ): EvidenceMcpServer => new EvidenceMcpServer( { @@ -269,6 +271,7 @@ const createMcpServer = ( : (evidence) => session.recordEvidence(evidence), delivery, compactPresentation, + omitOutputSchemas, ); /** @@ -283,6 +286,7 @@ export const createServer = ( const session = source.kind === "session" ? source.session : undefined; const analysis = source.kind === "session" ? source.session : source.analysis; const environment = snapshotEnvironment(options.environment ?? process.env); + const omitOutputSchemas = outputSchemasOmitted(environment); const providerEnvironment = captureProviderEnvironment( options.providerEnvironment, environment, @@ -330,6 +334,7 @@ export const createServer = ( session, delivery, selectCompactInputSchemaPresentation(environment), + omitOutputSchemas, ); const gdbSessions = new GdbSessionManager({ environment: providerEnvironment, @@ -587,6 +592,16 @@ const selectToolResultDelivery = ( return createToolResultDelivery(configured.value); }; +const outputSchemasOmitted = ( + environment: Readonly, +): boolean => { + const configured = parseMcpOutputSchemaProfile( + environment.REA_MCP_OUTPUT_SCHEMA_PROFILE, + ); + if (!configured.ok) throw configured.error; + return configured.value === "off"; +}; + const selectCompactInputSchemaPresentation = ( environment: Readonly, ): CompactInputSchemaPresentation | undefined => { diff --git a/src/server/registerEnhancedTools.ts b/src/server/registerEnhancedTools.ts index e1e27c127..32cc4b6a1 100644 --- a/src/server/registerEnhancedTools.ts +++ b/src/server/registerEnhancedTools.ts @@ -14,6 +14,7 @@ import type { BinarySessionPort } from "../application/binary/BinarySessionPort. import { EnhancedTools } from "../application/EnhancedTools.js"; import type { ValidatedEnhancedCall } from "../application/EnhancedToolTypes.js"; import { + executeComposedWorkflow, createWorkflowEvidence, workflowSnapshotRecord, recordWorkflowUnknowns, @@ -231,15 +232,20 @@ const executeEnhancedTool = async ( context, parameters, }); - const services = new EnhancedTools({ + const analysis: AnalysisOperationPort = { execute: (operation, operationParameters, executionOptions) => admittedAnalysis.execute(operation, operationParameters, { ...executionOptions, progress, }), - }); + }; const result = await logToolExecution(registration.logger, name, () => - services.executeValidated(validatedCall, context.mcpReq.signal), + executeComposedWorkflow(analysis, (observed) => + new EnhancedTools(observed).executeValidated( + validatedCall, + context.mcpReq.signal, + ), + ), ); await progress.report({ phase: name, @@ -253,8 +259,9 @@ const executeEnhancedTool = async ( target: registration.activeTarget?.(), operation: name, parameters, - result: result.value, + result: result.value.result, upstreamProfile: registration.analysisProfile?.(), + upstreamLimitations: result.value.limitations, }); const recorded = registration.recordEvidence?.(evidence); if (recorded !== undefined && !recorded.ok) @@ -273,7 +280,7 @@ const executeEnhancedTool = async ( } const unknowns = recordWorkflowUnknowns({ name, - result: result.value, + result: result.value.result, evidenceId: evidence.evidence_id, recordUnknown: registration.recordUnknown, }); diff --git a/src/server/toolResult.ts b/src/server/toolResult.ts index 5d40afb5b..77e57acf0 100644 --- a/src/server/toolResult.ts +++ b/src/server/toolResult.ts @@ -48,7 +48,6 @@ const nativeEvidenceContinuation = ( /** Immutable encoding and recovery policy shared by one MCP server. */ export class ToolResultDelivery { readonly resultBudgetBytes: number; - constructor(maximumResponseBytes: number) { this.resultBudgetBytes = maximumResponseBytes - 1024; Object.freeze(this); diff --git a/tests/boundary/idaStartupDiagnostics.test.ts b/tests/boundary/idaStartupDiagnostics.test.ts index 8f36543b0..9776caf66 100644 --- a/tests/boundary/idaStartupDiagnostics.test.ts +++ b/tests/boundary/idaStartupDiagnostics.test.ts @@ -1,6 +1,14 @@ import { describe, expect, it } from "vitest"; import { createIdaMcpConnection } from "../../src/ida/IdaMcpConnection.js"; +const launcherEnvironment = (): Readonly> => { + if (process.platform !== "win32") return {}; + const systemRoot = process.env.SYSTEMROOT; + if (systemRoot === undefined || systemRoot.length === 0) + throw new Error("The Windows Node fixture requires its actual SYSTEMROOT."); + return { SYSTEMROOT: systemRoot }; +}; + describe("IDA startup diagnostics", () => { it("preserves the failing launcher's stderr instead of reporting only a closed transport", async () => { const connection = createIdaMcpConnection( @@ -10,7 +18,7 @@ describe("IDA startup diagnostics", () => { "-e", 'process.stderr.write("FATAL: IDA runtime initialization failed\\n", () => process.exit(2));', ], - env: {}, + env: launcherEnvironment(), mode: "headless", timeoutMs: 1000, }, @@ -37,7 +45,7 @@ describe("IDA startup diagnostics", () => { "-e", 'process.stderr.write("x".repeat(20000) + "FATAL-END", () => process.exit(2));', ], - env: {}, + env: launcherEnvironment(), mode: "headless", timeoutMs: 1000, }, @@ -66,7 +74,7 @@ it("drops a leading partial UTF-8 character from the bounded stderr tail", async "-e", 'process.stderr.write("€".repeat(6000), () => process.exit(2));', ], - env: {}, + env: launcherEnvironment(), mode: "headless", timeoutMs: 1000, }, @@ -93,7 +101,7 @@ it("retains an in-budget undecodable leading byte without claiming tail truncati "-e", "process.stderr.write(Buffer.from([0x80, 0x61, 0x62]), () => process.exit(2));", ], - env: {}, + env: launcherEnvironment(), mode: "headless", timeoutMs: 1000, }, diff --git a/tests/boundary/mcp/agentKnownAnswerMcp.test.ts b/tests/boundary/mcp/agentKnownAnswerMcp.test.ts index b59dd80a8..11e8d365a 100644 --- a/tests/boundary/mcp/agentKnownAnswerMcp.test.ts +++ b/tests/boundary/mcp/agentKnownAnswerMcp.test.ts @@ -245,6 +245,334 @@ describe("source-produced parser comparison known answers", () => { }); }); +describe("Notes export known-answer evaluation", () => { + it("grades real analysis and traces while rejecting false values and unsupported citations", async () => { + const root = await createTestTempDirectory("rea-known-answer-notes-"); + const targets = await createAgentEvaluationFixtures(root, resolve(".")); + const claims = agentFixtureClaims("notes-export", targets); + if (claims === undefined) throw new Error("Missing Notes rubric"); + expect(claims.map(({ id }) => id)).toEqual([ + "renderer_api", + "renderer_ipc", + "main_ipc", + "preload", + "feature_modules", + "semantic_boundary", + ]); + const harness = await createApplicationMcpHarness(); + try { + const workflow = await notesWorkflow(harness, targets.notesElectron); + assertNotesAnswerFailures(workflow, claims); + assertNotesUnsupportedClaims(workflow, claims); + await assertNotesSeedBinding(harness, workflow, claims); + } finally { + await harness.close(); + } + }); + + it("binds Notes analysis and derived traces to the requested artifact path and bytes", async () => { + const root = await createTestTempDirectory( + "rea-known-answer-notes-identity-", + ); + const targets = await createAgentEvaluationFixtures(root, resolve(".")); + const claims = agentFixtureClaims("notes-export", targets); + if (claims === undefined) throw new Error("Missing Notes rubric"); + const foreign = join(root, "other-notes.asar"); + await cp(targets.notesElectron, foreign); + const harness = await createApplicationMcpHarness(); + try { + for (const path of [foreign, targets.notesElectron]) { + if (path === targets.notesElectron) { + await appendFile( + join(root, "notes-electron/renderer.js"), + "\n// changed shipped bytes\n", + ); + await createPackage(join(root, "notes-electron"), path); + } + const workflow = await notesWorkflow(harness, path); + const digest = notesEvidence(workflow, "analyze_javascript_application") + .subject?.digest.sha256; + expect(digest === targets.notesElectronSha256).toBe(path === foreign); + const assessment = grade( + workflow.events, + claimManifestByOperation(claims, workflow.evidence), + "analyze_javascript_application", + claims, + ).factualAssessment; + expect(assessment.status).toBe("failed"); + assertClaimFailure( + assessment, + "renderer_api", + "source_subject_mismatch", + ); + for (const id of ["feature_modules", "semantic_boundary"]) + assertClaimFailure(assessment, id, "source_links_mismatch"); + } + } finally { + await harness.close(); + } + }); +}); + +type NotesWorkflow = Awaited>; + +const notesEvidence = ( + workflow: NotesWorkflow, + operation: string, +): Evidence => { + const evidence = workflow.evidence.get(operation); + if (evidence === undefined) + throw new Error(`Missing Notes ${operation} Evidence`); + return evidence; +}; + +const assertClaimFailure = ( + assessment: ReturnType["factualAssessment"], + id: string, + failure: string, +) => { + expect(assessment.claims).toContainEqual( + expect.objectContaining({ + id, + failures: expect.arrayContaining([failure]), + }), + ); +}; + +const assertNotesAnswerFailures = ( + workflow: NotesWorkflow, + claims: readonly FixtureClaimExpectation[], +) => { + const manifest = claimManifestByOperation(claims, workflow.evidence); + const assess = (answer: unknown, events = workflow.events) => + grade(events, answer, "analyze_javascript_application", claims) + .factualAssessment; + expect(assess(manifest).status).toBe("passed"); + const cases = [ + { + id: "renderer_ipc", + patch: { + value: { + channel: "notes:delete", + operation: "send", + source: "preload.js", + }, + }, + failure: "value_mismatch", + }, + { + id: "semantic_boundary", + patch: { + value: { status: "found", coverage: "complete", relevant_unknowns: 0 }, + }, + failure: "value_mismatch", + }, + { + id: "renderer_api", + patch: { + evidence_id: notesEvidence(workflow, "trace_application_feature") + .evidence_id, + }, + failure: "source_operation_mismatch", + }, + ]; + for (const { id, patch, failure } of cases) { + const answer = { + claims: manifest.claims.map((claim) => + claim.id === id ? { ...claim, ...patch } : claim, + ), + }; + assertClaimFailure(assess(answer), id, failure); + } + const missingParent = assess( + manifest, + workflow.events.filter((event) => event.item.id !== "notes-analysis"), + ); + for (const id of ["feature_modules", "semantic_boundary"]) + assertClaimFailure(missingParent, id, "source_links_mismatch"); +}; + +const assertNotesUnsupportedClaims = ( + workflow: NotesWorkflow, + claims: readonly FixtureClaimExpectation[], +) => { + const manifest = claimManifestByOperation(claims, workflow.evidence); + const evidenceId = notesEvidence( + workflow, + "trace_application_feature", + ).evidence_id; + const unsupportedRuntime = { + claims: [ + ...manifest.claims, + { + id: "runtime_export", + value: "A click wrote notes.csv to the actual Downloads directory.", + evidence_id: evidenceId, + authority: "controlled-replay", + confidence: "observed", + }, + ], + }; + expect( + grade( + workflow.events, + unsupportedRuntime, + "analyze_javascript_application", + claims, + ).factualAssessment.manifestFailures, + ).toContain("unknown_claim:runtime_export"); + const prose = evaluateCodexEvents( + [ + ...workflow.events, + { + type: "item.completed", + item: { + type: "agent_message", + text: `Observed Evidence ${evidenceId}: notes:export, preload.js and main.js do not exist. The export certainly executed and wrote notes.csv; runtime behavior is known without further observation.`, + }, + }, + ], + "analyze_javascript_application", + { + requireEvidence: true, + requiredAnswerTermGroups: [["notes:export"], ["preload.js"], ["main.js"]], + fixtureClaims: claims, + }, + ); + expect(prose).toMatchObject({ + finalCitesEvidence: true, + answerTermCoverageMet: true, + factualCorrectness: "failed", + }); + expect(prose.factualAssessment.manifestFailures).toContain( + "invalid_final_claim_manifest", + ); +}; + +const assertNotesSeedBinding = async ( + harness: Awaited>, + workflow: NotesWorkflow, + claims: readonly FixtureClaimExpectation[], +) => { + const manifest = claimManifestByOperation(claims, workflow.evidence); + for (const match of ["contains", undefined] as const) { + const feature = await harness.client.callTool({ + name: "trace_application_feature", + arguments: { + application: notesEvidence(workflow, "analyze_javascript_application"), + seed: { + kind: "channel", + value: "notes:export", + ...(match === undefined ? {} : { match }), + case_sensitive: false, + }, + direction: "both", + }, + }); + expect(feature.isError).not.toBe(true); + const answer = { + claims: manifest.claims.map((claim) => + claim.id === "feature_modules" + ? { ...claim, evidence_id: resultEvidence(feature).evidence_id } + : claim, + ), + }; + const assessment = grade( + [ + ...workflow.events, + toolEvent( + `notes-feature-${match ?? "default"}`, + "trace_application_feature", + feature, + ), + ], + answer, + "analyze_javascript_application", + claims, + ).factualAssessment; + expect(assessment.status).toBe(match === undefined ? "passed" : "failed"); + if (match === "contains") { + assertClaimFailure( + assessment, + "feature_modules", + "source_assertion_mismatch", + ); + expect( + assessment.claims.find(({ id }) => id === "feature_modules")?.failures, + ).not.toContain("source_parameters_mismatch"); + } + } +}; + +const notesWorkflow = async ( + harness: Awaited>, + inputPath: string, +) => { + const analysis = await harness.client.callTool({ + name: "analyze_javascript_application", + arguments: { input_path: inputPath }, + }); + expect(analysis.isError).not.toBe(true); + const application = resultEvidence(analysis); + const feature = await harness.client.callTool({ + name: "trace_application_feature", + arguments: { + application, + seed: { + kind: "channel", + value: "notes:export", + match: "exact", + case_sensitive: false, + }, + direction: "both", + }, + }); + expect(feature.isError).not.toBe(true); + const semantic = await harness.client.callTool({ + name: "trace_javascript_semantics", + arguments: { + application, + query: { + seed: { kind: "property", name: "invoke" }, + direction: "forward-influence", + include_ambiguous_dynamic_edges: false, + expected: null, + }, + }, + }); + expect(semantic.isError).not.toBe(true); + return { + events: [ + toolEvent("notes-analysis", "analyze_javascript_application", analysis), + toolEvent("notes-feature", "trace_application_feature", feature), + toolEvent("notes-semantic", "trace_javascript_semantics", semantic), + ], + evidence: new Map([ + ["analyze_javascript_application", application], + ["trace_application_feature", resultEvidence(feature)], + ["trace_javascript_semantics", resultEvidence(semantic)], + ]), + }; +}; + +const claimManifestByOperation = ( + claims: readonly FixtureClaimExpectation[], + evidenceByOperation: ReadonlyMap, +) => ({ + claims: claims.map(({ id, expectedValue, source }) => { + const evidence = evidenceByOperation.get(source.operation); + if (evidence === undefined) + throw new Error(`Missing producer for claim ${id}`); + return { + id, + value: expectedValue, + evidence_id: evidence.evidence_id, + authority: evidence.authority, + confidence: evidence.confidence, + }; + }), +}); + const resultEvidence = (result: { readonly structuredContent?: unknown; }): Evidence => parseEvidence(result.structuredContent); diff --git a/tests/boundary/mcp/compactInputSchemaMcp.test.ts b/tests/boundary/mcp/compactInputSchemaMcp.test.ts index 5ca46c39b..8f8f7ee0c 100644 --- a/tests/boundary/mcp/compactInputSchemaMcp.test.ts +++ b/tests/boundary/mcp/compactInputSchemaMcp.test.ts @@ -4,6 +4,11 @@ import { Ajv2020 } from "ajv/dist/2020.js"; import { describe, expect, it } from "vitest"; import { TOOL_CONTRACTS } from "../../../src/contracts/toolContracts.js"; +import { + schemaArrays, + schemaChildren, + schemaMaps, +} from "../../../src/contracts/inputSchemaPresentation.js"; import { COMPACT_INPUT_SCHEMA_BUDGET_BYTES, parseMcpInputSchemaProfile, @@ -23,6 +28,45 @@ interface AdvertisedTool { const serializedBytes = (value: unknown): number => Buffer.byteLength(JSON.stringify(value), "utf8"); +const schemaDepth = (value: unknown, depth = 0): number => { + if (Array.isArray(value)) + return Math.max( + depth, + ...value.map((child) => schemaDepth(child, depth + 1)), + ); + if (typeof value !== "object" || value === null) return depth; + return Math.max( + depth, + ...Object.values(value).map((child) => schemaDepth(child, depth + 1)), + ); +}; + +const unsupportedDependencyKeywords = ( + schema: unknown, + path = "$", +): readonly string[] => { + if (Array.isArray(schema)) + return schema.flatMap((child, index) => + unsupportedDependencyKeywords(child, `${path}[${index}]`), + ); + if (typeof schema !== "object" || schema === null) return []; + return Object.entries(schema).flatMap(([key, value]) => { + const childPath = `${path}.${key}`; + if (key === "dependencies" || key === "dependentRequired") + return [childPath]; + if (["const", "enum", "default", "examples"].includes(key)) return []; + if (schemaMaps.has(key) && typeof value === "object" && value !== null) + return Object.entries(value).flatMap(([name, child]) => + unsupportedDependencyKeywords(child, `${childPath}.${name}`), + ); + if (schemaArrays.has(key) && Array.isArray(value)) + return unsupportedDependencyKeywords(value, childPath); + if (schemaChildren.has(key)) + return unsupportedDependencyKeywords(value, childPath); + return []; + }); +}; + const listToolsThroughTransport = async ( server: EvidenceMcpServer, ): Promise => { @@ -77,6 +121,12 @@ describe("MCP compact input schema profile", () => { serializedBytes(inputSchema) > COMPACT_INPUT_SCHEMA_BUDGET_BYTES, ); expect(oversized.map(({ name }) => name)).toEqual([]); + const tooDeep = tools.filter( + ({ inputSchema }) => schemaDepth(inputSchema) > 10, + ); + expect( + tooDeep.map(({ name, inputSchema }) => [name, schemaDepth(inputSchema)]), + ).toEqual([]); }); it("keeps output schemas and tool descriptions identical to the default profile", async () => { @@ -120,6 +170,31 @@ describe("MCP compact input schema profile", () => { } }); + it("omits Gemini-unsupported dependency keywords from both catalog profiles", async () => { + const full = fullServer(); + const compact = compactServer(); + registerCatalog(full); + registerCatalog(compact); + const [fullTools, compactTools] = await Promise.all([ + listToolsThroughTransport(full), + listToolsThroughTransport(compact), + ]); + for (const tools of [fullTools, compactTools]) + expect( + tools.flatMap(({ name, inputSchema }) => + unsupportedDependencyKeywords(inputSchema).map( + (path) => `${name}${path.slice(1)}`, + ), + ), + ).toEqual([]); + const sqlite = fullTools.find( + ({ name }) => name === "inspect_sqlite_database", + ); + expect(sqlite?.inputSchema.description).toContain( + "When row_limit is provided, also provide table.", + ); + }); + it("retains parameter types and avoids client-side reference expansion", async () => { const server = compactServer(); registerCatalog(server); @@ -136,13 +211,7 @@ describe("MCP compact input schema profile", () => { ({ name }) => name === "compare_managed_members", ); expect(comparison?.inputSchema.properties).toMatchObject({ - left: { - properties: { - normalized_result: { - type: ["null", "boolean", "object", "array", "number", "string"], - }, - }, - }, + left: { type: "object" }, }); const ledger = tools.find( ({ name }) => name === "build_reconstruction_obligation_ledger", diff --git a/tests/boundary/mcp/enhancedTools.04-uses-complete-inventories-inline.test.ts b/tests/boundary/mcp/enhancedTools.04-uses-complete-inventories-inline.test.ts index a08d35a34..c2c0c52e9 100644 --- a/tests/boundary/mcp/enhancedTools.04-uses-complete-inventories-inline.test.ts +++ b/tests/boundary/mcp/enhancedTools.04-uses-complete-inventories-inline.test.ts @@ -82,60 +82,62 @@ it("discovers declarations without counting duplicates or compiler bookkeeping", }); }); +const completeOverviewPort: AnalysisOperationPort = { + execute: (name) => { + switch (name) { + case "list_segments": + return Promise.resolve( + ok([ + { + name: "__TEXT", + start: "0x1000", + end: "0x1800", + image_base: "0x1000", + readable: null, + writable: null, + executable: null, + }, + { + name: "__DATA", + start: "0x1800", + end: "0x2000", + image_base: "0x1000", + readable: null, + writable: null, + executable: null, + }, + ]), + ); + case "current_document": + return Promise.resolve(ok("fixture")); + case "list_documents": + return Promise.resolve(ok(["New Document", "fixture"])); + case "list_strings": + return Promise.resolve( + ok( + Array.from({ length: 700 }, (_, index) => ({ + address: `0x${(0x30 + index).toString(16)}`, + value: `string-${index}`, + })), + ), + ); + case "list_procedures": { + return Promise.resolve( + ok([ + { address: "0x1", value: "first" }, + { address: "0x2", value: "last" }, + ]), + ); + } + default: + throw new Error(`Unexpected analysis operation: ${name}`); + } + }, +}; + describe("enhanced MCP tools", () => { it("returns the complete overview inline with exhaustive totals", async () => { - const client = await connect({ - execute: (name) => { - switch (name) { - case "list_segments": - return Promise.resolve( - ok([ - { - name: "__TEXT", - start: "0x1000", - end: "0x1800", - image_base: "0x1000", - readable: null, - writable: null, - executable: null, - }, - { - name: "__DATA", - start: "0x1800", - end: "0x2000", - image_base: "0x1000", - readable: null, - writable: null, - executable: null, - }, - ]), - ); - case "current_document": - return Promise.resolve(ok("fixture")); - case "list_documents": - return Promise.resolve(ok(["New Document", "fixture"])); - case "list_strings": - return Promise.resolve( - ok( - Array.from({ length: 700 }, (_, index) => ({ - address: `0x${(0x30 + index).toString(16)}`, - value: `string-${index}`, - })), - ), - ); - case "list_procedures": { - return Promise.resolve( - ok([ - { address: "0x1", value: "first" }, - { address: "0x2", value: "last" }, - ]), - ); - } - default: - throw new Error(`Unexpected analysis operation: ${name}`); - } - }, - }); + const client = await connect(completeOverviewPort); const result = jsonResult( await client.callTool({ name: "binary_overview", @@ -167,6 +169,48 @@ describe("enhanced MCP tools", () => { }); }); + it("returns binary counts while omitting unavailable segment coverage", async () => { + const client = await connect({ + execute: (name) => { + if (name === "current_document") + return Promise.resolve( + err( + new AnalysisCapabilityUnavailableError( + "fixture", + name, + "the provider has one bound document", + ), + ), + ); + if (name === "list_documents") + return Promise.resolve(ok(["/tmp/fixture.bin"])); + if (name === "list_segments") + return Promise.resolve( + err( + new AnalysisCapabilityUnavailableError( + "fixture", + name, + "segment enumeration is unsupported", + ), + ), + ); + if (name === "list_procedures" || name === "list_strings") + return Promise.resolve(ok([])); + throw new Error(`Unexpected analysis operation: ${name}`); + }, + }); + + expect( + jsonResult( + await client.callTool({ name: "binary_overview", arguments: {} }), + ), + ).toEqual({ + document: "/tmp/fixture.bin", + procedure_count: 0, + string_count: 0, + }); + }); + it("rejects paginated provider output", async () => { const client = await connect({ execute: () => diff --git a/tests/boundary/mcp/functionCallDirectionsMcp.test.ts b/tests/boundary/mcp/functionCallDirectionsMcp.test.ts new file mode 100644 index 000000000..b4034d036 --- /dev/null +++ b/tests/boundary/mcp/functionCallDirectionsMcp.test.ts @@ -0,0 +1,138 @@ +import { Client, InMemoryTransport } from "@modelcontextprotocol/client"; +import { Ajv2020 } from "ajv/dist/2020.js"; +import { expect, it } from "vitest"; + +import { FUNCTION_COMPARISON_EXAMPLE } from "../../../src/contracts/functionComparisonExample.js"; +import { createEvidence } from "../../../src/domain/evidence.js"; +import { functionDossierSchema } from "../../../src/domain/hopperValues.js"; +import type { RelationshipCoverage } from "../../../src/domain/relationshipCoverage.js"; +import { createServer } from "../../../src/server/createServer.js"; +import { createTestBinarySession } from "../../fixtures/binarySession.js"; + +const observe = (relationshipCoverage: RelationshipCoverage) => { + const template = FUNCTION_COMPARISON_EXAMPLE.left; + if (template.subject === null) + throw new Error("Expected artifact-bound fixture"); + const dossier = functionDossierSchema.parse(template.normalized_result); + return createEvidence( + { + path: template.subject.local_path, + sha256: template.subject.digest.sha256, + format: template.subject.format, + }, + template.provider, + { + operation: "analyze_function", + parameters: template.parameters, + result: { ...dossier, callees: [], unresolved_calls: [] }, + confidence: template.confidence, + authority: template.authority, + relationshipCoverage, + }, + ); +}; + +it("preserves directional gaps through advertised schemas and public path reconstruction", async () => { + const session = createTestBinarySession(() => { + throw new Error("Offline call paths must not launch a provider"); + }); + const server = createServer({ kind: "session", session }); + const client = new Client({ name: "function-call-directions", version: "1" }); + const [clientTransport, serverTransport] = + InMemoryTransport.createLinkedPair(); + try { + await server.connect(serverTransport); + await client.connect(clientTransport); + const advertised = (await client.listTools()).tools.find( + ({ name }) => name === "build_call_path", + ); + if (advertised === undefined) + throw new Error("Missing build_call_path contract"); + const ajv = new Ajv2020({ strict: false, validateFormats: false }); + expect(ajv.validateSchema(advertised.inputSchema)).toBe(true); + const validate = ajv.compile(advertised.inputSchema); + for (const { directions, status, exhaustive } of [ + { + directions: ["callers"] as ["callers"], + status: "not_found", + exhaustive: true, + }, + { + directions: ["callees"] as ["callees"], + status: "unknown", + exhaustive: false, + }, + { + directions: ["callers", "callees"] as ["callers", "callees"], + status: "unknown", + exhaustive: false, + }, + ]) { + const limitation = + "An observed call reference is absent from the inventory."; + const coverage: RelationshipCoverage = { + status: "partial", + limitations: [limitation], + limited_directions: directions, + }; + const observation = observe(coverage); + expect(session.recordEvidence(observation).ok).toBe(true); + const input = { + functions: [observation], + start: { address: "0x1000" }, + goal: { address: "0x9999" }, + }; + expect(validate(input), JSON.stringify(validate.errors)).toBe(true); + for (const limited_directions of [ + [], + ["callers", "callers"], + ["callees", "callees"], + ["callers", "callees", "callers"], + ["incoming"], + ]) + expect( + validate({ + ...input, + functions: [ + { + ...observation, + relationship_coverage: { + ...coverage, + limited_directions, + }, + }, + ], + }), + ).toBe(false); + const result = await client.callTool({ + name: "build_call_path", + arguments: input, + }); + if (result.isError) throw new Error(JSON.stringify(result)); + expect(result.structuredContent).toMatchObject({ + normalized_result: { + status, + search_scope: { exhaustive }, + limitations: expect.arrayContaining([limitation]), + }, + }); + const comparison = await client.callTool({ + name: "compare_functions", + arguments: { left: observation, right: observation }, + }); + if (comparison.isError) throw new Error(JSON.stringify(comparison)); + expect(comparison.structuredContent).toMatchObject({ + normalized_result: { + dimensions: expect.arrayContaining([ + expect.objectContaining({ dimension: "calls", status: "unknown" }), + ]), + }, + }); + // Each scenario owns its Evidence and residual-unknown ledger. + const closed = await session.close(); + if (!closed.ok) throw closed.error; + } + } finally { + await Promise.allSettled([session.close(), client.close(), server.close()]); + } +}); diff --git a/tests/boundary/mcp/functionComparisonMcp.test.ts b/tests/boundary/mcp/functionComparisonMcp.test.ts index 97f07dc1e..89b3f14a6 100644 --- a/tests/boundary/mcp/functionComparisonMcp.test.ts +++ b/tests/boundary/mcp/functionComparisonMcp.test.ts @@ -15,128 +15,157 @@ import { FUNCTION_COMPARISON_EXAMPLE } from "../../../src/contracts/functionComp import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js"; import { createServer } from "../../../src/server/createServer.js"; import { observed } from "../../fixtures/analysisExecution.js"; +import { reconstructionVerificationResultSchema } from "../../../src/domain/reconstructionVerificationSchemas.js"; -describe("function comparison reconstruction MCP integration", () => { - it.each([ - { comparisonStatus: "unchanged", verificationStatus: "pass" }, - { comparisonStatus: "changed", verificationStatus: "fail" }, - ] as const)( - "verifies public $comparisonStatus function comparison as $verificationStatus", - async ({ comparisonStatus, verificationStatus }) => { - let providerLaunches = 0; - const session = createTestBinarySession(() => { - providerLaunches += 1; - throw new Error("Function verification must not launch a provider"); - }); - const left = FUNCTION_COMPARISON_EXAMPLE.left; - const right = - comparisonStatus === "changed" - ? FUNCTION_COMPARISON_EXAMPLE.right - : createEvidence( - { - path: "/tmp/function-unchanged", - sha256: "2".repeat(64), - format: "mach-o", - }, - left.provider, - { - operation: left.operation, - parameters: left.parameters, - result: left.normalized_result, - confidence: left.confidence, - authority: left.authority, - ...(left.relationship_coverage === null - ? {} - : { relationshipCoverage: left.relationship_coverage }), - }, - ); - const server = createServer({ kind: "session", session }); - const client = new Client({ - name: "function-verification-test", - version: "1", - }); - const [clientTransport, serverTransport] = - InMemoryTransport.createLinkedPair(); - try { - await server.connect(serverTransport); - await client.connect(clientTransport); - const compared = await client.callTool({ - name: "compare_functions", - arguments: { left, right }, - }); - expect(compared.isError).not.toBe(true); - const comparison = parseEvidence(compared.structuredContent); - expect(comparison.normalized_result).toMatchObject({ - status: comparisonStatus, - }); - expect(comparison.parameters).toEqual({ - left_evidence_id: left.evidence_id, - right_evidence_id: right.evidence_id, - }); - expect(comparison.evidence_links).toEqual( - expect.arrayContaining([left.evidence_id, right.evidence_id]), - ); - - const verified = await client.callTool({ - name: "verify_reconstruction", - arguments: { - specification: { - name: "Function pseudocode comparison", - claims: [ - { - kind: "structural-function", - claim_id: "pseudocode", - title: "Pseudocode agrees", - comparison_evidence_id: comparison.evidence_id, - dimension: "pseudocode", - }, - ], +it.each([ + { comparisonStatus: "unchanged", verificationStatus: "pass" }, + { comparisonStatus: "changed", verificationStatus: "fail" }, +] as const)( + "verifies public $comparisonStatus function comparison as $verificationStatus", + async ({ comparisonStatus, verificationStatus }) => { + let providerLaunches = 0; + const session = createTestBinarySession(() => { + providerLaunches += 1; + throw new Error("Function verification must not launch a provider"); + }); + const left = FUNCTION_COMPARISON_EXAMPLE.left; + const right = + comparisonStatus === "changed" + ? FUNCTION_COMPARISON_EXAMPLE.right + : createEvidence( + { + path: "/tmp/function-unchanged", + sha256: "2".repeat(64), + format: "mach-o", }, - }, - }); - expect( - verified.isError, - JSON.stringify(verified.structuredContent), - ).not.toBe(true); - const verification = parseEvidence(verified.structuredContent); - expect(verification.normalized_result).toMatchObject({ - status: verificationStatus, - summary: { - total: 1, - passed: verificationStatus === "pass" ? 1 : 0, - failed: verificationStatus === "fail" ? 1 : 0, - unknown: 0, - }, - claims: { - items: [ - expect.objectContaining({ - status: verificationStatus, - observed_status: comparisonStatus, + left.provider, + { + operation: left.operation, + parameters: left.parameters, + result: left.normalized_result, + confidence: left.confidence, + authority: left.authority, + ...(left.relationship_coverage === null + ? {} + : { relationshipCoverage: left.relationship_coverage }), + }, + ); + const server = createServer({ kind: "session", session }); + const client = new Client({ name: "function-claims", version: "1" }); + const [clientTransport, serverTransport] = + InMemoryTransport.createLinkedPair(); + try { + await server.connect(serverTransport); + await client.connect(clientTransport); + const compared = await client.callTool({ + name: "compare_functions", + arguments: { left, right }, + }); + expect(compared.isError).not.toBe(true); + const comparison = parseEvidence(compared.structuredContent); + expect(comparison.limitations).not.toContainEqual( + expect.stringContaining( + "without address or generated-name normalization", + ), + ); + expect(comparison.normalized_result).toMatchObject({ + status: comparisonStatus, + dimensions: expect.arrayContaining( + ["pseudocode", "assembly"].map((dimension) => + expect.objectContaining({ + dimension, + limitations: expect.arrayContaining([ + expect.stringContaining( + "without address or generated-name normalization", + ), + ]), + }), + ), + ), + }); + expect(comparison.parameters).toEqual({ + left_evidence_id: left.evidence_id, + right_evidence_id: right.evidence_id, + }); + expect(comparison.evidence_links).toEqual( + expect.arrayContaining([left.evidence_id, right.evidence_id]), + ); + + const verified = await client.callTool({ + name: "verify_reconstruction", + arguments: { + specification: { + name: "Function pseudocode comparison", + claims: ["pseudocode", "assembly", "cfg", "overall"].map( + (dimension) => ({ + kind: "structural-function", + claim_id: dimension, + title: `${dimension} agrees`, comparison_evidence_id: comparison.evidence_id, - left_evidence_ids: [left.evidence_id], - right_evidence_ids: [right.evidence_id], + dimension, }), - ], - }, - }); - expect( - session - .exportEvidenceBundle() - .records.find( - ({ evidence_id: id }) => id === comparison.evidence_id, ), - ).toEqual(comparison); - } finally { - await Promise.allSettled([ - client.close(), - server.close(), - session.close(), - ]); - expect(providerLaunches).toBe(0); + }, + }, + }); + expect( + verified.isError, + JSON.stringify(verified.structuredContent), + ).not.toBe(true); + const verification = parseEvidence(verified.structuredContent); + expect(verification.normalized_result).toMatchObject({ + status: verificationStatus, + summary: { + total: 4, + passed: verificationStatus === "pass" ? 4 : 2, + failed: verificationStatus === "fail" ? 2 : 0, + unknown: 0, + }, + claims: { + items: expect.arrayContaining([ + expect.objectContaining({ + dimension: "pseudocode", + status: verificationStatus, + observed_status: comparisonStatus, + comparison_evidence_id: comparison.evidence_id, + left_evidence_ids: [left.evidence_id], + right_evidence_ids: [right.evidence_id], + limitations: expect.arrayContaining([ + expect.stringContaining( + "without address or generated-name normalization", + ), + ]), + }), + ]), + }, + }); + const claims = reconstructionVerificationResultSchema.parse( + verification.normalized_result, + ).claims.items; + for (const claim of claims) { + const caveats = claim.limitations.join(" "); + if (claim.dimension === "cfg") + expect(caveats).not.toMatch(/generated-name normalization/u); + else expect(caveats).toMatch(/generated-name normalization/u); + if (claim.dimension === "assembly" || claim.dimension === "overall") + expect(caveats).toMatch(/Assembly is opaque/u); + else expect(caveats).not.toMatch(/Assembly is opaque/u); } - }, - ); -}); + expect( + session + .exportEvidenceBundle() + .records.find(({ evidence_id: id }) => id === comparison.evidence_id), + ).toEqual(comparison); + } finally { + await Promise.allSettled([ + client.close(), + server.close(), + session.close(), + ]); + expect(providerLaunches).toBe(0); + } + }, +); describe("function comparison MCP integration", () => { it("records linked comparison Evidence and its residual unknown", async () => { diff --git a/tests/boundary/mcp/mcpDoctor.test.ts b/tests/boundary/mcp/mcpDoctor.test.ts index e9b68b36f..acfb2bb0e 100644 --- a/tests/boundary/mcp/mcpDoctor.test.ts +++ b/tests/boundary/mcp/mcpDoctor.test.ts @@ -10,12 +10,35 @@ import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { compareInventory, runProductionMcpDoctor, + runProductionMcpDoctorCli, } from "../../../src/mcpDoctor.js"; import { CATALOG_IDENTITY } from "../../../src/catalogIdentity.js"; afterEach(() => vi.useRealTimers()); describe("production MCP doctor", () => { + it("preserves the full-output envelope for startup failures", async () => { + const result = await runProductionMcpDoctorCli( + ["--json", "--full-output"], + { + dispatcherPath: resolve("tests/fixtures/mcpDoctorExit.mjs"), + packageRoot: process.cwd(), + }, + ); + expect(result.exitCode).toBe(1); + expect(JSON.parse(result.output)).toMatchObject({ + ok: false, + error: { + healthy: false, + checks: [{ name: "transport", ok: false }], + }, + meta: { + command: "rea mcp doctor", + duration: expect.stringMatching(/^\d+ms$/u), + }, + }); + }); + it("connects to the production stdio child and verifies the canonical catalog", async () => { const result = await runProductionMcpDoctor({ command: process.execPath, @@ -44,7 +67,9 @@ describe("production MCP doctor", () => { CATALOG_IDENTITY.counts.mcp_tools, ); }, 30_000); +}); +describe("production MCP doctor probe validation", () => { it("rejects a schema-invalid probe while retaining the discovered inventory", async () => { const root = await createTestTempDirectory("rea-mcp-doctor-reply-"); const snapshotPath = join(root, "reply.json"); @@ -127,7 +152,9 @@ describe("production MCP doctor", () => { duplicates: ["c"], }); }); +}); +describe("production MCP doctor child lifecycle", () => { it("captures startup exit diagnostics", async () => { const result = await runProductionMcpDoctor({ command: process.execPath, diff --git a/tests/boundary/mcp/outputSchemaProfile.test.ts b/tests/boundary/mcp/outputSchemaProfile.test.ts new file mode 100644 index 000000000..a77dddf8c --- /dev/null +++ b/tests/boundary/mcp/outputSchemaProfile.test.ts @@ -0,0 +1,115 @@ +import { Client, InMemoryTransport } from "@modelcontextprotocol/client"; +import { z } from "zod"; +import { expect, it } from "vitest"; + +import { + TOOL_CONTRACTS, + toolContract, +} from "../../../src/contracts/toolContracts.js"; +import { EvidenceMcpServer } from "../../../src/server/EvidenceMcpServer.js"; +import { toolRegistrationOptions } from "../../../src/server/toolRegistrationOptions.js"; +import { createToolResultDelivery } from "../../../src/server/toolResult.js"; +import type { ToolContract } from "../../../src/contracts/toolContractTypes.js"; + +const createServer = (omitOutputSchemas: boolean) => + new EvidenceMcpServer( + { name: "output-schema-fixture", version: "1" }, + { capabilities: {} }, + undefined, + createToolResultDelivery(10 * 1024 * 1024), + undefined, + omitOutputSchemas, + ); + +it("omits only output schema advertisements and keeps canonical MCP output validation", async () => { + const server = createServer(true); + const [clientTransport, serverTransport] = + InMemoryTransport.createLinkedPair(); + const client = new Client({ name: "schema-profile-test", version: "1" }); + const contract = { + ...toolContract("comment"), + name: "output_schema_fixture", + inputSchema: z.strictObject({}), + outputSchema: z.strictObject({ value: z.string() }), + } satisfies ToolContract; + server.registerTool( + contract.name, + toolRegistrationOptions(contract), + async () => ({ + content: [{ type: "text" as const, text: "fixture" }], + structuredContent: { value: "complete" }, + }), + ); + const invalidContract = { + ...contract, + name: "invalid_output_schema_fixture", + } satisfies ToolContract; + server.registerTool( + invalidContract.name, + toolRegistrationOptions(invalidContract), + async () => ({ + content: [{ type: "text" as const, text: "invalid fixture" }], + structuredContent: { value: 42 }, + }), + ); + + await server.connect(serverTransport); + await client.connect(clientTransport); + try { + const tools = (await client.listTools()).tools; + expect(tools).toHaveLength(2); + expect(tools.every((tool) => tool.outputSchema === undefined)).toBe(true); + const success = await client.callTool({ + name: contract.name, + arguments: {}, + }); + expect(success.isError).not.toBe(true); + expect(success.structuredContent).toEqual({ value: "complete" }); + const invalid = await client.callTool({ + name: invalidContract.name, + arguments: {}, + }); + expect(invalid.isError).toBe(true); + expect(invalid.content).toEqual([ + expect.objectContaining({ + type: "text", + text: expect.stringContaining("Output validation error"), + }), + ]); + } finally { + await client.close(); + await server.close(); + } +}); + +it("keeps a full catalog below a measured one-megabyte frame when output schemas are off", async () => { + const measure = async (omitOutputSchemas: boolean): Promise => { + const server = createServer(omitOutputSchemas); + const [clientTransport, serverTransport] = + InMemoryTransport.createLinkedPair(); + const client = new Client({ name: "catalog-size-test", version: "1" }); + for (const contract of TOOL_CONTRACTS) + server.registerTool( + contract.name, + toolRegistrationOptions(contract), + async () => ({ + content: [{ type: "text" as const, text: "unused" }], + }), + ); + await server.connect(serverTransport); + await client.connect(clientTransport); + try { + const listed = await client.listTools(); + return Buffer.byteLength(JSON.stringify(listed), "utf8"); + } finally { + await client.close(); + await server.close(); + } + }; + + const fullBytes = await measure(false); + const compactBytes = await measure(true); + expect(fullBytes).toBeGreaterThan(1024 * 1024); + expect(compactBytes).toBeLessThan(1024 * 1024); + expect(compactBytes).toBeLessThan(fullBytes); +}); diff --git a/tests/boundary/mcp/serverIdentity.test.ts b/tests/boundary/mcp/serverIdentity.test.ts index bd6f64471..934d0d18f 100644 --- a/tests/boundary/mcp/serverIdentity.test.ts +++ b/tests/boundary/mcp/serverIdentity.test.ts @@ -139,6 +139,20 @@ describe("server and catalog identity", () => { ); }); + it("changes catalog identity when advertised input examples change", () => { + const contract = TOOL_CONTRACTS.find((item) => item.examples.length > 0); + if (contract === undefined) throw new Error("expected an input example"); + const original = createCatalogDigests(); + const withoutExamples = createCatalogDigests( + TOOL_CONTRACTS.map((item) => + item === contract ? { ...item, examples: [] } : item, + ), + ); + expect(withoutExamples.tools_sha256).not.toBe(original.tools_sha256); + expect(withoutExamples.combined_sha256).not.toBe(original.combined_sha256); + expect(withoutExamples.prompts_sha256).toBe(original.prompts_sha256); + }); + it("reports unknown without a live comparison and distinguishes aligned from stale", () => { const unknown = createServerIdentity({ startedAt: "2026-07-13T00:00:00.000Z",