diff --git a/docs/reference/mcp-contracts.md b/docs/reference/mcp-contracts.md index ac29dd285..766510cfb 100644 --- a/docs/reference/mcp-contracts.md +++ b/docs/reference/mcp-contracts.md @@ -78,6 +78,12 @@ unknown metadata. Missing omission counts remain null. Summary output carries every matching record; actual transport admission still applies. Omitted `detail` preserves complete canonical bundle delivery; filters require summary. +`list_unknowns` returns every matching current head by default. For a smaller +inline response, supply `offset` and `limit`, then follow `next_offset` until it +is null. `total` counts all matching heads; each page preserves complete records +in stable unknown ID order. Filters apply before paging. Pages reflect current +session state, so keep the unknown ledger unchanged while traversing it. + For a portable exported bundle, the CLI shares this projection: `rea inspect-evidence-bundle '{"path":"/tmp/evidence.json","detail":"summary","filters":{"operation":"analyze_function"}}' --format json`. The CLI validates and materializes that local bundle under its existing memory diff --git a/docs/specs/documentation/organization/verification.md b/docs/specs/documentation/organization/verification.md index e30331e80..190de0f0b 100644 --- a/docs/specs/documentation/organization/verification.md +++ b/docs/specs/documentation/organization/verification.md @@ -1,7 +1,7 @@ # Verification after upstream integration -Integrated upstream 82cfcee19f18b0eeae9a330ba04e0411cdf85225, preserving its -new signal-recording guide, managed summary documentation and current translations. +Integrated upstream c462a4ebd, preserving its new signal-recording guide, managed +summary documentation, current translations and unknown-head pagination contract. - Documentation facts and README consistency checks passed. - Production VitePress build passed. @@ -15,6 +15,9 @@ new signal-recording guide, managed summary documentation and current translatio ten files. CLI runs need process-table access for actual ownership/cleanup verification; restricted execution failed with spawn EPERM, while the normal host run passed without changing timeouts or assertions. +- After the subsequent upstream advance, 14 tests across unknown-head pagination, + workflow replay and target-content framing passed. Static checks, documentation + facts and all 56 published routes passed again against that base. - Documentation CI and the manual build require the built-page verifier. An actual CI-planner regression ensures edits to the verifier select that lane. diff --git a/experiments/analysis-efficiency/unknown-pagination.md b/experiments/analysis-efficiency/unknown-pagination.md new file mode 100644 index 000000000..da84f4450 --- /dev/null +++ b/experiments/analysis-efficiency/unknown-pagination.md @@ -0,0 +1,60 @@ +# Selected unknown retrieval + +`list_unknowns` now accepts optional paging. Omitted limits still return all +matching heads. Selection happens before payload cloning in the ledger owner; +MCP returns complete selected records and the total matching count. + +## Bounded workflow comparison + +The task is to retrieve the first ten complete current unknown heads, in stable +ID order, from a session containing 200 heads. The complete-list workflow reads +all heads and selects ten locally. The selected-page workflow requests +`{"offset":0,"limit":10}`. Both use one real SDK MCP call through +`Client`/`InMemoryTransport`, with the same imported session and no provider. +All six executions returned identical selected records and the expected total. + +These are two workflows on the candidate implementation, **not** a historical +before/after binary comparison. The candidate was built from `f09bb8a99` plus +the paging changes. Runtime file-table SHA-256 for the preserved complete-list +copy: `06a457dac90912dceba88f945573897372701f6c6b2e0934d7028023b4646648`. +Node 24.18.0, locked MCP SDK 2.3.1, tokenx 1.3.0, macOS arm64. + +Flameox 0.2.9 ran three randomized paired blocks, seed 731, with a 30-second +and 1-GiB limit per execution. Each fresh process performed a first read and +five identical warm reads. The semantic oracle checked the selected answer +SHA-256, `c18d718a71323eb53712040b4b09ffebfc3497000d1ff8f58d57665638077310`. +Every execution and oracle passed. + +| Measurement | Complete list | Selected page | +| ----------------------------------------- | ------------: | ------------: | +| Calls / errors per task | 1 / 0 | 1 / 0 | +| Serialized response bytes | 646,855 | 32,603 | +| Estimated request tokens | 12 | 21 | +| Estimated response tokens | 151,097 | 7,647 | +| Estimated full catalog tokens | 764,512 | 764,512 | +| Median first-read elapsed milliseconds | 1,084.05 | 1,082.65 | +| Median warm-read elapsed milliseconds | 3.943 | 0.426 | +| Median first-read CPU milliseconds | 1,333.23 | 1,328.47 | +| Median warm-read CPU milliseconds | 5.012 | 0.591 | +| Median whole-process elapsed milliseconds | 1,752.40 | 1,664.42 | +| Median peak process-family MiB | 598.59 | 554.09 | + +Read timings include the MCP call and answer validation. First-read timings +exclude startup, imports, session preparation, handshake and catalog discovery; +the whole-process elapsed measurement includes those, all six reads, token +estimation and cleanup. Whole-process CPU was not measured. Peak RSS includes +preparation and imports; this workload has no child processes. Host caches were +not flushed. Three pairs and fifteen warm reads per workflow describe this +fixture only; they do not establish broad speed or memory improvements. + +Token counts estimate serialized request, response and catalog JSON, including +both MCP response representations. They are not measured model tokens. Host +schema injection, prompts, reasoning, retries and actual agent behavior are +unmeasured. The full catalog cost is unchanged between these workflows and can +dominate a first task. Smaller selected delivery does not remove that cost. + +Native Flameox evidence was preserved locally as +`14ea4c0ef6595a1b89645299d4e184a582d5e975f7916294d68c1f1b312608d0`, +analysis `f66e125179623591ce00b1ccf0d917c2e90512de55db1061269c042d65e1125d`. +An earlier 512-MiB capture exceeded its memory limit and is excluded from these +measurements. No production provider or model trial was run. diff --git a/scripts/verify-real-ghidra-mips-relocatable.mjs b/scripts/verify-real-ghidra-mips-relocatable.mjs index da8a821b1..700b93b8d 100755 --- a/scripts/verify-real-ghidra-mips-relocatable.mjs +++ b/scripts/verify-real-ghidra-mips-relocatable.mjs @@ -124,9 +124,7 @@ async function verifyFixture(byteOrder, kernelModule) { }); const call = async (name, args = {}) => requireMcpOperationResult( - await client.callTool({ name, arguments: args }, undefined, { - timeout: 180000, - }), + await client.callTool({ name, arguments: args }, { timeout: 180000 }), name, ); try { diff --git a/scripts/verify-real-ghidra-mips.mjs b/scripts/verify-real-ghidra-mips.mjs index 2b6f07c85..1c04ea978 100644 --- a/scripts/verify-real-ghidra-mips.mjs +++ b/scripts/verify-real-ghidra-mips.mjs @@ -171,7 +171,6 @@ try { const call = async (name, args = {}) => { const reply = await client.callTool( { name, arguments: args }, - undefined, { timeout: 180000 }, ); return requireMcpOperationResult(reply, name); @@ -184,7 +183,6 @@ try { name: "open_binary", arguments: { path: fixture.path, provider_id: "ghidra" }, }, - undefined, { timeout: 60000 }, ); const error = requireMcpToolError(rejected); @@ -423,7 +421,6 @@ try { snapshot_path: snapshotPath, }, }, - undefined, { timeout: 180000 }, ); requireMcpToolError(wrongSnapshot); diff --git a/scripts/verify/javascript/recovery.mjs b/scripts/verify/javascript/recovery.mjs index 1d9f30eb4..67ed72deb 100644 --- a/scripts/verify/javascript/recovery.mjs +++ b/scripts/verify/javascript/recovery.mjs @@ -41,9 +41,10 @@ const cli = async (...args) => { return parseEvidence(JSON.parse(stdout)); }; const call = async (name, args) => { - const result = await client.callTool({ name, arguments: args }, undefined, { - timeout: 150000, - }); + const result = await client.callTool( + { name, arguments: args }, + { timeout: 150000 }, + ); assert.notEqual(result.isError, true, mcpTextValue(result)); return parseEvidence(JSON.parse(mcpTextValue(result))); }; diff --git a/src/application/MipsElfMetadata.ts b/src/application/MipsElfMetadata.ts index 8f2d97115..8fd9d0e89 100644 --- a/src/application/MipsElfMetadata.ts +++ b/src/application/MipsElfMetadata.ts @@ -3,6 +3,7 @@ import type { MipsAbiFlags, MipsElfMetadata, } from "../domain/binaryTargetTypes.js"; +import { readElfFixedHeader } from "../domain/elfFixedHeader.js"; import { err, ok, type Result } from "../domain/result.js"; import { readMipsElfSharedLoad, @@ -64,21 +65,22 @@ export const readMipsElfMetadata = async ( const header = await read(0, 52, "ELF32 header"); if (!header.ok) return header; const bytes = header.value; + const fixedHeader = readElfFixedHeader(bytes, fileSize); + if (!fixedHeader.ok) return fixedHeader; const little = metadata.byteOrder === "little"; const u16 = (data: Buffer, offset: number): number => little ? data.readUInt16LE(offset) : data.readUInt16BE(offset); const u32 = (data: Buffer, offset: number): number => little ? data.readUInt32LE(offset) : data.readUInt32BE(offset); if ( - bytes.toString("hex", 0, 4) !== "7f454c46" || - bytes[4] !== 1 || - bytes[5] !== (little ? 1 : 2) || + fixedHeader.value.elfClass !== 32 || + fixedHeader.value.byteOrder !== metadata.byteOrder || u16(bytes, 18) !== 8 || u16(bytes, 16) !== metadata.type || u32(bytes, 36) !== metadata.flags ) return err("MIPS ELF header no longer matches the resolved identity"); - if (bytes[6] !== 1 || u32(bytes, 20) !== 1 || u16(bytes, 40) !== 52) + if (fixedHeader.value.headerSize !== 52) return err("invalid MIPS ELF32 version or header size"); const tables = [ diff --git a/src/application/binary/BinarySessionRecords.ts b/src/application/binary/BinarySessionRecords.ts index 5a6bb4dd0..456ca70ef 100644 --- a/src/application/binary/BinarySessionRecords.ts +++ b/src/application/binary/BinarySessionRecords.ts @@ -19,6 +19,10 @@ import type { UnknownStatus, UpdateUnknownInput, } from "../../domain/residualUnknown.js"; +import type { + UnknownPage, + UnknownQuery, +} from "../investigation/InvestigationRecordPort.js"; import { err, ok, type Result } from "../../domain/result.js"; import type { AnalysisExecution, @@ -350,6 +354,10 @@ export abstract class BinarySessionRecords { return this.#records.listUnknowns(filters); } + listUnknownsPage(query: UnknownQuery = {}): UnknownPage { + return this.#records.listUnknownsPage(query); + } + verifyUnknownResolution(unknownId: string): Result< { readonly valid: boolean; diff --git a/src/application/investigation/EvidenceLedger.ts b/src/application/investigation/EvidenceLedger.ts index 7a5cd9f63..ce806ad38 100644 --- a/src/application/investigation/EvidenceLedger.ts +++ b/src/application/investigation/EvidenceLedger.ts @@ -26,6 +26,7 @@ import { } from "../../domain/residualUnknown.js"; import { err, ok, type Result } from "../../domain/result.js"; import { compareUnicodeCodePoints } from "../../domain/unicodeCodePointOrder.js"; +import type { UnknownPage, UnknownQuery } from "./InvestigationRecordPort.js"; type EvidenceLedgerFailure = EvidenceIntegrityError; type RecordResult = Result<"added" | "duplicate", EvidenceLedgerFailure>; @@ -326,18 +327,39 @@ export class EvidenceLedger { readonly domain?: string; } = {}, ): ResidualUnknown[] { - return [...this.#unknownHeads.values()] + return this.listUnknownsPage(filters).items; + } + + /** Select a stable page of current heads before detaching record payloads. */ + listUnknownsPage(query: UnknownQuery = {}): UnknownPage { + const offset = query.offset ?? 0; + const matching = [...this.#unknownHeads.values()] .filter( (unknown) => - (filters.status === undefined || unknown.status === filters.status) && - (filters.severity === undefined || - unknown.severity === filters.severity) && - (filters.domain === undefined || unknown.domain === filters.domain), + (query.status === undefined || unknown.status === query.status) && + (query.severity === undefined || + unknown.severity === query.severity) && + (query.domain === undefined || unknown.domain === query.domain), ) .sort((left, right) => compareUnicodeCodePoints(left.unknown_id, right.unknown_id), - ) - .map((unknown) => structuredClone(unknown)); + ); + const start = Math.min(offset, matching.length); + const end = + query.limit === undefined + ? undefined + : start + Math.min(query.limit, matching.length - start); + const page = matching.slice(start, end); + const nextOffset = + offset <= matching.length && start + page.length < matching.length + ? start + page.length + : null; + return { + items: page.map((unknown) => structuredClone(unknown)), + total: matching.length, + offset, + next_offset: nextOffset, + }; } /** Return current resolution validity; imported invalid states are rejected. */ diff --git a/src/application/investigation/InvestigationRecordPort.ts b/src/application/investigation/InvestigationRecordPort.ts index efaf9700b..2ac18a207 100644 --- a/src/application/investigation/InvestigationRecordPort.ts +++ b/src/application/investigation/InvestigationRecordPort.ts @@ -38,6 +38,21 @@ export interface EvidenceUnknownWriter { ): Result; } +export type UnknownPage = { + readonly items: ResidualUnknown[]; + readonly total: number; + readonly offset: number; + readonly next_offset: number | null; +}; + +export interface UnknownQuery { + readonly status?: UnknownStatus; + readonly severity?: ResidualUnknown["severity"]; + readonly domain?: string; + readonly offset?: number; + readonly limit?: number; +} + /** Optimistic Unknown revisions and consistency verification. */ export interface UnknownRegistryPort { recordUnknown( @@ -51,6 +66,7 @@ export interface UnknownRegistryPort { readonly severity?: ResidualUnknown["severity"]; readonly domain?: string; }): ResidualUnknown[]; + listUnknownsPage(query?: UnknownQuery): UnknownPage; verifyUnknownResolution(unknownId: string): Result< { readonly valid: boolean; diff --git a/src/application/investigation/InvestigationRecords.ts b/src/application/investigation/InvestigationRecords.ts index 29b93e246..ea6edb46f 100644 --- a/src/application/investigation/InvestigationRecords.ts +++ b/src/application/investigation/InvestigationRecords.ts @@ -19,6 +19,8 @@ import type { EvidenceWriter, EvidenceUnknownWriter, UnknownRegistryPort, + UnknownPage, + UnknownQuery, } from "./InvestigationRecordPort.js"; /** One investigation's validated record owner, independent of binary execution. */ @@ -124,6 +126,10 @@ export class InvestigationRecords return this.#evidence.listUnknowns(filters); } + listUnknownsPage(query: UnknownQuery = {}): UnknownPage { + return this.#evidence.listUnknownsPage(query); + } + /** Verify registry consistency without asserting the answer is true. */ verifyUnknownResolution(unknownId: string): Result< { diff --git a/src/artifacts/ArtifactResourceScope.ts b/src/artifacts/ArtifactResourceScope.ts index 59fdcba36..7c0bcf2c2 100644 --- a/src/artifacts/ArtifactResourceScope.ts +++ b/src/artifacts/ArtifactResourceScope.ts @@ -6,6 +6,7 @@ import type { OwnedStagedOutput } from "../filesystem/OwnedStagedOutput.js"; import type { ArtifactReader } from "./ArtifactReader.js"; import { ArtifactReaderFailure } from "./ArtifactReader.js"; import type { SafeOutputTree } from "./SafeOutputTree.js"; +import type { PrivateRuntimeRoot } from "../process/PrivateRuntimeRoot.js"; /** One artifact resource and its format-specific cleanup owner. */ export type ArtifactResourceOwner = @@ -38,6 +39,11 @@ export type ArtifactResourceOwner = readonly kind: "staged-output"; readonly resource: string; readonly output: OwnedStagedOutput; + } + | { + readonly kind: "private-runtime-root"; + readonly resource: string; + readonly root: Pick; }; export type ArtifactCleanupAttempt = @@ -161,5 +167,8 @@ const cleanup = async (owner: ArtifactResourceOwner): Promise => { return; case "staged-output": await owner.output.close(); + return; + case "private-runtime-root": + await owner.root.close(); } }; diff --git a/src/artifacts/apple/InterfaceBuilderDecodeBudget.ts b/src/artifacts/apple/InterfaceBuilderDecodeBudget.ts index 8469d93f0..3593fa43e 100644 --- a/src/artifacts/apple/InterfaceBuilderDecodeBudget.ts +++ b/src/artifacts/apple/InterfaceBuilderDecodeBudget.ts @@ -170,8 +170,8 @@ export interface BinaryPlistExpansionOptions { readonly fail?: (kind: BinaryPlistExpansionFailure, message: string) => Error; /** * Structural bounds: "strict" keeps objects and the offset table before the - * trailer; "decoder" admits exactly what plist.parseBinary can read, so - * expansion is bounded without rejecting archives that decoder accepts. + * trailer; "decoder" follows the decoder's object-table boundary policy + * while still rejecting structures whose coercion would lose plist facts. */ readonly bounds?: "strict" | "decoder"; } @@ -267,6 +267,8 @@ export const estimateBinaryPlistExpansion = ( throw new TypeError("binary plist object offset is invalid"); const marker = bytes[offset] ?? 0; const type = marker >> 4; + if (entry.dictionaryKey && type !== 5 && type !== 6) + throw new TypeError("binary plist dictionary key is not a string"); let size = marker & 0x0f; let cursor = offset + 1; if (size === 0x0f && type !== 0 && type !== 8) { diff --git a/src/artifacts/apple/KeyedArchiveReader.test.ts b/src/artifacts/apple/KeyedArchiveReader.test.ts index b619788a1..a0b0f1b73 100644 --- a/src/artifacts/apple/KeyedArchiveReader.test.ts +++ b/src/artifacts/apple/KeyedArchiveReader.test.ts @@ -219,6 +219,37 @@ const keyedRoot = (payload: readonly Buffer[]): Buffer => describe("binary plist reference preflight", () => { const selection = { offset: 0, limit: 10 }; + it("rejects an array key that coerces to __proto__ before parsing", () => { + expect(() => + decodeKeyedArchiveBytes( + keyedRoot([ + plistArray([9]), + Buffer.from([0xd1, 0, 10, 0, 11]), + plistArray([12]), + Buffer.from([0xd1, 0, 13, 0, 14]), + plistString("__proto__"), + plistString("lost"), + plistString("value"), + ]), + selection, + ), + ).toThrow("binary plist dictionary key is not a string"); + }); + + it("rejects binary data dictionary keys", () => { + expect(() => + decodeKeyedArchiveBytes( + keyedRoot([ + plistArray([9]), + Buffer.from([0xd1, 0, 10, 0, 11]), + Buffer.from([0x41, 0xab]), + plistString("value"), + ]), + selection, + ), + ).toThrow("binary plist dictionary key is not a string"); + }); + it("reports a reference cycle as malformed, not as an oversized archive", () => { expect(() => decodeKeyedArchiveBytes(keyedRoot([plistArray([8])]), selection), diff --git a/src/browser/BrowserScenario.test.ts b/src/browser/BrowserScenario.test.ts index 2262f410a..ee736ee5e 100644 --- a/src/browser/BrowserScenario.test.ts +++ b/src/browser/BrowserScenario.test.ts @@ -80,6 +80,49 @@ const baseScenario = () => ({ }, }); +describe("browser scenario secret redaction", () => { + it("matches percent-escape hex case without case-folding raw secrets", () => { + const secrets = BrowserScenarioSecrets.resolve( + browserScenarioSchema.parse(baseScenario()), + { + REA_TEST_SESSION: "a/b?", + REA_TEST_PASSWORD: "password", + REA_TEST_COOKIE: "cookie", + }, + ); + if (secrets === undefined) throw new Error("Expected resolved secrets"); + + expect(secrets.redact("logged a%2fb%3f, a%2Fb%3F and a%2fb%3F")).toBe( + "logged [REDACTED:session], [REDACTED:session] and [REDACTED:session]", + ); + expect(secrets.redact("raw A/B? remains case-sensitive")).toBe( + "raw A/B? remains case-sensitive", + ); + expect( + secrets.redactBytes( + Buffer.concat([ + Buffer.from([0xff, 0]), + Buffer.from("a%2fb%3F"), + Buffer.from([0x80]), + ]), + ), + ).toEqual( + Buffer.concat([ + Buffer.from([0xff, 0]), + Buffer.from("[REDACTED:session]"), + Buffer.from([0x80]), + ]), + ); + expect( + secrets.sanitizeUrl("https://app.example.test/?token=a%2fb%3F").url, + ).toBe("https://app.example.test/?token=[REDACTED:session]"); + expect(secrets.fingerprint("stored a%2fb%3F")).toEqual({ + value_state: "redacted-secret", + value_sha256: null, + }); + }); +}); + describe("browserScenarioSchema", () => { it("accepts and normalizes a declared scenario", () => { const parsed = browserScenarioSchema.parse(baseScenario()); diff --git a/src/browser/BrowserScenarioSecrets.ts b/src/browser/BrowserScenarioSecrets.ts index 05fb99047..aa583c016 100644 --- a/src/browser/BrowserScenarioSecrets.ts +++ b/src/browser/BrowserScenarioSecrets.ts @@ -15,84 +15,229 @@ const REDACTION_PREFIX = "[REDACTED:"; const encodedSecretValues = (secret: string): ReadonlySet => new Set([ - secret, encodeURIComponent(secret), new URLSearchParams([["value", secret]]).toString().slice("value=".length), ]); -type SecretSpellings = "raw" | "url" | "bytes"; +type SecretSpellings = "url" | "text"; +interface SecretPattern { + readonly text: string; + readonly searchText: string; + readonly percentInsensitive: boolean; +} +interface SecretReplacement extends SecretPattern { + readonly id: string; +} +interface SecretNeedle { + readonly id: string; + readonly needle: Buffer; + readonly percentInsensitive: boolean; + readonly marker: Buffer; +} -const secretSpellings = ( - secret: string, - spellings: SecretSpellings, -): readonly string[] => { - if (secret === "") return []; - if (spellings === "raw") return [secret]; - const values = [...encodedSecretValues(secret)]; - if (spellings === "bytes") values.push(JSON.stringify(secret).slice(1, -1)); - return [...new Set(values)].filter((value) => value !== ""); +const normalizePercentEscapes = (value: string): string => { + if (!value.includes("%")) return value; + return value.replace(/%[0-9a-f]{2}/giu, (escape) => escape.toLowerCase()); }; -const declaredLiterals = ( +const secretPatterns = ( + secret: string, + spellings: SecretSpellings, +): readonly SecretPattern[] => { + if (secret === "") return []; + const patterns: SecretPattern[] = [ + { text: secret, searchText: secret, percentInsensitive: false }, + ...[...encodedSecretValues(secret)].map((text) => ({ + text, + searchText: normalizePercentEscapes(text), + percentInsensitive: true, + })), + ]; + if (spellings === "text") + patterns.push({ + text: JSON.stringify(secret).slice(1, -1), + searchText: JSON.stringify(secret).slice(1, -1), + percentInsensitive: false, + }); + const seen = new Set(); + return patterns.filter(({ text, percentInsensitive }) => { + if (text === "") return false; + const key = `${percentInsensitive ? "percent" : "literal"}:${text}`; + if (seen.has(key)) return false; + seen.add(key); + return true; + }); +}; + +const declaredPatterns = ( values: ReadonlyMap, spellings: SecretSpellings, -): readonly string[] => [ - ...new Set( - [...values.values()].flatMap((secret) => - secretSpellings(secret, spellings), - ), - ), -]; +): readonly SecretPattern[] => { + const seen = new Set(); + return [...values.values()] + .flatMap((secret) => secretPatterns(secret, spellings)) + .filter(({ text, percentInsensitive }) => { + const key = `${percentInsensitive ? "percent" : "literal"}:${text}`; + if (seen.has(key)) return false; + seen.add(key); + return true; + }); +}; -/** Pick a marker that does not itself contain a declared secret spelling. */ -const redactionMarker = (id: string, literals: readonly string[]): string => - [`${REDACTION_PREFIX}${id}]`, "[REDACTED]", "…", ""].find((candidate) => - literals.every((literal) => !candidate.includes(literal)), - ) ?? ""; - -const replaceSecretLiterals = ( - value: string, +const secretReplacements = ( values: ReadonlyMap, spellings: SecretSpellings, - literals: readonly string[], -): string => { - const replacements = [...values] +): readonly SecretReplacement[] => + [...values] .flatMap(([id, secret]) => - secretSpellings(secret, spellings).map((text) => ({ id, text })), + secretPatterns(secret, spellings).map((pattern) => ({ id, ...pattern })), ) .sort( (left, right) => right.text.length - left.text.length || (left.id < right.id ? -1 : left.id > right.id ? 1 : 0), ); - let output = value; - for (const replacement of replacements) - output = output.replaceAll( - replacement.text, - redactionMarker(replacement.id, literals), - ); - return output; + +const normalizePercentEscapesBytes = (value: Buffer): Buffer => { + if (value.indexOf(37) < 0) return value; + const normalized = Buffer.from(value); + for (let index = 0; index + 2 < normalized.length; index += 1) + if ( + normalized[index] === 37 && + isHexByte(normalized[index + 1]) && + isHexByte(normalized[index + 2]) + ) { + normalized[index + 1] = lowercaseHexByte(normalized[index + 1]!); + normalized[index + 2] = lowercaseHexByte(normalized[index + 2]!); + index += 2; + } + return normalized; +}; + +const isHexByte = (value: number | undefined): boolean => + value !== undefined && + ((value >= 48 && value <= 57) || + (value >= 65 && value <= 70) || + (value >= 97 && value <= 102)); + +const lowercaseHexByte = (value: number): number => + value >= 65 && value <= 70 ? value + 32 : value; + +const patternMatches = ( + value: string, + normalizedValue: string, + pattern: SecretPattern, +): boolean => + (pattern.percentInsensitive ? normalizedValue : value).includes( + pattern.searchText, + ); + +const containsSecretPattern = ( + value: string, + normalizedValue: string, + patterns: readonly SecretPattern[], +): boolean => { + if (patterns.length === 0) return false; + return patterns.some((pattern) => + patternMatches(value, normalizedValue, pattern), + ); +}; + +const containsSecretBytePattern = ( + value: Buffer, + normalizedValue: Buffer, + needles: readonly SecretNeedle[], +): boolean => + needles.some((needle) => + (needle.percentInsensitive ? normalizedValue : value).includes( + needle.needle, + ), + ); + +/** Pick a marker that does not itself contain a declared secret spelling. */ +const redactionMarker = ( + id: string, + patterns: readonly SecretPattern[], +): string => { + const candidates = [`${REDACTION_PREFIX}${id}]`, "[REDACTED]", "…", ""]; + return ( + candidates.find((candidate) => + patterns.every( + (pattern) => + !patternMatches( + candidate, + normalizePercentEscapes(candidate), + pattern, + ), + ), + ) ?? "" + ); +}; + +const replaceSecretLiterals = ( + value: string, + replacements: readonly SecretReplacement[], + patterns: readonly SecretPattern[], +): string => { + const normalizedValue = replacements.some( + ({ percentInsensitive }) => percentInsensitive, + ) + ? normalizePercentEscapes(value) + : value; + const parts: string[] = []; + let offset = 0; + while (offset < value.length) { + let next: + | ((typeof replacements)[number] & { readonly index: number }) + | undefined; + for (const replacement of replacements) { + const search = replacement.searchText; + const view = replacement.percentInsensitive ? normalizedValue : value; + const index = view.indexOf(search, offset); + if ( + index >= 0 && + (next === undefined || + index < next.index || + (index === next.index && replacement.text.length > next.text.length)) + ) + next = { ...replacement, index }; + } + if (next === undefined) break; + const marker = redactionMarker(next.id, patterns); + parts.push(value.slice(offset, next.index), marker); + offset = next.index + next.text.length; + } + if (parts.length === 0) return value; + parts.push(value.slice(offset)); + return parts.join(""); }; const secretNeedles = ( - values: ReadonlyMap, - literals: readonly string[], -): Array<{ - readonly id: string; - readonly needle: Buffer; - readonly marker: Buffer; -}> => - [...values].flatMap(([id, secret]) => - secretSpellings(secret, "bytes").map((text) => ({ - id, - needle: Buffer.from(text), - marker: Buffer.from(redactionMarker(id, literals)), - })), - ); + replacements: readonly SecretReplacement[], + patterns: readonly SecretPattern[], +): SecretNeedle[] => + replacements.map((pattern) => ({ + id: pattern.id, + needle: Buffer.from(pattern.searchText), + percentInsensitive: pattern.percentInsensitive, + marker: Buffer.from(redactionMarker(pattern.id, patterns)), + })); /** Resolved secret values kept only for one in-memory scenario session. */ export class BrowserScenarioSecrets { - private constructor(private readonly values: ReadonlyMap) {} + private readonly urlPatterns: readonly SecretPattern[]; + private readonly textPatterns: readonly SecretPattern[]; + private readonly urlReplacements: readonly SecretReplacement[]; + private readonly textReplacements: readonly SecretReplacement[]; + private readonly textNeedles: readonly SecretNeedle[]; + + private constructor(private readonly values: ReadonlyMap) { + this.urlPatterns = declaredPatterns(values, "url"); + this.textPatterns = declaredPatterns(values, "text"); + this.urlReplacements = secretReplacements(values, "url"); + this.textReplacements = secretReplacements(values, "text"); + this.textNeedles = secretNeedles(this.textReplacements, this.textPatterns); + } static resolve( scenario: BrowserScenario, @@ -127,18 +272,40 @@ export class BrowserScenarioSecrets { } redact(value: string): string { - const literals = declaredLiterals(this.values, "raw"); - const output = replaceSecretLiterals(value, this.values, "raw", literals); - return literals.some((literal) => output.includes(literal)) ? "" : output; + const patterns = this.textPatterns; + if (patterns.length === 0) return value; + const output = replaceSecretLiterals( + value, + this.textReplacements, + patterns, + ); + return containsSecretPattern( + output, + normalizePercentEscapes(output), + patterns, + ) + ? "" + : output; } - /** Replace declared UTF-8, URI/form, and JSON-escaped secrets without decoding binary data. */ + /** Replace declared UTF-8, URI/form, and JSON-escaped secret spellings. */ redactBytes(value: Buffer): Buffer { - const literals = declaredLiterals(this.values, "bytes"); - const candidates = secretNeedles(this.values, literals) + const patterns = this.textPatterns; + if (patterns.length === 0) return value; + const candidates = this.textNeedles; + const hasPercentPatterns = candidates.some( + ({ percentInsensitive }) => percentInsensitive, + ); + const normalizedValue = + hasPercentPatterns && value.indexOf(37) >= 0 + ? normalizePercentEscapesBytes(value) + : value; + const sortedCandidates = candidates .map((candidate) => ({ ...candidate, - index: value.indexOf(candidate.needle), + index: (candidate.percentInsensitive ? normalizedValue : value).indexOf( + candidate.needle, + ), })) .sort( (left, right) => @@ -148,13 +315,18 @@ export class BrowserScenarioSecrets { const parts: Buffer[] = []; let offset = 0; while (true) { - let next: (typeof candidates)[number] | undefined; - for (const candidate of candidates) { + let next: (typeof sortedCandidates)[number] | undefined; + for (const candidate of sortedCandidates) { if (candidate.index >= 0 && candidate.index < offset) - candidate.index = value.indexOf(candidate.needle, offset); + candidate.index = ( + candidate.percentInsensitive ? normalizedValue : value + ).indexOf(candidate.needle, offset); if ( candidate.index >= 0 && - (next === undefined || candidate.index < next.index) + (next === undefined || + candidate.index < next.index || + (candidate.index === next.index && + candidate.needle.length > next.needle.length)) ) next = candidate; } @@ -165,16 +337,23 @@ export class BrowserScenarioSecrets { if (parts.length === 0) return value; parts.push(value.subarray(offset)); const redacted = Buffer.concat(parts); - return literals.some((literal) => redacted.includes(Buffer.from(literal))) + const normalizedRedacted = + hasPercentPatterns && redacted.indexOf(37) >= 0 + ? normalizePercentEscapesBytes(redacted) + : redacted; + return containsSecretBytePattern(redacted, normalizedRedacted, candidates) ? Buffer.alloc(0) : redacted; } /** Remove only declared secret values from one URL and report that action. */ sanitizeUrl(value: string): SanitizedBrowserUrl { - const literals = declaredLiterals(this.values, "url"); - const output = replaceSecretLiterals(value, this.values, "url", literals); - if (literals.some((literal) => output.includes(literal))) + const patterns = this.urlPatterns; + if (patterns.length === 0) return sanitizeBrowserUrl(value); + const output = replaceSecretLiterals(value, this.urlReplacements, patterns); + if ( + containsSecretPattern(output, normalizePercentEscapes(output), patterns) + ) return { url: "", origin: null, @@ -189,14 +368,15 @@ export class BrowserScenarioSecrets { } fingerprint(value: string): BrowserStorageValueFingerprint { - for (const secret of this.values.values()) - if (secret !== "" && value.includes(secret)) - return { value_state: "redacted-secret", value_sha256: null }; + const patterns = this.textPatterns; + if ( + patterns.length > 0 && + containsSecretPattern(value, normalizePercentEscapes(value), patterns) + ) + return { value_state: "redacted-secret", value_sha256: null }; return { value_state: "hashed", - value_sha256: createHash("sha256") - .update(this.redact(value)) - .digest("hex"), + value_sha256: createHash("sha256").update(value).digest("hex"), }; } } diff --git a/src/browser/CdpEndpoint.ts b/src/browser/CdpEndpoint.ts index 0c4834e24..cf78df183 100644 --- a/src/browser/CdpEndpoint.ts +++ b/src/browser/CdpEndpoint.ts @@ -1,4 +1,4 @@ -import { request } from "node:http"; +import { request, type ClientRequest, type IncomingMessage } from "node:http"; import { z } from "zod"; @@ -29,6 +29,17 @@ const endpointTargetSchema = z.object({ }); const endpointTargetsSchema = z.array(endpointTargetSchema); +/** Per-response bounds for loopback CDP discovery documents. */ +const CDP_DISCOVERY_RESPONSE_LIMITS = { + bytes: 8 * 1024 * 1024, + timeoutMs: 10_000, +} as const; + +interface CdpDiscoveryResponseLimits { + readonly bytes: number; + readonly timeoutMs: number; +} + /** Validated direct CDP WebSocket bound to one discovered page target. */ interface CdpPageWebSocketEndpoint { readonly scope: "page"; @@ -291,29 +302,71 @@ export const readCdpJson = async ( url: URL, operation: BrowserObservationOperation, signal?: AbortSignal, + limits: CdpDiscoveryResponseLimits = CDP_DISCOVERY_RESPONSE_LIMITS, ): Promise => await new Promise((resolve, reject) => { if (signal?.aborted === true) { reject(new AnalysisCancelledError(operation)); return; } - const request_ = request( + let response: IncomingMessage | undefined; + let request_: ClientRequest | undefined; + let timer: NodeJS.Timeout | undefined; + let settled = false; + const removeAbort = (): void => + signal?.removeEventListener("abort", onAbort); + const finish = (error?: Error, value?: unknown): void => { + if (settled) return; + settled = true; + if (timer !== undefined) clearTimeout(timer); + removeAbort(); + if (error !== undefined) { + reject(error); + response?.destroy(); + request_?.destroy(); + } else resolve(value); + }; + const onAbort = (): void => { + if (signal?.aborted === true) + finish(new AnalysisCancelledError(operation)); + }; + request_ = request( url, - { method: "GET", signal, headers: { Accept: "application/json" } }, - (response) => { - if (response.statusCode !== 200) { - response.resume(); - reject( + { method: "GET", headers: { Accept: "application/json" } }, + (received) => { + response = received; + if (settled) { + received.destroy(); + return; + } + if (received.statusCode !== 200) { + finish( new BrowserObservationError(operation, "invalid_endpoint_response"), ); return; } const chunks: Buffer[] = []; - response.on("data", (chunk: Buffer) => chunks.push(chunk)); - response.on("end", () => { - const parsed = safeParseJson(Buffer.concat(chunks).toString("utf8")); + let bytes = 0; + received.on("data", (chunk: Buffer) => { + if (settled) return; + bytes += chunk.byteLength; + if (bytes > limits.bytes) { + finish( + new BrowserObservationError(operation, "payload_limit", { + detail: `CDP discovery response exceeds its ${String(limits.bytes)} byte limit.`, + }), + ); + return; + } + chunks.push(chunk); + }); + received.on("end", () => { + if (settled) return; + const parsed = safeParseJson( + Buffer.concat(chunks, bytes).toString("utf8"), + ); if (!parsed.ok) { - reject( + finish( new BrowserObservationError( operation, "invalid_endpoint_response", @@ -322,15 +375,41 @@ export const readCdpJson = async ( ); return; } - resolve(parsed.value); + finish(undefined, parsed.value); }); - response.on("error", (cause: unknown) => - reject(endpointFailure(cause, operation, signal)), + received.on("aborted", () => + finish( + new BrowserObservationError(operation, "invalid_endpoint_response"), + ), + ); + received.on("close", () => { + if (!settled && !received.complete) + finish( + new BrowserObservationError( + operation, + "invalid_endpoint_response", + ), + ); + }); + received.on("error", (cause: unknown) => + finish(endpointFailure(cause, operation, signal)), ); }, ); request_.on("error", (cause: unknown) => - reject(endpointFailure(cause, operation, signal)), + finish(endpointFailure(cause, operation, signal)), + ); + signal?.addEventListener("abort", onAbort, { once: true }); + onAbort(); + if (settled) return; + timer = setTimeout( + () => + finish( + new BrowserObservationError(operation, "timeout", { + detail: `CDP discovery response exceeded its ${String(limits.timeoutMs)} ms deadline.`, + }), + ), + limits.timeoutMs, ); request_.end(); }); diff --git a/src/browser/PlaywrightScenarioEvents.test.ts b/src/browser/PlaywrightScenarioEvents.test.ts index 4c96e8d3d..4e7841365 100644 --- a/src/browser/PlaywrightScenarioEvents.test.ts +++ b/src/browser/PlaywrightScenarioEvents.test.ts @@ -5,6 +5,77 @@ import { browserScenarioSchema } from "../domain/browserScenario.js"; import { BrowserScenarioSecrets } from "./BrowserScenarioSecrets.js"; import { PlaywrightScenarioEvents } from "./PlaywrightScenarioEvents.js"; +describe("declared secrets in scenario events", () => { + it("redacts encoded declared secrets from text evidence and fingerprints", () => { + const listeners = new Map void>(); + const socketListeners = new Map void>(); + const page = { + on: (name: string, listener: (value: unknown) => void) => { + listeners.set(name, listener); + }, + } as unknown as Page; + const secret = "a/b&c\n d"; + const scenario = browserScenarioSchema.parse({ + browser: { mode: "launch", executable_path: "/opt/chromium" }, + start_url: { url: "https://app.example.test/" }, + actions: [ + { step_id: "wait", action: "wait_for_timeout", duration_ms: 1 }, + ], + secrets: [ + { secret_id: "login_input", environment_variable: "REA_TEST_TOKEN" }, + ], + capture: { events: ["console", "page-errors", "websockets"] }, + }); + const secrets = BrowserScenarioSecrets.resolve(scenario, { + REA_TEST_TOKEN: secret, + }); + if (secrets === undefined) throw new Error("Expected resolved secrets"); + const events = new PlaywrightScenarioEvents({ + page, + enabled: new Set(["console", "page-errors", "websockets"]), + secrets, + }); + + listeners.get("console")?.({ + type: () => "log", + text: () => encodeURIComponent(secret), + location: () => ({ url: "https://app.example.test/app.js" }), + }); + const escapedSecret = JSON.stringify(secret).slice(1, -1); + const error = new Error(`failure ${escapedSecret}`); + error.stack = `failure ${escapedSecret}`; + listeners.get("pageerror")?.(error); + listeners.get("websocket")?.({ + url: () => "wss://app.example.test/socket", + on: (name: string, listener: (value: unknown) => void) => { + socketListeners.set(name, listener); + }, + }); + const formEncodedSecret = new URLSearchParams([["value", secret]]) + .toString() + .slice("value=".length); + socketListeners.get("framesent")?.({ payload: formEncodedSecret }); + + const items = events.result().items; + expect(items).toMatchObject([ + { kind: "console", text: "[REDACTED:login_input]" }, + { + kind: "page-error", + message: "failure [REDACTED:login_input]", + stack: "failure [REDACTED:login_input]", + }, + { kind: "websocket-opened" }, + { + kind: "websocket-frame-sent", + payload_text: "[REDACTED:login_input]", + }, + ]); + expect(secrets.fingerprint(`stored ${encodeURIComponent(secret)}`)).toEqual( + { value_state: "redacted-secret", value_sha256: null }, + ); + }); +}); + describe("PlaywrightScenarioEvents", () => { it("observes requested popup page errors independently of popup lifecycle events", () => { const listeners = new Map void>(); @@ -42,7 +113,7 @@ describe("PlaywrightScenarioEvents", () => { ]); }); - it("bounds oversized page errors before validating captured events", () => { + it("retains oversized page errors without truncating their text", () => { const listeners = new Map void>(); const page = { url: () => "https://app.example.test/", @@ -88,7 +159,7 @@ describe("PlaywrightScenarioEvents", () => { const error = new Error("m".repeat(70_000)); error.stack = "s".repeat(300_000); - expect(() => listeners.get("pageerror")?.(error)).not.toThrow(); + listeners.get("pageerror")?.(error); expect(events.result().items[0]).toMatchObject({ kind: "page-error", message: "m".repeat(70_000), diff --git a/src/contracts/sessionToolContracts.ts b/src/contracts/sessionToolContracts.ts index b3ea2e318..73fe37e64 100644 --- a/src/contracts/sessionToolContracts.ts +++ b/src/contracts/sessionToolContracts.ts @@ -139,7 +139,7 @@ export const SESSION_TOOL_CONTRACTS = [ ), session( "list_unknowns", - "List every current residual-unknown head in deterministic ID order, with optional exact status, severity, and domain filters. Results are complete and inline. This is read-only; unresolved, contradicted, and non-truth dispositions remain distinct.", + "List current residual-unknown heads in deterministic ID order, with optional exact status, severity, and domain filters. Results are complete inline by default; supply offset and limit to page the filtered order. total is the matching-head count and next_offset is null when the page exhausts the result. This is read-only; unresolved, contradicted, and non-truth dispositions remain distinct.", listUnknownsInputSchema, ), session( diff --git a/src/contracts/sessionToolSchemas.ts b/src/contracts/sessionToolSchemas.ts index 5ef55b967..f48a5bcfb 100644 --- a/src/contracts/sessionToolSchemas.ts +++ b/src/contracts/sessionToolSchemas.ts @@ -52,6 +52,22 @@ export const listUnknownsInputSchema = z.strictObject({ .optional(), severity: z.enum(["low", "medium", "high", "critical"]).optional(), domain: z.string().trim().min(1).optional(), + offset: z + .number() + .int() + .min(0) + .max(Number.MAX_SAFE_INTEGER) + .default(0) + .describe("Zero-based offset in the stable filtered unknown ID order."), + limit: z + .number() + .int() + .positive() + .max(Number.MAX_SAFE_INTEGER) + .optional() + .describe( + "Optional maximum number of heads to return. Omit to return every matching head from offset.", + ), }); /** Exact residual-unknown identity to revalidate. */ diff --git a/src/contracts/toolOutputSchemaGroups.ts b/src/contracts/toolOutputSchemaGroups.ts index 7dd02cf63..e828486ba 100644 --- a/src/contracts/toolOutputSchemaGroups.ts +++ b/src/contracts/toolOutputSchemaGroups.ts @@ -555,6 +555,8 @@ export const sessionOutputSchemas = { z.object({ items: z.array(residualUnknownSchema), total: z.number().int().min(0), + offset: z.number().int().min(0), + next_offset: z.number().int().min(0).nullable(), }), ), record_unknown: lifecycleResultOf(residualUnknownSchema), diff --git a/src/contracts/unknownContractExamples.ts b/src/contracts/unknownContractExamples.ts index f68c0120a..d3e690ead 100644 --- a/src/contracts/unknownContractExamples.ts +++ b/src/contracts/unknownContractExamples.ts @@ -2,7 +2,7 @@ import type { JsonValue } from "../domain/jsonValue.js"; /** Exact example inputs for residual-unknown lifecycle contracts. */ export const UNKNOWN_CONTRACT_EXAMPLES = { - list_unknowns: {}, + list_unknowns: { offset: 0, limit: 50 }, record_unknown: { question: "Does this branch require an unavailable external service?", severity: "medium", diff --git a/src/domain/android/androidApplication.ts b/src/domain/android/androidApplication.ts index eb6c671e7..c46fb9661 100644 --- a/src/domain/android/androidApplication.ts +++ b/src/domain/android/androidApplication.ts @@ -1,6 +1,9 @@ import { z } from "zod"; -import { parseArtifactInventoryEvidence } from "../artifactInventoryEvidence.js"; +import { + hasCompleteInventoryCoverage, + parseArtifactInventoryEvidence, +} from "../artifactInventoryEvidence.js"; import { digestSchema } from "../digests.js"; import { prefixedDigestSchema } from "../digests.js"; import { nativeRuntimeConvention } from "../nativeConvention.js"; @@ -62,6 +65,7 @@ export const androidApplicationProjectionResultSchema = z.strictObject({ bridge_candidate_coverage: bridgeCandidateCoverageSchema, coverage: z.strictObject({ status: z.enum(["complete-within-inventory", "partial"]), + // Count completeness is separate from the declared coverage status. inventory_complete: z.boolean(), }), limitations: z.array(z.string().min(1)), @@ -95,8 +99,8 @@ export const projectAndroidApplication = ( ); const components = classified; const limitations = [ - ...(!inventory.complete - ? ["Source inventory pages are incomplete; absence is unknown."] + ...(!hasCompleteInventoryCoverage(inventory) + ? ["Source inventory coverage is partial; absence is unknown."] : []), "Manifest, resource, signing, and bytecode semantics require a dedicated Android provider; this projection reports exact inventory paths and hashes only.", "Runtime families are inferred from inventory formats and paths; filename suffixes do not establish valid DEX or JVM class bytes.", diff --git a/src/domain/apple/appleApplication.test.ts b/src/domain/apple/appleApplication.test.ts index 8acde829f..9caf21fcb 100644 --- a/src/domain/apple/appleApplication.test.ts +++ b/src/domain/apple/appleApplication.test.ts @@ -138,6 +138,7 @@ const inventoryEvidence = ( occurrences: sortedOccurrences, edges: [], provenance: [], + coverage_status: "complete", limitations: [], }), }, diff --git a/src/domain/apple/appleApplication.ts b/src/domain/apple/appleApplication.ts index bbf6519ee..d46ddf26b 100644 --- a/src/domain/apple/appleApplication.ts +++ b/src/domain/apple/appleApplication.ts @@ -17,7 +17,10 @@ import { isSwiftRuntimeLibrary, nativeRuntimeConvention, } from "../nativeConvention.js"; -import { parseArtifactInventoryEvidence } from "../artifactInventoryEvidence.js"; +import { + hasCompleteInventoryCoverage, + parseArtifactInventoryEvidence, +} from "../artifactInventoryEvidence.js"; import { evidenceSchema } from "../evidence.js"; import { digestSchema } from "../digests.js"; import { prefixedDigestSchema } from "../digests.js"; @@ -88,6 +91,7 @@ export const appleApplicationProjectionResultSchema = z.strictObject({ bridge_candidate_coverage: bridgeCandidateCoverageSchema, coverage: z.strictObject({ status: z.enum(["complete-within-inventory", "partial"]), + // Count completeness is separate from the declared coverage status. inventory_complete: z.boolean(), }), limitations: z.array(z.string().min(1)), @@ -165,9 +169,10 @@ export const projectAppleApplication = ( ); const uninventoriedImage = rootFormat === "dmg" && inventory.occurrences.length === 1; - const complete = inventory.complete && !uninventoriedImage; + const inventoryComplete = hasCompleteInventoryCoverage(inventory); + const complete = inventoryComplete && !uninventoriedImage; const limitations = projectionLimitations({ - complete: inventory.complete, + complete: inventoryComplete, uninventoriedImage, roots, bundles, @@ -390,7 +395,7 @@ const projectionLimitations = (facts: { readonly bridgeCandidateCoverage: AppleApplicationProjectionResult["bridge_candidate_coverage"]; }): string[] => [ ...(!facts.complete - ? ["Source inventory pages are incomplete; absence is unknown."] + ? ["Source inventory coverage is partial; absence is unknown."] : []), ...(facts.uninventoriedImage ? [ diff --git a/src/domain/artifactComparison.ts b/src/domain/artifactComparison.ts index e47b28a22..9d7a766af 100644 --- a/src/domain/artifactComparison.ts +++ b/src/domain/artifactComparison.ts @@ -8,6 +8,7 @@ import type { } from "./artifactGraph.js"; import { evidenceSchema } from "./evidence.js"; import { + hasCompleteInventoryCoverage, parseArtifactInventoryEvidence, type InventorySet, } from "./artifactInventoryEvidence.js"; @@ -103,10 +104,8 @@ export const compareArtifacts = ( const right = parseArtifactInventoryEvidence(rightEvidence); const leftComplete = left.inventory.complete; const rightComplete = right.inventory.complete; - const leftCovered = - leftComplete && left.inventory.coverage_status === "complete"; - const rightCovered = - rightComplete && right.inventory.coverage_status === "complete"; + const leftCovered = hasCompleteInventoryCoverage(left.inventory); + const rightCovered = hasCompleteInventoryCoverage(right.inventory); const limitations = [ ...left.inventory.limitations.map((item) => `Left: ${item}`), ...right.inventory.limitations.map((item) => `Right: ${item}`), diff --git a/src/domain/artifactInventoryEvidence.ts b/src/domain/artifactInventoryEvidence.ts index 6d2528ec1..213364816 100644 --- a/src/domain/artifactInventoryEvidence.ts +++ b/src/domain/artifactInventoryEvidence.ts @@ -28,10 +28,16 @@ export interface InventorySet { readonly edges: readonly ArtifactInventoryResult["edges"][number][]; readonly integrityContradictions: readonly IntegrityContradiction[]; readonly limitations: readonly string[]; + /** True when merged node, occurrence, and edge counts match the manifest. */ readonly complete: boolean; readonly coverage_status: ArtifactInventoryCoverage; } +/** Whether all graph members are present and the producer reports full coverage. */ +export const hasCompleteInventoryCoverage = ( + inventory: InventorySet, +): boolean => inventory.complete && inventory.coverage_status === "complete"; + export interface ParsedInventorySet { readonly evidence: readonly Evidence[]; readonly inventory: InventorySet; diff --git a/src/domain/binaryTarget.fixture.ts b/src/domain/binaryTarget.fixture.ts index 3e2c46147..7d8a0a7f9 100644 --- a/src/domain/binaryTarget.fixture.ts +++ b/src/domain/binaryTarget.fixture.ts @@ -5,9 +5,16 @@ export const elf = ( machine: number, ): Buffer => { const bytes = Buffer.alloc(class_ === 1 ? 52 : 64); - bytes.set([0x7f, 0x45, 0x4c, 0x46, class_, endian]); - if (endian === 1) bytes.writeUInt16LE(machine, 18); - else bytes.writeUInt16BE(machine, 18); + bytes.set([0x7f, 0x45, 0x4c, 0x46, class_, endian, 1]); + if (endian === 1) { + bytes.writeUInt16LE(machine, 18); + bytes.writeUInt32LE(1, 20); + bytes.writeUInt16LE(class_ === 1 ? 52 : 64, class_ === 1 ? 40 : 52); + } else { + bytes.writeUInt16BE(machine, 18); + bytes.writeUInt32BE(1, 20); + bytes.writeUInt16BE(class_ === 1 ? 52 : 64, class_ === 1 ? 40 : 52); + } return bytes; }; diff --git a/src/domain/binaryTarget.mips.test.ts b/src/domain/binaryTarget.mips.test.ts index e646daee1..ded2f18bf 100644 --- a/src/domain/binaryTarget.mips.test.ts +++ b/src/domain/binaryTarget.mips.test.ts @@ -7,10 +7,14 @@ const header = (little: boolean, bits: 32 | 64 = 32): Buffer => { if (little) { bytes.writeUInt16LE(2, 16); bytes.writeUInt16LE(8, 18); + bytes.writeUInt32LE(1, 20); + bytes.writeUInt16LE(bits === 32 ? 52 : 64, bits === 32 ? 40 : 52); bytes.writeUInt32LE(0x70001001, bits === 32 ? 36 : 48); } else { bytes.writeUInt16BE(2, 16); bytes.writeUInt16BE(8, 18); + bytes.writeUInt32BE(1, 20); + bytes.writeUInt16BE(bits === 32 ? 52 : 64, bits === 32 ? 40 : 52); bytes.writeUInt32BE(0x70001001, bits === 32 ? 36 : 48); } return bytes; diff --git a/src/domain/binaryTarget.riscv.test.ts b/src/domain/binaryTarget.riscv.test.ts index fcc77cf89..8af185e12 100644 --- a/src/domain/binaryTarget.riscv.test.ts +++ b/src/domain/binaryTarget.riscv.test.ts @@ -13,10 +13,14 @@ const header = ( if (little) { bytes.writeUInt16LE(type, 16); bytes.writeUInt16LE(machine, 18); + bytes.writeUInt32LE(1, 20); + bytes.writeUInt16LE(bits === 32 ? 52 : 64, bits === 32 ? 40 : 52); bytes.writeUInt32LE(flags, bits === 32 ? 36 : 48); } else { bytes.writeUInt16BE(type, 16); bytes.writeUInt16BE(machine, 18); + bytes.writeUInt32BE(1, 20); + bytes.writeUInt16BE(bits === 32 ? 52 : 64, bits === 32 ? 40 : 52); bytes.writeUInt32BE(flags, bits === 32 ? 36 : 48); } return bytes; diff --git a/src/domain/binaryTarget.test.ts b/src/domain/binaryTarget.test.ts index 7d4cf322c..6164e8c22 100644 --- a/src/domain/binaryTarget.test.ts +++ b/src/domain/binaryTarget.test.ts @@ -26,6 +26,13 @@ describe("executable header parsing", () => { expect(result.ok && result.value).toMatchObject({ format, architecture }); }); + it("retains ELF32 x86-64 identity for the x32 ABI", () => { + expect(parseExecutableHeader(elf(1, 1, 62), "x64")).toMatchObject({ + ok: true, + value: { format: "elf", architecture: "x86_64" }, + }); + }); + it("distinguishes native PE applications, DLLs, and managed assemblies", () => { const application = parseExecutableHeader(pe(0x8664), "x64"); const library = parseExecutableHeader(pe(0x8664, 64, 0x2002), "x64"); diff --git a/src/domain/binaryTarget.ts b/src/domain/binaryTarget.ts index eca1174d2..48da65960 100644 --- a/src/domain/binaryTarget.ts +++ b/src/domain/binaryTarget.ts @@ -7,6 +7,7 @@ import type { UniversalMachOSlice, } from "./binaryTargetTypes.js"; import { err, ok, type Result } from "./result.js"; +import { readElfFixedHeader } from "./elfFixedHeader.js"; import { observeMzWindowsHeader, parseDosMzHeader } from "./dosMz.js"; import { decodePeMachine } from "./peMachine.js"; import { @@ -191,20 +192,14 @@ const parseElf = ( bytes: Buffer, fileSize: number, ): Result => { - if (bytes.length < 6) return err("truncated ELF header"); - if (bytes[4] !== 1 && bytes[4] !== 2) return err("unsupported ELF class"); - const little = bytes[5] === 1; - if (!little && bytes[5] !== 2) return err("unsupported ELF endianness"); - const headerSize = bytes[4] === 1 ? 52 : 64; - if (bytes.length < headerSize) - return err( - `truncated ELF header: an ELF${bytes[4] === 1 ? 32 : 64} header needs ${headerSize} bytes; the file has ${fileSize}`, - ); + const fixedHeader = readElfFixedHeader(bytes, fileSize); + if (!fixedHeader.ok) return fixedHeader; + const { elfClass, byteOrder } = fixedHeader.value; + const little = byteOrder === "little"; const machine = little ? bytes.readUInt16LE(18) : bytes.readUInt16BE(18); const architecture = elfArchitecture(machine); if (architecture === undefined) return err("unsupported ELF architecture"); if (architecture === "mips" || architecture === "riscv") { - const elfClass = bytes[4] === 1 ? 32 : 64; const flagsOffset = elfClass === 32 ? 36 : 48; const metadata: ElfFamilyMetadata = { elfClass, diff --git a/src/domain/elfFixedHeader.test.ts b/src/domain/elfFixedHeader.test.ts new file mode 100644 index 000000000..b902635ac --- /dev/null +++ b/src/domain/elfFixedHeader.test.ts @@ -0,0 +1,61 @@ +import { expect, it } from "vitest"; +import { readElfFixedHeader } from "./elfFixedHeader.js"; +import { elf } from "./binaryTarget.fixture.js"; + +it.each([ + ["ELF32 little endian", 1, 1], + ["ELF32 big endian", 1, 2], + ["ELF64 little endian", 2, 1], + ["ELF64 big endian", 2, 2], +] as const)( + "reads common fixed header commitments for %s", + (_name, cls, data) => { + const bytes = elf(cls, data, cls === 1 ? 3 : 62); + expect(readElfFixedHeader(bytes)).toEqual({ + ok: true, + value: { + elfClass: cls === 1 ? 32 : 64, + byteOrder: data === 1 ? "little" : "big", + headerSize: cls === 1 ? 52 : 64, + }, + }); + }, +); + +it.each([ + [ + "identification version", + (bytes: Buffer) => { + bytes[6] = 0; + }, + ], + [ + "header version", + (bytes: Buffer) => { + bytes.writeUInt32LE(0, 20); + }, + ], + [ + "undersized declared header", + (bytes: Buffer) => { + bytes.writeUInt16LE(51, 40); + }, + ], + [ + "declared header beyond file", + (bytes: Buffer) => { + bytes.writeUInt16LE(53, 40); + }, + ], +])("rejects an invalid %s", (_name, corrupt) => { + const bytes = elf(1, 1, 3); + corrupt(bytes); + expect(readElfFixedHeader(bytes)).toMatchObject({ ok: false }); +}); + +it("checks declared header size against the whole file, not the prefix", () => { + const bytes = elf(2, 1, 62); + bytes.writeUInt16LE(128, 52); + expect(readElfFixedHeader(bytes, 128)).toMatchObject({ ok: true }); + expect(readElfFixedHeader(bytes, 127)).toMatchObject({ ok: false }); +}); diff --git a/src/domain/elfFixedHeader.ts b/src/domain/elfFixedHeader.ts new file mode 100644 index 000000000..3b98976c8 --- /dev/null +++ b/src/domain/elfFixedHeader.ts @@ -0,0 +1,55 @@ +import { err, ok, type Result } from "./result.js"; + +export interface ElfFixedHeader { + readonly elfClass: 32 | 64; + readonly byteOrder: "little" | "big"; + readonly headerSize: number; +} + +/** + * Validate the common, fixed ELF header commitments. + * + * `bytes` may be a bounded prefix. `totalBytes` is the containing file size, + * so the declared header extent is checked against the file without requiring + * the whole declared extent to be present in this prefix. + */ +export const readElfFixedHeader = ( + bytes: Buffer, + totalBytes = bytes.length, +): Result => { + if (bytes.length >= 4 && bytes.toString("hex", 0, 4) !== "7f454c46") + return err("not an ELF image"); + if (bytes.length < 16) return err("ELF identification is truncated"); + const classByte = bytes[4]; + if (classByte !== 1 && classByte !== 2) return err("unsupported ELF class"); + const dataByte = bytes[5]; + if (dataByte !== 1 && dataByte !== 2) + return err("unsupported ELF endianness"); + if (bytes[6] !== 1) return err("unsupported ELF identification version"); + + const elfClass = classByte === 1 ? 32 : 64; + const minimumSize = elfClass === 32 ? 52 : 64; + if ( + !Number.isSafeInteger(totalBytes) || + totalBytes < bytes.length || + totalBytes < minimumSize || + bytes.length < minimumSize + ) + return err( + `truncated ELF header: an ELF${elfClass} header needs ${minimumSize} bytes; the file has ${totalBytes}`, + ); + const littleEndian = dataByte === 1; + const readU16 = (offset: number): number => + littleEndian ? bytes.readUInt16LE(offset) : bytes.readUInt16BE(offset); + const readU32 = (offset: number): number => + littleEndian ? bytes.readUInt32LE(offset) : bytes.readUInt32BE(offset); + if (readU32(20) !== 1) return err("unsupported ELF header version"); + const headerSize = readU16(elfClass === 32 ? 40 : 52); + if (headerSize < minimumSize || headerSize > totalBytes) + return err("invalid ELF header size"); + return ok({ + elfClass, + byteOrder: littleEndian ? "little" : "big", + headerSize, + }); +}; diff --git a/src/domain/harmony/harmonyApplication.ts b/src/domain/harmony/harmonyApplication.ts index 51dacd148..71755e828 100644 --- a/src/domain/harmony/harmonyApplication.ts +++ b/src/domain/harmony/harmonyApplication.ts @@ -1,6 +1,9 @@ import { z } from "zod"; -import { parseArtifactInventoryEvidence } from "../artifactInventoryEvidence.js"; +import { + hasCompleteInventoryCoverage, + parseArtifactInventoryEvidence, +} from "../artifactInventoryEvidence.js"; import { digestSchema } from "../digests.js"; import { prefixedDigestSchema } from "../digests.js"; import { nativeRuntimeConvention } from "../nativeConvention.js"; @@ -66,6 +69,7 @@ export const harmonyApplicationProjectionResultSchema = z.strictObject({ bridge_candidate_coverage: bridgeCandidateCoverageSchema, coverage: z.strictObject({ status: z.enum(["complete-within-inventory", "partial"]), + // Count completeness is separate from the declared coverage status. inventory_complete: z.boolean(), }), limitations: z.array(z.string().min(1)), @@ -108,8 +112,8 @@ export const projectHarmonyApplication = ( classified.native_libraries, ); const limitations = [ - ...(!inventory.complete - ? ["Source inventory pages are incomplete; absence is unknown."] + ...(!hasCompleteInventoryCoverage(inventory) + ? ["Source inventory coverage is partial; absence is unknown."] : []), ...(unavailable.length > 0 ? [ diff --git a/src/domain/mobileApplicationGraphProjection.ts b/src/domain/mobileApplicationGraphProjection.ts index 328388ebe..93e4143c5 100644 --- a/src/domain/mobileApplicationGraphProjection.ts +++ b/src/domain/mobileApplicationGraphProjection.ts @@ -3,7 +3,10 @@ import { createHash } from "node:crypto"; import canonicalize from "canonicalize"; import { z } from "zod"; -import type { ParsedInventorySet } from "./artifactInventoryEvidence.js"; +import { + hasCompleteInventoryCoverage, + type ParsedInventorySet, +} from "./artifactInventoryEvidence.js"; import { evidenceSchema } from "./evidence.js"; import { digestSchema } from "./digests.js"; import { prefixedDigestSchema } from "./digests.js"; @@ -49,10 +52,11 @@ export const projectionCoverage = ( budgetComplete: boolean, ): { status: "complete-within-inventory" | "partial"; + /** True when all manifest graph-member counts are present, independent of source coverage. */ inventory_complete: boolean; } => ({ status: - inventory.complete && budgetComplete + hasCompleteInventoryCoverage(inventory) && budgetComplete ? ("complete-within-inventory" as const) : ("partial" as const), inventory_complete: inventory.complete, diff --git a/src/evm/EvmoleInterfaceAdapter.test.ts b/src/evm/EvmoleInterfaceAdapter.test.ts index a8b53f280..ea12143a1 100644 --- a/src/evm/EvmoleInterfaceAdapter.test.ts +++ b/src/evm/EvmoleInterfaceAdapter.test.ts @@ -61,6 +61,13 @@ it.each([ functions: [{ selector: "aabbccdd", bytecodeOffset: -1, dispatch: "abi" }], }, { metadata: {} }, + { + metadata: { + bytecodeOffset: 1, + cborLength: 4, + entries: [{ key: "reported", value: { type: "integer", value: 1.5 } }], + }, + }, ])("rejects producer representations outside the selected profile", (raw) => { expect(() => projectEvmoleInterface(raw)).toThrow(); }); diff --git a/src/evm/EvmoleInterfaceAdapter.ts b/src/evm/EvmoleInterfaceAdapter.ts index da62d28b9..7ffd57853 100644 --- a/src/evm/EvmoleInterfaceAdapter.ts +++ b/src/evm/EvmoleInterfaceAdapter.ts @@ -9,7 +9,11 @@ const reportedMetadata = z.strictObject({ key: z.string(), value: z.discriminatedUnion("type", [ z.strictObject({ type: z.literal("string"), value: z.string() }), - z.strictObject({ type: z.literal("integer"), value: z.number() }), + z.strictObject({ + type: z.literal("integer"), + // Reported integers can exceed the safe range; fractions still violate the tag. + value: z.number().refine(Number.isInteger, "Expected an integer"), + }), z.strictObject({ type: z.literal("bytes"), value: z.string() }), z.strictObject({ type: z.literal("bool"), value: z.boolean() }), z.strictObject({ type: z.literal("undecoded"), value: z.string() }), diff --git a/src/flutter/DartElf.ts b/src/flutter/DartElf.ts index 48285737d..a9bf00fa5 100644 --- a/src/flutter/DartElf.ts +++ b/src/flutter/DartElf.ts @@ -5,6 +5,7 @@ import { SNAPSHOT_HASH_OFFSET, SNAPSHOT_HASH_PATTERN, } from "./FlutterPayloadScan.js"; +import { readElfFixedHeader } from "../domain/elfFixedHeader.js"; /** Bounds-checked little-endian reader over one loaded buffer. */ class BoundedReader { @@ -80,24 +81,13 @@ export const readElfHeader = ( totalBytes: number = bytes.length, ) => { const reader = new BoundedReader(bytes); - const magic = reader.u32(0); - if (magic !== 0x464c457f) return { failure: { reason: "not an ELF image" } }; - const elfClass = reader.u8(4); - if (elfClass !== 1 && elfClass !== 2) - return { failure: { reason: `unsupported ELF class ${String(elfClass)}` } }; - const is64 = elfClass === 2; + const fixedHeader = readElfFixedHeader(bytes, totalBytes); + if (!fixedHeader.ok) return { failure: { reason: fixedHeader.error } }; + const is64 = fixedHeader.value.elfClass === 64; const headerBytes = is64 ? ELF_HEADER_MAX_BYTES : 52; - if ( - bytes.length < headerBytes || - !Number.isSafeInteger(totalBytes) || - totalBytes < headerBytes - ) - return { failure: { reason: "ELF header is truncated" } }; - if (reader.u8(5) !== 1) + if (fixedHeader.value.byteOrder !== "little") return { failure: { reason: "unsupported ELF byte order" } }; - if (reader.u8(6) !== 1 || reader.u32(0x14) !== 1) - return { failure: { reason: "unsupported ELF version" } }; - if (reader.u16(is64 ? 0x34 : 0x28) !== headerBytes) + if (fixedHeader.value.headerSize !== headerBytes) return { failure: { reason: "unsupported ELF header size" } }; const e_type = reader.u16(0x10); if (e_type !== 1 && e_type !== 2 && e_type !== ET_DYN) diff --git a/src/go/GoElfImage.ts b/src/go/GoElfImage.ts index f02473aae..fae00c40c 100644 --- a/src/go/GoElfImage.ts +++ b/src/go/GoElfImage.ts @@ -6,31 +6,19 @@ import { type GoFileMapping, type GoZeroFillMapping, } from "./GoBinaryContainer.js"; +import { readElfFixedHeader } from "../domain/elfFixedHeader.js"; /** Read ELF build-info sections and file-backed program mappings. */ export const readGoElfImage = (bytes: Buffer): GoBinaryContainer => { - if (bytes.length < 16) - throw new GoBinaryFormatFailure( - "malformed", - "ELF identification is truncated", - ); - const bits = bytes[4] === 1 ? 32 : bytes[4] === 2 ? 64 : null; - if (bits === null || ![1, 2].includes(bytes[5] ?? 0) || bytes[6] !== 1) - throw new GoBinaryFormatFailure( - "malformed", - "ELF class, byte order, or version is invalid", - ); - const reader = new GoBinaryReader(bytes, bytes[5] === 1); + const fixedHeader = readElfFixedHeader(bytes); + if (!fixedHeader.ok) + throw new GoBinaryFormatFailure("malformed", fixedHeader.error); + const { elfClass: bits, headerSize } = fixedHeader.value; + const reader = new GoBinaryReader( + bytes, + fixedHeader.value.byteOrder === "little", + ); const wide = bits === 64; - const minimum = wide ? 64 : 52; - reader.range(0, minimum, "ELF header"); - if (reader.u32(20) !== 1 || reader.u16(wide ? 52 : 40) < minimum) - throw new GoBinaryFormatFailure( - "malformed", - "ELF header version or size is invalid", - ); - const headerSize = reader.u16(wide ? 52 : 40); - reader.range(0, headerSize, "ELF declared header"); const phOffset = reader.word(wide ? 32 : 28, bits); const shOffset = reader.word(wide ? 40 : 32, bits); const phWidth = reader.u16(wide ? 54 : 42); diff --git a/src/native/NativeMacOSProvider.ts b/src/native/NativeMacOSProvider.ts index 2ae6a7cb6..1d1c2a36c 100644 --- a/src/native/NativeMacOSProvider.ts +++ b/src/native/NativeMacOSProvider.ts @@ -84,16 +84,24 @@ import { inspectNativeMacho, } from "./NativeMachoInspection.js"; import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js"; +import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js"; +import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js"; +import type { PrivateRuntimeRoot } from "../process/PrivateRuntimeRoot.js"; import { createNativeUiHelperRuntime, type NativeUiHelperRuntime, } from "./NativeUiHelperRuntime.js"; +type SignatureRuntimeRootFactory = () => Promise< + Pick +>; + /** Read-only semantic provider composed from Xcode command-line utilities. */ export class NativeMacOSProvider implements AnalysisProvider { readonly #capabilities: readonly CapabilityDescriptor[]; readonly #environment: NodeJS.ProcessEnv; readonly #tracerFactory: () => NativeCallTracer; + readonly #privateRuntimeRootFactory: SignatureRuntimeRootFactory | undefined; constructor( environment: Readonly, @@ -102,10 +110,12 @@ export class NativeMacOSProvider implements AnalysisProvider { ), platform: NodeJS.Platform = process.platform, tracerFactory?: () => NativeCallTracer, + privateRuntimeRootFactory?: SignatureRuntimeRootFactory, ) { this.#environment = snapshotEnvironment(environment, platform); this.#tracerFactory = tracerFactory ?? (() => new LldbCallTracer(this.#environment)); + this.#privateRuntimeRootFactory = privateRuntimeRootFactory; this.#capabilities = nativeMacOSCapabilities(platform); } @@ -123,6 +133,7 @@ export class NativeMacOSProvider implements AnalysisProvider { this.runner, this.#tracerFactory(), this.#environment, + this.#privateRuntimeRootFactory, ); } } @@ -130,6 +141,7 @@ export class NativeMacOSProvider implements AnalysisProvider { class NativeMacOSClient implements AnalysisClient { readonly #pendingNativeUiRuntimes = new Set(); readonly #pendingCommands = new Set>(); + readonly #signatureResources = new ArtifactResourceScope(); #nativeUiTail: Promise = Promise.resolve(); #closed = false; @@ -138,6 +150,9 @@ class NativeMacOSClient implements AnalysisClient { private readonly runner: NativeCommandRunner, private readonly tracer: NativeCallTracer, private readonly environment: Readonly, + private readonly privateRuntimeRootFactory: + | SignatureRuntimeRootFactory + | undefined, ) {} async execute( @@ -270,6 +285,21 @@ class NativeMacOSClient implements AnalysisClient { const uiCleanupFailures = await this.#serializeNativeUi(() => this.#retryNativeUiCleanup(), ); + let signatureCleanup: AnalysisError | undefined; + try { + await this.#signatureResources.close(); + } catch (cause: unknown) { + const observation = ArtifactReaderFailure.cleanupObservation( + cause, + "signature-snapshot", + ); + signatureCleanup = new ProviderCleanupError( + IDENTITY.id, + observation.resources, + { reason: observation.reason }, + { operation: "close_binary", cause }, + ); + } const commandCleanup = await this.#retryNativeCommandCleanup(); let result = await this.tracer.close(); const cleanupErrors = [ @@ -277,6 +307,7 @@ class NativeMacOSClient implements AnalysisClient { ? [] : [this.#nativeUiCleanupFailure(uiCleanupFailures, "close_binary")]), ...(commandCleanup === undefined ? [] : [commandCleanup]), + ...(signatureCleanup === undefined ? [] : [signatureCleanup]), ]; for (const cleanupError of cleanupErrors) result = result.ok @@ -532,8 +563,10 @@ class NativeMacOSClient implements AnalysisClient { ): Promise> { return withSignatureTarget( this.target, + this.#signatureResources, (binding) => this.#inspectBoundSignature(binding, signal), signal, + this.privateRuntimeRootFactory, ); } diff --git a/src/native/SignatureTargetBinding.test.ts b/src/native/SignatureTargetBinding.test.ts new file mode 100644 index 000000000..a5d91f253 --- /dev/null +++ b/src/native/SignatureTargetBinding.test.ts @@ -0,0 +1,198 @@ +import { createHash } from "node:crypto"; +import { mkdtemp, rm, stat, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, expect, it } from "vitest"; + +import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js"; +import type { BinaryTarget } from "../domain/binaryTargetTypes.js"; +import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; +import { ok } from "../domain/result.js"; +import type { NativeCommandRunner } from "./CommandRunner.js"; +import type { NativeCallTracer } from "./LldbCallTracer.js"; +import { NativeMacOSProvider } from "./NativeMacOSProvider.js"; +import { withSignatureTarget } from "./SignatureTargetBinding.js"; + +const roots: string[] = []; +const targets: string[] = []; +const unusedRunner: NativeCommandRunner = { + async run() { + throw new Error("Changed target must be rejected before a command runs"); + }, +}; +const unusedTracer: NativeCallTracer = { + async trace() { + throw new Error("Signature inspection must not start a trace"); + }, + async close() { + return ok(null); + }, +}; + +afterEach(async () => { + await Promise.all( + [...roots, ...targets].map((path) => + rm(path, { recursive: true, force: true }), + ), + ); + roots.length = 0; + targets.length = 0; +}); + +const createTarget = async ( + bytes: string, + digest = bytes, +): Promise => { + const directory = await mkdtemp(join(tmpdir(), "rea-signature-target-")); + targets.push(directory); + const path = join(directory, "target.bin"); + await writeFile(path, bytes); + return { + path, + sha256: createHash("sha256").update(digest).digest("hex"), + kind: "executable", + format: "mach-o", + architecture: "arm64", + availableArchitectures: ["arm64"], + }; +}; + +const createRoot = async (failedAttempts = 0) => { + const path = await mkdtemp(join(tmpdir(), "rea-signature-root-")); + roots.push(path); + let attempts = 0; + return { + path, + async close() { + if (attempts++ < failedAttempts) { + throw new Error("injected root cleanup failure"); + } + await rm(path, { recursive: true, force: true }); + }, + }; +}; + +it("retains a snapshot root after cleanup fails alongside a digest mismatch", async () => { + const target = await createTarget("observed", "expected"); + const resources = new ArtifactResourceScope(); + const result = await withSignatureTarget( + target, + resources, + async () => ok("unused"), + undefined, + () => createRoot(1), + ); + + expect(result.ok).toBe(false); + if (result.ok) return; + expect(result.error.cleanupIncomplete).toBe(true); + expect(result.error.cleanupResources).toEqual(expect.arrayContaining(roots)); + expect(projectAnalysisError(result.error)).toMatchObject({ + details: { diagnostics: { primary_error: { code: "artifact_changed" } } }, + }); + + await resources.close(); + await expect(stat(roots[0]!)).rejects.toMatchObject({ code: "ENOENT" }); +}); + +it("returns a retained cleanup failure when the next call cannot retry its root", async () => { + const target = await createTarget("observed", "expected"); + const resources = new ArtifactResourceScope(); + const createFailingRoot = async () => { + const path = await mkdtemp(join(tmpdir(), "rea-signature-root-sticky-")); + roots.push(path); + return { + path, + close: async () => { + throw new Error("still blocked"); + }, + }; + }; + const first = await withSignatureTarget( + target, + resources, + async () => ok("unused"), + undefined, + createFailingRoot, + ); + expect(first.ok).toBe(false); + const second = await withSignatureTarget( + target, + resources, + async () => ok("must not run"), + undefined, + createRoot, + ); + expect(second.ok).toBe(false); + if (second.ok) return; + expect(second.error.cleanupIncomplete).toBe(true); + expect(second.error.cleanupResources).toEqual(expect.arrayContaining(roots)); +}); + +it("retries a retained signature root when the provider client closes", async () => { + const target = await createTarget("observed", "expected"); + const provider = new NativeMacOSProvider( + {}, + unusedRunner, + "darwin", + () => unusedTracer, + () => createRoot(2), + ); + const client = provider.createClient(target); + const result = await client.execute("inspect_signature", {}); + + expect(result.ok).toBe(false); + if (result.ok) return; + expect(result.error.cleanupIncomplete).toBe(true); + expect(result.error.cleanupResources).toEqual(expect.arrayContaining(roots)); + const firstClose = await client.close(); + expect(firstClose).toMatchObject({ + ok: false, + error: { cleanupIncomplete: true, cleanupResources: roots }, + }); + expect((await stat(roots[0]!)).isDirectory()).toBe(true); + await expect(client.close()).resolves.toMatchObject({ + ok: true, + value: null, + }); + await expect(stat(roots[0]!)).rejects.toMatchObject({ code: "ENOENT" }); +}); + +it("waits for an in-flight signature binding before provider close", async () => { + const target = await createTarget("observed", "expected"); + let rootFactoryEntered!: () => void; + let releaseRootFactory!: () => void; + const entered = new Promise( + (resolve) => (rootFactoryEntered = resolve), + ); + const gate = new Promise((resolve) => (releaseRootFactory = resolve)); + const provider = new NativeMacOSProvider( + {}, + unusedRunner, + "darwin", + () => unusedTracer, + async () => { + rootFactoryEntered(); + await gate; + return createRoot(); + }, + ); + const client = provider.createClient(target); + const operation = client.execute("inspect_signature", {}); + await entered; + + let closeSettled = false; + const closing = client.close().finally(() => { + closeSettled = true; + }); + await Promise.resolve(); + expect(closeSettled).toBe(false); + releaseRootFactory(); + await expect(operation).resolves.toMatchObject({ + ok: false, + error: { _tag: "AnalysisArtifactChangedError" }, + }); + await expect(closing).resolves.toMatchObject({ ok: true, value: null }); + expect(closeSettled).toBe(true); + await expect(stat(roots[0]!)).rejects.toMatchObject({ code: "ENOENT" }); +}); diff --git a/src/native/SignatureTargetBinding.ts b/src/native/SignatureTargetBinding.ts index ac94a5f06..29c8651e6 100644 --- a/src/native/SignatureTargetBinding.ts +++ b/src/native/SignatureTargetBinding.ts @@ -1,7 +1,6 @@ import { createHash } from "node:crypto"; import { constants, type BigIntStats } from "node:fs"; -import { lstat, mkdtemp, open, rm, type FileHandle } from "node:fs/promises"; -import { tmpdir } from "node:os"; +import { lstat, open, type FileHandle } from "node:fs/promises"; import { basename, join } from "node:path"; import type { BinaryTarget } from "../domain/binaryTargetTypes.js"; import { AnalysisError } from "../domain/analysisErrorBase.js"; @@ -9,10 +8,17 @@ import { AnalysisAccessDeniedError, AnalysisArtifactChangedError, AnalysisCancelledError, + AnalysisOutputError, AnalysisResourceConstraintError, } from "../domain/analysisErrorCore.js"; import { ProviderAdapterError } from "../domain/providerAdapterError.js"; +import { ProviderCleanupError } from "../domain/providerCleanupError.js"; +import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js"; import { err, ok, type Result } from "../domain/result.js"; +import { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js"; +import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js"; +import { PrivateRuntimeRoot } from "../process/PrivateRuntimeRoot.js"; +import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js"; import { NATIVE_MACOS_PROVIDER_IDENTITY } from "./NativeMacOSProviderMetadata.js"; /** Binding of commands to the selected executable's registered content/version. */ @@ -68,41 +74,126 @@ const signatureReadFailure = ( /** Inspect a digest-verified private copy, releasing it on every exit path. */ export const withSignatureTarget = async ( target: BinaryTarget, + resources: ArtifactResourceScope, inspect: ( binding: SignatureTargetBinding, ) => Promise>, signal?: AbortSignal, + createRoot: () => Promise> = () => + PrivateRuntimeRoot.create({ prefix: "rea-signature-" }), ): Promise> => { - let directory: string; try { - signal?.throwIfAborted(); - directory = await mkdtemp(join(tmpdir(), "rea-signature-")); + return await resources.run(async () => { + let root: Pick; + try { + signal?.throwIfAborted(); + root = await createRoot(); + } catch (cause: unknown) { + return err( + signal?.aborted + ? new AnalysisCancelledError("inspect_signature") + : snapshotFailure(target, "create", cause), + ); + } + const owner = { + kind: "private-runtime-root" as const, + root, + resource: root.path, + }; + let result: Result; + try { + const binding = await bindSignatureTarget( + target, + join(root.path, basename(target.path)), + resources, + signal, + ); + if (!binding.ok) result = binding; + else { + try { + result = await inspect(binding.value); + } catch (cause: unknown) { + result = err( + cause instanceof AnalysisError + ? cause + : new AnalysisOutputError( + "inspect_signature", + "Native output parsing failed", + { cause }, + ), + ); + } + } + } catch (cause: unknown) { + result = err( + cause instanceof AnalysisError + ? cause + : signatureReadFailure(target, cause), + ); + } + const cleanup = await resources.release(owner); + if (cleanup.kind === "failed") + return combineCleanupFailure( + target, + result, + [root.path], + cleanup.cause, + "cleanup", + ); + return result; + }); } catch (cause: unknown) { + if (cause instanceof AnalysisError) return err(cause); + const retained = + cause instanceof ArtifactReaderFailure && cause.cleanup !== undefined + ? cause.cleanup + : undefined; + if (retained !== undefined) + return combineCleanupFailure( + target, + err( + signal?.aborted + ? new AnalysisCancelledError("inspect_signature") + : signatureReadFailure(target, cause), + ), + retained.resources, + cause, + "retry", + ); return err( signal?.aborted ? new AnalysisCancelledError("inspect_signature") - : snapshotFailure(target, "create", cause), + : signatureReadFailure(target, cause), ); } - let result: Result; - let cleanup: Result; - try { - const binding = await bindSignatureTarget( - target, - join(directory, basename(target.path)), - signal, - ); - result = binding.ok ? await inspect(binding.value) : binding; - } finally { - try { - await rm(directory, { recursive: true, force: true }); - cleanup = ok(undefined); - } catch (cause: unknown) { - cleanup = err(snapshotFailure(target, "cleanup", cause)); - } - } - // Cleanup cannot replace the operation's original actionable failure. - return result.ok && !cleanup.ok ? cleanup : result; +}; + +const combineCleanupFailure = ( + target: BinaryTarget, + primary: Result, + resources: readonly string[], + cause: unknown, + phase: string, +): Result => { + const cleanup = new ProviderCleanupError( + NATIVE_MACOS_PROVIDER_IDENTITY.id, + resources, + { + reason: cause instanceof Error ? cause.message : String(cause), + phase: `signature-snapshot-${phase}`, + path: target.path, + }, + { operation: "inspect_signature", cause }, + ); + return primary.ok + ? err(cleanup) + : err( + analysisErrorWithCleanupFailure( + primary.error, + cleanup, + "inspect_signature", + ), + ); }; const snapshotFailure = ( @@ -180,22 +271,28 @@ const copySignatureBytes = async ( const bindSignatureTarget = async ( target: BinaryTarget, snapshotPath: string, + resources: ArtifactResourceScope, signal?: AbortSignal, ): Promise> => { - try { - signal?.throwIfAborted(); - const before = await lstat(target.path, { bigint: true }); - signal?.throwIfAborted(); - if (!before.isFile() || before.isSymbolicLink()) - return changedTarget( - target, - `Signature target is no longer a regular file: ${target.path}`, - ); - const file = await open( - target.path, - constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK, - ); + let sourceOwner: OwnedFileHandle | undefined; + let snapshotOwner: OwnedFileHandle | undefined; + const perform = async (): Promise< + Result + > => { try { + signal?.throwIfAborted(); + const before = await lstat(target.path, { bigint: true }); + signal?.throwIfAborted(); + if (!before.isFile() || before.isSymbolicLink()) + return changedTarget( + target, + `Signature target is no longer a regular file: ${target.path}`, + ); + const file = await open( + target.path, + constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK, + ); + sourceOwner = new OwnedFileHandle(file); const opened = await file.stat({ bigint: true }); signal?.throwIfAborted(); if (identity(opened) !== identity(before)) @@ -206,44 +303,71 @@ const bindSignatureTarget = async ( const snapshot = await snapshotIO(target, "create-file", () => open(snapshotPath, "wx", 0o600), ); - try { - const observed = await copySignatureBytes( + snapshotOwner = new OwnedFileHandle(snapshot); + const observed = await copySignatureBytes( + target, + { file, stat: opened }, + snapshot, + signal, + ); + if (observed !== target.sha256) + return changedTarget( target, - { file, stat: opened }, - snapshot, - signal, + `Signature target digest changed: ${target.path}; expected ${target.sha256}, observed ${observed}`, ); - if (observed !== target.sha256) - return changedTarget( - target, - `Signature target digest changed: ${target.path}; expected ${target.sha256}, observed ${observed}`, - ); - const after = await file.stat({ bigint: true }); - const current = await lstat(target.path, { bigint: true }); - signal?.throwIfAborted(); - if ( - identity(after) !== identity(opened) || - identity(current) !== identity(opened) - ) - return changedTarget( - target, - `Signature target changed while establishing its version: ${target.path}`, - ); - await snapshotIO(target, "seal", () => snapshot.chmod(0o400)); - return ok({ identity: identity(opened), snapshotPath }); - } finally { - await snapshotIO(target, "close", () => snapshot.close()); - } - } finally { - await file.close(); + const after = await file.stat({ bigint: true }); + const current = await lstat(target.path, { bigint: true }); + signal?.throwIfAborted(); + if ( + identity(after) !== identity(opened) || + identity(current) !== identity(opened) + ) + return changedTarget( + target, + `Signature target changed while establishing its version: ${target.path}`, + ); + await snapshotIO(target, "seal", () => snapshot.chmod(0o400)); + return ok({ identity: identity(opened), snapshotPath }); + } catch (cause: unknown) { + return err( + signal?.aborted + ? new AnalysisCancelledError("inspect_signature") + : signatureReadFailure(target, cause), + ); } - } catch (cause: unknown) { - return err( - signal?.aborted - ? new AnalysisCancelledError("inspect_signature") - : signatureReadFailure(target, cause), + }; + let result = await perform(); + const cleanupFailures: { + readonly path: string; + readonly cause: unknown; + }[] = []; + for (const [owner, path] of [ + [snapshotOwner, snapshotPath], + [sourceOwner, target.path], + ] as const) { + if (owner === undefined) continue; + const cleanup = await resources.release({ + kind: "file-handle", + handle: owner, + resource: path, + }); + if (cleanup.kind === "failed") + cleanupFailures.push({ path, cause: cleanup.cause }); + } + if (cleanupFailures.length > 0) { + const cause = new AggregateError( + cleanupFailures.map(({ cause: failure }) => failure), + "Signature snapshot file handles could not be closed", + ); + result = combineCleanupFailure( + target, + result, + cleanupFailures.map(({ path }) => path), + cause, + "close", ); } + return result; }; /** Reject target drift across signature command captures. */ diff --git a/src/server/registerSessionRecordTools.ts b/src/server/registerSessionRecordTools.ts index 41efe981e..a41d60cac 100644 --- a/src/server/registerSessionRecordTools.ts +++ b/src/server/registerSessionRecordTools.ts @@ -151,18 +151,14 @@ const registerListUnknownsTool = ({ listContract.name, toolRegistrationOptions(listContract), (input) => { - const all = session.listUnknowns({ + const page = session.listUnknownsPage({ ...(input.status === undefined ? {} : { status: input.status }), ...(input.severity === undefined ? {} : { severity: input.severity }), ...(input.domain === undefined ? {} : { domain: input.domain }), + offset: input.offset, + ...(input.limit === undefined ? {} : { limit: input.limit }), }); - return server.delivery.toCallToolResult( - ok({ - items: all, - total: all.length, - }), - listContract, - ); + return server.delivery.toCallToolResult(ok(page), listContract); }, ); }; diff --git a/tests/boundary/browser/cdpEndpointResponseBounds.test.ts b/tests/boundary/browser/cdpEndpointResponseBounds.test.ts new file mode 100644 index 000000000..ac1fb41e2 --- /dev/null +++ b/tests/boundary/browser/cdpEndpointResponseBounds.test.ts @@ -0,0 +1,158 @@ +import { createServer } from "node:http"; +import type { RequestListener, Server } from "node:http"; + +import { afterEach, describe, expect, it } from "vitest"; + +import { readCdpJson } from "../../../src/browser/CdpEndpoint.js"; + +const servers: Server[] = []; + +afterEach(async () => { + await Promise.all( + servers.splice(0).map( + (server) => + new Promise((resolve, reject) => { + server.closeAllConnections(); + server.close((error) => + error === undefined ? resolve() : reject(error), + ); + }), + ), + ); +}); + +describe("CDP discovery response bounds", () => { + it("accepts a complete JSON response exactly at the byte limit", async () => { + const body = JSON.stringify("x".repeat(14)); + expect(Buffer.byteLength(body)).toBe(16); + const server = await startServer((_request, response) => + response.end(body), + ); + + await expect( + readCdpJson(server.url, "list_browser_targets", undefined, { + bytes: 16, + timeoutMs: 1_000, + }), + ).resolves.toBe("x".repeat(14)); + }); + + it("rejects and closes an oversized chunked response without parsing a prefix", async () => { + let responseClosed: (() => void) | undefined; + const closed = new Promise((resolve) => { + responseClosed = resolve; + }); + const server = await startServer((_request, response) => { + response.once("close", () => responseClosed?.()); + response.write(JSON.stringify("x".repeat(15))); + }); + + await expect( + readCdpJson(server.url, "list_browser_targets", undefined, { + bytes: 16, + timeoutMs: 1_000, + }), + ).rejects.toMatchObject({ + _tag: "BrowserObservationError", + reason: "payload_limit", + }); + await closed; + }); + + it("ends a response that exceeds the total elapsed deadline", async () => { + let responseClosed: (() => void) | undefined; + let requestStarted: (() => void) | undefined; + const closed = new Promise((resolve) => { + responseClosed = resolve; + }); + const started = new Promise((resolve) => { + requestStarted = resolve; + }); + const server = await startServer((_request, response) => { + response.once("close", () => responseClosed?.()); + requestStarted?.(); + }); + const pending = readCdpJson(server.url, "list_browser_targets", undefined, { + bytes: 16, + timeoutMs: 50, + }); + const timeoutAssertion = expect(pending).rejects.toMatchObject({ + _tag: "BrowserObservationError", + reason: "timeout", + }); + await started; + + await timeoutAssertion; + await closed; + }); + + it("preserves caller cancellation and closes the response", async () => { + let responseClosed: (() => void) | undefined; + let requestStarted: (() => void) | undefined; + const closed = new Promise((resolve) => { + responseClosed = resolve; + }); + const started = new Promise((resolve) => { + requestStarted = resolve; + }); + const server = await startServer((_request, response) => { + response.once("close", () => responseClosed?.()); + requestStarted?.(); + }); + const controller = new AbortController(); + const pending = readCdpJson( + server.url, + "list_browser_targets", + controller.signal, + { + bytes: 16, + timeoutMs: 1_000, + }, + ); + await started; + controller.abort(); + + await expect(pending).rejects.toMatchObject({ + _tag: "AnalysisCancelledError", + operation: "list_browser_targets", + }); + await closed; + }); + + it("destroys an unsuccessful HTTP response", async () => { + let responseClosed: (() => void) | undefined; + const closed = new Promise((resolve) => { + responseClosed = resolve; + }); + const server = await startServer((_request, response) => { + response.once("close", () => responseClosed?.()); + response.writeHead(503).flushHeaders(); + }); + + await expect( + readCdpJson(server.url, "list_browser_targets", undefined, { + bytes: 16, + timeoutMs: 1_000, + }), + ).rejects.toMatchObject({ + _tag: "BrowserObservationError", + reason: "invalid_endpoint_response", + }); + await closed; + }); +}); + +const startServer = async ( + handler: RequestListener, +): Promise<{ readonly url: URL }> => { + const server = createServer(handler); + servers.push(server); + await new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(0, "127.0.0.1", resolve); + }); + const address = server.address(); + if (address === null || typeof address === "string") + throw new TypeError("Expected a TCP listener address"); + return { url: new URL(`http://127.0.0.1:${String(address.port)}`) }; +}; diff --git a/tests/boundary/filesystem/android/application.test.ts b/tests/boundary/filesystem/android/application.test.ts index c976b591c..28fa4d8c8 100644 --- a/tests/boundary/filesystem/android/application.test.ts +++ b/tests/boundary/filesystem/android/application.test.ts @@ -227,6 +227,9 @@ describe("case-distinct APK resource names", () => { const inventory = parseEvidence( await runProviderAnalysis(path, "inventory_artifact", {}), ); + expect(inventory.normalized_result).toMatchObject({ + coverage_status: "partial", + }); const result = projectAndroidApplicationEvidence({ inventory_evidence: [inventory], }); @@ -234,6 +237,13 @@ describe("case-distinct APK resource names", () => { requireSuccessfulProjection(result).normalized_result, ); expect(projection.root_format).toBe("apk"); + expect(projection.coverage).toEqual({ + status: "partial", + inventory_complete: true, + }); + expect(projection.limitations).toContain( + "Source inventory coverage is partial; absence is unknown.", + ); expect(projection.components.manifests).toHaveLength(1); expect(projection.components.dex).toHaveLength(1); expect(JSON.stringify(inventory)).toContain("res/2F.xml"); diff --git a/tests/boundary/filesystem/appleApplication.test.ts b/tests/boundary/filesystem/appleApplication.test.ts index 7cd50246a..00ef24ba5 100644 --- a/tests/boundary/filesystem/appleApplication.test.ts +++ b/tests/boundary/filesystem/appleApplication.test.ts @@ -75,6 +75,42 @@ async function createCompleteAppleProjection() { } describe("Apple application projection", () => { + it("keeps a case-colliding real IPA inventory partial", async () => { + const root = await createTestTempDirectory("rea-apple-case-collision-"); + const path = join(root, "CaseCollision.ipa"); + const writer = new ZipWriter(new Uint8ArrayWriter()); + await writer.add( + "Payload/Example.app/Contents/Info.plist", + new TextReader("upper"), + ); + await writer.add( + "Payload/Example.app/contents/Info.plist", + new TextReader("lower"), + ); + await writeFile(path, await writer.close()); + + const inventory = parseEvidence( + await runProviderAnalysis(path, "inventory_artifact", {}), + ); + expect(inventory.normalized_result).toMatchObject({ + coverage_status: "partial", + }); + const result = projectAppleApplicationEvidence({ + inventory_evidence: [inventory], + }); + const projection = appleApplicationProjectionResultSchema.parse( + requireSuccessfulProjection(result).normalized_result, + ); + + expect(projection.coverage).toEqual({ + status: "partial", + inventory_complete: true, + }); + expect(projection.limitations).toContain( + "Source inventory coverage is partial; absence is unknown.", + ); + }); + it("projects deterministic IPA components and bridge hypotheses from exact inventory Evidence", async () => { const root = await createTestTempDirectory("rea-apple-"); const path = join(root, "Fixture.ipa"); diff --git a/tests/boundary/filesystem/artifactComparison.test.ts b/tests/boundary/filesystem/artifactComparison.test.ts index 38bfde264..ca40596d3 100644 --- a/tests/boundary/filesystem/artifactComparison.test.ts +++ b/tests/boundary/filesystem/artifactComparison.test.ts @@ -172,6 +172,9 @@ describe("artifact inventory coverage", () => { authority: "shipped-artifact", }, ); + expect(parseArtifactInventoryEvidence(partial).inventory.complete).toBe( + true, + ); expect(compareArtifacts(partial, partial)).toMatchObject({ status: "unknown", summary: { unchanged: 0, unknown: 1 }, diff --git a/tests/boundary/filesystem/binaryTarget.test.ts b/tests/boundary/filesystem/binaryTarget.test.ts index d3f363efa..e56055069 100644 --- a/tests/boundary/filesystem/binaryTarget.test.ts +++ b/tests/boundary/filesystem/binaryTarget.test.ts @@ -20,6 +20,7 @@ import { } from "../../../src/application/AppBundleExecutable.js"; import { dosMz, + elf, fat, pe, thinMach, @@ -846,3 +847,40 @@ describe("DOS binary target I/O", () => { }); }); }); + +it.each([ + [ + "EI_VERSION", + (bytes: Buffer) => { + bytes[6] = 0; + }, + ], + [ + "e_version", + (bytes: Buffer) => { + bytes.writeUInt32LE(0, 20); + }, + ], + [ + "e_ehsize", + (bytes: Buffer) => { + bytes.writeUInt16LE(51, 40); + }, + ], +])( + "rejects an ELF with malformed %s during public target opening", + async (_name, corrupt) => { + const directory = await createTestTempDirectory("rea-invalid-elf-header-"); + const path = join(directory, "invalid.elf"); + const bytes = elf(1, 1, 3); + corrupt(bytes); + await writeFile(path, bytes); + expect(await parseBinaryTarget(path)).toMatchObject({ + ok: false, + error: { + _tag: "BinaryTargetError", + constraint: "file_format_rejected", + }, + }); + }, +); diff --git a/tests/boundary/filesystem/harmony/application.test.ts b/tests/boundary/filesystem/harmony/application.test.ts index 5f758c003..b2870e6c5 100644 --- a/tests/boundary/filesystem/harmony/application.test.ts +++ b/tests/boundary/filesystem/harmony/application.test.ts @@ -214,6 +214,34 @@ describe("HarmonyOS packaging boundaries", () => { }); describe("HarmonyOS inventory coverage", () => { + it("keeps a case-colliding inventory partial in the application projection", async () => { + const path = await writeApplicationZip( + "rea-harmony-case-collision-", + "Fixture.hap", + [ + { path: "module.json", content: '{"module":{}}' }, + { path: "MODULE.JSON", content: '{"module":{}}' }, + ], + ); + const inventory = await inventoryApplicationPackage(path); + expect(inventory.normalized_result).toMatchObject({ + coverage_status: "partial", + }); + + const projection = requireDeterministicProjection( + projectHarmonyApplicationEvidence, + inventory, + harmonyApplicationProjectionResultSchema, + ); + expect(projection.coverage).toEqual({ + status: "partial", + inventory_complete: true, + }); + expect(projection.limitations).toContain( + "Source inventory coverage is partial; absence is unknown.", + ); + }); + it.each(ZIP_PACKAGE_FORMATS)( "retains archive identity and ZIP coverage for %s", async (format) => { diff --git a/tests/boundary/go/goBinaryImage.test.ts b/tests/boundary/go/goBinaryImage.test.ts index 0dc486a96..24321ecc8 100644 --- a/tests/boundary/go/goBinaryImage.test.ts +++ b/tests/boundary/go/goBinaryImage.test.ts @@ -227,11 +227,6 @@ it("rejects ELF header tables overlapping the native header", () => { expect(() => readGoBinaryImage(fixture.bytes)).toThrowError( /header|overlap/i, ); - const declaredSize = createGoBinaryFixture(); - declaredSize.bytes.writeUInt16LE(8192, 52); - expect(() => readGoBinaryImage(declaredSize.bytes)).toThrowError( - /truncated|range/i, - ); }); it("rejects contradictory aliases when a metadata address maps to different file bytes", () => { diff --git a/tests/boundary/mcp/serverUnknownsMcp.test.ts b/tests/boundary/mcp/serverUnknownsMcp.test.ts index 41fc76c45..0a27324ec 100644 --- a/tests/boundary/mcp/serverUnknownsMcp.test.ts +++ b/tests/boundary/mcp/serverUnknownsMcp.test.ts @@ -4,6 +4,7 @@ import type { CallToolResult } from "@modelcontextprotocol/server"; import { afterEach, expect, it } from "vitest"; import { createTestBinarySession } from "../../fixtures/binarySession.js"; +import { toolContract } from "../../../src/contracts/toolContracts.js"; import type { AnalysisClient, AnalysisProvider, @@ -249,3 +250,158 @@ it("records capture disagreement as a contradicted unknown", async () => { ), ).toEqual(beforeRepeat); }); + +it("pages filtered unknown heads through the public MCP contract", async () => { + const session = createTestBinarySession(providerWithCapabilities([])); + const server = createServer({ kind: "session", session }); + const client = new Client({ name: "unknown-pagination", version: "1" }); + const [clientTransport, serverTransport] = + InMemoryTransport.createLinkedPair(); + resources.push(client, server); + await server.connect(serverTransport); + await client.connect(clientTransport); + + const created = [ + ["matching c", "high", "pagination"], + ["matching a", "high", "pagination"], + ["matching b", "high", "pagination"], + ["filtered severity", "medium", "pagination"], + ["filtered domain", "high", "other"], + ] as const; + for (const [question, severity, domain] of created) { + const result = session.recordUnknown({ + question, + severity, + domain, + required_authority: "analyst-inference", + required_confidence: "observed", + required_environment: null, + recommended_probes: [], + relationships: [], + supporting_evidence_ids: [], + contradicting_evidence_ids: [], + }); + expect(result.ok).toBe(true); + } + + const defaultPage = structured( + await client.callTool({ name: "list_unknowns", arguments: {} }), + ).result as { + items: { unknown_id: string }[]; + total: number; + offset: number; + next_offset: number | null; + }; + expect(defaultPage.items.map(({ unknown_id }) => unknown_id)).toEqual( + defaultPage.items.map(({ unknown_id }) => unknown_id).sort(), + ); + expect(defaultPage).toMatchObject({ total: 5, offset: 0, next_offset: null }); + + const detached = session.listUnknownsPage({ limit: 1 }); + const detachedHead = detached.items[0]; + if (detachedHead === undefined) throw new Error("Expected one unknown head"); + const originalQuestion = detachedHead.question; + Reflect.set(detachedHead, "question", "changed outside the ledger"); + expect( + session + .listUnknowns() + .find(({ unknown_id }) => unknown_id === detachedHead.unknown_id) + ?.question, + ).toBe(originalQuestion); + + const offsetWithoutLimit = structured( + await client.callTool({ + name: "list_unknowns", + arguments: { domain: "pagination", offset: 1 }, + }), + ).result as typeof defaultPage; + expect(offsetWithoutLimit).toMatchObject({ + total: 4, + offset: 1, + next_offset: null, + }); + expect(offsetWithoutLimit.items).toHaveLength(3); + + const noMatches = structured( + await client.callTool({ + name: "list_unknowns", + arguments: { domain: "no-such-domain", limit: 2 }, + }), + ).result as typeof defaultPage; + expect(noMatches).toMatchObject({ + items: [], + total: 0, + offset: 0, + next_offset: null, + }); + + const beyondEnd = structured( + await client.callTool({ + name: "list_unknowns", + arguments: { domain: "pagination", offset: 100, limit: 2 }, + }), + ).result as typeof defaultPage; + expect(beyondEnd).toMatchObject({ + total: 4, + offset: 100, + next_offset: null, + }); + expect(beyondEnd.items).toEqual([]); + + const maximumSafePage = structured( + await client.callTool({ + name: "list_unknowns", + arguments: { + offset: Number.MAX_SAFE_INTEGER, + limit: Number.MAX_SAFE_INTEGER, + }, + }), + ).result as typeof defaultPage; + expect(maximumSafePage).toMatchObject({ + total: 5, + offset: Number.MAX_SAFE_INTEGER, + next_offset: null, + }); + expect(maximumSafePage.items).toEqual([]); + + const request = { + name: "list_unknowns", + arguments: { domain: "pagination", severity: "high", limit: 2 }, + } as const; + const first = structured(await client.callTool(request)) + .result as typeof defaultPage; + expect(first).toMatchObject({ total: 3, offset: 0, next_offset: 2 }); + expect(first.items).toHaveLength(2); + const second = structured( + await client.callTool({ + ...request, + arguments: { ...request.arguments, offset: first.next_offset ?? 0 }, + }), + ).result as typeof defaultPage; + expect(second).toMatchObject({ total: 3, offset: 2, next_offset: null }); + expect(second.items).toHaveLength(1); + expect( + [...first.items, ...second.items].map(({ unknown_id }) => unknown_id), + ).toEqual( + session + .listUnknowns({ domain: "pagination", severity: "high" }) + .map(({ unknown_id }) => unknown_id), + ); + expect( + toolContract("list_unknowns").outputSchema.safeParse( + (await client.callTool(request)).structuredContent, + ).success, + ).toBe(true); + + for (const arguments_ of [ + { limit: 0 }, + { offset: -1 }, + { offset: Number.MAX_SAFE_INTEGER + 1 }, + { unexpected: true }, + ]) { + expect( + (await client.callTool({ name: "list_unknowns", arguments: arguments_ })) + .isError, + ).toBe(true); + } +}); diff --git a/tests/boundary/mcp/workflowSnapshotReplayMcp.test.ts b/tests/boundary/mcp/workflowSnapshotReplayMcp.test.ts index d5ecd00d2..b921fc224 100644 --- a/tests/boundary/mcp/workflowSnapshotReplayMcp.test.ts +++ b/tests/boundary/mcp/workflowSnapshotReplayMcp.test.ts @@ -412,7 +412,12 @@ describe("MCP investigation history snapshot replay", () => { toolContract("list_unknowns").outputSchema.parse( listed.structuredContent, ).result, - ).toEqual({ items: [resolved], total: 1 }); + ).toEqual({ + items: [resolved], + total: 1, + offset: 0, + next_offset: null, + }); }, ); }); diff --git a/tests/boundary/providers/native/nativeSignatureTargetBinding.test.ts b/tests/boundary/providers/native/nativeSignatureTargetBinding.test.ts index 25b9e0730..b622e2d97 100644 --- a/tests/boundary/providers/native/nativeSignatureTargetBinding.test.ts +++ b/tests/boundary/providers/native/nativeSignatureTargetBinding.test.ts @@ -15,6 +15,7 @@ import { import { dirname, join } from "node:path"; import { describe, expect, it } from "vitest"; import { NativeMacOSProvider } from "../../../../src/native/NativeMacOSProvider.js"; +import { ArtifactResourceScope } from "../../../../src/artifacts/ArtifactResourceScope.js"; import { withSignatureTarget, verifySignatureTarget, @@ -277,6 +278,7 @@ it("returns typed cancellation at binding entry and final verification", async ( await expect( withSignatureTarget( target, + new ArtifactResourceScope(), async () => { throw new Error("Cancelled acquisition must not invoke inspection"); }, @@ -286,14 +288,18 @@ it("returns typed cancellation at binding entry and final verification", async ( ok: false, error: { _tag: "AnalysisCancelledError" }, }); - await withSignatureTarget(target, async (binding) => { - const result = await verifySignatureTarget(target, binding, signal); - expect(result).toMatchObject({ - ok: false, - error: { _tag: "AnalysisCancelledError" }, - }); - return result; - }); + await withSignatureTarget( + target, + new ArtifactResourceScope(), + async (binding) => { + const result = await verifySignatureTarget(target, binding, signal); + expect(result).toMatchObject({ + ok: false, + error: { _tag: "AnalysisCancelledError" }, + }); + return result; + }, + ); }); it("preserves cancellation after a capture instead of classifying it as invalid output", async () => { @@ -348,21 +354,31 @@ it("copies and hashes the same bytes across multiple snapshot chunks", async () const bytes = Buffer.alloc(128 * 1024 + 13, 42); await writeFile(path, bytes); const target = await nativeMachoTargetForFile(path); - await withSignatureTarget(target, async (binding) => { - expect(await readFile(binding.snapshotPath)).toEqual(bytes); - return verifySignatureTarget(target, binding); - }); + await withSignatureTarget( + target, + new ArtifactResourceScope(), + async (binding) => { + expect(await readFile(binding.snapshotPath)).toEqual(bytes); + return verifySignatureTarget(target, binding); + }, + ); }); it("cleans the private snapshot when inspection throws", async () => { const { target } = await fixture(); let snapshotPath: string | undefined; - await expect( - withSignatureTarget(target, async (binding) => { + const result = await withSignatureTarget( + target, + new ArtifactResourceScope(), + async (binding) => { snapshotPath = binding.snapshotPath; throw new Error("Invalid parsed output"); - }), - ).rejects.toThrow("Invalid parsed output"); + }, + ); + expect(result).toMatchObject({ + ok: false, + error: { _tag: "AnalysisOutputError" }, + }); if (snapshotPath === undefined) throw new Error("Expected snapshot"); await expect(lstat(dirname(snapshotPath))).rejects.toMatchObject({ code: "ENOENT",