From 26525706fcd102355d1cd765138398067f32527f Mon Sep 17 00:00:00 2001 From: Nelglor Date: Tue, 13 Oct 2026 15:01:23 -0400 Subject: [PATCH] feat(managed): decode ldstr user strings in member inspection (#2162) * feat(managed): decode ldstr user strings in member inspection inspect_managed_members reported every ldstr operand as a bare 0x70 token, so an analyst could see that a method loads a literal but not which one. Answering "where is this path or key used" required a second tool. Read the optional #US stream and add a top-level user_strings inventory: each distinct token loaded by an inspected method body, once, in token order, with its #US file offset and decoded UTF-16 value. Instruction anchors and normalized CIL digests keep their build-local tokens. A missing #US stream or malformed item is reported against its token as partial coverage. Evidence recorded before this field reads user_strings as null. Co-Authored-By: Claude Opus 5.5 * fix(managed): collect user strings only from ldstr A malformed ldtoken can carry a 0x70 token, which the operand decoder classifies as a string operand. Only ldstr loads a literal, so collect user strings by opcode. Co-Authored-By: Claude Opus 5.5 * fix(managed): admit user-string tokens and failures before collecting them Charge each distinct ldstr token to the decode budget before adding it to the collection set, and admit each #US failure before retaining it. A missing #US stream is now one stream-level issue rather than one per token. Document why the #US terminal flag is not validated: Mono and Roslyn assemblies disagree with ECMA-335 II.24.2.4 and the runtime ignores it. Co-Authored-By: Claude Opus 5.5 * docs(managed): describe missing #US reporting and the unvalidated flag byte Co-Authored-By: Claude Opus 5.5 * test(managed): cover #US items that run past the stream end An odd length whose payload crosses the end of #US, and a two-byte length prefix that starts on the last #US byte, are both rejected as "User string leaves #US" beside a readable literal. The test fails if that bounds check is removed. Co-Authored-By: Claude Opus 5.5 * fix(managed): report reserved #US length prefixes as string failures The compressed-length reader shared by #Blob and #US hard-coded invalid-blob and "Metadata blob" for a reserved prefix, so a malformed user string was published as a blob failure. The caller now names the item it reads: #Blob keeps its code and message, and #US reports invalid-string. Co-Authored-By: Claude Opus 5.5 * fix(managed): append member inspection issues without argument spreading inspectManagedMembersBytes gathered every producer's issues with issues.push(...list), passing one argument per issue. A large malformed input, such as hundreds of thousands of distinct bad ldstr operands or MemberRef rows, overflows the stack with an untyped RangeError instead of returning partial coverage. All six sites now use appendManagedParseIssues, which appends one issue at a time. Co-Authored-By: Claude Opus 5.5 * fix(managed): classify truncated #US and #Blob length prefixes by item A two- or four-byte compressed-length prefix cut off by the end of the artifact went through the generic range check and was reported as invalid-row. The prefix reader now uses the item's own code for both reserved and truncated prefixes: invalid-string for #US, invalid-blob for #Blob. The unused generic range helper is removed. Co-Authored-By: Claude Opus 5.5 * fix(managed): retain user-string terminal flag conformance issues * test(managed): write ECMA-335 #US terminal flags and cover flag issues d5b7b0704 reports a #US terminal flag that differs from ECMA-335 II.24.2.4. The fixture heap always wrote 0, so its non-Latin-1 literals now read as nonconforming. The fixture writes the spec flag, and a new test flips one flag each way ("ok" set to 1, "it's" set to 0) and checks that both literals remain beside one issue each. Co-Authored-By: Claude Opus 5.5 * fix(managed): classify ldstr operands before collapsing a missing #US With no #US stream, every collected ldstr operand was counted as a #US reference and reported only as missing-stream, while the same non-0x70 operand got invalid-heap-index when the stream existed. parseUserStrings now owns which operands are user-string tokens: any other operand is invalid-heap-index either way, and only 0x70 tokens count toward the missing-stream issue. The heap reader drops its duplicate kind check. Co-Authored-By: Claude Opus 5.5 * fix(managed): state user-string limits in member comparisons compare_managed_members ignored user_strings, so a method whose ldstr literal changed at the same token, or legacy Evidence that never examined literals, could read as unchanged with no caveat. The comparison now adds an unknown limitation when either side predates user strings and a not-compared limitation when literals were observed. ExportedType and user-string inventories share one uncomparedInventory rule. Co-Authored-By: Claude Opus 5.5 * docs(managed): advertise ldstr literals in member inspection help The MCP tool description, the type_names input, and the CLI command and --type help now say that inspect_managed_members returns the #US string literals that ldstr loads, and that a type selection limits them to the literals the selected types' method bodies load. Co-Authored-By: Claude Opus 5.5 * fix(managed): read the empty #US entry at offset 0 ECMA-335 II.24.2.4 makes the first #US entry the empty blob: a zero length with no terminal flag byte. ldstr 0x70000000 loads it, but the odd-length check rejected it as invalid-string, dropping the literal and making coverage partial. Offset 0 with length 0 now reads as "". Co-Authored-By: Claude Opus 5.5 * fix(managed): reserve #US offset 0 and admit literals before scanning Offset 0 is the reserved empty #US entry, so a nonempty item there is now invalid-string instead of a published literal. The literal's projection is reserved before the terminal-flag scan, so an inadmissible item fails at the budget without reading every code unit. The scan's loop index no longer shadows the token's heap index. Co-Authored-By: Claude Opus 5.5 * fix(managed): report #US length prefixes wider than ECMA-335 assigns compressedLength accepted an overlong encoding, such as 0x80 0x05 for a 5-byte item, without comment. It now reports whether the width is the one ECMA-335 II.23.2 assigns, and the #US reader reports a non-minimal prefix as an invalid-string conformance issue beside the readable literal, like the terminal flag. readMetadataUserString returns every issue it observes. #Blob behavior is unchanged. Co-Authored-By: Claude Opus 5.5 * fix(managed): check the #US length width before the empty entry The offset-0 empty entry returned "" before the length-width check, so an overlong empty entry such as 0x80 0x00 read as complete. The width issue is now collected first and returned with the empty literal. Co-Authored-By: Claude Opus 5.5 * fix(managed): admit #US width issues only for returned literals The overlong-prefix issue was reserved before the fatal shape checks, so an item that then failed, such as 0x80 0x00 away from offset 0, leaked 512 bytes of budget per token. The width issue is now built and admitted only on paths that return the literal. Co-Authored-By: Claude Opus 5.5 * fix(managed): keep #US issues out of member metadata status User-string issues, including terminal-flag and length-prefix conformance differences, marked metadata.status partial, so compare_managed_members turned one-sided methods and fields into unknown even though every member row was examined. #US issues now make coverage partial while metadata.status describes the member inventory. Co-Authored-By: Claude Opus 5.5 * fix(managed): give user strings their own coverage #US issues entered the member coverage, so every consumer that reads it as member coverage (application graph member facts, comparison status) reported examined members as partial over a literal conformance difference. user_strings is now { state, literals, issues }: #US issues live there and never change the member coverage or metadata.status. parseUserStrings returns that inventory directly. compare_managed_members inspects with user strings skipped, since it never reads literals, so a large literal cannot exhaust the decode budget of a comparison. Its limitations state once that literals are not compared, and the ExportedType limitations return to their main form. Co-Authored-By: Claude Opus 5.5 * fix(managed): mark user strings partial when a body is not fully decoded With user strings owning their coverage, an inspected method body that was partial, malformed, or truncated no longer reached their state, so a literal after the decode stopped was missing from a "complete" inventory. parseUserStrings now marks the inventory partial unless every inspected body was fully decoded, keeping the literals it did observe. Co-Authored-By: Claude Opus 5.5 * fix(managed): mark user strings partial when a method row is omitted A MethodDef row that failed before its body was read was recorded as an issue and omitted, so its ldstr instructions were never scanned, yet the user-string inventory could still read as complete. parseUserStrings now takes the parseMethods result and stays partial whenever a method row failed, as well as when an inspected body was not fully decoded. Co-Authored-By: Claude Opus 5.5 * fix(managed): count only method rows that may hide a selected body Any parseMethods issue marked user strings partial, including an unreadable name on a method outside the type selection, so a fully scanned selection read as incomplete. parseMethods now reads each row's owner first and reports unscannedRows: rows that produced no method while their owner is selected or unknown. parseUserStrings uses that count. Co-Authored-By: Claude Opus 5.5 * fix(managed): treat unreadably named types as possibly selected Under a type_names selection, a type whose name could not be read may be a requested type, but its methods were skipped without counting toward unscannedRows, so an unexamined body left user strings complete. A method whose owner's name is unreadable now counts as possibly selected. Co-Authored-By: Claude Opus 5.5 * fix(managed): judge user-string scans by CIL decoding, not body status body.status is malformed for an invalid exception section even when every instruction decoded, so user strings read as partial although every ldstr was examined. The body reader now reports whether its CIL was fully scanned, independent of exception-region validity, and parseMethods folds unscanned bodies and unproduced rows into one unscannedMethods count that parseUserStrings uses instead of reinterpreting body.status. Co-Authored-By: Claude Opus 5.5 * fix(managed): stop deduplicating user-string issues The inventory passed its issues through uniqueManagedParseIssues, which serializes every issue into an unbudgeted map: about 42 MB of heap and 113 ms for 125,000 issues on Node 24. The issues are unique by construction (one failure or per-offset conformance issues per distinct token, one missing-stream issue), so the inventory now returns them, and its literals, without copies. Co-Authored-By: Claude Opus 5.5 * fix(managed): preserve selected string coverage unknowns --------- Co-authored-by: Claude Opus 5.5 Co-authored-by: morluto <76467478+morluto@users.noreply.github.com> --- docs/managed-code-analysis.md | 30 +- .../managed/ManagedMemberComparisonService.ts | 6 +- src/cli/managedCommands.ts | 4 +- src/contracts/managed/managedToolContracts.ts | 4 +- .../managed/managedWorkflowExamples.ts | 1 + src/domain/managed/managedArtifact.ts | 21 + .../managedMemberComparisonCoverage.ts | 3 + src/dotnet/ManagedMemberInspector.test.ts | 457 +++++++++++++++++- src/dotnet/ManagedMemberInspector.ts | 37 +- src/dotnet/ManagedMemberInspectorCore.ts | 3 + src/dotnet/ManagedMemberRows.ts | 44 +- src/dotnet/ManagedMetadataHeaps.test.ts | 28 +- src/dotnet/ManagedMetadataHeaps.ts | 192 ++++++-- src/dotnet/ManagedMetadataLayout.ts | 3 + src/dotnet/ManagedMethodBodyReader.ts | 46 +- src/dotnet/ManagedPe.fixture.ts | 140 ++++-- src/dotnet/ManagedReaderFailure.ts | 11 + src/dotnet/ManagedUserStrings.ts | 120 +++++ ...anagedMemberComparisonFromMetadata.test.ts | 24 + 19 files changed, 1054 insertions(+), 120 deletions(-) create mode 100644 src/dotnet/ManagedUserStrings.ts diff --git a/docs/managed-code-analysis.md b/docs/managed-code-analysis.md index 1d1a579e6..167d947c8 100644 --- a/docs/managed-code-analysis.md +++ b/docs/managed-code-analysis.md @@ -139,14 +139,40 @@ The result preserves the requested names in `selection.type_names` and reports unmatched names separately. If metadata names are unreadable, limitations retain that uncertainty. With no selection, `selection` is null and whole-assembly inspection keeps its existing resource policy. Selection restricts types, -fields, methods, call edges and field accesses; member references remain an -assembly-wide inventory, so output size is not guaranteed to be small. +fields, methods, call edges, field accesses and user strings; member references +remain an assembly-wide inventory, so output size is not guaranteed to be small. Coverage describes the selected members. Comparisons leave members outside the opposite observation's selection unknown, and application graphs report their selected scope as partial. Neither makes an absence claim about the whole assembly. Older member Evidence without `selection` remains readable. +### User strings + +`ldstr` anchors keep their build-local `0x70` tokens. `user_strings.literals` +lists each distinct token loaded by an inspected method body once, in token +order, with its `#US` file offset and decoded value. Values keep their stored +UTF-16 code units, including unpaired surrogates. Token `0x70000000` names the +heap's empty entry at offset 0 and reads as `""`; any other item there is +malformed. The terminal flag is checked against ECMA-335 II.24.2.4, and the +length prefix against the width II.23.2 assigns. A difference is reported with +its byte offset as a user-string issue while preserving the readable literal. +Toolchain conventions can differ; this conformance observation does not +establish runtime rejection. A missing `#US` stream is one stream-level issue; +a malformed item is reported against its token. User strings carry their own +coverage: any of these makes `user_strings.state` partial and lists the issue +in `user_strings.issues`, while the member `coverage` and `metadata.status` are +unaffected, and readable literals remain. The state is also partial when an +inspected method's CIL was not fully decoded (a malformed exception section +alone does not count) or a method that may belong to the inspected types was +not inspected (its row could not be read, or its type's name could not be read +under a `type_names` selection), because `ldstr` instructions there were not +observed. Normalized CIL digests do not include decoded values. +`compare_managed_members` does not decode or compare literals: an unchanged +method does not establish unchanged strings, and the comparison says so in its +limitations. Older member Evidence without `user_strings` reads as `null`: not +examined. + ### Edge target declaring types `member_refs[].parent_name`, `call_edges[].target_declaring_type`, and diff --git a/src/application/managed/ManagedMemberComparisonService.ts b/src/application/managed/ManagedMemberComparisonService.ts index 252feb8ca..04fae187f 100644 --- a/src/application/managed/ManagedMemberComparisonService.ts +++ b/src/application/managed/ManagedMemberComparisonService.ts @@ -92,7 +92,11 @@ export const compareManagedMemberPaths = async ( throw new EvidenceIntegrityError( `Managed artifact digest changed after open: expected ${target.sha256}, observed ${snapshot.sha256} at ${target.path}`, ); - const inspection = inspectManagedMembersBytes(snapshot.bytes, target); + // Member comparison never reads literals, so it does not spend the + // decode budget on them. + const inspection = inspectManagedMembersBytes(snapshot.bytes, target, { + userStrings: false, + }); const evidence = createEvidence(target, MANAGED_STATIC_PROVIDER, { operation: "inspect_managed_members", parameters: {}, diff --git a/src/cli/managedCommands.ts b/src/cli/managedCommands.ts index eee4eae5a..782549720 100644 --- a/src/cli/managedCommands.ts +++ b/src/cli/managedCommands.ts @@ -56,13 +56,13 @@ const registerInspectManagedMembers = ( ): void => { cli.command(CLI_COMMANDS.inspectManagedMembers, { description: - "Inspect PE/CLI metadata members, signatures, and CIL anchors without loading target code", + "Inspect PE/CLI metadata members, signatures, CIL anchors, and ldstr string literals without loading target code", args: z.object({ path: z.string().describe("Managed PE executable or assembly path"), }), options: z.object({ type: managedMembersInputSchema.shape.type_names.describe( - "Inspect only members declared by this exact type full name, such as System.Threading.Monitor; a nested type is named by its bare name, which selects every nested type of that name (repeatable)", + "Inspect only members declared by this exact type full name, and the string literals they load, such as System.Threading.Monitor; a nested type is named by its bare name, which selects every nested type of that name (repeatable)", ), }), run: ({ args, options }) => diff --git a/src/contracts/managed/managedToolContracts.ts b/src/contracts/managed/managedToolContracts.ts index 1e685a702..2c0088b9b 100644 --- a/src/contracts/managed/managedToolContracts.ts +++ b/src/contracts/managed/managedToolContracts.ts @@ -43,7 +43,7 @@ export const managedMembersInputSchema = managedTargetInputSchema.extend({ }) .meta({ uniqueItems: true }) .describe( - "Inspect only the fields, methods, call edges, and field accesses declared by these exact type full names; omit to inspect every type. Unmatched names are reported in selection.unmatched_type_names.", + "Inspect only the fields, methods, call edges, and field accesses declared by these exact type full names, and the user strings their method bodies load; omit to inspect every type. Unmatched names are reported in selection.unmatched_type_names.", ) .optional(), }); @@ -87,7 +87,7 @@ export const MANAGED_TOOL_CONTRACTS = [ name: "inspect_managed_members", ...toolContractMetadata("inspect_managed_members"), description: - "Inspect PE/CLI metadata members, signatures, raw CIL hashes, decoded-instruction-tuple hashes, separately reported exception regions, call edges, and field-access anchors without loading or executing target code. Select type_names to decode only the members of those exact declaring types, which keeps large assemblies such as a base class library within the representation budget. Metadata tokens are reported as build-local coordinates bound to the artifact SHA-256 and MVID; the tuple hash does not resolve them or fully commit control flow.", + "Inspect PE/CLI metadata members, signatures, raw CIL hashes, decoded-instruction-tuple hashes, separately reported exception regions, call edges, field-access anchors, and the #US string literals that ldstr loads, without loading or executing target code. Select type_names to decode only the members of those exact declaring types and the literals their bodies load, which keeps large assemblies such as a base class library within the representation budget. Metadata tokens are reported as build-local coordinates bound to the artifact SHA-256 and MVID; the tuple hash does not resolve them or fully commit control flow.", kind: "managed-provider", inputSchema: managedMembersInputSchema, outputSchema: memberOutputSchema, diff --git a/src/contracts/managed/managedWorkflowExamples.ts b/src/contracts/managed/managedWorkflowExamples.ts index 85297d671..6e5d836ec 100644 --- a/src/contracts/managed/managedWorkflowExamples.ts +++ b/src/contracts/managed/managedWorkflowExamples.ts @@ -53,6 +53,7 @@ const emptyManagedMembers = ( member_refs: [], call_edges: [], field_accesses: [], + user_strings: { state: "complete", literals: [], issues: [] }, selection: null, coverage: { state: "complete", issues: [] }, limitations: [], diff --git a/src/domain/managed/managedArtifact.ts b/src/domain/managed/managedArtifact.ts index 94eff2398..f41173ba6 100644 --- a/src/domain/managed/managedArtifact.ts +++ b/src/domain/managed/managedArtifact.ts @@ -219,6 +219,16 @@ const managedMemberReferenceSchema = z.object({ signature: managedSignatureSchema, }); +const managedUserStringSchema = z.object({ + token: tokenSchema, + offset: offsetSchema.describe("File offset of the #US item"), + value: z + .string() + .describe( + "Decoded UTF-16 code units, excluding the terminal flag byte; unpaired surrogates are kept", + ), +}); + const managedModuleReferenceSchema = z.object({ token: tokenSchema, row_offset: offsetSchema, @@ -457,6 +467,17 @@ export const managedMemberInspectionSchema = z.object({ member_refs: z.array(managedMemberReferenceSchema), call_edges: z.array(managedCallEdgeSchema), field_accesses: z.array(managedFieldAccessSchema), + // Distinct #US literals loaded by ldstr in the inspected method bodies, in + // token order, with their own coverage: #US issues never enter the member + // coverage. null means they were not examined, not that there were none. + user_strings: z + .object({ + state: z.enum(["complete", "partial"]), + literals: z.array(managedUserStringSchema), + issues: z.array(managedParseIssueSchema), + }) + .nullable() + .default(null), /** Caller-selected declaring types; null when every type was inspected. */ selection: z .object({ diff --git a/src/domain/managed/managedMemberComparisonCoverage.ts b/src/domain/managed/managedMemberComparisonCoverage.ts index 7f28cc88e..85da3b4eb 100644 --- a/src/domain/managed/managedMemberComparisonCoverage.ts +++ b/src/domain/managed/managedMemberComparisonCoverage.ts @@ -131,5 +131,8 @@ export const comparisonLimitations = ( limitations.push( "ExportedType rows, including type forwarders, are observed but not compared; an unchanged comparison does not establish an unchanged exported or forwarded type set.", ); + limitations.push( + "ldstr literals are not compared; normalized CIL keeps #US tokens rather than decoded text, so an unchanged method does not establish unchanged string literals.", + ); return limitations; }; diff --git a/src/dotnet/ManagedMemberInspector.test.ts b/src/dotnet/ManagedMemberInspector.test.ts index 9ada1bf55..3aee06e73 100644 --- a/src/dotnet/ManagedMemberInspector.test.ts +++ b/src/dotnet/ManagedMemberInspector.test.ts @@ -1,8 +1,18 @@ import { describe, expect, it } from "vitest"; import { inspectManagedMembersBytes } from "./ManagedMemberInspector.js"; -import { ManagedDecodeBudget } from "./ManagedDecodeBudget.js"; +import { + ManagedDecodeBudget, + managedDecodeBudget, +} from "./ManagedDecodeBudget.js"; +import { + metadataRowOffset, + readManagedMetadataLayout, + type ManagedMetadataLayout, +} from "./ManagedMetadataLayout.js"; +import { readManagedPeLayout } from "./ManagedPeReader.js"; import { edges } from "./ManagedMemberRows.js"; +import { parseUserStrings } from "./ManagedUserStrings.js"; import { signature, type ManagedMethod } from "./ManagedMemberInspectorCore.js"; import { ManagedReaderFailure } from "./ManagedReaderFailure.js"; import { managedMemberInspectionSchema } from "../domain/managed/managedArtifact.js"; @@ -11,9 +21,23 @@ import { comparisonLimitations } from "../domain/managed/managedMemberComparison import { buildManagedPeFixture, buildNativePeFixture, + fixtureUserStringHeap, managedPeFixtureTarget, } from "./ManagedPe.fixture.js"; +/** A tiny-header body that loads each `ldstr` token, discards it, and returns. */ +const ldstrBody = (tokens: readonly string[]): Buffer => { + const il = Buffer.concat([ + ...tokens.map((token) => { + const instruction = Buffer.from([0x72, 0, 0, 0, 0, 0x26]); + instruction.writeUInt32LE(Number.parseInt(token, 16), 1); + return instruction; + }), + Buffer.from([0x2a]), + ]); + return Buffer.concat([Buffer.from([(il.length << 2) | 2]), il]); +}; + describe("managed member inspection", () => { it("keeps metadata handles distinct from calls and field accesses", () => { const il = Buffer.from([ @@ -605,6 +629,433 @@ describe("managed exported types", () => { }); }); +describe("managed user strings", () => { + const inspect = (options: Parameters[0]) => { + const bytes = buildManagedPeFixture(options); + return inspectManagedMembersBytes(bytes, managedPeFixtureTarget(bytes)); + }; + + it("decodes each loaded literal once, in token order, without changing CIL tokens", () => { + const { heap, tokens } = fixtureUserStringHeap([ + "quicksave.peak", + "unused", + "Ünïcødé 𝄞", + "\ud800", + ]); + const [path, , unicode, lone] = tokens as [string, string, string, string]; + // 0x70000000 names the heap's empty entry at offset 0. + const empty = "0x70000000"; + const ilBody = ldstrBody([unicode, path, lone, path, empty]); + const result = inspect({ ilBody, userStringHeap: heap }); + + expect(result.user_strings?.literals).toEqual([ + { token: empty, offset: expect.any(Number), value: "" }, + { token: path, offset: expect.any(Number), value: "quicksave.peak" }, + { token: unicode, offset: expect.any(Number), value: "Ünïcødé 𝄞" }, + { token: lone, offset: expect.any(Number), value: "\ud800" }, + ]); + expect(result.coverage).toEqual({ state: "complete", issues: [] }); + expect(result.user_strings).toMatchObject({ + state: "complete", + issues: [], + }); + const body = result.methods[0]?.body; + expect(body?.anchors.map(({ operand }) => operand)).toEqual([ + unicode, + path, + lone, + path, + empty, + ]); + expect(body?.normalized_il_sha256).toBe( + inspect({ ilBody }).methods[0]?.body.normalized_il_sha256, + ); + expect(comparisonLimitations(result, result)).toContain( + "ldstr literals are not compared; normalized CIL keeps #US tokens rather than decoded text, so an unchanged method does not establish unchanged string literals.", + ); + }); + + it("ignores a 0x70 token carried by an instruction other than ldstr", () => { + const { heap, tokens } = fixtureUserStringHeap(["not loaded"]); + const token = Number.parseInt(tokens[0] ?? "", 16); + const ldtoken = Buffer.from([0xd0, 0, 0, 0, 0, 0x26, 0x2a]); + ldtoken.writeUInt32LE(token, 1); + const result = inspect({ + ilBody: Buffer.concat([ + Buffer.from([(ldtoken.length << 2) | 2]), + ldtoken, + ]), + userStringHeap: heap, + }); + + expect(result.methods[0]?.body.anchors[0]).toMatchObject({ + opcode: "ldtoken", + operand_kind: "string", + }); + expect(result.user_strings?.literals).toEqual([]); + }); + + it("admits ldstr tokens before collecting them", () => { + const layout = { userStrings: null } as unknown as ManagedMetadataLayout; + // Leave 1 KiB of the 64 MiB budget: too little for 100 distinct tokens. + managedDecodeBudget(layout).reserve(64 * 1024 * 1024 - 1024, null); + const method = { + body: { + anchors: Array.from({ length: 100 }, (_, index) => ({ + il_offset: index * 5, + opcode: "ldstr", + operand_kind: "string", + operand: `0x70${(index + 1).toString(16).padStart(6, "0")}`, + })), + }, + } as unknown as ManagedMethod; + + expect(() => + parseUserStrings(Buffer.alloc(0), layout, { + methods: [method], + unscannedMethods: 0, + }), + ).toThrow(ManagedReaderFailure); + }); + + it("retains readable literals beside malformed #US items", () => { + const { heap, tokens } = fixtureUserStringHeap(["ok", "bad", "worse"]); + const [ok, bad, worse] = tokens as [string, string, string]; + // "bad" encodes 7 bytes; an even length cannot end in the flag byte. + heap.writeUInt8(6, Number.parseInt(bad.slice(4), 16)); + // 0xe0 is a reserved compressed-length prefix. + heap.writeUInt8(0xe0, Number.parseInt(worse.slice(4), 16)); + const result = inspect({ + ilBody: ldstrBody([ok, bad, worse, "0x70ffffff", "0x06000001"]), + userStringHeap: heap, + }); + + expect(result.user_strings?.literals).toEqual([ + { token: ok, offset: expect.any(Number), value: "ok" }, + ]); + expect(result.user_strings?.state).toBe("partial"); + expect( + result.user_strings?.issues.map(({ code, scope }) => [code, scope]), + ).toEqual([ + ["invalid-heap-index", "metadata.#US:0x06000001"], + ["invalid-string", `metadata.#US:${bad}`], + ["invalid-string", `metadata.#US:${worse}`], + ["invalid-heap-index", "metadata.#US:0x70ffffff"], + ]); + }); + + it("rejects #US items that run past the end of the stream", () => { + const { heap: readable, tokens } = fixtureUserStringHeap(["ok"]); + // An odd length claiming 127 bytes with two left, then a two-byte length + // prefix whose second byte lies past the stream end. + const crossing = Buffer.from([0x7f, 0x41, 0x00]); + const heap = Buffer.concat([readable, crossing, Buffer.from([0x80])]); + const token = (offset: number) => + `0x70${offset.toString(16).padStart(6, "0")}`; + const crossingToken = token(readable.length); + const truncatedToken = token(readable.length + crossing.length); + const result = inspect({ + ilBody: ldstrBody([tokens[0] ?? "", crossingToken, truncatedToken]), + userStringHeap: heap, + }); + + expect(result.user_strings?.literals).toEqual([ + { token: tokens[0], offset: expect.any(Number), value: "ok" }, + ]); + expect(result.user_strings).toMatchObject({ + state: "partial", + issues: [crossingToken, truncatedToken].map((item) => ({ + code: "invalid-string", + scope: `metadata.#US:${item}`, + detail: "User string leaves #US", + })), + }); + }); + + it("leaves user strings unexamined when CLI metadata is unavailable or predates them", () => { + const native = buildNativePeFixture(); + expect( + inspectManagedMembersBytes(native, managedPeFixtureTarget(native)) + .user_strings, + ).toBeNull(); + + const { user_strings: _omitted, ...legacy } = inspect({}); + const parsed = managedMemberInspectionSchema.parse(legacy); + expect(parsed.user_strings).toBeNull(); + expect(comparisonLimitations(parsed, inspect({}))).toContain( + "ldstr literals are not compared; normalized CIL keeps #US tokens rather than decoded text, so an unchanged method does not establish unchanged string literals.", + ); + }); +}); + +describe("managed user string issues", () => { + const inspect = (options: Parameters[0]) => { + const bytes = buildManagedPeFixture(options); + return inspectManagedMembersBytes(bytes, managedPeFixtureTarget(bytes)); + }; + + it("reports a missing #US stream once and keeps the method", () => { + const result = inspect({ + // A MethodDef operand is not a #US reference, so it is reported alone. + ilBody: ldstrBody([ + "0x70000001", + "0x70000005", + "0x70000001", + "0x06000001", + ]), + }); + + expect(result.methods).toHaveLength(1); + expect(result.coverage.state).toBe("complete"); + expect(result.user_strings).toEqual({ + state: "partial", + literals: [], + issues: [ + { + code: "invalid-heap-index", + scope: "metadata.#US:0x06000001", + offset: null, + detail: "ldstr operand does not name an item inside #US", + }, + { + code: "missing-stream", + scope: "metadata.#US", + offset: null, + detail: + "2 distinct ldstr operand(s) reference #US, but the metadata has no #US stream", + }, + ], + }); + }); + + it("rejects a nonempty item at the reserved #US offset 0", () => { + // "A" with its flag, where only the empty entry may sit. + const result = inspect({ + ilBody: ldstrBody(["0x70000000"]), + userStringHeap: Buffer.from([3, 0x41, 0, 0]), + }); + + expect(result.user_strings?.literals).toEqual([]); + expect(result.user_strings).toMatchObject({ + state: "partial", + issues: [{ code: "invalid-string", scope: "metadata.#US:0x70000000" }], + }); + }); + + it.each([ + // "ok" is 5 bytes, which ECMA-335 encodes in one prefix byte, not 0x80 0x05. + ["0x70000001", [0, 0x80, 0x05, 0x6f, 0, 0x6b, 0, 0], "ok"], + // The empty entry at offset 0 is the single byte 0x00, not 0x80 0x00. + ["0x70000000", [0x80, 0x00], ""], + ])( + "keeps %s when its length prefix is wider than ECMA-335 assigns", + (token, heap, value) => { + const result = inspect({ + ilBody: ldstrBody([token]), + userStringHeap: Buffer.from(heap), + }); + + expect(result.user_strings?.literals).toEqual([ + { token, offset: expect.any(Number), value }, + ]); + expect(result.user_strings).toMatchObject({ + state: "partial", + issues: [ + { + code: "invalid-string", + scope: `metadata.#US:${token}`, + detail: expect.stringContaining("prefix wider than ECMA-335"), + }, + ], + }); + }, + ); + + it("admits no width issue for an item that then fails", () => { + // 64 overlong zero lengths away from offset 0: each item is fatal. + const heap = Buffer.concat([ + Buffer.from([0]), + ...Array.from({ length: 64 }, () => Buffer.from([0x80, 0x00])), + ]); + const layout = { + userStrings: { name: "#US", offset: 0, size: heap.length }, + } as unknown as ManagedMetadataLayout; + // Leave 24 KiB: enough for the failures, not for 512 bytes leaked each. + managedDecodeBudget(layout).reserve(64 * 1024 * 1024 - 24 * 1024, null); + const method = { + body: { + anchors: Array.from({ length: 64 }, (_, index) => ({ + il_offset: index * 5, + opcode: "ldstr", + operand_kind: "string", + operand: `0x70${(1 + index * 2).toString(16).padStart(6, "0")}`, + })), + }, + } as unknown as ManagedMethod; + + expect( + parseUserStrings(heap, layout, { methods: [method], unscannedMethods: 0 }) + .issues, + ).toHaveLength(64); + }); + + it("keeps literals whose terminal flag differs from ECMA-335 and reports each flag", () => { + const { heap, tokens } = fixtureUserStringHeap(["ok", "it's"]); + const [ok, quoted] = tokens as [string, string]; + // A short item is a one-byte length, the UTF-16 text, then the flag byte. + const flagAt = (token: string, value: string) => + Number.parseInt(token.slice(4), 16) + 1 + value.length * 2; + heap.writeUInt8(1, flagAt(ok, "ok")); + heap.writeUInt8(0, flagAt(quoted, "it's")); + const result = inspect({ + ilBody: ldstrBody([ok, quoted]), + userStringHeap: heap, + }); + + expect(result.coverage.state).toBe("complete"); + expect(result.user_strings?.literals.map(({ value }) => value)).toEqual([ + "ok", + "it's", + ]); + expect(result.user_strings).toMatchObject({ + state: "partial", + issues: [ok, quoted].map((token) => ({ + code: "invalid-string", + scope: `metadata.#US:${token}`, + detail: expect.stringContaining("terminal flag"), + })), + }); + }); +}); + +describe("managed user string scan completeness", () => { + const inspect = (options: Parameters[0]) => { + const bytes = buildManagedPeFixture(options); + return inspectManagedMembersBytes(bytes, managedPeFixtureTarget(bytes)); + }; + + it("marks user strings partial when a method body is not fully decoded", () => { + const { heap, tokens } = fixtureUserStringHeap(["ok", "later"]); + const [ok, later] = tokens as [string, string]; + // ldstr "ok", then an undefined opcode before ldstr "later". + const il = Buffer.from([ + 0x72, 0, 0, 0, 0, 0x26, 0xfe, 0xff, 0x72, 0, 0, 0, 0, 0x26, 0x2a, + ]); + il.writeUInt32LE(Number.parseInt(ok, 16), 1); + il.writeUInt32LE(Number.parseInt(later, 16), 9); + const result = inspect({ + ilBody: Buffer.concat([Buffer.from([(il.length << 2) | 2]), il]), + userStringHeap: heap, + }); + + expect(result.methods[0]?.body.status).toBe("malformed"); + expect(result.user_strings).toEqual({ + state: "partial", + literals: [{ token: ok, offset: expect.any(Number), value: "ok" }], + issues: [], + }); + }); + + it.each([ + ["outside", 1, "complete"], + ["inside", 2, "partial"], + ])( + "judges an unreadable method row %s the selection by its owner", + (_where, badRow, state) => { + const bytes = buildManagedPeFixture({ + additionalEmptyTypes: 1, + methodRowCount: 2, + }); + const cli = readManagedPeLayout(bytes); + if (cli.cli === null) throw new Error("fixture has no CLI header"); + const { rva, size } = cli.cli.metadata; + const layout = readManagedMetadataLayout( + bytes, + cli.rvaToOffset(rva, size, "cli.metadata"), + size, + ); + const write = (offset: number, width: number, value: number) => + width === 2 + ? bytes.writeUInt16LE(value, offset) + : bytes.writeUInt32LE(value, offset); + // Fixture.Program's methods now start at row 2, so Fixture.Empty0 owns row 1. + write( + metadataRowOffset(layout, 2, 2) + + 4 + + 2 * layout.stringIndexSize + + layout.codedIndexSize("TypeDefOrRef") + + layout.tableIndexSize(4), + layout.tableIndexSize(6), + 2, + ); + // An out-of-range name index makes that method row unreadable. + write( + metadataRowOffset(layout, 6, badRow) + 8, + layout.stringIndexSize, + 0xffff, + ); + const result = inspectManagedMembersBytes( + bytes, + managedPeFixtureTarget(bytes), + { typeNames: ["Fixture.Program"] }, + ); + + expect(result.user_strings?.state).toBe(state); + }, + ); + + it("keeps user strings complete when only the exception section is malformed", () => { + const { heap, tokens } = fixtureUserStringHeap(["ok"]); + // ldstr "ok"; pop; ret all decode before an exception section that + // holds no whole clause. + const il = Buffer.from([0x72, 0, 0, 0, 0, 0x26, 0x2a]); + il.writeUInt32LE(Number.parseInt(tokens[0] ?? "", 16), 1); + const header = Buffer.alloc(12); + header.writeUInt16LE(0x300b, 0); + header.writeUInt16LE(8, 2); + header.writeUInt32LE(il.length, 4); + const section = Buffer.alloc(17); + section[0] = 0x01; + section[1] = section.length; + const result = inspect({ + ilBody: Buffer.concat([header, il, Buffer.alloc(1), section]), + userStringHeap: heap, + }); + + expect(result.methods[0]?.body.status).toBe("malformed"); + expect(result.user_strings).toMatchObject({ + state: "complete", + literals: [{ value: "ok" }], + }); + }); + + it("marks user strings partial when selected type identity is unreadable", () => { + const layout = { + userStrings: { name: "#US", offset: 0, size: 1 }, + } as unknown as ManagedMetadataLayout; + expect( + parseUserStrings(Buffer.from([0]), layout, { + methods: [], + unscannedMethods: 0, + unreadableSelectedTypes: 1, + }), + ).toEqual({ state: "partial", literals: [], issues: [] }); + }); + + it("marks user strings partial when an inspected method row was not read", () => { + const layout = { + userStrings: { name: "#US", offset: 0, size: 1 }, + } as unknown as ManagedMetadataLayout; + // The omitted row's body, and any ldstr in it, was never scanned. + expect( + parseUserStrings(Buffer.from([0]), layout, { + methods: [], + unscannedMethods: 1, + }), + ).toEqual({ state: "partial", literals: [], issues: [] }); + }); +}); + describe("managed member decoding budget", () => { it("admits a wide-instruction method whose actual records fit the budget", () => { // Five-byte branches all target the next instruction. A final ret leaves @@ -1243,7 +1694,7 @@ describe("managed member type selection", () => { expect(result.field_accesses).toEqual(everything.field_accesses); expect(result.coverage).toEqual(everything.coverage); expect(result.limitations).toContain( - "Types, fields, methods, call edges, and field accesses cover only the selected declaring types; coverage describes that selection. Edge targets outside it keep their names but are not described.", + "Types, fields, methods, call edges, field accesses, and user strings cover only the selected declaring types; coverage describes that selection. Edge targets outside it keep their names but are not described.", ); }); @@ -1260,6 +1711,8 @@ describe("managed member type selection", () => { expect(result.selection?.unmatched_type_names).toEqual(["Fixture.Program"]); expect(result.coverage.state).toBe("partial"); + // Its methods were never inspected, so neither were their literals. + expect(result.user_strings?.state).toBe("partial"); expect(result.limitations).toContain( "1 TypeDef name(s) could not be read; a name in selection.unmatched_type_names may belong to one of them, so its absence is not observed.", ); diff --git a/src/dotnet/ManagedMemberInspector.ts b/src/dotnet/ManagedMemberInspector.ts index dba20ac57..1b018522a 100644 --- a/src/dotnet/ManagedMemberInspector.ts +++ b/src/dotnet/ManagedMemberInspector.ts @@ -19,6 +19,7 @@ import { } from "./ManagedMetadataInventory.js"; import { readManagedMetadataLayout } from "./ManagedMetadataLayout.js"; import { + appendManagedParseIssues, ManagedReaderFailure, uniqueManagedParseIssues, } from "./ManagedReaderFailure.js"; @@ -29,6 +30,7 @@ import { } from "./ManagedMemberInspectorCore.js"; import { edgeTargets } from "./ManagedMethodSpecs.js"; import { typeNameLookup } from "./ManagedTypeNames.js"; +import { parseUserStrings } from "./ManagedUserStrings.js"; import { edges, parseExportedTypes, @@ -70,6 +72,7 @@ const unavailable = ( }, types: [], exported_types: null, + user_strings: null, fields: [], methods: [], member_refs: [], @@ -149,11 +152,19 @@ const selectTypes = ( /** * Inspect metadata members and method bodies without loading target code. * Selected type names restrict full decoding to members of those exact types. + * A caller that never reads literals, such as member comparison, can skip + * user strings; they are then reported as not examined. */ export const inspectManagedMembersBytes = ( bytes: Buffer, target: BinaryTarget, - { typeNames }: { readonly typeNames?: readonly string[] } = {}, + { + typeNames, + userStrings: readUserStrings = true, + }: { + readonly typeNames?: readonly string[]; + readonly userStrings?: boolean; + } = {}, ): ManagedMemberInspection => { const pe = readManagedPeLayout(bytes); if (pe.cli === null) @@ -161,7 +172,7 @@ export const inspectManagedMembersBytes = ( const issues: ManagedParseIssue[] = []; try { const { layout, inventory } = readMemberInventory(bytes, pe, issues); - issues.push(...inventory.issues); + appendManagedParseIssues(issues, inventory.issues); const ranges = typeRanges(bytes, layout, issues); const { tokens: selected, @@ -169,16 +180,16 @@ export const inspectManagedMembersBytes = ( unreadableNames, } = selectTypes(ranges, typeNames); const parsedTypes = parseTypes(bytes, layout, ranges); - issues.push(...parsedTypes.issues); + appendManagedParseIssues(issues, parsedTypes.issues); const exportedTypes = parseExportedTypes(bytes, layout); - issues.push(...exportedTypes.issues); + appendManagedParseIssues(issues, exportedTypes.issues); const fields = parseFields(bytes, layout, ranges, selected); const memberRefs = parseMemberRefs( bytes, layout, typeNameLookup(bytes, layout, ranges, issues), ); - issues.push(...memberRefs.issues); + appendManagedParseIssues(issues, memberRefs.issues); const methods = parseMethods({ bytes, layout, @@ -186,13 +197,21 @@ export const inspectManagedMembersBytes = ( ranges, selected, }); - issues.push(...fields.issues, ...methods.issues); + appendManagedParseIssues(issues, fields.issues, methods.issues); + // #US literals sit beside the member inventory and carry their own + // coverage, so their issues never make member facts partial. + const userStrings = readUserStrings + ? parseUserStrings(bytes, layout, { + ...methods, + unreadableSelectedTypes: unreadableNames, + }) + : null; const targets = edgeTargets(bytes, layout, methods.methods, [ fields.core, memberRefs.core, methods.core, ]); - issues.push(...targets.issues); + appendManagedParseIssues(issues, targets.issues); const coverageIssues = uniqueManagedParseIssues(issues); const incompleteMembers = [...fields.fields, ...memberRefs.refs, ...methods.methods].some( @@ -238,6 +257,7 @@ export const inspectManagedMembersBytes = ( member_refs: memberRefs.refs, call_edges: related.callEdges, field_accesses: related.fieldAccesses, + user_strings: userStrings, selection, coverage: { state: @@ -250,11 +270,12 @@ export const inspectManagedMembersBytes = ( "Metadata tokens are build-local coordinates and are only meaningful with the reported artifact SHA-256 and MVID.", "CIL instruction anchors are decoded from file-backed method bodies only; no target assembly is loaded or executed.", "Exported types are reported from ExportedType rows as declared; forwarded types are not resolved in their implementing assemblies.", + "User strings are the #US literals loaded by ldstr in the inspected method bodies, with their own coverage; instruction anchors and normalized CIL digests keep their build-local tokens.", "Member signatures decode ECMA-335 primitive, class, valuetype, generic instantiation, generic variable, pointer, byref, single-dimension and general array, function pointer, custom modifier, and typed reference forms; class and valuetype operands keep their encoded TypeDefOrRefOrSpec values, and unsupported forms such as vararg sentinels retain raw signature hashes.", ...(selection === null ? [] : [ - "Types, fields, methods, call edges, and field accesses cover only the selected declaring types; coverage describes that selection. Edge targets outside it keep their names but are not described.", + "Types, fields, methods, call edges, field accesses, and user strings cover only the selected declaring types; coverage describes that selection. Edge targets outside it keep their names but are not described.", ]), ...(unreadableNames === 0 ? [] diff --git a/src/dotnet/ManagedMemberInspectorCore.ts b/src/dotnet/ManagedMemberInspectorCore.ts index 7776c0090..caac2e66e 100644 --- a/src/dotnet/ManagedMemberInspectorCore.ts +++ b/src/dotnet/ManagedMemberInspectorCore.ts @@ -35,6 +35,9 @@ export type ManagedMemberRef = ManagedMemberInspection["member_refs"][number]; export type ManagedCallEdge = ManagedMemberInspection["call_edges"][number]; export type ManagedFieldAccess = ManagedMemberInspection["field_accesses"][number]; +export type ManagedUserString = NonNullable< + ManagedMemberInspection["user_strings"] +>["literals"][number]; export type ManagedSignature = ManagedMethod["signature"]; export type ManagedMethodBody = ManagedMethod["body"]; export type ManagedInstructionAnchor = ManagedMethodBody["anchors"][number]; diff --git a/src/dotnet/ManagedMemberRows.ts b/src/dotnet/ManagedMemberRows.ts index 201ff1c45..700f723e6 100644 --- a/src/dotnet/ManagedMemberRows.ts +++ b/src/dotnet/ManagedMemberRows.ts @@ -269,16 +269,36 @@ export const parseMethods = ({ readonly methods: readonly ManagedMethod[]; readonly core: ReadonlyMap; readonly issues: readonly ManagedParseIssue[]; + /** + * Methods that may belong to a selected type but whose CIL was not fully + * scanned: rows that produced no method, or bodies whose decoding stopped. + */ + readonly unscannedMethods: number; } => { const methods: ManagedMethod[] = []; const core = new Map(); const issues: ManagedParseIssue[] = []; + let unscannedMethods = 0; const table = layout.table(6); const ownerRow = readManagedValue( () => memberDeclaringTypeRowLookup(bytes, layout, 6), issues, ); for (let row = 1; row <= (table?.rowCount ?? 0); row += 1) { + // The owner is read first: a row that produced no method hides a selected + // body only when its owner is selected, unknown, or unreadably named. + const owner = + ownerRow === undefined + ? undefined + : readManagedValue(() => ownerRow(row), issues); + const declared = owner === undefined ? undefined : ranges[owner - 1]; + const mayBeSelected = + selected === null || + owner === undefined || + isSelected(selected, declared) || + // A type whose name is unreadable may be one the caller requested. + declared?.fullName === null; + const produced = methods.length; readManagedValue(() => { const cursor = metadataRowCursor(bytes, layout, 6, row); const rva = cursor.readUInt32(); @@ -290,11 +310,6 @@ export const parseMethods = ({ cursor.readIndex(layout.stringIndexSize), layout.strings.size, ); - const owner = - ownerRow === undefined - ? 0 - : (readManagedValue(() => ownerRow(row), issues) ?? 0); - const declared = ranges[owner - 1]; const token = metadataToken(6, row); const member = { token, @@ -316,6 +331,14 @@ export const parseMethods = ({ ...member, genericArity: signatureGenericArity(decoded), }); + const read = methodBody( + bytes, + pe, + rva, + { implFlags, flags }, + managedDecodeBudget(layout), + ); + if (!read.scanned) unscannedMethods += 1; methods.push({ token, row_offset: cursor.start, @@ -326,17 +349,12 @@ export const parseMethods = ({ impl_flags: implFlags, flags, signature: decoded, - body: methodBody( - bytes, - pe, - rva, - { implFlags, flags }, - managedDecodeBudget(layout), - ), + body: read.body, }); }, issues); + if (mayBeSelected && methods.length === produced) unscannedMethods += 1; } - return { methods, core, issues }; + return { methods, core, issues, unscannedMethods }; }; /** diff --git a/src/dotnet/ManagedMetadataHeaps.test.ts b/src/dotnet/ManagedMetadataHeaps.test.ts index c94ae4055..ebc2a0d49 100644 --- a/src/dotnet/ManagedMetadataHeaps.test.ts +++ b/src/dotnet/ManagedMetadataHeaps.test.ts @@ -1,6 +1,9 @@ import { describe, expect, it } from "vitest"; -import { readMetadataBlob } from "./ManagedMetadataHeaps.js"; +import { + readMetadataBlob, + readMetadataUserString, +} from "./ManagedMetadataHeaps.js"; import type { ManagedMetadataLayout } from "./ManagedMetadataLayout.js"; describe("managed metadata heaps", () => { @@ -16,6 +19,7 @@ describe("managed metadata heaps", () => { strings: { name: "#Strings", offset: 0, size: 0 }, guid: { name: "#GUID", offset: 0, size: 0 }, blob: { name: "#Blob", offset: 0, size: bytes.length }, + userStrings: null, tables: new Map(), rowCounts: [], stringIndexSize: 2, @@ -30,4 +34,26 @@ describe("managed metadata heaps", () => { itemLength, ); }); + + it.each([[[0x80]], [[0xc0, 0x00, 0x00]]])( + "reports a #US length prefix cut off by the end of the artifact as a string failure", + (prefix) => { + // #US ends at the end of the artifact, inside a multi-byte length prefix. + const bytes = Buffer.from([0x00, ...prefix]); + const layout = { + userStrings: { name: "#US", offset: 0, size: bytes.length }, + } as unknown as ManagedMetadataLayout; + + expect(() => + readMetadataUserString(bytes, layout, "0x70000001"), + ).toThrowError( + expect.objectContaining({ + issue: expect.objectContaining({ + code: "invalid-string", + scope: "metadata.#US:0x70000001", + }), + }), + ); + }, + ); }); diff --git a/src/dotnet/ManagedMetadataHeaps.ts b/src/dotnet/ManagedMetadataHeaps.ts index 51b85576c..0987680f0 100644 --- a/src/dotnet/ManagedMetadataHeaps.ts +++ b/src/dotnet/ManagedMetadataHeaps.ts @@ -1,3 +1,4 @@ +import type { ManagedParseIssue } from "../domain/managed/managedArtifact.js"; import { createHash } from "node:crypto"; import { managedDecodeBudget } from "./ManagedDecodeBudget.js"; @@ -8,21 +9,6 @@ import { } from "./ManagedMetadataLayout.js"; import { managedFailure } from "./ManagedReaderFailure.js"; -const requireRange = ( - bytes: Buffer, - offset: number, - length: number, - scope: string, -): void => { - if (offset < 0 || length < 0 || offset > bytes.length - length) - throw managedFailure( - "invalid-row", - scope, - `${scope} leaves the artifact byte range`, - offset >= 0 ? offset : null, - ); -}; - /** Cursor for checked fixed-width metadata row decoding. */ export class MetadataRowCursor { #offset: number; @@ -69,36 +55,54 @@ export class MetadataRowCursor { } } +// Issue code and label for the heap item a compressed length prefixes. +const PREFIXED_ITEMS = { + blob: { code: "invalid-blob", label: "Metadata blob" }, + userString: { code: "invalid-string", label: "User string" }, +} as const; + const compressedLength = ( bytes: Buffer, offset: number, scope: string, -): { readonly length: number; readonly prefix: number } => { - requireRange(bytes, offset, 1, scope); + item: keyof typeof PREFIXED_ITEMS, +): { + readonly length: number; + readonly prefix: number; + /** Whether ECMA-335 II.23.2 assigns this width to the value. */ + readonly minimal: boolean; +} => { + const { code, label } = PREFIXED_ITEMS[item]; + const requirePrefix = (size: number): void => { + if (offset < 0 || offset > bytes.length - size) + throw managedFailure( + code, + scope, + `${label} length prefix leaves the artifact byte range`, + offset >= 0 ? offset : null, + ); + }; + requirePrefix(1); const first = bytes[offset] ?? 0; - if ((first & 0x80) === 0) return { length: first, prefix: 1 }; + if ((first & 0x80) === 0) return { length: first, prefix: 1, minimal: true }; if ((first & 0xc0) === 0x80) { - requireRange(bytes, offset, 2, scope); - return { - length: ((first & 0x3f) << 8) | (bytes[offset + 1] ?? 0), - prefix: 2, - }; + requirePrefix(2); + const length = ((first & 0x3f) << 8) | (bytes[offset + 1] ?? 0); + return { length, prefix: 2, minimal: length >= 0x80 }; } if ((first & 0xe0) === 0xc0) { - requireRange(bytes, offset, 4, scope); - return { - length: - (first & 0x1f) * 0x01_00_00_00 + - (bytes[offset + 1] ?? 0) * 0x01_00_00 + - (bytes[offset + 2] ?? 0) * 0x0100 + - (bytes[offset + 3] ?? 0), - prefix: 4, - }; + requirePrefix(4); + const length = + (first & 0x1f) * 0x01_00_00_00 + + (bytes[offset + 1] ?? 0) * 0x01_00_00 + + (bytes[offset + 2] ?? 0) * 0x0100 + + (bytes[offset + 3] ?? 0); + return { length, prefix: 4, minimal: length >= 0x4000 }; } throw managedFailure( - "invalid-blob", + code, scope, - "Metadata blob uses a reserved compressed-length prefix", + `${label} uses a reserved compressed-length prefix`, offset, ); }; @@ -172,7 +176,7 @@ export const readMetadataBlob = ( layout.blob.offset + Math.max(index, 0), ); const start = layout.blob.offset + index; - const decoded = compressedLength(bytes, start, "metadata.#Blob"); + const decoded = compressedLength(bytes, start, "metadata.#Blob", "blob"); if (decoded.length > heapExtent) throw managedFailure( "invalid-blob", @@ -191,6 +195,124 @@ export const readMetadataBlob = ( return bytes.subarray(content, content + decoded.length); }; +/** + * Read the #US literal named by a `0x70` user-string token: a compressed byte + * length, UTF-16LE code units, then one ECMA-335 terminal flag byte. Code units + * are kept as stored, including unpaired surrogates. `parseUserStrings` owns + * which `ldstr` operands are user-string tokens. + */ +export const readMetadataUserString = ( + bytes: Buffer, + layout: ManagedMetadataLayout, + token: string, +): { + readonly offset: number; + readonly value: string; + /** Conformance differences observed beside the readable literal. */ + readonly issues: readonly ManagedParseIssue[]; +} => { + const heap = layout.userStrings; + const scope = `metadata.#US:${token}`; + if (heap === null) + throw managedFailure( + "missing-stream", + scope, + "ldstr references #US, but the metadata has no #US stream", + ); + const index = Number.parseInt(token.slice(4), 16); + if (index >= heap.size) + throw managedFailure( + "invalid-heap-index", + scope, + "ldstr operand does not name an item inside #US", + heap.offset + index, + ); + const start = heap.offset + index; + const decoded = compressedLength(bytes, start, scope, "userString"); + const content = start + decoded.prefix; + if (content > heap.offset + heap.size - decoded.length) + throw managedFailure( + "invalid-string", + scope, + "User string leaves #US", + start, + ); + const budget = managedDecodeBudget(layout); + // A wider length prefix than the value needs still reads unambiguously, so + // it is reported like the flag rather than discarding the literal. It is + // admitted only on paths that return the literal, so a fatal item leaves + // no budget behind. + const conformance = (): ManagedParseIssue[] => { + if (decoded.minimal) return []; + budget.reserve(512, start); + return [ + { + code: "invalid-string", + scope, + offset: start, + detail: `User string length ${String(decoded.length)} uses a ${String(decoded.prefix)}-byte prefix wider than ECMA-335 II.23.2 assigns; readable UTF-16 is preserved and runtime acceptance is not assessed`, + }, + ]; + }; + // ECMA-335 II.24.2.4: offset 0 is the heap's empty entry, a zero length + // with no terminal flag byte; every other item carries the flag. + if (index === 0) { + if (decoded.length === 0) + return { offset: start, value: "", issues: conformance() }; + throw managedFailure( + "invalid-string", + scope, + `#US offset 0 is the reserved empty entry, but it encodes ${String(decoded.length)} byte(s)`, + start, + ); + } + if (decoded.length % 2 !== 1) + throw managedFailure( + "invalid-string", + scope, + "User string length is not UTF-16 code units plus one terminal flag byte", + start, + ); + const units = (decoded.length - 1) / 2; + // Admit the literal before scanning it: JSON spends at most six bytes per + // UTF-16 code unit. + budget.reserveString(units, start); + // Check the serialized flag independently of literal decoding. A difference + // from ECMA-335 is observable metadata, not proof of runtime rejection. + let expectedFlag = 0; + for (let unitIndex = 0; unitIndex < units; unitIndex++) { + const unit = bytes.readUInt16LE(content + unitIndex * 2); + if ( + unit > 0xff || + (unit >= 0x01 && unit <= 0x08) || + (unit >= 0x0e && unit <= 0x1f) || + unit === 0x27 || + unit === 0x2d || + unit === 0x7f + ) { + expectedFlag = 1; + break; + } + } + const flagOffset = content + units * 2; + const actualFlag = bytes[flagOffset]; + const issues = conformance(); + if (actualFlag !== expectedFlag) { + budget.reserve(512, flagOffset); + issues.push({ + code: "invalid-string", + scope, + offset: flagOffset, + detail: `User string terminal flag ${String(actualFlag)} differs from ECMA-335 II.24.2.4 value ${String(expectedFlag)}; readable UTF-16 is preserved and runtime acceptance is not assessed`, + }); + } + return { + offset: start, + value: bytes.subarray(content, flagOffset).toString("utf16le"), + issues, + }; +}; + /** Read one conventional GUID value from the one-based #GUID heap. */ export const readMetadataGuid = ( bytes: Buffer, diff --git a/src/dotnet/ManagedMetadataLayout.ts b/src/dotnet/ManagedMetadataLayout.ts index b7dc0a7ae..b38854645 100644 --- a/src/dotnet/ManagedMetadataLayout.ts +++ b/src/dotnet/ManagedMetadataLayout.ts @@ -78,6 +78,8 @@ export interface ManagedMetadataLayout { readonly strings: MetadataStream; readonly guid: MetadataStream; readonly blob: MetadataStream; + /** Optional: assemblies without `ldstr` literals may omit #US. */ + readonly userStrings: MetadataStream | null; readonly tables: ReadonlyMap; readonly rowCounts: readonly number[]; readonly stringIndexSize: 2 | 4; @@ -431,6 +433,7 @@ export const readManagedMetadataLayout = ( strings: requireStream(streams, "#Strings"), guid: requireStream(streams, "#GUID"), blob: requireStream(streams, "#Blob"), + userStrings: streams.get("#US") ?? null, tables, rowCounts, stringIndexSize: heaps.string, diff --git a/src/dotnet/ManagedMethodBodyReader.ts b/src/dotnet/ManagedMethodBodyReader.ts index 7c507cdd9..81f3543a2 100644 --- a/src/dotnet/ManagedMethodBodyReader.ts +++ b/src/dotnet/ManagedMethodBodyReader.ts @@ -76,17 +76,25 @@ export const methodBody = ( flags: 0, }, budget?: ManagedDecodeBudget, -): ManagedMethodBody => { - if (rva === 0) return emptyMethodBody(rva, "absent", null); - if ( +): { + readonly body: ManagedMethodBody; + /** + * Whether every CIL instruction was decoded, independent of exception-region + * validity. A body with no CIL, absent or native, has nothing left unscanned. + */ + readonly scanned: boolean; +} => { + // Without CIL there is nothing left unscanned. + if (rva === 0) + return { body: emptyMethodBody(rva, "absent", null), scanned: true }; + const native = (implementation.implFlags & 7) !== 0 || - (implementation.flags & 0x2000) !== 0 - ) - return emptyMethodBody( - rva, - "partial", - "CIL decoding is unavailable for native, runtime, unmanaged, or P/Invoke implementations", - ); + (implementation.flags & 0x2000) !== 0; + if (native) { + const issue = + "CIL decoding is unavailable for native, runtime, unmanaged, or P/Invoke implementations"; + return { body: emptyMethodBody(rva, "partial", issue), scanned: true }; + } try { const offset = pe.rvaToOffset(rva, 1, "method.body"); const methodExtent = pe.rvaAvailableBytes(rva, "method.body"); @@ -181,7 +189,10 @@ export const methodBody = ( ? "Instruction decoding stopped before the end of the method body" : null), }; - return budget === undefined ? body : admitManagedProjection(body, budget); + return { + body: budget === undefined ? body : admitManagedProjection(body, budget), + scanned: decoded.issue === null && decoded.truncated === 0, + }; } catch (cause: unknown) { if ( cause instanceof ManagedReaderFailure && @@ -193,11 +204,14 @@ export const methodBody = ( !(cause instanceof RangeError) ) throw cause; - return emptyMethodBody( - rva, - "malformed", - cause instanceof Error ? cause.message : "Method body parse failed", - ); + return { + body: emptyMethodBody( + rva, + "malformed", + cause instanceof Error ? cause.message : "Method body parse failed", + ), + scanned: false, + }; } finally { // IL and exception-section bookkeeping coexist until the body is built. budget?.releaseTransient(); diff --git a/src/dotnet/ManagedPe.fixture.ts b/src/dotnet/ManagedPe.fixture.ts index 7ea2b06b4..0d4cf4978 100644 --- a/src/dotnet/ManagedPe.fixture.ts +++ b/src/dotnet/ManagedPe.fixture.ts @@ -53,6 +53,8 @@ interface ManagedPeFixtureOptions { readonly targetFramework?: string | null; readonly typeName?: string; readonly typeNamespace?: string; + /** Raw #US stream bytes; omitted means the metadata has no #US stream. */ + readonly userStringHeap?: Buffer; readonly virtualSectionSize?: number; } @@ -416,6 +418,56 @@ const fixedStringAttributeBlob = (value: string): Buffer => { return Buffer.concat([Buffer.from([1, 0, bytes.length]), bytes, u16(0)]); }; +/** + * Lay out ECMA-335 #US items (compressed length, UTF-16LE, terminal flag byte) + * and return the `ldstr` token that names each value. + */ +// ECMA-335 II.24.2.4 terminal flag: 1 when a code unit has a high byte or a +// low byte in 0x01-0x08, 0x0E-0x1F, 0x27, 0x2D, or 0x7F. +const userStringFlag = (text: Buffer): number => { + for (let index = 0; index < text.length; index += 2) { + const unit = text.readUInt16LE(index); + if ( + unit > 0xff || + (unit >= 0x01 && unit <= 0x08) || + (unit >= 0x0e && unit <= 0x1f) || + unit === 0x27 || + unit === 0x2d || + unit === 0x7f + ) + return 1; + } + return 0; +}; + +/** A #US heap whose items carry the ECMA-335 terminal flag, and their tokens. */ +export const fixtureUserStringHeap = ( + values: readonly string[], +): { readonly heap: Buffer; readonly tokens: readonly string[] } => { + const items: Buffer[] = [Buffer.from([0])]; + const tokens: string[] = []; + let size = 1; + for (const value of values) { + const text = Buffer.from(value, "utf16le"); + const length = text.length + 1; + if (length > 0x3fff) + throw new RangeError("Managed PE fixture user string is too long"); + const prefix = + length < 0x80 + ? Buffer.from([length]) + : Buffer.from([0x80 | (length >> 8), length & 0xff]); + tokens.push(`0x70${size.toString(16).padStart(6, "0")}`); + const item = Buffer.concat([ + prefix, + text, + Buffer.from([userStringFlag(text)]), + ]); + items.push(item); + size += item.length; + } + return { heap: Buffer.concat(items), tokens }; +}; + const metadataStreamHeader = ( relativeOffset: number, size: number, @@ -428,12 +480,28 @@ const metadataStreamHeader = ( }; const metadataRoot = ( - tables: Buffer, - strings: Buffer, - guid: Buffer, - blob: Buffer, + { + tables, + strings, + guid, + blob, + userStrings, + }: { + readonly tables: Buffer; + readonly strings: Buffer; + readonly guid: Buffer; + readonly blob: Buffer; + readonly userStrings?: Buffer | undefined; + }, tableStreamName: "#~" | "#-" = "#~", ): Buffer => { + const streams: readonly (readonly [string, Buffer])[] = [ + [tableStreamName, tables], + ["#Strings", strings], + ...(userStrings === undefined ? [] : [["#US", userStrings] as const]), + ["#GUID", guid], + ["#Blob", blob], + ]; const version = Buffer.from("v4.0.30319\0\0", "utf8"); const header = Buffer.concat([ u32(0x424a_5342), @@ -443,34 +511,27 @@ const metadataRoot = ( u32(version.length), version, u16(0), - u16(4), + u16(streams.length), ]); - const streamHeaderSize = - metadataStreamHeader(0, 0, tableStreamName).length + - metadataStreamHeader(0, 0, "#Strings").length + - metadataStreamHeader(0, 0, "#GUID").length + - metadataStreamHeader(0, 0, "#Blob").length; + const streamHeaderSize = streams.reduce( + (total, [name]) => total + metadataStreamHeader(0, 0, name).length, + 0, + ); let offset = align4(header.length + streamHeaderSize); - const tablesOffset = offset; - offset = align4(offset + tables.length); - const stringsOffset = offset; - offset = align4(offset + strings.length); - const guidOffset = offset; - offset = align4(offset + guid.length); - const blobOffset = offset; - const headers = Buffer.concat([ - metadataStreamHeader(tablesOffset, tables.length, tableStreamName), - metadataStreamHeader(stringsOffset, strings.length, "#Strings"), - metadataStreamHeader(guidOffset, guid.length, "#GUID"), - metadataStreamHeader(blobOffset, blob.length, "#Blob"), - ]); - const root = Buffer.alloc(align4(blobOffset + blob.length)); + const placed = streams.map(([name, data]) => { + const at = offset; + offset = align4(offset + data.length); + return { name, data, at }; + }); + const headers = Buffer.concat( + placed.map(({ name, data, at }) => + metadataStreamHeader(at, data.length, name), + ), + ); + const root = Buffer.alloc(offset); header.copy(root, 0); headers.copy(root, header.length); - tables.copy(root, tablesOffset); - strings.copy(root, stringsOffset); - guid.copy(root, guidOffset); - blob.copy(root, blobOffset); + for (const { data, at } of placed) data.copy(root, at); return root; }; @@ -951,17 +1012,18 @@ const buildManagedFixtureMetadata = ( (stringSize === 4 ? 0x01 : 0) | (guidSize === 4 ? 0x02 : 0) | (blobSize === 4 ? 0x04 : 0); - const metadata = metadataRoot( - tablesStream( + const metadata = metadataRoot({ + tables: tablesStream( rows, options.metadataValidMaskExtra ?? 0n, heapFlags, options.metadataExtraData, ), - strings.toBuffer(), - guidHeap(options.mvid), - blobs.toBuffer(), - ); + strings: strings.toBuffer(), + guid: guidHeap(options.mvid), + blob: blobs.toBuffer(), + userStrings: options.userStringHeap, + }); if (options.corruptMetadataSignature === true) metadata.writeUInt32LE(0, 0); return metadata; }; @@ -1189,10 +1251,12 @@ export const buildManagedMemberPointerFixture = ( if (fieldPointers !== null) rows.set(3, fieldPointers.map(u16)); if (methodPointers !== null) rows.set(5, methodPointers.map(u16)); return metadataRoot( - tablesStream(rows, 0n, 0), - strings.toBuffer(), - guidHeap(), - blobs.toBuffer(), + { + tables: tablesStream(rows, 0n, 0), + strings: strings.toBuffer(), + guid: guidHeap(), + blob: blobs.toBuffer(), + }, options.streamName ?? "#-", ); }); diff --git a/src/dotnet/ManagedReaderFailure.ts b/src/dotnet/ManagedReaderFailure.ts index 5e16c0ca4..e360571b4 100644 --- a/src/dotnet/ManagedReaderFailure.ts +++ b/src/dotnet/ManagedReaderFailure.ts @@ -43,3 +43,14 @@ export const uniqueManagedParseIssues = ( ): readonly ManagedParseIssue[] => [ ...new Map(issues.map((issue) => [JSON.stringify(issue), issue])).values(), ]; + +/** + * Append issue lists one issue at a time. Spreading a list into `push` passes + * one argument per issue, which overflows the stack on large malformed inputs. + */ +export const appendManagedParseIssues = ( + issues: ManagedParseIssue[], + ...found: readonly (readonly ManagedParseIssue[])[] +): void => { + for (const list of found) for (const issue of list) issues.push(issue); +}; diff --git a/src/dotnet/ManagedUserStrings.ts b/src/dotnet/ManagedUserStrings.ts new file mode 100644 index 000000000..cce2f0b65 --- /dev/null +++ b/src/dotnet/ManagedUserStrings.ts @@ -0,0 +1,120 @@ +import { + admitManagedProjection, + managedDecodeBudget, +} from "./ManagedDecodeBudget.js"; +import type { + ManagedMemberInspection, + ManagedParseIssue, +} from "../domain/managed/managedArtifact.js"; +import type { + ManagedMethod, + ManagedUserString, +} from "./ManagedMemberInspectorCore.js"; +import type { ManagedMetadataLayout } from "./ManagedMetadataLayout.js"; +import { readMetadataUserString } from "./ManagedMetadataHeaps.js"; +import { readManagedValue } from "./ManagedReaderFailure.js"; + +type UserStringInventory = NonNullable; + +/** + * The literals with their own coverage, which never enters member coverage. + * Issues are unique by construction, so they are not deduplicated: each + * distinct token yields one failure or per-offset conformance issues, and a + * missing stream is one issue. + */ +const inventory = ( + literals: ManagedUserString[], + issues: ManagedParseIssue[], + scanned: boolean, +): UserStringInventory => ({ + state: issues.length === 0 && scanned ? "complete" : "partial", + literals, + issues, +}); + +/** Bookkeeping per distinct ldstr token: its set entry and its sorted slot. */ +const USER_STRING_TOKEN_BYTES = 128; + +/** Decode each distinct #US literal an inspected method loads, in token order. */ +export const parseUserStrings = ( + bytes: Buffer, + layout: ManagedMetadataLayout, + parsed: { + readonly methods: readonly ManagedMethod[]; + /** Methods that may be in scope but whose CIL was not fully scanned. */ + readonly unscannedMethods: number; + /** Unreadable selected type names may hide a requested type. */ + readonly unreadableSelectedTypes?: number; + }, +): UserStringInventory => { + const { methods } = parsed; + const budget = managedDecodeBudget(layout); + const scanned = + parsed.unscannedMethods === 0 && + (parsed.unreadableSelectedTypes ?? 0) === 0; + const userStrings: ManagedUserString[] = []; + const issues: ManagedParseIssue[] = []; + const tokens = new Set(); + try { + // Only ldstr loads a literal; a malformed ldtoken can carry a 0x70 token. + for (const method of methods) + for (const { opcode, operand } of method.body.anchors) + if (opcode === "ldstr" && operand !== null && !tokens.has(operand)) { + budget.admitTransient(USER_STRING_TOKEN_BYTES, null); + tokens.add(operand); + } + // Tokens are fixed-width lowercase hex, so string order is token order. + const references: string[] = []; + for (const token of [...tokens].sort()) { + if (token.startsWith("0x70")) references.push(token); + // An operand from another table names no #US item, with or without + // the stream. + else + issues.push( + admitManagedProjection( + { + code: "invalid-heap-index", + scope: `metadata.#US:${token}`, + offset: null, + detail: "ldstr operand does not name an item inside #US", + }, + budget, + ), + ); + } + if (references.length > 0 && layout.userStrings === null) { + // A missing stream is one fact about the metadata, not one per token. + issues.push( + admitManagedProjection( + { + code: "missing-stream", + scope: "metadata.#US", + offset: null, + detail: `${String(references.length)} distinct ldstr operand(s) reference #US, but the metadata has no #US stream`, + }, + budget, + ), + ); + return inventory(userStrings, issues, scanned); + } + for (const token of references) { + const failures: ManagedParseIssue[] = []; + readManagedValue(() => { + const { + offset, + value, + issues: observed, + } = readMetadataUserString(bytes, layout, token); + for (const issue of observed) failures.push(issue); + userStrings.push( + admitManagedProjection({ token, offset, value }, budget), + ); + }, failures); + for (const failure of failures) + issues.push(admitManagedProjection(failure, budget)); + } + return inventory(userStrings, issues, scanned); + } finally { + budget.releaseTransient(); + } +}; diff --git a/tests/boundary/providers/managed/managedMemberComparisonFromMetadata.test.ts b/tests/boundary/providers/managed/managedMemberComparisonFromMetadata.test.ts index 56014fbd4..c4c4e1e1a 100644 --- a/tests/boundary/providers/managed/managedMemberComparisonFromMetadata.test.ts +++ b/tests/boundary/providers/managed/managedMemberComparisonFromMetadata.test.ts @@ -313,6 +313,30 @@ describe("managed member comparison uncertainty", () => { }); }); +describe("managed member comparison beside user strings", () => { + it("keeps member deltas and coverage when only #US literals are nonconforming", () => { + // The method loads "ok", stored with flag 1 where ECMA-335 expects 0. + const options = { + ilBody: Buffer.from([(7 << 2) | 2, 0x72, 1, 0, 0, 0x70, 0x26, 0x2a]), + userStringHeap: Buffer.from([0, 5, 0x6f, 0, 0x6b, 0, 1]), + }; + const left = inspect(buildManagedPeFixture(options), "/tmp/left-us.dll"); + const right = inspect(buildManagedPeFixture(options), "/tmp/right-us.dll"); + const result = compareManagedMembers( + { evidenceId: left.evidenceId, result: { ...left.result, fields: [] } }, + { + evidenceId: right.evidenceId, + result: { ...right.result, methods: [] }, + }, + ); + + expect(left.result.metadata.status).toBe("complete"); + expect(left.result.coverage.state).toBe("complete"); + expect(left.result.user_strings?.state).toBe("partial"); + expect(result.summary).toMatchObject({ added: 1, removed: 1, unknown: 0 }); + }); +}); + describe("managed member comparison of undecoded signatures", () => { it("remaps renamed methods only when raw signatures and complete normalized IL match", () => { const signature = Buffer.from([0x00, 0x00, 0x7f]);