diff --git a/experiments/analysis-efficiency/process-trace-validation.md b/experiments/analysis-efficiency/process-trace-validation.md index 0a4f9b0f6..a32848596 100644 --- a/experiments/analysis-efficiency/process-trace-validation.md +++ b/experiments/analysis-efficiency/process-trace-validation.md @@ -2,13 +2,19 @@ A 6,000-event directed cycle previously overflowed the JavaScript stack inside the schema refinement. Cycle detection now uses iterative topological traversal. -Reachability is computed once per queried origin; cyclic inputs retain -independent declaration diagnostics without expanding all event pairs. Unordered -declarations retain group membership, with one diagnostic per overlapping group, -instead of allocating a pair set and repeating the same error for every pair. -Valid partial orders still receive complete pair validation. +Cyclic inputs retain independent declaration diagnostics without expanding all +event pairs. Unordered declarations retain group membership, with one diagnostic +per overlapping or conflicting group, instead of allocating a pair set and +repeating the same error for every pair. -## Bounded comparison +A unique topological order proves that every event pair is ordered, allowing +rank comparisons without retaining transitive closure. Other acyclic graphs +retain only one origin's forward and backward reachability sets at a time. +Complete pair validation preserves the missing count and first declaration-order +missing pair. General partial orders still require quadratic pair examination +and repeated graph traversal; they trade CPU time for bounded retained state. + +## Initial cycle fix comparison Flameox ran three randomized paired blocks (seed 917), each limited to 20 seconds and 512 MiB. Node 24.18.0 used a 384-MiB old-space limit on macOS arm64. Each @@ -17,7 +23,8 @@ known 6,000-event cycle failure with per-event not-before constraints and one unordered group containing every event. All six executions passed the embedded answer digest and expected rejection checks. -The baseline source is from `bb540826d`; the candidate includes this trace fix. +The baseline source is from `bb540826d`; the candidate is the initial cycle fix +in `10935b7b1`, which still memoized reachability sets for every queried origin. Both modules had TypeScript annotations removed with Node's identical built-in transform and used the same copied runtime dependencies. This isolates schema validation rather than comparing complete packaged CLI/MCP processes. @@ -44,8 +51,8 @@ the process but create fresh per-validation graph state. Whole-process elapsed includes startup, imports, all four validations, the cyclic-input check, token estimation and reporting. CPU includes startup through metrics publication; peak RSS covers the complete single-process workload, with no children. Host -caches were not flushed. Peak memory increased in this cohort. The remaining -valid-order closure and pair validation can still require quadratic resources. +caches were not flushed. Peak memory increased in this cohort. Its retained +valid-order closure still required quadratic memory, prompting the follow-up. Three pairs describe this fixture only. Preserved Flameox evidence: @@ -57,3 +64,50 @@ isolated runtime copy. Source-module SHA-256 values are `a7729faa2183b0e493b9ac4fc55f6feaee5b2e5294be13d1b29b57a44b899788` (baseline) and `725409ce84902125941251037a8a214ccc3c1eec577e484c164c361ca580d693` (candidate). + +## Bounded retention follow-up + +Flameox compared `10935b7b1` with the rank/streaming implementation in three +randomized paired blocks (seed 624), under the same host, Node, heap, memory and +20-second execution bounds. Each execution validated the chain above four times, +then a 1,200-event partial order with 600 unordered two-event layers and all four +edges between consecutive layers four times, then the same cyclic failure. +All six executions passed both parsed-answer digests and the cycle diagnostic. +Runtime dependencies were copied and isolated from concurrent checkout builds. + +| Median measurement | Memoized closure | Rank/streaming | +| ---------------------------------------- | ---------------: | -------------: | +| Chain first elapsed milliseconds | 166.11 | 11.37 | +| Chain warm elapsed milliseconds | 154.25 | 5.32 | +| Chain first CPU milliseconds | 210.13 | 24.38 | +| Chain warm CPU milliseconds | 184.76 | 16.84 | +| Partial order first elapsed milliseconds | 49.44 | 116.54 | +| Partial order warm elapsed milliseconds | 50.67 | 110.29 | +| Partial order first CPU milliseconds | 72.47 | 137.69 | +| Partial order warm CPU milliseconds | 70.65 | 121.77 | +| Capture process elapsed milliseconds | 901.24 | 557.30 | +| Process CPU milliseconds before report | 1,101.12 | 665.03 | +| Peak process-family MiB | 391.13 | 127.72 | + +First samples exclude module import and fixture preparation. Partial-order first +samples follow four chain validations, so they are first use of that fixture +within an already running process. Warm medians pool the three subsequent samples +from each of three processes. CPU and peak RSS cover the combined workload; +capture elapsed additionally includes Flameox capture startup and overhead. +Host caches were not flushed. The partial-order fixture is slower after this +change; these results establish a retention tradeoff, not universal acceleration. + +The partial-order input is 165,590 JSON bytes, estimated at 52,783 tokens by tokenx +1.3.0; its parsed-answer SHA-256 is +`41d06e593813e497221c36793bf00d5fece7055081895b5f6c123ba6ad9a8227`. +Chain bytes, token estimate and answer digest remain unchanged. Actual agent +tokens, transport, catalog injection, model reasoning and provider costs remain +unmeasured. Both cohorts are component measurements, not complete investigations. + +Preserved evidence: +`e29086af99afd8e67cbbd92e43d960ea40f73b4d318ea0cbca60f493575462b8`; +analysis `086133516f2acf031032d357a3ddb1ac611ce4f12556a24065a2becd567c5f7b`. +The baseline module SHA-256 is +`725409ce84902125941251037a8a214ccc3c1eec577e484c164c361ca580d693`; +the candidate module SHA-256 is +`2109f1096406e9ecac8bbf4f7ae8abb5113404790a9382487285d2831fcddf26`. diff --git a/src/domain/process/processTraceSpecification.test.ts b/src/domain/process/processTraceSpecification.test.ts index 9484e9075..ab649d2aa 100644 --- a/src/domain/process/processTraceSpecification.test.ts +++ b/src/domain/process/processTraceSpecification.test.ts @@ -95,8 +95,54 @@ describe("process trace specification", () => { }); expect(result.language.kind).toBe("partial_order"); }); +}); - it("counts only undeclared pairs after transitive, not-before and unordered relations", () => { +describe("process trace ordering validation", () => { + it("uses a unique topological order for transitive declarations and conflicts", () => { + const events = ["a", "b", "c"].map((id) => ({ + id, + source: "lifecycle" as const, + exact: { marker: id }, + cardinality: { kind: "optional" as const }, + })); + const accepted = processTraceSpecificationSchema.safeParse({ + events, + language: { + kind: "partial_order", + happens_before: [ + { before: "a", after: "b" }, + { before: "b", after: "c" }, + ], + }, + }); + expect(accepted.success).toBe(true); + + const rejected = processTraceSpecificationSchema.safeParse({ + events, + language: { + kind: "partial_order", + happens_before: [ + { before: "a", after: "b" }, + { before: "b", after: "c" }, + ], + not_before: [{ event: "a", anchor: "c" }], + unordered_groups: [{ events: ["a", "b", "c"] }], + }, + }); + if (rejected.success) throw new Error("Expected ordered pair conflicts"); + expect(rejected.error.issues).toEqual([ + expect.objectContaining({ + path: ["language", "not_before", 0], + message: "not-before constraint conflicts with happens-before order", + }), + expect.objectContaining({ + path: ["language", "unordered_groups", 0], + message: "ordered events cannot also be declared unordered", + }), + ]); + }); + + it("counts only undeclared pairs across transitive and disconnected relations", () => { const events = ["a", "b", "c", "d"].map((id) => ({ id, source: "lifecycle" as const, @@ -150,7 +196,82 @@ describe("process trace specification", () => { }), ]); }); +}); +describe("malformed process trace declarations", () => { + it("preserves independent diagnostics when event ids are duplicated", () => { + const result = processTraceSpecificationSchema.safeParse({ + events: [ + { + id: "a", + source: "lifecycle", + exact: { fixture: "first" }, + cardinality: { kind: "optional" }, + }, + { + id: "a", + source: "lifecycle", + exact: { fixture: "second" }, + cardinality: { kind: "optional" }, + }, + ], + language: { + kind: "partial_order", + unordered_groups: [{ events: ["a", "b"] }], + }, + }); + expect(result.success).toBe(false); + if (result.success) throw new Error("Expected a duplicate event id"); + expect(result.error.issues).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + path: ["events", 1, "id"], + message: "event ids must be unique", + }), + expect.objectContaining({ + path: ["language", "unordered_groups", 0, "events", 1], + message: "unknown event id: b", + }), + ]), + ); + }); + + it("keeps reachability diagnostics for unknown referenced successors", () => { + const result = processTraceSpecificationSchema.safeParse({ + events: ["a", "b"].map((id) => ({ + id, + source: "lifecycle", + exact: { fixture: id }, + cardinality: { kind: "optional" }, + })), + language: { + kind: "partial_order", + happens_before: [{ before: "a", after: "missing" }], + not_before: [{ event: "a", anchor: "missing" }], + }, + }); + expect(result.success).toBe(false); + if (result.success) throw new Error("Expected an unknown event reference"); + expect(result.error.issues).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + path: ["language", "happens_before", 0, "after"], + message: "unknown event id: missing", + }), + expect.objectContaining({ + path: ["language", "not_before", 0, "anchor"], + message: "unknown event id: missing", + }), + expect.objectContaining({ + path: ["language", "not_before", 0], + message: "not-before constraint conflicts with happens-before order", + }), + ]), + ); + }); +}); + +describe("finite trace specification validation", () => { it("rejects invalid ignores and unsatisfiable variants", () => { const base = specification(); expect( diff --git a/src/domain/process/processTraceSpecification.ts b/src/domain/process/processTraceSpecification.ts index 8e4e6fca6..09c14cf10 100644 --- a/src/domain/process/processTraceSpecification.ts +++ b/src/domain/process/processTraceSpecification.ts @@ -105,7 +105,17 @@ export type ProcessTraceSpecification = z.infer< typeof specificationShapeSchema >; type PartialOrderLanguage = z.infer; -type Reachability = (origin: string) => ReadonlySet; + +type AcyclicGraphFacts = + | { + readonly kind: "total-order"; + readonly rank: ReadonlyMap; + } + | { + readonly kind: "partial-order"; + readonly reverseAdjacency: ReadonlyMap>; + }; +type GraphFacts = { readonly kind: "cyclic" } | AcyclicGraphFacts; export const canonicalTraceJson = (value: unknown): string => { const serialized = canonicalize(value); @@ -195,38 +205,28 @@ const validateNotBefore = ( return pairs; }; -const validateNotBeforeOrder = ( - language: PartialOrderLanguage, - context: z.RefinementCtx, - reachable: Reachability, -): void => { - for (const [index, constraint] of language.not_before.entries()) - if (reachable(constraint.event).has(constraint.anchor)) - context.addIssue({ - code: "custom", - message: "not-before constraint conflicts with happens-before order", - path: ["language", "not_before", index], - }); -}; - -const graphHasCycle = ( +const classifyGraph = ( ids: readonly string[], adjacency: ReadonlyMap>, -): boolean => { +): GraphFacts => { const incoming = new Map(ids.map((id) => [id, 0])); + let hasUnknownSuccessor = false; for (const successors of adjacency.values()) for (const successor of successors) { const count = incoming.get(successor); - if (count !== undefined) incoming.set(successor, count + 1); + if (count === undefined) hasUnknownSuccessor = true; + else incoming.set(successor, count + 1); } const ready = [...incoming].flatMap(([id, count]) => count === 0 ? [id] : [], ); - let examined = 0; + const order: string[] = []; + let unique = true; while (ready.length > 0) { + if (ready.length !== 1) unique = false; const current = ready.pop(); - if (current === undefined) break; - examined++; + if (current === undefined) continue; + order.push(current); for (const successor of adjacency.get(current) ?? []) { const count = incoming.get(successor); if (count === undefined) continue; @@ -234,50 +234,67 @@ const graphHasCycle = ( if (count === 1) ready.push(successor); } } - return examined !== incoming.size; + if (order.length !== incoming.size) return { kind: "cyclic" }; + if (unique && order.length === ids.length && !hasUnknownSuccessor) { + return { + kind: "total-order", + rank: new Map(order.map((id, index) => [id, index])), + }; + } + const reverseAdjacency = new Map( + [...incoming.keys()].map((id) => [id, new Set()]), + ); + for (const [before, successors] of adjacency) + for (const after of successors) reverseAdjacency.get(after)?.add(before); + return { kind: "partial-order", reverseAdjacency }; }; -const createReachability = ( +const collectReachable = ( + origin: string, adjacency: ReadonlyMap>, -): Reachability => { - const reachable = new Map>(); - return (origin) => { - const cached = reachable.get(origin); - if (cached !== undefined) return cached; - const seen = new Set(); - const pending = [...(adjacency.get(origin) ?? [])]; - while (pending.length > 0) { - const current = pending.pop(); - if (current === undefined || seen.has(current)) continue; - seen.add(current); - for (const successor of adjacency.get(current) ?? []) - pending.push(successor); - } - reachable.set(origin, seen); - return seen; - }; -}; - -const isOrdered = ( - reachable: Reachability, - left: string, - right: string, -): boolean => reachable(left).has(right) || reachable(right).has(left); - -type OrderingRelations = { - readonly reachable: Reachability; - readonly notBeforePairs: ReadonlySet; +): ReadonlySet => { + const seen = new Set(); + const pending = [...(adjacency.get(origin) ?? [])]; + while (pending.length > 0) { + const current = pending.pop(); + if (current === undefined || seen.has(current)) continue; + seen.add(current); + for (const successor of adjacency.get(current) ?? []) + pending.push(successor); + } + return seen; }; const validateUnorderedGroups = ( language: PartialOrderLanguage, context: z.RefinementCtx, known: ReadonlySet, -): ((left: string, right: string) => boolean) => { +): { + readonly groups: readonly ReadonlySet[]; + hasPair(left: string, right: string): boolean; + markDeclaredPairConflicts( + pairs: ReadonlySet, + conflicts: boolean[], + ): void; + markSharedGroups(left: string, right: string, conflicts: boolean[]): void; +} => { const memberships = new Map>(); + const groups: Set[] = []; + const repeatedMemberships = new Map>(); for (const [groupIndex, group] of language.unordered_groups.entries()) { const path = ["language", "unordered_groups", groupIndex]; - const uniqueIds = new Set(group.events); + const uniqueIds = new Set(); + const repeatedIds = new Set(); + for (const id of group.events) { + if (uniqueIds.has(id)) repeatedIds.add(id); + else uniqueIds.add(id); + } + groups.push(uniqueIds); + for (const id of repeatedIds) { + const groupIndexes = repeatedMemberships.get(id) ?? new Set(); + groupIndexes.add(groupIndex); + repeatedMemberships.set(id, groupIndexes); + } if (uniqueIds.size !== group.events.length) context.addIssue({ code: "custom", @@ -301,51 +318,121 @@ const validateUnorderedGroups = ( path, }); } - return (left, right) => { + const markSharedGroups = ( + left: string, + right: string, + conflicts: boolean[], + ): void => { + if (left === right) { + for (const groupIndex of repeatedMemberships.get(left) ?? []) + conflicts[groupIndex] = true; + return; + } const rightGroups = memberships.get(right); for (const group of memberships.get(left) ?? []) - if (rightGroups?.has(group) === true) return true; - return false; + if (rightGroups?.has(group) === true) conflicts[group] = true; + }; + return { + groups, + hasPair(left, right) { + if (left === right) return memberships.has(left); + const rightGroups = memberships.get(right); + for (const group of memberships.get(left) ?? []) + if (rightGroups?.has(group) === true) return true; + return false; + }, + markDeclaredPairConflicts(pairs, conflicts) { + for (const key of pairs) { + const separator = key.indexOf("\0"); + const left = key.slice(0, separator); + const right = key.slice(separator + 1); + markSharedGroups(left, right, conflicts); + } + }, + markSharedGroups, }; }; -const validateUnorderedOrder = ( +const validateTotalOrderRelations = ( language: PartialOrderLanguage, + graph: Extract, + unorderedGroups: ReturnType, + notBeforePairs: ReadonlySet, context: z.RefinementCtx, - relations: OrderingRelations, ): void => { - for (const [groupIndex, group] of language.unordered_groups.entries()) - for (const [leftIndex, leftId] of group.events.entries()) { - for ( - let rightIndex = leftIndex + 1; - rightIndex < group.events.length; - rightIndex++ - ) { - const rightId = group.events[rightIndex]; - if (rightId === undefined) continue; - const key = pairKey(leftId, rightId); - if ( - isOrdered(relations.reachable, leftId, rightId) || - relations.notBeforePairs.has(key) - ) - context.addIssue({ - code: "custom", - message: "ordered events cannot also be declared unordered", - path: ["language", "unordered_groups", groupIndex], - }); + for (const [index, constraint] of language.not_before.entries()) { + const before = graph.rank.get(constraint.event); + const after = graph.rank.get(constraint.anchor); + if (before !== undefined && after !== undefined && before < after) + context.addIssue({ + code: "custom", + message: "not-before constraint conflicts with happens-before order", + path: ["language", "not_before", index], + }); + } + + const conflicts = new Array(language.unordered_groups.length).fill( + false, + ); + for (const [groupIndex, group] of unorderedGroups.groups.entries()) { + let comparableEvents = 0; + for (const id of group) + if (graph.rank.has(id) && ++comparableEvents === 2) { + conflicts[groupIndex] = true; + break; } - } + } + unorderedGroups.markDeclaredPairConflicts(notBeforePairs, conflicts); + addUnorderedConflictIssues(conflicts, context); }; -const validateCompletePairDeclarations = ( +const addUnorderedConflictIssues = ( + conflicts: readonly boolean[], + context: z.RefinementCtx, +): void => { + for (const [groupIndex, conflict] of conflicts.entries()) + if (conflict) + context.addIssue({ + code: "custom", + message: "ordered events cannot also be declared unordered", + path: ["language", "unordered_groups", groupIndex], + }); +}; + +const validatePartialOrderRelations = ( ids: readonly string[], - relations: OrderingRelations, - unorderedPair: (left: string, right: string) => boolean, + language: PartialOrderLanguage, + adjacency: ReadonlyMap>, + graph: Extract, + unorderedGroups: ReturnType, + notBeforePairs: ReadonlySet, context: z.RefinementCtx, ): void => { let missingPairCount = 0; let firstMissingPair = ""; + const groupConflicts = new Array( + language.unordered_groups.length, + ).fill(false); + unorderedGroups.markDeclaredPairConflicts(notBeforePairs, groupConflicts); + const notBeforeConflictFlags = new Uint8Array(language.not_before.length); + const constraintsBySource = new Map(); + for (const [index, constraint] of language.not_before.entries()) { + const indexes = constraintsBySource.get(constraint.event) ?? []; + indexes.push(index); + constraintsBySource.set(constraint.event, indexes); + } + + // Keep only one origin's forward and backward closure live at a time. Pair + // iteration stays in declaration order so the count and first missing pair + // retain the established diagnostic semantics. for (const [leftIndex, leftId] of ids.entries()) { + const forward = collectReachable(leftId, adjacency); + const backward = collectReachable(leftId, graph.reverseAdjacency); + for (const constraintIndex of constraintsBySource.get(leftId) ?? []) { + const constraint = language.not_before[constraintIndex]; + if (constraint !== undefined && forward.has(constraint.anchor)) + notBeforeConflictFlags[constraintIndex] = 1; + } for ( let rightIndex = leftIndex + 1; rightIndex < ids.length; @@ -353,17 +440,32 @@ const validateCompletePairDeclarations = ( ) { const rightId = ids[rightIndex]; if (rightId === undefined) continue; - if (isOrdered(relations.reachable, leftId, rightId)) continue; + const sameEvent = leftId === rightId; + const ordered = + !sameEvent && (forward.has(rightId) || backward.has(rightId)); const key = pairKey(leftId, rightId); - if ( - !relations.notBeforePairs.has(key) && - !unorderedPair(leftId, rightId) - ) { + const notBefore = notBeforePairs.has(key); + const unordered = unorderedGroups.hasPair(leftId, rightId); + if (ordered || notBefore) { + if (!sameEvent) + unorderedGroups.markSharedGroups(leftId, rightId, groupConflicts); + } + if (!ordered && !notBefore && !unordered) { if (missingPairCount === 0) firstMissingPair = `${leftId}/${rightId}`; missingPairCount++; } } } + + for (const [index, conflict] of notBeforeConflictFlags.entries()) + if (conflict === 1) + context.addIssue({ + code: "custom", + message: "not-before constraint conflicts with happens-before order", + path: ["language", "not_before", index], + }); + addUnorderedConflictIssues(groupConflicts, context); + if (missingPairCount > 0) context.addIssue({ code: "custom", @@ -383,31 +485,43 @@ const validatePartialOrder = ( const ids = specification.events.map(({ id }) => id); const known = new Set(ids); const adjacency = buildAdjacency(specification.language, context, known); - const cyclic = graphHasCycle(ids, adjacency); - if (cyclic) + const graph = classifyGraph(ids, adjacency); + if (graph.kind === "cyclic") context.addIssue({ code: "custom", message: "happens-before graph must be acyclic", path: ["language"], }); - const reachable = createReachability(adjacency); const notBeforePairs = validateNotBefore( specification.language, context, known, ); - const relations = { reachable, notBeforePairs }; - const unorderedPair = validateUnorderedGroups( + const unorderedGroups = validateUnorderedGroups( specification.language, context, known, ); // Preserve independent declaration diagnostics on cyclic input. Computing // all pair relations cannot establish a valid partial order in that case. - if (!cyclic) { - validateNotBeforeOrder(specification.language, context, reachable); - validateUnorderedOrder(specification.language, context, relations); - validateCompletePairDeclarations(ids, relations, unorderedPair, context); + if (graph.kind === "total-order") { + validateTotalOrderRelations( + specification.language, + graph, + unorderedGroups, + notBeforePairs, + context, + ); + } else if (graph.kind === "partial-order") { + validatePartialOrderRelations( + ids, + specification.language, + adjacency, + graph, + unorderedGroups, + notBeforePairs, + context, + ); } for (const [kind, values] of [ ["prefix", specification.language.prefix],