34 KiB

Raw Blame History

Analysis efficiency measurements

Tracks the experiments for #2016. The production CLI/MCP contracts and defaults remain the compatibility baseline. Downloaded artifacts, runtime copies, exported Evidence, transcripts and detailed JSON measurements belong outside the checkout. This document keeps the measured findings, verification scope and revision references; the scripts and corpus.json keep the reproduction inputs. Generated results/ is ignored.

Follow-up: native PE/DOS entry views and real model verification records the post-merge native continuation, including a small-function case where four selected replies cost more bytes than the complete dossier.

The 12 detailed checkpoint receipts were archived byte-for-byte before removal from the final PR tree. The archive review-receipts-00d51e0fc-20261012 is in the private experiment lab outside the checkout; its archive-index.json records source revision, filenames, sizes and SHA-256. Raw producer replies, model transcripts and exported Evidence remain in the same private lab. These local files are not required to build or use REA. Historical checkpoint commits retain their original receipts; no history was rewritten.

Optimization priorities and gradual migration

Agent token consumption and end-to-end task time are the primary objectives. Memory and CPU consumption are also measured and controlled. Correctness, Evidence provenance, caller choices and existing-user compatibility are acceptance conditions for every optimization.

Tokens: minimize unnecessary model-visible data while returning the requested facts and useful provenance inline. Count discovery, repeated calls, errors, retries and follow-up turns; report total/cached/uncached input and output separately. Compact JSON bytes are a delivery measurement, not a token count.
Time: measure completion of the same analyst question, including startup, provider work, transport, client processing and model corrections. Separate measured stages and cold/warm state; use repeated comparable trials before generalizing an observed latency improvement.
Memory: record aggregate process-tree peaks and limit/cleanup events, account for retained Evidence and representation expansion, and keep host safeguards.
CPU: record aggregate CPU time for the completed task. A one-CPU quota is an experiment control; it does not itself establish reduced CPU consumption.

Migrate domain by domain with focused real-artifact checks and relevant shared contract checks. The domain matrix is a growing roadmap; native PE and DOS, for example, remain unverified in this study. A passed lane does not imply all advertised platforms/providers have passed. Newly discovered REA bugs remain in the separate local handoff document for later team repair.

Old clients using existing CLI/MCP requests against the candidate retain the complete default path. New summary/view options require explicit selection. Actual old Evidence import/view/export checks preserve original authenticated records and identities. These checks establish the recorded baseline/workflow scope, not every historical release or third-party client. A new client targeting an older server must discover the actual schema and use a supported equivalent where available; this study does not implement a universal automatic fallback. The catalog intentionally gains more precise/additive schemas, so its bytes and digest are not frozen: measured runtime 259e99dd9 has about 2.6% larger complete catalog JSON than the frozen baseline. The separate post-integration receipt records the newer runtime, including upstream changes. Optional views can reduce model-visible result data; old complete requests do not silently switch delivery or acquire that reduction automatically.

Reproduce a measurement

Use the repository's pinned Node/npm and locked dependencies. Build each selected revision, then copy its dist/, bridge/, scripts/, skills/, package.json and package-lock.json to a separate runtime directory before editing production sources. That copy needs the matching locked dependencies. The initial frozen baseline is 42b0ed126df7d5b908ac369715378d5e519051b7; this study does not silently substitute a newer main or the previous Code Mode experiment.

The public artifacts and acquisition digests are in corpus.json. Supply their actual local paths and an optional JSON object of explicit provider environment settings. Do not add credentials to that file. Provider settings are inherited by the child, but the report persists only the selected settings' digest. Runs reject undeclared inherited REA/Hopper and Ghidra/Java configuration keys before starting a measured server or creating a model profile; include those settings explicitly in --providers. Both drivers use provider-environment.json and record its digest. The system/runtime environment still needs comparable controlled conditions; its contents are not serialized. Both drivers also fingerprint the effective provider PATH without recording its value, and the SDK comparator rejects different or unrecorded search paths. This detects different search configuration, not replacement of executable bytes at the same path. Historical receipts predate this guard, so their selected-settings digest alone does not certify that no undeclared provider setting was inherited.

export NODE_OPTIONS=--max-old-space-size=512
export UV_THREADPOOL_SIZE=1
export GOMAXPROCS=1
export GOMEMLIMIT=2GiB
python3 experiments/analysis-efficiency/resource-run.py -- \
  node experiments/analysis-efficiency/measure.mjs \
  --runtime-root /absolute/frozen-runtime \
  --revision 42b0ed126df7d5b908ac369715378d5e519051b7 \
  --artifact obsidian.asar --path /absolute/obsidian-1.14.4.asar \
  --providers /absolute/selected-providers.json \
  --output /absolute/new-measurement-directory

The wrapper requires Linux cgroup v2 and a working user systemd manager. It owns a unique transient scope, restricts the whole process tree to one CPU, caps memory at 4 GiB with no swap and 192 tasks, and samples memory/tasks/CPU every 100 ms. It stops its own scope above 3 GiB or below 4 GiB host available memory; startup requires 8 GiB available. Cleanup failures retain the owned scope's identity. Run heavy builds, providers and model trials serially. Sampled peaks can miss short spikes; cgroup limit events are separately reported.

An actual Codex + Chromium trial exceeded the initial 192-task allowance and was stopped automatically below 1 GiB sampled memory, with no OOM. The isolated browser model retry uses resource-run.py --task-ceiling 256 -- COMMAND... to account for both process trees; memory, swap and CPU controls are unchanged. Keep the original aborted run and cleanup receipt. This is experiment capacity, not a REA defect or a successful browser analysis.

What the numbers mean

summary.json records the selected artifact's before/after SHA-256, runtime file-table digest, dispatcher and lockfile digests, runner source digest, Node/SDK versions, catalog sizes, caller arguments and producer Evidence IDs. runtime-files.json identifies the exact compiled files measured. Actual provider settings and runtime files must stay fixed during a comparison.

Every tool call measures three intervals at the public SDK Transport boundary:

Client work before send, including first-use schema preparation.
Transport/server roundtrip, including framing, I/O, server work and client JSON parsing. This interval is not a Ghidra-only or provider-only timer.
Client work after the parsed response reaches onmessage.

The SDK call's total excludes the runner's subsequent encoding and disk writes. Result byte counts measure compact JSON of the parsed client result, not raw wire bytes or tokens. Progress notifications retain their actual arrival times; they do not imply more precise internal engine timings than the producer emits. Scope CPU time and aggregate process-tree memory come from the wrapper's final JSON line. Fresh processes have cold application caches; host page caches are not globally flushed.

--delivery summary is an explicit candidate workflow, initially for artifact inventory. It must be supported by the selected server's advertised schema. The default complete runs the existing calls unchanged. Compare the same analyst facts and complete source Evidence before crediting a smaller response. Projection can reduce delivery bytes without reducing upstream analysis time. For WASM, the runner validates a summary request against the actual advertised 2020-12 input schema, including SDK-generated references. Supporting servers return the first-call summary; older servers retain their complete path and use summary/export views after an observed transport constraint. Original errors remain in the receipts, and the complete source Evidence ID is a compared fact. --javascript-question selected analyzes only the exact Obsidian sources package.json and main.js, keeps inventory identity, and reads each observed main.js module by its exact node ID. Package metadata is not a module selector.

Compare paired runs with the same runner source and provider settings:

node experiments/analysis-efficiency/compare.mjs \
  --left /absolute/complete-run --right /absolute/summary-run \
  --output /absolute/new-comparison.json

This checks all selected analyst facts, including inventory source IDs, exact path records, manifest, counts, completeness, contradictions, coverage, unexplored branches and limitations. It excludes only delivery observation IDs and the named recovery-path description. Different settings, changed facts or failed workflows make the comparator fail. Binwalk's per-invocation region UUIDs are separately preserved with both original values and JSON paths; only those opaque lookup values are excluded from physical-region fact equality. Original producer replies are never rewritten. Selected JavaScript comparisons likewise preserve both values of the two named parent-heap-dependent byte budgets before excluding their values from fact equality; fixed limits and coverage rows remain checked. Counts include extra view calls and observed tool errors. Use repeated runs with alternating order before claiming a distribution or a speedup; one pair is only one observation.

For a completed WASM result exceeding the MCP response budget, the runner keeps the original error, exports retained Evidence through the public tool, and authenticates the exact reported record using the selected runtime's parser. The report names this recovery; it does not relabel the oversized inline reply as successful delivery.

Initial observations and exclusions

The frozen baseline has 2,326,609 bytes of complete catalog JSON, of which 494,207 bytes describe names, descriptions and input schemas. One Obsidian trial spent about 21.9 s before the first request was sent, then about 0.77 s in that request's transport/server roundtrip. Other cold runs differed substantially. These single observations identify a measurement boundary; they do not establish a speedup or a distribution.

The initial matrix exercised shipped Obsidian, NewPipe, ShareX, OpenWrt, SQLite WASM and jq artifacts. SQLite produced a complete result larger than the transport frame and requires an explicit recovery run. A first jq attempt hit the SDK's default 60 s timeout because the experimental runner used a superseded three-argument call signature. SDK 2.3.1 accepts request options as argument two; the corrected jq run completed real Ghidra analysis in about 265 s. That earlier driver timeout is excluded from REA regression claims. Provisional runs with different runner source digests are not one paired statistical cohort.

No cross-session engine database cache, default analyzer changes, external plugin loading, or broad plugin SDK is enabled by this experiment.

Obsidian delivery checkpoint

The two recorded pairs compare frozen baseline 42b0ed12 and candidate f1106181 in complete → summary and summary → complete order using the same runner source, bytes and provider settings. Both preserve every selected fact and the atomic inventory Evidence ID.

Measurement	Complete	Summary + exact path view
Tool-result compact JSON, including lifecycle calls	937,404 bytes	16,322 bytes
Tool calls per question	3	4
Complete catalog compact JSON	2,326,609 bytes	2,370,505 bytes
Workflow elapsed, first pair	11.04 s	10.23 s
Workflow elapsed, second pair	10.57 s	10.18 s

Reply JSON falls by about 98.3%; catalog JSON grows by about 1.9%. The two elapsed observations do not establish a statistical speedup. Inventory coverage and upstream byte acquisition are unchanged. The candidate remains opt-in; existing requests continue to deliver the complete inspection. Each run verifies the selected artifact's before/after digest and reports no observed OOM or task limit events. Source typing and projection improve selected delivery without introducing an engine database cache or an external plugin protocol.

Task-scoped analyzer experiment

The recorded jq pair explicitly disables Decompiler Switch Analysis through the existing REA_GHIDRA_DISABLED_ANALYZERS setting and compares it with the unchanged Ghidra default. Both use the same frozen runtime and real jq bytes, one CPU, a fresh ephemeral database and the same entry/main question. Completed main dossiers exceed the MCP frame in both runs; existing retained native views recover the selected facts, preserving the observed transport errors in the reports.

One disabled/default pair takes 174.77/276.23 seconds. Entry and main pseudocode, main strings and its block count agree. However, callee 0x40daa0 has 604 recovered body bytes in eight ranges by default, and 164 bytes in four ranges with the same 646-byte span after disabling switch analysis. Equal top-level counts do not establish call-graph equivalence. Keep the production default. Use the existing control only for a caller-selected task whose required facts have been checked; no general fast preset is introduced. A separate real coreutils/Ghidra probe rejects a nonexistent analyzer before bridge readiness, preserves the named failure and closes the session. Source bytes remain unchanged.

--schemas selected is a separate SDK 2.3.1 client experiment. It passes the exact live named tools/list definition to the SDK's public callTool option, retaining output validation. It does not change REA or the default runner mode. Selected definitions compile per call rather than preparing the entire cached catalog; report this client effect separately from engine analysis or model behavior.

Real Codex CLI trials

agent-trial.py accepts an explicit frozen runtime and revision, the same question templates in questions/, and optional explicit provider settings. It fingerprints the actual runtime, runs pinned local Codex CLI with a private temporary profile, copies only the authentication file with mode 0600, disables shell/web/multi-agent tools, and removes copied profile state afterward. Raw events and answers remain outside Git. A zero exit code is not a factual grade. Review the original tool calls, failed requests, result limitations and answer.

The optional --enabled-tools FILE supplies a list of exact names to the temporary Codex MCP configuration's enabled_tools, as described in the official MCP configuration. It leaves the REA catalog and other clients intact. Initial Obsidian trials showed parameter-correction attempts and different delivery choices even with the same question; filtering alone has not established a cost or quality win. Requested model aliases are recorded without inventing a resolved model identity.

The browser question uses --authority browser to permit only its explicitly selected public scenario in a fresh, owned headless profile. Browser observations and cleanup are graded separately from static artifact analysis.

Real artifact compatibility checkpoint

Six frozen baseline/pilot pairs exercise Obsidian, NewPipe, ShareX, OpenWrt, SQLite WASM and published historical WETH code through public MCP tools. All selected facts agree under the stated comparison rules. Obsidian complete delivery and atomic inventory Evidence identity remain unchanged. ShareX native-boundary claims remain scoped to the selected DLL; NewPipe retains a null DEX descriptor rather than inventing one.

SQLite's complete 15,382,128-code-unit WAT exceeds the MCP response budget in both revisions. The control recovers complete Evidence through file export; the pilot reads summary and four exact export forms inline, without running WABT again. Inline byte totals exclude file exports, so they do not establish a size win. OpenWrt comparison preserves both sets of upstream-generated region UUIDs before comparing offsets, extents, signatures, coverage and limitations. WETH is a published historical example; a live RPC request returned HTTP 403, so this is not live-chain verification. Every pair records unchanged target bytes and zero observed OOM/task-limit events. The Linux x64 results do not establish behavior for IDA, Hopper, macOS, Windows or Harmony.

Same-question model and selected-source follow-up

Two actual SQLite model trials preserve all 12 requested facts. The question, developer instruction, provider settings and tool allowlist hashes match. A description-only follow-up supplies the exact summary kind and its fields. Calls change from 42 to 3, workflow time from 292.64 to 63.79 seconds, and reported input tokens from 304,698 to 74,387. The follow-up reads no unnecessary WAT pages. Both keep the original completed analysis's transport error and authenticate the retained source/view identities. This is one stochastic model pair, not a statistical or general causal claim.

The selected JavaScript pair preserves all 24 observed main.js module identities and the exact two-source selection. The strict comparison initially differs on parent budget values derived from observed heap availability; both values and paths remain recorded separately. Other coverage facts, per-family rows, fixed worker limits, omitted/unknown counts, statistics and Electron observations agree. Coverage remains partial, with 36 unselected relevant files and 384 semantic unknowns. These static facts do not establish runtime behavior. The comparator does not rewrite Evidence.

Ten real CLI trial reviews cover native, managed, Android, firmware, Electron/JavaScript, WASM, EVM and browser tasks. The selected JavaScript model preserves core facts and partial coverage but omits full budget/family provenance. The browser producer captures complete DOM/accessibility/URL evidence and terminates its owned process; direct model answers lose two accessibility metadata fields to client context clipping and report them unknown. A requested under-development Codex code-mode configuration still produces a direct MCP call in the actual trace, so no executor/composition win is credited. Profile cleanup, original errors, actual calls, usage, facts and scope limits remain independently reviewed. Enabling a configuration flag does not prove that a capability was used.

Discovered REA defects are recorded separately for team repair at the user's request. This follow-up performs no additional REA bug fixes.

Nonempty managed boundaries and final verification

A second shipped managed library checks 11 ModuleRefs, 147 PInvoke declarations and 225 native implementations from ShareX HelpersLib 19.0.2. All complete normalized artifact and native-boundary fields match between baseline and pilot, including source Evidence identities. This covers nonempty declarations beyond the main ShareX DLL's empty import tables. It does not verify native exports, execute the assembly or resolve CLR dependencies. The curated receipt keeps identities, coverage, all ModuleRefs, the first three PInvoke records and digests of the complete boundary payloads. Full producer tables and comparison inputs stay in the private lab outputs.

The earlier verification includes 42 real WABT CLI/MCP cases and the complete real SQLite CLI JSON: the unchanged default returns the same original source Evidence ID, normalized result and raw result as the baseline exported MCP record. Source bytes are unchanged. The seven-file contract/composition run passes 51/52 assertions and hits the existing 20-second presentation-test deadline while sharing one CPU. That same unmodified assertion passes alone in 10.835 seconds. The original combined failure and isolated pass remain recorded; no test deadline or REA bug was patched. Build, type/lint/module checks and generated-document checks pass. No full check:pr or unverified provider/platform coverage is implied.

Actual old Evidence migration

The migration experiment imports the original baseline SQLite Evidence bundle into the pilot runtime, reads a typed summary and four export forms, and exports the retained bundle again. The complete old WASM record is deeply equal after import/export, with its original authenticated Evidence ID. The original baseline Obsidian complete inspection also works as inline input to the new inventory summary, retaining the old parent and atomic inventory identities and manifest. Original input files remain byte-identical.

The runtime explicitly selects an absent WABT directory and requests no producer operation. These views consume existing Evidence without another provider run; this is portable-result reuse through existing import/export, not an engine database cache. The five-call workflow completes in 5.458 seconds with a sampled aggregate peak of 700,542,976 bytes and no observed limit or OOM events.

To repeat it with genuine old public outputs:

python3 experiments/analysis-efficiency/resource-run.py -- \
  node experiments/analysis-efficiency/verify-migration.mjs \
  --runtime-root /absolute/frozen-pilot-runtime \
  --revision 259e99dd906290189217cf0fb1a19769bcf87101 \
  --bundle /absolute/baseline-exported-evidence.json \
  --inspection-reply /absolute/baseline-complete-inspection-reply.json \
  --output /absolute/new-migration-directory

PR integration verification

The separate post-integration verification pins runtime 0eb42a1eb8ab17aed04240174cb77612e733a1d8, integrated with main c22111eca2c4bc84a084f12711f405b3e7422a40. Conflict resolution preserves main's HAR API-surface views, portable capture admission, exact layout/JavaScript summary carriers and existing CLI tests alongside inventory/WASM views. Earlier real-model observations keep their original runtime fingerprints.

Build, the complete check static gate and generated documentation checks pass. Twelve focused files pass 123 assertions, including the presentation test run separately with its original deadline. Actual package creation, private-prefix installation and 42 real WABT CLI/MCP cases pass. No package is published. The complete generic package verifier and full local check:pr were not run.

Genuine old SQLite/Obsidian Evidence migration passes again with WABT unavailable. A fresh public SDK cohort preserves the original Obsidian default result size (937,404 compact JSON bytes), atomic inventory identity and every selected fact; explicit summary plus exact paths still returns 16,322 bytes. Complete catalog JSON is 2,359,082 bytes versus the original baseline's 2,326,609 (about 1.4% larger). That difference includes intervening main changes and does not isolate this PR's catalog contribution. One cohort is not a latency distribution.

Recorded integration scopes have no OOM/OOM-kill or hard-limit abort. The full source-check scope reports three memory.high events near its 3 GiB soft threshold. The subsequent cold pre-push type/lint scope reports 37 such events, with 3,221,155,840 sampled peak bytes and no hard-limit/OOM event. Heavy lanes stay serial with the existing CPU, memory and no-swap controls. No further REA bug repairs are included; separate local findings remain for team follow-up.

CI attribution

Review of the actual main and PR job logs separates two PR-related description/test mismatches from pre-existing main failures. The complete-graph wording check now accepts capitalization changes; the existing effects audit records why returning WASM execution/network metadata is not foreign execution. Those two files pass 11 focused assertions without production changes. Main's own CI already fails the same live prompt-completion assertion and Darwin packaged-completion checks; the affected implementation and fixture files are byte-identical here. Those failures remain unchanged for team review at the maintainer's request. CI failures are not all infrastructure faults, and the failed CI gate is not presented as passed.

Failure	Main `c22111eca`	Reviewed PR `8d6f7eb1c`
Empty prompt completion after open	main shard	PR shard
Packaged prompt completion, Darwin arm64	main package	PR package
Packaged prompt completion, Darwin x64	main package	PR package

src/server/promptCompletion.ts, src/application/binary/BinarySession.ts, tests/boundary/mcp/promptCompletion.test.ts and scripts/verify-package-prompts.mjs are byte-identical to that main revision. Main commit 07b36c148946f46b99a4ce04e55726e97d0ee03a introduces ready-provider-only completion; the failing fixtures expect populated completion immediately after opening. Whether those expectations or the readiness behavior should change is left for team review. Commit 00d51e0fcd1d1bba60bf56281725e82d525a27ae synchronizes only the two PR-related description checks.

Frozen runtime references

Review follow-up

Two substantive bot findings were reproduced and addressed before merge. Atomic inventory projections now reuse the owning subject/digest/format validation. Actual public SDK calls with original Obsidian data demonstrated the old acceptance of null subjects and inconsistent formats; the corrected runtime rejects both with invalid_request. Six boundary assertions also cover missing subject, wrong format and wrong digest across summary and node views.

The related three-file run passes 37 assertions, and full static/docs gates pass. Genuine baseline SQLite/Obsidian import/view/export still preserves whole original records and source/view identities with WABT unavailable. A new real Obsidian summary workflow preserves every selected fact, atomic source identity and 16,322 delivered JSON bytes; catalog bytes remain 2,359,082.

Both experimental drivers reject undeclared inherited provider controls before server/profile creation and log no values. Explicit settings still override the corresponding inherited values. This prevents claiming equal declared settings while silently inheriting a different analysis option; it does not certify every aspect of the inherited system/runtime environment. These are research-driver controls, with no production configuration changes.

The source/static scope records 98 memory.high events, no hard limit/OOM event, and a sampled peak of 3,221,209,088 bytes. The real migration/Obsidian scope peaks at 677,367,808 bytes with no limit/OOM events. All raw receipts remain outside Git.

A later WASM finding was also reproduced: portable sources with absent subject or a different subject path could report another artifact path. The producer and view now share an exact path/digest identity rule. Actual WABT inspection of the published VS Code Tree-sitter INI grammar (0.3.1, 4,716 bytes, SHA-256 1e8cdf3027209100fa54a43d0faed69721ae1116428164c8ad6dcc496a9aa6af) supplied the unchanged report for public SDK before/after checks. The old runtime accepts both inconsistencies; the corrected runtime rejects both with invalid_request and WABT unavailable. Its 26 focused assertions and type/lint checks pass, and genuine old records still round-trip unchanged. The verification scope peaks at 3,204,509,696 bytes with no observed limit/OOM event. The initial CSS producer exceeded the MCP reply budget as expected; that failed probe is preserved separately and is not credited as a completed identity reproduction.

These revisions identify different experiment stages; model measurements were not rerun or relabeled as post-integration measurements.

Stage	Source revision
Original baseline	`42b0ed126df7d5b908ac369715378d5e519051b7`
Inventory delivery pairs	`f110618184e99f89fa776bc5e59703a61baa7c57`
Initial WASM pilot	`0002caa07ffd7319672a21ce901dbfbc55e21aac`
Exact WASM summary guidance	`259e99dd906290189217cf0fb1a19769bcf87101`
Main-integrated package/migration verification	`0eb42a1eb8ab17aed04240174cb77612e733a1d8`

The baseline runtime file-table SHA-256 is 3a1765a6fec0031db0b288112d4db9bc0dd503c9c3aa0946f07ce57f7673ab4b; the integrated runtime file-table SHA-256 is e99d3bdc851822b77e7d5bfaa9b60c57e7b99bd66e27a739bdea2e9300560d5f. The integrated lockfile SHA-256 is 0a55042141899a442a39dce3d49755c65a861f86d1930bd4f9c6d89cd53f5eba. Actual old migration inputs have SHA-256 e1ff3d28801436bcc11693f79cc847a4f31234f49d88449f9e0d77215c3765fc (SQLite bundle) and 7cecd07012bb79f3afad2ea9e70357e056e9eab48c1aa13bbbd310cdcb43eb9f (Obsidian complete inspection reply). The source WASM identity ev_16b9f6e0fcefcab9603f14fd5e63ea00c4749accd94fef4a4e9fbb260d2e0140 and atomic inventory identity ev_22c17e52489f3ec0e7298a62da37bd985dbbec202d7208a0bb1c20d863790aed remain unchanged after the recorded import/view/export workflows.

Complete carrier identity and provider search configuration

The outer inspect_artifact carrier now uses the same owning subject/digest/format validator as its atomic inventory. Invalid imported/inline carriers are rejected before selected-view construction; the generic historical Evidence parser and valid complete records are unchanged. Six additional boundary assertions cover missing subject, wrong format and wrong digest across summary/node projections. The negative checks use the small keyed-archive workflow so the original large ZIP retention/capacity workflow keeps its existing deadline and distinct coverage. An initial large-workflow placement timed out and is retained as a failed attempt. The three public MCP workflows, build, fast static gate and generated docs pass. The final static/test scope observed one soft memory.high event, with no hard memory, task-limit, OOM or OOM-kill event; it is not a performance trial.

Both research drivers fingerprint the effective provider PATH, including an explicit caller override, without serializing its value or the ambient environment. The SDK comparator rejects absent or unequal search-path fingerprints. This records search configuration, not executable replacement at an unchanged path; provider/runtime identities and controlled host conditions still matter.

Actual public SDK calls using the original shipped Obsidian inspection reproduce accepted null/wrong-format outer subjects on frozen 2ef0467bb and invalid_request on the corrected compiled runtime. Two fresh Obsidian replays with different inherited search paths preserve selected facts and source identities, but the comparator rejects matching settings with only provider_search_path_sha256 different. The original records and failed/accepted replies stay in private lab receipts; no environment value is copied into them.

Resource-counter closeout qualification

The experimental wrapper now attempts a final cgroup event read before waiting or cleanup. Disappeared/incomplete counters or still-owned descendants make resource_counters_verified false; last observations remain available separately. The model grader treats missing final counters as unverified, never as zero, and any recorded soft/hard/task-limit event prevents an unrestricted resource pass. Lightweight parser probes cover missing/incomplete and nonzero counters; an actual short command confirms disappearance is reported as unknown. Regrading one already-correct SQLite model answer preserves its facts but rejects the old receipt's absent final resource counters. Earlier receipts record last-sampled observations, not verified final counters. Their bytes, facts and token reports remain original observations; resource-qualified performance conclusions need this additional closeout evidence. No analyzer/build/schema/package behavior is changed, so unrelated verification is not repeated for this research-driver fix.

The next bounded WASM delivery cohort is documented in first-call summary measurements.

34 KiB Raw Blame History