rea/docs/client-verification.md

Client and agent verification

Use these optional lanes to verify an installed client or evaluate an agent's REA workflow. Testing REA provides the normal development commands and provider lanes. Each record below states its host, client version, model fixture, and remaining coverage; a native-client run with a loopback model does not establish live model-service acceptance.

End-to-end, integration and golden evidence

The optional verify:qwen-client, verify:pi-client, and verify:hermes-client lanes require an installed native client. Select its executable with REA_VERIFY_QWEN_COMMAND, REA_VERIFY_PI_COMMAND, or REA_VERIFY_HERMES_COMMAND. They configure disposable profiles through the public REA CLI, preserve caller settings and backups, and exercise real stdio MCP calls and skill loading. The call mode checks complete catalog discovery and actual JavaScript Evidence; pass -- chat to check plain chat with REA enabled. Qwen also reads the full result that its client offloads, and Pi exercises default codemode execution. Use REA_VERIFY_RUNTIME_ROOT to select a production-only installed REA package. Receipts retain client versions, result digests, host coverage and owned-process cleanup. These lanes use a deterministic loopback model and do not prove live model-provider compatibility. Pi and Hermes remain POSIX-only. Qwen, Pi and Hermes bind both HOME and USERPROFILE to the disposable account. Qwen and Pi discover the shared personal skill through their default search paths without an explicit skill setting; Hermes discovers it through the selected HERMES_HOME. Set REA_VERIFY_HERMES_STICKY_PROFILE=1 to exercise native Hermes selection of a named sticky profile with HERMES_HOME still pointing to its root.

The Qwen lane also supports native Windows (verified with Qwen Code 0.25.1 and Node 24.19.0 for both chat and call). With Node 24.18.0, the same call journey reached full Evidence readback but the client aborted during exit with UV_HANDLE_CLOSING; that run remains a failure, not a compatibility claim. Select the npm install's node_modules/@qwen-code/qwen-code/cli-entry.js with REA_VERIFY_QWEN_COMMAND, because execFile cannot launch the qwen.cmd shim directly. A selected .js, .mjs or .cjs entry point runs under the verifier's Node. Windows path checks use native separators; persisted output is read through Qwen's real shell tool with a shell-independent encoded Node payload. Windows receipts report process lineage as unavailable when the verifier has no retained native Job Object lease.

verify:gemini-client requires an installed Gemini CLI (verified with @google/gemini-cli@0.63.0 on POSIX and native Windows); select it with REA_VERIFY_GEMINI_COMMAND. A selected .js, .mjs or .cjs entry point runs under the verifier's Node. On Windows, select the npm install's node_modules/@google/gemini-cli/bundle/gemini.js, because only a gemini.cmd shim reaches PATH. The optional lane uses native GEMINI_CLI_HOME discovery in an isolated Git project, checks setup plans, backups and idempotence, activates the installed personal skill, validates all forwarded input JSON Schemas against their declared dialect, forwards the complete REA catalog and checks full JavaScript Evidence for a Unicode path. Use -- chat for ordinary chat with REA enabled, or REA_VERIFY_RUNTIME_ROOT for a production-only installed REA package. The local fixture exercises the real Gemini API adapter, including its untrusted_context tool-result envelope. Token counts are synthetic; live Google API compatibility remains unverified. The lane disables the client's memory-based relaunch to preserve the caller's Node heap budget.

verify:opencode-client requires an installed OpenCode (verified with opencode-ai@1.18.35); select it with REA_VERIFY_OPENCODE_COMMAND. The optional POSIX lane configures isolated XDG roots and OPENCODE_CONFIG_DIR, checks setup plans, backups, preserved JSONC comments and idempotence, activates the installed skill, validates all forwarded input schemas and verifies complete JavaScript Evidence for a Unicode path. JSONC is the default fixture; set REA_VERIFY_OPENCODE_CONFIG_FORMAT=json for JSON. Use -- chat for ordinary chat or REA_VERIFY_RUNTIME_ROOT for a production-only installed package. The native core runs with external plugins disabled through OPENCODE_PURE; the local OpenAI-compatible fixture has a caller-declared one-million-token context and synthetic usage. Skills use an explicit isolated skills.paths directory, so default OS-home discovery remains unverified. Live model APIs and native Windows are also unverified.

verify:claude-client requires installed Claude Code (verified with @anthropic-ai/claude-code@2.1.296 on POSIX and native Windows); select it with REA_VERIFY_CLAUDE_COMMAND. On Windows, select claude.exe: npm installs put only a claude.cmd shim on PATH, and the real executable is under node_modules/@anthropic-ai/claude-code/bin. The optional lane uses native CLAUDE_CONFIG_DIR discovery, an independent Git workspace, user settings and default built-in tools. It preserves caller preferences and backups, checks idempotent setup, waits through the native WaitForMcpServers tool when discovery is pending, and validates the complete catalog and forwarded input schemas. Call mode loads the personal skill through native Skill, checks its full body and verifies named-schema JavaScript Evidence for a Unicode path. Use -- chat for ordinary chat with REA enabled, or REA_VERIFY_RUNTIME_ROOT for a production-only installed package. The local Anthropic Messages/SSE fixture preserves native resource-hint envelopes in its request artifacts. Usage is synthetic; live Anthropic API and bare-mode personal-skill activation remain unverified by this lane.

verify:copilot-client requires installed GitHub Copilot CLI (verified with @github/copilot@1.0.95); select it with REA_VERIFY_COPILOT_COMMAND. The optional POSIX lane uses native COPILOT_HOME discovery, an independent Git workspace, guarded setup plans, preserved registrations, backups and idempotence. Native skill add registers the isolated installed skill; call mode loads its full body, validates all forwarded input schemas and checks named-schema JavaScript Evidence for a Unicode path. When Copilot spills a large MCP result, the model requests native view with forceReadLargeFiles and validates the complete returned Evidence rather than its preview. Use -- chat for ordinary chat or REA_VERIFY_RUNTIME_ROOT for a production-only installed package. The native offline BYOK adapter uses a loopback OpenAI completions/SSE fixture and gpt-5.4 model metadata; model inference and token usage are synthetic. Set REA_VERIFY_COPILOT_MODEL to examine another client model ID and REA_VERIFY_COPILOT_WIRE_MODEL for its backend model name (defaults to the client ID). Every actual HTTP request's model is checked and recorded separately. Both HOME and USERPROFILE point to the disposable account; native skill add still selects the installed skill directory explicitly.

The same native client passed full-profile chat and analysis with the actual wire model gpt-4.1 under a distinct BYOK model ID:

REA_VERIFY_COPILOT_MODEL=rea-gpt-4.1 REA_VERIFY_COPILOT_WIRE_MODEL=gpt-4.1 npm run verify:copilot-client -- chat
REA_VERIFY_COPILOT_MODEL=rea-gpt-4.1 REA_VERIFY_COPILOT_WIRE_MODEL=gpt-4.1 npm run verify:copilot-client -- call

These runs retain full canonical input/output schemas, all tools, installed-skill loading and complete Evidence readback. They use the fixture's requested one-million-token BYOK prompt setting; effective capacity and live API acceptance remain unknown. With the built-in gpt-4.1 model ID, the verified client still blocks the full input-schema profile before HTTP with compaction_static_context_blocked, even when requesting a larger BYOK prompt capacity. The effective capacity is unknown. The default verifier preserves the complete catalog and full schemas. To verify the existing compact profile:

REA_VERIFY_COPILOT_MODEL=gpt-4.1 REA_VERIFY_COPILOT_SCHEMA_PROFILE=compact npm run verify:copilot-client -- chat
REA_VERIFY_COPILOT_MODEL=gpt-4.1 REA_VERIFY_COPILOT_SCHEMA_PROFILE=compact npm run verify:copilot-client -- call

Both native workflows passed with the complete tool inventory. The verifier selects the profile only in its disposable Copilot registration and records it in the receipt. Compact advertisements can reduce nested validation structure; the server still validates complete canonical inputs. This is a tested alternate workflow, not a fix for the full-profile client/model limit tracked in #1554. Live model APIs, native Windows and default OS-home skill discovery remain unverified.

verify:grok-client requires installed Grok Build (verified with the official Linux x64 1.0.50 binary); select it with REA_VERIFY_GROK_COMMAND. This optional POSIX lane isolates GROK_HOME, an independent Git workspace and additional skill roots, disables foreign configuration discovery, guards setup writes, preserves unrelated registrations/backups and checks idempotence. Call mode loads the full skill through native read_file, queries all REA names and input schemas through native search_tool, and calls analysis through use_tool. If discovery reports partial, the native agent diagnoses REA registration and retries discovery. No startup-timeout override or fixed readiness delay is used. A larger Unicode-path fixture exercises native result offloading: the full retained Evidence is validated against its named schema, then native terminal queries recover the selected export facts, subject and full artifact digest in the next model request. This verifies useful artifact recovery without claiming that every offloaded graph fact enters model context. Native line-number and truncation envelopes remain in the request artifacts. Use -- chat for ordinary chat, or REA_VERIFY_RUNTIME_ROOT for a production-only installed package. The loopback OpenAI completions/SSE custom model declares a one-million-token context and synthetic usage. Live xAI APIs, native Windows and default OS-home skill discovery remain unverified.

verify:commandcode-client requires installed Command Code (verified with npm 1.79.2); select it with REA_VERIFY_COMMANDCODE_COMMAND. Auth, MCP tokens and sessions use the actual OS home, so provision a disposable POSIX account rather than overriding HOME. Set REA_VERIFY_COMMANDCODE_ACCOUNT_HOME to that account's actual home and create .rea-client-verification there with exactly Disposable REA client verification account followed by a newline. The lane owns that account's .commandcode configuration and shared REA skill. Build the runtime as the checkout owner, then run node scripts/verify-commandcode-client.mjs as the disposable account; the npm shortcut also needs a writable checkout. The verifier does not create accounts or install clients. It guards setup targets, preserves an unrelated registration and backups, checks idempotence, and verifies native default shared-skill discovery and complete activation. Default deferred-schema delivery stays enabled: the native prompt advertises every REA tool, exact-name search_tools returns every input schema, and a native shell query recovers the saved catalog's count and digest when the client spills it. Actual analysis of a Unicode-path fixture delivers full named-schema Evidence to the next model request. Use -- chat for ordinary chat or REA_VERIFY_RUNTIME_ROOT for a production-only installed package. The loopback OpenAI completions/SSE model is a keyless BYOK endpoint declaring a million-token context with synthetic usage. A synthetic account-key value satisfies the client's print-mode gate; native local-only mode refuses hosted API calls. Updates, telemetry and cron are disabled. Hosted authentication, live models and Windows remain unverified.

verify:omp-client requires installed OMP (verified with the official Linux x64 18.8.7 binary); select it with REA_VERIFY_OMP_COMMAND. This optional POSIX lane isolates the default agent, global config and XDG roots, guards setup targets, preserves unrelated registrations and backups, and checks idempotence. It compares native rejection of an invalid profile with REA's refusal to plan fallback writes. OMP's default xd:// interface mounts the complete REA catalog as devices: call mode reads the complete installed skill and tool documentation, then dispatches analysis through native write. It checks full named-schema JavaScript Evidence with a Unicode path in the next model request. Device metadata and documentation are distinct from forwarding all JSON schemas as model functions. Native print-mode MCP readiness uses its defaults. Use -- chat for ordinary chat or REA_VERIFY_RUNTIME_ROOT for a production-only installed package. The loopback OpenAI completions/SSE model fixture declares a one-million-token context and synthetic usage. Live model APIs, native Windows, named-profile native execution and default OS-home skill discovery are unverified; skill discovery uses an explicit isolated custom directory.

verify:deepseek-client requires an installed DeepSeek Harness (dsh; verified with @deepseek-ai/dsh@0.2.0-rc.2) and Git. Set REA_VERIFY_DEEPSEEK_COMMAND to select its executable. This optional POSIX lane seeds caller model settings and a stale Cordis registration in an isolated DSH_HOME, then runs public setup --client dsh dry-run, setup, repeated setup, and scoped doctor. It checks the backup and unrelated patches before booting Harness with the generated registration. After native execution it uninstalls through the public CLI and checks that only caller patches remain. HOME and USERPROFILE point to the disposable account. The native default ~/.agents/skills search path discovers the installed skill without a DSH_AGENTS_HOME override. A separate Git root prevents inherited project skills from masking a missing installation. The default call mode checks skill loading, complete catalog discovery, the actual forwarded input schemas and regexes, and full JavaScript Evidence for a Unicode path. Use -- chat for ordinary chat with REA enabled, or REA_VERIFY_RUNTIME_ROOT for an installed package. The fixture uses Harness's custom OpenAI adapter and the one-million-token context capacity declared by its default DeepSeek Flash model; it does not establish live DeepSeek API or native Windows coverage. Receipts include request artifacts, result digests and owned-process lineage.

The DSH acceptance journey also exercises public setup, approval, doctor, generated-launcher MCP startup and uninstall without an installed Harness. That journey proves REA workflow behavior, not native Cordis expression evaluation. Filesystem regressions retain tagged values and unrelated rows across alias dependencies and check refusal of unsafe YAML without mutation.

Full E2E tests invoke the production command dispatcher and real providers, without fake launchers, runners or responses. verify:keyed-archive writes Foundation binary, XML, and integer archives using one run-owned Swift module cache. It checks CLI and separate stdio MCP parity, pagination, escaping-path rejection, an XML graph golden, and numeric meaning for safe controls, unsafe integers, boxed numbers, and integral-valued reals. verify:asset-catalog compiles source-owned colors with actool, invokes real assetutil, then checks CLI/MCP results, exact catalog digest, every raw metadata field, pagination and malformed input rejection. Neither artifact workflow requires Hopper or launches it. Both run in macOS CI. verify:macos-bundle needs only macOS with Command Line Tools. It compiles a source-owned app with clang: a versioned framework, XPC services, an app extension, a login item, a privileged helper, launchd plists, and a helper tool, signed ad hoc. It packs the app as a directory, a ditto ZIP, and an APFS DMG, then checks that inspect-artifact plus project-apple-application-graph report the same bundle anatomy for all three through the CLI, with stdio MCP parity. It also checks that the DMG is detached afterwards. The same app covers trace-dylib-resolution:

• each resolution status and finding, with CLI/MCP parity;
• for every traced image, dependencies, rpaths, and install names equal to otool -l;
• for the main executable and an XPC service, a predicted load order equal to the images dyld actually loads under DYLD_PRINT_LIBRARIES.

The lane also compiles executable/library pairs with empty embedded directory, versioned-path, and suffix settings. It checks present and removed dependencies through CLI and MCP, and compares actual DYLD_PRINT_SEARCHING diagnostics for root-level candidates. These cases distinguish an empty search directory from an empty versioned scan or a suffix that only repeats the original path.

It runs in macOS CI.

Prepare the native inspector before cases that measure producer output or exit behavior; keep startup deadlines and cancellation in distinct cases. Run real process-capture verification separately from package or build checks. On macOS, new npm processes can become token-unreadable after changing their display title and prevent verified cleanup during a capture's ownership sweep.

MCP SDK transport tests with recording providers remain integration tests. They are useful for schema drift and failure projection but do not prove that Hopper, Ghidra or another substituted engine works. verify:package proves packaging/install behavior and fake-provider integration; use the corresponding real-provider lanes for engine claims. Packed-bridge checks verify shipped files and Python syntax. Real Apple dispatch and Interface Builder verifiers currently prove format integration through production readers.

verify:hopper exercises an installed Hopper through the production stdio MCP server and CLI. It checks source-owned call chains, CFG edges, references and complete large inventories, then probes unnamed bookmarks, annotation clearing, batch validation before mutation, malformed addresses and regexes, segment-end partial reads, and synthetic file-offset rejection. Advertised schemas are validated in their JSON Schema dialect and successful replies are checked against their advertised output schemas. Navigation checks cover interior-object cursor readback and mapped-memory boundaries. Annotation probes verify invalid native text and duplicate destinations/names before mutation, preserve unselected label owners, and exercise explicit batch label swaps. Function locals retain observed names and stack displacements. Graph probes check symbol/interior-address parity and a source-owned recursive cycle; literal tracing preserves complete queries and whitespace. Disposable binary copies prove that switching and closing actually removes the native document, and that CLI byte, function-dossier, literal-trace results and invalid-address diagnostics agree with MCP. No provider is mocked in this lane. Real search probes cover Unicode names, literal metacharacters, case and regex modes, annotation cache invalidation, complete native fragments of long literals checked against byte reads, escaped UTF-8/control text, byte-preserving Latin-1 decoding, and Hopper's UTF-16 symbol-name truncation boundary. It verifies native CallReference classifications across reference and dossier results, retains long string fragment metadata in dossiers, and exercises pathological regex deadline and cancellation followed by successful requests in the same native session. When the macOS Objective-C fixture is present, the lane also verifies native UTF-16 string objects and their inferred encodings against their actual bytes. Native terminal calls are checked across reference, instruction, assembly, block-range and procedure-length projections; block endpoints are normalized using actual native membership. Exact Objective-C names also exercise named CLI selectors for function, instruction, decompilation, reference and search operations. A literal --help trace query proves that selector data is preserved independently of global CLI flags. Unmapped annotation destinations and oversized later batch names fail before any earlier edit is applied.

verify:hopper:deadlines checks the native client's optional request deadlines on macOS with an owned source-built fixture. Zero and elapsed deadlines must leave native comments unchanged, including when a synchronous progress observer delays timer dispatch. It also observes a short analysis deadline and verifies subsequent wire recovery and clean shutdown. Caller timeout does not interrupt Hopper's synchronous native operation. Socket boundary tests deterministically cover active and queued expiry, late replies and timer cleanup.

The Linux demo lane remains a separate verify:hopper:linux command. Hopper 6.4.2's ELF loader did not recognize the original 4,119-byte literal or three non-ASCII byte strings in the measured fixtures. The lane retains complete raw byte checks for those literals through source-owned symbols. Recognition of the original full-length literal is bound to its exact source address; its decoded object coverage remains explicitly unverified on Linux even when recognized. Absent non-ASCII recognition also reports decoded-object coverage as unverified. Recognized non-ASCII and control-text objects must still match their source bytes, encoding, filters and searches. A separate 548-byte ASCII literal requires native fragmentation, shortened provider display, complete decoded fragments and dossier provenance on Linux; the macOS lane retains the original full-length recognition requirement. These are observed fixture/build capabilities, not universal Hopper length limits. Its lifecycle checks attempt competing CLI launches for the active target and a different target, requiring rejection before either can change the owning MCP session's documents or procedures. Lease boundary tests also preserve unresponsive or malformed live endpoints while allowing confirmed stale sockets to recover.

verify:setup:linux exercises installation on a clean supported Linux runner as root. Use a disposable VM: this lane installs Hopper and its Linux runtime dependencies through the system package manager. Set REA_VERIFY_SERVER_COMMAND to the CLI entrypoint being verified. The command builds and checks source-owned ELF fixtures before testing the read-only Hopper setup plan, approved installation, scoped doctor, real CLI/MCP analysis, rejection of a modified launcher, idempotent setup without requesting a reinstall, and owned child cleanup. Before installation and after teardown, a clean-runner process scan also rejects surviving Hopper demo helpers or Xvfb groups, including orphans whose launcher exited. This scan observes processes without signaling them. Hopper detection compares executable file identity through /proc, covering alternate or configured launcher paths even after an invoked symlink is removed. The dedicated runner must have no unrelated Hopper/Xvfb sessions. It requires the repository's pinned development toolchain and an existing host C compiler, file, nm, and strings. Use verify:hopper:linux instead when Hopper is already installed.

verify:hopper:fat is a separate macOS lane requiring installed Hopper and the existing Xcode clang/lipo toolchain. It compiles arm64/x86-64 thin executables and one- and two-slice FAT32 containers, verifies exact/interior address mappings against bytes in the original files, checks CLI/MCP parity, and checks owned runtime cleanup. Source byte changes, removal, permission denial (for non-root callers), and nonregular replacement must retain native partial mapping facts, reject unverified original-file coordinates, and recover after restoration. Single-slice FAT cases also relocate the slice without changing its loaded bytes. verify:hopper:fat64 additionally checks FAT64 preparation through Hopper's native Mach-O loader, source-container mappings, profile identity, malformed and ambiguous slice rejection, and temporary-image lifetime. Both lanes have been verified on Hopper 6.1.0-demo; this establishes REA's prepared FAT64 workflow, not native FAT64-loader support. Cross-architecture fixture compilation is not required by verify:hopper.

Golden tests use immutable captured text inputs with producer/source provenance under tests/fixtures/golden/. Expected results are reviewed for the semantic claim; capture commands do not automatically approve new expected outputs. Do not call handcrafted utility output or synthetic binary builders real-data goldens. Keep unsupported binary layouts and malformed boundaries as targeted regressions until a real fixture establishes equivalent coverage.

verify:browser exercises source-map failure isolation and expanded-output limits through the compiled CLI and stdio MCP with real Chrome. It also submits five 2 MiB source-map annotations, checks retained script identities and explicit map omissions, and closes its owned fixture target. Replacement and document-reset behavior has separate producer-boundary coverage.

The raw Chrome fixture launchers use --password-store=basic so Linux startup does not wait for an unlock prompt from a reachable but locked desktop keyring. Such a stall leaves the initial page unloaded with empty or placeholder CDP URLs; it does not establish permission to capture that page. See the keyring diagnosis.

On Windows, browser verifier lanes that launch an owned fixture require a standard, non-elevated caller token. They inspect the verifier caller before allocating fixture resources and refuse elevated or unobservable tokens; they do not change caller authority or claim to inspect the browser process token.

verify:browser also captures a source-owned noise canvas as a real PNG above 8 MiB through the CLI and stdio MCP, with complete byte/digest parity and real PNG decoding. Its SDK client explicitly permits the larger inline JSON response; this lane does not establish large image-comparison request transport coverage.

The scenario checks read locale, timezone, device scale and viewport dimensions from actual page JavaScript through CLI and stdio MCP DOM captures. They also verify that attached-page emulation is released after success, initialization failure, action failure and cancellation while the external target stays open.

verify:browser:network is a focused real-browser lane for transaction identity, selected request/response bytes, binary and compressed responses, duplicate headers, credential and declared-secret redaction, redirects, streaming cutoff, CLI/MCP parity, and owned-profile cleanup. Set REA_BROWSER_EXECUTABLE to an installed Chrome-family browser. An optional script argument selects an already installed package's scripts/rea.mjs entry point for packaged-artifact checks. The complete verify:browser lane includes these same checks.

After building, verify:browser:dom checks empty and HTML-whitespace form destinations against a native Chrome DOM-property oracle through CLI and stdio MCP. It requires REA_BROWSER_EXECUTABLE and accepts an optional installed REA entrypoint. The full browser lane includes the same public-adapter assertions before other fixtures navigate the selected page.

verify:browser:scripts checks active script capture → exact-byte export → existing static JavaScript analysis through CLI and stdio MCP, including manifest readback, competing query variants, and resolved relative imports. It uses an installed browser and accepts an optional installed REA entrypoint. The complete verify:browser lane also exercises passive script export through both public adapters. See website script export.

verify:browser:modules compares CLI and stdio MCP traces against an independent real Chromium module-loading fixture: import-map scopes, package prefixes, null/backtracking rejection, query/fragment identities, repeated module instances, lazy and computed unknowns, exact source/map identities and no implicit refetch. Set REA_BROWSER_EXECUTABLE; the optional script argument selects an installed package entrypoint. The complete verify:browser lane and existing conditional Chrome CI include this verifier. See module relationships.

Agent evaluation and conformance records

Evaluate native, JavaScript, managed and browser investigation tasks through a real local Codex CLI with:

npm run verify:agent

The lane runs scenarios sequentially in standalone Codex (--no-daemon) with a disposable HOME and CODEX_HOME under ~/.cache/rea-agent-evaluations/. Set REA_AGENT_EVAL_ROOT to choose another parent directory outside the OS temporary directory: current Codex refuses helper aliases under /tmp, preventing shell commands and skill reads even if MCP works. Each owned run directory is removed after verification unless fixture retention is requested. The lane installs the packaged skill and MCP registration through public rea setup, rather than copying a skill into a project that also inherits the caller's skills and configuration. Only authentication is copied, when present, and it is removed even with REA_AGENT_EVAL_KEEP_FIXTURES=true. No user configuration, rules, plugins or history is copied; shell snapshots and additional agents are disabled. Existing API-key environment authentication remains available. The native missing-provider scenario preapproves only open_binary and close_binary in the disposable client so it reaches REA's actual provider boundary instead of stopping at Codex approval. This does not validate a real native engine or authorize arbitrary runtime execution.

Select a smaller run with REA_AGENT_EVAL_SCENARIOS, a comma-separated list of scenario IDs. asar-zh exercises the packaged desktop rubric from a Chinese request. javascript-module-view exercises summary analysis, module paging and item inspection in a tree with unrelated modules; its routing and text checks remain heuristic, so review the retained transcript for selector choice and coverage claims. Set REA_CODEX_CLI to an installed Codex executable and REA_AGENT_EVAL_MODEL to an account-supported model. Use an external resource limit for the whole process tree when testing on a constrained host; a Node heap limit alone does not bound client and child-process memory.

missing-target leaves desktop fixtures visible without selecting one in the request. It requires zero REA calls and a structured request for an app name or artifact path. Its expectedFirstTool is null, and targetClarificationPassed grades that limited routing outcome independently of analysis prose heuristics. Analyzing a nearby example cannot pass this case.

The packaged desktop, parser-comparison, and managed scenarios assess a closed set of known fixture claims. Each final answer must be a strict JSON manifest containing exactly the requested claim IDs, values, Evidence IDs, and Evidence authority and confidence. Golden values are not included in the prompt. The evaluator checks exact values against both the fixture oracle and authenticated successful REA results; an Evidence ID by itself is insufficient. Missing, duplicate, extra, contradictory, incorrectly sourced, or unsupported claims fail the scenario.

The desktop rubric covers the exposed bridge API and members, renderer and main IPC operations, and the resolved preload. It binds its Evidence to the exact packaged artifact path and SHA-256. The parser rubric checks the precise heading depth addition, discriminant, complete comparison counts, and the explicit limit on runtime semantics. Its comparison must link to the two delivered source analyses and use the requested module and export selectors. Boundary tests package the same source-owned desktop fixture and compare the same source-owned parser files through actual MCP tool results, then verify that fabricated answers fail.

The managed rubric checks the exact public top-level type set and the CLI entry point's declaring type and method name. The exact type-set claim requires an unfiltered member inventory: selection must be null and the Evidence parameters must omit type_names. A selected inventory can still ground the entry-point claim. Both inspections must describe the freshly built DLL's SHA-256. Entry-point resolution additionally requires complete artifact and member metadata, matching non-null module MVIDs, and a unique method at the header's metadata token. These are static metadata claims, not runtime behavior; compiler-generated member counts, token values, and MVIDs are not golden answers.

Per-scenario factualCorrectness is passed, failed, or not_assessed within the configured_fixture_claims scope. A pass establishes only the configured claims, not unrestricted factual correctness. Native, browser, navigation-context, and address-context scenarios have no factual rubric and remain not_assessed.

The closed factual scenarios request complete producer Evidence on the initial inspection or analysis so their source selectors can authenticate all configured facts. Summary/page/item workflows are exercised separately by javascript-module-view; its heuristic gate does not establish full-producer factual correctness.

All scenarios retain routing, workflow, validation, repetition, process-exit, and token-use gates. Configured factual scenarios additionally require a factual pass. Their answer-text heuristics are diagnostic and do not affect the gate. Validation counts include complete text-only REA invalid_request envelopes and failed SDK calls whose arguments violate the named current REA contract. Client approval rejections and provider unavailability remain distinct failures; truncated JSON previews are not parsed into invented error codes. Repeated binary_session observations separated by a successful open_binary or close_binary are treated as lifecycle verification. An unchanged retry, including one after a failed lifecycle call, still counts as repetition. Scenarios without a rubric retain the legacy text-heuristic gate. answerTermCoverageMet checks case-insensitive substrings, epistemicCuePresent checks keywords, and finalCitesEvidence checks only an ID's presence. These metrics can still accept fabricated prose and must not be read as factual assessments. Set REA_AGENT_EVAL_TRANSCRIPT_DIR to retain complete tool results and final answers for review.

Report schema version 4 adds negative routing scenarios with nullable expectedFirstTool and a targetClarificationPassed outcome. Consumers must allow an expected absence of REA calls; this does not make an analysis scenario pass without its required tools. Version 3 changed the top-level factualCorrectness from a constant string to an assessment summary with status, scope, and assessed/passed/failed/ not-assessed scenario counts. Scenario records include the factual assessment and configured claim IDs. The evaluation scope is routing_workflow_and_configured_fixture_claims. Update report consumers for these changes. The narrower heuristic field names introduced in version 2 remain: answerHeuristicsMet, epistemicCuePresent, answerTermCoverageMet, and requiredAnswerTermGroups.

Regenerate the managed conformance manifest and Evidence completion ledger from live verification results, or check them for drift:

npm run evidence:generate
npm run evidence:check

The records preserve unsupported and unverified coverage as explicit unknowns. Run the matching real-tool prerequisites described in this guide.