149 lines
10 KiB
Markdown
149 lines
10 KiB
Markdown
# Testing REA
|
|
|
|
REA's test suite follows one rule: use the lowest behavioral depth that can
|
|
prove a claim. Tests are grouped into native Vitest projects so ownership,
|
|
allowed dependencies, and runtime cost are visible from their paths.
|
|
|
|
## Behavioral depths
|
|
|
|
| Depth | Location | What it proves |
|
|
| ------------ | ----------------------- | ----------------------------------------------------------------------------------------------------- |
|
|
| Module | `src/**/*.test.ts` | One domain, service, or adapter through its narrow public surface |
|
|
| Composition | `tests/composition/**` | Provider-neutral session and registry wiring; filesystem access is limited to fixture materialization |
|
|
| Boundary | `tests/boundary/**` | Exactly one production filesystem, process, network, browser, CLI, or provider boundary |
|
|
| MCP boundary | `tests/boundary/mcp/**` | MCP transport and tool-contract boundaries with isolated sessions and explicit cleanup |
|
|
| Acceptance | `tests/acceptance/**` | A complete compiled CLI or MCP workflow |
|
|
| Conformance | `tests/conformance/**` | Shared provider contracts, parameterized by declared capabilities and explicit opt-outs |
|
|
| Evaluation | `tests/evaluation/**` | Deterministic evaluator parsing, scoring, and report generation |
|
|
|
|
Colocated domain tests may import only their owning domain and inward
|
|
dependencies. They do not construct sessions or servers and do not start
|
|
subprocesses or network listeners. Colocated application tests exercise one
|
|
service through explicit ports and recording adapters, never a CLI or MCP
|
|
entrypoint. Composition tests may assemble provider-neutral sessions and
|
|
registries but do not cross production filesystem, process, socket, or browser boundaries.
|
|
Boundary tests cross one production boundary. Only acceptance tests assemble
|
|
the complete runtime or invoke the compiled product surface.
|
|
|
|
Focused immutable builders and recording ports shared by one test family live
|
|
beside their production owner as `src/**/*.fixture.ts`. They are typechecked
|
|
with the suite and excluded from package builds; broader runtime and provider
|
|
fixtures remain under `tests/fixtures/**`.
|
|
|
|
`tests/process-global/**` is reserved for cases with a demonstrated dependency
|
|
on process-global state. Those tests run without file parallelism. Reusable,
|
|
test-scoped fixtures live under `tests/support/**`; immutable source artifacts
|
|
remain under `tests/fixtures/**`.
|
|
The process-global Vitest configuration contract rejects new direct temporary-root
|
|
creation outside the workspace seam and its narrowly documented boundary/package
|
|
exceptions.
|
|
|
|
Real Hopper, Ghidra, browser, package, managed-code, and controlled-replay
|
|
claims belong to their explicit `npm run verify:*` lanes. They are not inferred
|
|
from mocks or folded into the deterministic local gate. Real model trials are
|
|
manual; Vitest covers only deterministic evaluator logic.
|
|
|
|
## Real-toolchain verification lanes
|
|
|
|
Each real-toolchain command must require only the host tools needed to prove
|
|
its stated claim. Use a host-native fixture for host/provider acceptance, and
|
|
place optional cross-target formats or platform-specific runners in separate
|
|
commands. Check prerequisites before starting expensive work and name the
|
|
missing command, target, and lane in any failure message. A lane must not imply
|
|
that a host or target is covered when it was skipped.
|
|
|
|
| Ghidra lane | Supported runner/target | Additional local tools |
|
|
| ------------------------------------------ | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
|
|
| `npm run verify:ghidra` | Linux x64 with x86-64 ELF | Host C compiler, Ghidra 12.1.4, and full JDK 21 |
|
|
| `npm run verify:ghidra:aarch64-jump-table` | Any supported Ghidra host; AArch64 Linux ELF relocatable fixture | Clang with AArch64 target support, Ghidra 12.1.4, and full JDK 21 |
|
|
| `npm run verify:ghidra:cross-format` | Any supported Ghidra host; also analyzes AArch64 ELF, x86-64 PE, and x86-64 Mach-O | `clang`, LLD, and `lld-link` in addition to host-lane prerequisites |
|
|
| `npm run verify:ghidra:windows` | Controlled Windows x64 with native x86-64 PE | Ghidra 12.1.4, full JDK 21, and the Windows P0 fixture toolchain |
|
|
|
|
The cross-format Ghidra lane also analyzes an optimized AArch64 ELF switch
|
|
fixture. It checks the recovered case values against the source cases and
|
|
requires unresolved table bounds or case mappings to remain visible as
|
|
residual unknowns.
|
|
|
|
## Apple Interface Builder archives
|
|
|
|
`npm run verify:interface-builder` compiles the source-owned AppKit XIB into a
|
|
real `.nib` with Xcode `ibtool`, wraps it in a temporary app bundle, and checks
|
|
the decoded view hierarchy, outlet, action, evidence coverage, and truncation
|
|
status. Storyboard compilation additionally requires an installed iOS platform.
|
|
|
|
Keep the provider-specific acceptance path independent from optional
|
|
cross-compilers. Cross-format failures belong to the cross-format lane and must
|
|
not make native host acceptance unavailable.
|
|
|
|
## Developer commands
|
|
|
|
`npm test` runs every deterministic project once. The narrower feedback loops
|
|
are:
|
|
|
|
| Command | Scope |
|
|
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
|
|
| `npm run test:fast` | Domain, service, adapter, composition, boundary, MCP boundary, and conformance projects |
|
|
| `npm run test:local` | Changed and related tests without the build step; lightweight local feedback loop |
|
|
| `npm run test:boundary` | Boundary, MCP boundary, and process-global projects |
|
|
| `npm run test:mcp` | MCP boundary project only |
|
|
| `npm run test:acceptance` | Complete CLI and MCP acceptance workflows |
|
|
| `npm run test:changed` | Changed tests in non-serial projects via Vitest's import graph |
|
|
| `npm run test:watch` | Changed domain, service, adapter, composition, boundary, MCP boundary, conformance, and evaluation tests in watch mode |
|
|
| `npm run test:watch:all` | Changed tests from every deterministic project in watch mode |
|
|
| `npm run check:changed` | Cached static checks followed by changed tests |
|
|
| `npm run check:pr` | Static, generated-document, and complete deterministic PR gate |
|
|
|
|
Changed-test selection is a fast feedback aid, not release evidence. It can
|
|
miss behavior connected through runtime registration, generated data, shell
|
|
entrypoints, or other relationships that are absent from the import graph.
|
|
Use `npm run check:pr` before handing off a contribution.
|
|
|
|
Local full-suite Vitest runs are intentionally capped at one worker and one
|
|
project at a time. Boundary fixtures own real subprocesses, and this local cap
|
|
keeps aggregate memory predictable; CI retains the existing two-worker budget.
|
|
The pure domain/contracts and recording-port service projects share one worker
|
|
module context because their tests own no mutable runtime resources. MCP
|
|
boundary files also share the immutable server module graph while creating and
|
|
closing independent in-memory sessions. Adapter, composition, acceptance,
|
|
process-global, and other boundary projects retain per-file isolation.
|
|
`npm test`, `npm run docs:check`, and `npm run docs:generate` share
|
|
repository-local locks and fail fast when the same class of command is already
|
|
running. The `npm test` build is inside that lock. `check:pr` runs its test task
|
|
before starting documentation validation, so TypeDoc does not compete with the
|
|
full suite for memory.
|
|
|
|
Vitest and Node persistent compile caches are deliberately not enabled by
|
|
default. To evaluate repeated local runs, opt in for both cold and warm
|
|
measurements with an isolated cache:
|
|
|
|
```bash
|
|
NODE_COMPILE_CACHE=.cache/node-compile npm test
|
|
```
|
|
|
|
Do not report the warm result as a cold-suite improvement, and do not enable
|
|
the cache in coverage or benchmark CI without first showing that its
|
|
instrumentation remains equivalent.
|
|
|
|
## Coverage and timing
|
|
|
|
CI owns coverage. The aggregate floors are 65% statements, 60% branches, 60%
|
|
functions, and 68% lines. `src/domain/**` must reach 80% statements, 75%
|
|
branches, 75% functions, and 80% lines. `src/contracts/**` must reach 85%
|
|
statements, 80% branches, 80% functions, and 85% lines. Thresholds are
|
|
glob-specific rather than per-file and are never updated automatically.
|
|
Coverage does not replace named boundary, acceptance, or real-provider scenario
|
|
matrices.
|
|
|
|
CI runs four native Vitest shards without retries. Each shard emits a blob
|
|
report; the merge job produces aggregate coverage plus JUnit and JSON timing
|
|
reports, uploads them together, and writes the slowest files to the workflow
|
|
summary. Static checks, documentation, build, package verification, and
|
|
real-system lanes remain separate jobs so one kind of evidence cannot stand in
|
|
for another.
|
|
|
|
The PR acceptance target is a median `npm run check:pr` wall time below three
|
|
minutes across three warm-build runs on the benchmark host. Keep Vitest caches
|
|
cold unless separately identified. A PR that touches packaging or real-system
|
|
behavior also requires `npm run verify:package` and the applicable
|
|
`verify:*` lanes.
|