shifathosn
/
rea
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix(tests): separate Copilot model IDs from wire models (#1647) 

* fix(tests): separate Copilot model IDs from wire models

* fix(tests): inspect parsed reference import diagnostics
This commit is contained in:
Ysopwnable 2026-10-11 06:52:45 +08:00 committed by GitHub
parent 60fbb2f06e
commit b09dd333ab
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
3 changed files with 39 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
docs/installation.md
View File

@ -153,6 +153,13 @@ listed after the table because its connector is not one of these files:
For GitHub Copilot CLI 1.0.95, `gpt-4.1` model metadata can block ordinary
chat with REA's full input-schema profile before sending a model request
(`compaction_static_context_blocked`; [#1554](https://github.com/morluto/rea/issues/1554)).
For the native offline OpenAI-compatible BYOK adapter, full-profile chat and
analysis also passed with a distinct `COPILOT_PROVIDER_MODEL_ID` such as
`rea-gpt-4.1`, `COPILOT_PROVIDER_WIRE_MODEL=gpt-4.1`, and that distinct ID selected
through `copilot --model rea-gpt-4.1`. The local fixture requested
`COPILOT_PROVIDER_MAX_PROMPT_TOKENS=1000000`; configure the limit supported by
your backend. This keeps every full schema and the actual wire model intact.
It does not establish the effective client budget or fix the built-in-ID path.
The existing compact profile passed native chat and analysis with the complete
tool inventory on Linux using a loopback model fixture. After setup, add
`"REA_MCP_INPUT_SCHEMA_PROFILE": "compact"` to the `env` object of the

21
docs/testing.md
View File

@ -160,8 +160,25 @@ Use `-- chat` for ordinary
chat or `REA_VERIFY_RUNTIME_ROOT` for a production-only installed package.
The native offline BYOK adapter uses a loopback OpenAI completions/SSE fixture
and `gpt-5.4` model metadata; model inference and token usage are synthetic.
Set `REA_VERIFY_COPILOT_MODEL` to examine another model configuration. In the
verified client, `gpt-4.1` blocks the full input-schema profile before HTTP with
Set `REA_VERIFY_COPILOT_MODEL` to examine another client model ID and
`REA_VERIFY_COPILOT_WIRE_MODEL` for its backend model name (defaults to the client
ID). Every actual HTTP request's model is checked and recorded separately.
Both `HOME` and `USERPROFILE` point to the disposable account; native `skill add`
still selects the installed skill directory explicitly.
The same native client passed full-profile chat and analysis with the actual
wire model `gpt-4.1` under a distinct BYOK model ID:
```bash
REA_VERIFY_COPILOT_MODEL=rea-gpt-4.1 REA_VERIFY_COPILOT_WIRE_MODEL=gpt-4.1 npm run verify:copilot-client -- chat
REA_VERIFY_COPILOT_MODEL=rea-gpt-4.1 REA_VERIFY_COPILOT_WIRE_MODEL=gpt-4.1 npm run verify:copilot-client -- call
```
These runs retain full canonical input/output schemas, all tools, installed-skill
loading and complete Evidence readback. They use the fixture's requested
one-million-token BYOK prompt setting; effective capacity and live API acceptance
remain unknown. With the built-in `gpt-4.1` model ID, the verified client still
blocks the full input-schema profile before HTTP with
`compaction_static_context_blocked`, even when requesting a larger BYOK prompt
capacity. The effective capacity is unknown. The default verifier preserves
the complete catalog and full schemas. To verify the existing compact profile:

15
scripts/verify-copilot-client.mjs
View File

@ -22,6 +22,7 @@ const runtimeRoot = process.env.REA_VERIFY_RUNTIME_ROOT ?? repo;
const command = process.env.REA_VERIFY_COPILOT_COMMAND ?? "copilot";
const mode = process.argv[2] ?? "call";
const model = process.env.REA_VERIFY_COPILOT_MODEL ?? "gpt-5.4";
const wireModel = process.env.REA_VERIFY_COPILOT_WIRE_MODEL ?? model;
const inputSchemaProfile =
process.env.REA_VERIFY_COPILOT_SCHEMA_PROFILE ?? "full";
assert(
@ -50,6 +51,7 @@ await exec("git", ["init", "--quiet", workspace], { timeout: 30_000 });
const environment = {
PATH: process.env.PATH,
NODE_OPTIONS: process.env.NODE_OPTIONS,
HOME: account,
USERPROFILE: account,
COPILOT_HOME: profile,
COPILOT_OFFLINE: "true",
@ -60,7 +62,7 @@ const environment = {
COPILOT_PROVIDER_BEARER_TOKEN: "",
COPILOT_PROVIDER_HEADERS: "",
COPILOT_PROVIDER_MODEL_ID: model,
COPILOT_PROVIDER_WIRE_MODEL: model,
COPILOT_PROVIDER_WIRE_MODEL: wireModel,
COPILOT_PROVIDER_MAX_PROMPT_TOKENS: "1000000",
COPILOT_PROVIDER_MAX_OUTPUT_TOKENS: "4096",
REA_PROCESS_RUN_ID: verifier.run_id,
@ -118,10 +120,17 @@ const extractEvidence = (value) => {
}
return undefined;
};
const wireModels = [];
const fixture = await createOpenAiModelFixture({
directory: lab,
prefix: "copilot",
onRequest: async ({ body, tools, requestIndex }) => {
assert.equal(
body.model,
wireModel,
"Copilot must send the selected wire model",
);
wireModels.push(body.model);
// Auxiliary requests without tools cannot execute the analyst's workflow.
if (tools.length === 0)
return { role: "assistant", content: "REA fixture title" };
@ -424,6 +433,7 @@ try {
clientVersion,
mode,
model,
wireModel,
inputSchemaProfile,
passed: failure === undefined,
artifacts: lab,
@ -440,7 +450,7 @@ try {
capacityBoundary:
inputSchemaProfile === "compact"
? "Explicit compact profile: complete inventory and canonical server validation; advertised schemas may be reduced. This does not fix the full-profile gpt-4.1 context boundary."
: "gpt-4.1 model configuration blocks the full catalog before HTTP even with a ten-million-token requested prompt override; gpt-5.4 is the verified full-profile local fixture configuration",
: "Complete canonical input and output schemas. Client model ID and wire model are recorded separately; effective token capacity and live provider compatibility remain unknown.",
catalogSize,
schemasChecked,
spillReadRequested,
@ -448,6 +458,7 @@ try {
skillLoaded,
artifactDigest,
finalSeen,
wireModels,
requests: fixture.requests,
probes: fixture.probes,
fixtureFailure: fixture.failure,