An s1a probe exit 0 can still flip every case

John Burns

AI, Systems, Automation

The order check exited 0. The totals line said the task fits, and the same line said every case flipped. File-order accuracy was 8 out of 8. The reordered answers were 0 out of 8. A script that stops at the exit code will treat a position-biased answer as a fit.

That run was not a call to TypeSafe, and it was not a claim about Jev 1.13.0. The binary was s1a probe from ThinkFlowLab/system1-agents at commit 0ee1165 (4 November 2026). The checkout’s pyproject.toml still says version 0.1.0. The changelog’s newest dated section is 0.3.0, dated 30 October 2026, which is where --order was added. The process ran on Linux x86_64 under Python 3.13.5, with openjiuwen 0.1.18 from the documented pin jj-0.1.0 (93f154e). No production key was present. A local HTTP stand-in answered the decisions shape the client posts, and it always returned the first key in criteria with a one-hot probability.

The cases file had eight lines. Each offered a, b, and c, in that order, and accept named a. The rules said to pick the first text. --order reverse was the only order flag. Stderr was empty. Stdout ended with this line:

order: reverse (seed 0)   flips: 8/8 (100%)   accuracy file order: 100%   reordered: 0/8 (0%)   median ms: 1   verdict: fits

Each case row said ok for the file-order answer a, then c under the reversed ask, then FLIP. The exit code was 0 because verdict was fits. The flip rate is printed. It is not an input to the exit code.

What the exit is allowed to mean

docs/skills.md already says the verdict and the exit code come from the file order, and that nothing about that verdict changes when you add --order. The threshold in probe.py is the same one the fit probe has always used: at least eight cases, and at least 80 percent of the file-order answers in accept. Below eight cases the verdict string is too few cases, which is not fits, so the process exits 1. At or above 80 percent it exits 0. A flip does not move either number.

That sentence in the skills page is easy to read as a footnote. The totals line makes it operational. fits and flips: 8/8 (100%) are not two summaries of one fact. One is the gate. The other is a measurement the gate does not read. If you are teaching a bounded decision skill and you keep the probe as the fitness check, the exit code answers only “did the file order score clear 80 percent on at least eight cases?” It does not answer “did the chosen key survive a change in option order?”

The stand-in makes that split obvious because it has no other behavior. It reads criteria in the order the JSON object arrived and returns that first key. On the file-order request the keys were a, b, c, so it returned a, which the file called correct. On the reverse request the keys were c, b, a, so it returned c, which the file called wrong. Sixteen requests left the stand-in, two per case. The state object and the rules text were the same on both asks. Only the key order changed. A content-based answer would have returned a both times. This one did not, and the process still exited 0.

A second mode of the same stand-in returned b whenever b was offered. Against a file whose accept was b, reverse exited 0 as well, with flips: 0/8 (0%) and reordered accuracy 8/8. That is the result you want from a content-based check. It is also the same exit code as the run that flipped every case. The exit code cannot tell those two runs apart. The totals line can, if you read flips and reordered and do not stop at verdict.

The control that stays quiet

--order same:2 is the control the changelog describes: N more asks in the file order, so a flip there is the backend disagreeing with itself rather than a change in position. On the first-key stand-in, same:2 with --seed 7 flipped 0 of 8. The reordered column was a,a on every row, all marked WRONG, because accept was b in that file. Twenty-four requests went out, three per case, and every body listed a, b, c. The position bias was still there. The control cannot see it, because the order never moved.

That is the comparison the order check is for, and it only works if you run both. Reverse, on a first-key answer, flipped 8 of 8. Same-order re-asks flipped none. If you run only same:N and the flip rate is zero, you have learned that this backend repeated itself. You have not learned that it reads the option text. If you run only reverse and the flip rate is zero, you still want the same-order control when the backend is a served model that might sample or sit behind a load balancer. This stand-in did not sample. Its reverse flip rate was the position effect, and its same-order flip rate was zero, on the same eight cases and the same rules.

--seed does not change a reverse run. Seed 1 and seed 99 both sent a, b, c and then c, b, a. Both totals lines printed their own seed. The help text says the seed is for shuffle:N. The reverse path still stores whatever seed you passed, or 0 when you omit it, and the render always prints it. A difference in that number, on a reverse run, is not evidence that a different order was asked.

Test the exit against the flip line

Before you treat a zero exit as a fit, read the totals line and keep the JSON the --json flag writes. On the exit-0 reverse run the JSON file was written even though a later, failing probe also writes that file. The write happens after the model calls return, and the exit code is chosen after that write. A usage error does not create the file. A fit and a failed verdict both can. The presence of the file means the asks finished. It does not mean the verdict was fits.

The field to keep is the per-case asks list, not only flip_rate. On the exit-0 reverse run, ask 0 listed a, b, c and choice a. Ask 1 listed c, b, a and choice c. That pair is the whole finding. A flip rate of 1.0 with file-order accuracy of 1.0 means every accepted answer was the first key, and every reversed answer was the new first key. That pattern is position bias that happens to agree with the file. It is the case the exit code is least able to reject.

Do not lower the case count to make the run faster and then trust a zero flip rate. A one-option file, reversed, sent two requests whose key lists were identical. The stand-in returned that only key both times. The totals line said flips: 0/1 (0%), file-order accuracy 100 percent, and verdict: too few cases: 1 of 8. The exit was 1 because of the case count, not because a flip was detected. There was no other order to ask. A zero flip rate on a one-key option list is the probe saying the order could not change.

Gates that fire before any request

s1a probe --model random never reaches that check. The probe’s --model choices are jev, clm, laya, laya-served, and cua. random is not among them. The process exits 2 and prints argparse’s invalid-choice error, including the usage block. s1a run ticket_router --help on the same checkout lists random and rule next to the HTTP models. The builder notes in docs/skills.md tell you to probe, then run --model random. Those are different subcommands. Putting random on probe does not give you an offline fit score. It gives you a usage error before the cases file is scored and before a key is required.

The usage errors that do apply to probe are not one gate, and they do not all run before the file is read.

--seed or --json without --order is checked first. On a path that does not exist, both flags still print one line, --seed and --json only apply with --order, and exit 2. They do not say the file is missing. A missing file plus a bad --order prints the missing-file error and exits 2. It does not say the order string is wrong. The same bad string, sideways, against a readable file prints --order must be reverse, shuffle:N or same:N with 1 <= N <= 50 and exits 2, with no model load and no request. If you are debugging a flags wrapper, the first error you see is the first check that failed, not a summary of every bad flag.

An empty file is not in that usage class. With no key set, probe on an empty file and on a file of only blank lines exits 1 with the model-service config error: no Jev key, set TYPESAFE_API_KEY or OPENROUTER_API_KEY. The file had no invalid case, so read_cases returned an empty list, and the client is constructed before the empty summary is rendered. With a non-production marker pointing at the stand-in, the same empty file exited 1 with flips: 0/0 (0%) and verdict: too few cases: 0 of 8. The stand-in recorded no request. A zero-over-zero flip rate here means no case was asked. It does not mean the order check passed. The key error and the “too few cases” line are the same empty file, depending on whether the client could be built.

A malformed line still stops before that. A file whose state is a list exits 2 with the line number and the required shape: state object, options as key to text, rules text, accept as a list of keys. A file whose first line is not JSON exits 2 naming that line. Neither requested a key, and neither contacted the stand-in. The skills page says invalid cases stop before any model call. That held for those two files. It did not hold for a file with no cases at all, which is valid JSONL in the sense that it has no bad line, and which still demands a key.

shuffle:51 and shuffle:0 are usage errors, exit 2, and they name the bound 1 <= N <= 50. shuffle:01 is not. isdigit accepts the leading zero, int reads it as 1, and the run proceeds. The totals line keeps the text you typed, order: shuffle:01, rather than rewriting it to shuffle:1. On the first-key stand-in with seed 3, that one extra ask flipped 6 of 8 cases. Two rows did not flip, because the permutation left a first. Reverse on the same stand-in and the same file shape flipped every row. One shuffle is a sample of other orders, not a proof that no order would move the answer. If you need the position check, reverse is the run that changes every key’s place when there are two or more keys. Keep shuffle:N for a seeded sample, and do not treat a leading zero as a rejected flag.

A repeated key fails on decide and disappears in the file

s1a decide rejects a repeated --option key before it loads a model. The same key twice, pay, exited 2 with --option repeats key 'pay'. No request was sent. That gate is on the flag parser in cli.py. It is not applied to a probe cases file.

A JSONL line with "pay" written twice is still one JSON object. The parser keeps the last value. The probe accepted the line, and the stand-in recorded a request whose keys were pay and human only. The first description never left the file. Reverse then flipped: file order chose pay, the reversed ask chose human. The row said FLIP. The verdict was too few cases: 1 of 8, so the exit was 1 for the case count. Nothing in the stderr said a key had been collapsed. If you generate cases from a dict and the generator emits a repeated key, decide will stop you and probe will score the collapsed object. The option text you thought you asked is not the text that was scored.

What to keep from a probe run

Run reverse and same:N on the same cases file, with at least eight cases and at least two keys in every option list. Read flips, file-order accuracy, and reordered accuracy. Exit 0 means the file-order score cleared the fit threshold. It does not mean the reversed asks agreed. Keep the --json file and check that each ask’s option list actually changed when you asked for reverse. If the lists are identical, the flip rate cannot show position bias, which is what a one-key case and an empty file both look like in different ways: one sends the same list twice, the other sends nothing and, once a key is configured, still prints 0/0.

Do not point this checkout’s probe at --model random and read a usage error as a failed fit. Random is a run model on this commit. Probe will not start until --model is one of the five names in its help, and the default is jev, which requires TYPESAFE_API_KEY or OPENROUTER_API_KEY before it will score even an empty file. Set the URL only when you mean to leave the machine. A TypeSafe key without TYPESAFE_API_URL selects the public TypeSafe host. This note used the URL override so the client stayed on the stand-in.

The fit line and the flip line answer different questions. On this checkout, only one of them chooses the exit code.

Sources

Previous post

A Grok sandbox exit 0 does not mean Landlock applied