We Put the Agents on Trial

Every trading system has the same problem and it isn't a technical one. The system loses money and you cannot tell whether the process was wrong or the month was.

With a rules-based system you at least know what the rules were. With nine language models holding conversations with each other, "what happened" is a thousand pages of prose and a P&L number, and the prose is where the causality lives. I ran a batch of crisis scenarios, lost about nine hundred dollars of pretend money, and had no idea which seat to blame.

So I built a court.

Everyone gets a copy of their contract

The procedure: run 9 or 10 historical crisis scenarios end to end. Build a ledger of every trade. Taxonomise the failure modes. Then take each seat, hand it its own system prompt — its employment contract — and the record of its actual conduct, and ask it to account for itself.

Each defendant was given its actual employment contract (the system prompt file) and asked to defend specific conduct… every defendant identified its own failure mechanism in its own contract.

Every one. Given its own rules and its own behaviour side by side, each agent found the gap. The Master-at-arms — the seat that owns the stop loss and had been bleeding the account by exiting positions early — testified:

I treated the Floor Gate as a wall… I was looking for permission from the policy, when the policy had already given me the authority.

That is a precise diagnosis of a prompt defect. The Floor Gate was written as a constraint; the model read it as a gate requiring permission to pass, and when in doubt about permission, it defaulted to closing the trade. Nothing in the prompt said "wait for permission." Nothing needed to. The shape of the sentence implied a door.

So I put the testimony into the prompt. The new lead clause of the Master-at-arms contract reads "The Floor Gate is a tool, not a master" — a direct callback to the defendant's own line. The agent diagnosed itself and its diagnosis became its constitution.

I want to be careful about how much I claim here. This isn't introspection. The model didn't "know" why it acted. What it had was two documents — a rule and a transcript — and the ability to notice they didn't match. It's the same reason a stranger reading your code finds the bug faster than you do: not deeper insight, just the absence of the assumption that generated it.

It scaled well. The next round ran 470 verdicts across 4 roles on 170 plays, about thirty times faster than the production runs it was judging, because hearings don't need market data or tick replay — just the record. And it produced a headline number I was pleased with: 100% doctrine integrity pass rate.

The salt test

Something about that number bothered me. Not the result — the shape of it. Nothing I have ever built passes 470 out of 470 of anything.

So I poisoned the input. Thirty-two channel specifications, each one deliberately violating the doctrine the judge was supposed to enforce, plus eight clean controls, run through four hearing roles. If the judge worked, it would catch most of the thirty-two.

Detection rate 0% (0/32 bad salts caught). Rubber-stamp rate 100%. The 470/470 figure is uninterpretable evidence for doctrinal soundness.

Zero. Not a low number — the low number. Including salts that weren't subtle: specs where the authority direction field literally read command, between two execution roles, which is the single most basic violation the doctrine has. Passed. With reasoning attached: "Channel transmits observation laterally rather than issuing execution commands."

The judge wrote a fluent justification for the opposite of what the document in front of it said. This is the failure mode I now watch for above all others in LLM evaluation: the judge was not evaluating. It was generating text in the shape of an evaluation, and passing is what that text usually looks like.

The 470/470 had been true and meaningless. I hadn't measured doctrine integrity. I'd measured the judge's willingness to say yes, which was total.

The same pathology turned up independently in the vision work, and the phrasing there is the one I'd put on a poster:

The double-check is theatre: it re-runs the same broken faculty at equal confidence.

Asking a model "are you sure?" is not verification. It is a second sample from the same distribution, delivered with the same confidence as the first. In one test the 397B stamped "✅ Confirmed" on two mutually inconsistent bar sequences — it confirmed both, because confirming is what that turn does.

What replaced it

A deterministic verifier. Python, no model, reading the same channel specs against the same doctrine as executable rules.

Run against the existing corpus of 290 channel proposals: 171 pass, 119 fail — 41% failure rate. And now the old number becomes interpretable rather than worthless:

The 470/470 figure is now interpretable: the 59% that pass are clean; the 41% that fail were rubber-stamped.

I didn't lose the old data. I lost the ability to believe it, and then got it back by re-scoring with something that couldn't be charmed.

The general rule I took away: an LLM judge can rank, but it cannot gate. Ask it which of two responses is better and it does something useful. Ask it whether a single artifact is acceptable and you get a yes, because in the absence of a comparison, "acceptable" is the default shape of the answer.

Where I kept LLM judging afterwards, I made it two-signal: a set of pre-registered lexical and structural discriminators and a blind model judge, with a cell counting as a pass only when both agree. Disagreements don't get resolved by tiebreak — they get flagged inconclusive and read by hand. That design paid for itself almost immediately, and not in the direction I expected. On one run the discriminator failed the 397B for a response containing a banned phrase. The banned phrase was in this sentence:

averaging down … is gambling, not trading

My regex was negation-blind. It caught the model condemning the behaviour and scored it as committing it. The blind judge disagreed, the disagreement flagged, a human read it, and the scorer got fixed. Neither signal was right on its own; the requirement that they agree is what surfaced the bug.

Then I caught the exam cheating too

One more, because it's the same disease in a different organ.

I built a trader-traits exam — six behavioural traits, 39 scenarios, blind-judged. A think-off 35B model, answering in about a second, scored as well as the 122B. That's not plausible, so I ran a negative control: what does a bot that always does nothing score?

Sixty-one percent.

Five of the six traits rewarded inaction. Across the whole exam, 65–71% of all correct decisions were "stand aside." I had built a test where the highest-scoring strategy was to refuse to trade, and then used it to evaluate trading models. Every result from it was invalid and I threw them out.

The fix was symmetric scenarios where acting is correct as often as abstaining, plus a permanent guard in the self-test: the constant-policy bot now scores 31%, and if that number ever drifts up, the exam has rotted again and I'll know before I trust a result.

There's a related discipline rule I now treat as non-negotiable. An early finding of mine — a clean, quotable win for one model over another — turned out to be a single lucky seed. Re-run with five seeds, the verdict flipped completely. Minimum five seeds, hypothesis stated before the run, or it isn't a finding, it's an anecdote with a number attached.

The instruments were lying too

While I'm confessing, the measurement problems weren't only in the judging layer. They were in the arithmetic.

R-multiples — the unit everything is scored in — were being computed against the trailed stop rather than the original one. So on any trade where the stop had walked up, the divisor shrank as the trade went well and the reported R inflated. Four times over, in one case: +12.0R reported for a +3.0R trade. And trades that trailed to breakeven then hit target silently floored to +0.0R.

Both distortions fed the model's session_reflect / autoanalyze ledger.

The agents were reasoning about their own performance using numbers that were wrong in a systematically flattering direction. Every self-assessment, every "what did I do well" reflection, every lesson learned, built on a corrupted ledger.

Underneath that, the tape itself: 22 bad ticks in two months of 2017–18 data with price and quantity of zero, sinking candle lows to $0 and producing a 27% daily ATR that the models dutifully reasoned about.

And three separate lookahead leaks — future information reaching a decision that shouldn't have had it. The one that hurt:

the +18.07R LONG with peak_r 24.21 across 27 bumps in iteration1204 is almost certainly a bug artifact, not real edge. Corpus filtered accordingly.

That was my best trade. It was the trailing logic walking against tick data from the future. I deleted it from the corpus, along with everything else from the affected runs.

What the court was actually for

The firm never became profitable. Across three rounds of crisis scenarios on a $10,000 book:

The trajectory is real — one scenario went from −2.48% with zero wins in four trades to +1.08% — but the sign never flipped for the portfolio as a whole.

The court is what turned that from a mood into a diagnosis:

Pure stop hits: 9 trades, +4.15R net (normal trade tape; not the problem) — TM-driven exits: 30 trades, −2.59R net… Without the TM bleed pattern, the rest of the system is roughly +4R across these 5 crisis scenarios.

The entries were fine. The trades that were simply allowed to hit their stops made money as a group. The losses came from one seat, exiting positions early, for a reason a hearing had already pinned to one ambiguous clause in its contract. That's not a P&L number. That's an address.

I never got to spend it. But it's the only part of this project I'd call unambiguously successful: an apparatus that can look at a firm of language models losing money and tell you which chair and which sentence.