The Chartist's job was to look at a candlestick chart and describe its shape. A vision model, one PNG in, structural observations out — is this a rally or a range, where's the compression, is the recent action impulsive or corrective. The seats that trade can't see images, so the Chartist was their eyes.
I wanted to know how much the picture was worth. So I ran the same 30-question suite three ways: with the real chart, with the image removed entirely, and with the chart replaced by a cartoon duck.
In the production
D_png_v9_1Chartist view, the chart image contributes nothing. Swapping the real chart for a cartoon duck — or removing the image entirely — produces an identical Q1-Q30 score; the real chart is in fact a mild net-negative distractor (88/90 real vs 89/90 duck vs 89/90 no-image).
The vision model had not been reading the chart. It had been reading the text scaffold that accompanied the chart — the price series, the levels, the tape summary — and producing plausible chart-flavoured commentary on top of it. The image was decoration that I had spent weeks tuning, in a pipeline where a 27B vision model held a dedicated seat, on a dedicated machine.
The one question the real chart lost makes the mechanism unambiguous. On a rally scenario, duck and no-image both answer BULL correctly. With the real chart in context, the model flips to BEAR — "it read the pixels, they disagreed with the text scaffold, and it trusted the pixels." So it can see. It just sees badly enough that seeing hurts.
The instability is itself the proof
The companion finding is the part I'd actually hand to someone else, because it's a general test and it costs nothing:
A model reading pixels returns the same answer every time. Four different answers = the output is generated, not perceived. The instability is itself the proof.
Ask a vision model the same question about the same image four times. If it's perceiving, you get the same answer four times — the pixels didn't move. If you get four different answers, it isn't looking, it's sampling from a distribution of plausible answers about images like this one, and the variance is your evidence.
You don't need ground truth for this. You don't need a labelled benchmark. You need the same question twice and the discipline to notice the answers disagree. I had been running a single sample and reading the output for plausibility, which is exactly the failure mode the duck exposed: plausible is what these models produce whether or not they're grounded.
Two related results from the same stretch. The 397B was asked to confirm two mutually inconsistent bar sequences and stamped "✅ Confirmed" on both. And when a chart tool returned an ATR of exactly $0.00 for a below-threshold bar, the models read $0.00 as "low volatility" and traded on it. A null was interpreted as a measurement.
Can this model hold this seat?
The duck was one instance of the question the whole research repo existed to answer: given a seat with a job, a tool schema, and a one-minute deadline, which model can actually sit in it?
Not "which model is smarter." The seats have wildly different requirements. The Master-at-arms needs to wake often and decide fast. The Captain needs to hold six kinds of evidence at once and produce a judgment. The Chartist needs to describe a shape. Benchmark leaderboards tell you approximately nothing about any of that.
So I built the bench out of the job itself. Capture real production contexts — the actual prompt prefix a seat saw at a real decision point — and replay them against a candidate model. Score whether it calls the tool the production model called, whether it calls a tool at all, whether it truncates, and how long it took. Not correctness in the abstract: agreement, on the real distribution of situations that seat actually faces.
The headline result generalised further than I expected:
At 4-bit, tool-call discipline needs ≥~10B activated parameters.
A 35B mixture-of-experts with 3B active parameters, at 4-bit, fails: 12.6% of the time it emits
markdown prose instead of a tool call, plus 8–9% truncation. A 27B dense model — 27B active — holds.
A 122B MoE with 10B active holds. It isn't total parameter count and it isn't the quantization on its
own; it's how much active computation survives the compression. Structured output is the
first thing to go, and it goes silently: the model still writes something fluent, it just isn't a
tool call, and if you're not counting no_tool you'll never see it.
Some things that fell out of the sweep and surprised me:
- A 35% expert-pruned 264B model tied the full 397B trait-for-trait, at roughly four times the speed. The pruning removed capacity the trading seats never used.
- A 9B model at full FP16 was worse at tool discipline (33.6%
no_tool) than a 122B at 4-bit (0.84%). Precision is not the axis that matters here. - One 26B candidate returned HTTP 500 on 12.6% of chart-bearing contexts — not a quality problem, a deployability problem, and one you only find by replaying real contexts rather than clean benchmark prompts.
- Latency ordering inverted the size ordering constantly. A 27B dense at 63% agreement and 7.7 seconds lost its seat to a 122B at 62% and 2.0 seconds, because the seat's deadline is a candle, not a user's patience.
That last one is the whole discipline in one line. The document I keep coming back to is titled
time_is_the_dictator.md, and it was written after I asked myself what the actual
contribution of this work was. It gives the research-paper answer first, then knocks it over:
This framing is correct and useless.
He's trying to fit Opus-quality deliberation inside a 1m candle, on hardware he owns… That's not a research target. It's a real-time-systems target.
Quality per unit of wall-clock, on machines in my house. Once that's the objective function, half the model-selection literature stops applying and a bunch of unglamorous things — activated parameters, structured-output survival under quantization, p95 rather than p50 — become the only things that matter.
Two dials nobody talks about
While measuring seats I kept tripping over effects far larger than the model-choice effects I was actually studying.
Determinism is architecture times precision, and MoE loses. I wanted byte-identical reruns so I could tell a refactor from a regression. A dense 27B gives you that — identical output three times out of three, at full precision and at 4-bit. A mixture-of-experts at NVFP4 gives you zero out of three, diverging at around character 80 of the very first turn. It took eight commits in a single day to eliminate the alternative explanations one at a time — speculative decoding, mamba strategy, allreduce ordering, chunked prefill — before the answer settled: architecture decides whether you drift, precision only decides how fast. "Precision is the RATE dial." If you need to diff two runs to validate a change, that constrains your model choice before quality does.
Two words cut token usage by an order of magnitude. Prepending the literal string
"Decide quickly. " to a role prompt cut mean output tokens 12.8×, p99 by
42×, and took catastrophic blowups to zero. Not a rewrite — a prefix. Meanwhile the
same shared system prefix applied across seats cut Trade Manager output from 746 tokens to 385 and
raised Tactician from 1,404 to 1,608. There is no house style that's correct for every seat:
the register has to match the cognitive operation the seat performs. Defending a position and
integrating a market view are different jobs and they want different voices.
Both of these were bigger levers than any model swap I evaluated. I found them by accident, while looking for something else.
Discipline, or you'll fool yourself
Three rules I ended up enforcing on myself, each purchased with a wasted week.
Five seeds minimum, hypothesis pre-registered. One of my cleanest early findings — model A beats model B, quotable, tidy — was a single lucky seed. Re-run five times, the verdict flipped completely. Now: state what you expect before the run, five seeds minimum, or it's an anecdote with a number stapled to it.
Keep invalidated results, with banners. The project rule is that disproven hypotheses stay in the repo, labelled, rather than being deleted. Same for the model-behaviour notes: updated five times in a single day, always by appending dated annotations rather than rewriting — "The history of what we thought is itself the record." Deleting a wrong belief destroys the evidence of why you tried the thing you tried.
The best thing that rule ever did was let me retract myself. I published a finding that a 9B model "cannot apply rules" — 22% precision on a mechanical rule-following probe, a clean negative result. A month later I overturned it: same rules, same fixtures, 50 out of 50, once the model was allowed to reason before answering. My probe had forbidden reasoning in order to hit a latency and format target, and then I'd read the resulting failures as a statement about the model.
When a probe forbids reasoning to hit a latency/format SLO, its failures measure the envelope, not the model.
That correction only exists because the original was still sitting there to be corrected. Several findings in that directory carry banners like it. A research repo where nothing was ever wrong is a research repo that's been edited.
Run the negative control on your own instrument. Every measurement apparatus in this project that I bothered to poison turned out to be broken: the exam that scored a do-nothing bot at 61%, the judge that caught 0 of 32 deliberately-bad inputs, the vision pipeline that scored the same on a duck. Every single one passed casual inspection first. The apparatus is not exempt from the scepticism you apply to the results — it's where the scepticism pays best, because a broken instrument doesn't produce obviously wrong numbers. It produces encouraging ones.