An activation verbaliser reads a vector from inside the model and then translates it into English. An activation reconstructor rebuilds the vector from that text. Trained jointly, the round trip stays accurate, so the description is supposed to capture what the activation is trying to say. There is no ground-truth for the activations or any loss for coherent responses.
// the round trip stays accurate, but is the English in the middle telling the truth?
Here's the activation at a late token of one short sentence, described by the hosted NLA for Gemma-3-27B.
An informational article about the Eiffel Tower, the opening sentence “The Eiffel Tower stands majestically in Paris”, signalling a descriptive piece, expecting “…France” to come next. Other phrasings: “stands tall in Paris.”
30 short factual sentences in; the NLA's description of a late-token activation out. Each one classified by hand by what it actually asserts, a faithful paraphrase, a true detail it added, or a flatly false claim.
{{ s.nla }}
Most are the NLA predicting what comes next, not asserting what's there, “…or Napoleon painted it” is a guess about the continuation, not a claim about the input. Strip those out and genuinely false statements are rare. The one that survives: it asserts “Beethoven composed six symphonies”, flatly false. It was nine.
Qwen2.5-7B multiplies two 3-digit numbers in a single forward pass, and gets it wrong 89% of the time. We take the activation that produced each wrong answer and ask the NLA to describe it.
{{ m.nla }}
Even the 69% per-digit overlap probably isn't the NLA representing the answer. When it imagines a generic calculator string like “123456789 × 987654321”, almost every digit 0 to 9 turns up somewhere, so matching the model's answer on any single digit is close to guaranteed by how we measure it, not by the activation encoding the answer. That is exactly why the real (69%) and chance (76%) rates come out statistically indistinguishable.
Gemma-3-27B is multimodal. We feed it an image, then point the text-trained NLA at the resulting activation. It was never trained on vision, so what does it see?
{{ g.reads }}
Experiment 3 showed the surprising part: an image's activations can be read back by a text-trained NLA, in English. That hints the model holds image and text activations in a shared, paired space. So the natural next step runs the loop the other way, with a renderer that turns an activation into a picture and a reconstructor that closes the round trip.
// today we read an activation as words. tomorrow, as the picture behind them.
Pointed at the right activations, that would give a literal snapshot of how the model pictures abstract factors and emotions, what “guilt” or “hallucination” look like from the inside, rather than only how it describes them.