← Back to Blog

arXiv · 2026 · Preprint

MindAlign: giving brain signals a bridge to language

A two-stage approach to generating descriptions from brain activity, with a frozen language model.

Muxuan Liu

What if we could describe a silent thought?

When you look at a picture, you may silently describe one detail rather than the whole scene. MindAlign studies whether brain recordings can help recover that description. The task is controlled: participants view images and form Japanese descriptions silently during fMRI scanning. It is not unrestricted mind reading.

The idea: build a bridge in two steps

Brain recordings and language models speak very different numerical “languages.” MindAlign first learns a participant-specific mapping from brain activity to a compact representation of meaning—a semantic sketch. During training, image and language representations help guide this mapping.

The second stage turns that sketch, together with image features, into a short numerical prompt for LLaVA. The language model’s weights stay fixed; the connecting projection and fusion modules are trained. This separates learning to interpret a participant’s signals from learning how to guide the language model.

How we checked whether brain signals help

The study includes five participants. It compares brain-only input, brain-plus-image input, the original image-only LLaVA model, and a control that replaces brain input with random noise while retaining the image.

That last comparison matters: an image model can already produce a fluent caption. A useful brain-to-language system must show that the recorded brain signal contributes something beyond that visual information. Written descriptions collected after scanning provide the reference text.

What the results show

Brain-plus-image input generally improves the reported text-similarity scores over brain-only input. Real brain input also performs better than the random-input control in the subject-level results. For participant LD0006, BERTScore is 0.7800 with real brain signals and 0.7364 with random input (Table 2).

BERTScore compares representations of generated and reference text. A score of 0.78 is not 78% thought-reading accuracy. The results concern similarity in this task; they do not establish exact recovery of every word. Some generated descriptions still drift in meaning or introduce incorrect details.

Can the bridge be reused for another person?

The paper also pairs one participant’s first-stage encoder with another participant’s second-stage projector. The results suggest that the semantic-to-language part can be reused to some extent. The brain-to-semantics part still needs participant-specific training; this is not a system that works on an entirely new person without adaptation.

What to keep in mind

The main multimodal system receives the image as well as brain activity. The experiment uses a small group and a specific silent-description task, and the reference descriptions are reported after scanning. These conditions limit what we can infer about spontaneous inner speech.

MindAlign is an arXiv preprint. Its contribution is a modular way to connect limited brain data to an existing language model, with experiments testing what each input contributes. Code is not linked here because it is not yet available.