Can an EEG model answer questions asked in words?

• ID · question number: a one-hot code for each question the head was trained on. It cannot ask anything new.
• TPL · label template: a sentence frame filled with the label’s name, turned into a vector by a frozen text encoder.
• DESC · description: a sentence that describes the state without naming it, turned into a vector the same way.
• SHUF · shuffled templates: the template vectors attached to the wrong questions during training, by derangements declared in advance: a control for whether the head uses the link between a wording and its question.
• NUM · numeric code: for SSVEP, the frequency given as numbers, not words: a control without text.
• B-sh · one output per question: the same hidden layer with a fixed output per seen question, as in route 2; it cannot ask anything new either.
• CCA · no training: for SSVEP, canonical correlation analysis, which needs no training and can name any frequency: the reference an unseen frequency is held to.

A training frame for sleep, filled with a stage’s name: The scorer marked this 30-second epoch as N2 sleep.

SSVEP on BETA: a plain spectrum (L0) and frozen CBraMod features (L1). Sleep on BOAS: frozen CBraMod features (L1). Three training seeds each. The primary text encoder is bge-small-en-v1.5, frozen. The head is the same for every arm: standardised features, a small hidden layer adjusted by the question vector (FiLM), one output.

Three kinds of result, never pooled into one zero-shot number: a seen question in the wording the head trained with; the same question reworded; and a question the head never trained on. “Unseen” always means unseen by the EEG head, not by the text encoder: the frozen encoder has read the words, and for SSVEP every token of a held-out frequency also occurs in that rotation’s training texts. Unseen sleep questions are combinations of seen stages; unseen SSVEP questions are flicker frequencies held out in five rotations.

Each comparison is the language arm minus the reference, in percentage points (pp), with a 95% interval from resampling people, then training seeds, then held-out wordings. It gets two readings, both fixed before any fit. A difference is shown when the interval excludes zero. Against a margin of ±2 pp, it is equivalent when the whole interval lies inside, non-inferior when the bound against the language arm stays within the margin, and otherwise the margin is not met, which is not the same as a loss of 2 pp or more. For an unseen sleep question the comparison is R, the head’s remaining error (1 − AUROC) over that of the read-off from its own stage answers, with a margin of R < 1.25. The check of whether the head uses the link between wording and stage, the template arm against the shuffled templates, is a difference of AUROC. Among the primary comparisons, those with the numeric code, the neighbour average and the shuffled templates are read for a difference only; the secondary tables print both readings as the run computed them.

There are 35 pre-declared primary comparisons and no multiplicity correction, so at 95% about one in twenty comparisons with no true difference may show one by chance. Accuracy is averaged per person, then over people: it compares set-ups and is not a deployment error rate.

This is route 3 of the research plan on When not to act. “Jev-style” here means the interface of a decision model like TypeSafe’s Jev, applied to EEG: encode the recording once, then answer several explicit, typed questions about it, each with a probability. The plan started from a vision paper, Yu & Yao, 2026 · Visual Jev: Accurate and Efficient Decisions from Shared Visual Context ↗, which applies that idea to images. All three routes: Jev-style questions on EEG: the evidence →

In the training wording (P1)

The seen SSVEP task is hard for every head on these features: with question numbers the head reached 32.1% (28.8%–35.2%) on a plain spectrum and 24.2% on frozen CBraMod features, against a chance of 3.1%; CCA, which needs no training, reached 65.5%. On sleep every arm reached about the same balanced accuracy, 71.5% with question numbers against a chance of 20.0%. The levels below are descriptive: overlapping intervals are not ranked, and the paired differences above are the comparison.

Reworded (P2)

The same seen questions in wordings the head never trained with: a new sentence frame (a1, held-out frames resampled), a synonym of the label (a2: three per question, fixed, so these entries hold only for these three synonyms), or a new description (a3, held-out descriptions resampled). Reworded rows are never pooled with the unseen ones.

On sleep a new sentence frame cost nothing measurable: equivalent within ±2 pp (−0.46 pp, −1.33 to +0.16 pp). The three synonyms per stage scored −12.60 pp (−13.74 to −11.43 pp) and new descriptions −20.87 pp (−29.49 to −13.71 pp), a wide interval, because descriptions differ a lot from one another. On BETA every rewording scored lower, from −1.33 pp to −4.13 pp; only the synonyms on frozen CBraMod features stayed within the margin.

Can the head answer them? (P3)

The implicit names ranked windows in reverse: AUROC 0.317 for “asleep” and 0.244 for “light sleep”, below 0.5, so windows that really were asleep, or in light sleep, scored lower on these questions than the others. The explicit names did better (0.858 and 0.642) but stayed short of the read-off (0.980 and 0.922). Descriptions have no alignment control: the shuffled control is built from template vectors. Held-out sentence frames (S13, in the secondary panel) give the same result for all five template cells.

Is the link between wording and stage used? (P5)

The shuffled-template arm attaches the same template vectors to the wrong stages during training; the unseen questions keep their true vectors. If the head uses the link between a wording and its stage, the shuffled arm should do worse on them than the template arm.

For explicit names, yes: attaching the templates to the wrong stages lowered the unseen AUROC, TPL − SHUF +0.629 (+0.603 to +0.654). For the implicit names it was not shown: the shuffled templates scored higher, −0.378 (−0.501 to −0.253).

In the neighbour 3-way the unseen frequency competes with the trained detectors of its two neighbours (seen bias, stated). On a plain spectrum the template head fell below the mean chance of that test, 19.3% against 34.2%, and showed no difference against the numeric code. On frozen CBraMod features it beat the numeric code by +15.61 pp, but those features keep the stimulus phase and neighbouring targets differ in phase, so that comparison tests frequency and phase together. The plain spectrum, which carries no phase, is the clean frequency test.

Per-frequency and interior-only breakdowns were computed after the audits, and no independent audit covers them, so they are not published.

Why they failed, the reading revision 5 was built to test: a head trained on shuffled labels is still a random function of sleep features that are strongly organised by stage, so on new people its scores lean one way for everyone, up or down. A person bootstrap measures the spread between people, not between heads, so its 99.9% interval was too narrow for this question. The failures fit that reading: all were on BOAS, and one was below chance, which a leak cannot produce. It is not a proof that no leak exists: person-disjoint folds, no held-out label or wording in training and training-only scaling are asserted in the code, and both conformance audits check for leakage directly.

Decision S0b-6, approved by the owner: the SSVEP token rule says every token of a held-out frequency occurs in that rotation’s training texts. It stops the run for bge-small-en-v1.5 and all-MiniLM-L6-v2, and both passed in all 5 rotations. It cannot hold for multilingual-e5-small, whose tokenizer keeps one-decimal number pieces as single tokens while the held-out number itself is barred from training texts; for that encoder the rule was reported instead of stopping the run, and every Chinese-wording unseen-frequency entry says the frequency is a new token.

Engineering amendment EA-1, before the first fit: in the contiguous extrapolation band (S8) the neighbour average is not defined and the neighbour 3-way is relabelled, because its competitors are themselves held-out questions; failed fits are marked per cell, and none failed; the secondary scoring can resume after an interruption. No question, partition, wording, arm, metric, margin or gate changed.

The declared numeracy prediction was not activated. It said: if the primary text encoder orders frequencies poorly (Spearman ρ below 0.3 between frequency distance and embedding distance, for every training frame), the 8-way TPL − NUM flag will not read “TPL higher”. Its premise was false: ρ was 0.593, 0.563 and 0.526 for the three training frames of bge-small-en-v1.5, over 780 pairs. The flag reads “NUM higher” at both levels anyway; with the premise false, that is a description, not a confirmed prediction.

The gates

• Headroom: the question-number head’s seen balanced accuracy must lie above chance + 5 pp and below 95%. It passed on all three primary blocks; the levels are in the tables above.
• Triviality canary: a ridge classifier on per-channel mean and log-variance must not answer the seen task by itself. It passed on BETA (on the first fold’s 60 training people) and on BOAS and EESM19 (carried from route 2); none is declared for motor imagery, so its gate reads “not run”.
• Read-off floor: an unseen sleep cell counts only if the read-off’s AUROC interval lies above 0.55. 8 of 8 primary and 34 of 34 secondary cells passed.
• Shuffled-EEG scorer check: each primary aggregate, recomputed with the test features shuffled within each person, must sit within three Monte-Carlo standard errors of chance. 51 of 51 cells did (26 SSVEP, 25 BOAS).
• Fit check, on the first fold’s training rows: every conditioned SSVEP arm trained more than 5 pp below the one-output-per-question head, so the declared fallback of 60 epochs applies to every SSVEP arm; sleep, EESM19 and motor imagery stay at 30. No fit failed.

Four independent audits passed. They re-implement the definitions without importing the study’s scoring code: the conformance audit of the primary run passed 106 checks with 0 failed; the numerical audit of the primary made 2,270 comparisons with 0 mismatches; the conformance audit of the secondary run passed 231 checks with 0 failed; the numerical audit of the secondary made 18,619 comparisons, 29 of them outside the tolerance, every one explained.

All 787 texts the questions were asked with, in English and Chinese, for sleep, SSVEP and motor imagery: training and held-out sentence frames, label names and synonyms, descriptions, the names and descriptions of the unseen questions, and the negation frames. Two agents wrote them from a fixed brief before any score, the second writing every held-out description without seeing the first’s; they are not users’ wordings and were not tuned. The file also holds the leakage rules they passed, the held-out frequencies of every rotation and the extrapolation band, the declared derangements of the shuffled control, the numeric code of a frequency and the pinned text encoders. Text only, CC BY 4.0.

The wordings, as JSON ↓

The sleep stages’ names in the training frames, and the three synonyms each was reworded with:

• 'Unseen' means unseen by the EEG head; the text encoder has read these words, and every token of a held-out frequency also occurs in the training texts.
• Held-out sleep questions are combinations of seen stages ('asleep' is the complement of one), reported as compositional beside the read-off.
• Held-out SSVEP questions name a frequency, which CCA answers without training; interpolation tested, extrapolation secondary.
• In the neighbour 3-way the unseen frequency competes against trained detectors for its neighbours; at L1 that test also involves stimulus phase.
• Rewording and unseen results are separate rows and never pooled.
• Wordings were written by agents from a fixed brief before scoring, not by users and not tuned.
• Negation is a boundary probe; location and waveform-shape questions are not tested (no signal-level ground truth held).
• Frozen features and a small head; one recipe; not a ranking of text or EEG encoders.
• 35 pre-declared comparisons, not corrected for multiplicity.
• Questions about negation, location or waveform shape need signal-level ground truth. Writing more prompts does not supply it.
• Negation was not understood: the negated wordings ("not X", "non-REM sleep") scored as if they asked for X (AUROC 0.02-0.13 against the truth of the negated question; boundary probe S6, never a route sentence).
• Implicit names of the unseen sleep unions ("asleep", "light sleep") ranked windows in reverse (TPL AUROC 0.317 and 0.244, below 0.5); only the explicit names ("N1, N2, N3 or REM sleep") carried the composition.
• The neighbour 3-way at L1 involves stimulus phase as well as frequency; at L0 (a phase-free spectrum) it is the clean frequency test.
• Unseen SSVEP questions compete against trained detectors for their neighbours (seen bias, stated).
• The pre-run permutation canary failed twice (revisions 3 and 4) and was revised by the owner, who overrode revision 4's "no further revision"; revision 5 passed. Both failed records are kept and disclosed.
• For multilingual-e5-small each held-out frequency is a new token (S0b-6); S8 and S12 also involve new tokens.
• Secondary single-seed entries carry no training-run variance; EESM19 has 20 people; every EESM19 and single-seed entry was declared likely inconclusive.
• BETA numbers use the published 2-s windows and folds; Wearable-102 numbers are computed from the 2026-09-20 Tsinghua author mirror and claim no physical unit.
• BOAS: the three gaps and the attribution above; participants are pseudonymised in the public release; no result evaluates Bitbrain's headband or its automatic scoring.
• The head here is not a language model, and none of these results shows that EEG understands language.

Not run, and not reported

• revision-4 per-person centring diagnostic: not run (revision 5 part A.2; no fit exists)
• Ying scorer agreement; factorial cognitive sets: deferred (D10)
• E1 EEGNet trained jointly with the question vector: deferred (D6)
• third text encoder; LEAF-style Q-Former heads; route-1 calibration on language heads; user-written wordings: deferred (protocol.deferred)
• S8 TPL - NN(TPL): not defined (EA-1): the held-out band is contiguous, so no held-out frequency has two seen neighbours
• location and waveform-shape questions: not tested: no signal-level ground truth held
• per-union paired contrasts beyond the P3 entries and the per-union levels: not computed (W1)
• counts of people above or below zero on paired contrasts: allowed for BOAS but not computed by stage 2; not added
• per-person distributions, per-fold values, per-window scores: private by design

Not published

• Per-person, per-night, per-recording and per-fold values, per-window scores, features, head weights and fold assignments.
• The per-frequency and interior-only SSVEP breakdowns: computed after the audits, which do not cover them.
• The values of the pre-run checks, which are published as pass or fail and as counts.
• Measured compute, timed on a shared GPU.
• Demographics, recording dates and clock times, and any image of a real recording.

The text encoders

• BAAI/bge-small-en-v1.5 ↗: primary · MIT · 5c38ec7
• sentence-transformers/all-MiniLM-L6-v2 ↗: secondary, S4 · Apache-2.0 · 1110a24
• intfloat/multilingual-e5-small ↗: Chinese wordings, S5 · MIT · 614241f

Frozen at pinned commits and never redistributed.

The recordings

Features: frozen CBraMod (primary) and frozen REVE Large @ 317531c7, REVE-L (secondary).

Weights terms: REVE under the REVE Responsible Use License v1.0 (model versions: REVE Large @ 317531c7). No model’s authors endorse these results. REVE, arXiv:2510.21585 ↗

Methods and sources these results draw on

• Yu & Yao, 2026 · Visual Jev: Accurate and Efficient Decisions from Shared Visual Context ↗
• Perez et al., 2017 · FiLM: Visual Reasoning with a General Conditioning Layer ↗
• Lin, Zhang, Wu & Gao, 2006 · Frequency Recognition Based on CCA for SSVEP-Based BCIs, IEEE TBME 53:2610–2614 ↗
• Chen, Wang, Gao, Jung & Gao, 2015 · Filter bank CCA for implementing a high-speed SSVEP-based BCI, J. Neural Eng. 12:046008 ↗
• Wallace et al., 2019 · Do NLP Models Know Numbers? Probing Numeracy in Embeddings, EMNLP 2019 ↗
• Weller, Lawrie & Van Durme, 2024 · NevIR: Negation in Neural Information Retrieval, EACL 2024 ↗
• Xian, Lampert, Schiele & Akata · Zero-Shot Learning — A Comprehensive Evaluation of the Good, the Bad and the Ugly, TPAMI ↗

Data source: questions-in-language-update.json · schema bci-report-questions-in-language-update-v1.

Wordings: questions-in-language-wordings.json · schema bci-report-questions-in-language-wordings-v1.