BCI ReportSame dataset
The 20 people come from the same release, lab and protocol as the pilot; this is not evidence from an independent cohort.
Abstention · reliable decisions · Jev-style research plan
How often does an EEG decoder fire when nobody is giving a command?
On this site’s idle protocol, EEGNet made no false activation in 60 idle trials but detected 1 of 60 commands. On EEGMAT and BETA, route 1 of a Jev-style research plan measures when a decoder should decline to decide.
Measured on: ds003810; EEGMAT; BETA; ds005342; YSU asynchronous SSVEP-BCI dataset · Methods: EEGNet; LaBraM; CBraMod; ShallowFBCSPNet; Deep4Net; CSP+LDA; CCA
Short answer
It depends on how often the decoder acts at all: one that never acts never fires by mistake, so its false activations mean something only beside the commands it caught. In a cue-gated idle test with 4 people, the quietest decoder, EEGNet, made 0 / 60 false activations in idle trials but detected 1 / 60 commands, and for 2 of those people its threshold rejected every command. In a separate test with 20 other people, a CCA decoder with a threshold fixed on an earlier pilot accepted 748 / 960 windows when a command was intended, and 49 / 240, 93 / 240 and 107 / 480 in three states where none was. A threshold fitted instead on 96 of each person’s own windows accepted fewer non-control windows overall, and fewer in each state, but did not raise the share of commands both accepted and correct; the contrast cannot separate fitting the threshold to the person from simply having those labels.
0 / 60EEGNet: idle false activations, 4 people
No false activation, and 1 command in 60
Each threshold is chosen on calibration data only; for 2 of 4 people it rejected every command.
Both numbers, every method
Command detection against idle false activation
4 people; 60 idle and 60 command test trials in total; a command counts if it is detected within 3 seconds of its cue. Sorted by command detection.
| Method | Training | Commands detected ≤3 s | Idle false activations | Always abstained |
|---|---|---|---|---|
| ShallowFBCSPNet | Scratch · 20 epochs | 26 / 6043.3%43.3% of commands detected | 1 / 601.7% | 1of 4 people |
| CSP+LDA | Supervised fit | 14 / 6023.3%23.3% of commands detected | 0 / 600.0% | 1of 4 people |
| LaBraM | Frozen encoder + linear head | 11 / 6018.3%18.3% of commands detected | 2 / 603.3% | 0of 4 people |
| CBraMod | Frozen encoder + linear head | 11 / 6018.3%18.3% of commands detected | 1 / 601.7% | 1of 4 people |
| Deep4Net | Scratch · 20 epochs | 2 / 603.3%3.3% of commands detected | 1 / 601.7% | 1of 4 people |
| EEGNet | Scratch · 20 epochs | 1 / 601.7%1.7% of commands detected | 0 / 600.0% | 2of 4 people |
Explicit non-control · 20 people · rules fixed before scoring; global threshold from a 4-person pilot
Does it accept a window when nobody intends a command?
A second data set records the states between commands: looking at a central image with the flicker off, resting with the eyes on a white wall, and looking at the centre while the surrounding targets flicker. A fixed CCA decoder accepts or rejects each 1.5-second window as a command, under two rules fixed before any of these 20 people was scored: a global threshold fitted only on the calibration windows of the 4 pilot people below, and a personal threshold fitted on 96 labelled windows from each new person. Both rules score the same 96 held-out windows of each person.
+3.0 ppDetection balanced accuracy, personal minus global threshold, same 20 people
A personal threshold detected intent a little better — and produced no more correct commands
Paired interval +0.5 to +5.7 pp. 10 of 20 people improved, 8 got worse and 2 were unchanged. The gain came from fewer false acceptances: coverage of intended commands barely moved, and the share of command windows both accepted and correct went from 68.1% to 67.6%.
| Measure | Global threshold · fixed on the pilot | Personal threshold · own windows |
|---|---|---|
| Detection balanced accuracy, command against no command | 76.0%95% interval 71.6%–80.2% | 79.0%95% interval 74.2%–83.8% |
| Command windows accepted (coverage) | 748 / 96077.9% | 746 / 96077.7% |
| Accepted and correct (end to end) | 654 / 96068.1% | 649 / 96067.6% |
| Correct among accepted | 85.8%mean of each person’s rate | 85.1%mean of each person’s rate |
Before any rejection the frequency was recognised in 769 / 960 command windows under both rules: the decoder is the same, and only the rejection rule differs. Detection went up; correct commands did not.
| State | Global threshold · fixed on the pilot | Personal threshold · own windows |
|---|---|---|
| Central image, flicker offNS1 | 49 / 24020.4% | 32 / 24013.3% |
| Looking at a white wall, restingNS2 | 93 / 24038.8% | 79 / 24032.9% |
| Central image while the surrounding targets flickerNS3 | 107 / 48022.3% | 79 / 48016.5% |
| All three states, pooledevery non-control window | 249 / 96025.9% | 190 / 96019.8% |
The detection score pools the three states as they were collected, 12, 12 and 24 windows a person, so the flickering-centre state counts twice; the last row is that pool.
Limits of the non-control test
96 labels per person
The personal rule uses 96 labelled windows from each person, half with a command and half without. The comparison cannot separate fitting the threshold to the person from simply having those labels. The source does not say when those windows were recorded relative to the test windows, so calibrating earlier and using later is untested here.
Window rates
Separately collected 1.5-second windows in a fixed mix of states. They give window rates, not false activations per hour of continuous use, latency, or how often people actually issue commands.
No amplitude unit
The source states no physical unit for its EEG values. The decoder uses correlations, which are unit-free.
Bootstrap intervals
Whole-person bootstrap over the 20, conditional on the 4 pilot people and on each realised calibration split.
Development set: the 4-person pilot
Scored with per-person thresholds fitted on each person’s own calibration trials. Its 4 people supplied the global threshold above and are not among the 20. Pooled window counts.
| State | Windows |
|---|---|
| Frequency recognised, before rejection | 152 / 19279.2% |
| Accepted (coverage) | 130 / 19267.7% |
| Accepted and correct (end to end) | 121 / 19263.0% |
| Correct among accepted | 121 / 13093.1% |
| State | Accepted by mistake |
|---|---|
| Central image, flicker off | 9 / 4818.8% |
| Looking at a white wall, resting | 20 / 4841.7% |
| Central image while the surrounding targets flicker | 11 / 9611.5% |
Offline, window by window.
Why four people limit what can be published (English)
YSU asynchronous SSVEP-BCI dataset
Jing Zhao, Qian Zhang, Xinrui Wang, Xueshuo Liu, Jiaxin Li, Fengjie Fan, Zhenhu Liang and Xiaoli Li · An EEG dataset for studying asynchronous steady-state visual evoked potential (SSVEP) based brain computer interfaces, Brain-Apparatus Communication 3(1) (2024), doi:10.1080/27706710.2024.2418650; data at doi:10.6084/m9.figshare.24906300.v3.
Before the experiment, every participant signed written informed consent; the study was reviewed and approved by the ethics committee of Qinhuangdao First Hospital. This site uses all twenty-four people of the release, four as the development pilot and twenty as the extension, and publishes only cohort means, pooled window counts and how many people were helped or harmed.
Data source: extension-update.json · schema bci-report-extension-update-v1.
Data source: reviewed aggregate JSON · schema bci-report-context-update-v1.
Methods & limits
Limits of the idle test
4 people
At this size an average stays close to its parts: a published rate can be turned back into a count, and a count narrowed to one of two people.
Three minutes of idle
The test holds 180 seconds of idle, too little to estimate false activations per hour of unprompted rest.
Cue-gated
Detection restarts at every cue, so the decoder always knows a decision window has opened. A system that must notice intent on its own faces a harder problem than this protocol measures.
No probability quality on this protocol
These are thresholded decisions. Probability calibration is measured on EEGMAT and BETA, in the reliable-decisions results below.
The four-person arithmetic, in full (English)
ds005342 · Seated motor imagery · cue-gated replay
OpenNeuro ds005342 contributors · version 1.0.3, doi:10.18112/openneuro.ds005342.v1.0.3; associated study doi:10.3389/fninf.2022.961089. Original author credits are retained at the linked source.
The source README reports voluntary signed consent and approval by the ethics committee of Universidad Antonio Nariño. Only cohort aggregates appear here.
Data source: reviewed aggregate JSON · this protocol as CSV · research-preview-20260920.
Reliable decisions · route 1 of the plan below · EEGMAT and BETA
When should a decoder decline to decide?
Route 1 took the saved test scores of models this site already publishes — on EEGMAT, 36 people, mental arithmetic against rest, and on BETA, 70 people, a 40-target SSVEP speller — and asked which trials each decoder should have declined, under four rules fixed in advance. Full-coverage accuracies are the published ones: for the two LaBraM arms, the adaptation release’s seed-20260922 mean.
Four rules, fixed before scoring
- Fixed threshold (F): accept a trial when its calibrated top-class probability is at least 0.8. Temperature scaling, fitted on calibration scores, keeps every predicted label.
- Coverage target (S-cov): accept the most confident trials, with the cut-off set from calibration quantiles so that 80% should be accepted.
- Certified risk (S-risk): selection with guaranteed risk (SGR, δ = 0.05) certifies a cut-off whose error should stay at or below a target r*, here half the calibration error. A fold where nothing can be certified accepts nothing.
- Learned reject option (L): a logistic selector on the classifier’s top two scores and their margin, plus a spectral description of the signal that does not depend on the model, trained to predict which trials the classifier gets wrong.
Calibration scores never include a test person: in each outer fold the training people form four groups, and each group is scored by a model trained on the other three. Intervals are paired person-bootstrap 95% intervals over 2,000 resamples, with no multiplicity correction. A contrast is called a difference only when its interval excludes zero; differences of rates are in percentage points (pp).
A fixed threshold is a different policy for each method
The same cut-off — accept when the calibrated top-class probability is at least 0.8 — accepts very different shares of trials from different decoders, and from different people.
| Method | Balanced accuracy, everything accepted | Accepted at confidence ≥ 0.8 | Error among accepted | People with nothing accepted | Coverage when the cut-off targets 80% |
|---|---|---|---|---|---|
| Standard CCA | 63.1%the published value | 3,176 / 11,200coverage 28.4% · 95% interval 22.9%–34.2%28.4% of trials accepted | 4.3%95% interval 2.7%–6.8% | 0of 70 people | 79.7%95% interval 75.0%–84.0% |
| CBraMod | 33.7%the published value | 165 / 11,200coverage 1.5% · 95% interval 0.8%–2.3%1.5% of trials accepted | 22.4%95% interval 5.9%–45.1% | 38of 70 people | 76.4%95% interval 72.8%–80.0% |
| EEGNet | 55.8%the published value | 1,863 / 11,200coverage 16.6% · 95% interval 12.5%–21.0%16.6% of trials accepted | 5.5%95% interval 2.8%–9.1% | 10of 70 people | 80.5%95% interval 75.6%–85.1% |
| Pair | Difference | Reading |
|---|---|---|
| Standard CCA − CBraMod | +26.9 pp95% interval +21.7 to +32.4 pp | difference resolved |
| Standard CCA − EEGNet | +11.7 pp95% interval +7.8 to +16.2 pp | difference resolved |
| CBraMod − EEGNet | −15.2 pp95% interval −19.2 to −11.5 pp | difference resolved |
| Method | Balanced accuracy, everything accepted | Accepted at confidence ≥ 0.8 | Error among accepted | People with nothing accepted | Coverage when the cut-off targets 80% |
|---|---|---|---|---|---|
| Spectral ridge | 56.8%the published value | 3 / 2,160coverage 0.1% · 95% interval 0.0%–0.3%0.1% of trials accepted | 0 / 3wrong, of those acceptedUnstable: fewer than 10 accepted, so no rate is read from it | 33of 36 people | 80.6%95% interval 78.9%–82.3% |
| EEGNet | 67.6%the published value | 126 / 2,160coverage 5.8% · 95% interval 2.3%–10.6%5.8% of trials accepted | 42.1%95% interval 13.0%–61.4% | 19of 36 people | 79.1%95% interval 74.9%–83.2% |
| LaBraM · head only, encoder frozen | 56.9%the published seed-20260922 mean, adaptation release | 1 / 2,160coverage 0.05% · 95% interval 0.0%–0.1%0.05% of trials accepted | 0 / 1wrong, of those acceptedUnstable: fewer than 10 accepted, so no rate is read from it | 35of 36 people | 79.4%95% interval 77.7%–81.1% |
| LaBraM · LoRA rank 4 + head | 64.4%the published seed-20260922 mean, adaptation release | 44 / 2,160coverage 2.0% · 95% interval 0.9%–3.5%2.0% of trials accepted | 22.7%95% interval 6.3%–45.5% | 24of 36 people | 81.5%95% interval 79.1%–83.9% |
| Pair | Difference | Reading |
|---|---|---|
| Spectral ridge − EEGNet | −5.7 pp95% interval −10.5 to −2.1 pp | difference resolved |
| Spectral ridge − LaBraM (head only) | +0.1 pp95% interval 0.00 to +0.2 pp | no difference resolved |
| Spectral ridge − LaBraM (LoRA) | −1.9 pp95% interval −3.3 to −0.8 pp | difference resolved |
| EEGNet − LaBraM (head only) | +5.8 pp95% interval +2.2 to +10.6 pp | difference resolved |
| EEGNet − LaBraM (LoRA) | +3.8 pp95% interval +0.4 to +8.2 pp | difference resolved |
| LaBraM (head only) − LaBraM (LoRA) | −2.0 pp95% interval −3.4 to −0.9 pp | difference resolved |
Error at matched coverage
On BETA the smallest coverage the fixed threshold reached was 165 of 11,200 trials (1.5%, CBraMod’s). Each method then accepts its own 165 most confident trials, so the threshold drops out of the comparison:
| Method | Error among its most confident | Reading |
|---|---|---|
| Standard CCA | 0.6%95% interval 0.0%–2.3% | a descriptive level, no verdict |
| CBraMod | 22.4%95% interval 5.9%–45.1% | a descriptive level, no verdict |
| EEGNet | 8.5%95% interval 0.0%–28.5% | a descriptive level, no verdict |
| Standard CCA − CBraMod | −21.8 pp95% interval −44.6 to −5.1 pp | difference resolved |
| Standard CCA − EEGNet | −7.9 pp95% interval −28.2 to +1.7 pp | no difference resolved |
| CBraMod − EEGNet | +13.9 pp95% interval +5.3 to +22.2 pp | difference resolved |
On EEGMAT that common coverage is 1 window of 2,160: fewer than 10 accepted, so no verdict.
A learned reject option did not beat the model’s own calibrated confidence
Holding the classifier fixed, the learned reject option L — which also sees a spectral description of the signal — was set against the calibrated confidence S of the same classifier, on the same test trials. Two measures: the area under the risk–coverage curve (AURC; lower means the errors sit among the trials ranked least certain), and the error among the 80% of trials each ranks as most certain. L had a lower error than S at 80% for no method; where a difference was resolved, L erred more. By AURC, L ranked better only for CBraMod on BETA and worse for LaBraM (head only) on EEGMAT; every other contrast: no difference resolved.
| Method | AURC · confidence S | AURC · learned L | L − S, AURC | Error at 80% · S | Error at 80% · L | L − S, error at 80% |
|---|---|---|---|---|---|---|
| Spectral ridge | 0.38695% interval 0.348–0.428 | 0.39995% interval 0.359–0.440 | +0.01395% interval −0.006 to +0.029no difference resolved | 41.4%95% interval 38.9%–43.9% | 41.9%95% interval 39.2%–44.4% | +0.5 pp95% interval −1.2 to +1.7 ppno difference resolved |
| EEGNet | 0.29395% interval 0.211–0.368 | 0.30995% interval 0.238–0.379 | +0.01695% interval −0.014 to +0.050no difference resolved | 29.5%95% interval 23.9%–35.0% | 31.8%95% interval 27.0%–36.9% | +2.3 pp95% interval +0.6 to +4.1 ppdifference resolved |
| LaBraM · head only, encoder frozen | 0.39695% interval 0.351–0.441 | 0.42295% interval 0.378–0.466 | +0.02695% interval +0.005 to +0.046difference resolved | 42.9%95% interval 39.0%–45.8% | 43.6%95% interval 40.2%–46.6% | +0.6 pp95% interval −0.5 to +2.7 ppno difference resolved |
| LaBraM · LoRA rank 4 + head | 0.30895% interval 0.245–0.373 | 0.32395% interval 0.267–0.379 | +0.01595% interval −0.011 to +0.041no difference resolved | 33.2%95% interval 28.6%–37.4% | 35.1%95% interval 30.7%–39.8% | +1.9 pp95% interval +0.6 to +3.5 ppdifference resolved |
| Method | AURC · confidence S | AURC · learned L | L − S, AURC | Error at 80% · S | Error at 80% · L | L − S, error at 80% |
|---|---|---|---|---|---|---|
| Standard CCA | 0.14595% interval 0.105–0.194 | 0.14695% interval 0.107–0.196 | +0.00295% interval −0.004 to +0.008no difference resolved | 27.2%95% interval 20.8%–33.9% | 27.4%95% interval 21.1%–34.5% | +0.3 pp95% interval −0.3 to +1.0 ppno difference resolved |
| CBraMod | 0.49795% interval 0.430–0.566 | 0.47395% interval 0.413–0.538 | −0.02495% interval −0.039 to −0.007difference resolved | 61.7%95% interval 56.8%–66.5% | 61.6%95% interval 57.1%–66.4% | −0.03 pp95% interval −0.8 to +0.6 ppno difference resolved |
| EEGNet | 0.20895% interval 0.161–0.264 | 0.20195% interval 0.154–0.257 | −0.00795% interval −0.014 to +0.001no difference resolved | 34.8%95% interval 28.7%–41.4% | 35.6%95% interval 29.2%–42.4% | +0.8 pp95% interval +0.02 to +1.6 ppdifference resolved |
What declining buys, under the calibrated confidence
How much declining buys is read per protocol; the two are not compared: forty targets against two classes. On BETA, standard CCA’s error falls from 36.9% with everything accepted to 12.4% among its most certain half; on EEGMAT, EEGNet’s falls from 32.4% to 27.1%.
| Method | Everything accepted | Most certain 80% | Most certain 50% |
|---|---|---|---|
| EEGMAT | |||
| Spectral ridge | 43.2% | 41.4% | 39.5% |
| EEGNet | 32.4% | 29.5% | 27.1% |
| LaBraM · head only, encoder frozen | 43.1% | 42.9% | 40.7% |
| LaBraM · LoRA rank 4 + head | 35.6% | 33.2% | 29.5% |
| BETA | |||
| Standard CCA | 36.9% | 27.2% | 12.4% |
| CBraMod | 66.3% | 61.7% | 53.2% |
| EEGNet | 44.2% | 34.8% | 18.8% |
A certified error level did not reliably hold on new people
S-risk’s guarantee assumes calibration and test trials are exchangeable; across people they are not. On EEGMAT it certified no outer fold for any method, so nothing was accepted. On BETA it certified folds for every method, and in some certified folds the error on the new people exceeded that fold’s own target. The pooled error minus the target was not resolved from zero for any method.
| Method | Folds certified | Accepted | Error among accepted | Target r*, weighted over folds | Certified folds over their own target | Error minus target |
|---|---|---|---|---|---|---|
| BETA · 7 outer folds · 70 people | ||||||
| Standard CCA | 7 / 7 | 6,751 / 11,200coverage 60.3% | 17.0%95% interval 13.3%–21.6% | 18.5% | 4 / 7 | −1.5 pp95% interval −5.3 to +3.2 ppno difference resolved |
| CBraMod | 4 / 7 | 522 / 11,200coverage 4.7% | 33.1%95% interval 18.7%–48.5% | 34.9% | 2 / 4 | −1.8 pp95% interval −16.2 to +13.6 ppno difference resolved |
| EEGNet | 7 / 7 | 5,605 / 11,200coverage 50.0% | 18.9%95% interval 14.9%–23.7% | 23.3% | 2 / 7 | −4.4 pp95% interval −8.5 to +0.5 ppno difference resolved |
| EEGMAT · 5 outer folds · 36 people | ||||||
| Spectral ridge | 0 / 5 | 0 / 2,160nothing accepted | —not defined: nothing accepted | —no fold certified | —no fold certified | —not defined: nothing accepted |
| EEGNet | 0 / 5 | 0 / 2,160nothing accepted | —not defined: nothing accepted | —no fold certified | —no fold certified | —not defined: nothing accepted |
| LaBraM · head only, encoder frozen | 0 / 5 | 0 / 2,160nothing accepted | —not defined: nothing accepted | —no fold certified | —no fold certified | —not defined: nothing accepted |
| LaBraM · LoRA rank 4 + head | 0 / 5 | 0 / 2,160nothing accepted | —not defined: nothing accepted | —no fold certified | —no fold certified | —not defined: nothing accepted |
Each fold holds about ten people, so a fold’s error also carries sampling noise; the count of folds over target is descriptive.
A personal recalibration costs each new person labels
R0 uses the population temperature and no label from the new person; R1 fits a personal temperature on a labelled prefix of that person’s own trials. With 40 labels per new person on BETA, NLL and AURC fell for every method. With 10 per person on EEGMAT, NLL rose, with the interval excluding zero, for three of the four methods: many personal temperatures were degenerate, and they were kept, as the protocol requires. Re-setting only the coverage cut-off from that prefix, without its labels, moved EEGMAT coverage further from the 80% target with the 10-label prefix, resolved for three of the four methods.
| Method | Labelled prefix | Labels per new person | Change in NLL, R1 − R0 | Change in ECE | Change in AURC | People with a degenerate personal temperature | Label-free cut-off: change in distance from the 80% target |
|---|---|---|---|---|---|---|---|
| BETA | |||||||
| Standard CCA | block 1, one trial per target | 40 | −0.06595% interval −0.103 to −0.032difference resolved | −0.01995% interval −0.048 to +0.009no difference resolved | −0.01295% interval −0.022 to −0.005difference resolved | 1 / 70 | +1.0 pp95% interval −4.0 to +2.2 ppno difference resolved |
| CBraMod | block 1, one trial per target | 40 | −0.11395% interval −0.171 to −0.048difference resolved | −0.08395% interval −0.120 to −0.042difference resolved | −0.05295% interval −0.070 to −0.031difference resolved | 0 / 70 | −0.3 pp95% interval −4.4 to +3.8 ppno difference resolved |
| EEGNet | block 1, one trial per target | 40 | −0.08095% interval −0.135 to −0.020difference resolved | −0.08995% interval −0.125 to −0.048difference resolved | −0.01695% interval −0.027 to −0.004difference resolved | 0 / 70 | +2.6 pp95% interval −2.9 to +4.0 ppno difference resolved |
| EEGMAT | |||||||
| Spectral ridge | first 5 windows of each recording, then a one-window gap | 10 | +0.05495% interval +0.017 to +0.103difference resolved | +0.05995% interval +0.006 to +0.084difference resolved | +0.00695% interval −0.027 to +0.039no difference resolved | 9 / 36 | +10.6 pp95% interval +4.6 to +15.8 ppdifference resolved |
| Spectral ridge | first 10 windows of each recording, then a one-window gap | 20 | +0.02395% interval +0.003 to +0.047difference resolved | +0.03195% interval −0.006 to +0.061no difference resolved | +0.01395% interval −0.022 to +0.048no difference resolved | 9 / 36 | +1.7 pp95% interval −2.0 to +5.7 ppno difference resolved |
| EEGNet | first 5 windows of each recording, then a one-window gap | 10 | +0.33395% interval −0.051 to +0.881no difference resolved | −0.04095% interval −0.091 to +0.024no difference resolved | −0.08695% interval −0.154 to −0.014difference resolved | 8 / 36 | +5.8 pp95% interval −1.1 to +10.3 ppno difference resolved |
| EEGNet | first 10 windows of each recording, then a one-window gap | 20 | +0.02795% interval −0.088 to +0.185no difference resolved | −0.05195% interval −0.095 to +0.025no difference resolved | −0.10495% interval −0.171 to −0.030difference resolved | 5 / 36 | +1.7 pp95% interval −3.2 to +6.0 ppno difference resolved |
| LaBraM · head only, encoder frozen | first 5 windows of each recording, then a one-window gap | 10 | +0.04795% interval +0.016 to +0.082difference resolved | +0.05795% interval +0.012 to +0.082difference resolved | −0.01795% interval −0.052 to +0.018no difference resolved | 10 / 36 | +7.2 pp95% interval +2.1 to +11.1 ppdifference resolved |
| LaBraM · head only, encoder frozen | first 10 windows of each recording, then a one-window gap | 20 | −0.00395% interval −0.020 to +0.019no difference resolved | +0.00495% interval −0.033 to +0.032no difference resolved | −0.02995% interval −0.064 to +0.008no difference resolved | 10 / 36 | +2.7 pp95% interval −0.7 to +6.0 ppno difference resolved |
| LaBraM · LoRA rank 4 + head | first 5 windows of each recording, then a one-window gap | 10 | +0.93795% interval +0.184 to +1.924difference resolved | +0.03695% interval −0.008 to +0.078no difference resolved | −0.01995% interval −0.075 to +0.035no difference resolved | 9 / 36 | +6.9 pp95% interval +2.8 to +10.5 ppdifference resolved |
| LaBraM · LoRA rank 4 + head | first 10 windows of each recording, then a one-window gap | 20 | −0.02295% interval −0.053 to +0.019no difference resolved | −0.02095% interval −0.047 to +0.022no difference resolved | −0.07295% interval −0.118 to −0.022difference resolved | 4 / 36 | +1.8 pp95% interval −2.7 to +5.4 ppno difference resolved |
LoRA and probability quality, on EEGMAT
On EEGMAT, LaBraM adapted with rank-4 LoRA had a lower NLL than the head-only arm of the same recipe: calibrated −0.035 (95% interval −0.054 to −0.013). For ECE, no difference resolved (+0.015; −0.017 to +0.045). The LoRA arm is also more accurate, 64.4% against 56.9% balanced accuracy (seed 20260922, as the adaptation release publishes it; the three-seed means are on the model-adaptation page), so the NLL gap alone is not evidence of more honest probabilities. The adaptation itself is on the model-adaptation page.
Probability quality, with everything accepted
Negative log-likelihood (NLL), expected calibration error (ECE, 15 equal-mass bins) and the Brier score after temperature scaling, and NLL and ECE before it where the model outputs a probability. Lower is better throughout. NLL and the Brier score depend on the number of classes, so EEGMAT (2) and BETA (40) are not compared. Binned ECE is biased upward under person resampling, so a small ECE can sit at the lower end of its interval. In one outer fold of the LaBraM head-only arm the population temperature reached its search bound; it was flagged and kept.
| Method | NLL, calibrated | ECE, calibrated | Brier score, calibrated | NLL, raw | ECE, raw |
|---|---|---|---|---|---|
| EEGMAT | |||||
| Spectral ridge | 0.67995% interval 0.668–0.691 | 0.03795% interval 0.033–0.070 | 0.48695% interval 0.475–0.498 | —not applicable: no raw probability, not zero | —not applicable: no raw probability, not zero |
| EEGNet | 0.64895% interval 0.613–0.693 | 0.09995% interval 0.059–0.151 | 0.45095% interval 0.422–0.485 | 0.87495% interval 0.684–1.102 | 0.13595% interval 0.088–0.200 |
| LaBraM · head only, encoder frozen | 0.68395% interval 0.674–0.691 | 0.03695% interval 0.034–0.072 | 0.49095% interval 0.481–0.498 | 0.68395% interval 0.662–0.703 | 0.06195% interval 0.046–0.090 |
| LaBraM · LoRA rank 4 + head | 0.64895% interval 0.624–0.675 | 0.05195% interval 0.040–0.094 | 0.45695% interval 0.433–0.481 | 0.64795% interval 0.607–0.692 | 0.04395% interval 0.038–0.088 |
| BETA | |||||
| Standard CCA | 1.47695% interval 1.247–1.716 | 0.05095% interval 0.028–0.079 | 0.50495% interval 0.437–0.572 | —not applicable: no raw probability, not zero | —not applicable: no raw probability, not zero |
| CBraMod | 2.80995% interval 2.638–3.001 | 0.11995% interval 0.084–0.154 | 0.84395% interval 0.815–0.871 | —not applicable: no raw probability, not zero | —not applicable: no raw probability, not zero |
| EEGNet | 1.91295% interval 1.670–2.177 | 0.13695% interval 0.103–0.168 | 0.60795% interval 0.554–0.663 | 1.90795% interval 1.665–2.171 | 0.13695% interval 0.103–0.168 |
Robustness and exploratory arms, each read on its own
ds003810, motor imagery and rest: crude
10 people, two per test fold. Its contrasts below are secondary. The certified risk accepted nothing here for any method.
| Method | Balanced accuracy, everything accepted | Accepted at confidence ≥ 0.8 | People with nothing accepted | L − S, AURC | L − S, error at 80% | Folds certified (S-risk) |
|---|---|---|---|---|---|---|
| Spectral ridge | 54.9%the published value | 0 / 1,200coverage 0.0% | 10of 10 people | +0.00395% interval −0.031 to +0.032no difference resolved | +0.5 pp95% interval −1.1 to +2.3 ppno difference resolved | 0 / 5nothing accepted |
| CBraMod | 62.9%the published value | 7 / 1,200coverage 0.6% | 6of 10 people | +0.01395% interval −0.014 to +0.030no difference resolved | +0.7 pp95% interval −0.8 to +2.4 ppno difference resolved | 0 / 5nothing accepted |
| EEGNet | 71.0%the published value | 250 / 1,200coverage 20.8% | 0of 10 people | +0.03895% interval +0.009 to +0.061difference resolved | +0.9 pp95% interval −0.4 to +2.4 ppno difference resolved | 0 / 5nothing accepted |
Secondary seeds
Every other saved seed of the trained methods, each with its own calibration, cut-offs and selector, scored on that seed’s saved test predictions.
| Method | Dataset | Seed | L − S, AURC | L − S, error at 80% |
|---|---|---|---|---|
| EEGNet | EEGMAT | 20260920 | +0.00795% interval −0.029 to +0.041no difference resolved | +1.4 pp95% interval −0.6 to +3.2 ppno difference resolved |
| EEGNet | EEGMAT | 20260921 | −0.00195% interval −0.045 to +0.038no difference resolved | +2.3 pp95% interval 0.00 to +4.6 ppno difference resolved |
| LaBraM · head only, encoder frozen | EEGMAT | 20260923 | +0.04495% interval +0.013 to +0.074difference resolved | +1.9 pp95% interval +0.3 to +3.7 ppdifference resolved |
| LaBraM · head only, encoder frozen | EEGMAT | 20260924 | +0.02495% interval −0.001 to +0.050no difference resolved | +1.4 pp95% interval +0.1 to +2.8 ppdifference resolved |
| LaBraM · LoRA rank 4 + head | EEGMAT | 20260923 | +0.02195% interval −0.001 to +0.043no difference resolved | +2.0 pp95% interval +0.7 to +3.2 ppdifference resolved |
| LaBraM · LoRA rank 4 + head | EEGMAT | 20260924 | +0.02495% interval −0.004 to +0.050no difference resolved | +1.7 pp95% interval +0.6 to +3.2 ppdifference resolved |
| EEGNet | BETA | 20260919 | −0.00295% interval −0.011 to +0.007no difference resolved | +0.9 pp95% interval −0.1 to +1.8 ppno difference resolved |
| EEGNet | BETA | 20260920 | −0.00695% interval −0.013 to +0.002no difference resolved | +0.3 pp95% interval −0.5 to +1.0 ppno difference resolved |
| EEGNet | BETA | 20260921 | −0.01095% interval −0.019 to −0.001difference resolved | +0.1 pp95% interval −0.8 to +0.8 ppno difference resolved |
Two sensitivity arms: certified folds only
The matched-model holdout trains the model on three quarters of the training people and calibrates it on the remaining quarter, so calibration and test scores come from one model. Platt scaling on two classes and vector scaling on BETA replace temperature scaling and can change predicted labels.
| Method | Dataset | Arm | Folds certified | Certified folds over their own target |
|---|---|---|---|---|
| Spectral ridge | EEGMAT | matched-model holdout | 0 / 5 | —no fold certified |
| EEGNet | EEGMAT | matched-model holdout | 1 / 5 | 0 / 1 |
| LaBraM · head only, encoder frozen | EEGMAT | matched-model holdout | 0 / 5 | —no fold certified |
| LaBraM · LoRA rank 4 + head | EEGMAT | matched-model holdout | 0 / 5 | —no fold certified |
| Standard CCA | BETA | matched-model holdout | 7 / 7 | 3 / 7 |
| CBraMod | BETA | matched-model holdout | 6 / 7 | 3 / 6 |
| EEGNet | BETA | matched-model holdout | 7 / 7 | 1 / 7 |
| Spectral ridge | EEGMAT | Platt scaling | 0 / 5 | —no fold certified |
| EEGNet | EEGMAT | Platt scaling | 0 / 5 | —no fold certified |
| LaBraM · head only, encoder frozen | EEGMAT | Platt scaling | 0 / 5 | —no fold certified |
| LaBraM · LoRA rank 4 + head | EEGMAT | Platt scaling | 0 / 5 | —no fold certified |
| Standard CCA | BETA | vector scaling | 7 / 7 | 5 / 7 |
| CBraMod | BETA | vector scaling | 4 / 7 | 2 / 4 |
| EEGNet | BETA | vector scaling | 7 / 7 | 2 / 7 |
A reject head trained with its classifier (exploratory)
A SelectiveNet-style EEGNet, trained with its own selection head g (alpha 0.5, lambda 32, target coverage 0.8). Within the same model, g is set against the calibrated confidence of its own classifier head; a positive difference means g is worse.
| Dataset | g − confidence, AURC | g − confidence, error at 80% |
|---|---|---|
| EEGMAT | +0.01195% interval −0.012 to +0.035no difference resolved | +2.4 pp95% interval +0.8 to +4.2 ppdifference resolved |
| BETA | +0.08095% interval +0.061 to +0.099difference resolved | +1.9 pp95% interval +1.0 to +2.6 ppdifference resolved |
A cross-day label-budget arm on BNCI2015-001 stays unpublished while its source is held (see the holds register).
Limits of route 1
Method comparisons
Every protocol here is balanced or uniform by design: EEGMAT is half rest and half arithmetic, BETA uniform over its 40 targets, ds003810 balanced. The figures compare methods; none is a deployment error rate. On EEGMAT each class is its own recording, so recording-level differences are part of what any model or selector can use. BETA is cue-paced spelling with no idle state, so it gives no false-activation figure of any kind.
Nothing accepted, no error rate
A risk–coverage curve exists only where something is accepted. At zero coverage there is no error rate, not a zero one, and fewer than 10 accepted trials are flagged unstable.
A nominal guarantee
A certified selective risk assumes that calibration and test data are exchangeable. Across people they are not, so the guarantee is nominal: how often it fails is measured here, not assumed.
Out-of-distribution input
Rejecting likely errors on known classes is not detecting unfamiliar input: an uncertainty threshold may not flag an unseen class. Route 1 makes no out-of-distribution claim.
Calibration on fewer people
Calibration scores come from inner models that see about three quarters of the training people, so they may be less confident than the full model whose scores are tested. This approximation is part of every calibrated figure.
Published accuracies
Temperature scaling keeps each predicted label, so every full-coverage accuracy equals the site’s published value. The matched-model, map-sensitivity and jointly trained arms use other models or maps; their accuracies are not published and are never mixed with the site’s.
The difference rule
A contrast is called a difference only when its paired 95% person-bootstrap interval excludes zero, with no multiplicity correction; otherwise “no difference resolved”. One row is above another only where their 95% intervals do not overlap; descriptive levels carry no verdict.
Idle left out
On the idle protocol, route 1 gives per-window proportions at each decision time, never a false-activation rate per hour, in a different unit from this page’s idle figures, so they are left out.
ds003810 is crude
Ten people, two per test fold: bootstrap intervals are crude, and no person-level spread is published.
The recordings and the methods
EEGMAT
Igor Zyma, Ivan Seleznov, Anton Popov, Mariia Chernykh, Oleksii Shpenkov · EEG During Mental Arithmetic Tasks 1.0.0, PhysioNet, doi:10.13026/C2JQ1P. Study: Zyma et al. (2019), doi:10.3390/data4010014. PhysioNet platform: Pollard et al. (2026), doi:10.1038/s44360-026-00096-z.
Dataset record ↗ · Open Data Commons Attribution License 1.0
BETA
Bingchuan Liu et al. · BETA: A Large Benchmark Database Toward SSVEP-BCI Application (2020), doi:10.3389/fnins.2020.00627. Figshare 12264401 v3; mirror Bingchuan/BETA.
ds003810
Peterson et al. · OpenNeuro ds003810, version 2.0.2. Study: https://pmc.ncbi.nlm.nih.gov/articles/PMC9114495/
Methods these results rest on, checked against their records on 4 October 2026
- Guo, Pleiss, Sun & Weinberger, ICML 2017 · On calibration of modern neural networks ↗
- Geifman & El-Yaniv, 2017 · Selective classification for deep neural networks ↗
- Geifman & El-Yaniv, ICML 2019 · SelectiveNet: a deep neural network with an integrated reject option ↗
- Geifman, Uziel & El-Yaniv, ICLR 2019 · Bias-reduced uncertainty estimation for deep neural classifiers (AURC) ↗
- Nixon et al., 2019 · Measuring calibration in deep learning ↗
- Mulder, Valdenegro-Toro, Sburlea & de Jong, 2026 · The challenge of out-of-distribution detection in motor imagery BCIs ↗
Checks behind these figures: both stage-0 audits passed. The independent primary audit passed 11 of its 12 checks; the one it failed found a documentation error, not a computation error — the run log said a degenerate personal temperature falls back to the population one, while the code keeps it, as the protocol requires, and every number was computed and reproduced that way. The independent secondary audit passed all 15. Both re-implement the definitions without the study code; they stay in the private run root and are pinned by hash.
Data source: reliable-decisions-update.json · schema bci-report-reliable-decisions-update-v1.
A longer write-up of the BETA results is on Hugging Face: When should an EEG decoder abstain? ↗
Research plan · three routes run
Jev-style decision models for EEG
A decoder has more choices than a class label: it can act, wait for more evidence, or ask for calibration.
Status: all three routes have been run. The first route’s results are in the section above; the second’s and the third’s are on their own pages.
“Jev-style” here means the interface of a decision model like TypeSafe’s Jev, applied to EEG: encode the recording once, then answer several explicit, typed questions about it, each with a probability. The first route needs no new model: it rescores saved outputs of models this site already publishes. The second trained models that share one encoder (small EEGNets from scratch, and CBraMod frozen or adapted by LoRA) and are given each question by an identifier, never in language. The third asks the questions in words, through a frozen text encoder and a small head over frozen EEG features.
First · Run · results above
Reliable decisions
Compare a fixed threshold, a simple statistical policy and a learned reject option on the same data. Report error at matched coverage, acceptance per class, and probability quality — Brier score, log loss and reliability — with calibration data kept apart from test data. Recalibration is a separate experiment, with its label cost counted.
Published in the section above: error at matched coverage, log loss, the Brier score and calibration error. Not published: acceptance per class and reliability-diagram bins.
Then · Run · results on their own page
One representation, several questions
On the motor-imagery and sleep protocols separately, compare independent task models, a shared encoder with fixed heads, and the same encoder with a question-conditioned head, at matched data and compute.
Run on motor imagery (OpenBMI) and sleep (BOAS, with EESM19 as a crude replication). One secondary arm, CBraMod adapted by LoRA on sleep, ran after the other results were known; its design was fixed beforehand.
Later · Run · results on their own page
Questions in language
Compare task identifiers, label templates and natural-language descriptions. Paraphrases and unseen concepts are tested separately and never merged into one zero-shot number.
Run on SSVEP (BETA) and sleep (BOAS), with EESM19, OpenBMI motor imagery and Wearable-102 as secondary data. The pre-run permutation check failed twice and the owner revised it each time, the second time overriding its own “no further revision”; both failures are disclosed with the results.
Questions about negation, location or waveform shape need signal-level ground truth.
All three routes, with one answer each: Jev-style questions on EEG: the evidence →
Limits of the existing data
The sleep and P300 protocols on this site use balanced subsets. A probability estimated on them reflects that balance, so they serve to compare methods; a deployment error rate needs data in which each class occurs as often as in real use.
The first input with explicit non-control states is the asynchronous SSVEP data set above: a small pilot kept as the development set, then the rest of the same release, scored under rules fixed in advance.
Published work
External evidence as its authors report it, checked against the original records on 27 September 2026.
Geifman & El-Yaniv, 2017 · Selective classification for deep neural networks ↗
Shows A trained classifier can be given a reject option with a target risk set by the user, trading coverage for error.
Limits Evidence from image benchmarks. Its guarantee assumes test data that resemble the calibration data, which EEG across people and sessions often does not.
Geifman & El-Yaniv, ICML 2019 · SelectiveNet ↗
Shows Classification and rejection can be trained together, with risk at a chosen coverage as the explicit objective.
Limits Not EEG evidence. Its guarantees do not carry over to a new participant or device by default.
Chatterjee et al., 2026 · Making conformal predictors robust in healthcare settings ↗
Shows On EEG seizure classification, patient shift left standard conformal prediction short of its nominal coverage, and personalised calibration raised coverage substantially.
Limits One clinical task. It shows that coverage must be measured under the split that will actually be used.
Dai et al., 2026 · MTEEG: unified multi-task EEG analysis with low-rank adaptation ↗
Shows One pretrained EEG model can be adapted to several tasks at once with task-specific LoRA modules.
Limits A separate adapter per task is not the same as several questions sharing one computation over the same window.
Jiang et al., 2024 · NeuroLM: a universal multi-task foundation model for EEG ↗
Shows EEG tokens fed to a language model, with instruction tuning, support several EEG tasks in one model.
Limits Instruction-style EEG models already exist. A multi-task result is not evidence that arbitrary questions generalise.
Lu et al., 2025 · UniMind: LLMs for unified multi-task brain decoding ↗
Shows A language-model-based EEG model can select task-aware queries to decode several tasks.
Limits Its queries are internal tokens, not criteria a user writes.
Gijsen & Ritter, 2024 · EEG-language pretraining for label-efficient clinical phenotyping ↗
Shows EEG aligned with clinical reports enables zero-shot classification and retrieval in the authors’ clinical evaluations.
Limits Hospital EEG and clinical phenotypes. Nothing yet about wearable devices, few channels or arbitrary concepts.
Yu & Yao, 2026 · Visual Jev: Accurate and Efficient Decisions from Shared Visual Context ↗
Shows In vision, one encoding of an image and its context can serve several independent forced-choice questions run as a batch: the same encode-once, answer-many idea, applied to images. This plan started from it.
Limits Its own typed-head control gave no consistent accuracy advantage, and its training gains were concentrated on the task families represented in training. Its speed-ups were measured on images.
Where this connects
How much calibration? · Does pretraining help? · Fewer electrodes
Cite this page
BCI Report (2026). How often does an EEG decoder fire when nobody is giving a command? https://bci.report/topics/when-not-to-act/
Figures from releases reliable-decisions-update-20261004 (2026-10-04), extension-update-20261002 (2026-10-02), context-update-20260927 (2026-09-27), research-preview-20260920 (2026-09-20). Cite the upstream datasets as well: their credits are on this page.
Every release is archived on Zenodo: doi:10.5281/zenodo.23123296. BibTeX for the site and its releases → · CITATION.cff ↗