# What can a small pilot show, and what can it not?

Pilot studies · small cohorts, not ranked

Four small pilots, each its own dataset, task and metric. Injected orthosis errors in one cohort of eight people, including frozen EEGPT against matched classical features; inner speech in Greek and English, across people and languages; channels flat at a digital rail in an imagined-handwriting dataset, and what the EEG left over can show; and a stored-clip check on an emotion dataset.

- [Reviewed aggregate JSON ↓](https://bci.report/data/pilot-studies-update.json)
- 

Measured on: [Orthosis error-detection dataset (Kueper et al. 2024)](https://bci.report/datasets/orthosis-error-detection/); [BINGO cross-linguistic inner-speech dataset](https://bci.report/datasets/bingo-imagined-speech/); [EEG-fNIRS-based Handwriting Trajectory Dataset](https://bci.report/datasets/eeg-fnirs-handwriting/); [REFED emotion dataset (EEG-fNIRS)](https://bci.report/datasets/refed/)

## Short answer

Five findings from four pilots, none ranked. Orthosis device errors: fixed EEG features reach **73.0%** balanced accuracy in a person left out of training (95% interval **68.0%** to **77.7%**), against **51.4%** for a confound probe; the labels are recorded device events, so movement or device artifacts may contribute. Frozen EEGPT against matched classical features on the same eight people: **−0.6 pp** (interval **−4.0** to **+3.2 pp**): the interval spans zero, so neither is shown to be better, and the two are not shown to be equivalent. Inner speech in Greek and English (BINGO): every direction stays near the **6.7%** prior, within and across languages. Handwriting, **5** people: with flat and rail-touching channels masked, relative EEG reaches **26.81%** balanced accuracy on four coded labels against **25.00%** for the majority control, above the missingness-mask controls but not separated from elapsed time and trial order: a data-quality case study, not handwriting recognition. REFED emotion clips: EEG band power has a larger error than every control that uses no EEG, and when videos are shared, looking up the same video’s rating has the smallest.

Human · scalp EEG · recorded orthosis device events · Kueper et al.

## Orthosis errors: can EEG flag an injected device error in a person left out of training?

Locally executed experiment · offline · one cohort in three configurations, counted once

8 people wore an active orthosis that moved the right arm in elbow flexion and extension, and errors were injected into some of its movements. The task tells the recorded error events from standard ones, in the 0–0.5 s after each event: 2,407 events in 83 runs, 474 errors and 1,933 standard. Each person is held out whole, and a fixed probe fitted on the others scores them; results are averaged over the 8 people with equal weight. Three configurations of this one cohort are published, and it is counted once.

### The 64-channel classical baseline

**Balanced accuracy for a person left out of training, 64-channel configuration.** Dot: the mean over the eight people, each weighted equally; line: the 95% person-bootstrap interval; dashed line: chance. The class prior scores chance for everyone, so it has no interval. The permutation control is drawn at the mean of its replicates; the range in its label is not an interval.

- Fixed EEG features + logistic regression, 64 channels — 95% interval 68.0%–77.7%: 73.0%
- Direction/time confound probe (no EEG) — 95% interval 50.4%–52.2%: 51.4%
- Training class prior — a constant probability: chance for every person: 50.0%
- Label-permutation control — mean of 20 replicates, which ranged from 48.0% to 52.6%: not an interval: 50.1%

**73.0%** Fixed EEG features + logistic regression, 64 channels: balanced accuracy for a person left out of training. The confound probe scores 51.4%, the class prior 50.0%.

### Above every control, in people the probe was not trained on

Its 95% interval, 68.0% to 77.7%, lies above the confound probe’s interval, the class prior and every permutation replicate. Matched person by person, EEG minus the confound probe is +21.6 pp, with no interval computed; EEG was above the probe for 8 of 8 people.

The labels are recorded device events, and the result does not show that a person recognised the error: movement, muscle and device-injection artifacts may contribute, and beating a probe of movement direction and timing does not isolate a neural origin. Eight people limit what the interval can say, and this reanalysis does not reproduce the authors’ competition score.

### Frozen EEGPT against the same baseline on 58 channels

EEGPT supports 58 of the 64 channels; the other six (TP9, TP10, PO9, PO10, FT9, FT10) are left out, fixed before scoring. On those channels the same classical baseline scores 71.0% (95% interval 64.8% to 76.9%), also above its confound probe, the prior and every permutation replicate: a second configuration of the same people, not a replication, and no conclusion about how many channels are needed follows. Frozen EEGPT then replaces the classical features: the encoder’s weights stay unchanged, and its outputs feed the same probe, with the same people, folds and permutations.

**Balanced accuracy for a person left out of training, 58-channel configuration: frozen EEGPT against the matched classical features.** Dot: the mean over the eight people; line: the 95% person-bootstrap interval; dashed line: chance. The two arms share the same people, folds and probe.

- Frozen EEGPT + the same probe, 58 channels — 95% interval 66.2%–73.9%: 70.3%
- Fixed EEG features + logistic regression, 58 channels — 95% interval 64.8%–76.9%: 71.0%
- Direction/time confound probe (no EEG) — 95% interval 50.4%–52.3%: 51.4%
- Training class prior — a constant probability: chance for every person: 50.0%

**The paired difference, frozen EEGPT minus the matched classical features, in percentage points.** Dot: the mean of the eight people’s differences; line: its 95% person-bootstrap interval, which resamples each person’s difference; dashed line: no difference.

- Frozen EEGPT minus matched classical features — 95% interval −4.0 to +3.2 pp: −0.6 pp

**−0.6 pp** Frozen EEGPT minus the matched classical features, balanced accuracy, paired over the same 8 people; 95% interval −4.0 to +3.2 pp

### Neither shown better, nor shown equivalent

Frozen EEGPT scores 70.3% (66.2% to 73.9%) and the matched classical features 71.0%. The paired interval spans zero: neither arm is shown to be better, and the two are not shown to be equivalent. One short-window setup on eight people is not a verdict on EEG foundation models in general.

The same eight people as the 64-channel result: not additional coverage. The 64-channel scores were seen before this comparison was frozen, so it is a prospective diagnostic, not a blind confirmation. The probe is trained on the other seven people’s labels: supervised transfer, not zero-shot. Overlap between this dataset and EEGPT’s pretraining data has not been excluded, and the montage, filtering, reference and half-second window differ from its pretraining. The labels are recorded device events, so movement, muscle and device artifacts may contribute here too.

Every published metric of the three configurations, each the mean over the 8 people with its 95% interval where it has one. Errors are 19.7% of the events, so average precision is read against a constant guess’s 0.197. AUROC and average precision are secondary metrics.

| Arm | Balanced accuracy | AUROC | Average precision | Label-permutation control |
| --- | --- | --- | --- | --- |
| 64-channel configuration |  |  |  |  |
| Fixed EEG features + logistic regression, 64 channels | 73.0% (95% interval 68.0%–77.7%) | 0.812 (95% interval 0.751–0.863) | 0.585 (95% interval 0.492–0.672) | 50.1% (range 48.0% to 52.6%) |
| Direction/time confound probe (no EEG) | 51.4% (95% interval 50.4%–52.2%) | 0.524 (95% interval 0.507–0.541) | 0.228 (95% interval 0.208–0.249) | — |
| Training majority | 50.0% (the same for every person) | 0.500 (the same for every person) | 0.197 (95% interval 0.194–0.200) | — |
| Training class prior | 50.0% (the same for every person) | 0.500 (the same for every person) | 0.197 (95% interval 0.194–0.200) | — |
| 58-channel configuration · the same people and folds |  |  |  |  |
| Fixed EEG features + logistic regression, 58 channels | 71.0% (95% interval 64.8%–76.9%) | 0.793 (95% interval 0.719–0.854) | 0.544 (95% interval 0.438–0.654) | 50.1% (range 48.2% to 51.7%) |
| Frozen EEGPT + the same probe, 58 channels | 70.3% (95% interval 66.2%–73.9%) | 0.780 (95% interval 0.721–0.827) | 0.564 (95% interval 0.489–0.630) | 49.8% (range 47.6% to 52.2%) |
| Direction/time confound probe (no EEG) | 51.4% (95% interval 50.4%–52.3%) | 0.524 (95% interval 0.506–0.541) | 0.228 (95% interval 0.208–0.249) | — |
| Training majority | 50.0% (the same for every person) | 0.500 (the same for every person) | 0.197 (95% interval 0.194–0.200) | — |
| Training class prior | 50.0% (the same for every person) | 0.500 (the same for every person) | 0.197 (95% interval 0.194–0.200) | — |

A response-clean subset is published as counts only: 451 events, 446 errors and 5 standard events, with 3 people having both classes. Its scores stay out of every comparison.

Data: Niklas Kueper, Kartik Chari, Judith Bütefür, Julia Habenicht, Tobias Rossol, Su Kyoung Kim, Marc Tabie, Frank Kirchner and Elsa Andrea Kirchner, `doi:10.5281/zenodo.8345429`, [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). [Dataset page →](https://bci.report/datasets/orthosis-error-detection/) · [Credit, licence and consent ↓](https://bci.report/topics/pilot-studies/#credits)

Independently audited: every score and interval of the three configurations recomputed from the sealed predictions, and for the comparison, refits of the main and confound probes reproduced the sealed probabilities exactly. The encoder features were not re-extracted, and not every permutation probe was refitted.

Human · scalp EEG · inner speech in Greek and English · BINGO project

## Inner speech: can imagined words be decoded across people and across languages?

Locally executed experiment · offline · a subject field left out at a time · a negative result

17 subject fields, as the source defines them (not proven to be distinct people), each said 15 matched nouns silently, in Greek and in English, 10 times per noun and language: 5,100 trials. A fixed spectral baseline — relative band power from 19 channels at 300 Hz, and a ridge classifier — is trained on one language of the other fields (2,400 trials) and tested on the held-out field in the same language and in the other, from the 1.5 s after the fixation cross. Both languages of the held-out field stay out of every fit. Chance is the prior of one in fifteen, 6.7%.

**Balanced accuracy for a subject field left out of training, by training and test language.** Dot: the mean over the held-out fields, each weighted equally; line: its approximate descriptive 95% interval; dashed line: the prior, one in fifteen. Rust: within a language; green: across languages.

- Greek → Greek — within a language · 95% interval 5.8%–6.9%: 6.4%
- Greek → English — across languages · 95% interval 5.5%–7.6%: 6.6%
- English → English — within a language · 95% interval 5.8%–7.6%: 6.7%
- English → Greek — across languages · 95% interval 6.2%–7.9%: 7.1%

Means over the 17 held-out fields, each weighted equally, with approximate descriptive 95% intervals. The prior is 6.7% for balanced accuracy and 20.0% for top-3 accuracy. The shuffle column gives the mean of 20 training-label shuffles and the lowest and highest replicate means: an observed range, not an interval.

| Training → test language | Balanced accuracy | Top-3 accuracy | Order-only control | High-frequency ratio control | Training-label shuffles |
| --- | --- | --- | --- | --- | --- |
| Greek → Greek — within a language | 6.4% (95% interval 5.8%–6.9%) | 19.1% (95% interval 18.0%–20.0%) | 7.1% (95% interval 6.4%–7.9%) | 7.1% (95% interval 6.4%–7.8%) | 6.6% (range 5.5% to 7.5%) |
| Greek → English — across languages | 6.6% (95% interval 5.5%–7.6%) | 21.0% (95% interval 19.6%–22.4%) | 6.2% (95% interval 5.4%–7.0%) | 6.4% (95% interval 5.7%–7.1%) | 6.5% (range 5.4% to 7.3%) |
| English → English — within a language | 6.7% (95% interval 5.8%–7.6%) | 19.9% (95% interval 18.2%–21.6%) | 7.4% (95% interval 6.7%–8.1%) | 6.5% (95% interval 5.6%–7.5%) | 6.8% (range 6.1% to 7.8%) |
| English → Greek — across languages | 7.1% (95% interval 6.2%–7.9%) | 19.6% (95% interval 18.3%–20.9%) | 6.5% (95% interval 5.8%–7.3%) | 7.2% (95% interval 6.5%–8.0%) | 6.8% (range 5.5% to 7.7%) |

**Across languages minus within, balanced accuracy, in percentage points.** Dot: the mean over the held-out fields; line: its approximate descriptive 95% interval; dashed line: no difference.

- Mean over both training languages — 95% interval −0.6 to +1.3 pp: +0.3 pp
- Trained on Greek: English minus Greek — 95% interval −1.3 to +1.7 pp: +0.2 pp
- Trained on English: Greek minus English — 95% interval −0.9 to +1.7 pp: +0.4 pp

**+0.3 pp** Across languages minus within, balanced accuracy, the mean over both training languages; 95% interval −0.6 to +1.3 pp

### Near the prior in all four directions

Balanced accuracy runs from 6.4% (Greek → Greek) to 7.1% (English → Greek), and every interval holds the prior of 6.7%; top-3 accuracy runs from 19.1% to 21.0%, against its prior of 20.0%. The contrast’s interval spans zero: no superiority or equivalence between languages, and no significance test. A negative result for one fixed baseline in this setting: it does not show that inner speech cannot be decoded by other methods.

**Delivered signals, from the project’s deliverable [D3.1 – The BINGO Benchmarking Framework, version 2.0 ↗](https://bingo-project.gr/download/397):** The project documents the same preprocessing for both of its released datasets: band-pass filtering from 1 to 145 Hz, notch filtering at 50 Hz and its harmonics, Artifact Subspace Reconstruction (ASR), FORCe removal of ocular and muscle artifacts, and then segmentation into trials. The signals analysed here were therefore artifact-cleaned on the continuous recordings before any trial was cut. As the project states it; not re-verified against the delivered file.

The subject fields are the source’s operational groups; whether they are distinct people, and how recordings group across days, was not independently certified. Physical amplitude units, the named montage and the display latency are not established, and the window, though verified against the stored time vectors, does not certify causal or online processing or a brain-only origin of the signal. No foundation model or deep model was evaluated.

**Reported by the BINGO project, not run here.** On its other dataset, NATO-alphabet inner speech with 26 words, the project’s own leave-one-subject-out evaluation of EEGNet reports a mean accuracy of 3.85% against a chance level of 3.85%: at chance for people the model was not trained on. A different dataset, model and metric, so it is not comparable with the results above. [D3.1 – The BINGO Benchmarking Framework, version 3.0 ↗](https://bingo-project.gr/download/402)

Data: the BINGO (Brain Imagined-Speech Communication) project, CERTH, [datasets page ↗](https://bingo-project.gr/datasets/) · documentation: [D3.1 – The BINGO Benchmarking Framework, version 2.0 ↗](https://bingo-project.gr/download/397). No licence is stated by the source. [Dataset page →](https://bci.report/datasets/bingo-imagined-speech/) · [Credit, licence and consent ↓](https://bci.report/topics/pilot-studies/#credits)

Independently audited: every sealed prediction, aggregate, interval and contrast checked; the main and control models refitted with no difference, and the shuffle models reconstructed rather than refitted. The raw file was not re-extracted: the audit used cached features with verified lineage.

Human · scalp EEG · imagined handwriting, four released coded labels · competition dataset

## Handwriting: how much of a recording with flat or rail-touching channels is usable?

Locally executed experiment · offline · a person left out at a time · a data-quality and methods case study

5 people from the competition’s training split, with all 45 of their sessions: 1,620 events, of which 1,618 are scored; 2 events whose labels contradict each other stay quarantined. Each trial carries one of 4 released coded labels; which character or trajectory each label stands for is not established, so this is not handwriting recognition. Many of the recorded channels sit flat at the lower digital rail declared in the file header. The mask-aware analysis scores each trial only on the channels that pass a fixed availability rule, and sets that EEG against three controls that see no EEG: which channels are missing, the elapsed time and trial order, and both together. Each person is held out whole and scored by a model fitted on the other four; results are averaged over the 5 people with equal weight.

**Balanced accuracy for a person left out of training, four coded labels.** Dot: the mean over the five people, each weighted equally; line: the descriptive 95% person-bootstrap interval; dashed line: the training-majority control, which scores chance for every person. The mask-only control has no interval: with five people, its bounds would show individual people’s scores.

- Mask-aware relative EEG — 95% interval 26.30%–27.20%: 26.81%
- Missingness mask only — no EEG · interval not published: 24.99%
- Elapsed time / trial order — no EEG · 95% interval 24.03%–26.79%: 25.44%
- Mask + time — no EEG · 95% interval 23.90%–26.12%: 25.01%
- Training majority — chance for every person: 25.00%

**The paired difference, mask-aware relative EEG minus each control, in percentage points.** Dot: the mean of the five people’s differences; line: its descriptive 95% person-bootstrap interval, which resamples each person’s difference; dashed line: no difference.

- Minus the missingness mask — 95% interval +1.30 to +2.21 pp: +1.82 pp
- Minus elapsed time / trial order — 95% interval −0.12 to +3.07 pp · crosses zero: +1.37 pp
- Minus mask + time — 95% interval +0.43 to +3.05 pp: +1.80 pp

Means over the 5 held-out people, each weighted equally, with descriptive 95% intervals where they are published. The shuffle column gives the mean of 5 training-label shuffles, which keep each person’s and session’s labels together, and the lowest and highest replicate means: an observed range, not an interval, and a sensitivity check, not a significance test.

| Arm | Balanced accuracy | Training-label shuffles |
| --- | --- | --- |
| Mask-aware relative EEG | 26.81% (95% interval 26.30%–27.20%) | 25.20% (range 23.85% to 26.69%) |
| Missingness mask only — no EEG | 24.99% (interval not published) | 24.95% (range 24.88% to 25.00%) |
| Elapsed time / trial order — no EEG | 25.44% (95% interval 24.03%–26.79%) | 24.70% (range 24.30% to 25.16%) |
| Mask + time — no EEG | 25.01% (95% interval 23.90%–26.12%) | 24.71% (range 24.11% to 25.78%) |
| Training majority — no EEG | 25.00% (chance for every person) | — |

**26.81%** Mask-aware relative EEG: balanced accuracy for a person left out of training. The training-majority control scores 25.00%.

### Just above chance, and not separated from elapsed time and trial order

Its interval, 26.30% to 27.20%, lies above the majority control. Paired person by person, relative EEG minus the missingness mask is +1.82 pp (+1.30 to +2.21 pp) and minus mask + time +1.80 pp (+0.43 to +3.05 pp); minus elapsed time and trial order it is +1.37 pp, and that interval, −0.12 to +3.07 pp, crosses zero. Modest descriptive performance on four coded labels.

### What the masks leave

Every scored trial keeps between 27 and 38 of the 59 recorded EEG channels under the fixed availability rule — not constant, not touching either digital rail, numerically valid — so the recordings are not unusable as a whole. “Available” is that numerical rule, not certified clean EEG, and why the other channels sit at the rail is not known. Across 95,462 channel-trials, 50,811 carry the constant flag and 50,815 the lower-rail flag; the two flags overlap, so they are not summed. No upper-rail, non-finite or invalid-denominator flag occurred. These counts describe the five people analysed, not the whole dataset.

Scored trials by how many of the 59 recorded EEG channels are available.

| Available channels | Trials |
| --- | --- |
| 27 | 1,429 |
| 28 | 65 |
| 29 | 16 |
| 30 | 36 |
| 37 | 36 |
| 38 | 36 |

An earlier analysis of the same 5 people and 1,618 trials, with every channel on a common-average reference and log band power, scored 27.05% with logistic regression and 27.42% with shrinkage LDA. Its checks found a sample at a digital rail in 1,618 of 1,618 windows, and in 5 windows inspected afterwards, 148 of 295 channel-windows flat at the lower rail. It is the context for the mask-aware analysis, not a second cohort. Masking, referencing and the spectral representation changed together, so the difference between the two analyses cannot be attributed to masking, and no best model is chosen from them.

These five people were the development cohort of the earlier analysis, whose scores were seen before this design: a quality-informed diagnostic, not independent confirmation, and no population inference. The official ten-person test data were not used. The optical species, units and the clock between the two modalities are unresolved, so no fNIRS or fusion conclusion follows, and there is no online, physiological-origin, competition-standing or medical claim.

Data: [lasfk on Hugging Face · EEG-fNIRS-based Handwriting Trajectory Dataset ↗](https://huggingface.co/datasets/lasfk/EEG-fNIRS-based-Handwriting-Trajectory-Dataset), `revision 4036680b5fb0015472d55589da2f34c5a87a3c10`, [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) as declared on the dataset card, gated. No ethics-approval or informed-consent statement was found; the aggregates are published with that gap stated, by the owner’s decision. [Dataset page →](https://bci.report/datasets/eeg-fnirs-handwriting/) · [Credit, licence and consent ↓](https://bci.report/topics/pilot-studies/#credits)

Independently audited: all 120 stored models reconstructed and the 20 true-label models refitted, with no discrepancy. The shuffle models were not refitted, the raw samples were not reread and the features were not re-extracted.

Human · scalp EEG, stored clips · self-reported emotion · REFED

## Emotion ratings: what does a stimulus-identity control reveal?

Locally executed computational diagnostic · offline · two nominal folders · no interval

2 nominal participant folders watched the same 15 videos: 30 stored EEG clips, not as many independent people. For each clip the target is the mean of its stored valence and arousal ratings on the published raw scale from 1 to 255: descriptive self-report means, not SAM scores. Four fixed arms are scored by equal-axis mean absolute error, lower being better: EEG Ridge on relative band power at every channel position, and three controls that use no EEG — the training clips’ mean, the training target of the same video, and a ridge on stored duration and the numeric video ID. Two conditions: a folder held out with the videos shared (2 folds), and a folder held out together with 3 of the videos it is tested on (10 folds).

**Equal-axis mean absolute error with the videos shared and a folder held out, raw label units: lower is better.** Dot: the error over all clips of the condition; dashed line: the source global mean’s error. No interval: two nominal folders support no population inference.

- EEG Ridge (relative band power): 107.182
- Source global mean — uses no EEG: 40.287
- Source video lookup — uses no EEG: 27.325
- Duration / numeric-video-ID Ridge — uses no EEG: 37.910

**Equal-axis mean absolute error with a folder and its test videos held out together, raw label units: lower is better.** Dot: the error over all clips of the condition; dashed line: the source global mean’s error. No interval: two nominal folders support no population inference.

- EEG Ridge (relative band power): 91.984
- Source global mean — uses no EEG: 41.032
- Source video lookup — uses no EEG: 41.032
- Duration / numeric-video-ID Ridge — uses no EEG: 47.098

Equal-axis mean absolute error over the 30 clips of each condition, in raw label units; lower is better. In the second condition the lookup finds a matching video for 0 of 30 clips, so it falls back to the global mean.

| Arm | Shared videos, a folder held out | Folder and videos held out together |
| --- | --- | --- |
| EEG Ridge (relative band power) | 107.182 | 91.984 |
| Source global mean — uses no EEG | 40.287 | 41.032 |
| Source video lookup — uses no EEG | 27.325 | 41.032 |
| Duration / numeric-video-ID Ridge — uses no EEG | 37.910 | 47.098 |

**107.182** EEG Ridge, equal-axis mean absolute error with the videos shared; the same-video lookup scores 27.325, the global mean 40.287

### EEG band power has the largest error in both conditions

Held out together with the videos, EEG Ridge’s error is 91.984 against 41.032 for the global mean. With shared videos, looking up the same video’s training target has the smallest error: a familiar stimulus carries the prediction through its identity, which is what that control is there to show. Held out together, the lookup has no matching video and equals the global mean. This illustrates the confound; it does not establish a mechanism.

Two nominal folders sharing their video keys: the clips are not independent people, so no interval or significance test is reported, and no population, cross-day or physiological claim follows. The rating arrays had been inspected and their means computed before this design, so these are not untouched targets. Stored EEG units, the EEG-to-rating clock and the crop end points are unresolved: published video durations exceed the stored arrays by two seconds, and pairing whole arrays establishes no precise alignment. The numeric video ID is a key index, not an established presentation order.

Data: Xiaojun Ning, Jing Wang, Zhiyang Feng, Tianzuo Xin, Shuo Zhang, Shaoqi Zhang, Zheng Lian, Yi Ding, Youfang Lin and Ziyu Jia, `REFED2025/REFED-dataset · revision 0298355c2823946ac5800b2ec12a082bf2dcd01b`, [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/), gated. [Dataset page →](https://bci.report/datasets/refed/) · [Credit, licence and consent ↓](https://bci.report/topics/pilot-studies/#credits)

Independently audited: the sealed predictions reconstructed exactly, every metric recomputed independently and every preparation record verified. No raw signal or original annotation was reread, and no model was refitted.

Status only · no figure

## Not yet evaluated

These analyses have not been run: the evidence a valid evaluation needs is missing, and none of them is a result.

### Orthosis error-detection dataset: EMG and paired EEG–EMG analyses

**No EMG, TinyMyo or EEG–EMG fusion result exists for the Orthosis dataset.**

The exact clock anchors between the EMG sample rows and the EEG events, the latency and end points, and the meaning and units of the exported columns are not established. A nominal sampling rate alone does not establish the labels.

### EEG-fNIRS handwriting dataset: optical and fused analyses

**No fNIRS or EEG–fNIRS fusion result exists for the handwriting dataset.**

The optical species and column meanings, their units and a verified clock between the two modalities are not established.

### REFED: physiological, pointwise and fused analyses

**No physiological or precisely aligned fusion evaluation exists for REFED beyond the stored-clip computational diagnostic.**

Physical calibration, the crop boundaries, the mapping of the six optical components and paired timing are not established.

### Synchronized multimodal meditation and rest dataset (Zenodo): EEG phase labels and paired analyses

**No meditation EEG phase prediction or paired analysis exists.**

An explicit association between events and records, and clock anchors, are missing. One inspected record has a constant marker column, and that finding is not generalised to the other records. In this dataset RR means heartbeat RR intervals, not respiratory rate.

Methods & limits

## What these pilots can and cannot say

### How each result was produced

### Orthosis, classical features

The 64 EEG channels at 500 Hz with the declared header scaling, the recorded reference kept and the acceleration channels left out; continuous forward-only 1–30 Hz filtering that uses no future samples; the complete 0–0.5 s window (250 samples) after each event. 576 fixed amplitude and log band-power values per event (522 on the 58 channels), a training-only standardization and a balanced L2 logistic regression with C = 1, no tuning. Main-task runs only: 8 baseline runs, 82 targets without a following movement and 1 window crossing a boundary are left out, none depending on whether the person responded correctly. Every prediction was sealed before a separate scorer read the held-out labels.

### Orthosis, frozen EEGPT

The frozen 25,287,168-parameter EEGPT target encoder at its native input of 58 channels by 128 samples: the same filtered window, resampled from 500 to 256 Hz, microvolts divided by 1,000; its summary tokens, averaged over two temporal patches, feed the same probe. The encoder’s weights are unchanged: no fine-tuning and no LoRA. Across the 8 folds, 344 probes were fitted.

### BINGO, inner speech

Per trial and channel, max-absolute scaling and demeaning, then Hann periodograms; four relative bands partition 1–30 Hz across the 19 channels in source order, 76 values per trial, with no voltage calibration, montage interpolation or re-referencing. A source-only standardization and a ridge classifier with fixed alpha 1. Controls: an order-only control, a high-frequency ratio control, the class prior and 20 training-label shuffles per direction; all 816 prediction checkpoints were sealed before scoring. One incomplete field is left out by a complete-record rule set before scoring. The protocol, from the project’s D3.1, version 2.0: the word is shown at −2.5 s, removed for a countdown at −1.0 s, and the fixation cross at 0.0 s starts inner speech until 1.5 s; 15 nouns, 10 repetitions each, in fully randomized order.

### Handwriting, mask-aware EEG

A fixed window 0.5–4.5 seconds after the released EEG-relative onset (4,000 samples at 1,000 Hz), read from the 59 header-identified EEG channels only; the EOG, HECG and annotation channels and the optical samples are left out. Labels are the released event codes 200 to 203, mapped to 0 to 3. Before any transform, a channel-window is masked if it is constant, touches either header-declared digital rail or is not finite; the available channels are demeaned one by one, with no common-average reference and no interpolation. Welch spectra (1,000-sample Hann window, 500 overlap) give five half-open bands from 1 to 45 Hz, each divided by the sum of the five: 295 channel-major values per trial, zero where a channel is unavailable. Source-only standardization and a balanced multinomial logistic regression with C = 1, no tuning. The controls fit the same model on the availability bits and the available-channel fraction, on four elapsed-time and trial-order features, and on both. All 125 checkpoints, for 120 model fits and five majority controls, were sealed before a separate scorer read the held-out labels.

### REFED, stored clips

Five relative Welch band powers between 1 and 45 Hz, each divided by the channel’s total, at every channel position, with no channel selection or re-referencing; source-only standardization and Ridge with fixed alpha 1, no tuning, and predictions left unclipped. Every clip passed the numerical checks (1,920 channel–clip rows, 0 removed): finite, non-constant values and valid spectral denominators, which is numerical eligibility, not signal quality.

### What the intervals cover

Orthosis: pointwise 95% percentile intervals from 10,000 person-level bootstrap draws, conditional on the sealed predictions; with eight people they are wide, and the paired interval resamples each person’s difference rather than comparing two separate intervals. BINGO: approximate descriptive intervals from 10,000 field bootstrap draws over overlapping leave-one-field-out fits; they omit refit variability and are not calibrated population confidence intervals. Handwriting: descriptive 2.5–97.5% intervals from 10,000 fixed-seed person-bootstrap draws over the five people, conditional on the already fitted overlapping folds; they are not calibrated population confidence intervals and not evidence of superiority. Each bound is a percentile of resampled five-person means, not one person’s score; the mask-only control’s interval is not published, because its bounds would show individual people’s scores. REFED: none, because two nominal folders support no population inference. The permutation and shuffle ranges are the lowest and highest replicate means, not intervals, and those controls are diagnostics, not significance tests.

### Limits that hold for all four

- Small pilots, not ranked: each answers its own question on its own dataset, task and metric, and no result is set against another or against the core matrix.
- Nothing pooled: the three Orthosis results are one cohort, counted once, and no figure combines datasets, configurations, languages or conditions.
- Reanalyses executed locally by this site, not results reported by the datasets’ authors; the BINGO project’s own figure is printed as theirs, apart from ours.
- Offline only: no online, real-time, clinical or deployment claim.
- Fixed before scoring: no channel, feature or hyperparameter search, no score used to choose a setting, and every prediction sealed before scoring.
- Audit scope. Each result passed an independent audit; the Orthosis encoder features were not re-extracted, the BINGO raw file was not re-read, the handwriting shuffle models were not refitted, and no REFED model was refitted.

### Not published here

Per-person, per-field, per-folder, per-clip and per-record values of any kind, including per-person minima, maxima and per-fold scores; the response-clean subset’s scores; individual permutation and shuffle replicates; confusion matrices and per-class recall; the handwriting mask-only control’s interval; identifiers, labels, features, predictions, fitted parameters and checkpoints; EEGPT weights, original or converted; measured compute; and demographics. Why cohorts this small publish so little: [small cohorts, on the data-use page →](https://bci.report/data-use/#small-cohorts-pilots)

### Credits, licences, and consent

### Orthosis error-detection EEG and EMG dataset (Kueper et al. 2024)

Niklas Kueper, Kartik Chari, Judith Bütefür, Julia Habenicht, Tobias Rossol, Su Kyoung Kim, Marc Tabie, Frank Kirchner and Elsa Andrea Kirchner · EEG and EMG dataset for the detection of errors introduced by an active orthosis device (IJCAI’23 CC6 Competition), Zenodo, doi:10.5281/zenodo.8345429, CC BY 4.0. Paper: Niklas Kueper et al., EEG and EMG dataset for the detection of errors introduced by an active orthosis device, Frontiers in Human Neuroscience 18 (2024), doi:10.3389/fnhum.2024.1304311. Derived analysis by BCI Report; not endorsed by the authors.

**Ethics and consent, as the source states them:** Ethics approval: Approved by the Ethics Committee of the Department of Computer Science and Applied Cognitive Science of the Faculty of Engineering at the University of Duisburg-Essen; no approval number is given. Informed consent: The participants gave written informed consent and were told they could stop at any time.

[Dataset record ↗](https://doi.org/10.5281/zenodo.8345429) · [Paper ↗](https://doi.org/10.3389/fnhum.2024.1304311) · [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) · [Statement read from ↗](https://europepmc.org/article/PMC/PMC10839100)

### BINGO cross-linguistic inner-speech EEG dataset (Greek and English)

BINGO (Brain Imagined-Speech Communication) project, CERTH · cross-linguistic inner-speech EEG dataset, https://bingo-project.gr/datasets/; no licence is stated by the source. Documentation: Fotis P. Kalaganis, Kostas Georgiadis, Maria Kyrou, Spiros Nikolopoulos and Ioannis Kompatsiaris, D3.1 – The BINGO Benchmarking Framework, version 2.0, public deliverable (2025), https://bingo-project.gr/download/397. Derived analysis by BCI Report; not endorsed by the authors.

**Ethics and consent, as the source states them:** Ethics approval: No ethics-approval reference is published by the source. Informed consent: The project’s public deliverable D3.1, version 3.0, states that each participant signed a consent form before the sessions. It says so for the project’s 20-participant NATO-alphabet dataset, recorded in the same CERTH laboratory; the cross-language dataset’s own documentation, D3.1 version 2.0, has no ethics or consent statement.

[Dataset record ↗](https://bingo-project.gr/datasets/) · [D3.1 – The BINGO Benchmarking Framework, version 2.0 ↗](https://bingo-project.gr/download/397) · [D3.1 – The BINGO Benchmarking Framework, version 3.0 ↗](https://bingo-project.gr/download/402) · No licence stated

### EEG-fNIRS-based Handwriting Trajectory Dataset (Hugging Face)

lasfk on Hugging Face · EEG-fNIRS-based Handwriting Trajectory Dataset, revision 4036680b5fb0015472d55589da2f34c5a87a3c10, CC BY-NC-SA 4.0 as declared on the dataset card, gated access. Derived analysis by BCI Report; not endorsed by the uploader.

**Ethics and consent:** No ethics-approval or informed-consent statement was found. The dataset card has none; a participating team’s challenge paper, which identifies the competition as the ACM MM 2026 Multimodal Brain-Computer Interface Grand Challenge, has none; and the organisers’ paper is listed there as forthcoming. The aggregates are published with this gap stated, by the owner’s decision, to be rechecked when the organisers’ paper appears.

[Dataset record ↗](https://huggingface.co/datasets/lasfk/EEG-fNIRS-based-Handwriting-Trajectory-Dataset) · [A participating team’s challenge paper ↗](https://arxiv.org/abs/2608.03176v2) · [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)

### REFED: real-time dynamic labelled EEG-fNIRS emotion dataset (Ning et al. 2025)

Xiaojun Ning, Jing Wang, Zhiyang Feng, Tianzuo Xin, Shuo Zhang, Shaoqi Zhang, Zheng Lian, Yi Ding, Youfang Lin and Ziyu Jia · REFED: A Subject Real-time Dynamic Labeled EEG-fNIRS Synchronized Recorded Emotion Dataset, NeurIPS 2025 Datasets and Benchmarks Track. Data: REFED2025/REFED-dataset on Hugging Face, revision 0298355c2823946ac5800b2ec12a082bf2dcd01b, CC BY-NC-SA 4.0, gated. Derived analysis by BCI Report; not endorsed by the authors.

**Ethics and consent, as the source states them:** Ethics approval: Approved by the local Institutional Review Board; the paper’s datasheet names the approval IRB-CASIA. Informed consent: All participants gave informed consent to the recording of their physiological signals and rating scales, and agreed that the data would be used only for non-commercial scientific research.

[Dataset record ↗](https://huggingface.co/datasets/REFED2025/REFED-dataset) · [Paper ↗](https://papers.neurips.cc/paper_files/paper/2025/file/2bf0ed7c35d9d84128e7f7c72ab76402-Paper-Datasets_and_Benchmarks_Track.pdf) · [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)

### EEGPT, the model in the Orthosis comparison

Paper: [EEGPT: Pretrained Transformer for Universal and Reliable Representation of EEG Signals ↗](https://openreview.net/forum?id=XNiFuoqTuD); code: [BINE022/EEGPT ↗](https://github.com/BINE022/EEGPT) at `a0e0a8fad729e2ecf4eedb3a81548a6e6d48a705`; checkpoint `eegpt_mcae_58chs_4s_large4E.ckpt`, SHA-256 `9d63ebc80bba1ad5e3277df4eecfa05d2745e6243a12ff743c4c0a93930dfac0`.

Apache-2.0 covers the repository’s source code only; the checkpoint’s own licence is unresolved, and no original or converted weights are distributed.

Derived analyses by BCI Report, aggregates only; not endorsed by the data’s authors, and no raw data are redistributed.

Data source: [pilot-studies-update.json](https://bci.report/data/pilot-studies-update.json) · schema bci-report-pilot-studies-update-v1.

## Related questions

- Signals other than scalp EEG, human fNIRS and one macaque’s intracortical spiking, each read on its own: [Do simple baselines work on signals other than scalp EEG?](https://bci.report/topics/beyond-scalp-eeg/)
- Frozen foundation-model encoders against matched randomly initialized ones, and further checkpoints in one frozen recipe: [Does pretraining help EEG foundation models like LaBraM and CBraMod?](https://bci.report/topics/does-pretraining-help/)
- Its entry in the model directory, with its licence status: [EEGPT](https://bci.report/methods/#eegpt).

Keep exploring

Transfer

- [— Sensor transfer **Dry vs. wet electrodes**](https://bci.report/topics/dry-vs-wet/)
- [— Context transfer **Screen to VR**](https://bci.report/topics/screen-to-vr/)
- [— Montage **Fewer electrodes**](https://bci.report/topics/fewer-electrodes/)
- [— Motion robustness **On the move**](https://bci.report/topics/on-the-move/)
- [— Session transfer · same person **Later sessions**](https://bci.report/topics/later-sessions/)

Adapting models

- [— Calibration budget **How much calibration?**](https://bci.report/topics/calibration-budget/)
- [— Model adaptation **Which part to update?**](https://bci.report/topics/model-adaptation/)
- [— Representation controls **Does pretraining help?**](https://bci.report/topics/does-pretraining-help/)
- [— Shared encoder · route 2 **One model, several questions**](https://bci.report/topics/shared-encoder/)
- [— Questions in language · route 3 **Questions in language**](https://bci.report/topics/questions-in-language/)

Reliability & clinical

- [— Abstention · decision research **When not to act**](https://bci.report/topics/when-not-to-act/)
- [— Sleep staging · simple baselines **Sleep-stage balance**](https://bci.report/topics/sleep-staging/)
- [— Clinical research **Clinical groups**](https://bci.report/topics/clinical-groups/)

Other signals & pilots

- [— Other signals · human and macaque **Beyond scalp EEG**](https://bci.report/topics/beyond-scalp-eeg/)
- [— Pilot studies · small cohorts **Pilot studies**](https://bci.report/topics/pilot-studies/)

[All questions, and the map of which kinds of transfer have been measured →](https://bci.report/topics/)

## Cite this page

BCI Report (2026). *What can a small pilot show, and what can it not?* https://bci.report/topics/pilot-studies/

Figures from release [`pilot-studies-update-20261012`](https://bci.report/releases/#pilot-studies-update-20261012) (2026-10-12). Cite the upstream datasets as well: their credits are on this page.

Every release is archived on Zenodo: [doi:10.5281/zenodo.23123296](https://doi.org/10.5281/zenodo.23123296). [BibTeX for the site and its releases →](https://bci.report/api/#cite-heading) · [CITATION.cff ↗](https://raw.githubusercontent.com/twu3202/bci-report/main/CITATION.cff)

---
Markdown copy of https://bci.report/topics/pilot-studies/, generated from the published page. Figures are aggregate results; terms of use: https://bci.report/data-use/