Injected orthosis device errors against standard device events, EEG only: one cohort in three configurations
Orthosis error-detection dataset (Kueper et al. 2024): EEG decoding results
The EEG and EMG dataset for the detection of errors introduced by an active orthosis device (Zenodo 8345429) holds recordings from eight healthy subjects whose right arm an active orthosis moved in elbow flexion and extension; during some movements an error was deliberately introduced, and the orthosis briefly moved the opposite way. EEG was recorded with a 64-channel system at 500 Hz, alongside eight-channel EMG. Prepared for the IJCAI 2023 competition CC6 (IntEr-HRI) and released in three versions, the third the complete dataset, by Niklas Kueper and colleagues (DFKI and the University of Duisburg-Essen) under CC BY 4.0; described in Frontiers in Human Neuroscience (2024). Only the EEG is used here.
Also known asEEG and EMG dataset for the detection of errors introduced by an active orthosis device · IJCAI'23 CC6 IntEr-HRI competition dataset · Zenodo 8345429 · Kueper et al. (2024)
Fixed EEG features + logistic regression · 64 channels
73.0% (68.0%–77.7%)
Direction/time confound probe (no EEG)
51.4% (50.4%–52.2%)
0%25%50%75%100%
Balanced accuracy, 2 configurations. Dot: the estimate; line: 95% interval; dashed line: chance level where the payload records one.
Method
Condition
Metric
Value
People
Fixed EEG features + logistic regression · 64 channels
A person left out of training · 0–0.5 s after each recorded device event
Balanced accuracy
73.0%68.0%–77.7%
8
Fixed EEG features + logistic regression · 64 channels
A person left out of training · 0–0.5 s after each recorded device event
AUROC
0.8120.751–0.863
8
Fixed EEG features + logistic regression · 64 channels
A person left out of training · 0–0.5 s after each recorded device event
Average precision
0.5850.492–0.672
8
Direction/time confound probe (no EEG)reference or comparator, not a decoding model
A person left out of training · 0–0.5 s after each recorded device event
Balanced accuracy
51.4%50.4%–52.2%
8
Training class priorreference or comparator, not a decoding model
A person left out of training · 0–0.5 s after each recorded device event
Balanced accuracy
50.0%
8
Fixed EEG features + logistic regression · 64 channels
Minus the confound probe, matched person by person
Paired difference
+21.6 ppNo interval was computed for this difference. EEG was above the confound probe for 8 of 8 people.
8
Permutation control: training labels shuffled within person and run · Fixed EEG features + logistic regression · 64 channelsreference or comparator, not a decoding model
A person left out of training · 0–0.5 s after each recorded device event
Balanced accuracy
50.1%The mean of 20 replicates, whose means ranged from 48.0% to 52.6%: an observed range, not a confidence interval.
8
One cohort — 8 people, 83 runs, 2,407 recorded device events, 474 of them injected errors — published as three configurations and counted once. The labels are recorded device events: the results do not show that a person recognised an error, and movement, muscle or device artifacts may contribute. Errors are a minority of events, so average precision is read against that share. Outside every leaderboard and model ranking.
A response-clean subset is counts only: 451 events, 446 errors and 5 standard events, with 3 people having both classes, so its scores are not published.
Pilot studies · orthosis error events, the matched 58-channel classical configuration (the same cohort)
Fixed EEG features + logistic regression · 58 channels
71.0% (64.8%–76.9%)
Direction/time confound probe (no EEG)
51.4% (50.4%–52.3%)
0%25%50%75%100%
Balanced accuracy, 2 configurations. Dot: the estimate; line: 95% interval; dashed line: chance level where the payload records one.
Method
Condition
Metric
Value
People
Fixed EEG features + logistic regression · 58 channels
A person left out of training · 0–0.5 s after each recorded device event
Balanced accuracy
71.0%64.8%–76.9%
8
Fixed EEG features + logistic regression · 58 channels
A person left out of training · 0–0.5 s after each recorded device event
AUROC
0.7930.719–0.854
8
Fixed EEG features + logistic regression · 58 channels
A person left out of training · 0–0.5 s after each recorded device event
Average precision
0.5440.438–0.654
8
Direction/time confound probe (no EEG)reference or comparator, not a decoding model
A person left out of training · 0–0.5 s after each recorded device event
Balanced accuracy
51.4%50.4%–52.3%
8
Permutation control: training labels shuffled within person and run · Fixed EEG features + logistic regression · 58 channelsreference or comparator, not a decoding model
A person left out of training · 0–0.5 s after each recorded device event
Balanced accuracy
50.1%The mean of 20 replicates, whose means ranged from 48.2% to 51.7%: an observed range, not a confidence interval.
8
The same cohort, labels and caveats as the 64-channel baseline above, counted once. The 58 channels are the ones EEGPT supports, fixed before scoring. A second configuration of the same people, not a replication: the 64-channel scores were seen before it was frozen, and no conclusion about how many channels are needed follows.
Pilot studies · orthosis error events, frozen EEGPT against the matched classical features (58 channels, the same cohort)
EEGPT · frozen encoder + the same supervised probe · 58 channels
70.3% (66.2%–73.9%)
Fixed EEG features + logistic regression · 58 channels
71.0% (64.8%–76.9%)
0%25%50%75%100%
Balanced accuracy, 2 configurations. Dot: the estimate; line: 95% interval; dashed line: chance level where the payload records one.
Method
Condition
Metric
Value
People
EEGPT · frozen encoder + the same supervised probe · 58 channels
A person left out of training · 0–0.5 s after each recorded device event
Balanced accuracy
70.3%66.2%–73.9%
8
EEGPT · frozen encoder + the same supervised probe · 58 channels
A person left out of training · 0–0.5 s after each recorded device event
AUROC
0.7800.721–0.827
8
EEGPT · frozen encoder + the same supervised probe · 58 channels
A person left out of training · 0–0.5 s after each recorded device event
Average precision
0.5640.489–0.630
8
Fixed EEG features + logistic regression · 58 channels
A person left out of training · 0–0.5 s after each recorded device event
Balanced accuracy
71.0%64.8%–76.9%
8
EEGPT · frozen encoder + the same supervised probe · 58 channels
Minus the matched classical features, the same people and folds
Paired difference
−0.6 pp−4.0–+3.2 ppThe paired interval spans zero: neither arm is shown to be better, and the two are not shown to be equivalent.
8
Permutation control: training labels shuffled within person and run · EEGPT · frozen encoder + the same supervised probe · 58 channelsreference or comparator, not a decoding model
A person left out of training · 0–0.5 s after each recorded device event
Balanced accuracy
49.8%The mean of 20 replicates, whose means ranged from 47.6% to 52.2%: an observed range, not a confidence interval.
8
The same cohort, channels, folds and probe as the 58-channel configuration above, counted once. Supervised transfer: the frozen encoder’s outputs feed a probe trained on the other people’s labels, not zero-shot. Overlap between this dataset and EEGPT’s pretraining data has not been excluded, and the 64-channel classical scores were seen before this comparison was frozen. EEGPT’s checkpoint licence is unresolved and no original or converted weights are distributed; Apache-2.0 covers the repository’s code only. Not evidence about EEG foundation models in general.
Source and licence
Orthosis error-detection dataset (Kueper et al. 2024)
CreditNiklas Kueper, Kartik Chari, Judith Bütefür, Julia Habenicht, Tobias Rossol, Su Kyoung Kim, Marc Tabie, Frank Kirchner and Elsa Andrea Kirchner · EEG and EMG dataset for the detection of errors introduced by an active orthosis device (IJCAI’23 CC6 Competition), Zenodo, doi:10.5281/zenodo.8345429, CC BY 4.0. Paper: Niklas Kueper et al., EEG and EMG dataset for the detection of errors introduced by an active orthosis device, Frontiers in Human Neuroscience 18 (2024), doi:10.3389/fnhum.2024.1304311. Derived analysis by BCI Report; not endorsed by the authors.
BCI Report does not redistribute any recording. These are aggregate measurements computed by BCI Report under the licence above; the data remain their creators’.
BCI Report