{ "schema_version": "bci-report-shared-representation-update-v1", "release_id": "shared-representation-update-20261007", "generated_at": "2026-10-07", "status": "aggregate_preview", "metric_units": "balanced accuracy, AUROC, Cohen's kappa, shares and the arms' means are proportions in [0,1]; estimate_pp and interval_95_pp are percentage points of balanced accuracy (x minus y); log_r is the natural log of R; log loss is in nats per window; step_time_s and batch_time_s are seconds, indicative; counts of people, windows, parameters and multiply-accumulates are whole numbers", "scope": "Route 2 of the decision-research roadmap: shared encoders with fixed heads, question-conditioned heads and independent models at matched data and compute, on motor imagery (OpenBMI) and sleep (BOAS), with EESM19 as a crude replication. Every figure compares set-ups; none is a deployment error rate, and nothing is ranked where intervals overlap. Every BOAS figure travels with the three stated gaps and the attribution in boas_conditions.", "results": { "one-representation": { "id": "one-representation", "title": "One representation, several questions: shared encoders, fixed heads and question-conditioned heads", "run_id": "decision-route2-v1/20261006", "design": { "questions_asked_by": "Every question is a label-backed classification target from one dataset, asked by identifier: a learned 16-dimensional embedding of the question's index, never language.", "levels": { "E1": "EEGNet trained from scratch in its published configuration; the shared trunk has 2,096 parameters on motor imagery and 1,200 on sleep. Three seeds. Primary.", "L1": "Frozen CBraMod features with the heads trained on top. Three seeds. Primary.", "E2": "CBraMod adapted by LoRA (rank 4) together with the heads. One seed. Secondary; on sleep it ran after every other result was known (owner decision, 7 October 2026)." }, "arms": { "A": "One independent model per question.", "B-lin": "One shared encoder with one linear head per question: the fixed heads.", "B-sh": "One shared encoder, one shared hidden layer (width 66 on E1 motor imagery by the ±10% parameter rule, 64 elsewhere) and one output per question.", "C1": "B-sh plus FiLM conditioning from the question's 16-dimensional embedding: the question-conditioned head.", "-fz": "The same head on frozen CBraMod features; B-lin-fz-sgd trains the linear heads by SGD." }, "matching": "B-lin, B-sh and C1 share initialisation, batch order and step count, see the same windows and labels and run one encoder pass per window. A trains one encoder per question; that cost is what sharing saves, and the ledger counts it.", "folds": "Five person-disjoint outer folds per dataset (all sessions or nights of a person in one fold); no validation set, no early stopping; the final epoch is scored.", "metric": "Balanced accuracy averaged per person, then equally over people. It compares set-ups and is not a deployment error rate. P3 and S4 compare AUROC through log R, R = (1 - AUROC of the dedicated model) / (1 - AUROC of the read-out).", "single_seed": "A one-seed entry resamples people only and carries no training-run variance.", "canary": "Before any fit, a ridge classifier on each window's per-channel mean and log-variance, trained and tested within the first fold's training people only. A question it answers at a balanced accuracy of 0.85 or more is demoted: reported, never counted. No question was demoted.", "headroom_gate": "A question's entries count for a route sentence only if the 95% interval of the level's fixed-head (B-lin) arm lies above chance + 5 pp and below 95%; otherwise they are reported and labelled floor or ceiling. For P3 only the floor applies: the read-out's AUROC interval must lie above 0.55.", "freeze": "The protocol was frozen on 2026-10-06 at 07:17 UTC and amended eight times for engineering only (A1-A8) before the first stage-1 fit at 09:24 UTC; delta, the P3 cut, the canary and the gates are identical in every version. Outer-test labels were masked until scoring. Nothing was selected on test results; no fit failed and nothing was re-run.", "margin": { "delta_pp": 2.0, "p3_min_error_removed": 0.2, "fixed": "by the owner on 2026-10-06, at the freeze, before any pilot result" }, "flags": { "difference": "yes when the paired 95% interval excludes 0, with its direction (e.g. 'C1 higher'); otherwise 'no difference shown'", "margin": "'equivalent within delta' when the whole interval lies inside (-delta, +delta); otherwise 'non-inferior' when the relevant bound alone meets the margin (P1, P4, P5: upper bound of conditioned - fixed below delta; P2: lower bound of shared - independent above -delta; P3: lower bound of log R above log 0.8); otherwise 'margin not met'", "independence": "The two flags are computed separately and both are always printed. An interval such as [+0.4, +1.6] pp is 'difference: C1 higher' and 'equivalent within delta'; [-3, -1] pp is 'difference: B-lin higher' and 'non-inferior'.", "inconclusive": "an entry with 'no difference shown' and 'margin not met' is worded 'inconclusive at this sample size'" }, "margin_bound": { "upper": "P1, P4, P5 and their analogues: the upper bound of the more complex arm minus the simpler one below +delta", "lower": "P2, S10 and S12: the lower bound of shared minus independent above -delta", "log_r": "P3 and S4: the lower bound of log R above log 0.8" }, "route_sentence_rule": "stated for a domain only if (a) at least one primary question of the domain passes the canary and the headroom gate; (b) every such question has P1 margin flag 'non-inferior' or 'equivalent within delta'; (c) the ledger rule makes B-lin 'for less' than C1. The sentence must name any question where P1's difference flag is 'C1 higher', with its interval, and list the questions excluded by the gates.", "uncertainty": "Hierarchical paired bootstrap: 2,000 draws, seed 20261006; people with replacement, then seed indices per outer fold; percentile 95% intervals. Single-seed entries resample people only and carry no training-run variance.", "multiplicity": "21 pre-declared primary entries (P1 5, P2 5, P3 1, P4 5, P5 5), no correction, each with both flags; at 95% about one in 20 null entries may show a 'difference' by chance, which every write-up states. The route sentence uses an all-questions rule (intersection), which does not inflate its error. Everything else is secondary, exploratory or descriptive and labelled so." }, "datasets": { "openbmi": { "dataset": "OpenBMI", "domain": "motor imagery", "role": "primary", "description": "OpenBMI (Lee et al. 2019, GigaDB 100542), published 51-person cohort, 2 days each", "people": 51, "windows": { "unit": "2-s windows", "qc_passing": 31653 }, "questions": [ { "id": "MI-A", "question": "Imagining a hand movement, or resting?", "answers": [ "rest", "imagery" ], "role": "primary", "defined_on": "all windows", "windows": 31653, "people": 51, "counts": { "rest": 11356, "imagery": 20297 }, "shares": { "rest": 0.3587653618930275, "imagery": 0.6412346381069725 } }, { "id": "MI-B", "question": "Which hand?", "answers": [ "left", "right" ], "role": "primary", "defined_on": "offline (EEG_MI_train) trial windows only", "windows": 10199, "people": 51, "counts": { "left": 5099, "right": 5100 }, "shares": { "left": 0.49995097558584173, "right": 0.5000490244141582 } }, { "id": "MI-C", "question": "Offline or online run?", "answers": [ "offline", "online" ], "role": "secondary", "defined_on": "trial windows", "windows": 20297, "people": 51, "counts": { "offline": 10199, "online": 10098 }, "shares": { "offline": 0.5024880524215402, "online": 0.49751194757845985 } }, { "id": "MI-D", "question": "Day 1 or day 2?", "answers": [ "day1", "day2" ], "role": "secondary", "defined_on": "all windows", "windows": 31653, "people": 51, "counts": { "day1": 15867, "day2": 15786 }, "shares": { "day1": 0.5012794995735002, "day2": 0.49872050042649985 } } ], "canary": { "training_people": 40, "demotes_at": 0.85, "questions": [ { "id": "MI-A", "balanced_accuracy": 0.6607060088905852, "balanced_accuracy_unfiltered": 0.6509280636177694, "label": "passes canary" }, { "id": "MI-B", "balanced_accuracy": 0.5593661616161617, "balanced_accuracy_unfiltered": 0.6274911616161616, "label": "passes canary" }, { "id": "MI-C", "balanced_accuracy": 0.6011549105199789, "balanced_accuracy_unfiltered": 0.5813955028941106, "label": "passes canary" }, { "id": "MI-D", "balanced_accuracy": 0.4733311794962999, "balanced_accuracy_unfiltered": 0.5322482596426976, "label": "passes canary" } ] }, "route_sentence": { "sentence": "The fixed-heads sentence is not supported", "supported": false, "counted": [ "MI-A", "MI-B" ], "excluded": {}, "conditioned_head_higher": { "MI-A": [ 0.08817701894528421, 2.89441170995498 ] }, "margin_met_on_every_counted_question": false, "fixed_heads_for_less": true, "fixed_heads_lower_on": [ "parameters", "head_macs" ], "fixed_heads_higher_on": [] }, "entries": [ { "id": "P1", "role": "primary", "level": "E1", "question": "MI-A", "x": "C1", "y": "B-lin", "included_people": 51, "estimate_pp": 1.4754393517638422, "interval_95_pp": [ 0.08817701894528421, 2.89441170995498 ], "difference": "difference: C1 higher", "margin": "margin not met", "wording": "difference: C1 higher; margin not met", "single_seed": false, "gate": "pass", "mean_balanced_accuracy": { "C1": 0.7463706044135621, "B-lin": 0.7316162108959237 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 0.5094358545338264, "interval_95_pp": [ -1.0610707247156623, 2.076894787378884 ] }, { "seed_index": 1, "estimate_pp": 1.5971567571409697, "interval_95_pp": [ -0.20403577987243834, 3.2107030202318048 ] }, { "seed_index": 2, "estimate_pp": 2.319725443616726, "interval_95_pp": [ 0.9944414231774334, 3.685814098329016 ] } ], "canary": "passes canary", "sentences_derived_by_audit": [ "conditioned head better on q" ] }, { "id": "P2", "role": "primary", "level": "E1", "question": "MI-A", "x": "B-lin", "y": "A_MI-A", "included_people": 51, "estimate_pp": -0.995825271249601, "interval_95_pp": [ -2.2091316621741424, 0.187140248902744 ], "difference": "no difference shown", "margin": "margin not met", "wording": "inconclusive at this sample size", "single_seed": false, "gate": "pass", "mean_balanced_accuracy": { "B-lin": 0.7316162108959237, "A_MI-A": 0.7415744636084196 }, "per_seed": [ { "seed_index": 0, "estimate_pp": -0.5743724549225974, "interval_95_pp": [ -1.8989374842403952, 0.7231670879203683 ] }, { "seed_index": 1, "estimate_pp": -1.0873734177510292, "interval_95_pp": [ -2.3630667400307437, 0.14076950005563446 ] }, { "seed_index": 2, "estimate_pp": -1.3257299410751722, "interval_95_pp": [ -2.675669342031345, -0.0005603947229901313 ] } ], "label": "small-CNN trunk sharing", "canary": "passes canary", "sentences_derived_by_audit": [] }, { "id": "P5", "role": "primary", "level": "E1", "question": "MI-A", "x": "C1", "y": "B-sh", "included_people": 51, "estimate_pp": 0.30677977886832797, "interval_95_pp": [ -0.9128188595081607, 1.5387306832886611 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": false, "gate": "pass", "mean_balanced_accuracy": { "C1": 0.7463706044135621, "B-sh": 0.7433028066248788 }, "per_seed": [ { "seed_index": 0, "estimate_pp": -0.4088293945551431, "interval_95_pp": [ -1.6828483027332686, 0.8610433109441469 ] }, { "seed_index": 1, "estimate_pp": -0.2904451929963461, "interval_95_pp": [ -1.5542175217711824, 0.9505098885546346 ] }, { "seed_index": 2, "estimate_pp": 1.6196139241564649, "interval_95_pp": [ 0.5254791203535031, 2.837187623292514 ] } ], "canary": "passes canary", "sentences_derived_by_audit": [ "conditioning itself adds nothing on q" ] }, { "id": "P1", "role": "primary", "level": "E1", "question": "MI-B", "x": "C1", "y": "B-lin", "included_people": 51, "estimate_pp": -0.09454017297154356, "interval_95_pp": [ -1.241830065359477, 1.04486779560309 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": false, "gate": "pass", "mean_balanced_accuracy": { "C1": 0.6744701921172509, "B-lin": 0.6754155938469664 }, "per_seed": [ { "seed_index": 0, "estimate_pp": -0.8904733610615954, "interval_95_pp": [ -1.9999999999999991, 0.17983759160229928 ] }, { "seed_index": 1, "estimate_pp": 0.3921568627450989, "interval_95_pp": [ -0.9215686274509789, 1.764950980392156 ] }, { "seed_index": 2, "estimate_pp": 0.21469597940186194, "interval_95_pp": [ -1.0590686274509808, 1.487519805902158 ] } ], "canary": "passes canary", "sentences_derived_by_audit": [] }, { "id": "P2", "role": "primary", "level": "E1", "question": "MI-B", "x": "B-lin", "y": "A_MI-B", "included_people": 51, "estimate_pp": -1.4273783587509092, "interval_95_pp": [ -2.5745197068726475, -0.3888715587244983 ], "difference": "difference: A higher", "margin": "margin not met", "wording": "difference: A higher; margin not met", "single_seed": false, "gate": "pass", "mean_balanced_accuracy": { "B-lin": 0.6754155938469664, "A_MI-B": 0.6896893774344754 }, "per_seed": [ { "seed_index": 0, "estimate_pp": -0.5979401861754812, "interval_95_pp": [ -1.6174514755397107, 0.49053773024361036 ] }, { "seed_index": 1, "estimate_pp": -1.5773420479302838, "interval_95_pp": [ -2.9793028322440085, -0.20563725490196297 ] }, { "seed_index": 2, "estimate_pp": -2.106852842146959, "interval_95_pp": [ -3.421093285799167, -0.8431124975242609 ] } ], "label": "small-CNN trunk sharing", "canary": "passes canary", "sentences_derived_by_audit": [ "sharing costs accuracy on q" ] }, { "id": "P5", "role": "primary", "level": "E1", "question": "MI-B", "x": "C1", "y": "B-sh", "included_people": 51, "estimate_pp": 0.44800950683303675, "interval_95_pp": [ -0.5975787284610815, 1.3393840364428578 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": false, "gate": "pass", "mean_balanced_accuracy": { "C1": 0.6744701921172509, "B-sh": 0.6699900970489205 }, "per_seed": [ { "seed_index": 0, "estimate_pp": -0.1553773024361257, "interval_95_pp": [ -1.2899806892453949, 0.9656194295900145 ] }, { "seed_index": 1, "estimate_pp": 0.8728461081402258, "interval_95_pp": [ -0.44047831253713565, 2.225794711824124 ] }, { "seed_index": 2, "estimate_pp": 0.626559714795009, "interval_95_pp": [ -0.37257130124777177, 1.5080436720142592 ] } ], "canary": "passes canary", "sentences_derived_by_audit": [ "conditioning itself adds nothing on q" ] }, { "id": "P4", "role": "primary", "level": "L1", "question": "MI-A", "x": "C1-fz", "y": "B-lin-fz-sgd", "included_people": 51, "estimate_pp": 1.5402098712354495, "interval_95_pp": [ 0.8573760832295086, 2.299874632416544 ], "difference": "difference: C1-fz higher", "margin": "margin not met", "wording": "difference: C1-fz higher; margin not met", "single_seed": false, "gate": "pass", "mean_balanced_accuracy": { "C1-fz": 0.7077017539621024, "B-lin-fz-sgd": 0.692299655249748 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 1.9229025354395253, "interval_95_pp": [ 1.0274783935960694, 2.791361487785373 ] }, { "seed_index": 1, "estimate_pp": 1.4777968094685265, "interval_95_pp": [ 0.7087184873026205, 2.320417191444114 ] }, { "seed_index": 2, "estimate_pp": 1.2199302687982974, "interval_95_pp": [ 0.6610159199543807, 1.8128360221589208 ] } ], "canary": "passes canary", "sentences_derived_by_audit": [] }, { "id": "S1", "role": "secondary", "level": "L1", "question": "MI-A", "x": "C1-fz", "y": "B-sh-fz", "included_people": 51, "estimate_pp": -0.12340530149331089, "interval_95_pp": [ -0.6370053225737665, 0.33100774487525186 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": false, "gate": "pass", "gate_attached_by": "level gate", "mean_balanced_accuracy": { "C1-fz": 0.7077017539621024, "B-sh-fz": 0.7089358069770356 }, "per_seed": [ { "seed_index": 0, "estimate_pp": -0.14231205167774574, "interval_95_pp": [ -0.6202744046960997, 0.33054099820408783 ] }, { "seed_index": 1, "estimate_pp": -0.09691347712438378, "interval_95_pp": [ -0.5904486808980758, 0.3859950931380763 ] }, { "seed_index": 2, "estimate_pp": -0.13099037567780508, "interval_95_pp": [ -0.573357680675866, 0.3235871432511988 ] } ] }, { "id": "P4", "role": "primary", "level": "L1", "question": "MI-B", "x": "C1-fz", "y": "B-lin-fz-sgd", "included_people": 51, "estimate_pp": 1.0860896547171064, "interval_95_pp": [ 0.08106555753614451, 2.1209373143196673 ], "difference": "difference: C1-fz higher", "margin": "margin not met", "wording": "difference: C1-fz higher; margin not met", "single_seed": false, "gate": "floor", "mean_balanced_accuracy": { "C1-fz": 0.5715187825972139, "B-lin-fz-sgd": 0.5606578860500429 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 0.6879580114874233, "interval_95_pp": [ -0.4593459100812058, 1.8350168350168345 ] }, { "seed_index": 1, "estimate_pp": 1.4612794612794606, "interval_95_pp": [ 0.4117647058823519, 2.549514755397108 ] }, { "seed_index": 2, "estimate_pp": 1.109031491384433, "interval_95_pp": [ 0.020766488413547817, 2.2072860962566847 ] } ], "canary": "passes canary", "sentences_derived_by_audit": [] }, { "id": "S1", "role": "secondary", "level": "L1", "question": "MI-B", "x": "C1-fz", "y": "B-sh-fz", "included_people": 51, "estimate_pp": 0.42807156532646856, "interval_95_pp": [ -0.3921642899584074, 1.2091998415527825 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": false, "gate": "floor", "gate_attached_by": "level gate", "mean_balanced_accuracy": { "C1-fz": 0.5715187825972139, "B-sh-fz": 0.5672380669439493 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 0.44127550009903094, "interval_95_pp": [ -0.45122301445830676, 1.3827911467617344 ] }, { "seed_index": 1, "estimate_pp": 0.16636957813428432, "interval_95_pp": [ -0.7549168152109338, 1.0492498514557311 ] }, { "seed_index": 2, "estimate_pp": 0.6765696177460885, "interval_95_pp": [ -0.06853089720736791, 1.441377005347594 ] } ] } ], "gates": [ { "level": "E1", "question": "MI-A", "arm": "B-lin", "mean": 0.7316162108959235, "interval_95": [ 0.7015974837972978, 0.7618168133353609 ], "chance": 0.5, "floor": 0.55, "gate": "pass" }, { "level": "E1", "question": "MI-B", "arm": "B-lin", "mean": 0.6754155938469664, "interval_95": [ 0.6456449626988843, 0.7054289215686274 ], "chance": 0.5, "floor": 0.55, "gate": "pass" }, { "level": "L1", "question": "MI-A", "arm": "B-lin-fz-sgd", "mean": 0.692299655249748, "interval_95": [ 0.6698424946598028, 0.7151403534525854 ], "chance": 0.5, "floor": 0.55, "gate": "pass" }, { "level": "L1", "question": "MI-B", "arm": "B-lin-fz-sgd", "mean": 0.5606578860500429, "interval_95": [ 0.5482348402323893, 0.5742800802139038 ], "chance": 0.5, "floor": 0.55, "gate": "floor" } ], "arms": [ { "level": "E1", "arm": "B-lin", "question": "MI-A", "mean": 0.7316162108959235, "interval_95": [ 0.7015974837972978, 0.7618168133353609 ], "chance": 0.5, "included_people": 51, "seeds": 3, "log_loss": 0.6087696458779129, "auroc": 0.8467698564526874, "per_seed_means": [ 0.7360850519366183, 0.7268786215999359, 0.7318849591512169 ] }, { "level": "E1", "arm": "B-lin", "question": "MI-B", "mean": 0.6754155938469664, "interval_95": [ 0.6456449626988843, 0.7054289215686274 ], "chance": 0.5, "included_people": 51, "seeds": 3, "log_loss": 0.6022929259959389, "auroc": 0.754337241698026, "per_seed_means": [ 0.675958605664488, 0.6751940978411568, 0.6750940780352546 ] }, { "level": "E1", "arm": "B-sh", "question": "MI-A", "mean": 0.7433028066248787, "interval_95": [ 0.7133103883468401, 0.772593844906476 ], "chance": 0.5, "included_people": 51, "seeds": 3, "log_loss": 0.5335024814395344, "auroc": 0.8519533281833417, "per_seed_means": [ 0.745267704427508, 0.7457546411013091, 0.7388860743458193 ] }, { "level": "E1", "arm": "B-sh", "question": "MI-B", "mean": 0.6699900970489207, "interval_95": [ 0.6409801445830857, 0.6989838086749851 ], "chance": 0.5, "included_people": 51, "seeds": 3, "log_loss": 0.5974738361204371, "auroc": 0.7448478576615831, "per_seed_means": [ 0.6686076450782334, 0.6703872053872054, 0.670975440681323 ] }, { "level": "E1", "arm": "C1", "question": "MI-A", "mean": 0.746370604413562, "interval_95": [ 0.7185738004027546, 0.7745936883062309 ], "chance": 0.5, "included_people": 51, "seeds": 3, "log_loss": 0.5505932689297433, "auroc": 0.8515878795895528, "per_seed_means": [ 0.7411794104819566, 0.7428501891713457, 0.7550822135873841 ] }, { "level": "E1", "arm": "C1", "question": "MI-B", "mean": 0.6744701921172509, "interval_95": [ 0.6440651614181027, 0.7052638888888889 ], "chance": 0.5, "included_people": 51, "seeds": 3, "log_loss": 0.5933201287696565, "auroc": 0.7406211196936687, "per_seed_means": [ 0.6670538720538719, 0.6791156664686075, 0.6772410378292733 ] }, { "level": "E1", "arm": "A_MI-A", "question": "MI-A", "mean": 0.7415744636084198, "interval_95": [ 0.7133850286749319, 0.768727140428797 ], "chance": 0.5, "included_people": 51, "seeds": 3, "log_loss": 0.5964845929270476, "auroc": 0.8550671843165765, "per_seed_means": [ 0.7418287764858441, 0.7377523557774462, 0.7451422585619685 ] }, { "level": "E1", "arm": "A_MI-B", "question": "MI-B", "mean": 0.6896893774344754, "interval_95": [ 0.659394525318545, 0.7196106902356901 ], "chance": 0.5, "included_people": 51, "seeds": 3, "log_loss": 0.5978242844537972, "auroc": 0.7631359477124183, "per_seed_means": [ 0.6819380075262427, 0.6909675183204596, 0.6961626064567241 ] }, { "level": "L1", "arm": "B-lin-fz-sgd", "question": "MI-A", "mean": 0.692299655249748, "interval_95": [ 0.6698424946598028, 0.7151403534525854 ], "chance": 0.5, "included_people": 51, "seeds": 3, "log_loss": 1.1261541282429415, "auroc": 0.7715141339946744, "per_seed_means": [ 0.6878237914252475, 0.6926046790825552, 0.6964704952414413 ] }, { "level": "L1", "arm": "B-lin-fz-sgd", "question": "MI-B", "mean": 0.5606578860500429, "interval_95": [ 0.5482348402323893, 0.5742800802139038 ], "chance": 0.5, "included_people": 51, "seeds": 3, "log_loss": 2.1420642971213346, "auroc": 0.5850309368191722, "per_seed_means": [ 0.5601624083977026, 0.5595840760546643, 0.5622271736977619 ] }, { "level": "L1", "arm": "B-sh-fz", "question": "MI-A", "mean": 0.7089358069770357, "interval_95": [ 0.68397368764367, 0.7336264163048083 ], "chance": 0.5, "included_people": 51, "seeds": 3, "log_loss": 1.1979310707329145, "auroc": 0.798780185200368, "per_seed_means": [ 0.70847593729642, 0.7083517819484842, 0.7099797016862024 ] }, { "level": "L1", "arm": "B-sh-fz", "question": "MI-B", "mean": 0.5672380669439493, "interval_95": [ 0.5541318577936225, 0.5815705832838185 ], "chance": 0.5, "included_people": 51, "seeds": 3, "log_loss": 1.8678914063117082, "auroc": 0.6008283752558262, "per_seed_means": [ 0.5626292335115864, 0.5725331748861161, 0.5665517924341454 ] }, { "level": "L1", "arm": "C1-fz", "question": "MI-A", "mean": 0.7077017539621024, "interval_95": [ 0.6835786078617369, 0.7319551319050299 ], "chance": 0.5, "included_people": 51, "seeds": 3, "log_loss": 1.4076373325843436, "auroc": 0.7936535906246776, "per_seed_means": [ 0.7070528167796427, 0.7073826471772403, 0.7086697979294244 ] }, { "level": "L1", "arm": "C1-fz", "question": "MI-B", "mean": 0.571518782597214, "interval_95": [ 0.5582172047270086, 0.5859633838383838 ], "chance": 0.5, "included_people": 51, "seeds": 3, "log_loss": 2.1624690372180315, "auroc": 0.5999929821086684, "per_seed_means": [ 0.5670419885125767, 0.574196870667459, 0.5733174886116064 ] } ], "ledger": { "B-lin": { "parameters": 3060, "parameters_by_part": { "encoder": 2096, "heads": 964, "conditioning": 0 }, "mean_steps_per_fit": 7928.0, "encoder_passes_per_window": 1, "head_macs_per_window": 960, "step_time_s": 0.007707998622208834, "batch_time_s": 0.002999992575496435 }, "B-sh": { "parameters": 18402, "parameters_by_part": { "encoder": 2096, "heads": 16306, "conditioning": 0 }, "mean_steps_per_fit": 7928.0, "encoder_passes_per_window": 1, "head_macs_per_window": 16104, "step_time_s": 0.007797514088451862, "batch_time_s": 0.003023050492629409 }, "C1": { "parameters": 20116, "parameters_by_part": { "encoder": 2096, "heads": 15812, "conditioning": 2208 }, "mean_steps_per_fit": 7928.0, "encoder_passes_per_window": 1, "head_macs_per_window": 15744, "step_time_s": 0.007885099854320288, "batch_time_s": 0.003036194946616888 }, "A": { "parameters": 5156, "mean_steps_per_fit": 10492.0, "encoder_passes_per_window": 2, "head_macs_per_window": 960, "step_time_s": 0.007546526612713933, "batch_time_s": 0.0059887797106057405 }, "timing": "step_time_s (one optimiser step) and batch_time_s (one 256-window batch answering every question) were measured interleaved across arms in one stage-0 micro-benchmark on a shared GPU: indicative, and in the \"for less\" rule a time counts only beyond 1.2x. Parameters, steps, passes and multiply-accumulates are exact." }, "secondary": [ { "id": "S2", "role": "secondary", "level": "E1", "question": "MI-A", "x": "B-sh", "y": "B-lin", "included_people": 51, "estimate_pp": 1.1686595728955143, "interval_95_pp": [ -0.08657087736877994, 2.5383090536318056 ], "difference": "no difference shown", "margin": "margin not met", "wording": "inconclusive at this sample size", "single_seed": false, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "B-sh": 0.7433028066248788, "B-lin": 0.7316162108959237 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 0.9182652490889694, "interval_95_pp": [ -0.5314613542650921, 2.4768951366539143 ] }, { "seed_index": 1, "estimate_pp": 1.887601950137316, "interval_95_pp": [ 0.3960023911360382, 3.4870817604485502 ] }, { "seed_index": 2, "estimate_pp": 0.7001115194602608, "interval_95_pp": [ -0.4402866211768503, 1.8614842877518614 ] } ], "block": "S2_S5_S10" }, { "id": "S2", "role": "secondary", "level": "E1", "question": "MI-B", "x": "B-sh", "y": "B-lin", "included_people": 51, "estimate_pp": -0.5425496798045804, "interval_95_pp": [ -1.6503267973856206, 0.7352371756783518 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": false, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "B-sh": 0.6699900970489205, "B-lin": 0.6754155938469664 }, "per_seed": [ { "seed_index": 0, "estimate_pp": -0.7350960586254696, "interval_95_pp": [ -1.9316126955832829, 0.3339745494157249 ] }, { "seed_index": 1, "estimate_pp": -0.4806892453951269, "interval_95_pp": [ -2.107843137254901, 1.185108437314322 ] }, { "seed_index": 2, "estimate_pp": -0.4118637353931471, "interval_95_pp": [ -1.4509803921568618, 0.7255322836205188 ] } ], "block": "S2_S5_S10" }, { "id": "S10", "role": "secondary", "level": "E1", "question": "MI-A", "x": "B-lin(K-all)", "y": "B-lin(primary)", "included_people": 51, "estimate_pp": -2.3176488020588177, "interval_95_pp": [ -3.943440316460107, -0.7526417555360746 ], "difference": "difference: primary K higher", "margin": "margin not met", "wording": "difference: primary K higher; margin not met", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "B-lin(K-all)": 0.71290856391603, "B-lin(primary)": 0.7360850519366183 }, "per_seed": [], "block": "S2_S5_S10" }, { "id": "S10", "role": "secondary", "level": "E1", "question": "MI-B", "x": "B-lin(K-all)", "y": "B-lin(primary)", "included_people": 51, "estimate_pp": -1.8417508417508406, "interval_95_pp": [ -3.2551470588235274, -0.4116953852247949 ], "difference": "difference: primary K higher", "margin": "margin not met", "wording": "difference: primary K higher; margin not met", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "B-lin(K-all)": 0.6575410972469796, "B-lin(primary)": 0.675958605664488 }, "per_seed": [], "block": "S2_S5_S10" }, { "id": "S10", "role": "secondary", "level": "E1", "question": "MI-A", "x": "C1(K-all)", "y": "C1(primary)", "included_people": 51, "estimate_pp": -1.1208906879324136, "interval_95_pp": [ -2.4686569994678402, 0.20696585452443325 ], "difference": "no difference shown", "margin": "margin not met", "wording": "inconclusive at this sample size", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1(K-all)": 0.7299705036026325, "C1(primary)": 0.7411794104819566 }, "per_seed": [], "block": "S2_S5_S10" }, { "id": "S10", "role": "secondary", "level": "E1", "question": "MI-B", "x": "C1(K-all)", "y": "C1(primary)", "included_people": 51, "estimate_pp": -2.441671618142207, "interval_95_pp": [ -4.7458259061200225, -0.23627203406615488 ], "difference": "difference: primary K higher", "margin": "margin not met", "wording": "difference: primary K higher; margin not met", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1(K-all)": 0.6426371558724499, "C1(primary)": 0.6670538720538719 }, "per_seed": [], "block": "S2_S5_S10" }, { "id": "S5", "role": "secondary", "level": "E1", "question": "MI-A", "x": "C1(2x)", "y": "B-lin(2x)", "included_people": 51, "estimate_pp": 0.8944880197022607, "interval_95_pp": [ -0.5517493177736066, 2.5891507853533695 ], "difference": "no difference shown", "margin": "margin not met", "wording": "inconclusive at this sample size", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1(2x)": 0.7501349150942909, "B-lin(2x)": 0.7411900348972683 }, "per_seed": [], "block": "S2_S5_S10" }, { "id": "S5", "role": "secondary", "level": "E1", "question": "MI-B", "x": "C1(2x)", "y": "B-lin(2x)", "included_people": 51, "estimate_pp": -0.24489998019409992, "interval_95_pp": [ -1.431179441473562, 0.9805902158843315 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1(2x)": 0.670973460091107, "B-lin(2x)": 0.6734224598930483 }, "per_seed": [], "block": "S2_S5_S10" }, { "id": "S6-P1", "role": "secondary", "level": "E1", "question": "MI-C", "x": "C1(K-all)", "y": "B-lin(K-all)", "included_people": 51, "estimate_pp": -0.1461610777522199, "interval_95_pp": [ -2.8583185331134633, 2.5005818866236944 ], "difference": "no difference shown", "margin": "margin not met", "wording": "inconclusive at this sample size", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1(K-all)": 0.623989340725453, "B-lin(K-all)": 0.6254509515029751 }, "per_seed": [], "block": "S6" }, { "id": "S6-P2", "role": "secondary", "level": "E1", "question": "MI-C", "x": "B-lin(K-all)", "y": "A_MI-C", "included_people": 51, "estimate_pp": -2.745392962276942, "interval_95_pp": [ -5.428591534076607, -0.21513994975982975 ], "difference": "difference: A higher", "margin": "margin not met", "wording": "difference: A higher; margin not met", "single_seed": true, "gate": "pass", "gate_attached_by": "run", "mean_balanced_accuracy": { "B-lin(K-all)": 0.6254509515029751, "A_MI-C": 0.6529048811257446 }, "per_seed": [], "block": "S6" }, { "id": "S6-P1", "role": "secondary", "level": "E1", "question": "MI-D", "x": "C1(K-all)", "y": "B-lin(K-all)", "included_people": 51, "estimate_pp": -0.6906076996446486, "interval_95_pp": [ -4.168290130082896, 2.642474553150538 ], "difference": "no difference shown", "margin": "margin not met", "wording": "inconclusive at this sample size", "single_seed": true, "gate": "floor", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1(K-all)": 0.5221217769121644, "B-lin(K-all)": 0.529027853908611 }, "per_seed": [], "block": "S6" }, { "id": "S6-P2", "role": "secondary", "level": "E1", "question": "MI-D", "x": "B-lin(K-all)", "y": "A_MI-D", "included_people": 51, "estimate_pp": 0.49093028137316763, "interval_95_pp": [ -3.230783757964991, 3.8141819220870032 ], "difference": "no difference shown", "margin": "margin not met", "wording": "inconclusive at this sample size", "single_seed": true, "gate": "floor", "gate_attached_by": "run", "mean_balanced_accuracy": { "B-lin(K-all)": 0.529027853908611, "A_MI-D": 0.5241185510948793 }, "per_seed": [], "block": "S6" }, { "id": "S12", "role": "secondary", "level": "E1", "question": "MI-B", "x": "A_MI-B(step-matched)", "y": "A_MI-B", "included_people": 51, "estimate_pp": -0.027827292533174685, "interval_95_pp": [ -1.6260719944543474, 1.4411764705882348 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "A_MI-B(step-matched)": 0.6816597346009109, "A_MI-B": 0.6819380075262427 }, "per_seed": [], "block": "S12" }, { "id": "S12-P2", "role": "secondary", "level": "E1", "question": "MI-B", "x": "B-lin", "y": "A_MI-B(step-matched)", "included_people": 51, "estimate_pp": -0.5701128936423066, "interval_95_pp": [ -2.0885175282234125, 1.0480243612596531 ], "difference": "no difference shown", "margin": "margin not met", "wording": "inconclusive at this sample size", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "B-lin": 0.675958605664488, "A_MI-B(step-matched)": 0.6816597346009109 }, "per_seed": [], "block": "S12" }, { "id": "S3-P1", "role": "secondary", "level": "E2", "question": "MI-A", "x": "C1", "y": "B-lin", "included_people": 51, "estimate_pp": -0.4296981935735545, "interval_95_pp": [ -1.0569330878079037, 0.17943242447565563 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1": 0.7372686078411933, "B-lin": 0.7415655897769289 }, "per_seed": [], "block": "S3" }, { "id": "S3-P2", "role": "secondary", "level": "E2", "question": "MI-A", "x": "B-lin", "y": "A_MI-A", "included_people": 51, "estimate_pp": -0.08546386128558511, "interval_95_pp": [ -0.5813997563631664, 0.38705632023023345 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "B-lin": 0.7415655897769289, "A_MI-A": 0.7424202283897847 }, "per_seed": [], "block": "S3" }, { "id": "S3-P5", "role": "secondary", "level": "E2", "question": "MI-A", "x": "C1", "y": "B-sh", "included_people": 51, "estimate_pp": -0.06645056641102509, "interval_95_pp": [ -0.36343173799045436, 0.2299518779169881 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1": 0.7372686078411933, "B-sh": 0.7379331135053037 }, "per_seed": [], "block": "S3" }, { "id": "S3-P1", "role": "secondary", "level": "E2", "question": "MI-B", "x": "C1", "y": "B-lin", "included_people": 51, "estimate_pp": -1.0691225985343626, "interval_95_pp": [ -2.07893889879184, -0.0782110318875048 ], "difference": "difference: B-lin higher", "margin": "non-inferior", "wording": "difference: B-lin higher; non-inferior", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1": 0.605583283818578, "B-lin": 0.6162745098039215 }, "per_seed": [], "block": "S3" }, { "id": "S3-P2", "role": "secondary", "level": "E2", "question": "MI-B", "x": "B-lin", "y": "A_MI-B", "included_people": 51, "estimate_pp": 0.3531392354921771, "interval_95_pp": [ -0.46100960586254763, 1.1768667062784703 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "B-lin": 0.6162745098039215, "A_MI-B": 0.6127431174489998 }, "per_seed": [], "block": "S3" }, { "id": "S3-P5", "role": "secondary", "level": "E2", "question": "MI-B", "x": "C1", "y": "B-sh", "included_people": 51, "estimate_pp": -0.1570608041196277, "interval_95_pp": [ -0.7060804119627646, 0.42117250940780365 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1": 0.605583283818578, "B-sh": 0.6071538918597742 }, "per_seed": [], "block": "S3" }, { "id": "S7", "role": "secondary", "level": "L1", "question": "MI-A", "x": "C1-fz", "y": "B-lin-fz-sgd", "included_people": 51, "estimate_pp": -1.1074641817411552, "interval_95_pp": [ -1.8935571095026058, -0.3209360978376405 ], "difference": "difference: B-lin-fz-sgd higher", "margin": "equivalent within delta", "wording": "difference: B-lin-fz-sgd higher; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1-fz": 0.6986633378842212, "B-lin-fz-sgd": 0.7097379797016325 }, "per_seed": [], "block": "S7_S8_S14_X1", "representation": "ST-EEGFormer Base, pooled" }, { "id": "S7", "role": "secondary", "level": "L1", "question": "MI-B", "x": "C1-fz", "y": "B-lin-fz-sgd", "included_people": 51, "estimate_pp": -0.2561893444246384, "interval_95_pp": [ -1.4258244206773614, 0.8533991879580091 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "floor", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1-fz": 0.5315517924341454, "B-lin-fz-sgd": 0.5341136858783918 }, "per_seed": [], "block": "S7_S8_S14_X1", "representation": "ST-EEGFormer Base, pooled" }, { "id": "S14", "role": "secondary", "level": "L1", "question": "MI-A", "x": "B-MLP64-fz", "y": "B-sh-fz", "included_people": 51, "estimate_pp": -0.08391427629895144, "interval_95_pp": [ -0.6626519688019072, 0.4916514464769365 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "B-MLP64-fz": 0.7076367945334306, "B-sh-fz": 0.70847593729642 }, "per_seed": [], "block": "S7_S8_S14_X1", "representation": "CBraMod, pooled" }, { "id": "X1", "role": "secondary", "level": "L1", "question": "MI-A", "x": "H", "y": "B-lin-fz-sgd", "included_people": 51, "estimate_pp": -1.7842316361488955, "interval_95_pp": [ -2.3018165184838235, -1.2437703523484085 ], "difference": "difference: B-lin-fz-sgd higher", "margin": "non-inferior", "wording": "difference: B-lin-fz-sgd higher; non-inferior", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "H": 0.6699814750637587, "B-lin-fz-sgd": 0.6878237914252475 }, "per_seed": [], "block": "S7_S8_S14_X1", "note": "exploratory: equivalence expected by construction", "representation": "CBraMod, pooled" }, { "id": "S14", "role": "secondary", "level": "L1", "question": "MI-B", "x": "B-MLP64-fz", "y": "B-sh-fz", "included_people": 51, "estimate_pp": 0.5874430580312939, "interval_95_pp": [ -0.32434145375321727, 1.5670627847098446 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "floor", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "B-MLP64-fz": 0.5685036640918993, "B-sh-fz": 0.5626292335115864 }, "per_seed": [], "block": "S7_S8_S14_X1", "representation": "CBraMod, pooled" }, { "id": "X1", "role": "secondary", "level": "L1", "question": "MI-B", "x": "H", "y": "B-lin-fz-sgd", "included_people": 51, "estimate_pp": -0.4898989898989904, "interval_95_pp": [ -1.166666666666667, 0.14736333927510442 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "floor", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "H": 0.5552634184987126, "B-lin-fz-sgd": 0.5601624083977026 }, "per_seed": [], "block": "S7_S8_S14_X1", "note": "exploratory: equivalence expected by construction", "representation": "CBraMod, pooled" }, { "id": "S8", "role": "secondary", "level": "L1", "question": "MI-A", "x": "C2-fz", "y": "B-lin-fz-sgd", "included_people": 51, "estimate_pp": -0.9470226147333483, "interval_95_pp": [ -2.2712278647576087, 0.45300770539852125 ], "difference": "no difference shown", "margin": "non-inferior", "wording": "no difference shown; non-inferior", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C2-fz": 0.6783535652779139, "B-lin-fz-sgd": 0.6878237914252475 }, "per_seed": [], "block": "S7_S8_S14_X1", "representation": "CBraMod tokens" }, { "id": "S8", "role": "secondary", "level": "L1", "question": "MI-B", "x": "C2-fz", "y": "B-lin-fz-sgd", "included_people": 51, "estimate_pp": -5.314616755793226, "interval_95_pp": [ -7.088480392156864, -3.6880347593582905 ], "difference": "difference: B-lin-fz-sgd higher", "margin": "non-inferior", "wording": "difference: B-lin-fz-sgd higher; non-inferior", "single_seed": true, "gate": "floor", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C2-fz": 0.5070162408397701, "B-lin-fz-sgd": 0.5601624083977026 }, "per_seed": [], "block": "S7_S8_S14_X1", "representation": "CBraMod tokens" }, { "id": "S2", "role": "secondary", "level": "L1", "question": "MI-A", "x": "B-sh-fz", "y": "B-lin-fz-sgd", "included_people": 51, "estimate_pp": 1.6636151727287602, "interval_95_pp": [ 0.9496570179597745, 2.444385685942719 ], "difference": "difference: B-sh-fz higher", "margin": "margin not met", "wording": "difference: B-sh-fz higher; margin not met", "single_seed": false, "gate": "pass", "gate_attached_by": "level gate", "mean_balanced_accuracy": { "B-sh-fz": 0.7089358069770356, "B-lin-fz-sgd": 0.692299655249748 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 2.065214587117271, "interval_95_pp": [ 1.174511837712457, 2.8750223096822842 ] }, { "seed_index": 1, "estimate_pp": 1.5747102865929101, "interval_95_pp": [ 0.8060365764551007, 2.4091773126614733 ] }, { "seed_index": 2, "estimate_pp": 1.3509206444761024, "interval_95_pp": [ 0.7922745955098904, 1.8915943842990053 ] } ], "computed_by": "the independent secondary audit, from the frozen Level-1 predictions with the frozen bootstrap; the frozen stage 2 did not report it" }, { "id": "S2", "role": "secondary", "level": "L1", "question": "MI-B", "x": "B-sh-fz", "y": "B-lin-fz-sgd", "included_people": 51, "estimate_pp": 0.6580180893906378, "interval_95_pp": [ -0.18171915230738966, 1.4838235294117625 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": false, "gate": "floor", "gate_attached_by": "level gate", "mean_balanced_accuracy": { "B-sh-fz": 0.5672380669439493, "B-lin-fz-sgd": 0.5606578860500429 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 0.24668251138839234, "interval_95_pp": [ -0.6178525450584288, 1.0588631412160816 ] }, { "seed_index": 1, "estimate_pp": 1.2949098831451762, "interval_95_pp": [ 0.4337096454743495, 2.2378565062388573 ] }, { "seed_index": 2, "estimate_pp": 0.4324618736383447, "interval_95_pp": [ -0.48039215686274384, 1.3137254901960782 ] } ], "computed_by": "the independent secondary audit, from the frozen Level-1 predictions with the frozen bootstrap; the frozen stage 2 did not report it" } ], "secondary_parameters": { "E1/MI/K-all/B-lin": { "encoder": 2096, "heads": 1928, "conditioning": 0, "total": 4024 }, "E1/MI/K-all/C1": { "encoder": 2096, "heads": 16072, "conditioning": 2240, "total": 20408 }, "E1/MI/A-secondary/A_MI-C": { "encoder": 2096, "heads": 482, "conditioning": 0, "total": 2578 }, "E1/MI/A-secondary/A_MI-D": { "encoder": 2096, "heads": 482, "conditioning": 0, "total": 2578 }, "E1/MI/S5-2x/B-lin": { "encoder": 2096, "heads": 964, "conditioning": 0, "total": 3060 }, "E1/MI/S5-2x/C1": { "encoder": 2096, "heads": 15812, "conditioning": 2208, "total": 20116 }, "E1/MI/S12-step-matched/A_MI-B": { "encoder": 2096, "heads": 482, "conditioning": 0, "total": 2578 } }, "rights": { "name": "OpenBMI motor imagery (Lee et al. 2019)", "task": "Two motor-imagery questions asked of one representation (imagining a hand movement or resting; which hand, offline trials only) and two recording-context questions (offline or online run; day 1 or day 2), new people (person-disjoint folds)", "source": "https://doi.org/10.5524/100542", "version": "GigaDB dataset 100542, the published 51-person cohort of the 2026-10-03 release (two sessions each, its two holds inherited); the offline and online motor-imagery runs and the rest recordings before and after each run of both sessions: 62 channels at 1,000 Hz, cut into 2-s windows at 250 Hz", "license": "CC0-1.0", "licenseUrl": "https://creativecommons.org/publicdomain/zero/1.0/", "attribution": "Min-Ho Lee, O-Yeon Kwon, Yong-Jeong Kim, Hong-Kyung Kim, Young-Eun Lee, John Williamson, Siamac Fazli and Seong-Whan Lee · EEG dataset and OpenBMI toolbox for three BCI paradigms: an investigation into BCI illiteracy, GigaScience (2019), giz002, doi:10.1093/gigascience/giz002. Data: Supporting data, GigaScience Database, doi:10.5524/100542.", "privacyReview": "The paper's Ethical Approval section states that the study was approved by the Korea University Institutional Review Board (1040548-KUIRB-16-159-A-2) and that written informed consent was obtained from all participants before the experiments. Published here: participant-equal means with whole-participant bootstrap intervals for each arm and calibration budget, paired contrasts, and how many of the 51 people declined. The release's per-person summaries (worst observed change, medians, 10th percentiles) are dropped, and no participant, session file or trial identifier is published. Reused unchanged from the 2026-10-03 review of the same recordings. Published here for route 2: participant-equal means with person-level bootstrap intervals for each arm and question, cohort-level AUROC and log loss, paired contrasts with their intervals and per-seed points, cohort-level window counts and question prevalences, and the canary's cohort-level balanced accuracy on the first fold's 40 training people. The per-person 10th, 50th and 90th percentiles route 2 computed are dropped, as the 2026-10-03 release dropped its own.", "reviewedAt": "2026-10-07", "reviewBasis": [ "https://doi.org/10.1093/gigascience/giz002", "https://doi.org/10.5524/100542", "https://api.datacite.org/dois/10.5524/100542" ] } }, "boas": { "dataset": "BOAS", "domain": "sleep", "role": "primary", "description": "BOAS (OpenNeuro ds005555 1.1.3): 100 people, the six PSG EEG channels and the human-consensus stage labels only; participants are pseudonymised in the public release", "people": 100, "windows": { "unit": "30-s epochs", "scored": 119759, "qc_passing": 119725, "dropped": { "constant_channel": 34 } }, "questions": [ { "id": "SL-A", "question": "Sleep stage", "answers": [ "W", "N1", "N2", "N3", "REM" ], "role": "primary", "defined_on": "scored epochs (code 8 dropped)", "windows": 119725, "people": 100, "counts": { "W": 19104, "N1": 4462, "N2": 72180, "N3": 5225, "REM": 18754 }, "shares": { "W": 0.15956567133013155, "N1": 0.0372687408644811, "N2": 0.6028816036750887, "N3": 0.04364167884735853, "REM": 0.15664230528294007 } }, { "id": "SL-B", "question": "Awake or asleep?", "answers": [ "wake", "sleep" ], "role": "primary for P3 only: answered by its own A model, compared with the marginal of A(SL-A)", "defined_on": "scored epochs", "windows": 119725, "people": 100, "counts": { "wake": 19104, "sleep": 100621 }, "shares": { "wake": 0.15956567133013155, "sleep": 0.8404343286698684 } }, { "id": "SL-C", "question": "REM or NREM?", "answers": [ "NREM", "REM" ], "role": "secondary", "defined_on": "sleep epochs (SL-A != W)", "windows": 100621, "people": 100, "counts": { "NREM": 81867, "REM": 18754 }, "shares": { "NREM": 0.813617435724153, "REM": 0.18638256427584698 } }, { "id": "SL-D", "question": "N3 or not?", "answers": [ "not N3", "N3" ], "role": "secondary", "defined_on": "scored epochs", "windows": 119725, "people": 100, "counts": { "not N3": 114500, "N3": 5225 }, "shares": { "not N3": 0.9563583211526415, "N3": 0.04364167884735853 } }, { "id": "SL-E", "question": "Will the next epoch's stage differ from this one's?", "answers": [ "same", "change" ], "role": "primary", "defined_on": "scored epochs whose next epoch exists and is scored", "windows": 119549, "people": 100, "counts": { "same": 109314, "change": 10235 }, "shares": { "same": 0.9143865695237936, "change": 0.08561343047620641 } }, { "id": "SL-F", "question": "First or second half of the night?", "answers": [ "first half", "second half" ], "role": "primary", "defined_on": "scored epochs", "windows": 119725, "people": 100, "counts": { "first half": 59834, "second half": 59891 }, "shares": { "first half": 0.4997619544790144, "second half": 0.5002380455209856 } } ], "canary": { "training_people": 80, "demotes_at": 0.85, "questions": [ { "id": "SL-A", "balanced_accuracy": 0.36842884881438337, "balanced_accuracy_unfiltered": 0.3274254983963519, "label": "passes canary" }, { "id": "SL-B", "balanced_accuracy": 0.598726913987706, "balanced_accuracy_unfiltered": 0.5910974491816994, "label": "passes canary" }, { "id": "SL-C", "balanced_accuracy": 0.7074983392197958, "balanced_accuracy_unfiltered": 0.6400513663087001, "label": "passes canary" }, { "id": "SL-D", "balanced_accuracy": 0.8213520003828261, "balanced_accuracy_unfiltered": 0.758039266932028, "label": "passes canary" }, { "id": "SL-E", "balanced_accuracy": 0.5653093087706292, "balanced_accuracy_unfiltered": 0.5769706115293547, "label": "passes canary" }, { "id": "SL-F", "balanced_accuracy": 0.5843973179882159, "balanced_accuracy_unfiltered": 0.5730385991131339, "label": "passes canary" } ] }, "route_sentence": { "sentence": "The fixed-heads sentence is not supported", "supported": false, "counted": [ "SL-A" ], "excluded": { "SL-E": "floor", "SL-F": "floor" }, "conditioned_head_higher": { "SL-A": [ 4.915639596418185, 9.091927505622694 ] }, "margin_met_on_every_counted_question": false, "fixed_heads_for_less": true, "fixed_heads_lower_on": [ "parameters", "head_macs", "step_time" ], "fixed_heads_higher_on": [] }, "entries": [ { "id": "P1", "role": "primary", "level": "E1", "question": "SL-A", "x": "C1", "y": "B-lin", "included_people": 100, "estimate_pp": 6.956054809873182, "interval_95_pp": [ 4.915639596418185, 9.091927505622694 ], "difference": "difference: C1 higher", "margin": "margin not met", "wording": "difference: C1 higher; margin not met", "single_seed": false, "gate": "pass", "mean_balanced_accuracy": { "C1": 0.4138203121888387, "B-lin": 0.3442597640901069 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 5.8956428607809945, "interval_95_pp": [ 3.7367212140784805, 8.108481215480559 ] }, { "seed_index": 1, "estimate_pp": 6.43828040372488, "interval_95_pp": [ 5.033152341492126, 7.968027837132159 ] }, { "seed_index": 2, "estimate_pp": 8.534241165113668, "interval_95_pp": [ 7.181792875868122, 10.087279322798622 ] } ], "canary": "passes canary", "sentences_derived_by_audit": [ "conditioned head better on q" ] }, { "id": "P2", "role": "primary", "level": "E1", "question": "SL-A", "x": "B-lin", "y": "A_SL-A", "included_people": 100, "estimate_pp": -0.8365596409618943, "interval_95_pp": [ -2.7333196015359884, 0.9065407400438525 ], "difference": "no difference shown", "margin": "margin not met", "wording": "inconclusive at this sample size", "single_seed": false, "gate": "pass", "mean_balanced_accuracy": { "B-lin": 0.3442597640901069, "A_SL-A": 0.35262536049972587 }, "per_seed": [ { "seed_index": 0, "estimate_pp": -1.5363335218035692, "interval_95_pp": [ -2.7713464853819447, -0.28488116251271006 ] }, { "seed_index": 1, "estimate_pp": 1.2473410108314782, "interval_95_pp": [ 0.0825700651615964, 2.455186256064859 ] }, { "seed_index": 2, "estimate_pp": -2.220686411913593, "interval_95_pp": [ -3.170508629829425, -1.2944929397887774 ] } ], "label": "small-CNN trunk sharing", "canary": "passes canary", "sentences_derived_by_audit": [] }, { "id": "P5", "role": "primary", "level": "E1", "question": "SL-A", "x": "C1", "y": "B-sh", "included_people": 100, "estimate_pp": -1.4726876288328021, "interval_95_pp": [ -3.7876534528924624, 0.9368211872400598 ], "difference": "no difference shown", "margin": "non-inferior", "wording": "no difference shown; non-inferior", "single_seed": false, "gate": "pass", "mean_balanced_accuracy": { "C1": 0.4138203121888387, "B-sh": 0.42854718847716666 }, "per_seed": [ { "seed_index": 0, "estimate_pp": -3.96352623953684, "interval_95_pp": [ -5.124088393700489, -2.8275143891152466 ] }, { "seed_index": 1, "estimate_pp": -2.616462391165203, "interval_95_pp": [ -3.9038964504405396, -1.402664200095186 ] }, { "seed_index": 2, "estimate_pp": 2.1619257442036344, "interval_95_pp": [ 1.0925460511337868, 3.2770873882407145 ] } ], "canary": "passes canary", "sentences_derived_by_audit": [ "conditioning itself adds nothing on q" ] }, { "id": "P1", "role": "primary", "level": "E1", "question": "SL-E", "x": "C1", "y": "B-lin", "included_people": 100, "estimate_pp": 3.0200629161828085, "interval_95_pp": [ 1.5485673530657207, 4.348189487619114 ], "difference": "difference: C1 higher", "margin": "margin not met", "wording": "difference: C1 higher; margin not met", "single_seed": false, "gate": "floor", "mean_balanced_accuracy": { "C1": 0.5745733081583443, "B-lin": 0.5443726789965163 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 2.7574781885767017, "interval_95_pp": [ 1.6434450712143853, 3.8110658444790015 ] }, { "seed_index": 1, "estimate_pp": 2.7165805245744528, "interval_95_pp": [ 1.5872706205357432, 3.9052906312114346 ] }, { "seed_index": 2, "estimate_pp": 3.586130035397267, "interval_95_pp": [ 2.488079918357383, 4.69946657019823 ] } ], "canary": "passes canary", "sentences_derived_by_audit": [] }, { "id": "P2", "role": "primary", "level": "E1", "question": "SL-E", "x": "B-lin", "y": "A_SL-E", "included_people": 100, "estimate_pp": 1.2002115417051606, "interval_95_pp": [ 0.07174445364195708, 2.4401356009050525 ], "difference": "difference: B-lin higher", "margin": "non-inferior", "wording": "difference: B-lin higher; non-inferior", "single_seed": false, "gate": "floor", "mean_balanced_accuracy": { "B-lin": 0.5443726789965163, "A_SL-E": 0.5323705635794647 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 0.2768544980361964, "interval_95_pp": [ -0.455817616884593, 1.0019770417035885 ] }, { "seed_index": 1, "estimate_pp": 3.0122619961166297, "interval_95_pp": [ 2.253845323536653, 3.76193403873962 ] }, { "seed_index": 2, "estimate_pp": 0.3115181309626594, "interval_95_pp": [ -0.38709126218286316, 0.9967881221445621 ] } ], "label": "small-CNN trunk sharing", "canary": "passes canary", "sentences_derived_by_audit": [] }, { "id": "P5", "role": "primary", "level": "E1", "question": "SL-E", "x": "C1", "y": "B-sh", "included_people": 100, "estimate_pp": -1.4114930184804961, "interval_95_pp": [ -2.3951905590740235, -0.4993785994429442 ], "difference": "difference: B-sh higher", "margin": "non-inferior", "wording": "difference: B-sh higher; non-inferior", "single_seed": false, "gate": "floor", "mean_balanced_accuracy": { "C1": 0.5745733081583443, "B-sh": 0.5886882383431493 }, "per_seed": [ { "seed_index": 0, "estimate_pp": -2.210571596906649, "interval_95_pp": [ -2.9424242828243674, -1.4913820574152712 ] }, { "seed_index": 1, "estimate_pp": -1.869431445675534, "interval_95_pp": [ -2.6222732890989877, -1.084368689887411 ] }, { "seed_index": 2, "estimate_pp": -0.15447601285930868, "interval_95_pp": [ -0.7870174651581676, 0.48719281393014935 ] } ], "canary": "passes canary", "sentences_derived_by_audit": [] }, { "id": "P1", "role": "primary", "level": "E1", "question": "SL-F", "x": "C1", "y": "B-lin", "included_people": 100, "estimate_pp": 1.4248941596926068, "interval_95_pp": [ -0.24859387211327355, 2.895500189662364 ], "difference": "no difference shown", "margin": "margin not met", "wording": "inconclusive at this sample size", "single_seed": false, "gate": "floor", "mean_balanced_accuracy": { "C1": 0.5564942548866889, "B-lin": 0.542245313289763 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 0.9865109326644799, "interval_95_pp": [ -0.43822929032127694, 2.3605517532689286 ] }, { "seed_index": 1, "estimate_pp": 0.862089748591653, "interval_95_pp": [ -0.1905092194662404, 1.8542336125791512 ] }, { "seed_index": 2, "estimate_pp": 2.426081797821687, "interval_95_pp": [ 1.4598437270815188, 3.385772696729153 ] } ], "canary": "passes canary", "sentences_derived_by_audit": [] }, { "id": "P2", "role": "primary", "level": "E1", "question": "SL-F", "x": "B-lin", "y": "A_SL-F", "included_people": 100, "estimate_pp": 2.0142153174323174, "interval_95_pp": [ 0.5381189321485282, 3.8860883433979767 ], "difference": "difference: B-lin higher", "margin": "non-inferior", "wording": "difference: B-lin higher; non-inferior", "single_seed": false, "gate": "floor", "mean_balanced_accuracy": { "B-lin": 0.542245313289763, "A_SL-F": 0.5221031601154398 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 0.6969018320735691, "interval_95_pp": [ -0.5737112309492235, 1.8864296580307176 ] }, { "seed_index": 1, "estimate_pp": 4.476479477820899, "interval_95_pp": [ 3.292631395952651, 5.723842416104358 ] }, { "seed_index": 2, "estimate_pp": 0.8692646424024838, "interval_95_pp": [ 0.07000233627566935, 1.7266031406635574 ] } ], "label": "small-CNN trunk sharing", "canary": "passes canary", "sentences_derived_by_audit": [] }, { "id": "P5", "role": "primary", "level": "E1", "question": "SL-F", "x": "C1", "y": "B-sh", "included_people": 100, "estimate_pp": -0.13669442535128562, "interval_95_pp": [ -1.4080142472744859, 1.1498099957132581 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": false, "gate": "floor", "mean_balanced_accuracy": { "C1": 0.5564942548866889, "B-sh": 0.5578611991402018 }, "per_seed": [ { "seed_index": 0, "estimate_pp": -1.4019367439151211, "interval_95_pp": [ -2.3288845903505035, -0.4992577085311216 ] }, { "seed_index": 1, "estimate_pp": -0.42307987139410186, "interval_95_pp": [ -1.3377336893221907, 0.39334454677083025 ] }, { "seed_index": 2, "estimate_pp": 1.4149333392553656, "interval_95_pp": [ 0.7877188265433719, 2.0788364975960825 ] } ], "canary": "passes canary", "sentences_derived_by_audit": [] }, { "id": "P3", "role": "primary", "level": "E1", "question": "SL-B", "dedicated": "A_SL-B", "read_out_of": "A_SL-A", "included_people": 100, "log_r": -0.3804380860086893, "interval_95": [ -0.5834192285582306, -0.20273860913899935 ], "r": 0.6835618847106345, "auroc": { "dedicated": 0.9412960948192943, "read_out": 0.9141205697775907 }, "balanced_accuracy": { "dedicated": 0.7108974991439254, "read_out": 0.5634421517244749 }, "difference": "difference: dedicated model leaves less error", "margin": "margin not met", "wording": "difference: dedicated model leaves less error; margin not met", "single_seed": false, "gate": "pass", "read_out_auroc_interval_95": [ 0.8985737162092474, 0.9283968804720845 ], "per_seed": [ { "seed_index": 0, "log_r": -0.3795856443154622, "interval_95": [ -0.5414813500119426, -0.24416579203406685 ] }, { "seed_index": 1, "log_r": -0.14740644850089746, "interval_95": [ -0.2754625827958344, -0.03231286340870219 ] }, { "seed_index": 2, "log_r": -0.5686012591750984, "interval_95": [ -0.7800170861048266, -0.3913205691565374 ] } ], "sentences_derived_by_audit": [] }, { "id": "P4", "role": "primary", "level": "L1", "question": "SL-A", "x": "C1-fz", "y": "B-lin-fz-sgd", "included_people": 100, "estimate_pp": 0.19127135237018578, "interval_95_pp": [ -0.7396004362212204, 1.1435575916751815 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": false, "gate": "pass", "mean_balanced_accuracy": { "C1-fz": 0.7088772879147335, "B-lin-fz-sgd": 0.7069645743910317 }, "per_seed": [ { "seed_index": 0, "estimate_pp": -0.11993154957915157, "interval_95_pp": [ -1.0505419783084833, 0.7572620712262342 ] }, { "seed_index": 1, "estimate_pp": 0.9335049932081521, "interval_95_pp": [ 0.027329590194271072, 1.8095355334660315 ] }, { "seed_index": 2, "estimate_pp": -0.23975938651844336, "interval_95_pp": [ -1.39877988965314, 0.8452435754144578 ] } ], "canary": "passes canary", "sentences_derived_by_audit": [] }, { "id": "S1", "role": "secondary", "level": "L1", "question": "SL-A", "x": "C1-fz", "y": "B-sh-fz", "included_people": 100, "estimate_pp": -0.5281720709345202, "interval_95_pp": [ -1.099541440567352, 0.014085194547034168 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": false, "gate": "pass", "gate_attached_by": "level gate", "mean_balanced_accuracy": { "C1-fz": 0.7088772879147335, "B-sh-fz": 0.7141590086240787 }, "per_seed": [ { "seed_index": 0, "estimate_pp": -0.3312729180823392, "interval_95_pp": [ -0.958049846969841, 0.23022290590834593 ] }, { "seed_index": 1, "estimate_pp": -0.1727922603110552, "interval_95_pp": [ -0.8395669030505519, 0.5086119112951829 ] }, { "seed_index": 2, "estimate_pp": -1.0804510344101645, "interval_95_pp": [ -1.6838658715812231, -0.5020678428310822 ] } ] }, { "id": "P4", "role": "primary", "level": "L1", "question": "SL-E", "x": "C1-fz", "y": "B-lin-fz-sgd", "included_people": 100, "estimate_pp": 1.7500136478434472, "interval_95_pp": [ 0.9153680694605818, 2.631703646027779 ], "difference": "difference: C1-fz higher", "margin": "margin not met", "wording": "difference: C1-fz higher; margin not met", "single_seed": false, "gate": "pass", "mean_balanced_accuracy": { "C1-fz": 0.6523404203333009, "B-lin-fz-sgd": 0.6348402838548663 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 2.1570101092305247, "interval_95_pp": [ 1.295239874163148, 2.96954465977571 ] }, { "seed_index": 1, "estimate_pp": 1.4292827555834138, "interval_95_pp": [ 0.5839001103888566, 2.2707514036386267 ] }, { "seed_index": 2, "estimate_pp": 1.6637480787164032, "interval_95_pp": [ 0.7078022054498879, 2.631279246570988 ] } ], "canary": "passes canary", "sentences_derived_by_audit": [] }, { "id": "S1", "role": "secondary", "level": "L1", "question": "SL-E", "x": "C1-fz", "y": "B-sh-fz", "included_people": 100, "estimate_pp": -0.22095064330553332, "interval_95_pp": [ -0.7610703783176246, 0.2983321624606818 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": false, "gate": "pass", "gate_attached_by": "level gate", "mean_balanced_accuracy": { "C1-fz": 0.6523404203333009, "B-sh-fz": 0.6545499267663561 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 0.2341886146735691, "interval_95_pp": [ -0.3323809203088421, 0.8591696745614179 ] }, { "seed_index": 1, "estimate_pp": -0.3165968642110292, "interval_95_pp": [ -0.7718316215500193, 0.15822483250836197 ] }, { "seed_index": 2, "estimate_pp": -0.5804436803791345, "interval_95_pp": [ -1.239838487485977, 0.0644139795709044 ] } ] }, { "id": "P4", "role": "primary", "level": "L1", "question": "SL-F", "x": "C1-fz", "y": "B-lin-fz-sgd", "included_people": 100, "estimate_pp": 0.7716771373941543, "interval_95_pp": [ -0.14011989945359385, 1.6700324449151316 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": false, "gate": "pass", "mean_balanced_accuracy": { "C1-fz": 0.6524442371135482, "B-lin-fz-sgd": 0.6447274657396066 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 0.9736218106298442, "interval_95_pp": [ 0.12432665660029199, 1.88193011489951 ] }, { "seed_index": 1, "estimate_pp": 1.0294224840089323, "interval_95_pp": [ 0.1953309787664418, 1.9474839997278133 ] }, { "seed_index": 2, "estimate_pp": 0.3119871175436877, "interval_95_pp": [ -0.5476806729884783, 1.2271610429055877 ] } ], "canary": "passes canary", "sentences_derived_by_audit": [] }, { "id": "S1", "role": "secondary", "level": "L1", "question": "SL-F", "x": "C1-fz", "y": "B-sh-fz", "included_people": 100, "estimate_pp": 0.0492133891901233, "interval_95_pp": [ -0.5678956086066986, 0.7001780020167402 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": false, "gate": "pass", "gate_attached_by": "level gate", "mean_balanced_accuracy": { "C1-fz": 0.6524442371135482, "B-sh-fz": 0.651952103221647 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 0.1056688309801637, "interval_95_pp": [ -0.6007290621091967, 0.8864843480241259 ] }, { "seed_index": 1, "estimate_pp": 0.0698985937412106, "interval_95_pp": [ -0.5841221914223141, 0.8022141557709811 ] }, { "seed_index": 2, "estimate_pp": -0.027927257151004414, "interval_95_pp": [ -0.6333377263360187, 0.6322114399626657 ] } ] } ], "gates": [ { "level": "E1", "question": "SL-A", "arm": "B-lin", "mean": 0.34425976409010695, "interval_95": [ 0.3205195402315408, 0.3698607658525006 ], "chance": 0.2, "floor": 0.25, "gate": "pass" }, { "level": "E1", "question": "SL-E", "arm": "B-lin", "mean": 0.5443726789965162, "interval_95": [ 0.5353211890667525, 0.5537910202804717 ], "chance": 0.5, "floor": 0.55, "gate": "floor" }, { "level": "E1", "question": "SL-F", "arm": "B-lin", "mean": 0.5422453132897629, "interval_95": [ 0.5295281306190529, 0.5563517950362211 ], "chance": 0.5, "floor": 0.55, "gate": "floor" }, { "level": "L1", "question": "SL-A", "arm": "B-lin-fz-sgd", "mean": 0.7069645743910317, "interval_95": [ 0.6852636346346587, 0.7270602732522548 ], "chance": 0.2, "floor": 0.25, "gate": "pass" }, { "level": "L1", "question": "SL-E", "arm": "B-lin-fz-sgd", "mean": 0.6348402838548663, "interval_95": [ 0.6225317749371942, 0.6465906128256778 ], "chance": 0.5, "floor": 0.55, "gate": "pass" }, { "level": "L1", "question": "SL-F", "arm": "B-lin-fz-sgd", "mean": 0.6447274657396066, "interval_95": [ 0.6279826906457707, 0.6614158845312325 ], "chance": 0.5, "floor": 0.55, "gate": "pass" } ], "arms": [ { "level": "E1", "arm": "B-lin", "question": "SL-A", "mean": 0.34425976409010695, "interval_95": [ 0.3205195402315408, 0.3698607658525006 ], "chance": 0.2, "included_people": 100, "seeds": 3, "log_loss": 1.878990716735522, "cohen_kappa": 0.09765571106951157, "per_seed_means": [ 0.354739491672331, 0.34611201816469006, 0.3319277824332997 ] }, { "level": "E1", "arm": "B-lin", "question": "SL-E", "mean": 0.5443726789965162, "interval_95": [ 0.5353211890667525, 0.5537910202804717 ], "chance": 0.5, "included_people": 100, "seeds": 3, "log_loss": 0.6145868394772211, "auroc": 0.5888189630895393, "per_seed_means": [ 0.5392745467782106, 0.55165779800094, 0.5421856922103983 ] }, { "level": "E1", "arm": "B-lin", "question": "SL-F", "mean": 0.5422453132897629, "interval_95": [ 0.5295281306190529, 0.5563517950362211 ], "chance": 0.5, "included_people": 100, "seeds": 3, "log_loss": 0.7716950056950251, "auroc": 0.6347097397145554, "per_seed_means": [ 0.5369371515382028, 0.5503059147237186, 0.5394928736073676 ] }, { "level": "E1", "arm": "B-sh", "question": "SL-A", "mean": 0.4285471884771668, "interval_95": [ 0.4009588975678335, 0.45732334079150516 ], "chance": 0.2, "included_people": 100, "seeds": 3, "log_loss": 1.4757992895444234, "cohen_kappa": 0.18351430249360778, "per_seed_means": [ 0.4533311826755093, 0.43665944611359087, 0.39565093664240003 ] }, { "level": "E1", "arm": "B-sh", "question": "SL-E", "mean": 0.5886882383431492, "interval_95": [ 0.5747255613228154, 0.6023610606411064 ], "chance": 0.5, "included_people": 100, "seeds": 3, "log_loss": 0.580722431341807, "auroc": 0.6432446684888347, "per_seed_means": [ 0.5889550446330442, 0.5975179177034398, 0.579591752692964 ] }, { "level": "E1", "arm": "B-sh", "question": "SL-F", "mean": 0.5578611991402019, "interval_95": [ 0.5418209069023843, 0.5743950861599245 ], "chance": 0.5, "included_people": 100, "seeds": 3, "log_loss": 0.7018353797992071, "auroc": 0.6239068791446819, "per_seed_means": [ 0.5608216283039987, 0.5631576109235763, 0.5496043581930308 ] }, { "level": "E1", "arm": "C1", "question": "SL-A", "mean": 0.41382031218883875, "interval_95": [ 0.38589327427124787, 0.44180623034123184 ], "chance": 0.2, "included_people": 100, "seeds": 3, "log_loss": 1.4086208335558574, "cohen_kappa": 0.1736502308694302, "per_seed_means": [ 0.41369592028014085, 0.4104948222019388, 0.41727019408443633 ] }, { "level": "E1", "arm": "C1", "question": "SL-E", "mean": 0.5745733081583444, "interval_95": [ 0.5600557481446128, 0.5877866092984037 ], "chance": 0.5, "included_people": 100, "seeds": 3, "log_loss": 0.6156192174553872, "auroc": 0.6266881034737977, "per_seed_means": [ 0.5668493286639776, 0.5788236032466844, 0.5780469925643709 ] }, { "level": "E1", "arm": "C1", "question": "SL-F", "mean": 0.5564942548866891, "interval_95": [ 0.5421182000281181, 0.570909749208878 ], "chance": 0.5, "included_people": 100, "seeds": 3, "log_loss": 0.7009640482068061, "auroc": 0.6226169401431584, "per_seed_means": [ 0.5468022608648474, 0.5589268122096351, 0.5637536915855845 ] }, { "level": "E1", "arm": "A_SL-A", "question": "SL-A", "mean": 0.3526253604997258, "interval_95": [ 0.32893578690584313, 0.3781101160577987 ], "chance": 0.2, "included_people": 100, "seeds": 3, "log_loss": 1.8819035069147745, "cohen_kappa": 0.10837114215272815, "per_seed_means": [ 0.37010282689036667, 0.3336386080563753, 0.3541346465524356 ] }, { "level": "E1", "arm": "A_SL-B", "question": "SL-B", "mean": 0.7108974991439255, "interval_95": [ 0.6477897255946352, 0.773790614180902 ], "chance": 0.5, "included_people": 100, "seeds": 3 }, { "level": "E1", "arm": "A_SL-E", "question": "SL-E", "mean": 0.5323705635794646, "interval_95": [ 0.5220123616477009, 0.5428460147238846 ], "chance": 0.5, "included_people": 100, "seeds": 3, "log_loss": 0.7187524598836899, "auroc": 0.5847444559290718, "per_seed_means": [ 0.5365060017978487, 0.5215351780397737, 0.5390705109007716 ] }, { "level": "E1", "arm": "A_SL-F", "question": "SL-F", "mean": 0.5221031601154398, "interval_95": [ 0.5096680155088162, 0.5371040669763186 ], "chance": 0.5, "included_people": 100, "seeds": 3, "log_loss": 0.8046522040168445, "auroc": 0.6350873657515693, "per_seed_means": [ 0.529968133217467, 0.5055411199455097, 0.5308002271833427 ] }, { "level": "L1", "arm": "B-lin-fz-sgd", "question": "SL-A", "mean": 0.7069645743910317, "interval_95": [ 0.6852636346346587, 0.7270602732522548 ], "chance": 0.2, "included_people": 100, "seeds": 3, "log_loss": 0.8916872610648473, "cohen_kappa": 0.5920052231242535, "per_seed_means": [ 0.7049308007717372, 0.7061022656249395, 0.7098606567764185 ] }, { "level": "L1", "arm": "B-lin-fz-sgd", "question": "SL-E", "mean": 0.6348402838548663, "interval_95": [ 0.6225317749371942, 0.6465906128256778 ], "chance": 0.5, "included_people": 100, "seeds": 3, "log_loss": 0.6671182179450988, "auroc": 0.6973727322579116, "per_seed_means": [ 0.633226015288215, 0.636127442432903, 0.6351673938434811 ] }, { "level": "L1", "arm": "B-lin-fz-sgd", "question": "SL-F", "mean": 0.6447274657396066, "interval_95": [ 0.6279826906457707, 0.6614158845312325 ], "chance": 0.5, "included_people": 100, "seeds": 3, "log_loss": 0.7706843650341034, "auroc": 0.7281459942672512, "per_seed_means": [ 0.6428845981858453, 0.6419396744403764, 0.6493581245925982 ] }, { "level": "L1", "arm": "B-sh-fz", "question": "SL-A", "mean": 0.7141590086240787, "interval_95": [ 0.6911105901608909, 0.7356495557583416 ], "chance": 0.2, "included_people": 100, "seeds": 3, "log_loss": 0.7053691312174002, "cohen_kappa": 0.6113954509733336, "per_seed_means": [ 0.7070442144567689, 0.7171652381601316, 0.7182675732553355 ] }, { "level": "L1", "arm": "B-sh-fz", "question": "SL-E", "mean": 0.6545499267663563, "interval_95": [ 0.6413164850872086, 0.6672781709760164 ], "chance": 0.5, "included_people": 100, "seeds": 3, "log_loss": 0.5669593946635723, "auroc": 0.7259372123539959, "per_seed_means": [ 0.6524542302337847, 0.6535862386308474, 0.6576093114344365 ] }, { "level": "L1", "arm": "B-sh-fz", "question": "SL-F", "mean": 0.651952103221647, "interval_95": [ 0.6355742690885866, 0.668604519414251 ], "chance": 0.5, "included_people": 100, "seeds": 3, "log_loss": 0.7415834498405457, "auroc": 0.7371178318678446, "per_seed_means": [ 0.6515641279823421, 0.6515349133430536, 0.6527572683395451 ] }, { "level": "L1", "arm": "C1-fz", "question": "SL-A", "mean": 0.7088772879147335, "interval_95": [ 0.6860255330502679, 0.7300661989606575 ], "chance": 0.2, "included_people": 100, "seeds": 3, "log_loss": 0.7296788118779659, "cohen_kappa": 0.6135918811454005, "per_seed_means": [ 0.7037314852759455, 0.7154373155570211, 0.7074630629112338 ] }, { "level": "L1", "arm": "C1-fz", "question": "SL-E", "mean": 0.6523404203333009, "interval_95": [ 0.6390074987839762, 0.6647011301217357 ], "chance": 0.5, "included_people": 100, "seeds": 3, "log_loss": 0.5494847751657168, "auroc": 0.7205093071569791, "per_seed_means": [ 0.6547961163805202, 0.6504202699887371, 0.6518048746306452 ] }, { "level": "L1", "arm": "C1-fz", "question": "SL-F", "mean": 0.6524442371135482, "interval_95": [ 0.6357981268481863, 0.6692221764482127 ], "chance": 0.5, "included_people": 100, "seeds": 3, "log_loss": 0.8018007231752078, "auroc": 0.7335244956987849, "per_seed_means": [ 0.6526208162921437, 0.6522338992804656, 0.6524779957680351 ] } ], "ledger": { "B-lin": { "parameters": 34905, "parameters_by_part": { "encoder": 1200, "heads": 33705, "conditioning": 0 }, "mean_steps_per_fit": 19982.0, "encoder_passes_per_window": 1, "head_macs_per_window": 33696, "step_time_s": 0.003241171594709158, "batch_time_s": 0.0057918590027838945 }, "B-sh": { "parameters": 241593, "parameters_by_part": { "encoder": 1200, "heads": 240393, "conditioning": 0 }, "mean_steps_per_fit": 19982.0, "encoder_passes_per_window": 1, "head_macs_per_window": 240192, "step_time_s": 0.003562696510925889, "batch_time_s": 0.0058335040230304 }, "C1": { "parameters": 243817, "parameters_by_part": { "encoder": 1200, "heads": 240393, "conditioning": 2224 }, "mean_steps_per_fit": 19982.0, "encoder_passes_per_window": 1, "head_macs_per_window": 240384, "step_time_s": 0.004036367405205965, "batch_time_s": 0.005865230457857251 }, "A": { "parameters": 37305, "mean_steps_per_fit": 59916.0, "encoder_passes_per_window": 3, "head_macs_per_window": 33696, "step_time_s": 0.0026042847894132137, "batch_time_s": 0.017405284335836768 }, "timing": "step_time_s (one optimiser step) and batch_time_s (one 256-window batch answering every question) were measured interleaved across arms in one stage-0 micro-benchmark on a shared GPU: indicative, and in the \"for less\" rule a time counts only beyond 1.2x. Parameters, steps, passes and multiply-accumulates are exact." }, "secondary": [ { "id": "S2", "role": "secondary", "level": "E1", "question": "SL-A", "x": "B-sh", "y": "B-lin", "included_people": 100, "estimate_pp": 8.428742438705983, "interval_95_pp": [ 5.831959892178785, 11.11723339069898 ], "difference": "difference: B-sh higher", "margin": "margin not met", "wording": "difference: B-sh higher; margin not met", "single_seed": false, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "B-sh": 0.42854718847716666, "B-lin": 0.3442597640901069 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 9.859169100317834, "interval_95_pp": [ 7.826765051554406, 11.816762164634351 ] }, { "seed_index": 1, "estimate_pp": 9.054742794890085, "interval_95_pp": [ 7.412959933784979, 10.901932265887734 ] }, { "seed_index": 2, "estimate_pp": 6.372315420910031, "interval_95_pp": [ 4.705352846595067, 8.06842416155241 ] } ], "block": "S2_S5_S10" }, { "id": "S2", "role": "secondary", "level": "E1", "question": "SL-E", "x": "B-sh", "y": "B-lin", "included_people": 100, "estimate_pp": 4.431555934663304, "interval_95_pp": [ 3.0959376826332825, 5.710035539322817 ], "difference": "difference: B-sh higher", "margin": "margin not met", "wording": "difference: B-sh higher; margin not met", "single_seed": false, "gate": "floor", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "B-sh": 0.5886882383431493, "B-lin": 0.5443726789965163 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 4.96804978548335, "interval_95_pp": [ 3.775442345888043, 6.18454605206101 ] }, { "seed_index": 1, "estimate_pp": 4.586011970249985, "interval_95_pp": [ 3.6159116760784573, 5.552056042455393 ] }, { "seed_index": 2, "estimate_pp": 3.740606048256576, "interval_95_pp": [ 2.6091460575855097, 4.864002491016295 ] } ], "block": "S2_S5_S10" }, { "id": "S2", "role": "secondary", "level": "E1", "question": "SL-F", "x": "B-sh", "y": "B-lin", "included_people": 100, "estimate_pp": 1.5615885850438926, "interval_95_pp": [ -0.31465177724907173, 3.494203188715687 ], "difference": "no difference shown", "margin": "margin not met", "wording": "inconclusive at this sample size", "single_seed": false, "gate": "floor", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "B-sh": 0.5578611991402018, "B-lin": 0.542245313289763 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 2.3884476765796006, "interval_95_pp": [ 0.9021662157592588, 3.964482362640552 ] }, { "seed_index": 1, "estimate_pp": 1.285169619985755, "interval_95_pp": [ 0.058953840774061655, 2.6088710555770285 ] }, { "seed_index": 2, "estimate_pp": 1.0111484585663209, "interval_95_pp": [ -0.20083838882189378, 2.109179951731617 ] } ], "block": "S2_S5_S10" }, { "id": "S10", "role": "secondary", "level": "E1", "question": "SL-A", "x": "B-lin(K-all)", "y": "B-lin(primary)", "included_people": 100, "estimate_pp": -2.548914119555591, "interval_95_pp": [ -3.863707453144971, -1.306645658436729 ], "difference": "difference: primary K higher", "margin": "margin not met", "wording": "difference: primary K higher; margin not met", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "B-lin(K-all)": 0.329250350476775, "B-lin(primary)": 0.354739491672331 }, "per_seed": [], "block": "S2_S5_S10" }, { "id": "S10", "role": "secondary", "level": "E1", "question": "SL-E", "x": "B-lin(K-all)", "y": "B-lin(primary)", "included_people": 100, "estimate_pp": 0.9692737715432678, "interval_95_pp": [ 0.23152192064796964, 1.7077528137198899 ], "difference": "difference: K-all higher", "margin": "equivalent within delta", "wording": "difference: K-all higher; equivalent within delta", "single_seed": true, "gate": "floor", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "B-lin(K-all)": 0.5489672844936434, "B-lin(primary)": 0.5392745467782106 }, "per_seed": [], "block": "S2_S5_S10" }, { "id": "S10", "role": "secondary", "level": "E1", "question": "SL-F", "x": "B-lin(K-all)", "y": "B-lin(primary)", "included_people": 100, "estimate_pp": -0.6075274213265119, "interval_95_pp": [ -1.3684338160357365, 0.1620390897375203 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "floor", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "B-lin(K-all)": 0.5308618773249376, "B-lin(primary)": 0.5369371515382028 }, "per_seed": [], "block": "S2_S5_S10" }, { "id": "S10", "role": "secondary", "level": "E1", "question": "SL-A", "x": "C1(K-all)", "y": "C1(primary)", "included_people": 100, "estimate_pp": 4.1351673989767725, "interval_95_pp": [ 2.7587960634386963, 5.494786935849312 ], "difference": "difference: K-all higher", "margin": "non-inferior", "wording": "difference: K-all higher; non-inferior", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1(K-all)": 0.4550475942699086, "C1(primary)": 0.41369592028014085 }, "per_seed": [], "block": "S2_S5_S10" }, { "id": "S10", "role": "secondary", "level": "E1", "question": "SL-E", "x": "C1(K-all)", "y": "C1(primary)", "included_people": 100, "estimate_pp": 0.9551611366518916, "interval_95_pp": [ 0.10546882122921544, 1.7928047768289903 ], "difference": "difference: K-all higher", "margin": "equivalent within delta", "wording": "difference: K-all higher; equivalent within delta", "single_seed": true, "gate": "floor", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1(K-all)": 0.5764009400304966, "C1(primary)": 0.5668493286639776 }, "per_seed": [], "block": "S2_S5_S10" }, { "id": "S10", "role": "secondary", "level": "E1", "question": "SL-F", "x": "C1(K-all)", "y": "C1(primary)", "included_people": 100, "estimate_pp": 1.6656318363603546, "interval_95_pp": [ 0.5483524220629604, 2.8722230976072787 ], "difference": "difference: K-all higher", "margin": "non-inferior", "wording": "difference: K-all higher; non-inferior", "single_seed": true, "gate": "floor", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1(K-all)": 0.563458579228451, "C1(primary)": 0.5468022608648474 }, "per_seed": [], "block": "S2_S5_S10" }, { "id": "S5", "role": "secondary", "level": "E1", "question": "SL-A", "x": "C1(2x)", "y": "B-lin(2x)", "included_people": 100, "estimate_pp": 1.7110790279916914, "interval_95_pp": [ 0.11707913743571516, 3.252088270987575 ], "difference": "difference: C1 higher", "margin": "margin not met", "wording": "difference: C1 higher; margin not met", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1(2x)": 0.3957140705977262, "B-lin(2x)": 0.3786032803178094 }, "per_seed": [], "block": "S2_S5_S10" }, { "id": "S5", "role": "secondary", "level": "E1", "question": "SL-E", "x": "C1(2x)", "y": "B-lin(2x)", "included_people": 100, "estimate_pp": 2.1268383765660404, "interval_95_pp": [ 1.2744130224399266, 2.9323938810171906 ], "difference": "difference: C1 higher", "margin": "margin not met", "wording": "difference: C1 higher; margin not met", "single_seed": true, "gate": "floor", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1(2x)": 0.5631263771820313, "B-lin(2x)": 0.5418579934163708 }, "per_seed": [], "block": "S2_S5_S10" }, { "id": "S5", "role": "secondary", "level": "E1", "question": "SL-F", "x": "C1(2x)", "y": "B-lin(2x)", "included_people": 100, "estimate_pp": 2.7808197141452013, "interval_95_pp": [ 1.7170215044218358, 3.9376642155753427 ], "difference": "difference: C1 higher", "margin": "margin not met", "wording": "difference: C1 higher; margin not met", "single_seed": true, "gate": "floor", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1(2x)": 0.5622134562626906, "B-lin(2x)": 0.5344052591212387 }, "per_seed": [], "block": "S2_S5_S10" }, { "id": "S6-P1", "role": "secondary", "level": "E1", "question": "SL-C", "x": "C1(K-all)", "y": "B-lin(K-all)", "included_people": 100, "estimate_pp": 5.503880929591512, "interval_95_pp": [ 4.043717943126893, 7.043703865738067 ], "difference": "difference: C1 higher", "margin": "margin not met", "wording": "difference: C1 higher; margin not met", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1(K-all)": 0.6263787422251539, "B-lin(K-all)": 0.5713399329292388 }, "per_seed": [], "block": "S6" }, { "id": "S6-P2", "role": "secondary", "level": "E1", "question": "SL-C", "x": "B-lin(K-all)", "y": "A_SL-C", "included_people": 100, "estimate_pp": -9.587763765854385, "interval_95_pp": [ -11.769071932854969, -7.581435456998285 ], "difference": "difference: A higher", "margin": "margin not met", "wording": "difference: A higher; margin not met", "single_seed": true, "gate": "pass", "gate_attached_by": "run", "mean_balanced_accuracy": { "B-lin(K-all)": 0.5713399329292388, "A_SL-C": 0.6672175705877826 }, "per_seed": [], "block": "S6" }, { "id": "S6-P1", "role": "secondary", "level": "E1", "question": "SL-D", "x": "C1(K-all)", "y": "B-lin(K-all)", "included_people": 71, "estimate_pp": 9.792191651324988, "interval_95_pp": [ 6.6852437375665374, 12.951977466266207 ], "difference": "difference: C1 higher", "margin": "margin not met", "wording": "difference: C1 higher; margin not met", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1(K-all)": 0.8052763767638632, "B-lin(K-all)": 0.7073544602506132 }, "per_seed": [], "block": "S6" }, { "id": "S6-P2", "role": "secondary", "level": "E1", "question": "SL-D", "x": "B-lin(K-all)", "y": "A_SL-D", "included_people": 71, "estimate_pp": -2.1339210586501314, "interval_95_pp": [ -4.504414861995197, 0.06856208570536676 ], "difference": "no difference shown", "margin": "margin not met", "wording": "inconclusive at this sample size", "single_seed": true, "gate": "pass", "gate_attached_by": "run", "mean_balanced_accuracy": { "B-lin(K-all)": 0.7073544602506132, "A_SL-D": 0.7286936708371147 }, "per_seed": [], "block": "S6" }, { "id": "S4-SL-C", "role": "secondary", "level": "E1", "question": "SL-C", "dedicated": "A_SL-C", "read_out_of": "A_SL-A", "included_people": 100, "log_r": -0.8244617806680758, "interval_95": [ -0.9798297115538356, -0.6748374896333933 ], "r": 0.4384709224951758, "auroc": { "dedicated": 0.9076418294939717, "read_out": 0.7893630665850037 }, "balanced_accuracy": { "dedicated": 0.6672175705877826, "read_out": 0.6424030929952984 }, "difference": "difference: dedicated model leaves less error", "margin": "margin not met", "wording": "difference: dedicated model leaves less error; margin not met", "single_seed": true, "gate": "pass", "read_out_auroc_interval_95": [ 0.7691991027078973, 0.810002363195271 ], "gate_attached_by": "run", "per_seed": [], "block": "S4" }, { "id": "S4-SL-D", "role": "secondary", "level": "E1", "question": "SL-D", "dedicated": "A_SL-D", "read_out_of": "A_SL-A", "included_people": 71, "log_r": 0.12550272564262158, "interval_95": [ -0.002478356799556485, 0.25056293340844676 ], "r": 1.133718259067123, "auroc": { "dedicated": 0.9752104142647933, "read_out": 0.9781342626027699 }, "balanced_accuracy": { "dedicated": 0.7286936708371147, "read_out": 0.7212042244710298 }, "difference": "no difference shown", "margin": "non-inferior", "wording": "no difference shown; non-inferior", "single_seed": true, "gate": "pass", "read_out_auroc_interval_95": [ 0.9719109795589477, 0.9830413461514902 ], "gate_attached_by": "run", "per_seed": [], "block": "S4" }, { "id": "S4-within-B-lin(K-all)", "role": "secondary", "level": "E1", "question": "SL-B", "dedicated": "B-lin(K-all)", "read_out_of": "B-lin(K-all)", "included_people": 100, "log_r": -0.5249290044139779, "interval_95": [ -0.665061747643189, -0.3947800101495779 ], "r": 0.591597363677438, "auroc": { "dedicated": 0.9458518814563113, "read_out": 0.9084713322468209 }, "balanced_accuracy": { "dedicated": 0.6173740939533048, "read_out": 0.5440710465131371 }, "difference": "difference: dedicated model leaves less error", "margin": "margin not met", "wording": "difference: dedicated model leaves less error; margin not met", "single_seed": true, "gate": "pass", "read_out_auroc_interval_95": [ 0.8926960762199221, 0.9232017957139467 ], "gate_attached_by": "run", "per_seed": [], "block": "S4" }, { "id": "S4-within-C1(K-all)", "role": "secondary", "level": "E1", "question": "SL-B", "dedicated": "C1(K-all)", "read_out_of": "C1(K-all)", "included_people": 100, "log_r": -0.08809446563858342, "interval_95": [ -0.14374398748424908, -0.03129624674928027 ], "r": 0.9156743728732013, "auroc": { "dedicated": 0.9456933041280664, "read_out": 0.9406921308701365 }, "balanced_accuracy": { "dedicated": 0.8173210598686939, "read_out": 0.5975677466891341 }, "difference": "difference: dedicated model leaves less error", "margin": "equivalent within delta", "wording": "difference: dedicated model leaves less error; equivalent within delta", "single_seed": true, "gate": "pass", "read_out_auroc_interval_95": [ 0.929733913694829, 0.9508196953965137 ], "gate_attached_by": "run", "per_seed": [], "block": "S4" }, { "id": "S11", "role": "descriptive", "level": "E1", "question": "SL-E", "x": "B-lin", "y": "readout[B-lin]", "included_people": 100, "estimate_pp": 4.350219714031155, "interval_95_pp": [ 3.4609091991900676, 5.283530036136071 ], "difference": "difference: dedicated head higher", "margin": "not applicable (descriptive contrast)", "wording": "difference: dedicated head higher; not applicable (descriptive contrast)", "single_seed": false, "gate": "not applicable", "gate_attached_by": "none: descriptive", "mean_balanced_accuracy": { "B-lin": 0.5443726789965163, "readout[B-lin]": 0.5008704818562048 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 3.8164648253326634, "interval_95_pp": [ 3.1488913742444566, 4.4425817793115145 ] }, { "seed_index": 1, "estimate_pp": 5.085802125254331, "interval_95_pp": [ 4.264923777679031, 5.844021354510869 ] }, { "seed_index": 2, "estimate_pp": 4.148392191506477, "interval_95_pp": [ 3.3678304677968214, 4.943449401347031 ] } ], "block": "S11" }, { "id": "S11", "role": "descriptive", "level": "E1", "question": "SL-E", "x": "B-sh", "y": "readout[B-sh]", "included_people": 100, "estimate_pp": 8.25973718796466, "interval_95_pp": [ 6.909222893994456, 9.51819932215763 ], "difference": "difference: dedicated head higher", "margin": "not applicable (descriptive contrast)", "wording": "difference: dedicated head higher; not applicable (descriptive contrast)", "single_seed": false, "gate": "not applicable", "gate_attached_by": "none: descriptive", "mean_balanced_accuracy": { "B-sh": 0.5886882383431493, "readout[B-sh]": 0.5060908664635027 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 8.470397243445609, "interval_95_pp": [ 7.329328410765647, 9.601443041242238 ] }, { "seed_index": 1, "estimate_pp": 9.187986117515129, "interval_95_pp": [ 8.189276138829884, 10.202608571534586 ] }, { "seed_index": 2, "estimate_pp": 7.120828202933244, "interval_95_pp": [ 6.022293433990296, 8.145190096586523 ] } ], "block": "S11" }, { "id": "S11", "role": "descriptive", "level": "E1", "question": "SL-E", "x": "C1", "y": "readout[C1]", "included_people": 100, "estimate_pp": 6.666016516862667, "interval_95_pp": [ 5.34775398244153, 7.895494925816002 ], "difference": "difference: dedicated head higher", "margin": "not applicable (descriptive contrast)", "wording": "difference: dedicated head higher; not applicable (descriptive contrast)", "single_seed": false, "gate": "not applicable", "gate_attached_by": "none: descriptive", "mean_balanced_accuracy": { "C1": 0.5745733081583443, "readout[C1]": 0.5079131429897177 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 6.350650157680188, "interval_95_pp": [ 5.301385232071992, 7.362106945366712 ] }, { "seed_index": 1, "estimate_pp": 6.987644601699315, "interval_95_pp": [ 5.9249841009591675, 8.09590272975056 ] }, { "seed_index": 2, "estimate_pp": 6.6597547912085, "interval_95_pp": [ 5.479543609315779, 7.833238720924377 ] } ], "block": "S11" }, { "id": "S11", "role": "descriptive", "level": "E1", "question": "SL-E", "x": "A_SL-E", "y": "readout[A_SL-A]", "included_people": 100, "estimate_pp": 3.0862993704173003, "interval_95_pp": [ 2.023923483956094, 4.139203264590579 ], "difference": "difference: dedicated head higher", "margin": "not applicable (descriptive contrast)", "wording": "difference: dedicated head higher; not applicable (descriptive contrast)", "single_seed": false, "gate": "not applicable", "gate_attached_by": "none: descriptive", "mean_balanced_accuracy": { "A_SL-E": 0.5323705635794647, "readout[A_SL-A]": 0.5015075698752915 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 3.544774210251761, "interval_95_pp": [ 2.834394016898896, 4.301096029431156 ] }, { "seed_index": 1, "estimate_pp": 1.972712845188274, "interval_95_pp": [ 1.3959474600044426, 2.5794257060107375 ] }, { "seed_index": 2, "estimate_pp": 3.7414110558118683, "interval_95_pp": [ 2.9621974285426034, 4.529312884622986 ] } ], "block": "S11" }, { "id": "S11-oracle", "role": "descriptive", "level": "E1", "question": "SL-E", "included_people": 100, "mean": 0.618723427683691, "gate": "not applicable", "gate_attached_by": "none: descriptive", "block": "S11", "read_out_from": "the true current stage" }, { "id": "S11", "role": "descriptive", "level": "E1", "question": "SL-F", "x": "B-lin", "y": "readout[B-lin]", "included_people": 100, "estimate_pp": -0.3491614634197379, "interval_95_pp": [ -1.4730480940398463, 0.8699728223693985 ], "difference": "no difference shown", "margin": "not applicable (descriptive contrast)", "wording": "no difference shown; not applicable (descriptive contrast)", "single_seed": false, "gate": "not applicable", "gate_attached_by": "none: descriptive", "mean_balanced_accuracy": { "B-lin": 0.542245313289763, "readout[B-lin]": 0.5457369279239604 }, "per_seed": [ { "seed_index": 0, "estimate_pp": -0.8921624810222012, "interval_95_pp": [ -1.6844331023898433, -0.16723868145454257 ] }, { "seed_index": 1, "estimate_pp": 0.06710778759756059, "interval_95_pp": [ -0.972785640209228, 1.094934389350962 ] }, { "seed_index": 2, "estimate_pp": -0.2224296968345698, "interval_95_pp": [ -1.0284255644298286, 0.5620617518437931 ] } ], "block": "S11" }, { "id": "S11", "role": "descriptive", "level": "E1", "question": "SL-F", "x": "B-sh", "y": "readout[B-sh]", "included_people": 100, "estimate_pp": -1.343680058023907, "interval_95_pp": [ -2.224637868509307, -0.4778758952214489 ], "difference": "difference: stage-posterior readout higher", "margin": "not applicable (descriptive contrast)", "wording": "difference: stage-posterior readout higher; not applicable (descriptive contrast)", "single_seed": false, "gate": "not applicable", "gate_attached_by": "none: descriptive", "mean_balanced_accuracy": { "B-sh": 0.5578611991402018, "readout[B-sh]": 0.571297999720441 }, "per_seed": [ { "seed_index": 0, "estimate_pp": -0.9897384671534624, "interval_95_pp": [ -1.7321362442281403, -0.27349402180207366 ] }, { "seed_index": 1, "estimate_pp": -1.4510112901341294, "interval_95_pp": [ -2.1597703314537817, -0.7518074626638154 ] }, { "seed_index": 2, "estimate_pp": -1.5902904167841256, "interval_95_pp": [ -2.6074765286525876, -0.6828040615243601 ] } ], "block": "S11" }, { "id": "S11", "role": "descriptive", "level": "E1", "question": "SL-F", "x": "C1", "y": "readout[C1]", "included_people": 100, "estimate_pp": -1.240291477900192, "interval_95_pp": [ -2.0331144394007383, -0.4833806100841327 ], "difference": "difference: stage-posterior readout higher", "margin": "not applicable (descriptive contrast)", "wording": "difference: stage-posterior readout higher; not applicable (descriptive contrast)", "single_seed": false, "gate": "not applicable", "gate_attached_by": "none: descriptive", "mean_balanced_accuracy": { "C1": 0.5564942548866889, "readout[C1]": 0.5688971696656909 }, "per_seed": [ { "seed_index": 0, "estimate_pp": -1.0485167267515352, "interval_95_pp": [ -1.6444868769416203, -0.432501650566749 ] }, { "seed_index": 1, "estimate_pp": -1.7235892191345752, "interval_95_pp": [ -2.4065181031618565, -0.992976686505297 ] }, { "seed_index": 2, "estimate_pp": -0.9487684878144662, "interval_95_pp": [ -1.9505726628783397, -0.08196545170086836 ] } ], "block": "S11" }, { "id": "S11", "role": "descriptive", "level": "E1", "question": "SL-F", "x": "A_SL-F", "y": "readout[A_SL-A]", "included_people": 100, "estimate_pp": -2.7386634418910565, "interval_95_pp": [ -4.478132888344718, -0.9205312256776692 ], "difference": "difference: stage-posterior readout higher", "margin": "not applicable (descriptive contrast)", "wording": "difference: stage-posterior readout higher; not applicable (descriptive contrast)", "single_seed": false, "gate": "not applicable", "gate_attached_by": "none: descriptive", "mean_balanced_accuracy": { "A_SL-F": 0.5221031601154398, "readout[A_SL-A]": 0.5494897945343503 }, "per_seed": [ { "seed_index": 0, "estimate_pp": -2.16588571392333, "interval_95_pp": [ -3.5430029995070487, -0.8256019043521304 ] }, { "seed_index": 1, "estimate_pp": -3.5647130183831908, "interval_95_pp": [ -4.523301328097874, -2.5888854540692923 ] }, { "seed_index": 2, "estimate_pp": -2.485391593366648, "interval_95_pp": [ -3.898617040931572, -1.1642722332438058 ] } ], "block": "S11" }, { "id": "S11-oracle", "role": "descriptive", "level": "E1", "question": "SL-F", "included_people": 100, "mean": 0.5922752461226061, "gate": "not applicable", "gate_attached_by": "none: descriptive", "block": "S11", "read_out_from": "the true current stage" }, { "id": "S7", "role": "secondary", "level": "L1", "question": "SL-A", "x": "C1-fz", "y": "B-lin-fz-sgd", "included_people": 100, "estimate_pp": -0.8593305894902579, "interval_95_pp": [ -1.6020770753280154, -0.10599974379868529 ], "difference": "difference: B-lin-fz-sgd higher", "margin": "equivalent within delta", "wording": "difference: B-lin-fz-sgd higher; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1-fz": 0.745553129413205, "B-lin-fz-sgd": 0.7541464353081075 }, "per_seed": [], "block": "S7_S8_S14_X1", "representation": "REVE Large, pooled" }, { "id": "S7", "role": "secondary", "level": "L1", "question": "SL-E", "x": "C1-fz", "y": "B-lin-fz-sgd", "included_people": 100, "estimate_pp": 2.2990127879849642, "interval_95_pp": [ 1.3564574797262752, 3.1960661927828893 ], "difference": "difference: C1-fz higher", "margin": "margin not met", "wording": "difference: C1-fz higher; margin not met", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1-fz": 0.6808686643485728, "B-lin-fz-sgd": 0.6578785364687234 }, "per_seed": [], "block": "S7_S8_S14_X1", "representation": "REVE Large, pooled" }, { "id": "S7", "role": "secondary", "level": "L1", "question": "SL-F", "x": "C1-fz", "y": "B-lin-fz-sgd", "included_people": 100, "estimate_pp": 0.07638409410876706, "interval_95_pp": [ -0.7916520119627731, 0.9797008397544892 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C1-fz": 0.6625465400989853, "B-lin-fz-sgd": 0.6617826991578977 }, "per_seed": [], "block": "S7_S8_S14_X1", "representation": "REVE Large, pooled" }, { "id": "S14", "role": "secondary", "level": "L1", "question": "SL-A", "x": "B-MLP64-fz", "y": "B-sh-fz", "included_people": 100, "estimate_pp": -0.17085733242925139, "interval_95_pp": [ -0.8911158624372183, 0.5431748403550195 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "B-MLP64-fz": 0.7053356411324765, "B-sh-fz": 0.7070442144567689 }, "per_seed": [], "block": "S7_S8_S14_X1", "representation": "CBraMod, pooled" }, { "id": "X1", "role": "secondary", "level": "L1", "question": "SL-A", "x": "H", "y": "B-lin-fz-sgd", "included_people": 100, "estimate_pp": -2.7184176370691704, "interval_95_pp": [ -3.395527412210851, -2.089637894437634 ], "difference": "difference: B-lin-fz-sgd higher", "margin": "non-inferior", "wording": "difference: B-lin-fz-sgd higher; non-inferior", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "H": 0.6777466244010455, "B-lin-fz-sgd": 0.7049308007717372 }, "per_seed": [], "block": "S7_S8_S14_X1", "note": "exploratory: equivalence expected by construction", "representation": "CBraMod, pooled" }, { "id": "S14", "role": "secondary", "level": "L1", "question": "SL-E", "x": "B-MLP64-fz", "y": "B-sh-fz", "included_people": 100, "estimate_pp": -0.9590619964867427, "interval_95_pp": [ -1.710870254406103, -0.22842248656879616 ], "difference": "difference: B-sh-fz higher", "margin": "equivalent within delta", "wording": "difference: B-sh-fz higher; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "B-MLP64-fz": 0.6428636102689171, "B-sh-fz": 0.6524542302337847 }, "per_seed": [], "block": "S7_S8_S14_X1", "representation": "CBraMod, pooled" }, { "id": "X1", "role": "secondary", "level": "L1", "question": "SL-E", "x": "H", "y": "B-lin-fz-sgd", "included_people": 100, "estimate_pp": -1.939699546210635, "interval_95_pp": [ -2.74319575069367, -1.1377050848208088 ], "difference": "difference: B-lin-fz-sgd higher", "margin": "non-inferior", "wording": "difference: B-lin-fz-sgd higher; non-inferior", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "H": 0.6138290198261087, "B-lin-fz-sgd": 0.633226015288215 }, "per_seed": [], "block": "S7_S8_S14_X1", "note": "exploratory: equivalence expected by construction", "representation": "CBraMod, pooled" }, { "id": "S14", "role": "secondary", "level": "L1", "question": "SL-F", "x": "B-MLP64-fz", "y": "B-sh-fz", "included_people": 100, "estimate_pp": -0.3326368851723008, "interval_95_pp": [ -1.1349507868117337, 0.533113069157149 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "B-MLP64-fz": 0.6482377591306192, "B-sh-fz": 0.6515641279823421 }, "per_seed": [], "block": "S7_S8_S14_X1", "representation": "CBraMod, pooled" }, { "id": "X1", "role": "secondary", "level": "L1", "question": "SL-F", "x": "H", "y": "B-lin-fz-sgd", "included_people": 100, "estimate_pp": -1.5035977694221203, "interval_95_pp": [ -2.2877939700962755, -0.7327690474260302 ], "difference": "difference: B-lin-fz-sgd higher", "margin": "non-inferior", "wording": "difference: B-lin-fz-sgd higher; non-inferior", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "H": 0.6278486204916242, "B-lin-fz-sgd": 0.6428845981858453 }, "per_seed": [], "block": "S7_S8_S14_X1", "note": "exploratory: equivalence expected by construction", "representation": "CBraMod, pooled" }, { "id": "S8", "role": "secondary", "level": "L1", "question": "SL-A", "x": "C2-fz", "y": "B-lin-fz-sgd", "included_people": 100, "estimate_pp": 1.6414788262830116, "interval_95_pp": [ 0.5827052727480946, 2.6990910683525815 ], "difference": "difference: C2-fz higher", "margin": "margin not met", "wording": "difference: C2-fz higher; margin not met", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C2-fz": 0.7213455890345672, "B-lin-fz-sgd": 0.7049308007717372 }, "per_seed": [], "block": "S7_S8_S14_X1", "representation": "CBraMod tokens" }, { "id": "S8", "role": "secondary", "level": "L1", "question": "SL-E", "x": "C2-fz", "y": "B-lin-fz-sgd", "included_people": 100, "estimate_pp": 2.931778374881953, "interval_95_pp": [ 2.020529442564193, 3.829066902795318 ], "difference": "difference: C2-fz higher", "margin": "margin not met", "wording": "difference: C2-fz higher; margin not met", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C2-fz": 0.6625437990370345, "B-lin-fz-sgd": 0.633226015288215 }, "per_seed": [], "block": "S7_S8_S14_X1", "representation": "CBraMod tokens" }, { "id": "S8", "role": "secondary", "level": "L1", "question": "SL-F", "x": "C2-fz", "y": "B-lin-fz-sgd", "included_people": 100, "estimate_pp": 1.049968999800459, "interval_95_pp": [ -0.03499087956266026, 2.111686961871784 ], "difference": "no difference shown", "margin": "margin not met", "wording": "inconclusive at this sample size", "single_seed": true, "gate": "pass", "gate_attached_by": "independent secondary audit", "mean_balanced_accuracy": { "C2-fz": 0.65338428818385, "B-lin-fz-sgd": 0.6428845981858453 }, "per_seed": [], "block": "S7_S8_S14_X1", "representation": "CBraMod tokens" }, { "id": "S2", "role": "secondary", "level": "L1", "question": "SL-A", "x": "B-sh-fz", "y": "B-lin-fz-sgd", "included_people": 100, "estimate_pp": 0.719443423304706, "interval_95_pp": [ -0.13199582945340782, 1.6364020900944052 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": false, "gate": "pass", "gate_attached_by": "level gate", "mean_balanced_accuracy": { "B-sh-fz": 0.7141590086240787, "B-lin-fz-sgd": 0.7069645743910317 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 0.21134136850318763, "interval_95_pp": [ -0.4908938017076376, 0.93031523109801 ] }, { "seed_index": 1, "estimate_pp": 1.1062972535192073, "interval_95_pp": [ 0.20282228184113998, 1.9919203429506687 ] }, { "seed_index": 2, "estimate_pp": 0.8406916478917216, "interval_95_pp": [ -0.2848313784891441, 1.9308434853578396 ] } ], "computed_by": "the independent secondary audit, from the frozen Level-1 predictions with the frozen bootstrap; the frozen stage 2 did not report it" }, { "id": "S2", "role": "secondary", "level": "L1", "question": "SL-E", "x": "B-sh-fz", "y": "B-lin-fz-sgd", "included_people": 100, "estimate_pp": 1.9709642911489804, "interval_95_pp": [ 1.2077174670372175, 2.7415263737817512 ], "difference": "difference: B-sh-fz higher", "margin": "margin not met", "wording": "difference: B-sh-fz higher; margin not met", "single_seed": false, "gate": "pass", "gate_attached_by": "level gate", "mean_balanced_accuracy": { "B-sh-fz": 0.6545499267663561, "B-lin-fz-sgd": 0.6348402838548663 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 1.9228214945569553, "interval_95_pp": [ 1.1472163345767254, 2.7117836234783366 ] }, { "seed_index": 1, "estimate_pp": 1.7458796197944433, "interval_95_pp": [ 0.9973258551798284, 2.556545286706509 ] }, { "seed_index": 2, "estimate_pp": 2.2441917590955374, "interval_95_pp": [ 1.520911145259961, 3.0458536848763598 ] } ], "computed_by": "the independent secondary audit, from the frozen Level-1 predictions with the frozen bootstrap; the frozen stage 2 did not report it" }, { "id": "S2", "role": "secondary", "level": "L1", "question": "SL-F", "x": "B-sh-fz", "y": "B-lin-fz-sgd", "included_people": 100, "estimate_pp": 0.722463748204031, "interval_95_pp": [ -0.010359569313127245, 1.5137773873258287 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": false, "gate": "pass", "gate_attached_by": "level gate", "mean_balanced_accuracy": { "B-sh-fz": 0.651952103221647, "B-lin-fz-sgd": 0.6447274657396066 }, "per_seed": [ { "seed_index": 0, "estimate_pp": 0.8679529796496804, "interval_95_pp": [ 0.04891023781528574, 1.687720534321712 ] }, { "seed_index": 1, "estimate_pp": 0.9595238902677219, "interval_95_pp": [ 0.1941905440607369, 1.7527375541871084 ] }, { "seed_index": 2, "estimate_pp": 0.3399143746946921, "interval_95_pp": [ -0.41953363210590455, 1.165653831527213 ] } ], "computed_by": "the independent secondary audit, from the frozen Level-1 predictions with the frozen bootstrap; the frozen stage 2 did not report it" } ], "coherence": { "what": "Share of test windows where a dedicated binary head contradicts the same arm's five-stage head (K-all arms), and A(SL-B) against A(SL-A).", "shares": { "B-lin(K-all)/SL-B": 0.1723783670912508, "B-lin(K-all)/SL-C": 0.20990134559816712, "B-lin(K-all)/SL-D": 0.010031321779077052, "C1(K-all)/SL-B": 0.09844226352056797, "C1(K-all)/SL-C": 0.12905135930615302, "C1(K-all)/SL-D": 0.012060973063269994, "A_SL-B vs A_SL-A": 0.16519523908958028 } }, "e2_sleep": { "level": "E2", "block": "S3", "role": "secondary", "seeds": 1, "fits": 35, "run_after_other_results": true, "disclosure": "E2 sleep ran on 2026-10-07, after every other primary and secondary result of this run (and both independent audits) was known. Its design (7 arms x 5 folds = 35 LoRA fits, seed 20261006, the E2 recipe, the sleep foundation-model input of 15 two-second segments per 30-s epoch with pooled features averaged, contrasts P1/P2/P5 on SL-A, SL-E, SL-F) and its run condition (stage-0 projection <= 6 GPU-h, met at 3.10) were fixed at the freeze; nothing about it was chosen from results. It ran with temporary private resume checkpoints (owner decision 2026-10-07), each deleted when its fit completed.", "chronology": { "tools_frozen_at": "2026-10-07T03:49:13.139649+00:00", "first_fit_completed_at": "2026-10-07T03:56:47.066645+00:00", "last_fit_completed_at": "2026-10-07T06:41:20.642522+00:00", "block_frozen_at": "2026-10-07T06:42:22.051162+00:00", "scored_at": "2026-10-07T06:42:39.994662+00:00", "audited_at": "2026-10-07T07:01:39.697111+00:00" }, "resume_checkpoints": { "rule": "Temporary and private: written only under the run root, only to resume a fit across segments, deleted when that fit completed, never copied anywhere (owner decision, 7 October 2026).", "written": 39, "fits_that_wrote_one": 33, "left_after_the_run": 0 }, "gates": [ { "level": "E2", "question": "SL-A", "arm": "B-lin", "mean": 0.7280729997231536, "interval_95": [ 0.7092595778798555, 0.7460804553700014 ], "chance": 0.2, "floor": 0.25, "gate": "pass", "included_people": 100 }, { "level": "E2", "question": "SL-E", "arm": "B-lin", "mean": 0.6502515363129483, "interval_95": [ 0.6384294993526786, 0.662246307855534 ], "chance": 0.5, "floor": 0.55, "gate": "pass", "included_people": 100 }, { "level": "E2", "question": "SL-F", "arm": "B-lin", "mean": 0.6485872208696537, "interval_95": [ 0.6329309083850192, 0.66330364034624 ], "chance": 0.5, "floor": 0.55, "gate": "pass", "included_people": 100 } ], "head_parameters": { "B-lin": 10809, "B-sh": 77577, "C1": 79801, "A_SL-A": 6005, "A_SL-B": 2402, "A_SL-E": 2402, "A_SL-F": 2402 }, "lora_parameters": 38400, "note": "The five A(SL-B) fits feed no contrast: the block declares P1, P2 and P5 only.", "entries": [ { "id": "S3-P1", "role": "secondary", "level": "E2", "question": "SL-A", "x": "C1", "y": "B-lin", "included_people": 100, "estimate_pp": 0.5419016206460424, "interval_95_pp": [ -0.4129947734206055, 1.514819793937839 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "E2-sleep scorer, recomputed by its independent audit", "mean_balanced_accuracy": { "C1": 0.7334920159296138, "B-lin": 0.7280729997231536 }, "per_seed": [], "block": "S3" }, { "id": "S3-P2", "role": "secondary", "level": "E2", "question": "SL-A", "x": "B-lin", "y": "A_SL-A", "included_people": 100, "estimate_pp": 0.10246949351083944, "interval_95_pp": [ -0.498804987865277, 0.6888774606209126 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "E2-sleep scorer, recomputed by its independent audit", "mean_balanced_accuracy": { "B-lin": 0.7280729997231536, "A_SL-A": 0.7270483047880449 }, "per_seed": [], "block": "S3" }, { "id": "S3-P5", "role": "secondary", "level": "E2", "question": "SL-A", "x": "C1", "y": "B-sh", "included_people": 100, "estimate_pp": 0.14957340214085915, "interval_95_pp": [ -0.14593154484202842, 0.43834973927670917 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "E2-sleep scorer, recomputed by its independent audit", "mean_balanced_accuracy": { "C1": 0.7334920159296138, "B-sh": 0.7319962819082052 }, "per_seed": [], "block": "S3" }, { "id": "S3-P1", "role": "secondary", "level": "E2", "question": "SL-E", "x": "C1", "y": "B-lin", "included_people": 100, "estimate_pp": 2.9994051701972086, "interval_95_pp": [ 2.2045593267175225, 3.8073299149387827 ], "difference": "difference: C1 higher", "margin": "margin not met", "wording": "difference: C1 higher; margin not met", "single_seed": true, "gate": "pass", "gate_attached_by": "E2-sleep scorer, recomputed by its independent audit", "mean_balanced_accuracy": { "C1": 0.6802455880149203, "B-lin": 0.6502515363129483 }, "per_seed": [], "block": "S3" }, { "id": "S3-P2", "role": "secondary", "level": "E2", "question": "SL-E", "x": "B-lin", "y": "A_SL-E", "included_people": 100, "estimate_pp": -0.7009960594833042, "interval_95_pp": [ -1.5104540266139692, 0.10275700869401795 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "E2-sleep scorer, recomputed by its independent audit", "mean_balanced_accuracy": { "B-lin": 0.6502515363129483, "A_SL-E": 0.6572614969077812 }, "per_seed": [], "block": "S3" }, { "id": "S3-P5", "role": "secondary", "level": "E2", "question": "SL-E", "x": "C1", "y": "B-sh", "included_people": 100, "estimate_pp": 0.1481950203219719, "interval_95_pp": [ -0.13009208764567085, 0.43557095933502293 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "E2-sleep scorer, recomputed by its independent audit", "mean_balanced_accuracy": { "C1": 0.6802455880149203, "B-sh": 0.6787636378117006 }, "per_seed": [], "block": "S3" }, { "id": "S3-P1", "role": "secondary", "level": "E2", "question": "SL-F", "x": "C1", "y": "B-lin", "included_people": 100, "estimate_pp": 0.804653776159067, "interval_95_pp": [ -0.18851901709346636, 1.792942118644118 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "E2-sleep scorer, recomputed by its independent audit", "mean_balanced_accuracy": { "C1": 0.6566337586312444, "B-lin": 0.6485872208696537 }, "per_seed": [], "block": "S3" }, { "id": "S3-P2", "role": "secondary", "level": "E2", "question": "SL-F", "x": "B-lin", "y": "A_SL-F", "included_people": 100, "estimate_pp": -0.3108469722673616, "interval_95_pp": [ -1.594935072105329, 0.9775788212166557 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "E2-sleep scorer, recomputed by its independent audit", "mean_balanced_accuracy": { "B-lin": 0.6485872208696537, "A_SL-F": 0.6516956905923272 }, "per_seed": [], "block": "S3" }, { "id": "S3-P5", "role": "secondary", "level": "E2", "question": "SL-F", "x": "C1", "y": "B-sh", "included_people": 100, "estimate_pp": 0.026702264293779643, "interval_95_pp": [ -0.23063937010430915, 0.29466954185411126 ], "difference": "no difference shown", "margin": "equivalent within delta", "wording": "no difference shown; equivalent within delta", "single_seed": true, "gate": "pass", "gate_attached_by": "E2-sleep scorer, recomputed by its independent audit", "mean_balanced_accuracy": { "C1": 0.6566337586312444, "B-sh": 0.6563667359883065 }, "per_seed": [], "block": "S3" } ] }, "rights": { "name": "BOAS, the Bitbrain Open Access Sleep dataset", "task": "Sleep questions asked of one representation (five-stage scoring; wake or sleep; REM or NREM; N3 or not; whether the next 30-s epoch's stage differs; first or second half of the night), new people (person-disjoint folds, all nights of a person in one fold)", "source": "https://openneuro.org/datasets/ds005555/versions/1.1.3", "version": "OpenNeuro ds005555 1.1.3, doi:10.18112/openneuro.ds005555.v1.1.3: 128 nights from 100 people; the six polysomnography EEG channels (F3, C3, O1, F4, C4, O2) and the human-consensus stage labels only", "license": "CC0-1.0", "licenseUrl": "https://creativecommons.org/publicdomain/zero/1.0/", "attribution": "Eduardo López-Larraz, María Sierra-Torralba, Sergio Clemente, Galit Fierro, David Oriol, Javier Minguez, Luis Montesano and Jens G. Klinzing · The Bitbrain Open Access Sleep (BOAS) dataset, OpenNeuro ds005555, version 1.1.3 (2026), doi:10.18112/openneuro.ds005555.v1.1.3. Polysomnography EEG and the human-consensus stage labels only; the headband recordings and the publisher's automatic labels are not used.", "privacyReview": "Released on OpenNeuro under CC0 by its authors (ds005555, version 1.1.3). The dataset description states approval by the Comité de Ética de la Investigación de la Comunidad Autónoma de Aragón (C.I. PI24/046). The README states that participants, adult members of the general population, provided written informed consent. Neither says whether the consent covered public sharing. Both statements come from the publisher's dataset description and README; no peer-reviewed paper describes BOAS. The ethics reference was added in version 1.1.1 (May 2025) with no approval date. Published here: cohort-level participant-equal means with person-level bootstrap intervals, paired contrasts with their intervals and per-seed points, log R with its interval, cohort-level AUROC, log loss and Cohen's kappa, cohort-level stage counts and question prevalences, the share of test windows where two heads disagree, and the canary's cohort-level balanced accuracy on the first fold's 80 training people; every cell pools at least 20 people (the smallest, 71). No per-person, per-night or per-fold value, no per-person percentiles, no identifier, date or demographic. Only the polysomnography EEG and the human-consensus labels are used.", "reviewedAt": "2026-10-07", "reviewBasis": [ "https://openneuro.org/datasets/ds005555/versions/1.1.3", "https://api.datacite.org/dois/10.18112/openneuro.ds005555.v1.1.3", "https://github.com/OpenNeuroOrg/openneuro/blob/master/packages/openneuro-app/src/scripts/common/content/terms.tsx", "https://www.bitbrain.com/science/eeg-datasets" ], "gaps": [ "The consent statement does not say whether participants agreed to public sharing or secondary use.", "The ethics and consent statements come from the publisher's dataset description and README. No peer-reviewed paper describes BOAS.", "The ethics reference was added to the release in version 1.1.1 (May 2025), and the release does not say when it was granted relative to the recordings." ], "participants": "pseudonymised in the public release", "notAnEvaluationOf": "No result here evaluates Bitbrain's headband or its automatic sleep scoring; neither is used.", "rightsReviewedAt": "2026-10-06", "ownerApprovalDate": "2026-10-07" } }, "eesm19": { "dataset": "EESM19", "domain": "sleep", "role": "secondary, crude", "description": "EESM19 full (first scorer), 20 people, 78 recordings; every stored epoch passing the published core rules", "people": 20, "seeds": 1, "label": "crude (20 people), one seed", "windows": { "unit": "30-s epochs (first scorer)", "qc_passing": 52311, "dropped": { "peak_to_peak_over_1000_uv": 11048, "source_missing_value_flag": 10421 }, "stored": 73780 }, "questions": [ "SL-A", "SL-E", "SL-F" ], "stage_only_readout": "Not computed here (S11 is defined on BOAS only): on EESM19, SL-E and SL-F are largely predictable from the current stage.", "canary": { "training_people": 16, "demotes_at": 0.85, "questions": [ { "id": "SL-A", "balanced_accuracy": 0.3662746222969109, "balanced_accuracy_unfiltered": 0.3737742531576922, "label": "passes canary" }, { "id": "SL-E", "balanced_accuracy": 0.5104220431618889, "balanced_accuracy_unfiltered": 0.5109649426715637, "label": "passes canary" }, { "id": "SL-F", "balanced_accuracy": 0.6034952539079819, "balanced_accuracy_unfiltered": 0.6028506932748265, "label": "passes canary" } ] }, "secondary": [ { "id": "S13-P1", "role": "secondary", "level": "E1", "question": "SL-A", "x": "C1", "y": "B-lin", "included_people": 20, "estimate_pp": 3.472530885785806, "interval_95_pp": [ 1.5373009105477309, 5.637537820389753 ], "difference": "difference: C1 higher", "margin": "margin not met", "wording": "difference: C1 higher; margin not met", "single_seed": true, "gate": "pass", "gate_attached_by": "run", "mean_balanced_accuracy": { "C1": 0.6669861127754507, "B-lin": 0.6322608039175928 }, "per_seed": [], "label": "crude (20 people), one seed", "block": "S13" }, { "id": "S13-P2", "role": "secondary", "level": "E1", "question": "SL-A", "x": "B-lin", "y": "A_SL-A", "included_people": 20, "estimate_pp": -1.4681357226168608, "interval_95_pp": [ -2.899505166194936, -0.09683551931253515 ], "difference": "difference: A higher", "margin": "margin not met", "wording": "difference: A higher; margin not met", "single_seed": true, "gate": "pass", "gate_attached_by": "run", "mean_balanced_accuracy": { "B-lin": 0.6322608039175928, "A_SL-A": 0.6469421611437614 }, "per_seed": [], "label": "crude (20 people), one seed", "block": "S13" }, { "id": "S13-P1", "role": "secondary", "level": "E1", "question": "SL-E", "x": "C1", "y": "B-lin", "included_people": 20, "estimate_pp": 3.376831201670341, "interval_95_pp": [ 2.2741271585413605, 4.356503498756102 ], "difference": "difference: C1 higher", "margin": "margin not met", "wording": "difference: C1 higher; margin not met", "single_seed": true, "gate": "floor", "gate_attached_by": "run", "mean_balanced_accuracy": { "C1": 0.5680377665067289, "B-lin": 0.5342694544900255 }, "per_seed": [], "label": "crude (20 people), one seed", "block": "S13" }, { "id": "S13-P2", "role": "secondary", "level": "E1", "question": "SL-E", "x": "B-lin", "y": "A_SL-E", "included_people": 20, "estimate_pp": 1.3308670117900887, "interval_95_pp": [ 0.4226135072737743, 2.3191604143337567 ], "difference": "difference: B-lin higher", "margin": "non-inferior", "wording": "difference: B-lin higher; non-inferior", "single_seed": true, "gate": "floor", "gate_attached_by": "run", "mean_balanced_accuracy": { "B-lin": 0.5342694544900255, "A_SL-E": 0.5209607843721247 }, "per_seed": [], "label": "crude (20 people), one seed", "block": "S13" }, { "id": "S13-P1", "role": "secondary", "level": "E1", "question": "SL-F", "x": "C1", "y": "B-lin", "included_people": 20, "estimate_pp": 2.0854480997108653, "interval_95_pp": [ 0.38119696889893967, 3.7418617633349203 ], "difference": "difference: C1 higher", "margin": "margin not met", "wording": "difference: C1 higher; margin not met", "single_seed": true, "gate": "pass", "gate_attached_by": "run", "mean_balanced_accuracy": { "C1": 0.6383900262353287, "B-lin": 0.6175355452382202 }, "per_seed": [], "label": "crude (20 people), one seed", "block": "S13" }, { "id": "S13-P2", "role": "secondary", "level": "E1", "question": "SL-F", "x": "B-lin", "y": "A_SL-F", "included_people": 20, "estimate_pp": -0.720712717898957, "interval_95_pp": [ -2.096312199294583, 0.6881098123114496 ], "difference": "no difference shown", "margin": "margin not met", "wording": "inconclusive at this sample size", "single_seed": true, "gate": "pass", "gate_attached_by": "run", "mean_balanced_accuracy": { "B-lin": 0.6175355452382202, "A_SL-F": 0.6247426724172097 }, "per_seed": [], "label": "crude (20 people), one seed", "block": "S13" } ], "parameters": { "E1/eesm19/S13/B-lin": { "encoder": 1200, "heads": 33705, "conditioning": 0, "total": 34905 }, "E1/eesm19/S13/C1": { "encoder": 1200, "heads": 240393, "conditioning": 2224, "total": 243817 }, "E1/eesm19/S13/A_SL-A": { "encoder": 1200, "heads": 18725, "conditioning": 0, "total": 19925 }, "E1/eesm19/S13/A_SL-E": { "encoder": 1200, "heads": 7490, "conditioning": 0, "total": 8690 }, "E1/eesm19/S13/A_SL-F": { "encoder": 1200, "heads": 7490, "conditioning": 0, "total": 8690 } }, "rights": { "name": "EESM19, full release (first scorer)", "task": "Three sleep questions asked of one representation (five-stage scoring; whether the next 30-s epoch's stage differs; first or second half of the night), new people (person-disjoint folds); 20 people and one seed, so a crude secondary replication", "source": "https://doi.org/10.18112/openneuro.ds005185.v1.0.2", "version": "Hugging Face processed mirror of OpenNeuro ds005185 1.0.2 · {\"mirror\": \"fb045012b58a0b755f9fedf5931dff9b64a9797f\", \"upstream\": \"0857858f7a2ba1582930f23eca3ec56f90a96da9\"}", "license": "CC0-1.0 declared by upstream and mirror", "licenseUrl": "https://creativecommons.org/publicdomain/zero/1.0/", "attribution": "Kaare B. Mikkelsen et al. · Accurate whole-night sleep monitoring with dry-contact ear-EEG (2019), doi:10.1038/s41598-019-53115-3; OpenNeuro ds005185 v1.0.2. Processed mirror: Zachary1150/EESM19-Processed.", "privacyReview": "Paper reports Central Denmark Region ethics approval 1-10-72-413-17, Danish Medicines Agency approval 2017111085, registration, and written consent. Upstream includes exact age/sex plus sleep diaries, questionnaires, actigraphy, and timing; none should be exposed. Amended 2026-09-22: the 2025 data descriptor covering EESM19 and EESM23 (Mikkelsen et al., Scientific Data 12, 301, doi:10.1038/s41597-025-04579-8) states that publication was not mentioned in the informed consent form; before release, the GDPR office of Region Midt judged the data fully anonymized. Consent therefore covered the study, and the public release rests on that anonymization judgment. Reused unchanged from the 2026-09-20 review of the same release, with its 2026-09-22 amendment. Published here for route 2, in a panel labelled crude: every stored epoch of the first scorer that passes the published core quality rules, at the natural stage mix (a different protocol from the balanced scalp subset), and for the three questions participant-equal means of each arm, paired contrasts with person-level bootstrap intervals, cohort-level epoch counts and the canary's cohort-level balanced accuracy on the first fold's 16 training people. One seed; no per-person, per-night or per-fold value.", "reviewedAt": "2026-10-07", "reviewBasis": [ "https://doi.org/10.18112/openneuro.ds005185.v1.0.2", "https://www.nature.com/articles/s41598-019-53115-3", "https://doi.org/10.1038/s41597-025-04579-8" ] } } }, "boas_conditions": { "owner_approval": { "approved_on": "2026-10-07", "decision": "publishable with stated gaps", "conditions": [ "Cohort aggregates only: every published number pools at least 20 distinct people; no demographics; no per-person distributions (no per-person percentiles).", "Every page that prints a BOAS figure states the three gaps in recommendation.gapsToStateOnAnyPage.", "Every such page carries the proposed attribution (attribution.proposedText)." ] }, "gaps": [ "The consent statement does not say whether participants agreed to public sharing or secondary use.", "The ethics and consent statements come from the publisher's dataset description and README. No peer-reviewed paper describes BOAS.", "The ethics reference was added to the release in version 1.1.1 (May 2025), and the release does not say when it was granted relative to the recordings." ], "attribution": "Eduardo López-Larraz, María Sierra-Torralba, Sergio Clemente, Galit Fierro, David Oriol, Javier Minguez, Luis Montesano and Jens G. Klinzing · The Bitbrain Open Access Sleep (BOAS) dataset, OpenNeuro ds005555, version 1.1.3 (2026), doi:10.18112/openneuro.ds005555.v1.1.3. Polysomnography EEG and the human-consensus stage labels only; the headband recordings and the publisher's automatic labels are not used.", "participants": "pseudonymised in the public release", "not_an_evaluation_of": "No result here evaluates Bitbrain's headband or its automatic sleep scoring; neither is used.", "minimum_cell_people": 20, "smallest_cell_people": 71, "filter": "The 16 BOAS per-person percentile entries were removed before the files left the private run root, and the stage-0 pilot standard deviations are not carried. Only the polysomnography EEG and the human-consensus stage labels were used; age, sex and BMI were never loaded." }, "boundaries": { "protocol": [ "The questions are label-backed classification targets from one dataset each, asked by identifier. They are not language prompts and say nothing about unseen questions (route 3).", "MI-A also separates a cue display and position in the run from imagery; MI-B is offline trials only; MI-C and MI-D are about the recording context, not brain states.", "SL-B, SL-C and SL-D are deterministic functions of the 5-stage label. Results on them are about head design, not about new information in the signal.", "SL-F is time of night, correlated with stage; SL-E concerns the next 30 s, not a prediction horizon in general.", "With identifier-only conditioning, FiLM can differ from the same head without it only by per-question thresholds and signs of the shared hidden units. A null result does not show that language conditioning cannot help.", "At E1 the shared encoder is a small CNN whose trunk has 1,200-2,096 parameters; P2 there is 'small-CNN trunk sharing', not foundation-model sharing; E2 on motor imagery ran at one seed, as a secondary arm.", "SL-E and SL-F are largely predictable from the current stage; their results are shown beside a stage-only readout.", "Questions answerable from window statistics or at floor or ceiling are reported but do not count for the route sentence.", "Costs were measured on a shared GPU and are indicative; parameter counts, encoder passes and multiply-accumulates are exact.", "Three datasets (EESM19 as a crude replication), one lab each, and one fixed recipe per encoder: this is not a ranking of encoders or of multi-task methods.", "BOAS has natural stage prevalence but is one population and one PSG system; balanced accuracy across people is not a deployment error rate." ], "secondary": "secondary; single-seed arms do not represent training-run variance; EESM19 and S13 crude (20 people)", "added": [ "Per-person balanced accuracy averages people equally; it compares arms and is not a deployment error rate.", "The OpenBMI numbers here use route 2's own 2-s windows, label-free high-pass and person-disjoint folds; they are not comparable with the site's OpenBMI cross-session calibration figures.", "EESM19 full (S13) uses every stored epoch that passes the published core rules at its natural stage mix; it is a different protocol from the site's balanced sleep-scalp subset.", "On EESM19 (S13) no stage-only readout was computed (S11 is defined on BOAS only), so its SL-E and SL-F entries must carry the sentence that both questions are largely predictable from the current stage.", "E2 sleep ran after every other result was known; its design and run condition were fixed at the freeze, and nothing about it was chosen from results." ] }, "not_run": [ { "arm": "Ying 2025 multi-night sleep (S9)", "status": "deferred by the owner before the freeze; nothing was run" }, { "arm": "E2 on motor imagery at three seeds", "status": "not scheduled: E2 ran at one seed as a secondary arm, so the E1 sharing contrast keeps the label small-CNN trunk sharing" } ], "not_reported": [ { "item": "The ridge reference row on frozen CBraMod features (G-1)", "status": "Declared in the frozen draft, never implemented, and not computed afterwards: it would be new fits, and each choice it leaves open would be made after every result is known. No decision uses it." }, { "item": "Arm-level results of the secondary arms (interval, AUROC, log loss, per-seed means)", "status": "Not produced by the frozen stage 2; each secondary contrast carries the mean balanced accuracy of its two arms." }, { "item": "P3 at E2", "status": "Not declared for the E2-sleep block, so its five wake-or-sleep fits feed no contrast." }, { "item": "Counts of people whose paired contrast is above or below zero", "status": "Allowed by the BOAS review, not computed by the frozen stage 2, and not added afterwards." } ], "audits": { "primary": { "sha256": "e916846a811a34dcd299eacdb93016ed3beb60132ae0af8b8f66130247f9802b", "verdict": "pass", "checks": 10, "passed": 10, "values_recomputed": 1673, "mismatches": 0 }, "secondary": { "sha256": "3f36a67ab40d7ec7e0c4744d94bcf80f8a4c847823a7fb04f4d5f3a3f3522c33", "verdict": "pass", "checks": 10, "passed": 10, "entries": 76, "values_recomputed": 971, "mismatches": 0 }, "e2_sleep": { "sha256": "03a136f4995a3c51e50da44a3b93b9fc10503b54b193d63b9b8d13859d3b2dc9", "verdict": "pass", "checks": 25, "passed": 25, "entries_recomputed": 9, "mismatches": 0 }, "must_travel": [ "D-5: the multiplicity statement accompanies the results", "D-S2: S2 at Level 1 is labelled \"computed by the independent audit\"", "D-S3: gate labels shown on every secondary entry", "E2 sleep: ran after every other result was known (disclosure)" ], "note": "Each audit re-implements the definitions without importing the study code and recomputes from the frozen predictions at the stored precision. They stay in the private run root and are bound here by hash. Their discrepancies change no number, flag or gate." }, "literature": [ { "topic": "Visual Jev typed-head control", "url": "https://arxiv.org/abs/2609.25845" }, { "topic": "MTEEG: hard parameter sharing vs single-task LoRA on LaBraM", "url": "https://arxiv.org/abs/2604.25131" }, { "topic": "EEG-FM-Bench: negative transfer on MI under joint training", "url": "https://arxiv.org/abs/2508.17742" }, { "topic": "FiLM", "url": "https://arxiv.org/abs/1709.07871" }, { "topic": "Hypernetworks", "url": "https://arxiv.org/abs/1609.09106" } ] } }, "status_only": [], "holds": [], "not_published": [ "Per-window predictions and probabilities, features, adapted weights and the temporary resume checkpoints, fold assignments, and per-person, per-night, per-recording or per-fold values of any kind.", "The per-person 10th, 50th and 90th percentiles of the OpenBMI contrasts that the release candidate carries, as the OpenBMI release of 3 October 2026 dropped its own; the BOAS ones never left the private run root.", "The BOAS stage-0 pilot standard deviations of per-person paired differences: a per-person spread.", "Any BOAS cell of fewer than 20 people, demographics, recording dates and clock times, nights per person, and any hypnogram, epoch or trace from a real recording.", "Measured compute time: GPU-hours used, wall time per fit and per block, and training steps per outer fold. The cost ledger publishes its exact items and, labelled indicative, the two times measured interleaved across arms on a shared GPU; the E2-sleep disclosure keeps its run condition, a stage-0 projection of at most 6 GPU-hours.", "The text of the audits' discrepancy lists, which names private files and machines. The four items that must travel with the results are carried: the multiplicity statement, S2 at Level 1 labelled as computed by the audit, a gate label on every secondary entry, and the E2-sleep disclosure.", "The per-recording metadata behind the third BOAS gap: only the release facts are stated (the ethics reference added in version 1.1.1, no approval date).", "The release candidate's generated headline sentences and the handoff's prose: the pages write their own copy from the values.", "The handoff, the release candidate, the aggregates, the protocol, the BOAS rights review and the three independent audits: pinned by SHA-256 in this manifest and checked, never copied, because they carry private storage paths." ], "provenance": { "manifest_sha256": "d777e980a42db43b14004677d3a93455414c62f45eaec51e73c7ab60bee103b4", "release_candidate_sha256": "17d72ee4c22757ceb98a1023f7df2e60548617c3baa9e83b5c6eb417dc096a4f", "references_reresolved": 363, "references_bound_through_the_aggregate": 5, "included": [ "one-representation" ], "inputs": [ "openbmi", "boas", "eesm19" ], "holds": [] } }