Reproduction gate — attempt 1

Date: 2026-09-17 Method version: 0.1.0 Target: mouse wake vs isoflurane PCIst separation, DANDI:000458 Reference publication: Claar, Rembado et al. (2023), eLife 84630 Code: renzocom/PCIst, GPL-3.0, commit ed7d85389c559cc6b65bf20c10f619dfe0acf5e6 Verdict: PARTIAL — not passed.

---

What the gate is for

METHOD.md §9.1: before any reading is issued, the pipeline must reproduce a published result from the original authors' data using the original authors' code. The commitment is that a failure is published before anything else is.

This is that publication. The gate did not pass, and in failing it found three errors in the method document itself.

What was run

Six subjects — the six smallest assets in the dandiset, EEG-only sessions of 0.3–1.1 GB. Trial-averaged evoked responses from the 30-channel surface EEG array around cortical electrical stimulation of MOs, running trials excluded (the closest available match to the paper's quiet wakefulness), invalid trials excluded, baseline-corrected, passed to calc_PCIst unmodified.

Published values

Reproduced

The direction reproduces. The magnitude does not. Group awake is 42% of the published value, and two of six subjects separate the wrong way — in sub-543394, isoflurane scores higher than wake.

The one result that does reproduce

Stratifying by stimulation depth explains most of the gap. Only one of the six subjects received deep stimulation:

The single deep-stimulation subject lands inside the published error bar. The superficial subjects fall far short of theirs. So the pipeline can produce the published number on the condition the paper reports as producing it — on one animal — while the rest of the sample does not.

One subject is not a reproduction. It is a reason to run the deep-stimulation sessions in the large files, which is the next attempt.

Three errors this found in METHOD.md

1. resample is a required parameter and was not specified

PCIst constructs T×T recurrence matrices, so memory goes as the square of the samples in the response window. An 800 ms window at the native 2500 Hz demands roughly 9 GB and simply fails. The computation cannot run without a resample rate, and §2.1 named no rate at all.

Worse, the rate changes the answer: 250 Hz versus 500 Hz moved group means by about 25%. An unstated resample rate is an unreproducible reading.

2. The documented parameter set does not transfer

§2.1 gave one parameter set, from human TMS-EEG: baseline −400 to −50 ms, response 0 to 300 ms. The authors used −800 to −2 ms and +2 to +800 ms. Applying the human set to mouse SPES understates both values and the separation (1.51 versus 1.75). The standard needs a per-paradigm parameter set, not one set with a note that other paradigms differ.

3. avgref: true is wrong here — and it was our own instruction

§2.1 states that SPES with referential recording uses avgref: true. Applied to this data it degrades the result: the separation ratio falls from 1.75 to 1.28, and subjects separating correctly fall from four of six to three of six.

This is the most useful of the three findings, because it was not an omission. It was a prescription, written into the standard before any data had been run through it, and it is wrong for the first case tested.

Also recorded

• Selection bias. The six smallest assets are not a random sample. They are EEG-only sessions, mostly superficial stimulation, and three used a single stimulation current. The published N=31 sessions include the 12–27 GB sessions this attempt did not touch.
• Current confound. PCIst varies strongly and non-monotonically with stimulation current, and subjects were run at different currents (20–100 µA). Pooling across current, as this attempt did, is not what the paper did.
• Deposit versus paper. The paper reports 37 mice; the dandiset holds 23 subjects across 24 assets. The deposit is a subset, and any reading computed from it should say so.

What happens next

1. Run the deep-stimulation sessions in the large assets. Deep versus superficial is the largest reported effect after state, and this attempt had one deep animal.
2. Stratify by current rather than pooling.
3. Fix METHOD.md §2.1: add resample as required, add a mouse SPES parameter set, and withdraw the avgref prescription pending evidence.
4. Re-run, and publish attempt 2 here whichever way it goes.

Reproducing this

pip install --break-system-packages pyyaml jsonschema numpy scipy h5py remfile
git clone https://github.com/renzocom/PCIst.git ../PCIst
git -C ../PCIst checkout ed7d85389c559cc6b65bf20c10f619dfe0acf5e6
python3 gate/gate_multi.py     # downloads ~3.2 GB from DANDI
python3 gate/gate_refine.py

Raw outputs: gate_multi.json, gate_refine.json, gate_results.json.