Reproduction gate — attempt 2

Date: 2026-09-18 Method version: 0.2.0 Target: mouse PCIst, DANDI:000458, stratified by stimulation depth and current Reference publication: Claar, Rembado et al. (2023), eLife 84630 Code: renzocom/PCIst, GPL-3.0, commit ed7d85389c559cc6b65bf20c10f619dfe0acf5e6 Verdict: PARTIAL — the primary contrast reproduces; two subsidiary results do not.

Attempt 1 is at RESULT.md. It did not pass and found three errors in the method document. This attempt fixes what that one identified, covers the whole deposit instead of a convenience sample, and finds three more.

---

What changed from attempt 1

The change in coverage is the substantive one. Attempt 1 took the six smallest files because they fit in memory. Those six were almost entirely superficial stimulation, which is the condition the paper reports as producing the lower value — so attempt 1 compared a superficial-heavy sample against a published mean drawn mostly from deep stimulation, and concluded the pipeline undershot by 58%. It was measuring its own sampling.

A second selection error was found while fixing the first. The survey that chooses assets cast estim_current to an integer, which raised on the sessions recording n/a, and those sessions were silently skipped. That hid 13 of the 18 deep-stimulation assets. The first run of the survey reported 5; the correct number is 18. A crash would have been better than the exception that was caught and logged per-asset.

---

Reproduced against published

Intervals are standard error of the mean across sessions. Agreement is judged by whether the difference exceeds the combined standard error of the two means.

The headline result of the paper — that perturbational complexity separates wake from anesthesia, at the magnitudes reported — reproduces. Attempt 1's central failure is resolved, and it was a sampling artifact rather than a pipeline error.

The depth effect reproduces, and overshoots

Six sessions vary stimulation depth within the same animal, which controls for everything that differs between animals.

6 of 6 in the published direction; median ratio 3.25. The published ratio is 1.48 (52.7 / 35.6). The effect is real and considerably larger here than reported, which is the same discrepancy as the superficial row above seen from the other side: deep agrees, superficial comes in low.

State separation per session

16 of 18 deep-stimulation sessions separate wake from isoflurane in the published direction; median ratio 1.44. Attempt 1 managed 4 of 6.

The two that fail are sub-569068 (0.83) and sub-590480 (0.84). Both are anesthesia scoring higher than wake. No explanation is offered here; they are recorded so the next attempt has to account for them.

---

Three more errors this found

4. Stimulation current is a required reported parameter and is not in the method

PCIst under deep stimulation, awake, as a function of current:

An inverted U spanning a 5.4-fold range, peaking around 20–35 µA and falling off at both ends. Two readings taken at 20 µA and 100 µA on the same animal in the same state would differ by more than wake differs from anesthesia.

METHOD.md §2.1 does not require the stimulation current to be recorded, and §4 does not require it in the reading record. A perturbational reading without a stated stimulation intensity is not comparable to any other reading, and the register currently permits one. This is the same class of defect as the unspecified resample rate that attempt 1 found, and it is worse, because resample is a property of the analysis and current is a property of the measurement itself.

5. The quiet/active wakefulness ordering does not reproduce

Published: quiet wakefulness 55.2, active wakefulness 39.9 — quiet is higher. Reproduced, deep stimulation: quiet 46.76 ± 6.72, active 53.83 ± 7.86 — active is higher.

Both cells sit individually within error of their published counterparts, so this is a discrepancy in sign rather than a demonstrated inversion. But the ordering is a reported result and it does not reproduce, and the method document has no definition of behavioral state at all — it inherits whatever the deposit's is_running flag means. Attempt 1 assumed excluding running trials approximated quiet wakefulness and got closer to the published figure; with the full deposit that assumption no longer holds.

6. §9.1 does not say what passing means

This is the finding that matters most, and it is about the gate rather than the pipeline.

METHOD.md §9.1 requires the pipeline to "reproduce a published result." This attempt reproduces four published cells and fails one, reproduces the depth effect in direction while overshooting its magnitude, and separates 16 of 18 sessions. Under the clause as written, that is a pass, a failure, or a partial, depending on which sentence is read as the criterion.

A gate whose author can decide after seeing the numbers whether they passed is not a gate. §9.1 must state, before the next attempt is run: which published quantities are the target, what agreement is required of each, how many must agree, and what a reading is permitted to claim while the gate stands unpassed.

The verdict at the top of this file is stated as PARTIAL on the conservative reading — the primary contrast reproduces, a subsidiary one does not — but that choice is currently an author's judgment rather than a rule, which is the defect.

---

Also recorded

• The distribution is heavily right-skewed. Awake across all sessions: mean 36.44, median 21.56. The published SEM of 3.6 on n=31 implies a far tighter distribution than the one here. Something systematic differs — most likely trial or channel rejection, which the paper performs and this pipeline does not.
• 24 sessions, not 31. The deposit is a subset of the published sample. Any reading computed from it should say so.
• sub-569073 appears twice, as two sessions of differing size (13.20 GB and 26.73 GB), with very different values (5.85 and 93.92 awake). Sessions, not animals, are the unit here, as in the paper.
• No avgref. The prescription withdrawn in v0.2.0 was not reapplied.

Reproducing this

pip install --break-system-packages pyyaml jsonschema numpy scipy h5py remfile
git clone https://github.com/renzocom/PCIst.git ../PCIst
git -C ../PCIst checkout ed7d85389c559cc6b65bf20c10f619dfe0acf5e6
python3 gate/survey.py    # streams trials tables, ~90s
python3 gate/gate2.py     # streams 361 GB, ~45 min
python3 gate/analyze2.py

Nothing is downloaded. remfile serves HDF5 byte ranges over HTTP and h5py reads only the chunks it asks for; each trial is read once and accumulated into every cell it belongs to, so peak memory stays under 1 GB against a 27 GB asset.

Raw output: gate2.json (1,266 cells, 24 sessions), survey.json.

What happens next

1. Fix §2.1 and §4 to require stimulation current, and add it to the reading schema as a required field for perturbational readings.
2. Write the pass criterion into §9.1 before running attempt 3.
3. Define behavioral state in the method rather than inheriting it.
4. Implement trial and channel rejection matching the paper, which is the leading candidate for the skew and for the superficial shortfall.
5. Account for sub-569068 and sub-590480.