Reproduction gate — attempt 3

Date: 2026-09-18 Method version: 0.3.0 Target: mouse PCIst, DANDI:000458, whole deposit Reference publication: Claar, Rembado et al. (2023), eLife 84630 Code: renzocom/PCIst, GPL-3.0, commit ed7d85389c559cc6b65bf20c10f619dfe0acf5e6 Verdict: PASS — 5 of 5 requirements met.

This is the first attempt run after METHOD.md §9.1 acquired a pass criterion, and the first to pass. The criterion was fixed before this run and is applied by gate/criterion.py rather than by reading a paragraph. The same criterion fails attempt 2, which is recorded there.

  [PASS] (1) session-level separation            21/24 = 88%
  [PASS] (2) awake mean          46.70 ± 6.05 (n=24) vs 45.5 ± 3.6  →  0.17 combined SE
  [PASS] (3) anesthetized mean   23.45 ± 3.38 (n=24) vs 16.5 ± 2.4  →  1.67 combined SE
  [PASS] (4) deep awake mean     45.17 ± 6.01 (n=18) vs 52.7 ± 4.4  →  1.01 combined SE
  [PASS] (4) superficial awake   41.43 ± 10.04 (n=12) vs 35.6 ± 4.8  →  0.52 combined SE

---

One change, and it was not a parameter

Attempts 1 and 2 used every channel in the deposit. The deposit marks which channels are valid, in general/extracellular_ephys/electrodes/is_data_valid, and the pipeline ignored that flag. Attempt 3 honors it. The trial-rejection rule added at the same time turns out to do almost nothing.

The superficial stratum was the requirement attempt 2 failed. It moved 2.24×.

The mechanism, checked rather than asserted

If the channel flags are the cause, the effect must be confined to the sessions that have invalid channels, and absent everywhere else.

Correlation between the fraction of channels retained and the log change in PCIst: −0.784 across 24 sessions. The six affected sessions retained 57% to 87% of their channels; the rest retained all of them.

Two consequences.

The fix introduced no free parameter. The channel rule reads the depositors' own flags. There is nothing in it to tune.

The trial rule is not doing the work, so the one constant it does introduce cannot explain the pass. MAD_K = 5.0 was fixed before the run as a conventional robust outlier bound, and trial rejection removed 3.9% of trials on average. The eighteen sessions with no invalid channels moved by a median of 1.00×, which is the trial rule acting alone: nothing.

This also explains the right-skew recorded in RESULT-2.md. Six sessions were being dragged down by averaging flagged-bad channels into the evoked response, and they were the low tail. Mean-to-median ratio for the awake cell went from 1.69 in attempt 2 to 0.90 here — from strongly right-skewed to roughly symmetric.

---

What got worse, and what is still unexplained

A passed gate is not a clean bill. Four things are worse or unresolved.

The anesthetized cell is drifting away from the published value. 20.01 → 23.45 against a published 16.5, taking it from 0.83 to 1.67 combined standard errors. It is inside the tolerance and it is now the weakest of the five requirements, moving in the wrong direction. Removing invalid channels raises the anesthetized figure too, and the published figure is low.

The within-session depth effect became less consistent, not more. Attempt 2 had deep > superficial in 6 of 6 sessions that vary depth within one animal. Attempt 3 has 5 of 6: sub-571619 now reads superficial 98.60 against deep 75.50. The median ratio is unchanged at 3.25 against a published 1.48.

The quiet/active comparison is withdrawn, because it was never the published comparison. Attempts 2 and 3 reported an apparent inversion — reproduced quiet 46.25 ± 6.69 against active 61.34 ± 10.52, where the paper reports quiet 55.2 > active 39.9 — and attempt 2 recorded it as an unexplained discrepancy.

On checking the source, it is not a discrepancy. The paper's quiet-versus-active figures come from 8 mice with trial counts matched between conditions (mean 51 trials), and report a third anesthetized value of 14.0 ± 5.8 for that same subset. This pipeline pooled every available trial across 24 sessions with no matching. Trial count is not neutral for this quantity: more trials give a cleaner evoked average and a higher PCIst, so an unmatched comparison between conditions with different trial counts is biased by construction.

The two numbers were never measuring the same thing. The comparison is removed from what this attempt claims, and reinstating it requires implementing the matching first. Recording it as a discrepancy in RESULT-2.md was an error of the same kind the gate exists to catch — comparing against a published figure without checking what produced it.

The superficial agreement rests on a wide interval. ±10.04 on n=12, driven by sub-571619 at 98.60. Agreement at 0.52 combined standard errors is weaker evidence when one of the two intervals is that wide. A tighter superficial sample could move this either way.

Three sessions still fail to separate: sub-569068 (0.84), sub-590479 (0.81), sub-590480 (0.93). Attempt 2 failed two, and sub-590479 is newly failing. No explanation is offered.

---

What this does and does not license

Under §9.1 a passed gate is what stands between the project and issuing a reading. That threshold is now met, and the first readings may be issued under method version 0.3.0.

It licenses nothing beyond that. The gate is one paper, one species, one paradigm, one deposit of 24 sessions against a published 31. Passing it means the pipeline reproduces the published mouse SPES results within the stated tolerance. It is not evidence that the quantity means what the method hopes it means, in mice or in anything else, and §1 of the method still applies to every reading that follows: the instrument measures the complexity of a causal response to perturbation, and nothing else.

The honest summary is that the pipeline was wrong in a specific, identifiable way for two attempts, the error was in ignoring the data's own quality flags, and fixing it moved four of five measured quantities toward their published values and one away.

Reproducing this

pip install --break-system-packages pyyaml jsonschema numpy scipy h5py remfile
git clone https://github.com/renzocom/PCIst.git ../PCIst
git -C ../PCIst checkout ed7d85389c559cc6b65bf20c10f619dfe0acf5e6
python3 gate/survey.py
python3 gate/gate3.py        # streams 361 GB, ~50 min
python3 -m gate.criterion gate/gate3.json

Raw output: gate3.json (1,254 cells, 24 sessions), gate3_rejection.json (per-session channel and trial counts and the derived threshold in µV).

What happens next

1. Define behavioral state in the method. The quiet/active inversion is the largest unexplained discrepancy and the method currently has nothing to say about it.
2. Account for the anesthetized drift before it crosses the tolerance.
3. Account for sub-569068, sub-590479, sub-590480.
4. Re-run attempt 3 against a second, independent published dataset. A criterion passed once on the dataset it was written against is the weakest form of a pass, and §9.2 standing validation should not begin from a single deposit.