EXPERIMENT — A COMPLETED RUN WITH ITS ARTIFACTS

An Experiment is recorded here and executed somewhere else. The page owns the argument and the evidence; the code owns the run. Artifacts are the one thing in the vault that exists nowhere else, so they are laid out to be looked at — full-width plots, readable snippets, and a visible distinction between what is stored here and what is merely pointed at.

run-2026-09-14 · prereg-exclusions

⌘K

⌂

▤

▲

⌗

◻

EXPERIMENT

you set this — the app never changes it + artifact attach as evidence history

run-2026-09-14 · prereg-exclusions ran 14 Sep 2026, 11:04 · 6 min 22 s · pipeline v3

PURPOSE

See whether the pooled nap effect survives the preregistered exclusion rule at all — I expected it to collapse and wanted to know by how much before writing anything.

DESIGN written 9 Sep, before the run · revised twice · both explained

Varied: the inclusion rule only — all 41 datasets versus the 33 that survive the preregistered exclusions. Held constant: random-effects estimator, Hedges' g, the harmonised density definition, and the exclusion of within-subject designs. Measured as the pooled estimate with a profile-likelihood interval, plus Egger's intercept on each set. Known confound: four of the 41 share a first author and all four survive the exclusion rule, so the surviving set is more homogeneous than a random subset of the same size would be.

ARTIFACTS 6 items · 4 in the vault · 2 linked stored in the vault under 25 MB · larger is linked · overridable per item

funnel-all-41.png 412 KB · in vault

se 0.05

se 0.34

d 0.00.40.9

Asymmetric, as before. The excluded studies are the pale ones.

forest-prereg-33.png 301 KB · in vault

Pooled estimate in the bottom row. It did not move.

pooled-summary.csv 2 KB · in vault · first 6 rows

set,k,d,lo,hi,egger_p
all,41,0.44,0.34,0.54,0.004
prereg,33,0.41,0.29,0.53,0.011
large_n,14,0.33,0.19,0.47,0.210
small_n,19,0.52,0.38,0.66,0.002
prereg_large,11,0.31,0.15,0.47,0.330

wandb-panel.png 1.1 MB · in vault

screenshot — W&B run panel, sweep 3f2a

Kept because the sweep config is only legible in the UI.

! bootstrap-draws.parquet 2.4 GB · linked

~/research/nap-reanalysis/out/2026-09-14/
bootstrap-draws.parquet

Outside the vault. If this file moves or is cleaned up, the vault will not notice and this page will point at nothing.

copy into vault anyway checked 14 Sep · sha 4f2ac1

exclusion-counts inline table · in vault

OBSERVATIONS revision 2 · changed once, explained · 14 Sep

The pooled estimate barely moves: 0.44 → 0.41. Egger's intercept stays significant in both sets, so the asymmetry is real but it is not carrying the effect. The large-n stratum is lower (0.33) and its asymmetry disappears, which is the only part of this that behaves the way the bias story predicts. My reading: publication bias is present and small; something else accounts for the bulk of the effect.

earlier — “effect essentially unchanged, bias story dead” · revised after noticing the large-n stratum

WHERE IT RAN

repo nap-reanalysis

commit 8c41f0d · clean tree

entry analysis/pool.py --rule prereg

config configs/v3-prereg.yaml

w&b nap-reanalysis/3f2a91

out out/2026-09-14/

ATTACHED AS EVIDENCE 3 criteria · 2 hypotheses

each attachment carries its own note — the same run means different things to different criteria

POSITION HISTORY

14 SepObservations revised after the large-n stratum.

12 Sep2 quiet edits to the design.

9 SepDesign written. Status planned.

open full history →

RELATED RUNS

run-2026-09-06 · size-strata-b

same config lineage · v3

run-2026-08-14 · size-strata

superseded by the above

EXPERIMENT — DESIGNED, NOT YET RUN

The same page before anything exists to show. Purpose and design are the whole content, and that is a finished state: writing the design down before the run is the thing that makes the result mean something later. Empty regions say what will go in them rather than rendering as gaps.

draft · leave-one-lab-out

draft · leave-one-lab-out planned created 17 Sep · not run

PURPOSE

See whether the four same-author datasets matter at all. No claim attached — I want to know the size of the dependence before deciding whether it is worth a criterion.

DESIGN

Varied: which lab is held out, iterating over all nine contributing labs. Held constant: everything else in the v3 pipeline. Measured as the pooled estimate and its interval per held-out set, plus the spread of the nine estimates. Expect the Born-lab leave-out to move it most; if nothing moves more than 0.03 the dependence is not worth writing a criterion about.

ARTIFACTS

Nothing yet. Plots and snippets dropped here are stored in the vault; anything over 25 MB is linked with a warning.

OBSERVATIONS

Written after the run, and revisable — interpretation changes more often than data does.

WHERE IT WILL RUN

repo nap-reanalysis
entry analysis/loo.py (not written)
commit —

ATTACHED AS EVIDENCE

Nothing. Most runs never attach to a hypothesis, and this one was designed without a claim in mind on purpose.

CAME FROM

Four of the 41 share a first author — does that matter?

question · captured 16 Sep while reading run-2026-09-14

EXPERIMENT INBOX — COMPLETED, NOT YET INTERPRETED

The testing-side counterpart of the Question Inbox, and the same tone rule: the only number in the chrome is how many runs exist. Cheap runs accumulate; that is what cheap runs do. Sort, filter by project or attachment, search, or shuffle — the default gesture is looking through them, not clearing them.

consolidation-vault — experiments

Experiments

STATUS

EVIDENCE

PROJECT

⌕

SORT

RUN

PURPOSE

ARTIFACTS

EVIDENCE

WHEN

ARTIFACTS

TRIAGE — OR LEAVE IT

✎Write observationsO

▲Attach to a criterionE

?Capture a question from itQ

×AbandonD

WHERE IT RAN

j/k moveO observationsE attachQ capture {{ footerLabel }}