# LAB_EXPERIMENT_01 — Instruction tone × specification quality (PREREGISTRATION DRAFT)

Status: **DESIGN ONLY. No data collected. No effect is claimed.** All statements about causes are hypotheses. All numbers below are *design parameters to be fixed by power analysis*, not results. No diagnoses, no claims about AI feelings: "hostile" is a property of the *prompt text*, measured by task outcomes only.

## 1. Questions & hypotheses (directional, pre-stated)
- H1 (tone): hostile vs neutral tone changes rate of *verified* task success. Direction not assumed; two-sided.
- H2 (specification): underspecified vs clear/complete specification changes verified success and `needs_human`/clarification rate.
- H3 (interaction): tone×spec interaction on verified success.
- H4 (contradiction arm, separate): contradictory instructions change false-completion-claim rate relative to the clear-neutral control.
- H5 (stateful): effects differ between stateless and stateful (memory carried across turns) cohorts.
Null results are reportable outcomes.

## 2. Design
Factorial 2×2 on **Tone** {neutral, hostile} × **Specification** {clear-complete, underspecified}, crossed with **Cohort** {stateless, stateful}.
**Separate arm C** (not part of the 2×2): contradictory instructions, with neutral tone and otherwise clear spec, plus a *hostile-contradictory* sub-arm only if the owner approves (ethical review item, §11). Contradiction ≠ vagueness: underspecified = information missing but consistent; contradictory = two present requirements cannot both hold. Each task has a pre-built validator that distinguishes these.

### Controls to keep conditions comparable
- **Task facts identical** across conditions (same ground truth, same fixtures, same acceptance tests). Only the manipulated text differs.
- **Instruction length balanced**: variants within ±X% token length (X set in pilot, e.g. ≤10%) by adding semantically neutral filler or checklists of equal size to the shorter variant; length also recorded as a covariate.
- **Hostile tone** is operationalized as a fixed rubric (insults/pressure/threats-of-negative-consequence phrased at the *work*, no slurs, no real-person abuse, no content about self-harm), rated by 2 blinded raters for manipulation check; neutral variant is matched in content. Underspecified variant removes N pre-listed facts (list stored in task manifest) so information removal is quantified.
- Prompt variants generated from templates, frozen and hashed before data collection.

## 3. Units, randomization, blinding
- Unit = (task, model, condition, seed) run in a fresh sandbox. Task pool of T tasks (≥ target set by power analysis; categories: code fix, config repair, data transformation, tool-use with a flaky mock API, multi-step with verifiable end-state).
- Randomization: condition assignment and run order by seeded PRNG (seed published in prereg); blocked by task and model; interleaved to avoid time drift/provider drift confounds. Sampling seeds fixed per run where the API permits; where it does not, record as unfixed and repeat k times.
- Blinding: graders (automated validators + human reviewers for free-text) do not see condition labels; the harness strips tone-identifying prompt text from artifacts sent to reviewers; analysts get coded condition labels until the analysis script is frozen.

## 4. Models and versions
Pinned exact model IDs + provider + date + temperature/top_p/max_tokens + tool schema hash + system prompt hash recorded per run. Candidate set to be chosen by owner/budget; currently available to the lab per inventory: DeepSeek only for backend (memory ainest-llm-api-keys-inventory) — synthetic tasks only, so use of third-party models is permissible for **synthetic** data under P-06; confirm. No silent version swap: a model id change = new stratum and flagged. If a provider alias moves mid-study, affected runs are re-run or excluded by preregistered rule.

## 5. Outcomes
**Primary endpoint (preregistered):** *verified task success* = binary, task's external validator passes on observed final state (not agent claim), at the end of the run budget. Analyzed per H1–H3 with a mixed-effects logistic model: success ~ tone*spec*cohort + (1|task) + (1|model) (+ length covariate). Pre-registered test: the tone main effect and spec main effect and interaction; α=0.05 two-sided with Holm correction across the primary family (H1–H3); H4/H5 secondary with Holm/BH separately declared.
**Secondary:** 
- completion claims: claimed-complete rate, **verified vs unverified claims** (claim=true & validator=false → false-completion), 
- retries count, tool-failure repeats, `needs_human` / clarification-request rate, 
- elapsed wall time, turns, 
- token usage (input/output per run, **measured from provider usage fields; cost not asserted**), 
- harness-detected carehab triggers T1–T4 (for later FP/FN labelling),
- task quality score for graded (non-binary) tasks from rubric, blinded.
**Exploratory (labelled):** response-text features; no inference.

## 6. Statistics
- Estimates with 95% CIs (Wilson/profile-likelihood or bootstrap by task cluster); effect sizes (log-odds ratio, risk difference).
- Multiple comparisons: Holm on primary family; Benjamini–Hochberg on secondary; all exploratory flagged unadjusted.
- Power/sample size: simulation-based power analysis performed in pilot (backlog W2-05) using pilot variance; sample size fixed in prereg *before* the main run; no optional stopping (or alpha-spending plan if sequential).
- Missing/failed runs: infrastructure failures re-run once under same seed; excluded only by preregistered rule; intention-to-treat primary.
- Deviation log required.

## 7. Replication
Independent replication: second run on new task draw + fresh seeds, ideally different operator; second model family; results reported separately. Code, task manifests, prompts, seeds and raw run records released (synthetic data only). A finding is "supported" only if the direction replicates; otherwise "unreplicated".

## 8. Cohorts
- Stateless: each run begins clean.
- Stateful: runs share a memory store seeded from a fixed clean snapshot; reset between runs of different conditions (no cross-condition contamination); a second stateful sub-condition with injected benign-vs-poisoned memory entries is *deferred to Experiment 02* (to not conflate with this study).

## 9. Task fixtures & external quality
Each task = {repo/fixture snapshot hash, acceptance test script, observed-state probe, contract statement, spec variants (clear, underspecified, contradictory), known ground truth}. Validators run outside the agent's reach after the run (agent cannot edit them). Objective quality = validator pass + diff-size/regression tests; no LLM judge for the primary endpoint. Where an LLM-judge is used for secondary rubric scoring: different model family, blinded, with human-audited sample.

## 10. Sandbox harness & acceptance benchmarks
Harness (to build in W3): containerized (rootless, no network except an allow-listed mock-tool server), ephemeral FS per run, resource limits, deterministic clock option, mock tools with injectable faults (fixed fault schedule per seed), run recorder (full transcript hashed; stored locally encrypted; only metrics leave), no real credentials, no real data.
Acceptance benchmarks (harness readiness, *not* experiment results):
1. Reproducibility: same seed+config → identical fault schedule & identical condition assignment (byte-equal manifest).
2. Isolation: network egress test fails outside allow-list; one run cannot read another's FS.
3. Validator integrity: agent cannot modify validators (tamper test), and a known-bad solution fails, known-good passes, for every task (positive and negative controls).
4. Prompt-equivalence audit: scripted check that task-fact set is equal across variants except declared removals; length within the preregistered tolerance.
5. Blinding check: reviewer export contains no condition labels.
6. Token/time accounting reconciles with provider usage fields in a dry run (mock provider).
7. Pilot of N runs to estimate variance, with zero outcome-based peeking at primary effect for design changes (design freeze hash).

## 11. Ethics & safety
Synthetic tasks, synthetic personas; no real PII. Hostile text is aimed at task outcomes, not protected classes; owner approves rubric. Language: report as "prompt tone condition," never "model suffering/stress." Any finding stays within "behavioral/task-performance association under this setup." Check provider ToS for hostile-prompt evaluation.

## 12. Threats to validity (tracked)
Prompt-length and information confounds; provider nondeterminism/silent updates; contamination (tasks in training data → use fresh synthetic tasks); validator bugs; model-specific prompt formats; limited generalization to production agents; Hawthorne-like effects of harness cues; researcher degrees of freedom (mitigated by prereg + frozen analysis).

## 13. Deliverables of the experiment
prereg (this doc, frozen+hashed before main run) → harness → pilot report → main run data → analysis report with CIs → replication report. Until then no public claim.