Hypothesis Testing Experiment Log

Template Overview

Scientific experimentation requires rigorous pre-registration of hypotheses to prevent post-hoc rationalization (HARKing) and p-hacking. The Hypothesis Testing Experiment Log provides experimental scientists and data analysts with a formal protocol to document null and alternative hypotheses, statistical power parameters, operationalized variables, and empirical testing outcomes in clean, version-controlled Markdown.

Markdown Template Structure

Copy the block below or click Download .md File to add this template directly to your personal vault:

---
title: "Experiment: [Experiment Code / Test Name]"
date_initiated: [YYYY-MM-DD]
lead_researcher: "[[Lead Scientist / Analyst]]"
statistical_significance_alpha: 0.05
target_statistical_power_beta: 0.80
test_type: "[Two-Tailed T-Test / Mann-Whitney U / Chi-Square / Linear Regression]"
tags: [experimentation, statistics, hypothesis-testing, data-science, reproducibility]
---

# Hypothesis Testing Experiment Log — [Experiment Code]

## 1. Formal Hypotheses & Pre-Registration
- **Null Hypothesis (H0):** There is no difference in transcription processing latency between floating-point FP16 models and quantized INT4 models (`μ1 - μ2 = 0`).
- **Alternative Hypothesis (H1):** Quantized INT4 models exhibit statistically significant latency reduction compared to FP16 models (`μ1 - μ2 ≠ 0`).
- **Directionality:** Two-tailed test with pre-registered significance threshold `α = 0.05`.
- **Pre-Registration Timestamp:** Registered prior to running inference batch on [YYYY-MM-DD].

## 2. Experimental Variables & Controls
- **Independent Variable (IV):** Model Quantization Format (`FP16` vs `INT4`).
- **Dependent Variable (DV):** Processing Latency per 60-second audio benchmark chunk (measured in milliseconds).
- **Controlled Constants:** Hardware platform (Apple Silicon M3 Max, 36GB unified memory), ambient temperature (21°C), background daemon activity (idle).

## 3. Power Analysis & Sample Size Determination
- **Anticipated Effect Size:** Cohen's `d = 0.50` (Moderate effect).
- **Required Minimum Sample Size (N):** 64 samples per condition calculated via G*Power.
- **Actual Sample Size Collected:** [N = 80] runs per condition.

## 4. Empirical Data & Statistical Output
| Condition Group | Sample Size (N) | Mean Value (x̄) | Standard Deviation (s) | Median Value | Interquartile Range (IQR) |
|---|---|---|---|---|---|
| Control (FP16) | 80 | 1,420 ms | 85 ms | 1,415 ms | 110 ms |
| Experimental (INT4) | 80 | 480 ms | 32 ms | 478 ms | 42 ms |

## 5. Statistical Test Evaluation & Conclusion
- **Calculated Test Statistic:** `t = 91.24` (Degrees of Freedom: `df = 158`).
- **Observed P-Value:** `p < 0.0001` (Significantly below `α = 0.05`).
- **Calculated Effect Size:** Cohen's `d = 14.4` (Massive practical significance).
- **Decision:** **Reject Null Hypothesis (H0)** in favor of Alternative Hypothesis (H1).

## 6. Threats to Validity & Discussion
- **Internal Validity:** Ambient thermal throttling was ruled out by monitoring continuous CPU/GPU temperatures across all 160 consecutive benchmark runs.
- **External Validity:** Test was conducted on clean synthetic benchmark audio; performance on reverberant conference room audio requires separate empirical validation.

## 7. Next Actions & Code Commit
- [ ] Merge INT4 inference pipeline into main repository branch.
- [ ] Archive raw CSV benchmark data in `/data/experiments/exp-2026-10/`.

Filled Real-World Example

Here is a realistic example demonstrating how this template captures actual deliberations:

---
title: "Experiment: ANE-STREAM-04"
date_initiated: 2026-10-09
lead_researcher: "[[Dr. Henrik Lindqvist]]"
statistical_significance_alpha: 0.01
target_statistical_power_beta: 0.90
test_type: "Paired Two-Tailed T-Test"
tags: [experimentation, statistics, hypothesis-testing, data-science, reproducibility]
---

# Hypothesis Testing Experiment Log — ANE-STREAM-04

## 1. Formal Hypotheses
- **H0:** Core ML Apple Neural Engine scheduling provides zero reduction in battery power draw during continuous audio transcription compared to GPU execution (`μ_ANE - μ_GPU = 0`).
- **H1:** Core ML Apple Neural Engine scheduling significantly reduces system power draw in Watts compared to GPU execution (`μ_ANE - μ_GPU ≠ 0`).
- **Significance Level:** Strict `α = 0.01`.

## 2. Experimental Data
| Hardware Pipeline | N Samples | Mean Power (Watts) | Std Dev (W) |
|---|---|---|---|
| GPU Pipeline | 50 | 14.8 W | 1.2 W |
| Neural Engine Pipeline | 50 | 4.2 W | 0.4 W |

## 3. Statistical Test Result
- Paired T-Test: `t = 58.7`, `df = 49`, `p < 0.00001`.
- Decision: **Reject H0**. Neural engine scheduling demonstrates a 71.6% reduction in active battery consumption.

## 4. Next Steps
- Commit Neural Engine prioritization flag to production settings module.

When to Use and When to Avoid

When to Use

• Use whenever conducting formal hypothesis testing, A/B benchmarks, or scientific laboratory trials.
• Use to establish auditable pre-registration records that prevent p-hacking.
• Use to document statistical power calculations and sample size justifications.

When to Avoid

• Avoid using for qualitative open-ended interviews without numerical variables.
• Avoid using for routine operational maintenance logs.

On-Device AI Integration

Summarize your recording in Oyma with this local instruction:

Review the hypothesis testing log:
1. Verify the statistical test selection against the distribution characteristics of the data.
2. Check whether the calculated degrees of freedom and effect sizes align with reported sample sizes.
3. Draft a clean results paragraph in standard APA academic publication format.
Output in structured Markdown.

Your data remains securely on your Mac with zero cloud exposure.