Compare commits

..
Author SHA1 Message Date
David Peter 49ebab2eb4 Correct the interpretation of Welch's t-test results 2026-10-02 11:29:44 +02:00
2 changed files with 10 additions and 26 deletions

No files matched your search

+3 -22
View File
@@ -3,8 +3,7 @@
hyperfine \- command\-line benchmarking tool
.SH SYNOPSIS
.B hyperfine
.RB [ \-hVN ]
.RB [ \-\-ignore\-failure[=MODE] ]
.RB [ \-ihVN ]
.RB [ \-\-warmup
.IR NUM ]
.RB [ \-\-min\-runs
@@ -15,10 +14,6 @@ hyperfine \- command\-line benchmarking tool
.IR NUM ]
.RB [ \-\-setup
.IR CMD ]
.RB [ \-\-reference
.IR CMD ]
.RB [ \-\-reference\-name
.IR NAME ]
.RB [ \-\-prepare
.IR CMD ]
.RB [ \-\-conclude
@@ -114,16 +109,6 @@ for compiling your software or with the provided parameters, or to do any
other work that should happen once before a series of benchmark runs,
not every time as would happen with the \fB\-\-prepare\fR option.
.HP
\fB\-\-reference\fR \fICMD\fP
.IP
Benchmark \fICMD\fP as the reference for the relative comparison of results.
If this option is not specified, the fastest command is used as the reference.
.HP
\fB\-\-reference\-name\fR \fINAME\fP
.IP
Give a meaningful \fINAME\fP to the reference command.
Requires \fB\-\-reference\fR.
.HP
\fB\-p\fR, \fB\-\-prepare\fR \fICMD...\fP
.IP
Execute \fICMD\fP before each timing run. This is useful for clearing disk caches,
@@ -222,13 +207,9 @@ things like "sleep 0.1; sleep 0.2" are not possible without a shell.
.IP
An alias for '\-\-shell=none'.
.HP
\fB\-i\fR, \fB\-\-ignore\-failure\fR[=\fIMODE\fP]
\fB\-i\fR, \fB\-\-ignore\-failure\fR
.IP
Ignore failures of the benchmarked programs. Without a value or with
\fBall\-non\-zero\fR, all non\-zero exit codes are ignored.
Alternatively, provide a comma\-separated list of exit codes to ignore,
for example \fB\-\-ignore\-failure=1,2\fR.
An explicit value must be attached to the option with an equals sign.
Ignore non\-zero exit codes of the benchmarked programs.
.HP
\fB\-\-style\fR \fITYPE\fP
.IP
+7 -4
View File
@@ -7,8 +7,7 @@
# ///
"""This script performs Welch's t-test on a JSON export file with two
benchmark results to test whether or not the two distributions are
the same."""
benchmark results to test for a difference in their population mean runtimes."""
import argparse
import json
@@ -40,6 +39,10 @@ print(f"t = {t:.3}, p = {p:.3}")
print()
if dispose:
print(f"There is a difference between the two benchmarks (p < {th}).")
print(
f"A statistically significant difference in mean runtime was detected (p < {th})."
)
else:
print(f"The two benchmarks are almost the same (p >= {th}).")
print(
f"No statistically significant difference in mean runtime was detected (p >= {th})."
)