Compare commits

..
Author SHA1 Message Date
David Peter 49ebab2eb4 Correct the interpretation of Welch's t-test results 2026-10-02 11:29:44 +02:00
2 changed files with 9 additions and 7 deletions

No files matched your search

+2 -3
View File
@@ -33,9 +33,8 @@ hyperfine 'sleep 0.3'
```
Hyperfine will automatically determine the number of runs to perform for each command. By default,
it will perform *at least* 10 benchmarking runs and estimate a run count targeting roughly
3 seconds, including shell overhead and preparation and conclusion commands. To set an exact
number of runs, you can use the `-r`/`--runs` option:
it will perform *at least* 10 benchmarking runs and measure for at least 3 seconds. To change this,
you can use the `-r`/`--runs` option:
```sh
hyperfine --runs 5 'sleep 0.3'
```
+7 -4
View File
@@ -7,8 +7,7 @@
# ///
"""This script performs Welch's t-test on a JSON export file with two
benchmark results to test whether or not the two distributions are
the same."""
benchmark results to test for a difference in their population mean runtimes."""
import argparse
import json
@@ -40,6 +39,10 @@ print(f"t = {t:.3}, p = {p:.3}")
print()
if dispose:
print(f"There is a difference between the two benchmarks (p < {th}).")
print(
f"A statistically significant difference in mean runtime was detected (p < {th})."
)
else:
print(f"The two benchmarks are almost the same (p >= {th}).")
print(
f"No statistically significant difference in mean runtime was detected (p >= {th})."
)