Compare commits

...
Author SHA1 Message Date
David Peter 29ec4accd1 Document Linux peak RSS measurement limitation 2026-10-06 13:18:57 +02:00
David Peter 7395dde05d Document the new JSON export format in the changelog (#985)
* docs: document JSON schema v2 breaking changes

* docs: simplify JSON example introduction

* Update `--export-json` to schema version 2

Updated the JSON format for `--export-json` to schema version 2, which now includes metadata, per-run measurements, and statistical summaries. Consumers must update their code to accommodate the new structure.

* Revise breaking changes in CHANGELOG

Updated breaking changes section to clarify command execution and JSON export format.
2026-10-06 09:35:00 +02:00
David Peter e262cd6ea5 Support hardware counters in plotting scripts (#984) 2026-10-06 09:09:18 +02:00
5 changed files with 63 additions and 6 deletions

No files matched your search

+44 -2
View File
@@ -2,8 +2,50 @@
## Breaking changes ## Breaking changes
- Execute commands directly by default (`--shell=none`). Use `-S` (an alias for `--shell=default`) to restore the previous behavior (`sh` on Unix, `cmd.exe` on Windows), or select a shell with `--shell <SHELL>`. - Benchmarked commands are now always executed directly, without an intermediate shell (`--shell=none`). Use `-S` (an alias for `--shell=default`) to restore the previous behavior (`sh` on Unix, `cmd.exe` on Windows), or select a shell with `--shell <SHELL>`.
- Repurpose `-S` as an alias for `--shell=default`. It no longer accepts a shell argument; replace `-S <SHELL>` with `--shell <SHELL>`. - The JSON format for `--export-json` has changed (schema version 2). The new format now includes metadata, per-run measurements, and statistical summaries for all measured quantities, see #790 (@sharkdp). Code that reads those exported JSON files must be updated. For example, `results[i].mean` is now `results[i].summary.time_wall_clock.mean`, and `results[i].times` is replaced by `results[i].measurements[j].time_wall_clock.value`. The new structure looks like this:
```python
{
"schema_version": 2,
"metadata": {
"hyperfine_version": "1.21.0",
"start_time": "2026-10-06T12:00:00Z",
"platform": {"os": "Linux", "architecture": "x86_64"}
},
"results": [
{
"command": "sleep 1", # Actual command after parameter substitution
"name": "wait 1s", # Optional custom name, separate from the command
"parameters": {"duration": {"value": "1"}}, # Values are now objects
"measurements": [ # One entry per run, excluding warmups
{
"time_wall_clock": {"value": 1.0, "unit": "second"},
"time_user": {"value": 0.0, "unit": "second"},
"time_system": {"value": 0.0, "unit": "second"},
"memory_peak_resident": {"value": 1048576.0, "unit": "byte"},
"exit_code": 0
},
... # Further runs omitted
],
"summary": {
"time_wall_clock": {
"unit": "second",
"count": 2,
"mean": 1.0,
"stddev": 0.0,
"median": 1.0,
"min": 1.0,
"max": 1.0
}
# Other metrics have the same summary structure.
}
}
]
}
```
Times are always exported in seconds and memory in bytes.
# v1.21.0 # v1.21.0
+3 -1
View File
@@ -184,7 +184,9 @@ The JSON export includes the following metrics for each measured run (excluding
- Linux/macOS: Peak resident set size (RSS). This is the largest per-process peak among the - Linux/macOS: Peak resident set size (RSS). This is the largest per-process peak among the
command and child processes (whose usage is collected when their parents wait for them), command and child processes (whose usage is collected when their parents wait for them),
*not the simultaneous total memory of the full process tree*. *not the simultaneous total memory of the full process tree*. On Linux, measuring commands
with a very small peak RSS (below the peak inherited from hyperfine at startup, which can
be a few MiB) will currently result in that inherited value being reported instead.
- Windows: Currently not supported. - Windows: Currently not supported.
- **`cpu_cycles`**: CPU cycles consumed. - **`cpu_cycles`**: CPU cycles consumed.
+8
View File
@@ -7,6 +7,14 @@ hyperfine 'sleep 0.020' 'sleep 0.021' 'sleep 0.022' --export-json sleep.json
uv run plot_whisker.py sleep.json uv run plot_whisker.py sleep.json
``` ```
All plotting scripts accept `--metric` to select a metric. Hardware counters
(`cpu_cycles`, `instructions`, `cache_references`, `cache_misses`, and `branch_misses`)
are plotted as unscaled counts when available in the JSON export. For example:
```bash
uv run plot_histogram.py benchmark.json --metric instructions
```
### Prerequisites ### Prerequisites
Install [`uv`](https://docs.astral.sh/uv/getting-started/installation/) and run the Install [`uv`](https://docs.astral.sh/uv/getting-started/installation/) and run the
+2 -2
View File
@@ -54,14 +54,14 @@ parser.add_argument(
"--t-min", "--t-min",
dest="value_min", dest="value_min",
type=float, type=float,
help="Minimum value to display (seconds, or MiB for memory)", help="Minimum metric value to display",
) )
parser.add_argument( parser.add_argument(
"--max", "--max",
"--t-max", "--t-max",
dest="value_max", dest="value_max",
type=float, type=float,
help="Maximum value to display (seconds, or MiB for memory)", help="Maximum metric value to display",
) )
parser.add_argument( parser.add_argument(
"--log-count", "--log-count",
+6 -1
View File
@@ -5,6 +5,11 @@ METRICS = {
"time_user": ("User CPU time [s]", 1), "time_user": ("User CPU time [s]", 1),
"time_system": ("System CPU time [s]", 1), "time_system": ("System CPU time [s]", 1),
"memory_peak_resident": ("Peak resident memory [MiB]", 1024**2), "memory_peak_resident": ("Peak resident memory [MiB]", 1024**2),
"cpu_cycles": ("CPU cycles", 1),
"instructions": ("Completed CPU instructions", 1),
"cache_references": ("Cache references", 1),
"cache_misses": ("Cache misses", 1),
"branch_misses": ("Mispredicted branches", 1),
} }
@@ -14,7 +19,7 @@ def add_metric_argument(parser):
choices=METRICS, choices=METRICS,
default="time_wall_clock", default="time_wall_clock",
help="Metric to plot (default: time_wall_clock). " help="Metric to plot (default: time_wall_clock). "
"Times are in seconds; memory is in MiB.", "Times are in seconds; memory is in MiB; hardware counters are unscaled counts.",
) )