jkleinsc

#55728: build: add benchmark regression gauges and a version-history workflow

Merged
Created: Nov 10, 2026, 4:31:21 PM
Merged: Nov 16, 2026, 1:09:30 PM
1 comments
Target: main

Requested by John Kleinschmidt · Slack thread

Description of Change

This makes two additions to the crossbench benchmarks in script/benchmarks/.

Regression gauges. compare.mts now also sends electron.perf.regression and electron.perf.improvement. Each is a 0/1 gauge per metric and comparison that follows the comparator's existing verdict: it is direction-aware, needs p < 0.05 from both Welch's t-test and Mann-Whitney U, and needs at least the minimum relative change. The tags are metric, benchmark, comparison, platform, arch, electron_version and baseline_version, all low-cardinality, so a monitor such as max(last_4h):max:electron.perf.regression{*} by {metric,comparison} > 0 can alert on them. The existing series are unchanged.

Benchmarks History workflow. benchmarks-history.yml is dispatched by hand and shows performance across Electron majors. resolve-history.mts uses fiddle-core to resolve each requested major to its latest stable release. The workflow then runs one job per major through pipeline-segment-benchmarks.yml, six at a time, on the same ARC runner and container as the nightly run. Each job runs that major against a fixed anchor (default: 45.3.0, pinned so later per-major runs stay comparable; pass latest-stable to use the newest stable release instead), interleaved as the nightly run does. Runners are shared cloud VMs that change hosts between jobs, so raw numbers from different jobs can't be compared. The anchor-relative ratio can, so each job sends electron.perf.history.ratio.<metric> (candidate median / anchor median) alongside the raw electron.perf.history.<metric>. Both are tagged electron_major, electron_version, chromium_version, anchor_version, history_run_id, platform, arch and cpu_model. Uploading defaults to on for this workflow, and history jobs send no regression gauges or nightly series.

Stored results. Both workflows now also write each job's metadata.json, comparison.json and summary.md to the benchmarks blob container (https://dev-cdn.electronjs.org/benchmarks/), from a separate upload-results job. That job can't succeed until the paired electron/infra storage PR is applied and the benchmarks-upload environment is created.

Inputs and defaults: majors = 15-latest (ranges and lists, e.g. 30-45), anchor = 45.3.0 (pinned so later per-major runs stay comparable; latest-stable is still accepted), rounds = 5, suites = the nightly set, and upload-to-datadog = true.

Changes for old versions:

  • run.mts --best-effort stops running a suite the candidate failed in both of the first two rounds. It fails the job only when nothing was measured. History jobs also cap each invocation at 15 minutes.
  • The harness now probes protocol capabilities after ready. Before Electron 25, protocol has no methods until then, so registerFileProtocol used to be reported missing on Electron 15 to 22.
  • run.mts trims logs over 32 MiB. Electron 25's chromedriver once wrote 1.7 GB to a single story's log after losing its connection.

Electron 15 is the floor, because the harness needs node: imports. In local single-round runs, loading worked on 15, 18, 20, 22, 25, 28, 30, 33 and 36, and every suite ran on 15, 22, 25 and 30. Some limits apply: trace FCP and LCP need 22 or newer, trace resize needs 40 or newer, and protocol.handle stories need 25 or newer. The README has the details. A job should take about an hour (5 rounds at about 10 minutes each on the nightly runner), so the timeouts are 120 minutes for the job and 105 for the run step.

Checklist

Release Notes

Notes: none


Generated by Claude Code

Backports

No Backports Requested

This pull request doesn't have any backports requested or created for older release branches.

What are backports?

Backports are copies of changes made to the main branch that are applied to older release branches. They ensure that bug fixes and important changes are available in maintained older versions of Electron.

Semver Impact

Major
Breaking changes
Minor
New features
Patch
Bug fixes
None
Docs, tests, etc.

Semantic Versioning helps users understand the impact of updates:

  • Major (X.y.z): Breaking changes that may require code modifications
  • Minor (x.Y.z): New features that maintain backward compatibility
  • Patch (x.y.Z): Bug fixes that don't change the API
  • None: Changes that don't affect using facing parts of Electron