#55482: build: add crossbench benchmark harness and nightly comparison workflow
Requested by John Kleinschmidt · Slack thread
Description of Change
This adds the first phase of regular performance benchmarking, built on Chromium's crossbench runner. A new daily workflow downloads the latest nightly and the previous nightly, then runs them interleaved on one ARC Linux 32-core runner. It measures first paint, reload, resize, file loading across file://, registerFileProtocol, protocol.handle + fs and protocol.handle + net.fetch, Speedometer 3.1 and JetStream 2.2. Runners change hosts between jobs, so the workflow only compares builds within a single job. A change is flagged only when Welch's t-test and Mann-Whitney U both give p < 0.05 and the medians differ by at least 10%. Medians and deltas go to Datadog and the job summary. Dispatching the workflow with two versions also works as a bisect tool. Everything lives in script/benchmarks/ and reuses PGO's pinned benchmark server, so no machines stay up between jobs.
The 10% threshold is uncalibrated, so treat the first weeks of data as a noise-floor measurement, not alerts. The pipeline has only been run end to end locally: a 4-core VM, 3 rounds, Datadog dry run. It hasn't run yet on the ARC runner or in the build container with actions/setup-python. MotionMark is left out because the MotionMark revision PGO pins doesn't work with crossbench's MotionMark probe.
Checklist
- I have built and tested this change
- I have filled out the PR description
- I have reviewed and verified the changes
- tests are changed or added
Release Notes
Notes: none
Generated by Claude Code
Backports
No Backports Requested
This pull request doesn't have any backports requested or created for older release branches.
What are backports?
Backports are copies of changes made to the main branch that are applied to older release branches. They ensure that bug fixes and important changes are available in maintained older versions of Electron.
Semver Impact
Semantic Versioning helps users understand the impact of updates:
- Major (X.y.z): Breaking changes that may require code modifications
- Minor (x.Y.z): New features that maintain backward compatibility
- Patch (x.y.Z): Bug fixes that don't change the API
- None: Changes that don't affect using facing parts of Electron