Electron Releases

macOS release builds have been failing in the out/ffmpeg step with:

err: remote-exec <digest> failed: rpc error: code = FailedPrecondition
  desc = Failed to obtain input directory
  "out/x/sdk/xcode_links/MacOSX15.5.sdk/System/Cryptexes/...":
  Shard N: Object not found

The main out/Default build (22k+ remote actions using the same remote cluster) succeeds every time; only the 535-step out/ffmpeg build fails, consistently, across all four mac variants and multiple retries.

What we found

The two builds use different SDKs.

e build injects mac_sdk_path = "//out/<dir>/xcode_links/electron/MacOSX<ver>.sdk" into args.gn, pointing at the hermetic SDK that build-tools downloads (e-build.ts getGNArgs). out/Default is generated this way.

gn gen out/ffmpeg in this action is a raw invocation, so mac_sdk_path is unset and //build/config/mac/mac_sdk.gni falls through to sdk_info.py → xcrun -sdk macosx --show-sdk-path, which returns the GitHub runner's system Xcode SDK. The failed compile command in the CI log confirms -isysroot sdk/xcode_links/MacOSX15.5.sdk (system Xcode 16.4), not the hermetic SDK.

So the main build uploads the hermetic SDK's sysroot tree to CAS; the ffmpeg build needs a completely different one.

Why that surfaces as a CAS "object not found".

Siso precomputes the -isysroot directory as a single merkle subtree. Before using it, it calls FindMissingBlobs with only the subtree root digest; if the root is present it returns immediately and uploads the rest of the subtree in a background goroutine (build/tree_input.go subtree.init). Actions are dispatched without waiting for that background upload, with the subtree attached as TreeEntry{Digest: d, Store: nil} so the per-action uploader never re-checks its contents.

If the root digest is in CAS but some intermediate directory blob under it (here, Cryptexes / Cryptexes/OS) is not, the worker fails to materialize the input tree and Buildbarn returns FailedPrecondition. Siso treats FailedPrecondition as non-retryable and excluded from local fallback (build/run_remote.go runRemote), so the build hard-fails.

Why now.

The system-Xcode sysroot tree is only exercised by this release-only ffmpeg step. The run history shows no Publish MacOS runs between the last success (Apr 2) and the first failure (Apr 6). Our Buildbarn CAS is a fixed-size ring buffer (LocalBlobAccess), so this is consistent with the small intermediate Cryptexes* directory blobs rotating out on their shards during that gap while the root digest survived on a different shard — we did not directly observe the eviction, but wiping individual storage shards did not change the behaviour, which is what you'd expect if the root digest lives on a shard that wasn't wiped.

Fix

Grep mac_sdk_path out of out/Default/args.gn (which e build has already written by this point) and pass it to the ffmpeg gn gen. On non-mac the line isn't present, MAC_SDK_ARG is empty, and the args are unchanged.

This also means the release libffmpeg is no longer built against whatever Xcode the GitHub runner image happens to ship, which is probably what we wanted anyway.

The siso behaviour (root-only presence check + no fallback on FailedPrecondition for CAS-side Object not found) is worth raising upstream separately.

Notes: none