diff --git a/doc/source/arch_speculative_actions.rst b/doc/source/arch_speculative_actions.rst new file mode 100644 index 000000000..c713f69a2 --- /dev/null +++ b/doc/source/arch_speculative_actions.rst @@ -0,0 +1,502 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +.. _speculative_actions: + +Speculative Actions +=================== + +Speculative actions speed up rebuilds by pre-populating the action cache +with adapted versions of previously recorded build actions. When a dependency +changes, the individual compile and link commands from the previous build +are adapted with updated input digests and executed ahead of the actual +build, so that by the time recc runs the same commands, they hit the +action cache instead of being executed from scratch. + + +Overview +-------- + +A typical rebuild scenario: a developer modifies a low-level library +(e.g. a base SDK element). Every downstream element needs rebuilding +because its dependency changed. But the downstream elements' own source +code hasn't changed — only the dependency artifacts are different. +Speculative actions exploit this by: + +1. **Recording** subactions from the previous build (via recc through + the ``remote-apis-socket``) +2. **Generating** overlays that describe how each subaction's input files + relate to source elements and dependency artifacts +3. **Storing** the speculative actions on the artifact proto, keyed by + the element's weak cache key (stable across dependency version changes) +4. **Priming** the action cache on the next build by instantiating the + stored actions with current dependency digests and executing them + + +Subaction Recording +------------------- + +When an element builds with ``remote-apis-socket`` configured and +``CC: recc gcc`` as the compiler, each compiler invocation goes through +recc, which sends an ``Execute`` request to buildbox-casd's nested +server via the socket. buildbox-casd records each action digest as a +subaction. When the sandbox's ``StageTree`` session ends, the subaction +digests are returned in the ``StageTreeResponse`` and added to the +parent ``ActionResult.subactions`` field. + +BuildStream reads ``action_result.subactions`` after each sandbox +command execution (``SandboxREAPI._run()``) and accumulates them on +the sandbox object. After a successful build, ``Element._assemble()`` +transfers them to the element via ``_set_subaction_digests()``. + + +Overlay Generation +------------------ + +The ``SpeculativeActionsGenerator`` runs after the build queue. For each +element with subaction digests: + +1. Builds a **digest cache** mapping file content hashes to their origin: + + - **SOURCE** overlays: files from the element's own source tree + - **ARTIFACT** overlays: files from dependency artifacts + - SOURCE takes priority over ARTIFACT when the same digest appears + in both + +2. For each subaction, fetches the ``Action`` proto and traverses its + input tree to find all file digests. Each digest that matches the + cache produces an ``Overlay`` recording: + + - The overlay type (SOURCE, ARTIFACT, or ACTION) + - The source element name (or producing action's base digest hash + for ACTION overlays) + - The file path within the source/artifact tree + - The target digest to replace + +3. Generates **ACTION overlays** for inter-subaction dependencies, both + within the element and across dependency elements: + + - **Intra-element**: subactions are processed in order; after each, + the generator fetches its ``ActionResult`` to learn what it produced. + Later subactions whose input digests match get ACTION overlays + (e.g., link's ``main.o`` linked to the compile that produced it). + - **Cross-element**: for each dependency with stored ``SpeculativeActions``, + the generator fetches ActionResults of the dependency's subactions + and seeds the output map. If the current element's subaction input + contains an intermediate file produced by a dependency's subaction + (not in the artifact — those are ARTIFACT overlays), a cross-element + ACTION overlay is created with ``source_element`` set to the + dependency name. Subsequently, a copy of the SpeculativeAction that is referenced by the ACTION overlay, is added to the list of speculative actions with its element field set to its originating element. We then evaluate that SA in the same way and copy in further SA's as needed, setting element to the dependency if it isn't set. We only need to walk the list of SAs of the dependency, which by definition is complete. This approach makes the SA list self-sufficient, at the cost of some duplication. + +4. Stores the ``SpeculativeActions`` proto on the artifact, which is + saved under both the strong and weak cache keys. + + +Weak Key Lookup +--------------- + +The weak cache key includes everything about the element itself (sources, +environment, build commands, sandbox config) but only dependency **names** +(not their cache keys). This means: + +- When a dependency is rebuilt with new content, the downstream element's + weak key remains **stable** +- The speculative actions stored under the weak key from the previous + build are still **reachable** +- When the element's own sources or configuration change, the weak key + changes, correctly **invalidating** stale speculative actions + + +Overlay Fallback Resolution +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When the same file digest appears in both a dependency's source tree +and its artifact (e.g. a header file), both SOURCE and ARTIFACT +overlays are generated. At instantiation time, they are tried in +priority order: **SOURCE first, then ACTION, then ARTIFACT**. + +- **SOURCE** overlays enable the earliest resolution — as soon as + the element's sources are fetched, before any build completes. +- **ACTION** overlays resolve intermediate files (e.g. ``.o`` files) + that are produced by prior subactions but not present in artifacts. + They are tried before ARTIFACT because they provide a more direct + resolution path for intermediate files. +- **ARTIFACT** overlays serve as a fallback when sources are not + available (dependency not rebuilding this invocation — its artifact + is already cached). + +Overlay data availability at priming time: + +- If a referenced element is **not rebuilding**: its sources/artifacts + haven't changed, so the overlay's target digest remains valid and + ARTIFACT resolution succeeds from the cached artifact. +- If a referenced element **is rebuilding**: its old artifact is + invalidated (new strong key), so ARTIFACT resolution returns None. + SOURCE resolution may succeed if the Fetch queue has already run. + If neither resolves, the subaction is deferred until the dependency's + sources become available or its artifact is cached. + + +Action Instantiation +-------------------- + +The ``SpeculativeActionInstantiator`` adapts stored actions for the +current dependency versions: + +0. **Already-instantiated check**: if the base action's hash is found + in the global ``instantiated_actions`` dict, returns the previously + adapted digest immediately (avoids redundant work when multiple + elements reference the same dependency subaction) +1. Fetches the base action from CAS +2. Resolves each overlay with fallback (first resolved wins per target + digest), in priority order **SOURCE > ACTION > ARTIFACT**: + + - **SOURCE** overlays: finds the current file digest in the element's + source tree by path + - **ACTION** overlays: looks up the producing subaction's adapted + digest in the global ``instantiated_actions`` dict, then fetches + the ``ActionResult`` from the action cache to find the output + file's current digest. If the producing action was never + instantiated, the overlay is dropped gracefully. + - **ARTIFACT** overlays: finds the current file digest in the + dependency's artifact tree by path + +3. Builds a digest replacement map (old hash → new digest), skipping + when old hash == new digest. If the replacement map is empty, the + SA is marked as done. +4. Recursively traverses the action's input tree, replacing file digests +5. Stores the modified action in CAS +6. If no digests changed, returns the base action digest (already cached) + + +Pipeline Integration +-------------------- + +The scheduler queue order with speculative actions enabled:: + + Pull → Fetch → Priming → Build → Generation → Push + +**Pull Queue**: For elements not cached by strong key, also pulls the +weak key artifact proto from remotes. This is a lightweight pull — just +the metadata, not the full artifact files. + +**Priming Queue** (``SpeculativeCachePrimingQueue``): Runs before the +build queue. Uses the PENDING state to hold elements while their +dependencies build, running background priming concurrently. + +Elements without stored SpeculativeActions skip this queue entirely. +Elements that are already buildable (all deps cached) get a single +priming pass as a job. Elements with unbuilt dependencies enter as +PENDING: + +1. ``register_pending_element``: sets a per-dep callback + (``_set_build_dep_cached_callback``) and launches background + priming in the scheduler's thread pool +2. **Background priming**: pre-fetches CAS blobs, instantiates + subactions whose overlays are resolvable from already-cached deps, + submits them fire-and-forget (reads first stream response to + confirm acceptance, then drops the stream). Each instantiated + action is recorded in the global ``instantiated_actions`` dict. +3. **Per-dep callback**: as each dependency becomes cached, the + callback triggers incremental priming — newly resolvable ARTIFACT + and ACTION overlays are resolved and submitted +4. **Final pass** (element becomes buildable): all dependencies are + built, all ``ActionResults`` are in the action cache. Remaining + ACTION overlays are resolved via the global ``instantiated_actions`` + dict. Remaining subactions are submitted fire-and-forget. +5. Element proceeds to BuildQueue with all actions primed + +Unchanged actions (instantiated digest equals base digest) skip +submission — they are already in the action cache from the previous +build. + +**Global instantiated_actions**: A shared dict +(``base_action_hash → adapted_action_digest``) accessible to all +elements during priming. When element A instantiates a subaction, +the mapping is immediately visible to element B's priming. This +enables cross-element ACTION overlay resolution — element B can look +up element A's adapted subaction digest to find intermediate files +(e.g. generated headers) that aren't in artifacts. The dict is +protected by a threading lock for write access; reads are safe under +the GIL. + +**Build Queue**: Builds elements as usual. When recc runs a compile or +link command, it checks the action cache first. If priming succeeded, +the adapted action is already cached → **action cache hit**. + +**Generation Queue** (``SpeculativeActionGenerationQueue``): Runs after +the build queue. Generates overlays from newly recorded subactions and +stores them for future priming. + + +Example Scenarios +----------------- + +The following scenarios illustrate how speculative actions behave across +different dependency change patterns. In each case, "unchanged" means +the element's own sources did not change (its weak key is stable), so +its stored SA is available for priming. + + +Single dependency change +~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + base (sources CHANGED) → liba (unchanged) → app (unchanged) + +The most common CI scenario: a low-level element is modified, all +downstream elements need rebuilding. + +- **base**: weak key changed → no SA available → builds from scratch. +- **liba**: weak key unchanged → SA available. + + - SOURCE overlays (liba's own ``.c`` files): resolve immediately. + - ARTIFACT overlays (base's headers): deferred until base builds. + - When base finishes → per-dep callback fires → ARTIFACT overlays + resolve → liba's compile actions are submitted fire-and-forget. + - ACTION overlays (intra-element, e.g. ``ar`` consuming ``.o`` files + from compile): resolve sequentially from ``instantiated_actions`` + once the compile actions complete. + +- **app**: same pattern — waits for liba, then resolves and submits. + +Result: every downstream element gets full cache hits on all subactions. + + +Cross-element intermediate files +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + codegen (unchanged) → liba (unchanged) + | | + generates gen.h compiles with gen.h + +codegen's build produces ``gen.h`` as a subaction output. liba's +compile subaction uses ``gen.h`` as an input, tracked by a cross-element +ACTION overlay. + +- **codegen**: weak key unchanged → SA available → primed. Its compile + action is recorded in ``instantiated_actions``. +- **liba**: ACTION overlay for ``gen.h`` looks up codegen's subaction + in ``instantiated_actions`` → found → fetches ``ActionResult`` from + AC → resolves ``gen.h``'s adapted digest → submitted. + +Result: the global ``instantiated_actions`` dict enables cross-element +resolution of intermediate files. Without the global dict (per-element +state only), liba would not see codegen's adapted digest. + + +Intra-element action chains (compile → archive) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + base (sources CHANGED) → liba (unchanged) + compile: base.h + liba.c → liba.o + archive: ar rcs libliba.a liba.o + +liba's archive action depends on ``.o`` files produced by liba's own +compile actions. These ``.o`` files are intermediate — they are not +installed in artifacts. + +- **liba**: priming processes subactions in order. + + 1. Compile action: ARTIFACT overlay for ``base.h`` deferred until + base builds. When base finishes → resolves → submitted → + recorded in ``instantiated_actions``. + 2. Archive action: ACTION overlay for ``liba.o`` looks up compile's + hash in ``instantiated_actions`` → found → fetches + ``ActionResult`` → resolves ``liba.o`` → submitted. + +Result: the full compile → archive chain fires as soon as base completes. +Downstream elements that depend on ``libliba.a`` via ARTIFACT overlays +resolve once liba's artifact is cached. + + +Changed element breaks the action chain +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + codegen (sources CHANGED) → liba (unchanged) → app (unchanged) + | | + generates gen.h compiles with gen.h + +When codegen's sources change, its weak key changes, so its SA is +unavailable and codegen builds from scratch. Its subactions are never +primed and do not appear in ``instantiated_actions``. + +- **liba**: ACTION overlay for ``gen.h`` references codegen's subaction + → ``instantiated_actions.get(...)`` returns None → **overlay dropped**. + If ``gen.h`` is also installed in codegen's artifact, an ARTIFACT + overlay exists as fallback → resolves once codegen finishes building. + If ``gen.h`` is truly intermediate (not in the artifact), that + specific compile action cannot be adapted and falls back to full + execution during liba's build. + +- **liba finishes priming**: its adapted actions are recorded in + ``instantiated_actions``. From this point, all downstream elements + (app, etc.) can resolve ACTION overlays referencing liba's subactions. + +Result: one level of delay (codegen must build before the chain +resumes), but the chain propagates from liba onward. See +:ref:`referenced_speculative_actions` for a future optimization that +would eliminate this delay. + + +Multiple source changes in a chain +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + base (sources CHANGED) → liba (sources CHANGED) → app (unchanged) + +Both base and liba have changed sources, so both have changed weak keys +and no SAs available. Both build from scratch. + +- **app**: weak key unchanged → SA available. + + - ARTIFACT overlays for liba: deferred until liba builds → resolve. + - ACTION overlays for liba's subactions: liba was never primed → + ``instantiated_actions`` has no entries for liba's subactions → + **overlays dropped**. App's compile actions that depend on liba's + intermediate files (e.g. ``.o`` files not in the artifact) cannot + be adapted. + +Result: app gets cache hits for subactions that only depend on +artifacts (the common case), but misses on subactions that depend on +intermediate files from liba. This is a graceful degradation — those +subactions execute normally during app's build. + + +Dependency adapted but sources unchanged +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + base (deps CHANGED, not sources) → liba (unchanged) → app (unchanged) + +base's own sources didn't change (weak key stable), but one of base's +dependencies changed, so base has a different strong key. + +- **base**: weak key unchanged → SA available → primed. Adapted + actions recorded in ``instantiated_actions``. +- **liba**: ACTION overlays for base's subactions → found in + ``instantiated_actions`` (populated by base's priming) → resolve. +- **app**: similarly resolves via ``instantiated_actions``. + +Result: the global dict ensures that base's adapted digests propagate +to all downstream elements, even though base's artifact hasn't changed +content-wise. Without the global dict, liba would fail to look up +base's adapted digests. + + + +Scaling Considerations +---------------------- + +Execute calls are full builds +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Each adapted action runs a full build command (e.g. ``gcc -c``) through +buildbox-run. For N elements with M subactions each, that's N×M Execute +calls competing for CPU with the actual build queue. + +**Mitigation**: With remote execution, priming fans out across a cluster. +Locally, casd's ``--jobs`` flag limits concurrent executions. + +FetchTree calls are sequential +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The pre-fetch phase does one ``FetchTree`` per base action. For an +element with many subactions, this is many sequential calls. + +**Mitigation**: Batch ``FetchTree`` calls or parallelize them. Could +also collect all directory digests and issue a single +``FetchMissingBlobs``. + +CAS storage growth +~~~~~~~~~~~~~~~~~~ + +Every adapted action produces new directory trees in CAS. Most content +is shared (CAS deduplication), but root directories and Action protos +are unique per adaptation. CAS quota management handles eviction. + +Priming stale SA is wasteful +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If an element's build commands changed, its SA may produce adapted +actions that don't match what recc computes. The weak key includes +build configuration, so this only happens when the element itself +changed — in which case the SA is correctly invalidated. + + +.. _referenced_speculative_actions: + +Future Optimizations +-------------------- + +1. **ReferencedSpeculativeActions**: Store + ``repeated ReferencedSpeculativeActions`` on the SA proto — pointers + (``element_name``, ``sa_digest``) to dependency elements' SAs. This + enables a downstream element to instantiate a dependency's SA even + when the dependency's weak key changed (its sources changed). + + Consider this scenario:: + + codegen (sources CHANGED) → liba (unchanged) + | | + generates gen.h compiles with gen.h + + Currently, codegen's weak key changes, so its SA is unavailable. + liba's ACTION overlay for ``gen.h`` is dropped because codegen was + never primed and ``instantiated_actions`` has no entry for codegen's + subaction. liba must wait for codegen to build before the ARTIFACT + fallback (if ``gen.h`` is installed) or full execution (if ``gen.h`` + is truly intermediate) can proceed. + + With ReferencedSAs, liba's artifact would store a reference to + codegen's SA from the previous build. During priming, liba could + retrieve codegen's SA, instantiate codegen's subactions (adapting + them with codegen's new sources), and populate + ``instantiated_actions`` with codegen's adapted digests. The ACTION + overlay for ``gen.h`` would then resolve immediately, eliminating + the one-level delay. + + The benefit is most pronounced when a low-level element with + generated intermediate files (headers, protocol buffer outputs, + code-generated sources) changes frequently and has many downstream + dependents. The cost is additional complexity in SA storage and + retrieval, plus the overhead of instantiating dependency SAs during + priming. Whether this trade-off is worthwhile depends on real-world + profiling of rebuild patterns. + +2. **Topological prioritization**: Prime elements in build order + (dependencies first) to maximize the chance priming completes before + building starts. + +3. **Selective priming**: Skip cheap actions (fast link steps), prioritize + expensive ones (long compilations). Only skip when it doesn't break + SA chains. + +4. **Batch FetchTree**: Collect all input root digests and fetch in + parallel or in a single batch. + +5. **Storage**: Store SAs more efficiently so that they can be pulled + down efficiently. + +6. **Generation**: Find a way to make the output tree to input tree + matching more efficient. diff --git a/doc/source/main_architecture.rst b/doc/source/main_architecture.rst index cff4d7428..143ddd38a 100644 --- a/doc/source/main_architecture.rst +++ b/doc/source/main_architecture.rst @@ -30,4 +30,5 @@ This section provides details on the overall BuildStream architecture. arch_caches arch_sandboxing arch_remote_execution + arch_speculative_actions diff --git a/src/buildstream/_artifactcache.py b/src/buildstream/_artifactcache.py index c8328f109..b5c0bb20d 100644 --- a/src/buildstream/_artifactcache.py +++ b/src/buildstream/_artifactcache.py @@ -456,6 +456,69 @@ def _pull_artifact_storage(self, element, key, artifact_digest, remote, pull_bui return True + # pull_artifact_proto(): + # + # Pull only the artifact proto (metadata) for an element by key. + # + # This is a lightweight pull that fetches just the artifact proto + # from the remote, without fetching files, buildtrees, or other + # large blobs. Used by the speculative actions priming path to + # retrieve the SA digest reference from a previous build's artifact. + # + # Args: + # element (Element): The element whose artifact proto to pull + # key (str): The cache key to pull by (typically the weak key) + # + # Returns: + # (bool): True if the proto was pulled, False if not found + # + def pull_artifact_proto(self, element, key): + project = element._get_project() + + artifact_name = element.get_artifact_name(key=key) + uri = REMOTE_ASSET_ARTIFACT_URN_TEMPLATE.format(artifact_name) + + index_remotes, storage_remotes = self.get_remotes(project.name, False) + + # Resolve the artifact name to a digest via index remotes + artifact_digest = None + for remote in index_remotes: + remote.init() + try: + response = remote.fetch_blob([uri]) + if response: + artifact_digest = response.blob_digest + break + except AssetCacheError: + continue + + if not artifact_digest: + return False + + # Fetch the artifact blob via casd (handles remote fetching) + try: + if storage_remotes: + self.cas.fetch_blobs(storage_remotes[0], [artifact_digest]) + else: + return False + except (BlobNotFound, CASRemoteError): + return False + + # Parse and write the artifact proto to local cache + try: + artifact = artifact_pb2.Artifact() + with self.cas.open(artifact_digest, "rb") as f: + artifact.ParseFromString(f.read()) + + artifact_path = os.path.join(self._basedir, artifact_name) + os.makedirs(os.path.dirname(artifact_path), exist_ok=True) + with utils.save_file_atomic(artifact_path, mode="wb") as f: + f.write(artifact.SerializeToString()) + + return True + except (FileNotFoundError, OSError): + return False + # _query_remote() # # Args: @@ -473,3 +536,111 @@ def _query_remote(self, ref, remote): return bool(response) except AssetCacheError as e: raise ArtifactError("{}".format(e), temporary=True) from e + + # store_speculative_actions(): + # + # Store SpeculativeActions for an element's artifact. + # + # Stores using both the artifact proto field (backward compat) and + # a weak key reference (stable across dependency version changes). + # + # Args: + # artifact (Artifact): The artifact to attach speculative actions to + # spec_actions (SpeculativeActions): The speculative actions proto + # weak_key (str): Optional weak cache key for stable lookup + # + def store_speculative_actions(self, artifact, spec_actions, weak_key=None): + + # Store the speculative actions proto in CAS + spec_actions_digest = self.cas.store_proto(spec_actions) + + # Set the speculative_actions field on the artifact proto + artifact_proto = artifact._get_proto() + artifact_proto.speculative_actions.CopyFrom(spec_actions_digest) + + # Save the updated artifact proto under all keys (strong + weak). + # The artifact was originally stored under both keys; we must update + # both so that lookup_speculative_actions_by_weak_key() can find the + # SA when the strong key changes but the weak key remains stable. + element = artifact._element + keys = set() + keys.add(artifact.get_extract_key()) + if artifact.weak_key: + keys.add(artifact.weak_key) + serialized = artifact_proto.SerializeToString() + for key in keys: + ref = element.get_artifact_name(key) + proto_path = os.path.join(self._basedir, ref) + with open(proto_path, mode="w+b") as f: + f.write(serialized) + + # lookup_speculative_actions_by_weak_key(): + # + # Look up SpeculativeActions by element and weak key. + # + # Loads the artifact proto stored under the weak key ref and reads + # its speculative_actions digest. This works even when the element + # is not cached under its strong key (the common priming scenario: + # dependency changed, strong key differs, but weak key is stable + # so the artifact from the previous build is still reachable). + # + # Args: + # element (Element): The element to look up SA for + # weak_key (str): The weak cache key + # + # Returns: + # SpeculativeActions proto or None if not available + # + def lookup_speculative_actions_by_weak_key(self, element, weak_key): + from ._protos.buildstream.v2 import speculative_actions_pb2 + from ._protos.buildstream.v2 import artifact_pb2 + + if not weak_key: + return None + + # Load the artifact proto stored under the weak key ref + artifact_ref = element.get_artifact_name(key=weak_key) + proto_path = os.path.join(self._basedir, artifact_ref) + try: + with open(proto_path, mode="r+b") as f: + artifact_proto = artifact_pb2.Artifact() + artifact_proto.ParseFromString(f.read()) + except FileNotFoundError: + return None + + # Read the speculative_actions digest from the artifact proto + if not artifact_proto.HasField("speculative_actions"): + return None + + return self.cas.fetch_proto( + artifact_proto.speculative_actions, speculative_actions_pb2.SpeculativeActions + ) + + # get_speculative_actions(): + # + # Retrieve SpeculativeActions for an element's artifact. + # + # First tries the weak key path (stable across dependency version + # changes), then falls back to the artifact proto field. + # + # Args: + # artifact (Artifact): The artifact to get speculative actions from + # weak_key (str): Optional weak cache key for stable lookup + # + # Returns: + # SpeculativeActions proto or None if not available + # + def get_speculative_actions(self, artifact): + from ._protos.buildstream.v2 import speculative_actions_pb2 + + # Load from artifact proto's speculative_actions digest field + artifact_proto = artifact._get_proto() + if not artifact_proto: + return None + + # Check if speculative_actions field is set + if not artifact_proto.HasField("speculative_actions"): + return None + + # Fetch the speculative actions from CAS + return self.cas.fetch_proto(artifact_proto.speculative_actions, speculative_actions_pb2.SpeculativeActions) diff --git a/src/buildstream/_cas/cascache.py b/src/buildstream/_cas/cascache.py index 68fd4b610..92c640f28 100644 --- a/src/buildstream/_cas/cascache.py +++ b/src/buildstream/_cas/cascache.py @@ -703,6 +703,100 @@ def _send_directory(self, remote, digest): def get_cache_usage(self): return self._cache_usage_monitor.get_cache_usage() + # fetch_proto(): + # + # Fetch a protobuf message from CAS by digest and parse it. + # + # Args: + # digest (Digest): The digest of the proto message + # proto_class: The protobuf message class to parse into + # + # Returns: + # The parsed protobuf message, or None if not found + # + def fetch_proto(self, digest, proto_class): + if not digest or not digest.hash: + return None + + try: + with self.open(digest, mode="rb") as f: + proto_instance = proto_class() + proto_instance.ParseFromString(f.read()) + return proto_instance + except FileNotFoundError: + return None + except Exception: + return None + + # store_proto(): + # + # Store a protobuf message in CAS. + # + # Args: + # proto: The protobuf message instance + # instance_name (str): Optional casd instance_name for remote CAS + # + # Returns: + # (Digest): The digest of the stored proto + # + def store_proto(self, proto, instance_name=None): + buffer = proto.SerializeToString() + return self.add_object(buffer=buffer, instance_name=instance_name) + + # fetch_action(): + # + # Fetch an Action proto from CAS. + # + # Args: + # action_digest (Digest): The digest of the Action + # + # Returns: + # Action proto or None if not found + # + def fetch_action(self, action_digest): + return self.fetch_proto(action_digest, remote_execution_pb2.Action) + + # store_action(): + # + # Store an Action proto in CAS. + # + # Args: + # action (Action): The Action proto + # instance_name (str): Optional casd instance_name + # + # Returns: + # (Digest): The digest of the stored action + # + def store_action(self, action, instance_name=None): + return self.store_proto(action, instance_name=instance_name) + + # fetch_directory(): + # + # Fetch a Directory proto from CAS (not the full tree). + # + # Args: + # directory_digest (Digest): The digest of the Directory + # + # Returns: + # Directory proto or None if not found + # + def fetch_directory_proto(self, directory_digest): + return self.fetch_proto(directory_digest, remote_execution_pb2.Directory) + + # store_directory(): + # + # Store a Directory proto in CAS. + # + # Args: + # directory (Directory): The Directory proto + # instance_name (str): Optional casd instance_name + # + # Returns: + # (Digest): The digest of the stored directory + # + def store_directory_proto(self, directory, instance_name=None): + return self.store_proto(directory, instance_name=instance_name) + # _CASCacheUsage # diff --git a/src/buildstream/_context.py b/src/buildstream/_context.py index 48e8eff43..3ea819358 100644 --- a/src/buildstream/_context.py +++ b/src/buildstream/_context.py @@ -31,7 +31,7 @@ from ._remotespec import RemoteSpec, RemoteExecutionSpec from ._sourcecache import SourceCache from ._cas import CASCache, CASDProcessManager, CASLogLevel -from .types import _CacheBuildTrees, _PipelineSelection, _SchedulerErrorAction, _SourceUriPolicy +from .types import _CacheBuildTrees, _PipelineSelection, _SchedulerErrorAction, _SourceUriPolicy, _SpeculativeActionMode from ._workspaces import Workspaces, WorkspaceProjectCache from .node import Node, MappingNode @@ -164,6 +164,9 @@ def __init__(self, *, use_casd: bool = True) -> None: # What to do when a build fails in non interactive mode self.sched_error_action: Optional[str] = None + # Speculative actions mode + self.speculative_actions_mode: _SpeculativeActionMode = _SpeculativeActionMode.NONE + # Maximum jobs per build self.build_max_jobs: Optional[int] = None @@ -451,13 +454,24 @@ def load(self, config: Optional[str] = None) -> None: # Load scheduler config scheduler = defaults.get_mapping("scheduler") - scheduler.validate_keys(["on-error", "fetchers", "builders", "pushers", "network-retries"]) + scheduler.validate_keys(["on-error", "fetchers", "builders", "pushers", "network-retries", "speculative-actions"]) self.sched_error_action = scheduler.get_enum("on-error", _SchedulerErrorAction) self.sched_fetchers = scheduler.get_int("fetchers") self.sched_builders = scheduler.get_int("builders") self.sched_pushers = scheduler.get_int("pushers") self.sched_network_retries = scheduler.get_int("network-retries") + # Load speculative actions config + # Accepts mode string (none/prime-only/source-artifact/intra-element/full) + # or boolean for backward compatibility (True → full, False → none) + try: + self.speculative_actions_mode = scheduler.get_enum("speculative-actions", _SpeculativeActionMode) + except Exception: + self.speculative_actions_mode = ( + _SpeculativeActionMode.FULL if scheduler.get_bool("speculative-actions") + else _SpeculativeActionMode.NONE + ) + # Load build config build = defaults.get_mapping("build") build.validate_keys(["max-jobs", "retry-failed", "dependencies"]) diff --git a/src/buildstream/_protos/build/bazel/remote/execution/v2/remote_execution.proto b/src/buildstream/_protos/build/bazel/remote/execution/v2/remote_execution.proto index 31e20dcf4..a34596bbe 100644 --- a/src/buildstream/_protos/build/bazel/remote/execution/v2/remote_execution.proto +++ b/src/buildstream/_protos/build/bazel/remote/execution/v2/remote_execution.proto @@ -981,8 +981,8 @@ message SymlinkNode { // serializing, but care should be taken to avoid shortcuts. For instance, // concatenating two messages to merge them may produce duplicate fields. message Digest { - // The hash, represented as a lowercase hexadecimal string, padded with - // leading zeroes up to the hash function length. + // The hash. In the case of SHA-256, it will always be a lowercase hex string + // exactly 64 characters long. string hash = 1; // The size of the blob, in bytes. @@ -1220,6 +1220,13 @@ message ActionResult { // The details of the execution that originally produced this result. ExecutedActionMetadata execution_metadata = 9; + + // The digests of Actions that were executed as nested executions during + // this action (e.g., compiler invocations via recc). Each digest references + // an Action that was stored in the CAS during execution. This allows clients + // to retrieve the full dependency tree of actions that contributed to this + // result. + repeated Digest subactions = 99; } // An `OutputFile` is similar to a @@ -1433,20 +1440,6 @@ message ExecuteRequest { // length of the action digest hash and the digest functions announced // in the server's capabilities. DigestFunction.Value digest_function = 9; - - // A hint to the server to request inlining stdout in the - // [ActionResult][build.bazel.remote.execution.v2.ActionResult] message. - bool inline_stdout = 10; - - // A hint to the server to request inlining stderr in the - // [ActionResult][build.bazel.remote.execution.v2.ActionResult] message. - bool inline_stderr = 11; - - // A hint to the server to inline the contents of the listed output files. - // Each path needs to exactly match one file path in either `output_paths` or - // `output_files` (DEPRECATED since v2.1) in the - // [Command][build.bazel.remote.execution.v2.Command] message. - repeated string inline_output_files = 12; } // A `LogFile` is a log stored in the CAS. @@ -1682,7 +1675,7 @@ message BatchUpdateBlobsRequest { bytes data = 2; // The format of `data`. Must be `IDENTITY`/unspecified, or one of the - // compressors advertised by the + // compressors advertised by the // [CacheCapabilities.supported_batch_compressors][build.bazel.remote.execution.v2.CacheCapabilities.supported_batch_compressors] // field. Compressor.Value compressor = 3; diff --git a/src/buildstream/_protos/build/bazel/remote/execution/v2/remote_execution_pb2.py b/src/buildstream/_protos/build/bazel/remote/execution/v2/remote_execution_pb2.py index 569ec22aa..357b3f7fa 100644 --- a/src/buildstream/_protos/build/bazel/remote/execution/v2/remote_execution_pb2.py +++ b/src/buildstream/_protos/build/bazel/remote/execution/v2/remote_execution_pb2.py @@ -32,7 +32,7 @@ from buildstream._protos.google.rpc import status_pb2 as google_dot_rpc_dot_status__pb2 -DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n6build/bazel/remote/execution/v2/remote_execution.proto\x12\x1f\x62uild.bazel.remote.execution.v2\x1a\x1f\x62uild/bazel/semver/semver.proto\x1a\x1cgoogle/api/annotations.proto\x1a#google/longrunning/operations.proto\x1a\x19google/protobuf/any.proto\x1a\x1egoogle/protobuf/duration.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1egoogle/protobuf/wrappers.proto\x1a\x17google/rpc/status.proto\"\xa6\x02\n\x06\x41\x63tion\x12?\n\x0e\x63ommand_digest\x18\x01 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x42\n\x11input_root_digest\x18\x02 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12*\n\x07timeout\x18\x06 \x01(\x0b\x32\x19.google.protobuf.Duration\x12\x14\n\x0c\x64o_not_cache\x18\x07 \x01(\x08\x12\x0c\n\x04salt\x18\t \x01(\x0c\x12;\n\x08platform\x18\n \x01(\x0b\x32).build.bazel.remote.execution.v2.PlatformJ\x04\x08\x03\x10\x06J\x04\x08\x08\x10\t\"\xae\x04\n\x07\x43ommand\x12\x11\n\targuments\x18\x01 \x03(\t\x12[\n\x15\x65nvironment_variables\x18\x02 \x03(\x0b\x32<.build.bazel.remote.execution.v2.Command.EnvironmentVariable\x12\x18\n\x0coutput_files\x18\x03 \x03(\tB\x02\x18\x01\x12\x1e\n\x12output_directories\x18\x04 \x03(\tB\x02\x18\x01\x12\x14\n\x0coutput_paths\x18\x07 \x03(\t\x12?\n\x08platform\x18\x05 \x01(\x0b\x32).build.bazel.remote.execution.v2.PlatformB\x02\x18\x01\x12\x19\n\x11working_directory\x18\x06 \x01(\t\x12\x1e\n\x16output_node_properties\x18\x08 \x03(\t\x12_\n\x17output_directory_format\x18\t \x01(\x0e\x32>.build.bazel.remote.execution.v2.Command.OutputDirectoryFormat\x1a\x32\n\x13\x45nvironmentVariable\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t\"R\n\x15OutputDirectoryFormat\x12\r\n\tTREE_ONLY\x10\x00\x12\x12\n\x0e\x44IRECTORY_ONLY\x10\x01\x12\x16\n\x12TREE_AND_DIRECTORY\x10\x02\"{\n\x08Platform\x12\x46\n\nproperties\x18\x01 \x03(\x0b\x32\x32.build.bazel.remote.execution.v2.Platform.Property\x1a\'\n\x08Property\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t\"\x9a\x02\n\tDirectory\x12\x38\n\x05\x66iles\x18\x01 \x03(\x0b\x32).build.bazel.remote.execution.v2.FileNode\x12\x43\n\x0b\x64irectories\x18\x02 \x03(\x0b\x32..build.bazel.remote.execution.v2.DirectoryNode\x12>\n\x08symlinks\x18\x03 \x03(\x0b\x32,.build.bazel.remote.execution.v2.SymlinkNode\x12H\n\x0fnode_properties\x18\x05 \x01(\x0b\x32/.build.bazel.remote.execution.v2.NodePropertiesJ\x04\x08\x04\x10\x05\"+\n\x0cNodeProperty\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t\"\xaf\x01\n\x0eNodeProperties\x12\x41\n\nproperties\x18\x01 \x03(\x0b\x32-.build.bazel.remote.execution.v2.NodeProperty\x12)\n\x05mtime\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12/\n\tunix_mode\x18\x03 \x01(\x0b\x32\x1c.google.protobuf.UInt32Value\"\xbe\x01\n\x08\x46ileNode\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x37\n\x06\x64igest\x18\x02 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x15\n\ris_executable\x18\x04 \x01(\x08\x12H\n\x0fnode_properties\x18\x06 \x01(\x0b\x32/.build.bazel.remote.execution.v2.NodePropertiesJ\x04\x08\x03\x10\x04J\x04\x08\x05\x10\x06\"V\n\rDirectoryNode\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x37\n\x06\x64igest\x18\x02 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\"{\n\x0bSymlinkNode\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0e\n\x06target\x18\x02 \x01(\t\x12H\n\x0fnode_properties\x18\x04 \x01(\x0b\x32/.build.bazel.remote.execution.v2.NodePropertiesJ\x04\x08\x03\x10\x04\"*\n\x06\x44igest\x12\x0c\n\x04hash\x18\x01 \x01(\t\x12\x12\n\nsize_bytes\x18\x02 \x01(\x03\"\xdd\x05\n\x16\x45xecutedActionMetadata\x12\x0e\n\x06worker\x18\x01 \x01(\t\x12\x34\n\x10queued_timestamp\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12:\n\x16worker_start_timestamp\x18\x03 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12>\n\x1aworker_completed_timestamp\x18\x04 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12?\n\x1binput_fetch_start_timestamp\x18\x05 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12\x43\n\x1finput_fetch_completed_timestamp\x18\x06 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12=\n\x19\x65xecution_start_timestamp\x18\x07 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12\x41\n\x1d\x65xecution_completed_timestamp\x18\x08 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12=\n\x1avirtual_execution_duration\x18\x0c \x01(\x0b\x32\x19.google.protobuf.Duration\x12\x41\n\x1doutput_upload_start_timestamp\x18\t \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12\x45\n!output_upload_completed_timestamp\x18\n \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12\x30\n\x12\x61uxiliary_metadata\x18\x0b \x03(\x0b\x32\x14.google.protobuf.Any\"\xa7\x05\n\x0c\x41\x63tionResult\x12\x41\n\x0coutput_files\x18\x02 \x03(\x0b\x32+.build.bazel.remote.execution.v2.OutputFile\x12P\n\x14output_file_symlinks\x18\n \x03(\x0b\x32..build.bazel.remote.execution.v2.OutputSymlinkB\x02\x18\x01\x12G\n\x0foutput_symlinks\x18\x0c \x03(\x0b\x32..build.bazel.remote.execution.v2.OutputSymlink\x12L\n\x12output_directories\x18\x03 \x03(\x0b\x32\x30.build.bazel.remote.execution.v2.OutputDirectory\x12U\n\x19output_directory_symlinks\x18\x0b \x03(\x0b\x32..build.bazel.remote.execution.v2.OutputSymlinkB\x02\x18\x01\x12\x11\n\texit_code\x18\x04 \x01(\x05\x12\x12\n\nstdout_raw\x18\x05 \x01(\x0c\x12>\n\rstdout_digest\x18\x06 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x12\n\nstderr_raw\x18\x07 \x01(\x0c\x12>\n\rstderr_digest\x18\x08 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12S\n\x12\x65xecution_metadata\x18\t \x01(\x0b\x32\x37.build.bazel.remote.execution.v2.ExecutedActionMetadataJ\x04\x08\x01\x10\x02\"\xd2\x01\n\nOutputFile\x12\x0c\n\x04path\x18\x01 \x01(\t\x12\x37\n\x06\x64igest\x18\x02 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x15\n\ris_executable\x18\x04 \x01(\x08\x12\x10\n\x08\x63ontents\x18\x05 \x01(\x0c\x12H\n\x0fnode_properties\x18\x07 \x01(\x0b\x32/.build.bazel.remote.execution.v2.NodePropertiesJ\x04\x08\x03\x10\x04J\x04\x08\x06\x10\x07\"~\n\x04Tree\x12\x38\n\x04root\x18\x01 \x01(\x0b\x32*.build.bazel.remote.execution.v2.Directory\x12<\n\x08\x63hildren\x18\x02 \x03(\x0b\x32*.build.bazel.remote.execution.v2.Directory\"\xcc\x01\n\x0fOutputDirectory\x12\x0c\n\x04path\x18\x01 \x01(\t\x12<\n\x0btree_digest\x18\x03 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x1f\n\x17is_topologically_sorted\x18\x04 \x01(\x08\x12\x46\n\x15root_directory_digest\x18\x05 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.DigestJ\x04\x08\x02\x10\x03\"}\n\rOutputSymlink\x12\x0c\n\x04path\x18\x01 \x01(\t\x12\x0e\n\x06target\x18\x02 \x01(\t\x12H\n\x0fnode_properties\x18\x04 \x01(\x0b\x32/.build.bazel.remote.execution.v2.NodePropertiesJ\x04\x08\x03\x10\x04\"#\n\x0f\x45xecutionPolicy\x12\x10\n\x08priority\x18\x01 \x01(\x05\"&\n\x12ResultsCachePolicy\x12\x10\n\x08priority\x18\x01 \x01(\x05\"\xce\x03\n\x0e\x45xecuteRequest\x12\x15\n\rinstance_name\x18\x01 \x01(\t\x12\x19\n\x11skip_cache_lookup\x18\x03 \x01(\x08\x12>\n\raction_digest\x18\x06 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12J\n\x10\x65xecution_policy\x18\x07 \x01(\x0b\x32\x30.build.bazel.remote.execution.v2.ExecutionPolicy\x12Q\n\x14results_cache_policy\x18\x08 \x01(\x0b\x32\x33.build.bazel.remote.execution.v2.ResultsCachePolicy\x12N\n\x0f\x64igest_function\x18\t \x01(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.Value\x12\x15\n\rinline_stdout\x18\n \x01(\x08\x12\x15\n\rinline_stderr\x18\x0b \x01(\x08\x12\x1b\n\x13inline_output_files\x18\x0c \x03(\tJ\x04\x08\x02\x10\x03J\x04\x08\x04\x10\x05J\x04\x08\x05\x10\x06\"Z\n\x07LogFile\x12\x37\n\x06\x64igest\x18\x01 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x16\n\x0ehuman_readable\x18\x02 \x01(\x08\"\xd0\x02\n\x0f\x45xecuteResponse\x12=\n\x06result\x18\x01 \x01(\x0b\x32-.build.bazel.remote.execution.v2.ActionResult\x12\x15\n\rcached_result\x18\x02 \x01(\x08\x12\"\n\x06status\x18\x03 \x01(\x0b\x32\x12.google.rpc.Status\x12U\n\x0bserver_logs\x18\x04 \x03(\x0b\x32@.build.bazel.remote.execution.v2.ExecuteResponse.ServerLogsEntry\x12\x0f\n\x07message\x18\x05 \x01(\t\x1a[\n\x0fServerLogsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\x37\n\x05value\x18\x02 \x01(\x0b\x32(.build.bazel.remote.execution.v2.LogFile:\x02\x38\x01\"a\n\x0e\x45xecutionStage\"O\n\x05Value\x12\x0b\n\x07UNKNOWN\x10\x00\x12\x0f\n\x0b\x43\x41\x43HE_CHECK\x10\x01\x12\n\n\x06QUEUED\x10\x02\x12\r\n\tEXECUTING\x10\x03\x12\r\n\tCOMPLETED\x10\x04\"\xb5\x02\n\x18\x45xecuteOperationMetadata\x12\x44\n\x05stage\x18\x01 \x01(\x0e\x32\x35.build.bazel.remote.execution.v2.ExecutionStage.Value\x12>\n\raction_digest\x18\x02 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x1a\n\x12stdout_stream_name\x18\x03 \x01(\t\x12\x1a\n\x12stderr_stream_name\x18\x04 \x01(\t\x12[\n\x1apartial_execution_metadata\x18\x05 \x01(\x0b\x32\x37.build.bazel.remote.execution.v2.ExecutedActionMetadata\"$\n\x14WaitExecutionRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\"\x8a\x02\n\x16GetActionResultRequest\x12\x15\n\rinstance_name\x18\x01 \x01(\t\x12>\n\raction_digest\x18\x02 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x15\n\rinline_stdout\x18\x03 \x01(\x08\x12\x15\n\rinline_stderr\x18\x04 \x01(\x08\x12\x1b\n\x13inline_output_files\x18\x05 \x03(\t\x12N\n\x0f\x64igest_function\x18\x06 \x01(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.Value\"\xdb\x02\n\x19UpdateActionResultRequest\x12\x15\n\rinstance_name\x18\x01 \x01(\t\x12>\n\raction_digest\x18\x02 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x44\n\raction_result\x18\x03 \x01(\x0b\x32-.build.bazel.remote.execution.v2.ActionResult\x12Q\n\x14results_cache_policy\x18\x04 \x01(\x0b\x32\x33.build.bazel.remote.execution.v2.ResultsCachePolicy\x12N\n\x0f\x64igest_function\x18\x05 \x01(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.Value\"\xbf\x01\n\x17\x46indMissingBlobsRequest\x12\x15\n\rinstance_name\x18\x01 \x01(\t\x12=\n\x0c\x62lob_digests\x18\x02 \x03(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12N\n\x0f\x64igest_function\x18\x03 \x01(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.Value\"a\n\x18\x46indMissingBlobsResponse\x12\x45\n\x14missing_blob_digests\x18\x02 \x03(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\"\xee\x02\n\x17\x42\x61tchUpdateBlobsRequest\x12\x15\n\rinstance_name\x18\x01 \x01(\t\x12R\n\x08requests\x18\x02 \x03(\x0b\x32@.build.bazel.remote.execution.v2.BatchUpdateBlobsRequest.Request\x12N\n\x0f\x64igest_function\x18\x05 \x01(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.Value\x1a\x97\x01\n\x07Request\x12\x37\n\x06\x64igest\x18\x01 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x0c\n\x04\x64\x61ta\x18\x02 \x01(\x0c\x12\x45\n\ncompressor\x18\x03 \x01(\x0e\x32\x31.build.bazel.remote.execution.v2.Compressor.Value\"\xda\x01\n\x18\x42\x61tchUpdateBlobsResponse\x12U\n\tresponses\x18\x01 \x03(\x0b\x32\x42.build.bazel.remote.execution.v2.BatchUpdateBlobsResponse.Response\x1ag\n\x08Response\x12\x37\n\x06\x64igest\x18\x01 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\"\n\x06status\x18\x02 \x01(\x0b\x32\x12.google.rpc.Status\"\x8b\x02\n\x15\x42\x61tchReadBlobsRequest\x12\x15\n\rinstance_name\x18\x01 \x01(\t\x12\x38\n\x07\x64igests\x18\x02 \x03(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12Q\n\x16\x61\x63\x63\x65ptable_compressors\x18\x03 \x03(\x0e\x32\x31.build.bazel.remote.execution.v2.Compressor.Value\x12N\n\x0f\x64igest_function\x18\x04 \x01(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.Value\"\xac\x02\n\x16\x42\x61tchReadBlobsResponse\x12S\n\tresponses\x18\x01 \x03(\x0b\x32@.build.bazel.remote.execution.v2.BatchReadBlobsResponse.Response\x1a\xbc\x01\n\x08Response\x12\x37\n\x06\x64igest\x18\x01 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x0c\n\x04\x64\x61ta\x18\x02 \x01(\x0c\x12\x45\n\ncompressor\x18\x04 \x01(\x0e\x32\x31.build.bazel.remote.execution.v2.Compressor.Value\x12\"\n\x06status\x18\x03 \x01(\x0b\x32\x12.google.rpc.Status\"\xdc\x01\n\x0eGetTreeRequest\x12\x15\n\rinstance_name\x18\x01 \x01(\t\x12<\n\x0broot_digest\x18\x02 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x11\n\tpage_size\x18\x03 \x01(\x05\x12\x12\n\npage_token\x18\x04 \x01(\t\x12N\n\x0f\x64igest_function\x18\x05 \x01(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.Value\"k\n\x0fGetTreeResponse\x12?\n\x0b\x64irectories\x18\x01 \x03(\x0b\x32*.build.bazel.remote.execution.v2.Directory\x12\x17\n\x0fnext_page_token\x18\x02 \x01(\t\"/\n\x16GetCapabilitiesRequest\x12\x15\n\rinstance_name\x18\x01 \x01(\t\"\xe3\x02\n\x12ServerCapabilities\x12N\n\x12\x63\x61\x63he_capabilities\x18\x01 \x01(\x0b\x32\x32.build.bazel.remote.execution.v2.CacheCapabilities\x12V\n\x16\x65xecution_capabilities\x18\x02 \x01(\x0b\x32\x36.build.bazel.remote.execution.v2.ExecutionCapabilities\x12:\n\x16\x64\x65precated_api_version\x18\x03 \x01(\x0b\x32\x1a.build.bazel.semver.SemVer\x12\x33\n\x0flow_api_version\x18\x04 \x01(\x0b\x32\x1a.build.bazel.semver.SemVer\x12\x34\n\x10high_api_version\x18\x05 \x01(\x0b\x32\x1a.build.bazel.semver.SemVer\"\x8f\x01\n\x0e\x44igestFunction\"}\n\x05Value\x12\x0b\n\x07UNKNOWN\x10\x00\x12\n\n\x06SHA256\x10\x01\x12\x08\n\x04SHA1\x10\x02\x12\x07\n\x03MD5\x10\x03\x12\x07\n\x03VSO\x10\x04\x12\n\n\x06SHA384\x10\x05\x12\n\n\x06SHA512\x10\x06\x12\x0b\n\x07MURMUR3\x10\x07\x12\x0e\n\nSHA256TREE\x10\x08\x12\n\n\x06\x42LAKE3\x10\t\"7\n\x1d\x41\x63tionCacheUpdateCapabilities\x12\x16\n\x0eupdate_enabled\x18\x01 \x01(\x08\"\xac\x01\n\x14PriorityCapabilities\x12W\n\npriorities\x18\x01 \x03(\x0b\x32\x43.build.bazel.remote.execution.v2.PriorityCapabilities.PriorityRange\x1a;\n\rPriorityRange\x12\x14\n\x0cmin_priority\x18\x01 \x01(\x05\x12\x14\n\x0cmax_priority\x18\x02 \x01(\x05\"P\n\x1bSymlinkAbsolutePathStrategy\"1\n\x05Value\x12\x0b\n\x07UNKNOWN\x10\x00\x12\x0e\n\nDISALLOWED\x10\x01\x12\x0b\n\x07\x41LLOWED\x10\x02\"F\n\nCompressor\"8\n\x05Value\x12\x0c\n\x08IDENTITY\x10\x00\x12\x08\n\x04ZSTD\x10\x01\x12\x0b\n\x07\x44\x45\x46LATE\x10\x02\x12\n\n\x06\x42ROTLI\x10\x03\"\xeb\x04\n\x11\x43\x61\x63heCapabilities\x12O\n\x10\x64igest_functions\x18\x01 \x03(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.Value\x12h\n action_cache_update_capabilities\x18\x02 \x01(\x0b\x32>.build.bazel.remote.execution.v2.ActionCacheUpdateCapabilities\x12Z\n\x1b\x63\x61\x63he_priority_capabilities\x18\x03 \x01(\x0b\x32\x35.build.bazel.remote.execution.v2.PriorityCapabilities\x12\"\n\x1amax_batch_total_size_bytes\x18\x04 \x01(\x03\x12j\n\x1esymlink_absolute_path_strategy\x18\x05 \x01(\x0e\x32\x42.build.bazel.remote.execution.v2.SymlinkAbsolutePathStrategy.Value\x12P\n\x15supported_compressors\x18\x06 \x03(\x0e\x32\x31.build.bazel.remote.execution.v2.Compressor.Value\x12]\n\"supported_batch_update_compressors\x18\x07 \x03(\x0e\x32\x31.build.bazel.remote.execution.v2.Compressor.Value\"\xd1\x02\n\x15\x45xecutionCapabilities\x12N\n\x0f\x64igest_function\x18\x01 \x01(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.Value\x12\x14\n\x0c\x65xec_enabled\x18\x02 \x01(\x08\x12^\n\x1f\x65xecution_priority_capabilities\x18\x03 \x01(\x0b\x32\x35.build.bazel.remote.execution.v2.PriorityCapabilities\x12!\n\x19supported_node_properties\x18\x04 \x03(\t\x12O\n\x10\x64igest_functions\x18\x05 \x03(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.Value\"6\n\x0bToolDetails\x12\x11\n\ttool_name\x18\x01 \x01(\t\x12\x14\n\x0ctool_version\x18\x02 \x01(\t\"\xed\x01\n\x0fRequestMetadata\x12\x42\n\x0ctool_details\x18\x01 \x01(\x0b\x32,.build.bazel.remote.execution.v2.ToolDetails\x12\x11\n\taction_id\x18\x02 \x01(\t\x12\x1a\n\x12tool_invocation_id\x18\x03 \x01(\t\x12!\n\x19\x63orrelated_invocations_id\x18\x04 \x01(\t\x12\x17\n\x0f\x61\x63tion_mnemonic\x18\x05 \x01(\t\x12\x11\n\ttarget_id\x18\x06 \x01(\t\x12\x18\n\x10\x63onfiguration_id\x18\x07 \x01(\t2\xb9\x02\n\tExecution\x12\x8e\x01\n\x07\x45xecute\x12/.build.bazel.remote.execution.v2.ExecuteRequest\x1a\x1d.google.longrunning.Operation\"1\x82\xd3\xe4\x93\x02+\"&/v2/{instance_name=**}/actions:execute:\x01*0\x01\x12\x9a\x01\n\rWaitExecution\x12\x35.build.bazel.remote.execution.v2.WaitExecutionRequest\x1a\x1d.google.longrunning.Operation\"1\x82\xd3\xe4\x93\x02+\"&/v2/{name=operations/**}:waitExecution:\x01*0\x01\x32\xd6\x03\n\x0b\x41\x63tionCache\x12\xd7\x01\n\x0fGetActionResult\x12\x37.build.bazel.remote.execution.v2.GetActionResultRequest\x1a-.build.bazel.remote.execution.v2.ActionResult\"\\\x82\xd3\xe4\x93\x02V\x12T/v2/{instance_name=**}/actionResults/{action_digest.hash}/{action_digest.size_bytes}\x12\xec\x01\n\x12UpdateActionResult\x12:.build.bazel.remote.execution.v2.UpdateActionResultRequest\x1a-.build.bazel.remote.execution.v2.ActionResult\"k\x82\xd3\xe4\x93\x02\x65\x1aT/v2/{instance_name=**}/actionResults/{action_digest.hash}/{action_digest.size_bytes}:\raction_result2\x9b\x06\n\x19\x43ontentAddressableStorage\x12\xbc\x01\n\x10\x46indMissingBlobs\x12\x38.build.bazel.remote.execution.v2.FindMissingBlobsRequest\x1a\x39.build.bazel.remote.execution.v2.FindMissingBlobsResponse\"3\x82\xd3\xe4\x93\x02-\"(/v2/{instance_name=**}/blobs:findMissing:\x01*\x12\xbc\x01\n\x10\x42\x61tchUpdateBlobs\x12\x38.build.bazel.remote.execution.v2.BatchUpdateBlobsRequest\x1a\x39.build.bazel.remote.execution.v2.BatchUpdateBlobsResponse\"3\x82\xd3\xe4\x93\x02-\"(/v2/{instance_name=**}/blobs:batchUpdate:\x01*\x12\xb4\x01\n\x0e\x42\x61tchReadBlobs\x12\x36.build.bazel.remote.execution.v2.BatchReadBlobsRequest\x1a\x37.build.bazel.remote.execution.v2.BatchReadBlobsResponse\"1\x82\xd3\xe4\x93\x02+\"&/v2/{instance_name=**}/blobs:batchRead:\x01*\x12\xc8\x01\n\x07GetTree\x12/.build.bazel.remote.execution.v2.GetTreeRequest\x1a\x30.build.bazel.remote.execution.v2.GetTreeResponse\"X\x82\xd3\xe4\x93\x02R\x12P/v2/{instance_name=**}/blobs/{root_digest.hash}/{root_digest.size_bytes}:getTree0\x01\x32\xbd\x01\n\x0c\x43\x61pabilities\x12\xac\x01\n\x0fGetCapabilities\x12\x37.build.bazel.remote.execution.v2.GetCapabilitiesRequest\x1a\x33.build.bazel.remote.execution.v2.ServerCapabilities\"+\x82\xd3\xe4\x93\x02%\x12#/v2/{instance_name=**}/capabilitiesB\xb4\x01\n\x1f\x62uild.bazel.remote.execution.v2B\x14RemoteExecutionProtoP\x01ZQgithub.com/bazelbuild/remote-apis/build/bazel/remote/execution/v2;remoteexecution\xa2\x02\x03REX\xaa\x02\x1f\x42uild.Bazel.Remote.Execution.V2b\x06proto3') +DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n6build/bazel/remote/execution/v2/remote_execution.proto\x12\x1f\x62uild.bazel.remote.execution.v2\x1a\x1f\x62uild/bazel/semver/semver.proto\x1a\x1cgoogle/api/annotations.proto\x1a#google/longrunning/operations.proto\x1a\x19google/protobuf/any.proto\x1a\x1egoogle/protobuf/duration.proto\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1egoogle/protobuf/wrappers.proto\x1a\x17google/rpc/status.proto\"\xa6\x02\n\x06\x41\x63tion\x12?\n\x0e\x63ommand_digest\x18\x01 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x42\n\x11input_root_digest\x18\x02 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12*\n\x07timeout\x18\x06 \x01(\x0b\x32\x19.google.protobuf.Duration\x12\x14\n\x0c\x64o_not_cache\x18\x07 \x01(\x08\x12\x0c\n\x04salt\x18\t \x01(\x0c\x12;\n\x08platform\x18\n \x01(\x0b\x32).build.bazel.remote.execution.v2.PlatformJ\x04\x08\x03\x10\x06J\x04\x08\x08\x10\t\"\xae\x04\n\x07\x43ommand\x12\x11\n\targuments\x18\x01 \x03(\t\x12[\n\x15\x65nvironment_variables\x18\x02 \x03(\x0b\x32<.build.bazel.remote.execution.v2.Command.EnvironmentVariable\x12\x18\n\x0coutput_files\x18\x03 \x03(\tB\x02\x18\x01\x12\x1e\n\x12output_directories\x18\x04 \x03(\tB\x02\x18\x01\x12\x14\n\x0coutput_paths\x18\x07 \x03(\t\x12?\n\x08platform\x18\x05 \x01(\x0b\x32).build.bazel.remote.execution.v2.PlatformB\x02\x18\x01\x12\x19\n\x11working_directory\x18\x06 \x01(\t\x12\x1e\n\x16output_node_properties\x18\x08 \x03(\t\x12_\n\x17output_directory_format\x18\t \x01(\x0e\x32>.build.bazel.remote.execution.v2.Command.OutputDirectoryFormat\x1a\x32\n\x13\x45nvironmentVariable\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t\"R\n\x15OutputDirectoryFormat\x12\r\n\tTREE_ONLY\x10\x00\x12\x12\n\x0e\x44IRECTORY_ONLY\x10\x01\x12\x16\n\x12TREE_AND_DIRECTORY\x10\x02\"{\n\x08Platform\x12\x46\n\nproperties\x18\x01 \x03(\x0b\x32\x32.build.bazel.remote.execution.v2.Platform.Property\x1a\'\n\x08Property\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t\"\x9a\x02\n\tDirectory\x12\x38\n\x05\x66iles\x18\x01 \x03(\x0b\x32).build.bazel.remote.execution.v2.FileNode\x12\x43\n\x0b\x64irectories\x18\x02 \x03(\x0b\x32..build.bazel.remote.execution.v2.DirectoryNode\x12>\n\x08symlinks\x18\x03 \x03(\x0b\x32,.build.bazel.remote.execution.v2.SymlinkNode\x12H\n\x0fnode_properties\x18\x05 \x01(\x0b\x32/.build.bazel.remote.execution.v2.NodePropertiesJ\x04\x08\x04\x10\x05\"+\n\x0cNodeProperty\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t\"\xaf\x01\n\x0eNodeProperties\x12\x41\n\nproperties\x18\x01 \x03(\x0b\x32-.build.bazel.remote.execution.v2.NodeProperty\x12)\n\x05mtime\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12/\n\tunix_mode\x18\x03 \x01(\x0b\x32\x1c.google.protobuf.UInt32Value\"\xbe\x01\n\x08\x46ileNode\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x37\n\x06\x64igest\x18\x02 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x15\n\ris_executable\x18\x04 \x01(\x08\x12H\n\x0fnode_properties\x18\x06 \x01(\x0b\x32/.build.bazel.remote.execution.v2.NodePropertiesJ\x04\x08\x03\x10\x04J\x04\x08\x05\x10\x06\"V\n\rDirectoryNode\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x37\n\x06\x64igest\x18\x02 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\"{\n\x0bSymlinkNode\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0e\n\x06target\x18\x02 \x01(\t\x12H\n\x0fnode_properties\x18\x04 \x01(\x0b\x32/.build.bazel.remote.execution.v2.NodePropertiesJ\x04\x08\x03\x10\x04\"*\n\x06\x44igest\x12\x0c\n\x04hash\x18\x01 \x01(\t\x12\x12\n\nsize_bytes\x18\x02 \x01(\x03\"\xdd\x05\n\x16\x45xecutedActionMetadata\x12\x0e\n\x06worker\x18\x01 \x01(\t\x12\x34\n\x10queued_timestamp\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12:\n\x16worker_start_timestamp\x18\x03 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12>\n\x1aworker_completed_timestamp\x18\x04 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12?\n\x1binput_fetch_start_timestamp\x18\x05 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12\x43\n\x1finput_fetch_completed_timestamp\x18\x06 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12=\n\x19\x65xecution_start_timestamp\x18\x07 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12\x41\n\x1d\x65xecution_completed_timestamp\x18\x08 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12=\n\x1avirtual_execution_duration\x18\x0c \x01(\x0b\x32\x19.google.protobuf.Duration\x12\x41\n\x1doutput_upload_start_timestamp\x18\t \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12\x45\n!output_upload_completed_timestamp\x18\n \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12\x30\n\x12\x61uxiliary_metadata\x18\x0b \x03(\x0b\x32\x14.google.protobuf.Any\"\xe4\x05\n\x0c\x41\x63tionResult\x12\x41\n\x0coutput_files\x18\x02 \x03(\x0b\x32+.build.bazel.remote.execution.v2.OutputFile\x12P\n\x14output_file_symlinks\x18\n \x03(\x0b\x32..build.bazel.remote.execution.v2.OutputSymlinkB\x02\x18\x01\x12G\n\x0foutput_symlinks\x18\x0c \x03(\x0b\x32..build.bazel.remote.execution.v2.OutputSymlink\x12L\n\x12output_directories\x18\x03 \x03(\x0b\x32\x30.build.bazel.remote.execution.v2.OutputDirectory\x12U\n\x19output_directory_symlinks\x18\x0b \x03(\x0b\x32..build.bazel.remote.execution.v2.OutputSymlinkB\x02\x18\x01\x12\x11\n\texit_code\x18\x04 \x01(\x05\x12\x12\n\nstdout_raw\x18\x05 \x01(\x0c\x12>\n\rstdout_digest\x18\x06 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x12\n\nstderr_raw\x18\x07 \x01(\x0c\x12>\n\rstderr_digest\x18\x08 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12S\n\x12\x65xecution_metadata\x18\t \x01(\x0b\x32\x37.build.bazel.remote.execution.v2.ExecutedActionMetadata\x12;\n\nsubactions\x18\x63 \x03(\x0b\x32\'.build.bazel.remote.execution.v2.DigestJ\x04\x08\x01\x10\x02\"\xd2\x01\n\nOutputFile\x12\x0c\n\x04path\x18\x01 \x01(\t\x12\x37\n\x06\x64igest\x18\x02 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x15\n\ris_executable\x18\x04 \x01(\x08\x12\x10\n\x08\x63ontents\x18\x05 \x01(\x0c\x12H\n\x0fnode_properties\x18\x07 \x01(\x0b\x32/.build.bazel.remote.execution.v2.NodePropertiesJ\x04\x08\x03\x10\x04J\x04\x08\x06\x10\x07\"~\n\x04Tree\x12\x38\n\x04root\x18\x01 \x01(\x0b\x32*.build.bazel.remote.execution.v2.Directory\x12<\n\x08\x63hildren\x18\x02 \x03(\x0b\x32*.build.bazel.remote.execution.v2.Directory\"\xcc\x01\n\x0fOutputDirectory\x12\x0c\n\x04path\x18\x01 \x01(\t\x12<\n\x0btree_digest\x18\x03 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x1f\n\x17is_topologically_sorted\x18\x04 \x01(\x08\x12\x46\n\x15root_directory_digest\x18\x05 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.DigestJ\x04\x08\x02\x10\x03\"}\n\rOutputSymlink\x12\x0c\n\x04path\x18\x01 \x01(\t\x12\x0e\n\x06target\x18\x02 \x01(\t\x12H\n\x0fnode_properties\x18\x04 \x01(\x0b\x32/.build.bazel.remote.execution.v2.NodePropertiesJ\x04\x08\x03\x10\x04\"#\n\x0f\x45xecutionPolicy\x12\x10\n\x08priority\x18\x01 \x01(\x05\"&\n\x12ResultsCachePolicy\x12\x10\n\x08priority\x18\x01 \x01(\x05\"\x83\x03\n\x0e\x45xecuteRequest\x12\x15\n\rinstance_name\x18\x01 \x01(\t\x12\x19\n\x11skip_cache_lookup\x18\x03 \x01(\x08\x12>\n\raction_digest\x18\x06 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12J\n\x10\x65xecution_policy\x18\x07 \x01(\x0b\x32\x30.build.bazel.remote.execution.v2.ExecutionPolicy\x12Q\n\x14results_cache_policy\x18\x08 \x01(\x0b\x32\x33.build.bazel.remote.execution.v2.ResultsCachePolicy\x12N\n\x0f\x64igest_function\x18\t \x01(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.ValueJ\x04\x08\x02\x10\x03J\x04\x08\x04\x10\x05J\x04\x08\x05\x10\x06\"Z\n\x07LogFile\x12\x37\n\x06\x64igest\x18\x01 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x16\n\x0ehuman_readable\x18\x02 \x01(\x08\"\xd0\x02\n\x0f\x45xecuteResponse\x12=\n\x06result\x18\x01 \x01(\x0b\x32-.build.bazel.remote.execution.v2.ActionResult\x12\x15\n\rcached_result\x18\x02 \x01(\x08\x12\"\n\x06status\x18\x03 \x01(\x0b\x32\x12.google.rpc.Status\x12U\n\x0bserver_logs\x18\x04 \x03(\x0b\x32@.build.bazel.remote.execution.v2.ExecuteResponse.ServerLogsEntry\x12\x0f\n\x07message\x18\x05 \x01(\t\x1a[\n\x0fServerLogsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\x37\n\x05value\x18\x02 \x01(\x0b\x32(.build.bazel.remote.execution.v2.LogFile:\x02\x38\x01\"a\n\x0e\x45xecutionStage\"O\n\x05Value\x12\x0b\n\x07UNKNOWN\x10\x00\x12\x0f\n\x0b\x43\x41\x43HE_CHECK\x10\x01\x12\n\n\x06QUEUED\x10\x02\x12\r\n\tEXECUTING\x10\x03\x12\r\n\tCOMPLETED\x10\x04\"\xb5\x02\n\x18\x45xecuteOperationMetadata\x12\x44\n\x05stage\x18\x01 \x01(\x0e\x32\x35.build.bazel.remote.execution.v2.ExecutionStage.Value\x12>\n\raction_digest\x18\x02 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x1a\n\x12stdout_stream_name\x18\x03 \x01(\t\x12\x1a\n\x12stderr_stream_name\x18\x04 \x01(\t\x12[\n\x1apartial_execution_metadata\x18\x05 \x01(\x0b\x32\x37.build.bazel.remote.execution.v2.ExecutedActionMetadata\"$\n\x14WaitExecutionRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\"\x8a\x02\n\x16GetActionResultRequest\x12\x15\n\rinstance_name\x18\x01 \x01(\t\x12>\n\raction_digest\x18\x02 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x15\n\rinline_stdout\x18\x03 \x01(\x08\x12\x15\n\rinline_stderr\x18\x04 \x01(\x08\x12\x1b\n\x13inline_output_files\x18\x05 \x03(\t\x12N\n\x0f\x64igest_function\x18\x06 \x01(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.Value\"\xdb\x02\n\x19UpdateActionResultRequest\x12\x15\n\rinstance_name\x18\x01 \x01(\t\x12>\n\raction_digest\x18\x02 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x44\n\raction_result\x18\x03 \x01(\x0b\x32-.build.bazel.remote.execution.v2.ActionResult\x12Q\n\x14results_cache_policy\x18\x04 \x01(\x0b\x32\x33.build.bazel.remote.execution.v2.ResultsCachePolicy\x12N\n\x0f\x64igest_function\x18\x05 \x01(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.Value\"\xbf\x01\n\x17\x46indMissingBlobsRequest\x12\x15\n\rinstance_name\x18\x01 \x01(\t\x12=\n\x0c\x62lob_digests\x18\x02 \x03(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12N\n\x0f\x64igest_function\x18\x03 \x01(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.Value\"a\n\x18\x46indMissingBlobsResponse\x12\x45\n\x14missing_blob_digests\x18\x02 \x03(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\"\xee\x02\n\x17\x42\x61tchUpdateBlobsRequest\x12\x15\n\rinstance_name\x18\x01 \x01(\t\x12R\n\x08requests\x18\x02 \x03(\x0b\x32@.build.bazel.remote.execution.v2.BatchUpdateBlobsRequest.Request\x12N\n\x0f\x64igest_function\x18\x05 \x01(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.Value\x1a\x97\x01\n\x07Request\x12\x37\n\x06\x64igest\x18\x01 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x0c\n\x04\x64\x61ta\x18\x02 \x01(\x0c\x12\x45\n\ncompressor\x18\x03 \x01(\x0e\x32\x31.build.bazel.remote.execution.v2.Compressor.Value\"\xda\x01\n\x18\x42\x61tchUpdateBlobsResponse\x12U\n\tresponses\x18\x01 \x03(\x0b\x32\x42.build.bazel.remote.execution.v2.BatchUpdateBlobsResponse.Response\x1ag\n\x08Response\x12\x37\n\x06\x64igest\x18\x01 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\"\n\x06status\x18\x02 \x01(\x0b\x32\x12.google.rpc.Status\"\x8b\x02\n\x15\x42\x61tchReadBlobsRequest\x12\x15\n\rinstance_name\x18\x01 \x01(\t\x12\x38\n\x07\x64igests\x18\x02 \x03(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12Q\n\x16\x61\x63\x63\x65ptable_compressors\x18\x03 \x03(\x0e\x32\x31.build.bazel.remote.execution.v2.Compressor.Value\x12N\n\x0f\x64igest_function\x18\x04 \x01(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.Value\"\xac\x02\n\x16\x42\x61tchReadBlobsResponse\x12S\n\tresponses\x18\x01 \x03(\x0b\x32@.build.bazel.remote.execution.v2.BatchReadBlobsResponse.Response\x1a\xbc\x01\n\x08Response\x12\x37\n\x06\x64igest\x18\x01 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x0c\n\x04\x64\x61ta\x18\x02 \x01(\x0c\x12\x45\n\ncompressor\x18\x04 \x01(\x0e\x32\x31.build.bazel.remote.execution.v2.Compressor.Value\x12\"\n\x06status\x18\x03 \x01(\x0b\x32\x12.google.rpc.Status\"\xdc\x01\n\x0eGetTreeRequest\x12\x15\n\rinstance_name\x18\x01 \x01(\t\x12<\n\x0broot_digest\x18\x02 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x11\n\tpage_size\x18\x03 \x01(\x05\x12\x12\n\npage_token\x18\x04 \x01(\t\x12N\n\x0f\x64igest_function\x18\x05 \x01(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.Value\"k\n\x0fGetTreeResponse\x12?\n\x0b\x64irectories\x18\x01 \x03(\x0b\x32*.build.bazel.remote.execution.v2.Directory\x12\x17\n\x0fnext_page_token\x18\x02 \x01(\t\"/\n\x16GetCapabilitiesRequest\x12\x15\n\rinstance_name\x18\x01 \x01(\t\"\xe3\x02\n\x12ServerCapabilities\x12N\n\x12\x63\x61\x63he_capabilities\x18\x01 \x01(\x0b\x32\x32.build.bazel.remote.execution.v2.CacheCapabilities\x12V\n\x16\x65xecution_capabilities\x18\x02 \x01(\x0b\x32\x36.build.bazel.remote.execution.v2.ExecutionCapabilities\x12:\n\x16\x64\x65precated_api_version\x18\x03 \x01(\x0b\x32\x1a.build.bazel.semver.SemVer\x12\x33\n\x0flow_api_version\x18\x04 \x01(\x0b\x32\x1a.build.bazel.semver.SemVer\x12\x34\n\x10high_api_version\x18\x05 \x01(\x0b\x32\x1a.build.bazel.semver.SemVer\"\x8f\x01\n\x0e\x44igestFunction\"}\n\x05Value\x12\x0b\n\x07UNKNOWN\x10\x00\x12\n\n\x06SHA256\x10\x01\x12\x08\n\x04SHA1\x10\x02\x12\x07\n\x03MD5\x10\x03\x12\x07\n\x03VSO\x10\x04\x12\n\n\x06SHA384\x10\x05\x12\n\n\x06SHA512\x10\x06\x12\x0b\n\x07MURMUR3\x10\x07\x12\x0e\n\nSHA256TREE\x10\x08\x12\n\n\x06\x42LAKE3\x10\t\"7\n\x1d\x41\x63tionCacheUpdateCapabilities\x12\x16\n\x0eupdate_enabled\x18\x01 \x01(\x08\"\xac\x01\n\x14PriorityCapabilities\x12W\n\npriorities\x18\x01 \x03(\x0b\x32\x43.build.bazel.remote.execution.v2.PriorityCapabilities.PriorityRange\x1a;\n\rPriorityRange\x12\x14\n\x0cmin_priority\x18\x01 \x01(\x05\x12\x14\n\x0cmax_priority\x18\x02 \x01(\x05\"P\n\x1bSymlinkAbsolutePathStrategy\"1\n\x05Value\x12\x0b\n\x07UNKNOWN\x10\x00\x12\x0e\n\nDISALLOWED\x10\x01\x12\x0b\n\x07\x41LLOWED\x10\x02\"F\n\nCompressor\"8\n\x05Value\x12\x0c\n\x08IDENTITY\x10\x00\x12\x08\n\x04ZSTD\x10\x01\x12\x0b\n\x07\x44\x45\x46LATE\x10\x02\x12\n\n\x06\x42ROTLI\x10\x03\"\xeb\x04\n\x11\x43\x61\x63heCapabilities\x12O\n\x10\x64igest_functions\x18\x01 \x03(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.Value\x12h\n action_cache_update_capabilities\x18\x02 \x01(\x0b\x32>.build.bazel.remote.execution.v2.ActionCacheUpdateCapabilities\x12Z\n\x1b\x63\x61\x63he_priority_capabilities\x18\x03 \x01(\x0b\x32\x35.build.bazel.remote.execution.v2.PriorityCapabilities\x12\"\n\x1amax_batch_total_size_bytes\x18\x04 \x01(\x03\x12j\n\x1esymlink_absolute_path_strategy\x18\x05 \x01(\x0e\x32\x42.build.bazel.remote.execution.v2.SymlinkAbsolutePathStrategy.Value\x12P\n\x15supported_compressors\x18\x06 \x03(\x0e\x32\x31.build.bazel.remote.execution.v2.Compressor.Value\x12]\n\"supported_batch_update_compressors\x18\x07 \x03(\x0e\x32\x31.build.bazel.remote.execution.v2.Compressor.Value\"\xd1\x02\n\x15\x45xecutionCapabilities\x12N\n\x0f\x64igest_function\x18\x01 \x01(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.Value\x12\x14\n\x0c\x65xec_enabled\x18\x02 \x01(\x08\x12^\n\x1f\x65xecution_priority_capabilities\x18\x03 \x01(\x0b\x32\x35.build.bazel.remote.execution.v2.PriorityCapabilities\x12!\n\x19supported_node_properties\x18\x04 \x03(\t\x12O\n\x10\x64igest_functions\x18\x05 \x03(\x0e\x32\x35.build.bazel.remote.execution.v2.DigestFunction.Value\"6\n\x0bToolDetails\x12\x11\n\ttool_name\x18\x01 \x01(\t\x12\x14\n\x0ctool_version\x18\x02 \x01(\t\"\xed\x01\n\x0fRequestMetadata\x12\x42\n\x0ctool_details\x18\x01 \x01(\x0b\x32,.build.bazel.remote.execution.v2.ToolDetails\x12\x11\n\taction_id\x18\x02 \x01(\t\x12\x1a\n\x12tool_invocation_id\x18\x03 \x01(\t\x12!\n\x19\x63orrelated_invocations_id\x18\x04 \x01(\t\x12\x17\n\x0f\x61\x63tion_mnemonic\x18\x05 \x01(\t\x12\x11\n\ttarget_id\x18\x06 \x01(\t\x12\x18\n\x10\x63onfiguration_id\x18\x07 \x01(\t2\xb9\x02\n\tExecution\x12\x8e\x01\n\x07\x45xecute\x12/.build.bazel.remote.execution.v2.ExecuteRequest\x1a\x1d.google.longrunning.Operation\"1\x82\xd3\xe4\x93\x02+\"&/v2/{instance_name=**}/actions:execute:\x01*0\x01\x12\x9a\x01\n\rWaitExecution\x12\x35.build.bazel.remote.execution.v2.WaitExecutionRequest\x1a\x1d.google.longrunning.Operation\"1\x82\xd3\xe4\x93\x02+\"&/v2/{name=operations/**}:waitExecution:\x01*0\x01\x32\xd6\x03\n\x0b\x41\x63tionCache\x12\xd7\x01\n\x0fGetActionResult\x12\x37.build.bazel.remote.execution.v2.GetActionResultRequest\x1a-.build.bazel.remote.execution.v2.ActionResult\"\\\x82\xd3\xe4\x93\x02V\x12T/v2/{instance_name=**}/actionResults/{action_digest.hash}/{action_digest.size_bytes}\x12\xec\x01\n\x12UpdateActionResult\x12:.build.bazel.remote.execution.v2.UpdateActionResultRequest\x1a-.build.bazel.remote.execution.v2.ActionResult\"k\x82\xd3\xe4\x93\x02\x65\x1aT/v2/{instance_name=**}/actionResults/{action_digest.hash}/{action_digest.size_bytes}:\raction_result2\x9b\x06\n\x19\x43ontentAddressableStorage\x12\xbc\x01\n\x10\x46indMissingBlobs\x12\x38.build.bazel.remote.execution.v2.FindMissingBlobsRequest\x1a\x39.build.bazel.remote.execution.v2.FindMissingBlobsResponse\"3\x82\xd3\xe4\x93\x02-\"(/v2/{instance_name=**}/blobs:findMissing:\x01*\x12\xbc\x01\n\x10\x42\x61tchUpdateBlobs\x12\x38.build.bazel.remote.execution.v2.BatchUpdateBlobsRequest\x1a\x39.build.bazel.remote.execution.v2.BatchUpdateBlobsResponse\"3\x82\xd3\xe4\x93\x02-\"(/v2/{instance_name=**}/blobs:batchUpdate:\x01*\x12\xb4\x01\n\x0e\x42\x61tchReadBlobs\x12\x36.build.bazel.remote.execution.v2.BatchReadBlobsRequest\x1a\x37.build.bazel.remote.execution.v2.BatchReadBlobsResponse\"1\x82\xd3\xe4\x93\x02+\"&/v2/{instance_name=**}/blobs:batchRead:\x01*\x12\xc8\x01\n\x07GetTree\x12/.build.bazel.remote.execution.v2.GetTreeRequest\x1a\x30.build.bazel.remote.execution.v2.GetTreeResponse\"X\x82\xd3\xe4\x93\x02R\x12P/v2/{instance_name=**}/blobs/{root_digest.hash}/{root_digest.size_bytes}:getTree0\x01\x32\xbd\x01\n\x0c\x43\x61pabilities\x12\xac\x01\n\x0fGetCapabilities\x12\x37.build.bazel.remote.execution.v2.GetCapabilitiesRequest\x1a\x33.build.bazel.remote.execution.v2.ServerCapabilities\"+\x82\xd3\xe4\x93\x02%\x12#/v2/{instance_name=**}/capabilitiesB\xb4\x01\n\x1f\x62uild.bazel.remote.execution.v2B\x14RemoteExecutionProtoP\x01ZQgithub.com/bazelbuild/remote-apis/build/bazel/remote/execution/v2;remoteexecution\xa2\x02\x03REX\xaa\x02\x1f\x42uild.Bazel.Remote.Execution.V2b\x06proto3') _globals = globals() _builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals) @@ -99,97 +99,97 @@ _globals['_EXECUTEDACTIONMETADATA']._serialized_start=2282 _globals['_EXECUTEDACTIONMETADATA']._serialized_end=3015 _globals['_ACTIONRESULT']._serialized_start=3018 - _globals['_ACTIONRESULT']._serialized_end=3697 - _globals['_OUTPUTFILE']._serialized_start=3700 - _globals['_OUTPUTFILE']._serialized_end=3910 - _globals['_TREE']._serialized_start=3912 - _globals['_TREE']._serialized_end=4038 - _globals['_OUTPUTDIRECTORY']._serialized_start=4041 - _globals['_OUTPUTDIRECTORY']._serialized_end=4245 - _globals['_OUTPUTSYMLINK']._serialized_start=4247 - _globals['_OUTPUTSYMLINK']._serialized_end=4372 - _globals['_EXECUTIONPOLICY']._serialized_start=4374 - _globals['_EXECUTIONPOLICY']._serialized_end=4409 - _globals['_RESULTSCACHEPOLICY']._serialized_start=4411 - _globals['_RESULTSCACHEPOLICY']._serialized_end=4449 - _globals['_EXECUTEREQUEST']._serialized_start=4452 - _globals['_EXECUTEREQUEST']._serialized_end=4914 - _globals['_LOGFILE']._serialized_start=4916 - _globals['_LOGFILE']._serialized_end=5006 - _globals['_EXECUTERESPONSE']._serialized_start=5009 - _globals['_EXECUTERESPONSE']._serialized_end=5345 - _globals['_EXECUTERESPONSE_SERVERLOGSENTRY']._serialized_start=5254 - _globals['_EXECUTERESPONSE_SERVERLOGSENTRY']._serialized_end=5345 - _globals['_EXECUTIONSTAGE']._serialized_start=5347 - _globals['_EXECUTIONSTAGE']._serialized_end=5444 - _globals['_EXECUTIONSTAGE_VALUE']._serialized_start=5365 - _globals['_EXECUTIONSTAGE_VALUE']._serialized_end=5444 - _globals['_EXECUTEOPERATIONMETADATA']._serialized_start=5447 - _globals['_EXECUTEOPERATIONMETADATA']._serialized_end=5756 - _globals['_WAITEXECUTIONREQUEST']._serialized_start=5758 - _globals['_WAITEXECUTIONREQUEST']._serialized_end=5794 - _globals['_GETACTIONRESULTREQUEST']._serialized_start=5797 - _globals['_GETACTIONRESULTREQUEST']._serialized_end=6063 - _globals['_UPDATEACTIONRESULTREQUEST']._serialized_start=6066 - _globals['_UPDATEACTIONRESULTREQUEST']._serialized_end=6413 - _globals['_FINDMISSINGBLOBSREQUEST']._serialized_start=6416 - _globals['_FINDMISSINGBLOBSREQUEST']._serialized_end=6607 - _globals['_FINDMISSINGBLOBSRESPONSE']._serialized_start=6609 - _globals['_FINDMISSINGBLOBSRESPONSE']._serialized_end=6706 - _globals['_BATCHUPDATEBLOBSREQUEST']._serialized_start=6709 - _globals['_BATCHUPDATEBLOBSREQUEST']._serialized_end=7075 - _globals['_BATCHUPDATEBLOBSREQUEST_REQUEST']._serialized_start=6924 - _globals['_BATCHUPDATEBLOBSREQUEST_REQUEST']._serialized_end=7075 - _globals['_BATCHUPDATEBLOBSRESPONSE']._serialized_start=7078 - _globals['_BATCHUPDATEBLOBSRESPONSE']._serialized_end=7296 - _globals['_BATCHUPDATEBLOBSRESPONSE_RESPONSE']._serialized_start=7193 - _globals['_BATCHUPDATEBLOBSRESPONSE_RESPONSE']._serialized_end=7296 - _globals['_BATCHREADBLOBSREQUEST']._serialized_start=7299 - _globals['_BATCHREADBLOBSREQUEST']._serialized_end=7566 - _globals['_BATCHREADBLOBSRESPONSE']._serialized_start=7569 - _globals['_BATCHREADBLOBSRESPONSE']._serialized_end=7869 - _globals['_BATCHREADBLOBSRESPONSE_RESPONSE']._serialized_start=7681 - _globals['_BATCHREADBLOBSRESPONSE_RESPONSE']._serialized_end=7869 - _globals['_GETTREEREQUEST']._serialized_start=7872 - _globals['_GETTREEREQUEST']._serialized_end=8092 - _globals['_GETTREERESPONSE']._serialized_start=8094 - _globals['_GETTREERESPONSE']._serialized_end=8201 - _globals['_GETCAPABILITIESREQUEST']._serialized_start=8203 - _globals['_GETCAPABILITIESREQUEST']._serialized_end=8250 - _globals['_SERVERCAPABILITIES']._serialized_start=8253 - _globals['_SERVERCAPABILITIES']._serialized_end=8608 - _globals['_DIGESTFUNCTION']._serialized_start=8611 - _globals['_DIGESTFUNCTION']._serialized_end=8754 - _globals['_DIGESTFUNCTION_VALUE']._serialized_start=8629 - _globals['_DIGESTFUNCTION_VALUE']._serialized_end=8754 - _globals['_ACTIONCACHEUPDATECAPABILITIES']._serialized_start=8756 - _globals['_ACTIONCACHEUPDATECAPABILITIES']._serialized_end=8811 - _globals['_PRIORITYCAPABILITIES']._serialized_start=8814 - _globals['_PRIORITYCAPABILITIES']._serialized_end=8986 - _globals['_PRIORITYCAPABILITIES_PRIORITYRANGE']._serialized_start=8927 - _globals['_PRIORITYCAPABILITIES_PRIORITYRANGE']._serialized_end=8986 - _globals['_SYMLINKABSOLUTEPATHSTRATEGY']._serialized_start=8988 - _globals['_SYMLINKABSOLUTEPATHSTRATEGY']._serialized_end=9068 - _globals['_SYMLINKABSOLUTEPATHSTRATEGY_VALUE']._serialized_start=9019 - _globals['_SYMLINKABSOLUTEPATHSTRATEGY_VALUE']._serialized_end=9068 - _globals['_COMPRESSOR']._serialized_start=9070 - _globals['_COMPRESSOR']._serialized_end=9140 - _globals['_COMPRESSOR_VALUE']._serialized_start=9084 - _globals['_COMPRESSOR_VALUE']._serialized_end=9140 - _globals['_CACHECAPABILITIES']._serialized_start=9143 - _globals['_CACHECAPABILITIES']._serialized_end=9762 - _globals['_EXECUTIONCAPABILITIES']._serialized_start=9765 - _globals['_EXECUTIONCAPABILITIES']._serialized_end=10102 - _globals['_TOOLDETAILS']._serialized_start=10104 - _globals['_TOOLDETAILS']._serialized_end=10158 - _globals['_REQUESTMETADATA']._serialized_start=10161 - _globals['_REQUESTMETADATA']._serialized_end=10398 - _globals['_EXECUTION']._serialized_start=10401 - _globals['_EXECUTION']._serialized_end=10714 - _globals['_ACTIONCACHE']._serialized_start=10717 - _globals['_ACTIONCACHE']._serialized_end=11187 - _globals['_CONTENTADDRESSABLESTORAGE']._serialized_start=11190 - _globals['_CONTENTADDRESSABLESTORAGE']._serialized_end=11985 - _globals['_CAPABILITIES']._serialized_start=11988 - _globals['_CAPABILITIES']._serialized_end=12177 + _globals['_ACTIONRESULT']._serialized_end=3758 + _globals['_OUTPUTFILE']._serialized_start=3761 + _globals['_OUTPUTFILE']._serialized_end=3971 + _globals['_TREE']._serialized_start=3973 + _globals['_TREE']._serialized_end=4099 + _globals['_OUTPUTDIRECTORY']._serialized_start=4102 + _globals['_OUTPUTDIRECTORY']._serialized_end=4306 + _globals['_OUTPUTSYMLINK']._serialized_start=4308 + _globals['_OUTPUTSYMLINK']._serialized_end=4433 + _globals['_EXECUTIONPOLICY']._serialized_start=4435 + _globals['_EXECUTIONPOLICY']._serialized_end=4470 + _globals['_RESULTSCACHEPOLICY']._serialized_start=4472 + _globals['_RESULTSCACHEPOLICY']._serialized_end=4510 + _globals['_EXECUTEREQUEST']._serialized_start=4513 + _globals['_EXECUTEREQUEST']._serialized_end=4900 + _globals['_LOGFILE']._serialized_start=4902 + _globals['_LOGFILE']._serialized_end=4992 + _globals['_EXECUTERESPONSE']._serialized_start=4995 + _globals['_EXECUTERESPONSE']._serialized_end=5331 + _globals['_EXECUTERESPONSE_SERVERLOGSENTRY']._serialized_start=5240 + _globals['_EXECUTERESPONSE_SERVERLOGSENTRY']._serialized_end=5331 + _globals['_EXECUTIONSTAGE']._serialized_start=5333 + _globals['_EXECUTIONSTAGE']._serialized_end=5430 + _globals['_EXECUTIONSTAGE_VALUE']._serialized_start=5351 + _globals['_EXECUTIONSTAGE_VALUE']._serialized_end=5430 + _globals['_EXECUTEOPERATIONMETADATA']._serialized_start=5433 + _globals['_EXECUTEOPERATIONMETADATA']._serialized_end=5742 + _globals['_WAITEXECUTIONREQUEST']._serialized_start=5744 + _globals['_WAITEXECUTIONREQUEST']._serialized_end=5780 + _globals['_GETACTIONRESULTREQUEST']._serialized_start=5783 + _globals['_GETACTIONRESULTREQUEST']._serialized_end=6049 + _globals['_UPDATEACTIONRESULTREQUEST']._serialized_start=6052 + _globals['_UPDATEACTIONRESULTREQUEST']._serialized_end=6399 + _globals['_FINDMISSINGBLOBSREQUEST']._serialized_start=6402 + _globals['_FINDMISSINGBLOBSREQUEST']._serialized_end=6593 + _globals['_FINDMISSINGBLOBSRESPONSE']._serialized_start=6595 + _globals['_FINDMISSINGBLOBSRESPONSE']._serialized_end=6692 + _globals['_BATCHUPDATEBLOBSREQUEST']._serialized_start=6695 + _globals['_BATCHUPDATEBLOBSREQUEST']._serialized_end=7061 + _globals['_BATCHUPDATEBLOBSREQUEST_REQUEST']._serialized_start=6910 + _globals['_BATCHUPDATEBLOBSREQUEST_REQUEST']._serialized_end=7061 + _globals['_BATCHUPDATEBLOBSRESPONSE']._serialized_start=7064 + _globals['_BATCHUPDATEBLOBSRESPONSE']._serialized_end=7282 + _globals['_BATCHUPDATEBLOBSRESPONSE_RESPONSE']._serialized_start=7179 + _globals['_BATCHUPDATEBLOBSRESPONSE_RESPONSE']._serialized_end=7282 + _globals['_BATCHREADBLOBSREQUEST']._serialized_start=7285 + _globals['_BATCHREADBLOBSREQUEST']._serialized_end=7552 + _globals['_BATCHREADBLOBSRESPONSE']._serialized_start=7555 + _globals['_BATCHREADBLOBSRESPONSE']._serialized_end=7855 + _globals['_BATCHREADBLOBSRESPONSE_RESPONSE']._serialized_start=7667 + _globals['_BATCHREADBLOBSRESPONSE_RESPONSE']._serialized_end=7855 + _globals['_GETTREEREQUEST']._serialized_start=7858 + _globals['_GETTREEREQUEST']._serialized_end=8078 + _globals['_GETTREERESPONSE']._serialized_start=8080 + _globals['_GETTREERESPONSE']._serialized_end=8187 + _globals['_GETCAPABILITIESREQUEST']._serialized_start=8189 + _globals['_GETCAPABILITIESREQUEST']._serialized_end=8236 + _globals['_SERVERCAPABILITIES']._serialized_start=8239 + _globals['_SERVERCAPABILITIES']._serialized_end=8594 + _globals['_DIGESTFUNCTION']._serialized_start=8597 + _globals['_DIGESTFUNCTION']._serialized_end=8740 + _globals['_DIGESTFUNCTION_VALUE']._serialized_start=8615 + _globals['_DIGESTFUNCTION_VALUE']._serialized_end=8740 + _globals['_ACTIONCACHEUPDATECAPABILITIES']._serialized_start=8742 + _globals['_ACTIONCACHEUPDATECAPABILITIES']._serialized_end=8797 + _globals['_PRIORITYCAPABILITIES']._serialized_start=8800 + _globals['_PRIORITYCAPABILITIES']._serialized_end=8972 + _globals['_PRIORITYCAPABILITIES_PRIORITYRANGE']._serialized_start=8913 + _globals['_PRIORITYCAPABILITIES_PRIORITYRANGE']._serialized_end=8972 + _globals['_SYMLINKABSOLUTEPATHSTRATEGY']._serialized_start=8974 + _globals['_SYMLINKABSOLUTEPATHSTRATEGY']._serialized_end=9054 + _globals['_SYMLINKABSOLUTEPATHSTRATEGY_VALUE']._serialized_start=9005 + _globals['_SYMLINKABSOLUTEPATHSTRATEGY_VALUE']._serialized_end=9054 + _globals['_COMPRESSOR']._serialized_start=9056 + _globals['_COMPRESSOR']._serialized_end=9126 + _globals['_COMPRESSOR_VALUE']._serialized_start=9070 + _globals['_COMPRESSOR_VALUE']._serialized_end=9126 + _globals['_CACHECAPABILITIES']._serialized_start=9129 + _globals['_CACHECAPABILITIES']._serialized_end=9748 + _globals['_EXECUTIONCAPABILITIES']._serialized_start=9751 + _globals['_EXECUTIONCAPABILITIES']._serialized_end=10088 + _globals['_TOOLDETAILS']._serialized_start=10090 + _globals['_TOOLDETAILS']._serialized_end=10144 + _globals['_REQUESTMETADATA']._serialized_start=10147 + _globals['_REQUESTMETADATA']._serialized_end=10384 + _globals['_EXECUTION']._serialized_start=10387 + _globals['_EXECUTION']._serialized_end=10700 + _globals['_ACTIONCACHE']._serialized_start=10703 + _globals['_ACTIONCACHE']._serialized_end=11173 + _globals['_CONTENTADDRESSABLESTORAGE']._serialized_start=11176 + _globals['_CONTENTADDRESSABLESTORAGE']._serialized_end=11971 + _globals['_CAPABILITIES']._serialized_start=11974 + _globals['_CAPABILITIES']._serialized_end=12163 # @@protoc_insertion_point(module_scope) diff --git a/src/buildstream/_protos/build/bazel/remote/execution/v2/remote_execution_pb2.pyi b/src/buildstream/_protos/build/bazel/remote/execution/v2/remote_execution_pb2.pyi index 14badbac9..99a57f913 100644 --- a/src/buildstream/_protos/build/bazel/remote/execution/v2/remote_execution_pb2.pyi +++ b/src/buildstream/_protos/build/bazel/remote/execution/v2/remote_execution_pb2.pyi @@ -177,7 +177,7 @@ class ExecutedActionMetadata(_message.Message): def __init__(self, worker: _Optional[str] = ..., queued_timestamp: _Optional[_Union[_timestamp_pb2.Timestamp, _Mapping]] = ..., worker_start_timestamp: _Optional[_Union[_timestamp_pb2.Timestamp, _Mapping]] = ..., worker_completed_timestamp: _Optional[_Union[_timestamp_pb2.Timestamp, _Mapping]] = ..., input_fetch_start_timestamp: _Optional[_Union[_timestamp_pb2.Timestamp, _Mapping]] = ..., input_fetch_completed_timestamp: _Optional[_Union[_timestamp_pb2.Timestamp, _Mapping]] = ..., execution_start_timestamp: _Optional[_Union[_timestamp_pb2.Timestamp, _Mapping]] = ..., execution_completed_timestamp: _Optional[_Union[_timestamp_pb2.Timestamp, _Mapping]] = ..., virtual_execution_duration: _Optional[_Union[_duration_pb2.Duration, _Mapping]] = ..., output_upload_start_timestamp: _Optional[_Union[_timestamp_pb2.Timestamp, _Mapping]] = ..., output_upload_completed_timestamp: _Optional[_Union[_timestamp_pb2.Timestamp, _Mapping]] = ..., auxiliary_metadata: _Optional[_Iterable[_Union[_any_pb2.Any, _Mapping]]] = ...) -> None: ... class ActionResult(_message.Message): - __slots__ = ("output_files", "output_file_symlinks", "output_symlinks", "output_directories", "output_directory_symlinks", "exit_code", "stdout_raw", "stdout_digest", "stderr_raw", "stderr_digest", "execution_metadata") + __slots__ = ("output_files", "output_file_symlinks", "output_symlinks", "output_directories", "output_directory_symlinks", "exit_code", "stdout_raw", "stdout_digest", "stderr_raw", "stderr_digest", "execution_metadata", "subactions") OUTPUT_FILES_FIELD_NUMBER: _ClassVar[int] OUTPUT_FILE_SYMLINKS_FIELD_NUMBER: _ClassVar[int] OUTPUT_SYMLINKS_FIELD_NUMBER: _ClassVar[int] @@ -189,6 +189,7 @@ class ActionResult(_message.Message): STDERR_RAW_FIELD_NUMBER: _ClassVar[int] STDERR_DIGEST_FIELD_NUMBER: _ClassVar[int] EXECUTION_METADATA_FIELD_NUMBER: _ClassVar[int] + SUBACTIONS_FIELD_NUMBER: _ClassVar[int] output_files: _containers.RepeatedCompositeFieldContainer[OutputFile] output_file_symlinks: _containers.RepeatedCompositeFieldContainer[OutputSymlink] output_symlinks: _containers.RepeatedCompositeFieldContainer[OutputSymlink] @@ -200,7 +201,8 @@ class ActionResult(_message.Message): stderr_raw: bytes stderr_digest: Digest execution_metadata: ExecutedActionMetadata - def __init__(self, output_files: _Optional[_Iterable[_Union[OutputFile, _Mapping]]] = ..., output_file_symlinks: _Optional[_Iterable[_Union[OutputSymlink, _Mapping]]] = ..., output_symlinks: _Optional[_Iterable[_Union[OutputSymlink, _Mapping]]] = ..., output_directories: _Optional[_Iterable[_Union[OutputDirectory, _Mapping]]] = ..., output_directory_symlinks: _Optional[_Iterable[_Union[OutputSymlink, _Mapping]]] = ..., exit_code: _Optional[int] = ..., stdout_raw: _Optional[bytes] = ..., stdout_digest: _Optional[_Union[Digest, _Mapping]] = ..., stderr_raw: _Optional[bytes] = ..., stderr_digest: _Optional[_Union[Digest, _Mapping]] = ..., execution_metadata: _Optional[_Union[ExecutedActionMetadata, _Mapping]] = ...) -> None: ... + subactions: _containers.RepeatedCompositeFieldContainer[Digest] + def __init__(self, output_files: _Optional[_Iterable[_Union[OutputFile, _Mapping]]] = ..., output_file_symlinks: _Optional[_Iterable[_Union[OutputSymlink, _Mapping]]] = ..., output_symlinks: _Optional[_Iterable[_Union[OutputSymlink, _Mapping]]] = ..., output_directories: _Optional[_Iterable[_Union[OutputDirectory, _Mapping]]] = ..., output_directory_symlinks: _Optional[_Iterable[_Union[OutputSymlink, _Mapping]]] = ..., exit_code: _Optional[int] = ..., stdout_raw: _Optional[bytes] = ..., stdout_digest: _Optional[_Union[Digest, _Mapping]] = ..., stderr_raw: _Optional[bytes] = ..., stderr_digest: _Optional[_Union[Digest, _Mapping]] = ..., execution_metadata: _Optional[_Union[ExecutedActionMetadata, _Mapping]] = ..., subactions: _Optional[_Iterable[_Union[Digest, _Mapping]]] = ...) -> None: ... class OutputFile(_message.Message): __slots__ = ("path", "digest", "is_executable", "contents", "node_properties") @@ -259,26 +261,20 @@ class ResultsCachePolicy(_message.Message): def __init__(self, priority: _Optional[int] = ...) -> None: ... class ExecuteRequest(_message.Message): - __slots__ = ("instance_name", "skip_cache_lookup", "action_digest", "execution_policy", "results_cache_policy", "digest_function", "inline_stdout", "inline_stderr", "inline_output_files") + __slots__ = ("instance_name", "skip_cache_lookup", "action_digest", "execution_policy", "results_cache_policy", "digest_function") INSTANCE_NAME_FIELD_NUMBER: _ClassVar[int] SKIP_CACHE_LOOKUP_FIELD_NUMBER: _ClassVar[int] ACTION_DIGEST_FIELD_NUMBER: _ClassVar[int] EXECUTION_POLICY_FIELD_NUMBER: _ClassVar[int] RESULTS_CACHE_POLICY_FIELD_NUMBER: _ClassVar[int] DIGEST_FUNCTION_FIELD_NUMBER: _ClassVar[int] - INLINE_STDOUT_FIELD_NUMBER: _ClassVar[int] - INLINE_STDERR_FIELD_NUMBER: _ClassVar[int] - INLINE_OUTPUT_FILES_FIELD_NUMBER: _ClassVar[int] instance_name: str skip_cache_lookup: bool action_digest: Digest execution_policy: ExecutionPolicy results_cache_policy: ResultsCachePolicy digest_function: DigestFunction.Value - inline_stdout: bool - inline_stderr: bool - inline_output_files: _containers.RepeatedScalarFieldContainer[str] - def __init__(self, instance_name: _Optional[str] = ..., skip_cache_lookup: bool = ..., action_digest: _Optional[_Union[Digest, _Mapping]] = ..., execution_policy: _Optional[_Union[ExecutionPolicy, _Mapping]] = ..., results_cache_policy: _Optional[_Union[ResultsCachePolicy, _Mapping]] = ..., digest_function: _Optional[_Union[DigestFunction.Value, str]] = ..., inline_stdout: bool = ..., inline_stderr: bool = ..., inline_output_files: _Optional[_Iterable[str]] = ...) -> None: ... + def __init__(self, instance_name: _Optional[str] = ..., skip_cache_lookup: bool = ..., action_digest: _Optional[_Union[Digest, _Mapping]] = ..., execution_policy: _Optional[_Union[ExecutionPolicy, _Mapping]] = ..., results_cache_policy: _Optional[_Union[ResultsCachePolicy, _Mapping]] = ..., digest_function: _Optional[_Union[DigestFunction.Value, str]] = ...) -> None: ... class LogFile(_message.Message): __slots__ = ("digest", "human_readable") diff --git a/src/buildstream/_protos/buildstream/v2/artifact.proto b/src/buildstream/_protos/buildstream/v2/artifact.proto index 28d006f0f..57628faa7 100644 --- a/src/buildstream/_protos/buildstream/v2/artifact.proto +++ b/src/buildstream/_protos/buildstream/v2/artifact.proto @@ -93,4 +93,7 @@ message Artifact { repeated string marked_directories = 4; }; SandboxState buildsandbox = 18; // optional + + // digest of a SpeculativeActions message (from speculative_actions.proto) + build.bazel.remote.execution.v2.Digest speculative_actions = 19; // optional } diff --git a/src/buildstream/_protos/buildstream/v2/artifact_pb2.py b/src/buildstream/_protos/buildstream/v2/artifact_pb2.py index a81006af5..757781a18 100644 --- a/src/buildstream/_protos/buildstream/v2/artifact_pb2.py +++ b/src/buildstream/_protos/buildstream/v2/artifact_pb2.py @@ -26,7 +26,7 @@ from buildstream._protos.google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2 -DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1d\x62uildstream/v2/artifact.proto\x12\x0e\x62uildstream.v2\x1a\x36\x62uild/bazel/remote/execution/v2/remote_execution.proto\x1a\x1cgoogle/api/annotations.proto\"\xa6\t\n\x08\x41rtifact\x12\x0f\n\x07version\x18\x01 \x01(\x05\x12\x15\n\rbuild_success\x18\x02 \x01(\x08\x12\x13\n\x0b\x62uild_error\x18\x03 \x01(\t\x12\x1b\n\x13\x62uild_error_details\x18\x04 \x01(\t\x12\x12\n\nstrong_key\x18\x05 \x01(\t\x12\x10\n\x08weak_key\x18\x06 \x01(\t\x12\x16\n\x0ewas_workspaced\x18\x07 \x01(\x08\x12\x36\n\x05\x66iles\x18\x08 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x37\n\nbuild_deps\x18\t \x03(\x0b\x32#.buildstream.v2.Artifact.Dependency\x12<\n\x0bpublic_data\x18\n \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12.\n\x04logs\x18\x0b \x03(\x0b\x32 .buildstream.v2.Artifact.LogFile\x12:\n\tbuildtree\x18\x0c \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x38\n\x07sources\x18\r \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x43\n\x12low_diversity_meta\x18\x0e \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x44\n\x13high_diversity_meta\x18\x0f \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x12\n\nstrict_key\x18\x10 \x01(\t\x12:\n\tbuildroot\x18\x11 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12;\n\x0c\x62uildsandbox\x18\x12 \x01(\x0b\x32%.buildstream.v2.Artifact.SandboxState\x1a\x63\n\nDependency\x12\x14\n\x0cproject_name\x18\x01 \x01(\t\x12\x14\n\x0c\x65lement_name\x18\x02 \x01(\t\x12\x11\n\tcache_key\x18\x03 \x01(\t\x12\x16\n\x0ewas_workspaced\x18\x04 \x01(\x08\x1aP\n\x07LogFile\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x37\n\x06\x64igest\x18\x02 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x1a\xdd\x01\n\x0cSandboxState\x12Q\n\x0b\x65nvironment\x18\x01 \x03(\x0b\x32<.build.bazel.remote.execution.v2.Command.EnvironmentVariable\x12\x19\n\x11working_directory\x18\x02 \x01(\t\x12\x43\n\x12subsandbox_digests\x18\x03 \x03(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x1a\n\x12marked_directories\x18\x04 \x03(\tb\x06proto3') +DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1d\x62uildstream/v2/artifact.proto\x12\x0e\x62uildstream.v2\x1a\x36\x62uild/bazel/remote/execution/v2/remote_execution.proto\x1a\x1cgoogle/api/annotations.proto\"\xec\t\n\x08\x41rtifact\x12\x0f\n\x07version\x18\x01 \x01(\x05\x12\x15\n\rbuild_success\x18\x02 \x01(\x08\x12\x13\n\x0b\x62uild_error\x18\x03 \x01(\t\x12\x1b\n\x13\x62uild_error_details\x18\x04 \x01(\t\x12\x12\n\nstrong_key\x18\x05 \x01(\t\x12\x10\n\x08weak_key\x18\x06 \x01(\t\x12\x16\n\x0ewas_workspaced\x18\x07 \x01(\x08\x12\x36\n\x05\x66iles\x18\x08 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x37\n\nbuild_deps\x18\t \x03(\x0b\x32#.buildstream.v2.Artifact.Dependency\x12<\n\x0bpublic_data\x18\n \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12.\n\x04logs\x18\x0b \x03(\x0b\x32 .buildstream.v2.Artifact.LogFile\x12:\n\tbuildtree\x18\x0c \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x38\n\x07sources\x18\r \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x43\n\x12low_diversity_meta\x18\x0e \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x44\n\x13high_diversity_meta\x18\x0f \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x12\n\nstrict_key\x18\x10 \x01(\t\x12:\n\tbuildroot\x18\x11 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12;\n\x0c\x62uildsandbox\x18\x12 \x01(\x0b\x32%.buildstream.v2.Artifact.SandboxState\x12\x44\n\x13speculative_actions\x18\x13 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x1a\x63\n\nDependency\x12\x14\n\x0cproject_name\x18\x01 \x01(\t\x12\x14\n\x0c\x65lement_name\x18\x02 \x01(\t\x12\x11\n\tcache_key\x18\x03 \x01(\t\x12\x16\n\x0ewas_workspaced\x18\x04 \x01(\x08\x1aP\n\x07LogFile\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x37\n\x06\x64igest\x18\x02 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x1a\xdd\x01\n\x0cSandboxState\x12Q\n\x0b\x65nvironment\x18\x01 \x03(\x0b\x32<.build.bazel.remote.execution.v2.Command.EnvironmentVariable\x12\x19\n\x11working_directory\x18\x02 \x01(\t\x12\x43\n\x12subsandbox_digests\x18\x03 \x03(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x1a\n\x12marked_directories\x18\x04 \x03(\tb\x06proto3') _globals = globals() _builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals) @@ -34,11 +34,11 @@ if not _descriptor._USE_C_DESCRIPTORS: DESCRIPTOR._loaded_options = None _globals['_ARTIFACT']._serialized_start=136 - _globals['_ARTIFACT']._serialized_end=1326 - _globals['_ARTIFACT_DEPENDENCY']._serialized_start=921 - _globals['_ARTIFACT_DEPENDENCY']._serialized_end=1020 - _globals['_ARTIFACT_LOGFILE']._serialized_start=1022 - _globals['_ARTIFACT_LOGFILE']._serialized_end=1102 - _globals['_ARTIFACT_SANDBOXSTATE']._serialized_start=1105 - _globals['_ARTIFACT_SANDBOXSTATE']._serialized_end=1326 + _globals['_ARTIFACT']._serialized_end=1396 + _globals['_ARTIFACT_DEPENDENCY']._serialized_start=991 + _globals['_ARTIFACT_DEPENDENCY']._serialized_end=1090 + _globals['_ARTIFACT_LOGFILE']._serialized_start=1092 + _globals['_ARTIFACT_LOGFILE']._serialized_end=1172 + _globals['_ARTIFACT_SANDBOXSTATE']._serialized_start=1175 + _globals['_ARTIFACT_SANDBOXSTATE']._serialized_end=1396 # @@protoc_insertion_point(module_scope) diff --git a/src/buildstream/_protos/buildstream/v2/artifact_pb2.pyi b/src/buildstream/_protos/buildstream/v2/artifact_pb2.pyi index 7eb4d7550..7681772b2 100644 --- a/src/buildstream/_protos/buildstream/v2/artifact_pb2.pyi +++ b/src/buildstream/_protos/buildstream/v2/artifact_pb2.pyi @@ -8,7 +8,7 @@ from typing import ClassVar as _ClassVar, Iterable as _Iterable, Mapping as _Map DESCRIPTOR: _descriptor.FileDescriptor class Artifact(_message.Message): - __slots__ = ("version", "build_success", "build_error", "build_error_details", "strong_key", "weak_key", "was_workspaced", "files", "build_deps", "public_data", "logs", "buildtree", "sources", "low_diversity_meta", "high_diversity_meta", "strict_key", "buildroot", "buildsandbox") + __slots__ = ("version", "build_success", "build_error", "build_error_details", "strong_key", "weak_key", "was_workspaced", "files", "build_deps", "public_data", "logs", "buildtree", "sources", "low_diversity_meta", "high_diversity_meta", "strict_key", "buildroot", "buildsandbox", "speculative_actions") class Dependency(_message.Message): __slots__ = ("project_name", "element_name", "cache_key", "was_workspaced") PROJECT_NAME_FIELD_NUMBER: _ClassVar[int] @@ -56,6 +56,7 @@ class Artifact(_message.Message): STRICT_KEY_FIELD_NUMBER: _ClassVar[int] BUILDROOT_FIELD_NUMBER: _ClassVar[int] BUILDSANDBOX_FIELD_NUMBER: _ClassVar[int] + SPECULATIVE_ACTIONS_FIELD_NUMBER: _ClassVar[int] version: int build_success: bool build_error: str @@ -74,4 +75,5 @@ class Artifact(_message.Message): strict_key: str buildroot: _remote_execution_pb2.Digest buildsandbox: Artifact.SandboxState - def __init__(self, version: _Optional[int] = ..., build_success: bool = ..., build_error: _Optional[str] = ..., build_error_details: _Optional[str] = ..., strong_key: _Optional[str] = ..., weak_key: _Optional[str] = ..., was_workspaced: bool = ..., files: _Optional[_Union[_remote_execution_pb2.Digest, _Mapping]] = ..., build_deps: _Optional[_Iterable[_Union[Artifact.Dependency, _Mapping]]] = ..., public_data: _Optional[_Union[_remote_execution_pb2.Digest, _Mapping]] = ..., logs: _Optional[_Iterable[_Union[Artifact.LogFile, _Mapping]]] = ..., buildtree: _Optional[_Union[_remote_execution_pb2.Digest, _Mapping]] = ..., sources: _Optional[_Union[_remote_execution_pb2.Digest, _Mapping]] = ..., low_diversity_meta: _Optional[_Union[_remote_execution_pb2.Digest, _Mapping]] = ..., high_diversity_meta: _Optional[_Union[_remote_execution_pb2.Digest, _Mapping]] = ..., strict_key: _Optional[str] = ..., buildroot: _Optional[_Union[_remote_execution_pb2.Digest, _Mapping]] = ..., buildsandbox: _Optional[_Union[Artifact.SandboxState, _Mapping]] = ...) -> None: ... + speculative_actions: _remote_execution_pb2.Digest + def __init__(self, version: _Optional[int] = ..., build_success: bool = ..., build_error: _Optional[str] = ..., build_error_details: _Optional[str] = ..., strong_key: _Optional[str] = ..., weak_key: _Optional[str] = ..., was_workspaced: bool = ..., files: _Optional[_Union[_remote_execution_pb2.Digest, _Mapping]] = ..., build_deps: _Optional[_Iterable[_Union[Artifact.Dependency, _Mapping]]] = ..., public_data: _Optional[_Union[_remote_execution_pb2.Digest, _Mapping]] = ..., logs: _Optional[_Iterable[_Union[Artifact.LogFile, _Mapping]]] = ..., buildtree: _Optional[_Union[_remote_execution_pb2.Digest, _Mapping]] = ..., sources: _Optional[_Union[_remote_execution_pb2.Digest, _Mapping]] = ..., low_diversity_meta: _Optional[_Union[_remote_execution_pb2.Digest, _Mapping]] = ..., high_diversity_meta: _Optional[_Union[_remote_execution_pb2.Digest, _Mapping]] = ..., strict_key: _Optional[str] = ..., buildroot: _Optional[_Union[_remote_execution_pb2.Digest, _Mapping]] = ..., buildsandbox: _Optional[_Union[Artifact.SandboxState, _Mapping]] = ..., speculative_actions: _Optional[_Union[_remote_execution_pb2.Digest, _Mapping]] = ...) -> None: ... diff --git a/src/buildstream/_protos/buildstream/v2/speculative_actions.proto b/src/buildstream/_protos/buildstream/v2/speculative_actions.proto new file mode 100644 index 000000000..399c224a6 --- /dev/null +++ b/src/buildstream/_protos/buildstream/v2/speculative_actions.proto @@ -0,0 +1,69 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +syntax = "proto3"; + +package buildstream.v2; + +import "build/bazel/remote/execution/v2/remote_execution.proto"; + +// SpeculativeActions: Metadata for cache priming via speculative execution +// +// This message stores overlay information that allows BuildStream to: +// 1. Instantiate previously-recorded Actions with current dependency versions +// 2. Submit these adapted Actions to prime the Remote Execution ActionCache +// 3. Speed up builds when dependencies change but not the element itself +message SpeculativeActions { + // Speculative actions for this element's build + repeated SpeculativeAction actions = 1; + + // Overlays mapping artifact file digests to their sources + // Enables downstream elements to resolve dependencies without fetching sources + repeated Overlay artifact_overlays = 2; + + message SpeculativeAction { + // Original action digest from the recorded build + build.bazel.remote.execution.v2.Digest base_action_digest = 1; + + // Overlays to apply when instantiating this action + repeated Overlay overlays = 2; + } + + message Overlay { + enum OverlayType { + SOURCE = 0; // From element's source tree + ARTIFACT = 1; // From dependency element's artifact output + ACTION = 2; // Output of a prior subaction + } + + OverlayType type = 1; + + // Element name providing the source + // Empty string means the element itself (self-reference) + // For ACTION overlays: the element containing the producing subaction + // (empty string = same element; populated for cross-element ACTION overlays) + string source_element = 2; + + // Path within source tree, artifact, or ActionResult output files + string source_path = 4; + + // The digest that should be replaced in the action's input tree + // When instantiating, find all occurrences of this digest and replace + // with the current digest of the file at source_path + build.bazel.remote.execution.v2.Digest target_digest = 5; + + // For ACTION overlays: base_action_digest of the producing subaction + // Used to look up the producing subaction's ActionResult during priming + build.bazel.remote.execution.v2.Digest source_action_digest = 3; + } +} diff --git a/src/buildstream/_protos/buildstream/v2/speculative_actions_pb2.py b/src/buildstream/_protos/buildstream/v2/speculative_actions_pb2.py new file mode 100644 index 000000000..d82d74535 --- /dev/null +++ b/src/buildstream/_protos/buildstream/v2/speculative_actions_pb2.py @@ -0,0 +1,43 @@ +# -*- coding: utf-8 -*- +# Generated by the protocol buffer compiler. DO NOT EDIT! +# NO CHECKED-IN PROTOBUF GENCODE +# source: buildstream/v2/speculative_actions.proto +# Protobuf Python Version: 5.29.0 +"""Generated protocol buffer code.""" +from google.protobuf import descriptor as _descriptor +from google.protobuf import descriptor_pool as _descriptor_pool +from google.protobuf import runtime_version as _runtime_version +from google.protobuf import symbol_database as _symbol_database +from google.protobuf.internal import builder as _builder +_runtime_version.ValidateProtobufRuntimeVersion( + _runtime_version.Domain.PUBLIC, + 5, + 29, + 0, + '', + 'buildstream/v2/speculative_actions.proto' +) +# @@protoc_insertion_point(imports) + +_sym_db = _symbol_database.Default() + + +from buildstream._protos.build.bazel.remote.execution.v2 import remote_execution_pb2 as build_dot_bazel_dot_remote_dot_execution_dot_v2_dot_remote__execution__pb2 + + +DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n(buildstream/v2/speculative_actions.proto\x12\x0e\x62uildstream.v2\x1a\x36\x62uild/bazel/remote/execution/v2/remote_execution.proto\"\xf6\x04\n\x12SpeculativeActions\x12\x45\n\x07\x61\x63tions\x18\x01 \x03(\x0b\x32\x34.buildstream.v2.SpeculativeActions.SpeculativeAction\x12\x45\n\x11\x61rtifact_overlays\x18\x02 \x03(\x0b\x32*.buildstream.v2.SpeculativeActions.Overlay\x1a\x96\x01\n\x11SpeculativeAction\x12\x43\n\x12\x62\x61se_action_digest\x18\x01 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12<\n\x08overlays\x18\x02 \x03(\x0b\x32*.buildstream.v2.SpeculativeActions.Overlay\x1a\xb8\x02\n\x07Overlay\x12\x44\n\x04type\x18\x01 \x01(\x0e\x32\x36.buildstream.v2.SpeculativeActions.Overlay.OverlayType\x12\x16\n\x0esource_element\x18\x02 \x01(\t\x12\x13\n\x0bsource_path\x18\x04 \x01(\t\x12>\n\rtarget_digest\x18\x05 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\x12\x45\n\x14source_action_digest\x18\x03 \x01(\x0b\x32\'.build.bazel.remote.execution.v2.Digest\"3\n\x0bOverlayType\x12\n\n\x06SOURCE\x10\x00\x12\x0c\n\x08\x41RTIFACT\x10\x01\x12\n\n\x06\x41\x43TION\x10\x02\x62\x06proto3') + +_globals = globals() +_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals) +_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'buildstream.v2.speculative_actions_pb2', _globals) +if not _descriptor._USE_C_DESCRIPTORS: + DESCRIPTOR._loaded_options = None + _globals['_SPECULATIVEACTIONS']._serialized_start=117 + _globals['_SPECULATIVEACTIONS']._serialized_end=747 + _globals['_SPECULATIVEACTIONS_SPECULATIVEACTION']._serialized_start=282 + _globals['_SPECULATIVEACTIONS_SPECULATIVEACTION']._serialized_end=432 + _globals['_SPECULATIVEACTIONS_OVERLAY']._serialized_start=435 + _globals['_SPECULATIVEACTIONS_OVERLAY']._serialized_end=747 + _globals['_SPECULATIVEACTIONS_OVERLAY_OVERLAYTYPE']._serialized_start=696 + _globals['_SPECULATIVEACTIONS_OVERLAY_OVERLAYTYPE']._serialized_end=747 +# @@protoc_insertion_point(module_scope) diff --git a/src/buildstream/_protos/buildstream/v2/speculative_actions_pb2.pyi b/src/buildstream/_protos/buildstream/v2/speculative_actions_pb2.pyi new file mode 100644 index 000000000..6155b15e1 --- /dev/null +++ b/src/buildstream/_protos/buildstream/v2/speculative_actions_pb2.pyi @@ -0,0 +1,44 @@ +from build.bazel.remote.execution.v2 import remote_execution_pb2 as _remote_execution_pb2 +from google.protobuf.internal import containers as _containers +from google.protobuf.internal import enum_type_wrapper as _enum_type_wrapper +from google.protobuf import descriptor as _descriptor +from google.protobuf import message as _message +from typing import ClassVar as _ClassVar, Iterable as _Iterable, Mapping as _Mapping, Optional as _Optional, Union as _Union + +DESCRIPTOR: _descriptor.FileDescriptor + +class SpeculativeActions(_message.Message): + __slots__ = ("actions", "artifact_overlays") + class SpeculativeAction(_message.Message): + __slots__ = ("base_action_digest", "overlays") + BASE_ACTION_DIGEST_FIELD_NUMBER: _ClassVar[int] + OVERLAYS_FIELD_NUMBER: _ClassVar[int] + base_action_digest: _remote_execution_pb2.Digest + overlays: _containers.RepeatedCompositeFieldContainer[SpeculativeActions.Overlay] + def __init__(self, base_action_digest: _Optional[_Union[_remote_execution_pb2.Digest, _Mapping]] = ..., overlays: _Optional[_Iterable[_Union[SpeculativeActions.Overlay, _Mapping]]] = ...) -> None: ... + class Overlay(_message.Message): + __slots__ = ("type", "source_element", "source_path", "target_digest", "source_action_digest") + class OverlayType(int, metaclass=_enum_type_wrapper.EnumTypeWrapper): + __slots__ = () + SOURCE: _ClassVar[SpeculativeActions.Overlay.OverlayType] + ARTIFACT: _ClassVar[SpeculativeActions.Overlay.OverlayType] + ACTION: _ClassVar[SpeculativeActions.Overlay.OverlayType] + SOURCE: SpeculativeActions.Overlay.OverlayType + ARTIFACT: SpeculativeActions.Overlay.OverlayType + ACTION: SpeculativeActions.Overlay.OverlayType + TYPE_FIELD_NUMBER: _ClassVar[int] + SOURCE_ELEMENT_FIELD_NUMBER: _ClassVar[int] + SOURCE_PATH_FIELD_NUMBER: _ClassVar[int] + TARGET_DIGEST_FIELD_NUMBER: _ClassVar[int] + SOURCE_ACTION_DIGEST_FIELD_NUMBER: _ClassVar[int] + type: SpeculativeActions.Overlay.OverlayType + source_element: str + source_path: str + target_digest: _remote_execution_pb2.Digest + source_action_digest: _remote_execution_pb2.Digest + def __init__(self, type: _Optional[_Union[SpeculativeActions.Overlay.OverlayType, str]] = ..., source_element: _Optional[str] = ..., source_path: _Optional[str] = ..., target_digest: _Optional[_Union[_remote_execution_pb2.Digest, _Mapping]] = ..., source_action_digest: _Optional[_Union[_remote_execution_pb2.Digest, _Mapping]] = ...) -> None: ... + ACTIONS_FIELD_NUMBER: _ClassVar[int] + ARTIFACT_OVERLAYS_FIELD_NUMBER: _ClassVar[int] + actions: _containers.RepeatedCompositeFieldContainer[SpeculativeActions.SpeculativeAction] + artifact_overlays: _containers.RepeatedCompositeFieldContainer[SpeculativeActions.Overlay] + def __init__(self, actions: _Optional[_Iterable[_Union[SpeculativeActions.SpeculativeAction, _Mapping]]] = ..., artifact_overlays: _Optional[_Iterable[_Union[SpeculativeActions.Overlay, _Mapping]]] = ...) -> None: ... diff --git a/src/buildstream/_protos/buildstream/v2/speculative_actions_pb2_grpc.py b/src/buildstream/_protos/buildstream/v2/speculative_actions_pb2_grpc.py new file mode 100644 index 000000000..7c3546642 --- /dev/null +++ b/src/buildstream/_protos/buildstream/v2/speculative_actions_pb2_grpc.py @@ -0,0 +1,24 @@ +# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT! +"""Client and server classes corresponding to protobuf-defined services.""" +import grpc +import warnings + + +GRPC_GENERATED_VERSION = '1.69.0' +GRPC_VERSION = grpc.__version__ +_version_not_supported = False + +try: + from grpc._utilities import first_version_is_lower + _version_not_supported = first_version_is_lower(GRPC_VERSION, GRPC_GENERATED_VERSION) +except ImportError: + _version_not_supported = True + +if _version_not_supported: + raise RuntimeError( + f'The grpc package installed is at version {GRPC_VERSION},' + + f' but the generated code in buildstream/v2/speculative_actions_pb2_grpc.py depends on' + + f' grpcio>={GRPC_GENERATED_VERSION}.' + + f' Please upgrade your grpc module to grpcio>={GRPC_GENERATED_VERSION}' + + f' or downgrade your generated code using grpcio-tools<={GRPC_VERSION}.' + ) diff --git a/src/buildstream/_scheduler/__init__.py b/src/buildstream/_scheduler/__init__.py index d37e47b6f..0b849045f 100644 --- a/src/buildstream/_scheduler/__init__.py +++ b/src/buildstream/_scheduler/__init__.py @@ -23,6 +23,8 @@ from .queues.artifactpushqueue import ArtifactPushQueue from .queues.pullqueue import PullQueue from .queues.cachequeryqueue import CacheQueryQueue +from .queues.speculativeactiongenerationqueue import SpeculativeActionGenerationQueue +from .queues.speculativecacheprimingqueue import SpeculativeCachePrimingQueue from .scheduler import Scheduler, SchedStatus from .jobs import ElementJob, JobStatus diff --git a/src/buildstream/_scheduler/queues/speculativeactiongenerationqueue.py b/src/buildstream/_scheduler/queues/speculativeactiongenerationqueue.py new file mode 100644 index 000000000..3c0fc57a6 --- /dev/null +++ b/src/buildstream/_scheduler/queues/speculativeactiongenerationqueue.py @@ -0,0 +1,117 @@ +# +# Copyright 2025 The Apache Software Foundation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +SpeculativeActionGenerationQueue +================================= + +Queue for generating SpeculativeActions after element builds. + +This queue runs after BuildQueue to: +1. Extract subaction digests from built elements +2. Generate SOURCE and ARTIFACT overlays +3. Store SpeculativeActions with the artifact +""" + +# Local imports +from . import Queue, QueueStatus +from ..jobs import JobStatus + + +# A queue which generates speculative actions for built elements +# +class SpeculativeActionGenerationQueue(Queue): + + action_name = "Generating overlays" + complete_name = "Overlays generated" + resources = [] # No special resources needed + + def get_process_func(self): + return SpeculativeActionGenerationQueue._generate_overlays + + def status(self, element): + # Only process elements that were successfully built + # and have subaction digests + if not element._cached_success(): + return QueueStatus.SKIP + + # Check if element has subaction digests + subaction_digests = element._get_subaction_digests() + if not subaction_digests: + return QueueStatus.SKIP + + return QueueStatus.READY + + def done(self, _, element, result, status): + if status is JobStatus.FAIL: + # Generation is best-effort, don't fail the build + pass + + # Result contains the SpeculativeActions that were generated + # The artifact cache has already been updated in the child process + + @staticmethod + def _generate_overlays(element): + """ + Generate SpeculativeActions for an element. + + Args: + element: The element to generate overlays for + + Returns: + Number of actions generated, or None if skipped + """ + from ..._speculative_actions.generator import SpeculativeActionsGenerator + + # Get subaction digests + subaction_digests = element._get_subaction_digests() + if not subaction_digests: + return None + + # Get the context and caches + context = element._get_context() + cas = context.get_cascache() + artifactcache = context.artifactcache + + # Get dependencies to resolve overlays + from ...types import _Scope + + dependencies = list(element._dependencies(_Scope.BUILD, recurse=False)) + + # Get action cache service for ACTION overlay generation + # (only needed for intra-element and full modes) + from ...types import _SpeculativeActionMode + + mode = context.speculative_actions_mode + casd = context.get_casd() + ac_service = None + if mode in (_SpeculativeActionMode.INTRA_ELEMENT, _SpeculativeActionMode.FULL): + ac_service = casd.get_ac_service() if casd else None + + # Generate overlays + generator = SpeculativeActionsGenerator(cas, ac_service=ac_service, artifactcache=artifactcache) + spec_actions = generator.generate_speculative_actions( + element, subaction_digests, dependencies, mode=mode + ) + + if not spec_actions or not spec_actions.actions: + return 0 + + # Store with the artifact, using weak key for stable lookup + artifact = element._get_artifact() + weak_key = element._get_weak_cache_key() + artifactcache.store_speculative_actions(artifact, spec_actions, weak_key=weak_key) + + return len(spec_actions.actions) diff --git a/src/buildstream/_scheduler/queues/speculativecacheprimingqueue.py b/src/buildstream/_scheduler/queues/speculativecacheprimingqueue.py new file mode 100644 index 000000000..9550402e3 --- /dev/null +++ b/src/buildstream/_scheduler/queues/speculativecacheprimingqueue.py @@ -0,0 +1,489 @@ +# +# Copyright 2025 The Apache Software Foundation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +SpeculativeCachePrimingQueue +============================= + +Queue for priming the ActionCache with speculative actions. + +This queue runs BEFORE BuildQueue and uses the PENDING state to hold +elements while their dependencies build. While an element waits, +background priming runs fire-and-forget — submitting adapted actions +to casd for execution. As each dependency completes, per-dep callbacks +trigger incremental overlay resolution, unlocking more subactions. + +When all dependencies are cached and the element becomes buildable, +a final priming pass resolves remaining ACTION overlays and the +element is released to the BuildQueue. By then, most adapted actions +are already in the action cache — recc gets cache hits. + +Elements without stored SpeculativeActions skip this queue entirely. + +Cross-element ACTION overlay resolution uses a global +``_instantiated_actions`` dict (base_action_hash → adapted_action_digest) +shared across all elements. When element A's priming instantiates a +subaction, the mapping is immediately visible to element B's priming, +enabling cross-element intermediate file resolution. +""" + +import threading + +# Local imports +from . import Queue, QueueStatus +from ..jobs import JobStatus +from ..resources import ResourceType + + +class SpeculativeCachePrimingQueue(Queue): + + action_name = "Priming cache" + complete_name = "Cache primed" + resources = [ResourceType.UPLOAD] + + # Global shared state: maps base_action_hash -> adapted_action_digest + # Populated by all elements during priming, enabling cross-element + # ACTION overlay resolution. + _instantiated_actions = {} + _instantiated_actions_lock = threading.Lock() + + # Elements whose priming has completed (all passes done). + # Used to determine if an ACTION overlay's producing element has + # finished priming — if so and the action isn't in _instantiated_actions, + # the overlay can be permanently dropped from the SA proto. + _primed_elements = set() + + def get_process_func(self): + # Runs when element is READY (buildable) — final priming pass + return SpeculativeCachePrimingQueue._final_prime_pass + + def status(self, element): + if element._cached(): + # Already cached — no priming needed. Record as primed so + # downstream elements know this element's actions won't appear + # in _instantiated_actions. + SpeculativeCachePrimingQueue._primed_elements.add(element.name) + return QueueStatus.SKIP + + weak_key = element._get_weak_cache_key() + if not weak_key: + SpeculativeCachePrimingQueue._primed_elements.add(element.name) + return QueueStatus.SKIP + + context = element._get_context() + artifactcache = context.artifactcache + spec_actions = artifactcache.lookup_speculative_actions_by_weak_key(element, weak_key) + if not spec_actions or not spec_actions.actions: + SpeculativeCachePrimingQueue._primed_elements.add(element.name) + return QueueStatus.SKIP + + # Has SAs. If not buildable, enter PENDING — background + # priming will run while we wait for dependencies. + if not element._buildable(): + return QueueStatus.PENDING + + # Already buildable — run final priming pass as a job + return QueueStatus.READY + + def register_pending_element(self, element): + # Register per-dep callback for incremental overlay resolution + element._set_build_dep_cached_callback(self._on_dep_cached) + + # Register per-dep callback for when a dependency finishes + # priming — its adapted actions are now in _instantiated_actions + # and downstream elements can resolve cross-element ACTION overlays + element._set_build_dep_primed_callback(self._on_dep_primed) + + # Also register buildable callback so we get re-enqueued + # when the element becomes fully buildable + element._set_buildable_callback(self._enqueue_element) + + # Launch background priming immediately in the scheduler's + # thread pool — fire-and-forget independent subactions while + # we wait for dependencies + self._scheduler.loop.call_soon(self._launch_background_priming, element) + + def _launch_background_priming(self, element): + self._scheduler.loop.run_in_executor( + None, SpeculativeCachePrimingQueue._background_prime, element + ) + + def _on_dep_cached(self, element, dep): + """Called each time a build dependency of element becomes cached. + + Launches incremental priming in the background — newly resolvable + ARTIFACT overlays (dep's artifact now cached) and ACTION overlays + (dep's subaction results now in AC) can be resolved and submitted. + """ + self._scheduler.loop.call_soon( + self._launch_incremental_prime, element, dep + ) + + def _on_dep_primed(self, element, dep): + """Called each time a build dependency finishes priming. + + The dep's adapted action digests are now in _instantiated_actions. + Launches incremental priming to resolve cross-element ACTION + overlays that reference the dep's subactions. + """ + self._scheduler.loop.call_soon( + self._launch_incremental_prime, element, dep + ) + + def _launch_incremental_prime(self, element, dep): + self._scheduler.loop.run_in_executor( + None, SpeculativeCachePrimingQueue._incremental_prime, element, dep + ) + + def done(self, _, element, result, status): + if status is JobStatus.FAIL: + return + + if result: + primed, skipped, total = result + if skipped: + element.info(f"Primed {primed}/{total} actions ({skipped} skipped)") + else: + element.info(f"Primed {primed}/{total} actions") + + # Record element as primed so other elements can determine + # whether ACTION overlay producers have finished. + with SpeculativeCachePrimingQueue._instantiated_actions_lock: + SpeculativeCachePrimingQueue._primed_elements.add(element.name) + + # Notify reverse build deps that this element finished priming — + # its adapted actions are now in _instantiated_actions and + # downstream elements can resolve cross-element ACTION overlays. + element._notify_build_deps_primed() + + # Clear priming state and per-dep callbacks + element._set_build_dep_cached_callback(None) + element._set_build_dep_primed_callback(None) + element.__priming_submitted = None + element.__priming_spec_actions = None + element.__priming_resolved = None + element.__priming_ac_cache = None + + # ----------------------------------------------------------------- + # Background priming (runs in thread pool while element is PENDING) + # ----------------------------------------------------------------- + + @staticmethod + def _background_prime(element): + """Initial background priming pass. + + Fire-and-forget subactions whose overlays can be resolved from + already-cached deps. Defer everything else. + """ + SpeculativeCachePrimingQueue._do_prime_pass(element) + + @staticmethod + def _incremental_prime(element, dep): + """Incremental priming after a dependency becomes cached. + + Re-attempt overlay resolution — the newly cached dep may unlock + ARTIFACT overlays or ACTION overlays. + """ + SpeculativeCachePrimingQueue._do_prime_pass(element) + + @staticmethod + def _do_prime_pass(element): + """Core priming logic shared by background and incremental passes. + + Iterates over all subactions, skipping already-submitted ones. + For each remaining subaction, attempts to resolve all overlays. + If resolvable, instantiates and submits fire-and-forget. + + ACTION overlay resolution uses the global _instantiated_actions + dict. For each ACTION overlay: + - If the producing action is in _instantiated_actions → check + AC for the ActionResult; if not yet available, defer the SA + - If the producing action is NOT in _instantiated_actions AND + its source_element has finished priming → the producing + action will never appear; remove the overlay from the proto + - If the producing action is NOT in _instantiated_actions AND + its source_element has NOT finished priming → skip for now, + it may appear on a later pass + + Dropped overlays are removed directly from the in-memory SA + proto (which is discarded after the build). + """ + from ..._speculative_actions.instantiator import SpeculativeActionInstantiator + from ..._protos.buildstream.v2 import speculative_actions_pb2 + from ..._protos.build.bazel.remote.execution.v2 import remote_execution_pb2 + + context = element._get_context() + cas = context.get_cascache() + artifactcache = context.artifactcache + + # Use the cached spec_actions proto if available (mutations and + # _resolved_cache attributes must survive across passes). Only + # deserialize from CAS on the first pass. + spec_actions = getattr(element, "_SpeculativeCachePrimingQueue__priming_spec_actions", None) + if spec_actions is None: + weak_key = element._get_weak_cache_key() + spec_actions = artifactcache.lookup_speculative_actions_by_weak_key(element, weak_key) + if not spec_actions or not spec_actions.actions: + return + + # Recover or initialize per-element state + submitted = getattr(element, "_SpeculativeCachePrimingQueue__priming_submitted", None) or set() + # Per-SA resolution caches: {base_action_hash -> {target_hash -> new_digest}} + resolved_caches = getattr(element, "_SpeculativeCachePrimingQueue__priming_resolved", None) or {} + # AC result cache: avoids redundant GetActionResult gRPCs across passes. + # Maps adapted_digest_hash -> ActionResult (or False for "checked, not found"). + ac_cache = getattr(element, "_SpeculativeCachePrimingQueue__priming_ac_cache", None) or {} + + # Pre-fetch CAS blobs only on first pass + if not submitted: + SpeculativeCachePrimingQueue._prefetch_cas_blobs( + element, spec_actions, cas, artifactcache + ) + + # Build element lookup + from ...types import _Scope + + dependencies = list(element._dependencies(_Scope.BUILD, recurse=True)) + element_lookup = {dep.name: dep for dep in dependencies} + element_lookup[element.name] = element + + # Get services + casd = context.get_casd() + exec_service = casd.get_exec_service() + if not exec_service: + return + + ac_service = casd.get_ac_service() + instantiator = SpeculativeActionInstantiator(cas, artifactcache, ac_service=ac_service) + + # References to global state (reads are GIL-safe) + instantiated_actions = SpeculativeCachePrimingQueue._instantiated_actions + primed_elements = SpeculativeCachePrimingQueue._primed_elements + + for spec_action in spec_actions.actions: + base_hash = spec_action.base_action_digest.hash + + if base_hash in submitted: + continue + + # Check ACTION overlay resolvability against global state, + # removing overlays that will never resolve. + resolvable = True + to_remove = [] + for i, overlay in enumerate(spec_action.overlays): + if overlay.type != speculative_actions_pb2.SpeculativeActions.Overlay.ACTION: + continue + + source_hash = overlay.source_action_digest.hash + adapted = instantiated_actions.get(source_hash) + + if adapted is not None: + # Producing action was instantiated — check if + # result is in AC (using cache to avoid redundant + # gRPC calls across passes) + cached_result = ac_cache.get(adapted.hash) + if cached_result is None: + # Not in cache — query AC + if ac_service: + try: + request = remote_execution_pb2.GetActionResultRequest( + action_digest=adapted, + ) + action_result = ac_service.GetActionResult(request) + if action_result: + ac_cache[adapted.hash] = action_result + else: + # Not yet complete — defer (don't cache + # negative result, it may complete later) + resolvable = False + break + except Exception: + resolvable = False + break + elif cached_result is False: + # Previously checked and not found + resolvable = False + break + # else: cached_result is a valid ActionResult, proceed + else: + # Not in instantiated_actions — check if the + # producing element has finished priming + source_elem = overlay.source_element or element.name + if source_elem in primed_elements: + # Element finished priming without instantiating + # this action — it will never appear. Mark for + # removal from the proto. + to_remove.append(i) + # else: source element not yet primed, skip for now + + # Remove dropped overlays from the proto (reverse order to + # preserve indices) + for i in reversed(to_remove): + del spec_action.overlays[i] + + if not resolvable: + continue + + try: + # Get or create per-SA resolution cache + sa_cache = resolved_caches.setdefault(base_hash, {}) + action_digest = instantiator.instantiate_action( + spec_action, element, element_lookup, + instantiated_actions=instantiated_actions, + resolved_cache=sa_cache, + ) + + if not action_digest: + continue + + # Record in global state (write-locked) + with SpeculativeCachePrimingQueue._instantiated_actions_lock: + instantiated_actions[base_hash] = action_digest + + # Skip unchanged actions (already in AC from previous build) + if action_digest.hash == base_hash: + submitted.add(base_hash) + continue + + SpeculativeCachePrimingQueue._submit_action_async( + exec_service, action_digest, element + ) + element.info( + f"Submitted action {action_digest.hash[:8]} " + f"(base {base_hash[:8]})" + ) + submitted.add(base_hash) + + except Exception as e: + element.warn(f"Failed to prime action: {e}") + continue + + # Store per-element state for next pass + element.__priming_submitted = submitted + element.__priming_spec_actions = spec_actions + element.__priming_resolved = resolved_caches + element.__priming_ac_cache = ac_cache + + # ----------------------------------------------------------------- + # Final priming pass (runs as a job when element becomes READY) + # ----------------------------------------------------------------- + + @staticmethod + def _final_prime_pass(element): + """Final priming pass when element is buildable. + + All deps are built, so all ActionResults are in AC. + Resolve any remaining ACTION overlays and submit. + """ + # Run the same logic — it will pick up where background left off. + # By now all deps are built, so _primed_elements contains all + # producing elements. Any ACTION overlay whose source_element + # is in _primed_elements but whose action is not in + # _instantiated_actions will be removed from the proto. + SpeculativeCachePrimingQueue._do_prime_pass(element) + + # Count results + submitted = getattr(element, "_SpeculativeCachePrimingQueue__priming_submitted", None) or set() + spec_actions = getattr(element, "_SpeculativeCachePrimingQueue__priming_spec_actions", None) + if not spec_actions: + return (0, 0, 0) + + total = len(spec_actions.actions) + primed = len(submitted) + skipped = total - primed + + return (primed, skipped, total) + + # ----------------------------------------------------------------- + # Utility methods + # ----------------------------------------------------------------- + + @staticmethod + def _prefetch_cas_blobs(element, spec_actions, cas, artifactcache): + """Pre-fetch all CAS blobs needed for instantiation. + + Fetches base action blobs in a single batch, then deduplicates + input root digests and fetches directory trees concurrently. + """ + from concurrent.futures import ThreadPoolExecutor, as_completed + + project = element._get_project() + _, storage_remotes = artifactcache.get_remotes(project.name, False) + remote = storage_remotes[0] if storage_remotes else None + + if not remote: + return + + base_action_digests = [ + sa.base_action_digest + for sa in spec_actions.actions + if sa.base_action_digest.hash + ] + if base_action_digests: + try: + cas.fetch_blobs(remote, base_action_digests, allow_partial=True) + except Exception: + pass + + # Collect and deduplicate input root digests + unique_roots = {} # hash -> digest + for digest in base_action_digests: + try: + action = cas.fetch_action(digest) + if action and action.HasField("input_root_digest"): + root = action.input_root_digest + if root.hash not in unique_roots: + unique_roots[root.hash] = root + except Exception: + pass + + if not unique_roots: + return + + # Fetch directory trees concurrently + def _fetch_tree(root_digest): + try: + cas.fetch_directory(remote, root_digest) + except Exception: + pass + + with ThreadPoolExecutor(max_workers=min(16, len(unique_roots))) as pool: + futures = [pool.submit(_fetch_tree, d) for d in unique_roots.values()] + for f in as_completed(futures): + pass # Errors handled inside _fetch_tree + + @staticmethod + def _submit_action_async(exec_service, action_digest, element): + """Submit an Execute request fire-and-forget style. + + Reads the first response from the stream to confirm the action + was accepted by casd, then returns. The action continues + executing asynchronously in casd and its result will appear in + the action cache when complete. + """ + try: + from ..._protos.build.bazel.remote.execution.v2 import remote_execution_pb2 + + request = remote_execution_pb2.ExecuteRequest( + action_digest=action_digest, + skip_cache_lookup=False, + ) + + # Read first response to confirm acceptance, then drop the stream + stream = exec_service.Execute(request) + next(stream, None) + + except Exception as e: + element.warn(f"Failed to submit priming action: {e}") diff --git a/src/buildstream/_speculative_actions/__init__.py b/src/buildstream/_speculative_actions/__init__.py new file mode 100644 index 000000000..a94e55216 --- /dev/null +++ b/src/buildstream/_speculative_actions/__init__.py @@ -0,0 +1,30 @@ +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Speculative Actions - Cache Priming Infrastructure +=================================================== + +This module implements the Speculative Actions feature for BuildStream, +which enables predictive cache priming by recording and replaying compiler +invocations with updated dependency versions. + +Key Components: +- generator: Generates SpeculativeActions and artifact overlays after builds +- instantiator: Applies overlays to instantiate actions before builds +""" + +from .generator import SpeculativeActionsGenerator +from .instantiator import SpeculativeActionInstantiator + +__all__ = ["SpeculativeActionsGenerator", "SpeculativeActionInstantiator"] diff --git a/src/buildstream/_speculative_actions/generator.py b/src/buildstream/_speculative_actions/generator.py new file mode 100644 index 000000000..dfd6738d0 --- /dev/null +++ b/src/buildstream/_speculative_actions/generator.py @@ -0,0 +1,530 @@ +# +# Copyright 2025 The Apache Software Foundation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +SpeculativeActionsGenerator +============================ + +Generates SpeculativeActions and artifact overlays after element builds. + +This module is responsible for: +1. Extracting subaction digests from ActionResult +2. Traversing action input trees to find all file digests +3. Resolving digests to their source elements (SOURCE > ACTION > ARTIFACT priority) +4. Creating overlays for each digest +5. Generating artifact_overlays for the element's output files +6. Tracking inter-subaction output dependencies via ACTION overlays +""" + +from typing import Dict, Tuple + + +class SpeculativeActionsGenerator: + """ + Generates SpeculativeActions from element builds. + + This class analyzes completed element builds to extract subactions and + generate overlay metadata that describes how to adapt inputs for future + builds. + """ + + def __init__(self, cas, ac_service=None, artifactcache=None): + """ + Initialize the generator. + + Args: + cas: The CAS cache for fetching actions and directories + ac_service: Optional ActionCache service stub for fetching + ActionResults of prior subactions (needed for ACTION overlays) + artifactcache: Optional artifact cache for loading dependency + SpeculativeActions (needed for cross-element ACTION overlays) + """ + self._cas = cas + self._ac_service = ac_service + self._artifactcache = artifactcache + # Cache for digest.hash -> list of (element, path, type) lookups + # Multiple entries per digest enable fallback resolution: + # SOURCE overlays are tried first, then ACTION, then ARTIFACT. + self._digest_cache: Dict[str, list] = {} + # Artifact file entries for the element being processed, + # collected during _build_digest_cache to avoid re-traversal + # in _generate_artifact_overlays. + # List of (digest_hash, digest_size) tuples. + self._own_artifact_entries: list = [] + + def generate_speculative_actions(self, element, subaction_digests, dependencies, mode=None): + """ + Generate SpeculativeActions for an element build. + + This is the main entry point for overlay generation. It processes + all subactions from the element's build and generates overlays + for each. + + Args: + element: The element that was built + subaction_digests: List of Action digests from the build (from ActionResult.subactions) + dependencies: List of dependency elements (for resolving artifact overlays) + mode: _SpeculativeActionMode controlling which overlay types to generate. + None defaults to FULL for backward compatibility. + + Returns: + A SpeculativeActions message containing: + - actions: SpeculativeActions with overlays for each subaction + - artifact_overlays: Overlays mapping artifact file digests to sources + """ + from .._protos.buildstream.v2 import speculative_actions_pb2 + from .._protos.build.bazel.remote.execution.v2 import remote_execution_pb2 + from ..types import _SpeculativeActionMode + + if mode is None: + mode = _SpeculativeActionMode.FULL + + spec_actions = speculative_actions_pb2.SpeculativeActions() + + # Build digest lookup tables from element sources and dependencies + self._build_digest_cache(element, dependencies) + + # Track outputs from prior subactions for ACTION overlay generation + # Maps file_digest_hash -> (source_element, producing_action_digest, output_path) + prior_outputs = {} + + # Seed prior_outputs with dependency subaction outputs for + # cross-element ACTION overlays (full mode only). + # These enable earlier resolution than ARTIFACT overlays: an + # ACTION overlay resolves when the dep is primed (via + # instantiated_actions), while an ARTIFACT overlay only resolves + # when the dep is fully built. This parallelism is the core + # benefit of speculative actions. + if mode == _SpeculativeActionMode.FULL: + if self._ac_service and self._artifactcache: + self._seed_dependency_outputs(dependencies, prior_outputs) + + # Generate overlays for each subaction + for subaction_digest in subaction_digests: + spec_action, input_digests = self._generate_action_overlays(element, subaction_digest) + + # Generate ACTION overlays for digests that match prior subaction outputs + # but weren't already resolved as SOURCE or ARTIFACT. + # Requires intra-element or full mode. + if mode in (_SpeculativeActionMode.INTRA_ELEMENT, _SpeculativeActionMode.FULL): + if self._ac_service and prior_outputs and input_digests: + # Collect hashes already covered by SOURCE/ARTIFACT overlays + already_overlaid = set() + if spec_action: + for overlay in spec_action.overlays: + already_overlaid.add(overlay.target_digest.hash) + + for digest_hash, digest_size in input_digests: + if digest_hash in prior_outputs and digest_hash not in already_overlaid: + source_element, producing_action_digest, output_path = prior_outputs[digest_hash] + # Create ACTION overlay + if spec_action is None: + spec_action = speculative_actions_pb2.SpeculativeActions.SpeculativeAction() + spec_action.base_action_digest.CopyFrom(subaction_digest) + overlay = speculative_actions_pb2.SpeculativeActions.Overlay() + overlay.type = speculative_actions_pb2.SpeculativeActions.Overlay.ACTION + overlay.source_element = source_element + overlay.source_action_digest.CopyFrom(producing_action_digest) + overlay.source_path = output_path + overlay.target_digest.hash = digest_hash + overlay.target_digest.size_bytes = digest_size + spec_action.overlays.append(overlay) + + # Sort overlays: SOURCE > ACTION > ARTIFACT + # This ensures the instantiator tries SOURCE first, then + # ACTION (intermediate files), then ARTIFACT as fallback. + if spec_action: + type_priority = { + speculative_actions_pb2.SpeculativeActions.Overlay.SOURCE: 0, + speculative_actions_pb2.SpeculativeActions.Overlay.ACTION: 1, + speculative_actions_pb2.SpeculativeActions.Overlay.ARTIFACT: 2, + } + spec_action.overlays.sort(key=lambda o: type_priority.get(o.type, 99)) + spec_actions.actions.append(spec_action) + + # Fetch this subaction's ActionResult and record its outputs + # for subsequent subactions (intra-element and full modes) + if mode in (_SpeculativeActionMode.INTRA_ELEMENT, _SpeculativeActionMode.FULL): + if self._ac_service: + self._record_subaction_outputs(subaction_digest, prior_outputs) + + # Generate artifact overlays for the element's output files + artifact_overlays = self._generate_artifact_overlays(element) + spec_actions.artifact_overlays.extend(artifact_overlays) + + return spec_actions + + def _record_subaction_outputs(self, action_digest, prior_outputs, source_element=""): + """ + Fetch a subaction's ActionResult from the action cache and record + its output file digests for subsequent subaction ACTION overlay generation. + + Args: + action_digest: The action digest to look up (stored on ACTION overlays) + prior_outputs: Dict to update with file_digest_hash -> (source_element, action_digest, path) + source_element: Element name for cross-element overlays ("" = same element) + """ + try: + from .._protos.build.bazel.remote.execution.v2 import remote_execution_pb2 + + request = remote_execution_pb2.GetActionResultRequest( + action_digest=action_digest, + ) + action_result = self._ac_service.GetActionResult(request) + if action_result: + for output_file in action_result.output_files: + prior_outputs[output_file.digest.hash] = ( + source_element, action_digest, output_file.path + ) + except Exception: + pass + + def _seed_dependency_outputs(self, dependencies, prior_outputs): + """ + Seed prior_outputs with subaction outputs from dependency elements. + + For each dependency that has stored SpeculativeActions, fetch the + ActionResult of each subaction and record its output files. This + enables cross-element ACTION overlays: if the current element's + subaction input tree contains a file that was produced by a + dependency's subaction, the overlay will reference it. + + Cross-element ACTION overlays enable earlier resolution than + ARTIFACT overlays: they resolve when the dep is primed (via + instantiated_actions), while ARTIFACT overlays only resolve + when the dep is fully built. + + Args: + dependencies: List of dependency elements + prior_outputs: Dict to seed with file_digest_hash -> + (source_element, action_digest, path) + """ + for dep in dependencies: + try: + if not dep._cached(): + continue + + artifact = dep._get_artifact() + if not artifact or not artifact.cached(): + continue + + dep_sa = self._artifactcache.get_speculative_actions(artifact) + if not dep_sa: + continue + + for spec_action in dep_sa.actions: + self._record_subaction_outputs( + spec_action.base_action_digest, + prior_outputs, + source_element=dep.name, + ) + except Exception: + pass + + def _build_digest_cache(self, element, dependencies): + """ + Build a cache mapping file digests to their source elements. + + Multiple entries per digest are stored to enable fallback + resolution at instantiation time (SOURCE > ACTION > ARTIFACT). + + Args: + element: The element being processed + dependencies: List of dependency elements + """ + self._digest_cache.clear() + self._own_artifact_entries.clear() + + # Index element's own sources (highest priority) + self._index_element_sources(element, element) + + # Index dependency sources — enables SOURCE overlays for dep + # files (e.g. headers) that exist in both source and artifact. + # At instantiation, SOURCE is tried first; if the dep's sources + # aren't fetched (dep not rebuilding), ARTIFACT is used instead. + for dep in dependencies: + self._index_element_sources(dep, dep) + + # Index dependency artifacts + for dep in dependencies: + self._index_element_artifact(dep) + + # Index element's own artifact and collect entries for + # artifact_overlays generation (avoids re-traversal later) + self._index_element_artifact(element, collect_entries=True) + + def _index_element_sources(self, element, source_element): + """ + Index all file digests in an element's source tree. + + Args: + element: The element whose sources to index + source_element: The element to record as the source + """ + # Get the element's source directory + try: + # Check if element has any sources + if not any(element.sources()): + return + + # Access the private __sources attribute to get ElementSources + sources = element._Element__sources + if not sources or not sources.cached(): + return + + source_dir = sources.get_files() + if not source_dir: + return + + # Traverse the source directory and index all files with full paths + self._traverse_directory_with_paths( + source_dir._get_digest(), source_element.name, "SOURCE", "" # Start with empty path + ) + except Exception as e: + # Gracefully handle missing sources + pass + + def _index_element_artifact(self, element, collect_entries=False): + """ + Index all file digests in an element's artifact output. + + Args: + element: The element whose artifact to index + collect_entries: If True, also collect (digest_hash, digest_size) + tuples in self._own_artifact_entries for artifact_overlays + """ + try: + # Check if element is cached + if not element._cached(): + return + + # Get the artifact object + artifact = element._get_artifact() + if not artifact or not artifact.cached(): + return + + # Get the artifact files directory + files_dir = artifact.get_files() + if not files_dir: + return + + # Traverse the artifact files directory with full paths + self._traverse_directory_with_paths( + files_dir._get_digest(), element.name, "ARTIFACT", "", + collect_entries=collect_entries, + ) + except Exception as e: + # Gracefully handle missing artifacts + pass + + def _traverse_directory_with_paths(self, directory_digest, element_name, overlay_type, current_path, + collect_entries=False): + """ + Recursively traverse a Directory tree and index all file digests with full paths. + + Args: + directory_digest: The Directory digest to traverse + element_name: The element name to associate with found files + overlay_type: Either "SOURCE" or "ARTIFACT" + current_path: Current relative path from root (e.g., "src/foo") + collect_entries: If True, also append to self._own_artifact_entries + """ + try: + directory = self._cas.fetch_directory_proto(directory_digest) + if not directory: + return + + # Index all files in this directory with full paths + for file_node in directory.files: + digest_hash = file_node.digest.hash + # Build full relative path + file_path = file_node.name if not current_path else f"{current_path}/{file_node.name}" + + entry = (element_name, file_path, overlay_type) + if digest_hash not in self._digest_cache: + self._digest_cache[digest_hash] = [entry] + else: + # Avoid duplicate (same element, same path, same type) + if entry not in self._digest_cache[digest_hash]: + self._digest_cache[digest_hash].append(entry) + + if collect_entries: + self._own_artifact_entries.append( + (digest_hash, file_node.digest.size_bytes) + ) + + # Recursively traverse subdirectories + for dir_node in directory.directories: + # Build path for subdirectory + subdir_path = dir_node.name if not current_path else f"{current_path}/{dir_node.name}" + self._traverse_directory_with_paths( + dir_node.digest, element_name, overlay_type, subdir_path, + collect_entries=collect_entries, + ) + except Exception as e: + # Gracefully handle errors + pass + + def _generate_action_overlays(self, element, action_digest): + """ + Generate overlays for a single subaction. + + Args: + element: The element being processed + action_digest: The Action digest to generate overlays for + + Returns: + Tuple of (SpeculativeAction proto or None, input_digests set) + """ + from .._protos.buildstream.v2 import speculative_actions_pb2 + + # Fetch the action from CAS + action = self._cas.fetch_action(action_digest) + if not action: + return None, set() + + spec_action = speculative_actions_pb2.SpeculativeActions.SpeculativeAction() + spec_action.base_action_digest.CopyFrom(action_digest) + + # Extract all file digests from the action's input tree + input_digests = self._extract_digests_from_action(action) + + # Resolve each digest to overlays (may produce multiple per digest + # for fallback resolution: SOURCE > ACTION > ARTIFACT) + for digest in input_digests: + overlays = self._resolve_digest_to_overlays(digest, element) + spec_action.overlays.extend(overlays) + + return (spec_action if spec_action.overlays else None), input_digests + + def _extract_digests_from_action(self, action): + """ + Extract all unique file digests from an Action's input tree. + + Args: + action: Action proto + + Returns: + Set of file digests (as Digest protos) + """ + digests = set() + + if not action.HasField("input_root_digest"): + return digests + + # Traverse the input root directory tree + self._collect_file_digests(action.input_root_digest, digests) + + return digests + + def _collect_file_digests(self, directory_digest, digests_set): + """ + Recursively collect all file digests from a directory tree. + + Args: + directory_digest: Directory digest to traverse + digests_set: Set to add found digests to + """ + try: + directory = self._cas.fetch_directory_proto(directory_digest) + if not directory: + return + + # Collect file digests + for file_node in directory.files: + # Store the digest as a tuple (hash, size) for set uniqueness + digests_set.add((file_node.digest.hash, file_node.digest.size_bytes)) + + # Recursively traverse subdirectories + for dir_node in directory.directories: + self._collect_file_digests(dir_node.digest, digests_set) + except: + pass + + def _resolve_digest_to_overlays(self, digest_tuple, element): + """ + Resolve a file digest to Overlay protos. + + Returns multiple overlays when the same digest appears in both + source and artifact trees, enabling fallback resolution at + instantiation time (SOURCE tried first, then ARTIFACT). + + Args: + digest_tuple: Tuple of (hash, size_bytes) + element: The element being processed + + Returns: + List of Overlay protos (SOURCE first, then ARTIFACT), or empty list + """ + from .._protos.buildstream.v2 import speculative_actions_pb2 + + digest_hash = digest_tuple[0] + digest_size = digest_tuple[1] + + entries = self._digest_cache.get(digest_hash) + if not entries: + return [] + + overlays = [] + for element_name, file_path, overlay_type in entries: + overlay = speculative_actions_pb2.SpeculativeActions.Overlay() + overlay.target_digest.hash = digest_hash + overlay.target_digest.size_bytes = digest_size + overlay.source_path = file_path + + if overlay_type == "SOURCE": + overlay.type = speculative_actions_pb2.SpeculativeActions.Overlay.SOURCE + overlay.source_element = "" if element_name == element.name else element_name + elif overlay_type == "ARTIFACT": + overlay.type = speculative_actions_pb2.SpeculativeActions.Overlay.ARTIFACT + overlay.source_element = element_name + else: + continue + + overlays.append(overlay) + + # Sort: SOURCE first, then ARTIFACT — instantiator tries in order. + # ACTION overlays are added separately in generate_speculative_actions() + # and the final sort there establishes SOURCE > ACTION > ARTIFACT. + type_priority = { + speculative_actions_pb2.SpeculativeActions.Overlay.SOURCE: 0, + speculative_actions_pb2.SpeculativeActions.Overlay.ARTIFACT: 2, + } + overlays.sort(key=lambda o: type_priority.get(o.type, 99)) + + return overlays + + def _generate_artifact_overlays(self, element): + """ + Generate artifact_overlays for the element's output files. + + Uses _own_artifact_entries collected during _build_digest_cache + to avoid re-traversing the artifact tree. + + Args: + element: The element with the artifact + + Returns: + List of Overlay protos + """ + overlays = [] + for digest_hash, digest_size in self._own_artifact_entries: + resolved = self._resolve_digest_to_overlays( + (digest_hash, digest_size), element + ) + # For artifact_overlays, take the highest-priority overlay + if resolved: + overlays.append(resolved[0]) + return overlays diff --git a/src/buildstream/_speculative_actions/instantiator.py b/src/buildstream/_speculative_actions/instantiator.py new file mode 100644 index 000000000..085974db0 --- /dev/null +++ b/src/buildstream/_speculative_actions/instantiator.py @@ -0,0 +1,503 @@ +# +# Copyright 2025 The Apache Software Foundation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +SpeculativeActionInstantiator +============================== + +Instantiates SpeculativeActions by applying overlays. + +This module is responsible for: +1. Checking if the action is already instantiated (via global instantiated_actions) +2. Fetching base actions from CAS +3. Applying SOURCE, ACTION, and ARTIFACT overlays (in that priority order) +4. Replacing file digests in action input trees +5. Storing modified actions back to CAS +""" + + + + +class SpeculativeActionInstantiator: + """ + Instantiate SpeculativeActions by applying overlays. + + This class takes speculative actions and adapts them to current + dependency versions by replacing file digests according to overlays. + """ + + def __init__(self, cas, artifactcache, ac_service=None): + """ + Initialize the instantiator. + + Args: + cas: The CAS cache + artifactcache: The artifact cache + ac_service: Optional ActionCache service stub for resolving + cross-element ACTION overlays + """ + self._cas = cas + self._artifactcache = artifactcache + self._ac_service = ac_service + # Cache parsed Directory protos to avoid redundant CAS reads. + # Many overlays reference files in the same directory trees, + # so intermediate Directory protos are fetched repeatedly. + self._dir_cache = {} # type: dict[str, object] + + def instantiate_action(self, spec_action, element, element_lookup, + instantiated_actions=None, resolved_cache=None): + """ + Instantiate a SpeculativeAction by applying overlays. + + Previously resolved overlays can be passed in via resolved_cache + to avoid re-resolving overlays that succeeded on a prior pass but + whose SA couldn't be fully instantiated yet (e.g. an ACTION + overlay was deferred). + + Args: + spec_action: SpeculativeAction proto (may be mutated: overlays + removed by the priming queue) + element: Element being primed + element_lookup: Dict mapping element names to Element objects + instantiated_actions: Optional dict mapping base_action_hash -> adapted_action_digest + (global across all elements, populated by the priming queue) + resolved_cache: Optional dict of {target_digest_hash -> new_digest} + from prior passes, updated in-place with new resolutions + + Returns: + Digest of instantiated action, or None if overlays cannot be applied + """ + # Step 0: Check if already instantiated (e.g. by another element's priming) + base_hash = spec_action.base_action_digest.hash + if instantiated_actions is not None and base_hash in instantiated_actions: + return instantiated_actions[base_hash] + + # Fetch the base action + base_action = self._cas.fetch_action(spec_action.base_action_digest) + if not base_action: + return None + + # Get cached build dependency cache keys for optimization + # Skip overlays for dependencies that haven't changed + cached_dep_keys = self._get_cached_dependency_keys(element) + + # Start with a copy of the base action + from .._protos.build.bazel.remote.execution.v2 import remote_execution_pb2 + + action = remote_execution_pb2.Action() + action.CopyFrom(base_action) + + # Seed digest_replacements from the resolution cache (if provided). + # This avoids re-resolving overlays that succeeded on a prior + # pass but whose SA couldn't be fully instantiated yet. + if resolved_cache is None: + resolved_cache = {} + digest_replacements = dict(resolved_cache) + skipped_count = 0 + applied_count = 0 + + # Resolve overlays with fallback. Multiple overlays may target + # the same digest (e.g. SOURCE + ARTIFACT for the same dep file). + # They are stored in priority order (SOURCE first); once a target + # is resolved, subsequent overlays for it are skipped. + for overlay in spec_action.overlays: + # Skip if this target was already resolved (by a higher-priority + # overlay or from the resolution cache) + if overlay.target_digest.hash in digest_replacements: + continue + + # Optimization: Skip overlays for dependencies with unchanged cache keys + # (only applies to SOURCE/ARTIFACT overlays with a source_element) + if overlay.source_element and self._should_skip_overlay(overlay, element, cached_dep_keys): + skipped_count += 1 + continue + + replacement = self._resolve_overlay(overlay, element, element_lookup, instantiated_actions=instantiated_actions) + if replacement: + # replacement is (old_digest, new_digest) + digest_replacements[replacement[0].hash] = replacement[1] + applied_count += 1 + + # Update the resolution cache in-place for the next pass + resolved_cache.update(digest_replacements) + + # Check if any replacements actually change a digest + modified = any( + old_hash != new_digest.hash + for old_hash, new_digest in digest_replacements.items() + ) + + # Log optimization results + if skipped_count > 0: + element.info(f"Skipped {skipped_count} overlays (unchanged dependencies), applied {applied_count}") + + if not modified: + # No changes needed, return base action digest + return spec_action.base_action_digest + + # Apply digest replacements to the action's input tree + if action.HasField("input_root_digest"): + new_root_digest = self._replace_digests_in_tree(action.input_root_digest, digest_replacements) + if new_root_digest: + action.input_root_digest.CopyFrom(new_root_digest) + + # Store the modified action and return its digest + return self._cas.store_action(action) + + def _get_cached_dependency_keys(self, element): + """ + Get cache keys for build dependencies from the cached artifact. + + Args: + element: The element being primed + + Returns: + Dict mapping element_name -> cache_key from artifact.build_deps + """ + dep_keys = {} + + try: + artifact = element._get_artifact() + if not artifact or not artifact.cached(): + return dep_keys + + artifact_proto = artifact._get_proto() + if not artifact_proto: + return dep_keys + + # Extract cache keys from build_deps + for build_dep in artifact_proto.build_deps: + dep_keys[build_dep.element_name] = build_dep.cache_key + + except Exception: + # If we can't get the keys, just continue without optimization + pass + + return dep_keys + + def _should_skip_overlay(self, overlay, element, cached_dep_keys): + """ + Check if an overlay can be skipped because the dependency hasn't changed. + + Args: + overlay: Overlay proto + element: Element being primed + cached_dep_keys: Dict of element_name -> cache_key from cached artifact + + Returns: + bool: True if overlay can be skipped + """ + from .._protos.buildstream.v2 import speculative_actions_pb2 + + # Never skip ACTION overlays via this optimization — they use + # subaction indices, not element names + if overlay.type == speculative_actions_pb2.SpeculativeActions.Overlay.ACTION: + return False + + # Only skip for dependency overlays (source_element is not empty and not self) + if not overlay.source_element or overlay.source_element == element.name: + return False + + # Check if we have a cached key for this dependency + cached_key = cached_dep_keys.get(overlay.source_element) + if not cached_key: + return False + + # Get the current dependency element + from ..types import _Scope + + for dep in element._dependencies(_Scope.BUILD, recurse=False): + if dep.name == overlay.source_element: + current_key = dep._get_cache_key() + # Skip overlay if cache keys match (dependency unchanged) + if current_key == cached_key: + return True + break + + return False + + def _resolve_overlay(self, overlay, element, element_lookup, instantiated_actions=None): + """ + Resolve an overlay to get current file digest. + + Args: + overlay: Overlay proto + element: Current element + element_lookup: Dict mapping element names to Element objects + instantiated_actions: Optional dict mapping base_action_hash -> adapted_action_digest + + Returns: + Tuple of (old_digest, new_digest) or None + """ + from .._protos.buildstream.v2 import speculative_actions_pb2 + + if overlay.type == speculative_actions_pb2.SpeculativeActions.Overlay.SOURCE: + return self._resolve_source_overlay(overlay, element, element_lookup) + elif overlay.type == speculative_actions_pb2.SpeculativeActions.Overlay.ACTION: + return self._resolve_action_overlay(overlay, instantiated_actions) + elif overlay.type == speculative_actions_pb2.SpeculativeActions.Overlay.ARTIFACT: + return self._resolve_artifact_overlay(overlay, element, element_lookup) + + return None + + def _resolve_source_overlay(self, overlay, element, element_lookup): + """ + Resolve a SOURCE overlay to get current source file digest. + + Args: + overlay: Overlay proto + element: Current element + element_lookup: Dict mapping element names to Element objects + + Returns: + Tuple of (old_digest, new_digest) or None + """ + # Determine source element (empty = self) + if overlay.source_element == "": + source_element = element + else: + # Look up the source element by name + source_element = element_lookup.get(overlay.source_element) + if not source_element: + return None + + # Get current digest from source files + try: + # Check if element has any sources + if not any(source_element.sources()): + return None + + # Access the private __sources attribute + sources = source_element._Element__sources + if not sources or not sources.cached(): + return None + + source_dir = sources.get_files() + if not source_dir: + return None + + # Find the file in the source tree by full path + current_digest = self._find_file_by_path(source_dir._get_digest(), overlay.source_path) + + if current_digest: + return (overlay.target_digest, current_digest) + except Exception as e: + pass + + return None + + def _resolve_artifact_overlay(self, overlay, element, element_lookup): + """ + Resolve an ARTIFACT overlay to get current artifact file digest. + + Args: + overlay: Overlay proto + element: Current element + element_lookup: Dict mapping element names to Element objects + + Returns: + Tuple of (old_digest, new_digest) or None + """ + # Look up the artifact element + artifact_element = element_lookup.get(overlay.source_element) + if not artifact_element: + return None + + try: + # Check if element is cached + if not artifact_element._cached(): + return None + + # Get the artifact object + artifact = artifact_element._get_artifact() + if not artifact or not artifact.cached(): + return None + + # Get speculative actions to trace back to source + spec_actions = self._artifactcache.get_speculative_actions(artifact) + if spec_actions and spec_actions.artifact_overlays: + # Trace through artifact_overlays to find the ultimate source + for art_overlay in spec_actions.artifact_overlays: + if art_overlay.target_digest.hash == overlay.target_digest.hash: + # Found the mapping - now resolve the source overlay + return self._resolve_overlay(art_overlay, artifact_element, element_lookup) + + # Fallback: directly look up file in artifact + files_dir = artifact.get_files() + if not files_dir: + return None + + current_digest = self._find_file_by_path(files_dir._get_digest(), overlay.source_path) + + if current_digest: + return (overlay.target_digest, current_digest) + + except Exception as e: + pass + + return None + + def _resolve_action_overlay(self, overlay, instantiated_actions): + """ + Resolve an ACTION overlay using the global instantiated_actions map. + + Looks up the producing subaction's adapted digest in + instantiated_actions, then fetches the ActionResult from the + action cache to find the output file's current digest. + + Works for both intra-element and cross-element ACTION overlays, + since instantiated_actions is global across all elements. + + Args: + overlay: Overlay proto with type ACTION + instantiated_actions: Dict mapping base_action_hash -> adapted_action_digest + + Returns: + Tuple of (old_digest, new_digest) or None + """ + source_hash = overlay.source_action_digest.hash + + # Step 1: Look up the adapted digest for the producing action + adapted_digest = None + if instantiated_actions: + adapted_digest = instantiated_actions.get(source_hash) + + if adapted_digest is None: + # Producing action was never instantiated — drop this overlay + return None + + # Step 2: Fetch ActionResult using the adapted digest from AC + if self._ac_service: + try: + from .._protos.build.bazel.remote.execution.v2 import remote_execution_pb2 + + request = remote_execution_pb2.GetActionResultRequest( + action_digest=adapted_digest, + ) + action_result = self._ac_service.GetActionResult(request) + if action_result: + for output_file in action_result.output_files: + if output_file.path == overlay.source_path: + return (overlay.target_digest, output_file.digest) + except Exception: + pass + + return None + + def _cached_fetch_directory(self, digest): + """Fetch a Directory proto, using the in-memory cache.""" + cached = self._dir_cache.get(digest.hash) + if cached is not None: + return cached + directory = self._cas.fetch_directory_proto(digest) + if directory is not None: + self._dir_cache[digest.hash] = directory + return directory + + def _find_file_by_path(self, directory_digest, file_path): + """ + Find a file in a directory tree by full relative path. + + Args: + directory_digest: Directory to search + file_path: Full relative path (e.g., "src/foo/bar.c") + + Returns: + File digest or None + """ + try: + # Split path into components + if not file_path: + return None + + parts = file_path.split("/") + current_digest = directory_digest + + # Navigate through directories + for i, part in enumerate(parts[:-1]): # All but the last (filename) + directory = self._cached_fetch_directory(current_digest) + if not directory: + return None + + # Find the subdirectory + found = False + for dir_node in directory.directories: + if dir_node.name == part: + current_digest = dir_node.digest + found = True + break + + if not found: + return None + + # Now find the file + filename = parts[-1] + directory = self._cached_fetch_directory(current_digest) + if not directory: + return None + + for file_node in directory.files: + if file_node.name == filename: + return file_node.digest + + except Exception as e: + pass + + return None + + def _replace_digests_in_tree(self, directory_digest, replacements): + """ + Replace file digests in a directory tree. + + Args: + directory_digest: Root directory digest + replacements: Dict of old_hash -> new_digest + + Returns: + New directory digest or None + """ + try: + directory = self._cached_fetch_directory(directory_digest) + if not directory: + return None + + from .._protos.build.bazel.remote.execution.v2 import remote_execution_pb2 + + new_directory = remote_execution_pb2.Directory() + new_directory.CopyFrom(directory) + + modified = False + + # Replace file digests + for i, file_node in enumerate(new_directory.files): + if file_node.digest.hash in replacements: + new_directory.files[i].digest.CopyFrom(replacements[file_node.digest.hash]) + modified = True + + # Recursively process subdirectories + for i, dir_node in enumerate(new_directory.directories): + new_subdir_digest = self._replace_digests_in_tree(dir_node.digest, replacements) + if new_subdir_digest and new_subdir_digest.hash != dir_node.digest.hash: + new_directory.directories[i].digest.CopyFrom(new_subdir_digest) + modified = True + + if modified: + # Store the modified directory + return self._cas.store_directory_proto(new_directory) + else: + # No changes, return original + return directory_digest + except: + return None diff --git a/src/buildstream/_stream.py b/src/buildstream/_stream.py index a475bdb41..cba21ed84 100644 --- a/src/buildstream/_stream.py +++ b/src/buildstream/_stream.py @@ -41,13 +41,15 @@ BuildQueue, PullQueue, ArtifactPushQueue, + SpeculativeActionGenerationQueue, + SpeculativeCachePrimingQueue, ) from .element import Element from ._profile import Topics, PROFILER from ._project import ProjectRefStorage from ._remotespec import RemoteSpec from ._state import State -from .types import _KeyStrength, _PipelineSelection, _Scope, _HostMount +from .types import _KeyStrength, _PipelineSelection, _Scope, _HostMount, _SpeculativeActionMode from .plugin import Plugin from . import utils, node, _yaml, _site, _pipeline @@ -429,8 +431,21 @@ def build( self._add_queue(FetchQueue(self._scheduler, skip_cached=True)) + sa_mode = self._context.speculative_actions_mode + + if sa_mode != _SpeculativeActionMode.NONE: + # Priming queue: For each element, instantiate and submit its speculative + # actions to warm the remote ActionCache BEFORE the element reaches BuildQueue. + # Must come after FetchQueue so sources are available for resolving SOURCE overlays. + self._add_queue(SpeculativeCachePrimingQueue(self._scheduler)) + self._add_queue(BuildQueue(self._scheduler, imperative=True)) + if sa_mode not in (_SpeculativeActionMode.NONE, _SpeculativeActionMode.PRIME_ONLY): + # Generation queue: After each build, extract subactions and generate + # overlays so future builds can benefit from cache priming. + self._add_queue(SpeculativeActionGenerationQueue(self._scheduler)) + if self._artifacts.has_push_remotes(): self._add_queue(ArtifactPushQueue(self._scheduler, skip_uncached=True)) diff --git a/src/buildstream/data/userconfig.yaml b/src/buildstream/data/userconfig.yaml index 76af0d6c8..3155fe80a 100644 --- a/src/buildstream/data/userconfig.yaml +++ b/src/buildstream/data/userconfig.yaml @@ -74,6 +74,18 @@ scheduler: # on-error: quit + # Speculative actions mode for cache priming. + # Controls which overlay types are generated and whether priming is enabled. + # Modes (each includes capabilities of previous modes): + # none - Disabled entirely + # prime-only - Use existing SAs to prime, don't generate new ones + # source-artifact - Generate SOURCE and ARTIFACT overlays (no AC calls) + # intra-element - Also generate intra-element ACTION overlays + # full - Also generate cross-element ACTION overlays + # Also accepts True (= full) or False (= none) for backward compatibility. + # + speculative-actions: none + # # Build related configuration diff --git a/src/buildstream/element.py b/src/buildstream/element.py index aede8f1ea..b4dcd1434 100644 --- a/src/buildstream/element.py +++ b/src/buildstream/element.py @@ -90,7 +90,7 @@ from .sandbox import _SandboxFlags, SandboxCommandError from .sandbox._config import SandboxConfig from .sandbox._sandboxremote import SandboxRemote -from .types import _Scope, _CacheBuildTrees, _KeyStrength, OverlapAction, _DisplayKey +from .types import _Scope, _CacheBuildTrees, _KeyStrength, OverlapAction, _DisplayKey, _SpeculativeActionMode from ._artifact import Artifact from ._elementproxy import ElementProxy from ._elementsources import ElementSources @@ -300,6 +300,8 @@ def __init__( self.__required_callback = None # Callback to Queues self.__can_query_cache_callback = None # Callback to PullQueue/FetchQueue self.__buildable_callback = None # Callback to BuildQueue + self.__build_dep_cached_callback = None # Callback to PrimingQueue (per-dep, on cached) + self.__build_dep_primed_callback = None # Callback to PrimingQueue (per-dep, on primed) self.__resolved_initial_state = False # Whether the initial state of the Element has been resolved @@ -307,6 +309,9 @@ def __init__( self.__variables: Optional[Variables] = None self.__dynamic_public_guard = Lock() + # Speculative actions support + self.__subaction_digests = [] # Subaction digests from the build's ActionResult + if artifact_key: self.__initialize_from_artifact_key(artifact_key) else: @@ -1725,6 +1730,9 @@ def _assemble(self): collect = self.assemble(sandbox) # pylint: disable=assignment-from-no-return self.__set_build_result(success=True, description="succeeded") + + # Collect subaction digests recorded during the build + self._set_subaction_digests(sandbox._get_subaction_digests()) except (ElementError, SandboxCommandError) as e: # Shelling into a sandbox is useful to debug this error e.sandbox = True @@ -1803,6 +1811,43 @@ def _cache_artifact(self, sandbox, collect): "unable to collect artifact contents".format(collect) ) + # _set_subaction_digests(): + # + # Set the subaction digests captured from the build's ActionResult. + # This is called after a successful build to store compiler invocations. + # + # Args: + # subaction_digests: List of Digest protos from ActionResult.subactions + # + def _set_subaction_digests(self, subaction_digests): + self.__subaction_digests = list(subaction_digests) if subaction_digests else [] + + # _get_subaction_digests(): + # + # Get the subaction digests from the build's ActionResult. + # + # Returns: + # List of Digest protos, or empty list if none + # + def _get_subaction_digests(self): + return self.__subaction_digests + + # _get_weak_cache_key(): + # + # Get the weak cache key for this element. + # + # Used by speculative actions for stable lookup: the weak key includes + # everything about the element itself (sources, env, commands, sandbox) + # but only dependency *names* (not their cache keys), making it stable + # across dependency version changes while still changing when the + # element's own sources or configuration change. + # + # Returns: + # (str): The weak cache key, or None if not yet computed + # + def _get_weak_cache_key(self): + return self.__weak_cache_key + # _fetch_done() # # Indicates that fetching the sources for this element has been done. @@ -1919,6 +1964,21 @@ def _load_artifact(self, *, pull, strict=None): artifact._cached = False pulled = False + # For speculative actions: if the element is not cached (will need + # building), pull the weak key artifact proto so the priming queue + # can retrieve stored SpeculativeActions from a previous build. + # This is a lightweight pull — only the artifact proto metadata, + # not the full artifact files. The SA data itself and the base + # Actions will be fetched lazily by casd when needed. + if ( + pull + and not artifact.cached() + and context.speculative_actions_mode != _SpeculativeActionMode.NONE + and self.__weak_cache_key + and not self.__artifacts.contains(self, self.__weak_cache_key) + ): + self.__artifacts.pull_artifact_proto(self, self.__weak_cache_key) + self.__artifact = artifact return pulled elif self.__pull_pending: @@ -2431,6 +2491,53 @@ def _set_can_query_cache_callback(self, callback): def _set_buildable_callback(self, callback): self.__buildable_callback = callback + # _set_build_dep_cached_callback() + # + # Set a callback invoked each time a build dependency becomes cached. + # Unlike _set_buildable_callback (which fires only when ALL deps are + # cached), this fires incrementally — once per completed dep. + # + # Used by SpeculativeCachePrimingQueue for incremental overlay + # resolution: each completed dep may unlock ARTIFACT overlays or + # ACTION overlays whose producing subactions just finished. + # + # Args: + # callback (callable) - Called with (element, dep) where dep is + # the just-cached dependency + # + def _set_build_dep_cached_callback(self, callback): + self.__build_dep_cached_callback = callback + + # _set_build_dep_primed_callback() + # + # Set a callback invoked each time a build dependency finishes + # priming (its speculative actions have been instantiated and + # submitted to the action cache). + # + # Unlike _set_build_dep_cached_callback (which fires when a dep + # becomes cached after building), this fires earlier — when a + # dep's priming completes. This enables downstream elements to + # re-evaluate cross-element ACTION overlays sooner, since the + # dep's adapted action digests are now in instantiated_actions. + # + # Args: + # callback (callable) - Called with (element, dep) where dep is + # the just-primed dependency + # + def _set_build_dep_primed_callback(self, callback): + self.__build_dep_primed_callback = callback + + # _notify_build_deps_primed() + # + # Notify reverse build dependencies that this element has finished + # priming. Called by SpeculativeCachePrimingQueue.done() after an + # element's priming completes. + # + def _notify_build_deps_primed(self): + for rdep in self.__reverse_build_deps: + if rdep.__build_dep_primed_callback is not None: + rdep.__build_dep_primed_callback(rdep, self) + # _set_depth() # # Set the depth of the Element. @@ -2476,6 +2583,11 @@ def _update_ready_for_runtime_and_cached(self): rdep.__build_deps_uncached -= 1 assert not rdep.__build_deps_uncached < 0 + # Notify priming queue of each completed dep for + # incremental overlay resolution + if rdep.__build_dep_cached_callback is not None: + rdep.__build_dep_cached_callback(rdep, self) + if rdep._buildable(): rdep.__update_cache_key_non_strict() @@ -3338,6 +3450,7 @@ def __update_cache_keys(self): ] self.__weak_cache_key = self._calculate_cache_key(dependencies) + context = self._get_context() # Calculate the strict cache key diff --git a/src/buildstream/sandbox/_sandboxreapi.py b/src/buildstream/sandbox/_sandboxreapi.py index be210e450..0360301cd 100644 --- a/src/buildstream/sandbox/_sandboxreapi.py +++ b/src/buildstream/sandbox/_sandboxreapi.py @@ -105,6 +105,10 @@ def _run(self, command, *, flags, cwd, env): cwd, action_result.output_directories, action_result.output_files, failure=action_result.exit_code != 0 ) + # Collect subaction digests recorded during nested execution (if any) + if action_result.subactions: + self._collect_subaction_digests(action_result.subactions) + # Non-zero exit code means a normal error during the build: # the remote execution system has worked correctly but the command failed. return action_result.exit_code diff --git a/src/buildstream/sandbox/sandbox.py b/src/buildstream/sandbox/sandbox.py index e266a95c3..75e69e23b 100644 --- a/src/buildstream/sandbox/sandbox.py +++ b/src/buildstream/sandbox/sandbox.py @@ -86,6 +86,7 @@ def __init__(self, context: "Context", project: "Project", **kwargs): self.__mount_sources = {} # type: Dict[str, str] self.__allow_run = True self.__subsandboxes = [] # type: List[Sandbox] + self.__subaction_digests = [] # Subaction digests collected from ActionResults # Plugin element full name for logging plugin = kwargs.get("plugin", None) @@ -562,6 +563,29 @@ def _clean_directory(self, path): def _get_element_name(self): return self.__element_name + # _collect_subaction_digests() + # + # Store subaction digests from an ActionResult. + # + # Called by sandbox implementations after executing an action + # to collect subaction digests recorded by trexe. + # + # Args: + # subactions: Iterable of Digest protos from ActionResult.subactions + # + def _collect_subaction_digests(self, subactions): + self.__subaction_digests.extend(subactions) + + # _get_subaction_digests() + # + # Get subaction digests collected during sandbox execution. + # + # Returns: + # (list): List of Digest protos + # + def _get_subaction_digests(self): + return self.__subaction_digests + # _disable_run() # # Raise exception if `Sandbox.run()` is called. diff --git a/src/buildstream/types.py b/src/buildstream/types.py index 0cc2106b3..6ee4ec921 100644 --- a/src/buildstream/types.py +++ b/src/buildstream/types.py @@ -241,6 +241,30 @@ class _SchedulerErrorAction(FastEnum): TERMINATE = "terminate" +# _SpeculativeActionMode() +# +# Graduated modes for speculative actions, controlling which overlay +# types are generated and whether priming is enabled. Each mode +# includes all capabilities of the previous modes. +# +class _SpeculativeActionMode(FastEnum): + + # Speculative actions disabled entirely + NONE = "none" + + # Use existing SAs to prime the cache, but don't generate new ones + PRIME_ONLY = "prime-only" + + # Generate SOURCE and ARTIFACT overlays only (no AC calls during generation) + SOURCE_ARTIFACT = "source-artifact" + + # Also generate intra-element ACTION overlays (AC calls for own subactions) + INTRA_ELEMENT = "intra-element" + + # Full mode: also generate cross-element ACTION overlays + FULL = "full" + + # _CacheBuildTrees() # # When to cache build trees diff --git a/tests/integration/project/elements/speculative/README.md b/tests/integration/project/elements/speculative/README.md new file mode 100644 index 000000000..7ff73b6cd --- /dev/null +++ b/tests/integration/project/elements/speculative/README.md @@ -0,0 +1,97 @@ +# Speculative Actions Test Project + +This directory contains test elements for verifying the Speculative Actions PoC implementation. + +## Test Elements + +The test project consists of a 3-element dependency chain that uses trexe to record subactions: + +``` +trexe.bst (provides /usr/bin/trexe from buildbox) + ↓ +base.bst (depends on base.bst from project + trexe) + ↓ +middle.bst (depends on speculative/base.bst + trexe) + ↓ +top.bst (depends on speculative/middle.bst + trexe) +``` + +### Element Details + +- **trexe.bst**: Imports trexe binary from buildbox build directory +- **base.bst**: Uses `trexe -- cat` to process base.txt, recording file operations as subactions +- **middle.bst**: Uses trexe to combine files from sources and base dependency +- **top.bst**: Uses trexe to aggregate files from entire dependency chain (top + middle + base) + +**Key Feature**: Each element uses `trexe --input -- ` to wrap simple file operations (cat, echo, wc). Each trexe invocation records the operation as a subaction in the ActionResult with explicit input declarations. This is essential for the Speculative Actions PoC to have actual subactions to extract and process. + +**Why simple commands?**: Using `cat` and shell commands instead of compilation keeps the test fast and simple while still exercising the full subaction recording mechanism. The `--input` flags explicitly declare dependencies, which is what the PoC needs to trace. + +## Test Scenarios + +### 1. Basic Build (`test_speculative_actions_basic`) +- Builds the full chain from scratch +- Verifies all artifacts are created correctly +- Checks that speculative actions are generated and stored + +### 2. Rebuild with Source Change (`test_speculative_actions_rebuild_with_source_change`) +- Builds initial chain +- Modifies `top.txt` source file +- Rebuilds and verifies only necessary elements are rebuilt +- **Key test**: In future, speculative actions from middle.bst should help adapt cached artifacts + +### 3. Dependency Chain (`test_speculative_actions_dependency_chain`) +- Builds each element independently +- Verifies dependency relationships work correctly +- Confirms artifacts from dependencies are accessible + +## Manual Testing + +To manually test the project: + +```bash +# From buildstream root +cd /workspace/buildstream + +# Build the full chain +bst --directory tests/integration/project build speculative/top.bst + +# Check the artifact +bst --directory tests/integration/project artifact checkout speculative/top.bst --directory /tmp/checkout +ls -la /tmp/checkout + +# Modify source and rebuild +echo "Modified content" > tests/integration/project/files/speculative/top.txt +bst --directory tests/integration/project build speculative/top.bst +``` + +## Integration with Speculative Actions PoC + +The PoC implementation includes: + +1. **Generator** (`src/buildstream/_speculative_actions/generator.py`): + - Runs after BuildQueue + - Extracts subactions from ActionResult + - Traverses directory trees to identify file sources + - Creates overlay metadata + +2. **Instantiator** (`src/buildstream/_speculative_actions/instantiator.py`): + - Runs during cache priming (before BuildQueue) + - Reads speculative actions from artifacts + - Creates adapted actions with overlays applied + - Submits to Remote Execution + +3. **Queue Integration**: + - `SpeculativeActionGenerationQueue`: Generates actions after builds + - `SpeculativeCachePrimingQueue`: Instantiates and submits actions before builds + +## Future Enhancements + +Currently, the tests verify basic functionality. Future additions should verify: + +- [ ] Speculative actions are stored in artifact proto's `speculative_actions` field +- [ ] Actions can be retrieved from CAS using the stored digest +- [ ] Overlays correctly identify SOURCE vs ARTIFACT types +- [ ] Cache key optimization skips overlay generation for strong cache hits +- [ ] Weak cache hits benefit from speculative action reuse +- [ ] Remote execution successfully executes adapted actions diff --git a/tests/integration/project/elements/speculative/app-chained.bst b/tests/integration/project/elements/speculative/app-chained.bst new file mode 100644 index 000000000..63d197eaf --- /dev/null +++ b/tests/integration/project/elements/speculative/app-chained.bst @@ -0,0 +1,35 @@ +kind: autotools +description: | + Multi-file application for testing ACTION overlay chaining. + + Same as app.bst but also depends on slow-dep.bst. The slow + dependency keeps this element not-buildable for long enough that + fire-and-forget compile actions complete and their results appear + in the action cache. This allows subsequent priming passes to + resolve ACTION overlays on the link step. + +build-depends: +- filename: base/base-debian.bst + config: + digest-environment: RECC_REMOTE_PLATFORM_chrootRootDigest +- recc/recc.bst +- speculative/dep.bst +- speculative/slow-dep.bst + +sources: +- kind: tar + url: project_dir:/files/speculative/multifile.tar.gz + ref: 1242f38c2b92574bf851fcf51c83a50087debb953aa302763b4e72339a345ab5 + +sandbox: + remote-apis-socket: + path: /tmp/casd.sock + +environment: + CC: recc gcc + RECC_LOG_LEVEL: debug + RECC_LOG_DIRECTORY: .recc-log + RECC_DEPS_GLOBAL_PATHS: 1 + RECC_NO_PATH_REWRITE: 1 + RECC_LINK: 1 + RECC_SERVER: unix:/tmp/casd.sock diff --git a/tests/integration/project/elements/speculative/app.bst b/tests/integration/project/elements/speculative/app.bst new file mode 100644 index 000000000..cd45e4f69 --- /dev/null +++ b/tests/integration/project/elements/speculative/app.bst @@ -0,0 +1,40 @@ +kind: autotools +description: | + Multi-file application for speculative actions priming test. + + Compiles main.c (includes dep.h from dep element) and util.c + (only includes local common.h) through recc. This produces 3 + subactions: compile main.c, compile util.c, link. + + When dep.bst changes: + - main.c compile action needs instantiation (dep.h digest changed) + - util.c compile action stays stable (no dep files in input tree) + - link action needs instantiation (main.o changed) + + So priming should produce 2 cache hits (main.c + link adapted) and + 1 direct cache hit (util.c unchanged). + +build-depends: +- filename: base/base-debian.bst + config: + digest-environment: RECC_REMOTE_PLATFORM_chrootRootDigest +- recc/recc.bst +- speculative/dep.bst + +sources: +- kind: tar + url: project_dir:/files/speculative/multifile.tar.gz + ref: 1242f38c2b92574bf851fcf51c83a50087debb953aa302763b4e72339a345ab5 + +sandbox: + remote-apis-socket: + path: /tmp/casd.sock + +environment: + CC: recc gcc + RECC_LOG_LEVEL: debug + RECC_LOG_DIRECTORY: .recc-log + RECC_DEPS_GLOBAL_PATHS: 1 + RECC_NO_PATH_REWRITE: 1 + RECC_LINK: 1 + RECC_SERVER: unix:/tmp/casd.sock diff --git a/tests/integration/project/elements/speculative/base.bst b/tests/integration/project/elements/speculative/base.bst new file mode 100644 index 000000000..7d288b5e4 --- /dev/null +++ b/tests/integration/project/elements/speculative/base.bst @@ -0,0 +1,29 @@ +kind: autotools +description: | + Base element using recc for subaction recording. + Compiles amhello through recc via remote-apis-socket so each + compiler invocation is recorded as a subaction. + +build-depends: +- filename: base/base-debian.bst + config: + digest-environment: RECC_REMOTE_PLATFORM_chrootRootDigest +- recc/recc.bst + +sources: +- kind: tar + url: project_dir:/files/amhello.tar.gz + ref: 534a884bc1974ffc539a9c215e35c4217b6f666a134cd729e786b9c84af99650 + +sandbox: + remote-apis-socket: + path: /tmp/casd.sock + +environment: + CC: recc gcc + RECC_LOG_LEVEL: debug + RECC_LOG_DIRECTORY: .recc-log + RECC_DEPS_GLOBAL_PATHS: 1 + RECC_NO_PATH_REWRITE: 1 + RECC_LINK: 1 + RECC_SERVER: unix:/tmp/casd.sock diff --git a/tests/integration/project/elements/speculative/dep.bst b/tests/integration/project/elements/speculative/dep.bst new file mode 100644 index 000000000..a8332a99c --- /dev/null +++ b/tests/integration/project/elements/speculative/dep.bst @@ -0,0 +1,9 @@ +kind: import +description: | + Dependency element providing dep.h header. + Changing the header content triggers rebuilds of downstream + elements while their weak keys remain stable. + +sources: +- kind: local + path: files/speculative/dep-files diff --git a/tests/integration/project/elements/speculative/middle.bst b/tests/integration/project/elements/speculative/middle.bst new file mode 100644 index 000000000..442ca86ee --- /dev/null +++ b/tests/integration/project/elements/speculative/middle.bst @@ -0,0 +1,21 @@ +kind: manual +description: | + Middle element in the speculative actions dependency chain. + Depends on base (which uses recc for subaction recording). + +build-depends: +- base/base-debian.bst +- speculative/base.bst + +sources: +- kind: local + path: files/speculative + +config: + build-commands: + - | + test -f /usr/bin/hello && echo "base dependency available" + install-commands: + - | + mkdir -p %{install-root}/usr/share/speculative + cp middle.txt %{install-root}/usr/share/speculative/middle.txt diff --git a/tests/integration/project/elements/speculative/slow-dep.bst b/tests/integration/project/elements/speculative/slow-dep.bst new file mode 100644 index 000000000..33b01f8c0 --- /dev/null +++ b/tests/integration/project/elements/speculative/slow-dep.bst @@ -0,0 +1,19 @@ +kind: manual +description: | + Slow dependency that takes time to build. + Used to keep downstream elements not-buildable while fire-and-forget + priming actions complete, enabling ACTION overlay resolution. + +build-depends: +- filename: base/base-debian.bst + +sources: +- kind: local + path: files/speculative/slow-dep-files + +config: + install-commands: + - | + sleep 2 + mkdir -p %{install-root}/usr/lib/speculative + cp slow.txt %{install-root}/usr/lib/speculative/slow.txt diff --git a/tests/integration/project/elements/speculative/top.bst b/tests/integration/project/elements/speculative/top.bst new file mode 100644 index 000000000..0c6fd3c42 --- /dev/null +++ b/tests/integration/project/elements/speculative/top.bst @@ -0,0 +1,23 @@ +kind: manual +description: | + Top element in the speculative actions dependency chain. + Depends on middle → base. Verifies the full chain builds. + +build-depends: +- base/base-debian.bst +- speculative/middle.bst + +sources: +- kind: local + path: files/speculative + +config: + build-commands: + - | + test -f /usr/bin/hello && echo "base dependency available" + test -f /usr/share/speculative/middle.txt && echo "middle dependency available" + install-commands: + - | + mkdir -p %{install-root}/usr/share/speculative + cp top.txt %{install-root}/usr/share/speculative/top.txt + cp /usr/share/speculative/middle.txt %{install-root}/usr/share/speculative/from-middle.txt diff --git a/tests/integration/project/files/speculative/base.txt b/tests/integration/project/files/speculative/base.txt new file mode 100644 index 000000000..2dceab0e2 --- /dev/null +++ b/tests/integration/project/files/speculative/base.txt @@ -0,0 +1 @@ +This is the base file diff --git a/tests/integration/project/files/speculative/dep-files/usr/include/speculative/dep.h b/tests/integration/project/files/speculative/dep-files/usr/include/speculative/dep.h new file mode 100644 index 000000000..3e31f82a9 --- /dev/null +++ b/tests/integration/project/files/speculative/dep-files/usr/include/speculative/dep.h @@ -0,0 +1,4 @@ +#ifndef DEP_H +#define DEP_H +#define DEP_VERSION 1 +#endif diff --git a/tests/integration/project/files/speculative/dep.txt b/tests/integration/project/files/speculative/dep.txt new file mode 100644 index 000000000..360062981 --- /dev/null +++ b/tests/integration/project/files/speculative/dep.txt @@ -0,0 +1 @@ +dep version 1 diff --git a/tests/integration/project/files/speculative/middle.txt b/tests/integration/project/files/speculative/middle.txt new file mode 100644 index 000000000..f0fed4c61 --- /dev/null +++ b/tests/integration/project/files/speculative/middle.txt @@ -0,0 +1 @@ +This is the middle file diff --git a/tests/integration/project/files/speculative/multifile.tar.gz b/tests/integration/project/files/speculative/multifile.tar.gz new file mode 100644 index 000000000..5199dd972 Binary files /dev/null and b/tests/integration/project/files/speculative/multifile.tar.gz differ diff --git a/tests/integration/project/files/speculative/slow-dep-files/slow.txt b/tests/integration/project/files/speculative/slow-dep-files/slow.txt new file mode 100644 index 000000000..086e0352c --- /dev/null +++ b/tests/integration/project/files/speculative/slow-dep-files/slow.txt @@ -0,0 +1 @@ +slow dependency v1 diff --git a/tests/integration/project/files/speculative/top.txt b/tests/integration/project/files/speculative/top.txt new file mode 100644 index 000000000..47022bdcf --- /dev/null +++ b/tests/integration/project/files/speculative/top.txt @@ -0,0 +1 @@ +This is the top file version 1 diff --git a/tests/integration/speculative_actions.py b/tests/integration/speculative_actions.py new file mode 100644 index 000000000..c22432c9a --- /dev/null +++ b/tests/integration/speculative_actions.py @@ -0,0 +1,520 @@ +# +# Copyright 2025 The Apache Software Foundation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Pylint doesn't play well with fixtures and dependency injection from pytest +# pylint: disable=redefined-outer-name + +import io +import os +import re +import tarfile +import pytest + +from buildstream._testing import cli_integration as cli # pylint: disable=unused-import +from buildstream._testing.integration import assert_contains +from buildstream._testing._utils.site import HAVE_SANDBOX + + +pytestmark = pytest.mark.integration + + +DATA_DIR = os.path.join(os.path.dirname(os.path.realpath(__file__)), "project") + + +def _parse_queue_processed(output, queue_name): + """Parse 'processed N' count for a queue from the pipeline summary.""" + pattern = rf"{re.escape(queue_name)} Queue:\s+processed\s+(\d+)" + match = re.search(pattern, output) + if match: + return int(match.group(1)) + return None + + +# NOTE: Test ordering matters. The integration cache (including casd's action +# cache) is shared across all tests in this module. The generation test must +# run first to get a fresh casd without action cache hits from prior builds. +# Pytest runs tests in file order by default. + + +@pytest.mark.datafiles(DATA_DIR) +@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox") +def test_speculative_actions_generation(cli, datafiles): + """ + Build with speculative-actions enabled and verify: + 1. recc executed actions remotely (subactions recorded) + 2. The generation queue processed at least one element + 3. Artifact was produced correctly + + This test must run first in the module to avoid casd action cache + hits from prior builds that would prevent remote execution. + """ + project = str(datafiles) + element_name = "speculative/base.bst" + + cli.configure({"scheduler": {"speculative-actions": True}}) + + result = cli.run( + project=project, + args=["--cache-buildtrees", "always", "build", element_name], + ) + if result.exit_code != 0: + cli.run( + project=project, + args=[ + "shell", "--build", "--use-buildtree", element_name, + "--", "sh", "-c", + "cat config.log .recc-log/* */.recc-log/* 2>/dev/null", + ], + ) + assert result.exit_code == 0 + build_output = result.stderr + + # Verify recc executed remotely + result = cli.run( + project=project, + args=[ + "shell", "--build", "--use-buildtree", element_name, + "--", "sh", "-c", "cat src/.recc-log/recc.buildbox*", + ], + ) + assert result.exit_code == 0 + assert "Executing action remotely" in result.output, ( + "recc did not execute remotely — got action cache hits instead" + ) + + # Verify artifact + checkout = os.path.join(cli.directory, "checkout") + result = cli.run( + project=project, + args=["artifact", "checkout", element_name, "--directory", checkout], + ) + assert result.exit_code == 0 + assert_contains(checkout, ["/usr", "/usr/bin", "/usr/bin/hello"]) + + # Verify the generation queue processed at least one element + assert "Generating overlays Queue:" in build_output, ( + "Generation queue not in pipeline summary — " + "speculative-actions config not applied?" + ) + processed = _parse_queue_processed(build_output, "Generating overlays") + assert processed is not None, ( + "Could not parse generation queue stats from pipeline summary" + ) + assert processed > 0, ( + "Generation queue processed 0 elements — no subactions found" + ) + + +@pytest.mark.datafiles(DATA_DIR) +@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox") +def test_speculative_actions_dependency_chain(cli, datafiles): + """ + Build the full 3-element dependency chain: base -> middle -> top. + """ + project = str(datafiles) + element_name = "speculative/top.bst" + + result = cli.run( + project=project, + args=["--cache-buildtrees", "always", "build", element_name], + ) + assert result.exit_code == 0 + + checkout = os.path.join(cli.directory, "checkout") + result = cli.run( + project=project, + args=["artifact", "checkout", element_name, "--directory", checkout], + ) + assert result.exit_code == 0 + assert os.path.exists( + os.path.join(checkout, "usr", "share", "speculative", "top.txt") + ) + assert os.path.exists( + os.path.join(checkout, "usr", "share", "speculative", "from-middle.txt") + ) + + +@pytest.mark.datafiles(DATA_DIR) +@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox") +def test_speculative_actions_rebuild_with_source_change(cli, datafiles): + """ + Full speculative actions roundtrip: + 1. Build base element with recc (subactions recorded, overlays generated) + 2. Modify source (patch main.c in the amhello tarball) + 3. Rebuild and verify the modified source was picked up + 4. Verify generation queue runs on the rebuild (new subactions for + the changed source) + """ + project = str(datafiles) + element_name = "speculative/base.bst" + + cli.configure({"scheduler": {"speculative-actions": True}}) + + # --- First build --- + result = cli.run( + project=project, + args=["--cache-buildtrees", "always", "build", element_name], + ) + assert result.exit_code == 0 + + # --- Modify source: patch main.c in the amhello tarball --- + original_tar = os.path.join(project, "files", "amhello.tar.gz") + + members = {} + with tarfile.open(original_tar, "r:gz") as tf: + for member in tf.getmembers(): + if member.isfile(): + members[member.name] = (member, tf.extractfile(member).read()) + else: + members[member.name] = (member, None) + + main_c_name = "amhello/src/main.c" + member, content = members[main_c_name] + new_content = content.replace( + b'puts ("Hello World!");', + b'puts ("Hello Speculative World!");', + ) + assert new_content != content, "Source modification failed" + + with tarfile.open(original_tar, "w:gz") as tf: + for name, (m, data) in members.items(): + if data is not None: + if name == main_c_name: + data = new_content + m.size = len(data) + tf.addfile(m, io.BytesIO(data)) + else: + tf.addfile(m) + + # Delete cached artifact and re-track source + result = cli.run(project=project, args=["artifact", "delete", element_name]) + assert result.exit_code == 0 + result = cli.run(project=project, args=["source", "track", element_name]) + assert result.exit_code == 0 + + # --- Second build with modified source --- + result = cli.run( + project=project, + args=["--cache-buildtrees", "always", "build", element_name], + ) + assert result.exit_code == 0 + rebuild_output = result.stderr + + # Verify the rebuild produced a new artifact + checkout = os.path.join(cli.directory, "checkout-rebuild") + result = cli.run( + project=project, + args=["artifact", "checkout", element_name, "--directory", checkout], + ) + assert result.exit_code == 0 + assert os.path.exists(os.path.join(checkout, "usr", "bin", "hello")) + + # Verify the generation queue ran on the rebuild. + # The source changed so recc builds with different inputs → new Execute + # requests → new subactions recorded. + processed = _parse_queue_processed(rebuild_output, "Generating overlays") + if processed is not None: + assert processed > 0, ( + "Generation queue processed 0 on rebuild — " + "expected new subactions after source change" + ) + + +@pytest.mark.datafiles(DATA_DIR) +@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox") +def test_speculative_actions_priming(cli, datafiles): + """ + End-to-end priming test with partial cache hits. + + app.bst is a multi-file autotools project compiled through recc: + - main.c includes dep.h (from dep.bst) and common.h (local) + - util.c includes only common.h (local) + - link step combines main.o and util.o + + This produces 3 subactions: compile main.c, compile util.c, link. + + When dep.bst changes (dep.h updated): + - main.c compile: needs instantiation (dep.h digest changed) + - util.c compile: stays stable (no dep files in input tree) + - link: needs instantiation (main.o changed) + + So we expect: + - Priming queue processes app (finds SA by stable weak key) + - On rebuild, recc sees a mix of cache hits (from priming) and + possibly some direct hits (unchanged actions) + """ + project = str(datafiles) + app_element = "speculative/app.bst" + + cli.configure({"scheduler": {"speculative-actions": True}}) + + # --- First build: generate speculative actions for app --- + result = cli.run( + project=project, + args=["--cache-buildtrees", "always", "build", app_element], + ) + if result.exit_code != 0: + cli.run( + project=project, + args=[ + "shell", "--build", "--use-buildtree", app_element, + "--", "sh", "-c", + "cat config.log .recc-log/* */.recc-log/* 2>/dev/null", + ], + ) + assert result.exit_code == 0 + + # Verify SA generation and count remote executions + first_build_output = result.stderr + gen_processed = _parse_queue_processed(first_build_output, "Generating overlays") + assert gen_processed is not None and gen_processed > 0, ( + "First build did not generate speculative actions" + ) + + # Check first build recc log: should have remote executions + result = cli.run( + project=project, + args=[ + "shell", "--build", "--use-buildtree", app_element, + "--", "sh", "-c", "cat src/.recc-log/recc.buildbox*", + ], + ) + assert result.exit_code == 0 + first_recc_log = result.output + first_remote_execs = first_recc_log.count("Executing action remotely") + assert first_remote_execs >= 3, ( + f"Expected at least 3 remote executions (2 compiles + 1 link), " + f"got {first_remote_execs}" + ) + + # --- Modify dep: change dep.h header --- + dep_header = os.path.join( + project, "files", "speculative", "dep-files", + "usr", "include", "speculative", "dep.h", + ) + with open(dep_header, "w") as f: + f.write("#ifndef DEP_H\n#define DEP_H\n#define DEP_VERSION 2\n#endif\n") + + # --- Second build: priming + rebuild --- + result = cli.run( + project=project, + args=["--cache-buildtrees", "always", "build", app_element], + ) + assert result.exit_code == 0 + rebuild_output = result.stderr + + # Verify priming queue ran for app + primed = _parse_queue_processed(rebuild_output, "Priming cache") + assert primed is not None and primed > 0, ( + "Priming queue did not process app — SA not found by weak key?" + ) + + # Check rebuild recc log: should have cache hits from priming + result = cli.run( + project=project, + args=[ + "shell", "--build", "--use-buildtree", app_element, + "--", "sh", "-c", "cat src/.recc-log/recc.buildbox*", + ], + ) + assert result.exit_code == 0 + rebuild_recc_log = result.output + cache_hits = rebuild_recc_log.count("Action Cache hit") + remote_execs = rebuild_recc_log.count("Executing action remotely") + + print( + f"Priming result: {cache_hits} cache hits, " + f"{remote_execs} remote executions " + f"(first build had {first_remote_execs} remote executions)" + ) + + # With fire-and-forget, priming may still be in-flight when recc + # submits the same action. The RE system deduplicates — recc waits + # for the already-running execution rather than starting a new one. + # This shows up as "Executing action remotely" in the recc log, not + # a cache hit, but is still correct behavior. + # + # Verify that primed action digests match recc's by comparing the + # digests logged during priming with those in the recc buildbox log. + primed_digests = set( + re.findall(r"Submitted action ([0-9a-f]+)", rebuild_output) + ) + recc_digests = set( + re.findall(r"Action Digest: ([0-9a-f]+)/", rebuild_recc_log) + ) + # Truncate both to 8 chars for comparison + primed_short = {d[:8] for d in primed_digests} + recc_short = {d[:8] for d in recc_digests} + + matching = primed_short & recc_short + print( + f"Digest match: {len(matching)} of {len(primed_short)} primed " + f"actions found in recc's {len(recc_short)} actions" + ) + + # At least some primed digests should match recc's. + # Unmatched primed actions indicate overlays that didn't resolve + # correctly (e.g. ACTION overlays whose producers hadn't completed). + assert len(matching) > 0, ( + f"No primed action digests match recc's actions. " + f"Primed: {primed_short}, Recc: {recc_short}" + ) + + # The total should account for all actions + assert cache_hits + remote_execs >= first_remote_execs, ( + f"Expected at least {first_remote_execs} total actions " + f"(hits + execs), got {cache_hits + remote_execs}" + ) + + +@pytest.mark.datafiles(DATA_DIR) +@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox") +def test_speculative_actions_action_overlay_chaining(cli, datafiles): + """ + End-to-end test for ACTION overlay chaining with a slow dependency. + + app-chained.bst depends on dep.bst (fast, header change) and + slow-dep.bst (5s sleep). The slow dependency keeps app-chained + not-buildable while the priming queue re-enqueues: + + 1. First pass: compile actions submitted fire-and-forget, link + deferred (ACTION overlay unresolvable — compile not in AC yet) + 2. Re-enqueue passes: slow-dep still building, compile completes + in AC, ACTION overlay resolves, link submitted fire-and-forget + 3. slow-dep completes, app-chained becomes buildable, released + to build queue with all actions primed + + This demonstrates that the iterative priming + re-enqueue mechanism + correctly chains ACTION overlays across subactions. + """ + project = str(datafiles) + app_element = "speculative/app-chained.bst" + + cli.configure({"scheduler": {"speculative-actions": True}}) + + # --- First build: generate speculative actions --- + result = cli.run( + project=project, + args=["--cache-buildtrees", "always", "build", app_element], + ) + if result.exit_code != 0: + cli.run( + project=project, + args=[ + "shell", "--build", "--use-buildtree", app_element, + "--", "sh", "-c", + "cat config.log .recc-log/* */.recc-log/* 2>/dev/null", + ], + ) + assert result.exit_code == 0 + first_build_output = result.stderr + + gen_processed = _parse_queue_processed(first_build_output, "Generating overlays") + assert gen_processed is not None and gen_processed > 0, ( + "First build did not generate speculative actions" + ) + + # Count first build remote executions + result = cli.run( + project=project, + args=[ + "shell", "--build", "--use-buildtree", app_element, + "--", "sh", "-c", "cat src/.recc-log/recc.buildbox*", + ], + ) + assert result.exit_code == 0 + first_remote_execs = result.output.count("Executing action remotely") + assert first_remote_execs >= 3, ( + f"Expected at least 3 remote executions, got {first_remote_execs}" + ) + + # --- Modify dep: change dep.h header --- + dep_header = os.path.join( + project, "files", "speculative", "dep-files", + "usr", "include", "speculative", "dep.h", + ) + with open(dep_header, "w") as f: + f.write("#ifndef DEP_H\n#define DEP_H\n#define DEP_VERSION 2\n#endif\n") + + # Also modify slow-dep so it needs rebuilding on the second build. + # This keeps app-chained not-buildable while slow-dep rebuilds, + # giving background priming time to resolve ACTION overlays. + slow_dep_file = os.path.join( + project, "files", "speculative", "slow-dep-files", "slow.txt", + ) + with open(slow_dep_file, "w") as f: + f.write("slow dependency v2\n") + + # --- Second build: priming with slow dependency --- + result = cli.run( + project=project, + args=["--cache-buildtrees", "always", "build", app_element], + ) + assert result.exit_code == 0 + rebuild_output = result.stderr + + # Verify priming queue processed app-chained + primed = _parse_queue_processed(rebuild_output, "Priming cache") + assert primed is not None and primed > 0, ( + "Priming queue did not process app-chained" + ) + + # Extract primed action digests + primed_digests = set( + re.findall(r"Submitted action ([0-9a-f]+)", rebuild_output) + ) + + # Check rebuild recc log + result = cli.run( + project=project, + args=[ + "shell", "--build", "--use-buildtree", app_element, + "--", "sh", "-c", "cat src/.recc-log/recc.buildbox*", + ], + ) + assert result.exit_code == 0 + rebuild_recc_log = result.output + cache_hits = rebuild_recc_log.count("Action Cache hit") + remote_execs = rebuild_recc_log.count("Executing action remotely") + + # Extract recc action digests + recc_digests = set( + re.findall(r"Action Digest: ([0-9a-f]+)/", rebuild_recc_log) + ) + primed_short = {d[:8] for d in primed_digests} + recc_short = {d[:8] for d in recc_digests} + matching = primed_short & recc_short + + print( + f"Chained priming result: {cache_hits} cache hits, " + f"{remote_execs} remote executions " + f"(first build had {first_remote_execs} remote executions)" + ) + print( + f"Digest match: {len(matching)} of {len(primed_short)} primed " + f"actions found in recc's {len(recc_short)} actions" + ) + print(f" Primed: {sorted(primed_short)}") + print(f" Recc: {sorted(recc_short)}") + + # With slow-dep rebuilding (10s), app-chained enters priming as + # PENDING. Background priming submits the compile fire-and-forget. + # Per-dep callback on dep completion may resolve more overlays. + # Final pass when buildable resolves remaining ACTION overlays. + # We expect at least the compile action digest to match recc's. + assert len(matching) >= 1, ( + f"Expected at least 1 primed action to match recc's, " + f"got {len(matching)}. Primed: {primed_short}, Recc: {recc_short}" + ) diff --git a/tests/integration/verify_speculative_test.sh b/tests/integration/verify_speculative_test.sh new file mode 100755 index 000000000..b96c45b65 --- /dev/null +++ b/tests/integration/verify_speculative_test.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +# +# Manual verification script for Speculative Actions test project +# +# This script provides a quick way to verify the test project works +# without running the full pytest suite. + +set -e + +PROJECT_DIR="/workspace/buildstream/tests/integration/project" +CHECKOUT_DIR="/tmp/speculative-test-checkout" + +echo "=== Speculative Actions Manual Test ===" +echo "" + +# Check we're in the right place +if [ ! -d "$PROJECT_DIR" ]; then + echo "ERROR: Project directory not found: $PROJECT_DIR" + exit 1 +fi + +cd /workspace/buildstream + +echo "Step 1: Clean any existing artifacts..." +rm -rf ~/.cache/buildstream/artifacts/test || true +rm -rf "$CHECKOUT_DIR" || true + +echo "" +echo "Step 2: Show element info..." +bst --directory "$PROJECT_DIR" show speculative/top.bst + +echo "" +echo "Step 3: Build the full chain (base -> middle -> top)..." +bst --directory "$PROJECT_DIR" build speculative/top.bst + +echo "" +echo "Step 4: Checkout the artifact..." +bst --directory "$PROJECT_DIR" artifact checkout speculative/top.bst --directory "$CHECKOUT_DIR" + +echo "" +echo "Step 5: Verify artifact contents..." +echo "Files in checkout:" +ls -la "$CHECKOUT_DIR" + +echo "" +echo "Content of top.txt:" +cat "$CHECKOUT_DIR/top.txt" + +echo "" +echo "Content of from-middle.txt:" +cat "$CHECKOUT_DIR/from-middle.txt" + +echo "" +echo "Content of from-base.txt:" +cat "$CHECKOUT_DIR/from-base.txt" + +echo "" +echo "=== Test passed! ===" +echo "" +echo "Next steps:" +echo " 1. Modify tests/integration/project/files/speculative/top.txt" +echo " 2. Re-run: bst --directory $PROJECT_DIR build speculative/top.bst" +echo " 3. Verify only top.bst rebuilds (middle.bst and base.bst should be cached)" diff --git a/tests/speculative_actions/__init__.py b/tests/speculative_actions/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/speculative_actions/test_generator_unit.py b/tests/speculative_actions/test_generator_unit.py new file mode 100644 index 000000000..67d4b34b2 --- /dev/null +++ b/tests/speculative_actions/test_generator_unit.py @@ -0,0 +1,406 @@ +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Unit tests for SpeculativeActionsGenerator. + +These tests construct Action + Directory protos in-memory and verify +that the Generator correctly produces overlays. No sandbox needed. +""" + +import hashlib +import pytest + +from buildstream._protos.build.bazel.remote.execution.v2 import remote_execution_pb2 +from buildstream._protos.buildstream.v2 import speculative_actions_pb2 + + +def _make_digest(content): + """Create a Digest proto from content bytes.""" + digest = remote_execution_pb2.Digest() + digest.hash = hashlib.sha256(content).hexdigest() + digest.size_bytes = len(content) + return digest + + +class FakeCAS: + """In-memory CAS for testing without a real CAS daemon.""" + + def __init__(self): + self._blobs = {} # hash -> bytes + self._directories = {} # hash -> Directory proto + self._actions = {} # hash -> Action proto + + def store_directory_proto(self, directory): + data = directory.SerializeToString() + digest = _make_digest(data) + self._directories[digest.hash] = directory + self._blobs[digest.hash] = data + return digest + + def fetch_directory_proto(self, digest): + return self._directories.get(digest.hash) + + def store_action(self, action): + data = action.SerializeToString() + digest = _make_digest(data) + self._actions[digest.hash] = action + self._blobs[digest.hash] = data + return digest + + def fetch_action(self, digest): + return self._actions.get(digest.hash) + + def store_proto(self, proto): + data = proto.SerializeToString() + return _make_digest(data) + + def fetch_proto(self, digest, proto_class): + data = self._blobs.get(digest.hash) + if data is None: + return None + proto = proto_class() + proto.ParseFromString(data) + return proto + + +class FakeSourceDir: + """Fake source directory with a digest.""" + + def __init__(self, digest): + self._digest = digest + + def _get_digest(self): + return self._digest + + +class FakeSources: + """Fake ElementSources.""" + + def __init__(self, files_dir): + self._files_dir = files_dir + self._cached = True + + def cached(self): + return self._cached + + def get_files(self): + return self._files_dir + + +class FakeArtifact: + """Fake Artifact.""" + + def __init__(self, files_dir, is_cached=True): + self._files_dir = files_dir + self._cached = is_cached + + def cached(self): + return self._cached + + def get_files(self): + return self._files_dir + + +class FakeElement: + """Fake Element for testing Generator without a real Element.""" + + def __init__(self, name, sources=None, artifact=None): + self.name = name + self._Element__sources = sources + self._artifact = artifact + + def sources(self): + if self._Element__sources: + yield True # Just needs to be non-empty + + def _cached(self): + return self._artifact is not None and self._artifact.cached() + + def _get_artifact(self): + return self._artifact + + +def _build_source_tree(cas, files): + """Build a CAS directory tree from a dict of {path: content_bytes}. + + Args: + cas: FakeCAS instance + files: Dict mapping relative paths to content bytes + + Returns: + Digest of root directory + """ + # Group files by directory + dirs = {} + for path, content in files.items(): + parts = path.rsplit("/", 1) + if len(parts) == 1: + dirname, filename = "", parts[0] + else: + dirname, filename = parts + dirs.setdefault(dirname, []).append((filename, content)) + + # Build leaf directories first, then work up + dir_digests = {} + + # Sort paths by depth (deepest first) + all_dirs = set() + for path in files: + parts = path.split("/") + for i in range(len(parts) - 1): + all_dirs.add("/".join(parts[: i + 1])) + all_dirs.add("") # root + + # Process deepest directories first, root ("") always last + non_root = sorted((d for d in all_dirs if d), key=lambda d: -d.count("/")) + non_root.append("") + + for dirpath in non_root: + directory = remote_execution_pb2.Directory() + + # Add files in this directory + for filename, content in dirs.get(dirpath, []): + file_node = directory.files.add() + file_node.name = filename + file_node.digest.CopyFrom(_make_digest(content)) + + # Add subdirectories + for child_dir, child_digest in sorted(dir_digests.items()): + # Check if child_dir is a direct subdirectory of dirpath + if dirpath == "": + if "/" not in child_dir: + dir_node = directory.directories.add() + dir_node.name = child_dir + dir_node.digest.CopyFrom(child_digest) + else: + prefix = dirpath + "/" + if child_dir.startswith(prefix) and "/" not in child_dir[len(prefix) :]: + dir_node = directory.directories.add() + dir_node.name = child_dir[len(prefix) :] + dir_node.digest.CopyFrom(child_digest) + + digest = cas.store_directory_proto(directory) + dir_digests[dirpath] = digest + + return dir_digests[""] + + +def _build_action(cas, input_root_digest): + """Build an Action proto with the given input root.""" + action = remote_execution_pb2.Action() + action.input_root_digest.CopyFrom(input_root_digest) + return cas.store_action(action) + + +class TestGeneratorOverlayProduction: + """Test that Generator correctly produces overlays from subactions.""" + + def test_generates_source_overlays(self): + """Files found in element sources should produce SOURCE overlays.""" + from buildstream._speculative_actions.generator import SpeculativeActionsGenerator + + cas = FakeCAS() + + # Create source files + source_files = { + "main.c": b'int main() { return 0; }', + "util.h": b'#pragma once\nvoid util();', + } + source_root = _build_source_tree(cas, source_files) + sources = FakeSources(FakeSourceDir(source_root)) + + # Create an action that uses these source files in its input tree + action_input = _build_source_tree(cas, { + "src/main.c": b'int main() { return 0; }', + "src/util.h": b'#pragma once\nvoid util();', + }) + action_digest = _build_action(cas, action_input) + + element = FakeElement("test-element.bst", sources=sources) + generator = SpeculativeActionsGenerator(cas) + + spec_actions = generator.generate_speculative_actions(element, [action_digest], []) + + assert spec_actions is not None + assert len(spec_actions.actions) == 1 + + action = spec_actions.actions[0] + # Should have overlays for the source files found in the action input + assert len(action.overlays) > 0 + # All overlays should be SOURCE type + for overlay in action.overlays: + assert overlay.type == speculative_actions_pb2.SpeculativeActions.Overlay.SOURCE + + def test_generates_artifact_overlays_for_dependencies(self): + """Files from dependency artifacts should produce ARTIFACT overlays.""" + from buildstream._speculative_actions.generator import SpeculativeActionsGenerator + + cas = FakeCAS() + + # Create a dependency artifact with library files + dep_files = { + "lib/libfoo.so": b'fake-shared-object-content', + } + dep_root = _build_source_tree(cas, dep_files) + dep_artifact = FakeArtifact(FakeSourceDir(dep_root)) + dep_element = FakeElement("dep.bst", artifact=dep_artifact) + + # Create element sources (no overlap with dep) + source_files = { + "main.c": b'int main() { return 0; }', + } + source_root = _build_source_tree(cas, source_files) + sources = FakeSources(FakeSourceDir(source_root)) + + # Create an action that uses both source files and dep artifacts + action_input = _build_source_tree(cas, { + "src/main.c": b'int main() { return 0; }', + "lib/libfoo.so": b'fake-shared-object-content', + }) + action_digest = _build_action(cas, action_input) + + element = FakeElement("test-element.bst", sources=sources) + generator = SpeculativeActionsGenerator(cas) + + spec_actions = generator.generate_speculative_actions(element, [action_digest], [dep_element]) + + assert spec_actions is not None + assert len(spec_actions.actions) == 1 + + action = spec_actions.actions[0] + overlay_types = {o.type for o in action.overlays} + # Should have both SOURCE and ARTIFACT overlays + assert speculative_actions_pb2.SpeculativeActions.Overlay.SOURCE in overlay_types + assert speculative_actions_pb2.SpeculativeActions.Overlay.ARTIFACT in overlay_types + + def test_source_priority_over_artifact(self): + """When same digest exists in both source and artifact, both overlays + are generated with SOURCE first for fallback resolution.""" + from buildstream._speculative_actions.generator import SpeculativeActionsGenerator + + cas = FakeCAS() + + shared_content = b'shared-file-content' + shared_hash = _make_digest(shared_content).hash + + # Create element sources with the shared file + source_root = _build_source_tree(cas, { + "shared.h": shared_content, + }) + sources = FakeSources(FakeSourceDir(source_root)) + + # Create dependency artifact with the same file + dep_root = _build_source_tree(cas, { + "include/shared.h": shared_content, + }) + dep_artifact = FakeArtifact(FakeSourceDir(dep_root)) + dep_element = FakeElement("dep.bst", artifact=dep_artifact) + + # Action uses the shared file + action_input = _build_source_tree(cas, { + "shared.h": shared_content, + }) + action_digest = _build_action(cas, action_input) + + element = FakeElement("test-element.bst", sources=sources) + generator = SpeculativeActionsGenerator(cas) + + spec_actions = generator.generate_speculative_actions(element, [action_digest], [dep_element]) + + assert len(spec_actions.actions) == 1 + action = spec_actions.actions[0] + # Both SOURCE and ARTIFACT overlays should be generated for the + # same target digest, with SOURCE first for priority resolution + matching = [o for o in action.overlays if o.target_digest.hash == shared_hash] + assert len(matching) >= 2 + assert matching[0].type == speculative_actions_pb2.SpeculativeActions.Overlay.SOURCE + assert matching[1].type == speculative_actions_pb2.SpeculativeActions.Overlay.ARTIFACT + + def test_no_overlays_for_unknown_digests(self): + """Digests not found in sources or artifacts should produce no overlays.""" + from buildstream._speculative_actions.generator import SpeculativeActionsGenerator + + cas = FakeCAS() + + # Empty sources + source_root = _build_source_tree(cas, {}) + sources = FakeSources(FakeSourceDir(source_root)) + + # Action with files not in any source + action_input = _build_source_tree(cas, { + "unknown.bin": b'mystery-content', + }) + action_digest = _build_action(cas, action_input) + + element = FakeElement("test-element.bst", sources=sources) + generator = SpeculativeActionsGenerator(cas) + + spec_actions = generator.generate_speculative_actions(element, [action_digest], []) + + # No overlays should be generated (action with no overlays is excluded) + assert len(spec_actions.actions) == 0 + + def test_multiple_subactions(self): + """Multiple subaction digests should each produce a SpeculativeAction.""" + from buildstream._speculative_actions.generator import SpeculativeActionsGenerator + + cas = FakeCAS() + + source_files = { + "a.c": b'void a() {}', + "b.c": b'void b() {}', + } + source_root = _build_source_tree(cas, source_files) + sources = FakeSources(FakeSourceDir(source_root)) + + # Two separate actions + action1_input = _build_source_tree(cas, {"src/a.c": b'void a() {}'}) + action1_digest = _build_action(cas, action1_input) + + action2_input = _build_source_tree(cas, {"src/b.c": b'void b() {}'}) + action2_digest = _build_action(cas, action2_input) + + element = FakeElement("test-element.bst", sources=sources) + generator = SpeculativeActionsGenerator(cas) + + spec_actions = generator.generate_speculative_actions( + element, [action1_digest, action2_digest], [] + ) + + assert len(spec_actions.actions) == 2 + + def test_element_artifact_overlays_generated(self): + """artifact_overlays should be generated for cached element output.""" + from buildstream._speculative_actions.generator import SpeculativeActionsGenerator + + cas = FakeCAS() + + source_files = {"main.c": b'int main() { return 0; }'} + source_root = _build_source_tree(cas, source_files) + sources = FakeSources(FakeSourceDir(source_root)) + + # Element also has a cached artifact + artifact_files = {"bin/main": b'compiled-binary'} + artifact_root = _build_source_tree(cas, artifact_files) + artifact = FakeArtifact(FakeSourceDir(artifact_root)) + + element = FakeElement("test-element.bst", sources=sources, artifact=artifact) + + # No subactions, just check artifact_overlays + generator = SpeculativeActionsGenerator(cas) + spec_actions = generator.generate_speculative_actions(element, [], []) + + # No subaction overlays but artifact_overlays may be present + # (bin/main is not in source, so it won't be resolved) + assert spec_actions is not None diff --git a/tests/speculative_actions/test_instantiator_unit.py b/tests/speculative_actions/test_instantiator_unit.py new file mode 100644 index 000000000..9a60a2df8 --- /dev/null +++ b/tests/speculative_actions/test_instantiator_unit.py @@ -0,0 +1,434 @@ +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Unit tests for SpeculativeActionInstantiator. + +Given overlays and new file digests, verify correct digest replacements +in action input trees. No sandbox needed. +""" + +import hashlib +import pytest + +from buildstream._protos.build.bazel.remote.execution.v2 import remote_execution_pb2 +from buildstream._protos.buildstream.v2 import speculative_actions_pb2 + + +def _make_digest(content): + """Create a Digest proto from content bytes.""" + digest = remote_execution_pb2.Digest() + digest.hash = hashlib.sha256(content).hexdigest() + digest.size_bytes = len(content) + return digest + + +class FakeCAS: + """In-memory CAS for testing.""" + + def __init__(self): + self._blobs = {} + self._directories = {} + self._actions = {} + + def store_directory_proto(self, directory): + data = directory.SerializeToString() + digest = _make_digest(data) + self._directories[digest.hash] = directory + self._blobs[digest.hash] = data + return digest + + def fetch_directory_proto(self, digest): + return self._directories.get(digest.hash) + + def store_action(self, action): + data = action.SerializeToString() + digest = _make_digest(data) + self._actions[digest.hash] = action + self._blobs[digest.hash] = data + return digest + + def fetch_action(self, digest): + return self._actions.get(digest.hash) + + def store_proto(self, proto): + data = proto.SerializeToString() + return _make_digest(data) + + def fetch_proto(self, digest, proto_class): + data = self._blobs.get(digest.hash) + if data is None: + return None + proto = proto_class() + proto.ParseFromString(data) + return proto + + +class FakeSourceDir: + def __init__(self, digest): + self._digest = digest + + def _get_digest(self): + return self._digest + + +class FakeSources: + def __init__(self, files_dir): + self._files_dir = files_dir + self._cached = True + + def cached(self): + return self._cached + + def get_files(self): + return self._files_dir + + +class FakeArtifact: + def __init__(self, files_dir=None, is_cached=True, proto=None): + self._files_dir = files_dir + self._cached = is_cached + self._proto = proto + + def cached(self): + return self._cached + + def get_files(self): + return self._files_dir + + def _get_proto(self): + return self._proto + + +class FakeArtifactCache: + def __init__(self): + self._spec_actions = {} + + def get_speculative_actions(self, artifact, structural_key=None): + return self._spec_actions.get(id(artifact)) + + def store_speculative_actions(self, artifact, spec_actions, structural_key=None): + self._spec_actions[id(artifact)] = spec_actions + + +class FakeElement: + def __init__(self, name, sources=None, artifact=None, project_name="project"): + self.name = name + self.project_name = project_name + self._Element__sources = sources + self._artifact = artifact + + def sources(self): + if self._Element__sources: + yield True + + def _cached(self): + return self._artifact is not None and self._artifact.cached() + + def _get_artifact(self): + return self._artifact + + def _dependencies(self, scope, recurse=False): + return [] + + def _get_cache_key(self): + return "fake-cache-key" + + def info(self, msg): + pass + + def warn(self, msg): + pass + + +def _build_source_tree(cas, files): + """Build a CAS directory tree from a dict of {path: content_bytes}.""" + dirs = {} + for path, content in files.items(): + parts = path.rsplit("/", 1) + if len(parts) == 1: + dirname, filename = "", parts[0] + else: + dirname, filename = parts + dirs.setdefault(dirname, []).append((filename, content)) + + dir_digests = {} + all_dirs = set() + for path in files: + parts = path.split("/") + for i in range(len(parts) - 1): + all_dirs.add("/".join(parts[: i + 1])) + all_dirs.add("") + + # Process deepest directories first, root ("") always last + non_root = sorted((d for d in all_dirs if d), key=lambda d: -d.count("/")) + non_root.append("") + + for dirpath in non_root: + directory = remote_execution_pb2.Directory() + for filename, content in dirs.get(dirpath, []): + file_node = directory.files.add() + file_node.name = filename + file_node.digest.CopyFrom(_make_digest(content)) + + for child_dir, child_digest in sorted(dir_digests.items()): + if dirpath == "": + if "/" not in child_dir: + dir_node = directory.directories.add() + dir_node.name = child_dir + dir_node.digest.CopyFrom(child_digest) + else: + prefix = dirpath + "/" + if child_dir.startswith(prefix) and "/" not in child_dir[len(prefix) :]: + dir_node = directory.directories.add() + dir_node.name = child_dir[len(prefix) :] + dir_node.digest.CopyFrom(child_digest) + + digest = cas.store_directory_proto(directory) + dir_digests[dirpath] = digest + + return dir_digests[""] + + +class TestInstantiatorDigestReplacement: + """Test that Instantiator correctly replaces digests in action trees.""" + + def test_replaces_source_digest(self): + """SOURCE overlay should replace old digest with current source digest.""" + from buildstream._speculative_actions.instantiator import SpeculativeActionInstantiator + + cas = FakeCAS() + artifactcache = FakeArtifactCache() + + old_content = b'old source content' + new_content = b'new source content' + old_digest = _make_digest(old_content) + new_digest = _make_digest(new_content) + + # Build the original action input tree with old content + input_root = _build_source_tree(cas, {"main.c": old_content}) + action = remote_execution_pb2.Action() + action.input_root_digest.CopyFrom(input_root) + action_digest = cas.store_action(action) + + # Build current source tree with new content + new_source_root = _build_source_tree(cas, {"main.c": new_content}) + sources = FakeSources(FakeSourceDir(new_source_root)) + element = FakeElement("test.bst", sources=sources) + + # Create a SpeculativeAction with SOURCE overlay + spec_action = speculative_actions_pb2.SpeculativeActions.SpeculativeAction() + spec_action.base_action_digest.CopyFrom(action_digest) + overlay = spec_action.overlays.add() + overlay.type = speculative_actions_pb2.SpeculativeActions.Overlay.SOURCE + overlay.source_element = "" # self + overlay.source_path = "main.c" + overlay.target_digest.CopyFrom(old_digest) + + instantiator = SpeculativeActionInstantiator(cas, artifactcache) + result_digest = instantiator.instantiate_action(spec_action, element, {}) + + assert result_digest is not None + # The result should be a new action (different digest since content changed) + assert result_digest.hash != action_digest.hash + + # Verify the new action has the updated input tree + new_action = cas.fetch_action(result_digest) + assert new_action is not None + new_root = cas.fetch_directory_proto(new_action.input_root_digest) + assert new_root is not None + assert len(new_root.files) == 1 + assert new_root.files[0].digest.hash == new_digest.hash + + def test_unchanged_digest_returns_base(self): + """When no digests actually change, return the base action digest.""" + from buildstream._speculative_actions.instantiator import SpeculativeActionInstantiator + + cas = FakeCAS() + artifactcache = FakeArtifactCache() + + content = b'same content' + digest = _make_digest(content) + + input_root = _build_source_tree(cas, {"main.c": content}) + action = remote_execution_pb2.Action() + action.input_root_digest.CopyFrom(input_root) + action_digest = cas.store_action(action) + + # Sources have the same content + source_root = _build_source_tree(cas, {"main.c": content}) + sources = FakeSources(FakeSourceDir(source_root)) + element = FakeElement("test.bst", sources=sources) + + spec_action = speculative_actions_pb2.SpeculativeActions.SpeculativeAction() + spec_action.base_action_digest.CopyFrom(action_digest) + overlay = spec_action.overlays.add() + overlay.type = speculative_actions_pb2.SpeculativeActions.Overlay.SOURCE + overlay.source_element = "" + overlay.source_path = "main.c" + overlay.target_digest.CopyFrom(digest) + + instantiator = SpeculativeActionInstantiator(cas, artifactcache) + result_digest = instantiator.instantiate_action(spec_action, element, {}) + + # Should return the base action digest (no modifications) + assert result_digest.hash == action_digest.hash + + def test_missing_base_action_returns_none(self): + """If the base action can't be fetched, return None.""" + from buildstream._speculative_actions.instantiator import SpeculativeActionInstantiator + + cas = FakeCAS() + artifactcache = FakeArtifactCache() + + # Create a digest for a non-existent action + fake_digest = _make_digest(b'does-not-exist') + + spec_action = speculative_actions_pb2.SpeculativeActions.SpeculativeAction() + spec_action.base_action_digest.CopyFrom(fake_digest) + + element = FakeElement("test.bst") + instantiator = SpeculativeActionInstantiator(cas, artifactcache) + result = instantiator.instantiate_action(spec_action, element, {}) + + assert result is None + + def test_replaces_in_nested_directories(self): + """Digests in nested directory trees should be replaced.""" + from buildstream._speculative_actions.instantiator import SpeculativeActionInstantiator + + cas = FakeCAS() + artifactcache = FakeArtifactCache() + + old_content = b'old nested file' + new_content = b'new nested file' + old_digest = _make_digest(old_content) + new_digest = _make_digest(new_content) + + # Build nested input tree + input_root = _build_source_tree(cas, {"src/lib/util.c": old_content}) + action = remote_execution_pb2.Action() + action.input_root_digest.CopyFrom(input_root) + action_digest = cas.store_action(action) + + # New sources with updated content + source_root = _build_source_tree(cas, {"lib/util.c": new_content}) + sources = FakeSources(FakeSourceDir(source_root)) + element = FakeElement("test.bst", sources=sources) + + spec_action = speculative_actions_pb2.SpeculativeActions.SpeculativeAction() + spec_action.base_action_digest.CopyFrom(action_digest) + overlay = spec_action.overlays.add() + overlay.type = speculative_actions_pb2.SpeculativeActions.Overlay.SOURCE + overlay.source_element = "" + overlay.source_path = "lib/util.c" + overlay.target_digest.CopyFrom(old_digest) + + instantiator = SpeculativeActionInstantiator(cas, artifactcache) + result_digest = instantiator.instantiate_action(spec_action, element, {}) + + assert result_digest is not None + assert result_digest.hash != action_digest.hash + + def test_multiple_overlays_applied(self): + """Multiple overlays should all be applied to the same action.""" + from buildstream._speculative_actions.instantiator import SpeculativeActionInstantiator + + cas = FakeCAS() + artifactcache = FakeArtifactCache() + + old_a = b'old a.c' + old_b = b'old b.c' + new_a = b'new a.c' + new_b = b'new b.c' + + input_root = _build_source_tree(cas, {"a.c": old_a, "b.c": old_b}) + action = remote_execution_pb2.Action() + action.input_root_digest.CopyFrom(input_root) + action_digest = cas.store_action(action) + + source_root = _build_source_tree(cas, {"a.c": new_a, "b.c": new_b}) + sources = FakeSources(FakeSourceDir(source_root)) + element = FakeElement("test.bst", sources=sources) + + spec_action = speculative_actions_pb2.SpeculativeActions.SpeculativeAction() + spec_action.base_action_digest.CopyFrom(action_digest) + + overlay_a = spec_action.overlays.add() + overlay_a.type = speculative_actions_pb2.SpeculativeActions.Overlay.SOURCE + overlay_a.source_element = "" + overlay_a.source_path = "a.c" + overlay_a.target_digest.CopyFrom(_make_digest(old_a)) + + overlay_b = spec_action.overlays.add() + overlay_b.type = speculative_actions_pb2.SpeculativeActions.Overlay.SOURCE + overlay_b.source_element = "" + overlay_b.source_path = "b.c" + overlay_b.target_digest.CopyFrom(_make_digest(old_b)) + + instantiator = SpeculativeActionInstantiator(cas, artifactcache) + result_digest = instantiator.instantiate_action(spec_action, element, {}) + + assert result_digest is not None + assert result_digest.hash != action_digest.hash + + # Verify both files were replaced + new_action = cas.fetch_action(result_digest) + new_root = cas.fetch_directory_proto(new_action.input_root_digest) + file_hashes = {f.name: f.digest.hash for f in new_root.files} + assert file_hashes["a.c"] == _make_digest(new_a).hash + assert file_hashes["b.c"] == _make_digest(new_b).hash + + +class TestInstantiatorArtifactOverlay: + """Test ARTIFACT overlay resolution.""" + + def test_resolves_artifact_overlay_from_dep(self): + """ARTIFACT overlay should resolve file digest from dependency artifact.""" + from buildstream._speculative_actions.instantiator import SpeculativeActionInstantiator + + cas = FakeCAS() + artifactcache = FakeArtifactCache() + + old_lib = b'old-lib-content' + new_lib = b'new-lib-content' + old_digest = _make_digest(old_lib) + new_digest = _make_digest(new_lib) + + # Build original action + input_root = _build_source_tree(cas, {"lib/libfoo.so": old_lib}) + action = remote_execution_pb2.Action() + action.input_root_digest.CopyFrom(input_root) + action_digest = cas.store_action(action) + + # Dependency element with updated artifact + dep_artifact_root = _build_source_tree(cas, {"lib/libfoo.so": new_lib}) + dep_artifact = FakeArtifact(FakeSourceDir(dep_artifact_root)) + dep_element = FakeElement("dep.bst", artifact=dep_artifact) + + element = FakeElement("test.bst") + element_lookup = {"dep.bst": dep_element} + + spec_action = speculative_actions_pb2.SpeculativeActions.SpeculativeAction() + spec_action.base_action_digest.CopyFrom(action_digest) + overlay = spec_action.overlays.add() + overlay.type = speculative_actions_pb2.SpeculativeActions.Overlay.ARTIFACT + overlay.source_element = "dep.bst" + overlay.source_path = "lib/libfoo.so" + overlay.target_digest.CopyFrom(old_digest) + + instantiator = SpeculativeActionInstantiator(cas, artifactcache) + result_digest = instantiator.instantiate_action(spec_action, element, element_lookup) + + assert result_digest is not None + assert result_digest.hash != action_digest.hash diff --git a/tests/speculative_actions/test_pipeline_integration.py b/tests/speculative_actions/test_pipeline_integration.py new file mode 100644 index 000000000..4869a78ef --- /dev/null +++ b/tests/speculative_actions/test_pipeline_integration.py @@ -0,0 +1,1595 @@ +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Pipeline integration tests for speculative actions. + +These tests exercise the full generate → store → retrieve → instantiate +pipeline using in-memory fakes for CAS and artifact cache. No sandbox +or real trexe binary needed — subaction digests are constructed directly +from proto objects. + +The scenario modeled: + 1. "Build" an element by constructing Action protos with known input trees + 2. Run the Generator to produce SpeculativeActions from those subactions + 3. Store SpeculativeActions via the artifact cache (weak key path) + 4. Simulate a source change (new file content) + 5. Retrieve SpeculativeActions and run the Instantiator + 6. Verify the instantiated action has the updated file digests +""" + +import hashlib +import os +import tempfile +import pytest + +from buildstream._protos.build.bazel.remote.execution.v2 import remote_execution_pb2 +from buildstream._protos.buildstream.v2 import speculative_actions_pb2 +from buildstream._speculative_actions.generator import SpeculativeActionsGenerator +from buildstream._speculative_actions.instantiator import SpeculativeActionInstantiator + + +# --------------------------------------------------------------------------- +# Shared test helpers +# --------------------------------------------------------------------------- + +def _make_digest(content): + """Create a Digest proto from content bytes.""" + digest = remote_execution_pb2.Digest() + digest.hash = hashlib.sha256(content).hexdigest() + digest.size_bytes = len(content) + return digest + + +class FakeCAS: + """In-memory CAS that supports the operations used by generator and instantiator.""" + + def __init__(self): + self._blobs = {} # hash -> bytes + self._directories = {} # hash -> Directory proto + self._actions = {} # hash -> Action proto + + def store_directory_proto(self, directory): + data = directory.SerializeToString() + digest = _make_digest(data) + self._directories[digest.hash] = directory + self._blobs[digest.hash] = data + return digest + + def fetch_directory_proto(self, digest): + return self._directories.get(digest.hash) + + def store_action(self, action): + data = action.SerializeToString() + digest = _make_digest(data) + self._actions[digest.hash] = action + self._blobs[digest.hash] = data + return digest + + def fetch_action(self, digest): + return self._actions.get(digest.hash) + + def store_proto(self, proto): + data = proto.SerializeToString() + digest = _make_digest(data) + self._blobs[digest.hash] = data + return digest + + def fetch_proto(self, digest, proto_class): + data = self._blobs.get(digest.hash) + if data is None: + return None + proto = proto_class() + proto.ParseFromString(data) + return proto + + +class FakeSourceDir: + def __init__(self, digest): + self._digest = digest + + def _get_digest(self): + return self._digest + + +class FakeSources: + def __init__(self, files_dir): + self._files_dir = files_dir + self._cached = True + + def cached(self): + return self._cached + + def get_files(self): + return self._files_dir + + +class FakeArtifactProto: + """Minimal artifact proto supporting HasField and speculative_actions.""" + + def __init__(self): + self._speculative_actions = None + self.build_deps = [] + + def HasField(self, name): + if name == "speculative_actions": + return self._speculative_actions is not None + return False + + @property + def speculative_actions(self): + return self._speculative_actions + + @speculative_actions.setter + def speculative_actions(self, value): + self._speculative_actions = value + + +class FakeArtifact: + def __init__(self, files_dir=None, is_cached=True, element=None): + self._files_dir = files_dir + self._cached = is_cached + self._element = element + + def cached(self): + return self._cached + + def get_files(self): + return self._files_dir + + def _get_proto(self): + return None + + def get_extract_key(self): + return "extract-key" + + +class FakeProject: + def __init__(self, name="test-project"): + self.name = name + + +class FakeElement: + def __init__(self, name, sources=None, artifact=None, project_name="project"): + self.name = name + self.project_name = project_name + self._Element__sources = sources + self._artifact = artifact + self._project = FakeProject() + + def sources(self): + if self._Element__sources: + yield True + + def _cached(self): + return self._artifact is not None and self._artifact.cached() + + def _get_artifact(self): + return self._artifact + + def _get_project(self): + return self._project + + def _dependencies(self, scope, recurse=False): + return [] + + def _get_cache_key(self): + return "fake-cache-key" + + def get_artifact_name(self, key): + return "{}/{}/{}".format(self._project.name, self.name, key) + + def info(self, msg): + pass + + def warn(self, msg): + pass + + +def _build_source_tree(cas, files): + """Build a CAS directory tree from {path: content_bytes}, return root Digest.""" + dirs = {} + for path, content in files.items(): + parts = path.rsplit("/", 1) + if len(parts) == 1: + dirname, filename = "", parts[0] + else: + dirname, filename = parts + dirs.setdefault(dirname, []).append((filename, content)) + + dir_digests = {} + all_dirs = set() + for path in files: + parts = path.split("/") + for i in range(len(parts) - 1): + all_dirs.add("/".join(parts[: i + 1])) + all_dirs.add("") + + non_root = sorted((d for d in all_dirs if d), key=lambda d: -d.count("/")) + non_root.append("") + + for dirpath in non_root: + directory = remote_execution_pb2.Directory() + for filename, content in dirs.get(dirpath, []): + file_node = directory.files.add() + file_node.name = filename + file_node.digest.CopyFrom(_make_digest(content)) + + for child_dir, child_digest in sorted(dir_digests.items()): + if dirpath == "": + if "/" not in child_dir: + dir_node = directory.directories.add() + dir_node.name = child_dir + dir_node.digest.CopyFrom(child_digest) + else: + prefix = dirpath + "/" + if child_dir.startswith(prefix) and "/" not in child_dir[len(prefix):]: + dir_node = directory.directories.add() + dir_node.name = child_dir[len(prefix):] + dir_node.digest.CopyFrom(child_digest) + + digest = cas.store_directory_proto(directory) + dir_digests[dirpath] = digest + + return dir_digests[""] + + +def _build_action(cas, input_root_digest): + """Build an Action proto with the given input root, store it, return Digest.""" + action = remote_execution_pb2.Action() + action.input_root_digest.CopyFrom(input_root_digest) + return cas.store_action(action) + + +class FakeArtifactCache: + """Artifact cache backed by a temp directory, using real file paths like the production code.""" + + def __init__(self, cas, basedir): + self.cas = cas + self._basedir = basedir + self._by_artifact = {} # id(artifact) -> SpeculativeActions + + def store_speculative_actions(self, artifact, spec_actions, weak_key=None): + # Store proto in CAS + spec_actions_digest = self.cas.store_proto(spec_actions) + + # Store by artifact identity (for get_speculative_actions without weak_key) + self._by_artifact[id(artifact)] = spec_actions + + # Store weak key reference + if weak_key: + element = artifact._element + project = element._get_project() + sa_ref = "{}/{}/speculative-{}".format(project.name, element.name, weak_key) + sa_ref_path = os.path.join(self._basedir, sa_ref) + os.makedirs(os.path.dirname(sa_ref_path), exist_ok=True) + with open(sa_ref_path, mode="w+b") as f: + f.write(spec_actions.SerializeToString()) + + def get_speculative_actions(self, artifact, weak_key=None): + if weak_key is not None: + if not weak_key: + return None + element = artifact._element + project = element._get_project() + sa_ref = "{}/{}/speculative-{}".format(project.name, element.name, weak_key) + sa_ref_path = os.path.join(self._basedir, sa_ref) + if os.path.exists(sa_ref_path): + spec_actions = speculative_actions_pb2.SpeculativeActions() + with open(sa_ref_path, mode="r+b") as f: + spec_actions.ParseFromString(f.read()) + return spec_actions + return None + + # No weak_key provided: lookup by artifact identity + # (used by _seed_dependency_outputs which passes just the artifact) + return self._by_artifact.get(id(artifact)) + + +# --------------------------------------------------------------------------- +# Tests +# --------------------------------------------------------------------------- + +class TestGenerateStoreRetrieveInstantiate: + """Full pipeline: generate overlays, store, retrieve, instantiate with changed sources.""" + + def test_source_change_roundtrip(self, tmp_path): + """ + Scenario: element has source file main.c. A build records a subaction + whose input tree contains main.c. After the build, we generate and + store SpeculativeActions. Later, main.c changes. We retrieve the + stored SA and instantiate — the action's input tree should now + reference the new main.c digest. + """ + cas = FakeCAS() + artifactcache = FakeArtifactCache(cas, str(tmp_path)) + + # --- Build phase (v1) --- + v1_content = b'int main() { return 0; }' + v1_digest = _make_digest(v1_content) + + # Element sources contain main.c v1 + source_root_v1 = _build_source_tree(cas, {"main.c": v1_content}) + sources_v1 = FakeSources(FakeSourceDir(source_root_v1)) + + element = FakeElement("app.bst", sources=sources_v1) + + # The build produced a subaction whose input tree includes main.c + subaction_input = _build_source_tree(cas, {"main.c": v1_content}) + subaction_digest = _build_action(cas, subaction_input) + + # --- Generate phase --- + generator = SpeculativeActionsGenerator(cas) + spec_actions = generator.generate_speculative_actions(element, [subaction_digest], []) + + assert len(spec_actions.actions) == 1 + assert len(spec_actions.actions[0].overlays) == 1 + overlay = spec_actions.actions[0].overlays[0] + assert overlay.type == speculative_actions_pb2.SpeculativeActions.Overlay.SOURCE + assert overlay.source_path == "main.c" + assert overlay.target_digest.hash == v1_digest.hash + + # --- Store phase --- + weak_key = "fake-weak-key-v1" + artifact = FakeArtifact(element=element) + artifactcache.store_speculative_actions(artifact, spec_actions, weak_key=weak_key) + + # --- Source change (v2) --- + v2_content = b'int main() { return 42; }' + v2_digest = _make_digest(v2_content) + source_root_v2 = _build_source_tree(cas, {"main.c": v2_content}) + sources_v2 = FakeSources(FakeSourceDir(source_root_v2)) + + # New element state with updated sources (same weak key because + # in real life, the weak key for downstream elements is stable + # across dependency version changes — here we're the leaf element + # whose source changed, so in practice this SA would be stored + # under a *different* weak key. But the retrieve+instantiate + # logic is the same.) + element_v2 = FakeElement("app.bst", sources=sources_v2) + artifact_v2 = FakeArtifact(element=element_v2) + + # --- Retrieve phase --- + retrieved = artifactcache.get_speculative_actions(artifact_v2, weak_key=weak_key) + assert retrieved is not None + assert len(retrieved.actions) == 1 + + # --- Instantiate phase --- + instantiator = SpeculativeActionInstantiator(cas, artifactcache) + result_digest = instantiator.instantiate_action(retrieved.actions[0], element_v2, {}) + + assert result_digest is not None + # The action should be different (new input digest) + assert result_digest.hash != subaction_digest.hash + + # Verify the new action's input tree has main.c with v2 content digest + new_action = cas.fetch_action(result_digest) + assert new_action is not None + new_root = cas.fetch_directory_proto(new_action.input_root_digest) + assert new_root is not None + assert len(new_root.files) == 1 + assert new_root.files[0].name == "main.c" + assert new_root.files[0].digest.hash == v2_digest.hash + + def test_dependency_artifact_change_roundtrip(self, tmp_path): + """ + Scenario: element depends on dep.bst whose artifact provides libfoo.so. + A build records a subaction using both main.c (source) and libfoo.so + (from dep). After storing SA, dep.bst is rebuilt with new libfoo.so. + Instantiation should produce an action with the new libfoo.so digest. + """ + cas = FakeCAS() + artifactcache = FakeArtifactCache(cas, str(tmp_path)) + + # --- Build phase (v1) --- + src_content = b'#include "foo.h"\nint main() { foo(); }' + lib_v1 = b'libfoo-v1-content' + lib_v1_digest = _make_digest(lib_v1) + + # Element sources + source_root = _build_source_tree(cas, {"main.c": src_content}) + sources = FakeSources(FakeSourceDir(source_root)) + + # Dependency artifact with libfoo.so v1 + dep_artifact_root_v1 = _build_source_tree(cas, {"lib/libfoo.so": lib_v1}) + dep_artifact_v1 = FakeArtifact(FakeSourceDir(dep_artifact_root_v1)) + dep_element_v1 = FakeElement("dep.bst", artifact=dep_artifact_v1) + + element = FakeElement("app.bst", sources=sources) + + # Subaction input tree has both source file and dep library + subaction_input = _build_source_tree(cas, { + "main.c": src_content, + "lib/libfoo.so": lib_v1, + }) + subaction_digest = _build_action(cas, subaction_input) + + # --- Generate --- + generator = SpeculativeActionsGenerator(cas) + spec_actions = generator.generate_speculative_actions( + element, [subaction_digest], [dep_element_v1] + ) + + assert len(spec_actions.actions) == 1 + overlays = spec_actions.actions[0].overlays + overlay_types = {o.type for o in overlays} + assert speculative_actions_pb2.SpeculativeActions.Overlay.SOURCE in overlay_types + assert speculative_actions_pb2.SpeculativeActions.Overlay.ARTIFACT in overlay_types + + # --- Store --- + weak_key = "fake-weak-key-app" + artifact = FakeArtifact(element=element) + artifactcache.store_speculative_actions(artifact, spec_actions, weak_key=weak_key) + + # --- Dependency change (v2) --- + lib_v2 = b'libfoo-v2-content' + lib_v2_digest = _make_digest(lib_v2) + dep_artifact_root_v2 = _build_source_tree(cas, {"lib/libfoo.so": lib_v2}) + dep_artifact_v2 = FakeArtifact(FakeSourceDir(dep_artifact_root_v2)) + dep_element_v2 = FakeElement("dep.bst", artifact=dep_artifact_v2) + + # Element sources unchanged + element_v2 = FakeElement("app.bst", sources=sources) + artifact_v2 = FakeArtifact(element=element_v2) + + # --- Retrieve --- + retrieved = artifactcache.get_speculative_actions(artifact_v2, weak_key=weak_key) + assert retrieved is not None + + # --- Instantiate --- + element_lookup = {"dep.bst": dep_element_v2} + instantiator = SpeculativeActionInstantiator(cas, artifactcache) + result_digest = instantiator.instantiate_action( + retrieved.actions[0], element_v2, element_lookup + ) + + assert result_digest is not None + assert result_digest.hash != subaction_digest.hash + + # Verify: main.c unchanged, libfoo.so updated to v2 + new_action = cas.fetch_action(result_digest) + new_root = cas.fetch_directory_proto(new_action.input_root_digest) + + # Collect all files recursively + all_files = {} + self._collect_files(cas, new_root, "", all_files) + + assert all_files["main.c"] == _make_digest(src_content).hash + assert all_files["lib/libfoo.so"] == lib_v2_digest.hash + + def test_no_change_returns_base_action(self, tmp_path): + """ + When sources haven't changed between generate and instantiate, + the instantiator should return the base action digest unchanged. + """ + cas = FakeCAS() + artifactcache = FakeArtifactCache(cas, str(tmp_path)) + + content = b'unchanged source' + source_root = _build_source_tree(cas, {"file.c": content}) + sources = FakeSources(FakeSourceDir(source_root)) + element = FakeElement("app.bst", sources=sources) + + subaction_input = _build_source_tree(cas, {"file.c": content}) + subaction_digest = _build_action(cas, subaction_input) + + # Generate and store + generator = SpeculativeActionsGenerator(cas) + spec_actions = generator.generate_speculative_actions(element, [subaction_digest], []) + + weak_key = "unchanged-key" + artifact = FakeArtifact(element=element) + artifactcache.store_speculative_actions(artifact, spec_actions, weak_key=weak_key) + + # Retrieve and instantiate with same sources + retrieved = artifactcache.get_speculative_actions(artifact, weak_key=weak_key) + instantiator = SpeculativeActionInstantiator(cas, artifactcache) + result_digest = instantiator.instantiate_action(retrieved.actions[0], element, {}) + + # Should return the original action digest (no modifications needed) + assert result_digest.hash == subaction_digest.hash + + def test_multiple_subactions_roundtrip(self, tmp_path): + """ + Multiple subactions from a single build should each be independently + instantiatable after a source change. + """ + cas = FakeCAS() + artifactcache = FakeArtifactCache(cas, str(tmp_path)) + + v1_a = b'void a_v1() {}' + v1_b = b'void b_v1() {}' + + source_root = _build_source_tree(cas, {"a.c": v1_a, "b.c": v1_b}) + sources = FakeSources(FakeSourceDir(source_root)) + element = FakeElement("app.bst", sources=sources) + + # Two subactions, each using a different source file + sub1_input = _build_source_tree(cas, {"a.c": v1_a}) + sub1_digest = _build_action(cas, sub1_input) + sub2_input = _build_source_tree(cas, {"b.c": v1_b}) + sub2_digest = _build_action(cas, sub2_input) + + generator = SpeculativeActionsGenerator(cas) + spec_actions = generator.generate_speculative_actions( + element, [sub1_digest, sub2_digest], [] + ) + assert len(spec_actions.actions) == 2 + + weak_key = "multi-sub" + artifact = FakeArtifact(element=element) + artifactcache.store_speculative_actions(artifact, spec_actions, weak_key=weak_key) + + # Change both source files + v2_a = b'void a_v2() {}' + v2_b = b'void b_v2() {}' + source_root_v2 = _build_source_tree(cas, {"a.c": v2_a, "b.c": v2_b}) + sources_v2 = FakeSources(FakeSourceDir(source_root_v2)) + element_v2 = FakeElement("app.bst", sources=sources_v2) + artifact_v2 = FakeArtifact(element=element_v2) + + retrieved = artifactcache.get_speculative_actions(artifact_v2, weak_key=weak_key) + instantiator = SpeculativeActionInstantiator(cas, artifactcache) + + # Both actions should be instantiatable + for i, spec_action in enumerate(retrieved.actions): + result = instantiator.instantiate_action(spec_action, element_v2, {}) + assert result is not None + assert result.hash != [sub1_digest, sub2_digest][i].hash + + new_action = cas.fetch_action(result) + new_root = cas.fetch_directory_proto(new_action.input_root_digest) + # Each action should have exactly one file with the v2 digest + assert len(new_root.files) == 1 + expected_hash = _make_digest([v2_a, v2_b][i]).hash + assert new_root.files[0].digest.hash == expected_hash + + def test_nested_source_tree_roundtrip(self, tmp_path): + """ + Source files in nested directories should be correctly tracked + through generate and instantiate. + """ + cas = FakeCAS() + artifactcache = FakeArtifactCache(cas, str(tmp_path)) + + v1 = b'nested file v1' + source_root = _build_source_tree(cas, {"src/lib/util.c": v1}) + sources = FakeSources(FakeSourceDir(source_root)) + element = FakeElement("app.bst", sources=sources) + + # Subaction has the same nested file + sub_input = _build_source_tree(cas, {"src/lib/util.c": v1}) + sub_digest = _build_action(cas, sub_input) + + generator = SpeculativeActionsGenerator(cas) + spec_actions = generator.generate_speculative_actions(element, [sub_digest], []) + assert len(spec_actions.actions) == 1 + + weak_key = "nested" + artifact = FakeArtifact(element=element) + artifactcache.store_speculative_actions(artifact, spec_actions, weak_key=weak_key) + + # Change the nested file + v2 = b'nested file v2' + source_root_v2 = _build_source_tree(cas, {"src/lib/util.c": v2}) + sources_v2 = FakeSources(FakeSourceDir(source_root_v2)) + element_v2 = FakeElement("app.bst", sources=sources_v2) + artifact_v2 = FakeArtifact(element=element_v2) + + retrieved = artifactcache.get_speculative_actions(artifact_v2, weak_key=weak_key) + instantiator = SpeculativeActionInstantiator(cas, artifactcache) + result = instantiator.instantiate_action(retrieved.actions[0], element_v2, {}) + + assert result is not None + assert result.hash != sub_digest.hash + + # Verify nested file was updated + new_action = cas.fetch_action(result) + all_files = {} + self._collect_files(cas, cas.fetch_directory_proto(new_action.input_root_digest), "", all_files) + assert all_files["src/lib/util.c"] == _make_digest(v2).hash + + def test_weak_key_isolation(self, tmp_path): + """ + Different weak keys should store and retrieve independent SA sets, + modeling how different element configurations get separate SA entries. + """ + cas = FakeCAS() + artifactcache = FakeArtifactCache(cas, str(tmp_path)) + + content_a = b'content for config A' + content_b = b'content for config B' + + # Store SA under key A + source_root_a = _build_source_tree(cas, {"file.c": content_a}) + sources_a = FakeSources(FakeSourceDir(source_root_a)) + element_a = FakeElement("app.bst", sources=sources_a) + sub_a = _build_action(cas, _build_source_tree(cas, {"file.c": content_a})) + + generator = SpeculativeActionsGenerator(cas) + sa_a = generator.generate_speculative_actions(element_a, [sub_a], []) + artifact_a = FakeArtifact(element=element_a) + artifactcache.store_speculative_actions(artifact_a, sa_a, weak_key="key-A") + + # Store SA under key B + source_root_b = _build_source_tree(cas, {"file.c": content_b}) + sources_b = FakeSources(FakeSourceDir(source_root_b)) + element_b = FakeElement("app.bst", sources=sources_b) + sub_b = _build_action(cas, _build_source_tree(cas, {"file.c": content_b})) + + sa_b = generator.generate_speculative_actions(element_b, [sub_b], []) + artifact_b = FakeArtifact(element=element_b) + artifactcache.store_speculative_actions(artifact_b, sa_b, weak_key="key-B") + + # Retrieve each independently + ret_a = artifactcache.get_speculative_actions(artifact_a, weak_key="key-A") + ret_b = artifactcache.get_speculative_actions(artifact_b, weak_key="key-B") + + assert ret_a is not None + assert ret_b is not None + + # They should reference different base actions + assert ret_a.actions[0].base_action_digest.hash != ret_b.actions[0].base_action_digest.hash + + # Key A should not return key B's data + ret_missing = artifactcache.get_speculative_actions(artifact_a, weak_key="key-nonexistent") + assert ret_missing is None + + def test_priming_scenario(self, tmp_path): + """ + Models the priming queue's core scenario: + + 1. Element app.bst depends on dep.bst + 2. app.bst is built with dep v1 — subactions recorded, SA generated + and stored under app's weak key + 3. dep.bst is rebuilt with new content (v2) + 4. app.bst needs rebuilding (strict key changed), but its weak key + is stable (only dep names, not cache keys) + 5. Priming: retrieve SA by weak key, instantiate each action with + dep v2's artifact digests, verify the adapted actions have the + correct updated digests + + This is the core value of speculative actions: adapting cached + build actions to new dependency versions without rebuilding. + """ + cas = FakeCAS() + artifactcache = FakeArtifactCache(cas, str(tmp_path)) + + # --- Initial build: app depends on dep v1 --- + app_src = b'#include "dep.h"\nint main() { return dep(); }' + dep_header_v1 = b'int dep(void); /* v1 */' + dep_lib_v1 = b'dep-object-code-v1' + + app_source_root = _build_source_tree(cas, {"main.c": app_src}) + app_sources = FakeSources(FakeSourceDir(app_source_root)) + + dep_artifact_root_v1 = _build_source_tree(cas, { + "include/dep.h": dep_header_v1, + "lib/libdep.o": dep_lib_v1, + }) + dep_artifact_v1 = FakeArtifact(FakeSourceDir(dep_artifact_root_v1)) + dep_element_v1 = FakeElement("dep.bst", artifact=dep_artifact_v1) + + app_element = FakeElement("app.bst", sources=app_sources) + + # Subactions from app's build: compile (uses main.c + dep.h) and + # link (uses main.o + libdep.o) + compile_input = _build_source_tree(cas, { + "main.c": app_src, + "include/dep.h": dep_header_v1, + }) + compile_action = _build_action(cas, compile_input) + + link_input = _build_source_tree(cas, { + "main.o": b'app-object-code', + "lib/libdep.o": dep_lib_v1, + }) + link_action = _build_action(cas, link_input) + + # Generate SA from both subactions + generator = SpeculativeActionsGenerator(cas) + spec_actions = generator.generate_speculative_actions( + app_element, [compile_action, link_action], [dep_element_v1] + ) + + assert len(spec_actions.actions) == 2, ( + f"Expected 2 speculative actions (compile + link), got {len(spec_actions.actions)}" + ) + + # Store under app's weak key + weak_key = "app-weak-key" + app_artifact = FakeArtifact(element=app_element) + artifactcache.store_speculative_actions( + app_artifact, spec_actions, weak_key=weak_key + ) + + # --- dep.bst rebuilt with v2 --- + dep_header_v2 = b'int dep(void); /* v2 - added feature */' + dep_lib_v2 = b'dep-object-code-v2' + + dep_artifact_root_v2 = _build_source_tree(cas, { + "include/dep.h": dep_header_v2, + "lib/libdep.o": dep_lib_v2, + }) + dep_artifact_v2 = FakeArtifact(FakeSourceDir(dep_artifact_root_v2)) + dep_element_v2 = FakeElement("dep.bst", artifact=dep_artifact_v2) + + # app's sources unchanged, weak key stable + app_element_v2 = FakeElement("app.bst", sources=app_sources) + app_artifact_v2 = FakeArtifact(element=app_element_v2) + + # --- Priming: retrieve and instantiate --- + retrieved = artifactcache.get_speculative_actions( + app_artifact_v2, weak_key=weak_key + ) + assert retrieved is not None + assert len(retrieved.actions) == 2 + + element_lookup = {"dep.bst": dep_element_v2} + instantiator = SpeculativeActionInstantiator(cas, artifactcache) + + adapted_actions = [] + for spec_action in retrieved.actions: + result = instantiator.instantiate_action( + spec_action, app_element_v2, element_lookup + ) + assert result is not None + adapted_actions.append(result) + + # Verify compile action: main.c unchanged, dep.h updated to v2 + compile_result = cas.fetch_action(adapted_actions[0]) + compile_files = {} + self._collect_files( + cas, + cas.fetch_directory_proto(compile_result.input_root_digest), + "", compile_files, + ) + assert compile_files["main.c"] == _make_digest(app_src).hash + assert compile_files["include/dep.h"] == _make_digest(dep_header_v2).hash + + # Verify link action: libdep.o updated to v2 + link_result = cas.fetch_action(adapted_actions[1]) + link_files = {} + self._collect_files( + cas, + cas.fetch_directory_proto(link_result.input_root_digest), + "", link_files, + ) + assert link_files["lib/libdep.o"] == _make_digest(dep_lib_v2).hash + + @staticmethod + def _collect_files(cas, directory, prefix, result): + """Recursively collect {path: digest_hash} from a Directory proto.""" + if directory is None: + return + for f in directory.files: + path = f.name if not prefix else "{}/{}".format(prefix, f.name) + result[path] = f.digest.hash + for d in directory.directories: + subpath = d.name if not prefix else "{}/{}".format(prefix, d.name) + subdir = cas.fetch_directory_proto(d.digest) + TestGenerateStoreRetrieveInstantiate._collect_files(cas, subdir, subpath, result) + + +# --------------------------------------------------------------------------- +# Fake ActionCache service for ACTION overlay tests +# --------------------------------------------------------------------------- + +class FakeACService: + """Fake ActionCache service that returns stored ActionResults.""" + + def __init__(self): + self._results = {} # action_digest_hash -> ActionResult proto + + def store_action_result(self, action_digest, action_result): + self._results[action_digest.hash] = action_result + + def GetActionResult(self, request): + return self._results.get(request.action_digest.hash) + + +# --------------------------------------------------------------------------- +# ACTION overlay tests +# --------------------------------------------------------------------------- + +class TestActionOverlays: + """Tests for ACTION overlay generation and instantiation (cross-subaction output chaining).""" + + def test_action_overlay_generated_for_prior_output(self, tmp_path): + """ + Scenario: compile subaction produces main.o. Link subaction's input + tree contains main.o. Generator should create an ACTION overlay on + the link subaction pointing to the compile subaction's output. + """ + cas = FakeCAS() + ac_service = FakeACService() + + # --- Build phase --- + app_src = b'int main() { return 0; }' + main_o = b'compiled-object-code' + main_o_digest = _make_digest(main_o) + + # Element sources + source_root = _build_source_tree(cas, {"main.c": app_src}) + sources = FakeSources(FakeSourceDir(source_root)) + element = FakeElement("app.bst", sources=sources) + + # Compile subaction: input has main.c, output has main.o + compile_input = _build_source_tree(cas, {"main.c": app_src}) + compile_action_digest = _build_action(cas, compile_input) + + # Store compile's ActionResult with main.o as output + compile_result = remote_execution_pb2.ActionResult() + output_file = compile_result.output_files.add() + output_file.path = "main.o" + output_file.digest.CopyFrom(main_o_digest) + ac_service.store_action_result(compile_action_digest, compile_result) + + # Link subaction: input has main.o (output of compile) + link_input = _build_source_tree(cas, {"main.o": main_o}) + link_action_digest = _build_action(cas, link_input) + + # --- Generate with ac_service --- + generator = SpeculativeActionsGenerator(cas, ac_service=ac_service) + spec_actions = generator.generate_speculative_actions( + element, [compile_action_digest, link_action_digest], [] + ) + + # Compile should have SOURCE overlay for main.c + assert len(spec_actions.actions) >= 2 + compile_sa = spec_actions.actions[0] + assert any( + o.type == speculative_actions_pb2.SpeculativeActions.Overlay.SOURCE + for o in compile_sa.overlays + ) + + # Link should have ACTION overlay for main.o + link_sa = spec_actions.actions[1] + action_overlays = [ + o for o in link_sa.overlays + if o.type == speculative_actions_pb2.SpeculativeActions.Overlay.ACTION + ] + assert len(action_overlays) == 1 + ao = action_overlays[0] + assert ao.source_action_digest.hash == compile_action_digest.hash + assert ao.source_path == "main.o" + assert ao.target_digest.hash == main_o_digest.hash + + def test_action_overlay_not_generated_when_covered_by_source(self, tmp_path): + """ + If a file in the input tree is already resolved as a SOURCE overlay, + it should NOT get a duplicate ACTION overlay even if it matches a + prior subaction output. + """ + cas = FakeCAS() + ac_service = FakeACService() + + # main.c appears both in sources AND as output of subaction 0 + src_content = b'int main() { return 0; }' + src_digest = _make_digest(src_content) + + source_root = _build_source_tree(cas, {"main.c": src_content}) + sources = FakeSources(FakeSourceDir(source_root)) + element = FakeElement("app.bst", sources=sources) + + # Subaction 0: some action that happens to output main.c + sub0_input = _build_source_tree(cas, {"other.c": b'other'}) + sub0_digest = _build_action(cas, sub0_input) + sub0_result = remote_execution_pb2.ActionResult() + out = sub0_result.output_files.add() + out.path = "main.c" + out.digest.CopyFrom(src_digest) + ac_service.store_action_result(sub0_digest, sub0_result) + + # Subaction 1: uses main.c + sub1_input = _build_source_tree(cas, {"main.c": src_content}) + sub1_digest = _build_action(cas, sub1_input) + + generator = SpeculativeActionsGenerator(cas, ac_service=ac_service) + spec_actions = generator.generate_speculative_actions( + element, [sub0_digest, sub1_digest], [] + ) + + # The second subaction should only have a SOURCE overlay, not ACTION + sub1_sa = [sa for sa in spec_actions.actions if sa.base_action_digest.hash == sub1_digest.hash] + assert len(sub1_sa) == 1 + for overlay in sub1_sa[0].overlays: + assert overlay.type != speculative_actions_pb2.SpeculativeActions.Overlay.ACTION + + def test_action_overlay_instantiation_with_instantiated_actions(self, tmp_path): + """ + Instantiate an ACTION overlay using instantiated_actions from a prior + subaction's priming, with the ActionResult in the AC. + """ + cas = FakeCAS() + ac_service = FakeACService() + artifactcache = FakeArtifactCache(cas, str(tmp_path)) + + # Build an action whose input tree has main.o + old_main_o = b'old-object-code' + old_main_o_digest = _make_digest(old_main_o) + link_input = _build_source_tree(cas, {"main.o": old_main_o}) + link_action_digest = _build_action(cas, link_input) + + # Create a SpeculativeAction with an ACTION overlay + # Use a fake compile action digest as the producing action's base + compile_base_digest = _make_digest(b'fake-compile-action-base') + # The adapted digest (what was actually executed after priming) + compile_adapted_digest = _make_digest(b'fake-compile-action-adapted') + spec_action = speculative_actions_pb2.SpeculativeActions.SpeculativeAction() + spec_action.base_action_digest.CopyFrom(link_action_digest) + overlay = spec_action.overlays.add() + overlay.type = speculative_actions_pb2.SpeculativeActions.Overlay.ACTION + overlay.source_action_digest.CopyFrom(compile_base_digest) + overlay.source_path = "main.o" + overlay.target_digest.CopyFrom(old_main_o_digest) + + # Simulate: compile subaction was instantiated and executed, + # producing new main.o — result is in AC under adapted digest + new_main_o = b'new-object-code' + new_main_o_digest = _make_digest(new_main_o) + compile_result = remote_execution_pb2.ActionResult() + out = compile_result.output_files.add() + out.path = "main.o" + out.digest.CopyFrom(new_main_o_digest) + ac_service.store_action_result(compile_adapted_digest, compile_result) + + # Global instantiated_actions: base -> adapted + instantiated_actions = {compile_base_digest.hash: compile_adapted_digest} + + element = FakeElement("app.bst") + instantiator = SpeculativeActionInstantiator(cas, artifactcache, ac_service=ac_service) + result_digest = instantiator.instantiate_action( + spec_action, element, {}, + instantiated_actions=instantiated_actions, + ) + + assert result_digest is not None + assert result_digest.hash != link_action_digest.hash + + # Verify the action's input tree has the new main.o digest + new_action = cas.fetch_action(result_digest) + new_root = cas.fetch_directory_proto(new_action.input_root_digest) + assert new_root.files[0].name == "main.o" + assert new_root.files[0].digest.hash == new_main_o_digest.hash + + def test_action_overlay_full_roundtrip(self, tmp_path): + """ + Full roundtrip: generate ACTION overlays, store, retrieve, + instantiate with instantiated_actions from sequential priming execution. + + Models the compile→link scenario where dep.h changes, causing + main.o to change, which should be chained to the link action. + """ + cas = FakeCAS() + ac_service = FakeACService() + artifactcache = FakeArtifactCache(cas, str(tmp_path)) + + # --- Build phase (v1) --- + app_src = b'#include "dep.h"\nint main() { return dep(); }' + dep_header_v1 = b'int dep(void); /* v1 */' + main_o_v1 = b'main-object-v1' + main_o_v1_digest = _make_digest(main_o_v1) + + source_root = _build_source_tree(cas, {"main.c": app_src}) + sources = FakeSources(FakeSourceDir(source_root)) + + dep_artifact_root = _build_source_tree(cas, {"include/dep.h": dep_header_v1}) + dep_artifact = FakeArtifact(FakeSourceDir(dep_artifact_root)) + dep_element_v1 = FakeElement("dep.bst", artifact=dep_artifact) + + element = FakeElement("app.bst", sources=sources) + + # Compile: uses main.c + dep.h, produces main.o + compile_input = _build_source_tree(cas, { + "main.c": app_src, + "include/dep.h": dep_header_v1, + }) + compile_digest = _build_action(cas, compile_input) + + compile_result = remote_execution_pb2.ActionResult() + out = compile_result.output_files.add() + out.path = "main.o" + out.digest.CopyFrom(main_o_v1_digest) + ac_service.store_action_result(compile_digest, compile_result) + + # Link: uses main.o + link_input = _build_source_tree(cas, {"main.o": main_o_v1}) + link_digest = _build_action(cas, link_input) + + # --- Generate --- + generator = SpeculativeActionsGenerator(cas, ac_service=ac_service) + spec_actions = generator.generate_speculative_actions( + element, [compile_digest, link_digest], [dep_element_v1] + ) + + assert len(spec_actions.actions) == 2 + + # Verify link has ACTION overlay + link_sa = spec_actions.actions[1] + action_overlays = [ + o for o in link_sa.overlays + if o.type == speculative_actions_pb2.SpeculativeActions.Overlay.ACTION + ] + assert len(action_overlays) == 1 + + # --- Store --- + weak_key = "app-weak" + artifact = FakeArtifact(element=element) + artifactcache.store_speculative_actions(artifact, spec_actions, weak_key=weak_key) + + # --- dep changes (v2) --- + dep_header_v2 = b'int dep(void); /* v2 */' + dep_artifact_root_v2 = _build_source_tree(cas, {"include/dep.h": dep_header_v2}) + dep_artifact_v2 = FakeArtifact(FakeSourceDir(dep_artifact_root_v2)) + dep_element_v2 = FakeElement("dep.bst", artifact=dep_artifact_v2) + + element_v2 = FakeElement("app.bst", sources=sources) + artifact_v2 = FakeArtifact(element=element_v2) + + # --- Retrieve --- + retrieved = artifactcache.get_speculative_actions(artifact_v2, weak_key=weak_key) + assert retrieved is not None + + # --- Sequential instantiation (simulating priming queue) --- + element_lookup = {"dep.bst": dep_element_v2} + instantiator = SpeculativeActionInstantiator(cas, artifactcache, ac_service=ac_service) + instantiated_actions = {} + + # 1) Instantiate compile action (SOURCE + ARTIFACT overlays) + compile_result_digest = instantiator.instantiate_action( + retrieved.actions[0], element_v2, element_lookup, + instantiated_actions=instantiated_actions, + ) + assert compile_result_digest is not None + + # Record in instantiated_actions (as the priming queue would) + instantiated_actions[compile_digest.hash] = compile_result_digest + + # Simulate compile execution producing new main.o + # Store the result in the AC under the adapted digest + main_o_v2 = b'main-object-v2' + main_o_v2_digest = _make_digest(main_o_v2) + compile_v2_result = remote_execution_pb2.ActionResult() + out = compile_v2_result.output_files.add() + out.path = "main.o" + out.digest.CopyFrom(main_o_v2_digest) + ac_service.store_action_result(compile_result_digest, compile_v2_result) + + # 2) Instantiate link action (ACTION overlay resolves via + # instantiated_actions + AC lookup) + link_result_digest = instantiator.instantiate_action( + retrieved.actions[1], element_v2, element_lookup, + instantiated_actions=instantiated_actions, + ) + assert link_result_digest is not None + assert link_result_digest.hash != link_digest.hash + + # Verify link action's input tree has new main.o + link_action = cas.fetch_action(link_result_digest) + link_root = cas.fetch_directory_proto(link_action.input_root_digest) + assert link_root.files[0].name == "main.o" + assert link_root.files[0].digest.hash == main_o_v2_digest.hash + + def test_no_action_overlays_without_ac_service(self, tmp_path): + """ + When ac_service is None, no ACTION overlays should be generated + (backward compatibility). + """ + cas = FakeCAS() + + src_content = b'int main() { return 0; }' + main_o = b'object-code' + + source_root = _build_source_tree(cas, {"main.c": src_content}) + sources = FakeSources(FakeSourceDir(source_root)) + element = FakeElement("app.bst", sources=sources) + + compile_input = _build_source_tree(cas, {"main.c": src_content}) + compile_digest = _build_action(cas, compile_input) + + link_input = _build_source_tree(cas, {"main.o": main_o}) + link_digest = _build_action(cas, link_input) + + # No ac_service — should behave exactly as before + generator = SpeculativeActionsGenerator(cas) + spec_actions = generator.generate_speculative_actions( + element, [compile_digest, link_digest], [] + ) + + # Compile has SOURCE overlay, link has no overlays (main.o unresolved) + assert len(spec_actions.actions) == 1 # only compile + for sa in spec_actions.actions: + for o in sa.overlays: + assert o.type != speculative_actions_pb2.SpeculativeActions.Overlay.ACTION + + +class TestCrossElementActionOverlays: + """Tests for cross-element ACTION overlays (dependency subaction output chaining).""" + + def test_cross_element_action_overlay_generated(self, tmp_path): + """ + Scenario: dep.bst has a codegen subaction that produces gen.h. + app.bst's compile subaction uses gen.h in its input tree. + Generator should create a cross-element ACTION overlay pointing + to dep.bst's codegen subaction. + """ + cas = FakeCAS() + ac_service = FakeACService() + artifactcache = FakeArtifactCache(cas, str(tmp_path)) + + # --- dep.bst was built, generated SAs --- + gen_h_content = b'/* generated header v1 */' + gen_h_digest = _make_digest(gen_h_content) + + # dep's codegen subaction produced gen.h + dep_codegen_input = _build_source_tree(cas, {"schema.xml": b''}) + dep_codegen_digest = _build_action(cas, dep_codegen_input) + + dep_codegen_result = remote_execution_pb2.ActionResult() + out = dep_codegen_result.output_files.add() + out.path = "gen.h" + out.digest.CopyFrom(gen_h_digest) + ac_service.store_action_result(dep_codegen_digest, dep_codegen_result) + + # dep's artifact contains gen.h (installed) + dep_artifact_root = _build_source_tree(cas, {"include/gen.h": gen_h_content}) + dep_artifact = FakeArtifact(FakeSourceDir(dep_artifact_root)) + dep_element = FakeElement("dep.bst", artifact=dep_artifact) + + # dep's stored SpeculativeActions + dep_sa = speculative_actions_pb2.SpeculativeActions() + dep_spec_action = dep_sa.actions.add() + dep_spec_action.base_action_digest.CopyFrom(dep_codegen_digest) + dep_sa_artifact = FakeArtifact(element=dep_element) + artifactcache.store_speculative_actions(dep_sa_artifact, dep_sa, weak_key="dep-weak") + + # Patch dep_artifact to return the stored SA + dep_artifact._sa = dep_sa + original_get_sa = artifactcache.get_speculative_actions + def get_sa_with_dep(artifact, weak_key=None): + if hasattr(artifact, '_sa'): + return artifact._sa + return original_get_sa(artifact, weak_key=weak_key) + artifactcache.get_speculative_actions = get_sa_with_dep + + # --- app.bst build: compile uses gen.h from dep --- + app_src = b'#include "gen.h"\nint main() {}' + app_source_root = _build_source_tree(cas, {"main.c": app_src}) + app_sources = FakeSources(FakeSourceDir(app_source_root)) + app_element = FakeElement("app.bst", sources=app_sources) + + # app's compile subaction input has main.c and gen.h + compile_input = _build_source_tree(cas, { + "main.c": app_src, + "include/gen.h": gen_h_content, + }) + compile_digest = _build_action(cas, compile_input) + + # --- Generate SAs for app --- + generator = SpeculativeActionsGenerator( + cas, ac_service=ac_service, artifactcache=artifactcache + ) + spec_actions = generator.generate_speculative_actions( + app_element, [compile_digest], [dep_element] + ) + + assert len(spec_actions.actions) == 1 + overlays = spec_actions.actions[0].overlays + + # main.c should be SOURCE overlay + source_overlays = [ + o for o in overlays + if o.type == speculative_actions_pb2.SpeculativeActions.Overlay.SOURCE + ] + assert len(source_overlays) == 1 + assert source_overlays[0].source_path == "main.c" + + # gen.h could be ARTIFACT (from dep's artifact tree) or ACTION + # (from dep's codegen subaction output). ARTIFACT takes priority + # in the digest cache, but gen.h in the input tree at include/gen.h + # has the same content digest as dep's codegen output. + # Since SOURCE/ARTIFACT are checked first, gen.h at include/gen.h + # should be an ARTIFACT overlay (dep's artifact has it). + # But the gen.h digest also matches dep's codegen output — since + # ARTIFACT already covers it, no ACTION overlay should be created. + action_overlays = [ + o for o in overlays + if o.type == speculative_actions_pb2.SpeculativeActions.Overlay.ACTION + ] + artifact_overlays = [ + o for o in overlays + if o.type == speculative_actions_pb2.SpeculativeActions.Overlay.ARTIFACT + ] + assert len(artifact_overlays) == 1 + assert artifact_overlays[0].source_path == "include/gen.h" + assert len(action_overlays) == 0 # Covered by ARTIFACT + + def test_cross_element_action_overlay_for_intermediate_file(self, tmp_path): + """ + When a dependency subaction produces an intermediate file that is + NOT in the dependency's artifact but IS in the current element's + subaction input tree, a cross-element ACTION overlay should be + generated (since ARTIFACT can't cover it). + """ + cas = FakeCAS() + ac_service = FakeACService() + artifactcache = FakeArtifactCache(cas, str(tmp_path)) + + # dep.bst: codegen produces intermediate.h but only installs final.h + intermediate_content = b'/* intermediate */' + intermediate_digest = _make_digest(intermediate_content) + + dep_codegen_input = _build_source_tree(cas, {"schema.xml": b''}) + dep_codegen_digest = _build_action(cas, dep_codegen_input) + + dep_result = remote_execution_pb2.ActionResult() + out = dep_result.output_files.add() + out.path = "intermediate.h" + out.digest.CopyFrom(intermediate_digest) + ac_service.store_action_result(dep_codegen_digest, dep_result) + + # dep's artifact only has final.h (intermediate.h not installed) + dep_artifact_root = _build_source_tree(cas, {"include/final.h": b'/* final */'}) + dep_artifact = FakeArtifact(FakeSourceDir(dep_artifact_root)) + dep_element = FakeElement("dep.bst", artifact=dep_artifact) + + # dep's stored SA + dep_sa = speculative_actions_pb2.SpeculativeActions() + dep_spec = dep_sa.actions.add() + dep_spec.base_action_digest.CopyFrom(dep_codegen_digest) + dep_artifact._sa = dep_sa + def get_sa(artifact, weak_key=None): + if hasattr(artifact, '_sa'): + return artifact._sa + return None + artifactcache.get_speculative_actions = get_sa + + # app.bst compile uses intermediate.h (somehow available in sandbox) + app_src = b'#include "intermediate.h"' + app_source_root = _build_source_tree(cas, {"main.c": app_src}) + app_sources = FakeSources(FakeSourceDir(app_source_root)) + app_element = FakeElement("app.bst", sources=app_sources) + + compile_input = _build_source_tree(cas, { + "main.c": app_src, + "intermediate.h": intermediate_content, + }) + compile_digest = _build_action(cas, compile_input) + + # Generate + generator = SpeculativeActionsGenerator( + cas, ac_service=ac_service, artifactcache=artifactcache + ) + spec_actions = generator.generate_speculative_actions( + app_element, [compile_digest], [dep_element] + ) + + assert len(spec_actions.actions) == 1 + overlays = spec_actions.actions[0].overlays + + # intermediate.h is not in sources or dep artifact → ACTION overlay + action_overlays = [ + o for o in overlays + if o.type == speculative_actions_pb2.SpeculativeActions.Overlay.ACTION + ] + assert len(action_overlays) == 1 + ao = action_overlays[0] + assert ao.source_element == "dep.bst" + assert ao.source_action_digest.hash == dep_codegen_digest.hash + assert ao.source_path == "intermediate.h" + assert ao.target_digest.hash == intermediate_digest.hash + + def test_cross_element_action_overlay_instantiation(self, tmp_path): + """ + Instantiate a cross-element ACTION overlay by looking up the + producing subaction's ActionResult from the action cache. + """ + cas = FakeCAS() + ac_service = FakeACService() + artifactcache = FakeArtifactCache(cas, str(tmp_path)) + + # Old intermediate file in the action's input tree + old_content = b'/* old intermediate */' + old_digest = _make_digest(old_content) + action_input = _build_source_tree(cas, {"intermediate.h": old_content}) + action_digest = _build_action(cas, action_input) + + # The producing subaction's new ActionResult (dep was rebuilt) + dep_codegen_digest = _make_digest(b'dep-codegen-action') + new_content = b'/* new intermediate */' + new_digest = _make_digest(new_content) + new_result = remote_execution_pb2.ActionResult() + out = new_result.output_files.add() + out.path = "intermediate.h" + out.digest.CopyFrom(new_digest) + ac_service.store_action_result(dep_codegen_digest, new_result) + + # Build a SpeculativeAction with cross-element ACTION overlay + spec_action = speculative_actions_pb2.SpeculativeActions.SpeculativeAction() + spec_action.base_action_digest.CopyFrom(action_digest) + overlay = spec_action.overlays.add() + overlay.type = speculative_actions_pb2.SpeculativeActions.Overlay.ACTION + overlay.source_element = "dep.bst" + overlay.source_action_digest.CopyFrom(dep_codegen_digest) + overlay.source_path = "intermediate.h" + overlay.target_digest.CopyFrom(old_digest) + + element = FakeElement("app.bst") + instantiator = SpeculativeActionInstantiator( + cas, artifactcache, ac_service=ac_service + ) + # Cross-element: the dep's codegen action was instantiated and + # its result is in AC. instantiated_actions maps base -> adapted + # (in this case, same digest since we stored under dep_codegen_digest). + instantiated_actions = {dep_codegen_digest.hash: dep_codegen_digest} + result_digest = instantiator.instantiate_action( + spec_action, element, {}, + instantiated_actions=instantiated_actions, + ) + + assert result_digest is not None + assert result_digest.hash != action_digest.hash + + new_action = cas.fetch_action(result_digest) + new_root = cas.fetch_directory_proto(new_action.input_root_digest) + assert new_root.files[0].name == "intermediate.h" + assert new_root.files[0].digest.hash == new_digest.hash + + +# --------------------------------------------------------------------------- +# Speculative action mode tests +# --------------------------------------------------------------------------- + +class TestSpeculativeActionModes: + """Tests verifying that each mode generates the correct overlay types.""" + + def _build_compile_link_scenario(self, cas, ac_service): + """Build a compile→link scenario with source, artifact, and action overlays. + + Returns (element, dep_element, subaction_digests, dependencies) + """ + app_src = b'int main() { return dep(); }' + dep_header = b'int dep(void);' + main_o = b'main-object-code' + main_o_digest = _make_digest(main_o) + + source_root = _build_source_tree(cas, {"main.c": app_src}) + sources = FakeSources(FakeSourceDir(source_root)) + + dep_artifact_root = _build_source_tree(cas, {"include/dep.h": dep_header}) + dep_artifact = FakeArtifact(FakeSourceDir(dep_artifact_root)) + dep_element = FakeElement("dep.bst", artifact=dep_artifact) + + element = FakeElement("app.bst", sources=sources) + + # Compile: uses main.c + dep.h, produces main.o + compile_input = _build_source_tree(cas, { + "main.c": app_src, + "include/dep.h": dep_header, + }) + compile_digest = _build_action(cas, compile_input) + + compile_result = remote_execution_pb2.ActionResult() + out = compile_result.output_files.add() + out.path = "main.o" + out.digest.CopyFrom(main_o_digest) + ac_service.store_action_result(compile_digest, compile_result) + + # Link: uses main.o (output of compile) + link_input = _build_source_tree(cas, {"main.o": main_o}) + link_digest = _build_action(cas, link_input) + + return element, dep_element, [compile_digest, link_digest], [dep_element] + + def test_source_artifact_mode_no_action_overlays(self, tmp_path): + """source-artifact mode should produce only SOURCE and ARTIFACT overlays.""" + from buildstream.types import _SpeculativeActionMode + + cas = FakeCAS() + ac_service = FakeACService() + + element, dep_element, subaction_digests, dependencies = \ + self._build_compile_link_scenario(cas, ac_service) + + generator = SpeculativeActionsGenerator(cas, ac_service=ac_service) + spec_actions = generator.generate_speculative_actions( + element, subaction_digests, dependencies, + mode=_SpeculativeActionMode.SOURCE_ARTIFACT, + ) + + # Should have spec_actions for subactions with SOURCE/ARTIFACT overlays + assert len(spec_actions.actions) >= 1 + + # No ACTION overlays should exist in any spec_action + for sa in spec_actions.actions: + for overlay in sa.overlays: + assert overlay.type != speculative_actions_pb2.SpeculativeActions.Overlay.ACTION, \ + "source-artifact mode should not produce ACTION overlays" + + def test_intra_element_mode_has_action_overlays(self, tmp_path): + """intra-element mode should produce ACTION overlays for within-element chains.""" + from buildstream.types import _SpeculativeActionMode + + cas = FakeCAS() + ac_service = FakeACService() + + element, dep_element, subaction_digests, dependencies = \ + self._build_compile_link_scenario(cas, ac_service) + + generator = SpeculativeActionsGenerator(cas, ac_service=ac_service) + spec_actions = generator.generate_speculative_actions( + element, subaction_digests, dependencies, + mode=_SpeculativeActionMode.INTRA_ELEMENT, + ) + + # Should have 2 spec_actions (compile + link) + assert len(spec_actions.actions) == 2 + + # The link action should have an ACTION overlay for main.o + link_sa = spec_actions.actions[1] + action_overlays = [ + o for o in link_sa.overlays + if o.type == speculative_actions_pb2.SpeculativeActions.Overlay.ACTION + ] + assert len(action_overlays) == 1 + assert action_overlays[0].source_path == "main.o" + + # ACTION overlays should be intra-element only (source_element empty) + for sa in spec_actions.actions: + for overlay in sa.overlays: + if overlay.type == speculative_actions_pb2.SpeculativeActions.Overlay.ACTION: + assert overlay.source_element == "", \ + "intra-element mode should not produce cross-element ACTION overlays" + + def test_full_mode_has_cross_element_action_overlays(self, tmp_path): + """full mode should produce cross-element ACTION overlays from dep subactions.""" + from buildstream.types import _SpeculativeActionMode + + cas = FakeCAS() + ac_service = FakeACService() + artifactcache = FakeArtifactCache(cas, str(tmp_path)) + + # dep element has a subaction that produces intermediate.h + dep_intermediate = b'/* generated header */' + dep_intermediate_digest = _make_digest(dep_intermediate) + + dep_compile_input = _build_source_tree(cas, {"gen.c": b'void gen() {}'}) + dep_compile_digest = _build_action(cas, dep_compile_input) + + dep_result = remote_execution_pb2.ActionResult() + out = dep_result.output_files.add() + out.path = "intermediate.h" + out.digest.CopyFrom(dep_intermediate_digest) + ac_service.store_action_result(dep_compile_digest, dep_result) + + # Create dep artifact and store dep SA on it + dep_element_obj = FakeElement("dep.bst") + dep_artifact = FakeArtifact(element=dep_element_obj) + + dep_sa = speculative_actions_pb2.SpeculativeActions() + dep_spec = dep_sa.actions.add() + dep_spec.base_action_digest.CopyFrom(dep_compile_digest) + artifactcache.store_speculative_actions(dep_artifact, dep_sa) + + # Current element uses intermediate.h in its compile input + source_root = _build_source_tree(cas, {"main.c": b'#include "intermediate.h"'}) + sources = FakeSources(FakeSourceDir(source_root)) + element = FakeElement("app.bst", sources=sources) + + compile_input = _build_source_tree(cas, { + "main.c": b'#include "intermediate.h"', + "intermediate.h": dep_intermediate, + }) + compile_digest = _build_action(cas, compile_input) + + # dep_element must use the SAME artifact object so + # get_speculative_actions finds the stored SA + dep_element_obj._artifact = dep_artifact + dep_element = dep_element_obj + + generator = SpeculativeActionsGenerator( + cas, ac_service=ac_service, artifactcache=artifactcache + ) + spec_actions = generator.generate_speculative_actions( + element, [compile_digest], [dep_element], + mode=_SpeculativeActionMode.FULL, + ) + + assert len(spec_actions.actions) == 1 + + # Should have a cross-element ACTION overlay for intermediate.h + action_overlays = [ + o for o in spec_actions.actions[0].overlays + if o.type == speculative_actions_pb2.SpeculativeActions.Overlay.ACTION + ] + assert len(action_overlays) == 1 + assert action_overlays[0].source_element == "dep.bst" + assert action_overlays[0].source_path == "intermediate.h" + + def test_mode_backward_compat_bool(self): + """Boolean True/False should map to full/none modes.""" + from buildstream.types import _SpeculativeActionMode + + # Verify enum values exist and are distinct + assert _SpeculativeActionMode.NONE.value == "none" + assert _SpeculativeActionMode.PRIME_ONLY.value == "prime-only" + assert _SpeculativeActionMode.SOURCE_ARTIFACT.value == "source-artifact" + assert _SpeculativeActionMode.INTRA_ELEMENT.value == "intra-element" + assert _SpeculativeActionMode.FULL.value == "full" + + def test_source_artifact_mode_fewer_ac_calls(self, tmp_path): + """source-artifact mode should make zero AC calls during generation.""" + from buildstream.types import _SpeculativeActionMode + + cas = FakeCAS() + + # Use a counting AC service to verify zero calls + class CountingACService: + def __init__(self): + self.call_count = 0 + self._results = {} + def store_action_result(self, action_digest, action_result): + self._results[action_digest.hash] = action_result + def GetActionResult(self, request): + self.call_count += 1 + return self._results.get(request.action_digest.hash) + + ac_service = CountingACService() + + element, dep_element, subaction_digests, dependencies = \ + self._build_compile_link_scenario(cas, ac_service) + + # source-artifact mode: generator should NOT use ac_service + generator = SpeculativeActionsGenerator(cas, ac_service=ac_service) + spec_actions = generator.generate_speculative_actions( + element, subaction_digests, dependencies, + mode=_SpeculativeActionMode.SOURCE_ARTIFACT, + ) + + assert ac_service.call_count == 0, \ + f"source-artifact mode should make 0 AC calls, got {ac_service.call_count}" + + def test_intra_element_mode_limited_ac_calls(self, tmp_path): + """intra-element mode should only make AC calls for own subactions.""" + from buildstream.types import _SpeculativeActionMode + + cas = FakeCAS() + + class CountingACService: + def __init__(self): + self.call_count = 0 + self._results = {} + def store_action_result(self, action_digest, action_result): + self._results[action_digest.hash] = action_result + def GetActionResult(self, request): + self.call_count += 1 + return self._results.get(request.action_digest.hash) + + ac_service = CountingACService() + + element, dep_element, subaction_digests, dependencies = \ + self._build_compile_link_scenario(cas, ac_service) + + # intra-element mode: should call AC for own subactions only + # (2 subactions = 2 _record_subaction_outputs calls) + generator = SpeculativeActionsGenerator(cas, ac_service=ac_service) + spec_actions = generator.generate_speculative_actions( + element, subaction_digests, dependencies, + mode=_SpeculativeActionMode.INTRA_ELEMENT, + ) + + # Should be exactly N calls for N subactions (no dep seeding) + assert ac_service.call_count == len(subaction_digests), \ + f"intra-element mode should make {len(subaction_digests)} AC calls " \ + f"(one per own subaction), got {ac_service.call_count}" diff --git a/tests/speculative_actions/test_weak_key.py b/tests/speculative_actions/test_weak_key.py new file mode 100644 index 000000000..25672d791 --- /dev/null +++ b/tests/speculative_actions/test_weak_key.py @@ -0,0 +1,211 @@ +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Tests for the speculative actions weak key lookup. + +The weak cache key is used for speculative actions lookup because it is: +- Stable across dependency version changes (only dep names, not cache keys) +- Changing when the element's own sources change +- Changing when build commands change +- Changing when environment changes +- Changing when sandbox config changes + +This mirrors Element._calculate_cache_key() with weak-mode dependencies +(only [project_name, name] per dependency). +""" + +import pytest +from buildstream._cachekey import generate_key + + +# These helpers mirror the structure of Element._calculate_cache_key() to +# verify the properties of the weak key as used for speculative actions. +# The actual weak key is computed by Element.__update_cache_keys() using +# _calculate_cache_key(dependencies) where dependencies are [project, name] +# pairs (in non-strict mode). + +def _make_weak_key_dict( + plugin_name="autotools", + plugin_key=None, + sources_key="abc123", + dep_names=None, + sandbox=None, + environment=None, + public=None, +): + """Helper to construct a dict that mirrors the weak cache key inputs. + + This doesn't replicate _calculate_cache_key exactly, but captures the + same structural properties for testing key stability/invalidation. + """ + if plugin_key is None: + plugin_key = { + "build-commands": ["make"], + "install-commands": ["make install"], + } + if dep_names is None: + dep_names = [["project", "base.bst"], ["project", "dep-a.bst"]] + if environment is None: + environment = {"PATH": "/usr/bin"} + if public is None: + public = {} + + cache_key_dict = { + "core-artifact-version": 1, + "element-plugin-key": plugin_key, + "element-plugin-name": plugin_name, + "element-plugin-version": 0, + "sources": sources_key, + "public": public, + "fatal-warnings": [], + } + if sandbox is not None: + cache_key_dict["sandbox"] = sandbox + cache_key_dict["environment"] = environment + + # Weak dependencies: only [project, name] pairs (no cache keys) + cache_key_dict["dependencies"] = sorted(dep_names) + + return cache_key_dict + + +class TestWeakKeyStability: + """Verify key stability: same inputs produce same key.""" + + def test_same_inputs_same_key(self): + """Identical inputs must produce the same key.""" + dict1 = _make_weak_key_dict() + dict2 = _make_weak_key_dict() + assert generate_key(dict1) == generate_key(dict2) + + def test_stable_across_dependency_version_changes(self): + """Key uses dependency names only, not their cache keys. + + When a dependency is rebuilt with different content, the weak key + remains stable because it only records [project, name] pairs. + """ + # Same dep names → same key, regardless of what version was built + dict1 = _make_weak_key_dict(dep_names=[["proj", "dep.bst"]]) + dict2 = _make_weak_key_dict(dep_names=[["proj", "dep.bst"]]) + assert generate_key(dict1) == generate_key(dict2) + + def test_dependency_order_irrelevant(self): + """Dependency names are sorted, so ordering doesn't matter.""" + dict1 = _make_weak_key_dict(dep_names=[["proj", "a.bst"], ["proj", "b.bst"]]) + dict2 = _make_weak_key_dict(dep_names=[["proj", "b.bst"], ["proj", "a.bst"]]) + assert generate_key(dict1) == generate_key(dict2) + + +class TestWeakKeyInvalidation: + """Verify key changes when element configuration changes.""" + + def test_changes_when_source_changes(self): + """Different source content must produce a different key. + + Unlike the old structural key, the weak key includes source + digests, so changing source code correctly invalidates it. + """ + key1 = generate_key(_make_weak_key_dict(sources_key="source-v1")) + key2 = generate_key(_make_weak_key_dict(sources_key="source-v2")) + assert key1 != key2 + + def test_changes_when_build_commands_change(self): + """Different build commands must produce a different key.""" + key1 = generate_key( + _make_weak_key_dict(plugin_key={"build-commands": ["make"]}) + ) + key2 = generate_key( + _make_weak_key_dict(plugin_key={"build-commands": ["cmake --build ."]}) + ) + assert key1 != key2 + + def test_changes_when_install_commands_change(self): + """Different install commands must produce a different key.""" + key1 = generate_key( + _make_weak_key_dict(plugin_key={"install-commands": ["make install"]}) + ) + key2 = generate_key( + _make_weak_key_dict(plugin_key={"install-commands": ["make install DESTDIR=/foo"]}) + ) + assert key1 != key2 + + def test_changes_when_dependency_names_change(self): + """Adding a dependency must change the key.""" + key1 = generate_key( + _make_weak_key_dict(dep_names=[["proj", "base.bst"]]) + ) + key2 = generate_key( + _make_weak_key_dict(dep_names=[["proj", "base.bst"], ["proj", "extra.bst"]]) + ) + assert key1 != key2 + + def test_changes_when_dependency_removed(self): + """Removing a dependency must change the key.""" + key1 = generate_key( + _make_weak_key_dict(dep_names=[["proj", "base.bst"], ["proj", "dep.bst"]]) + ) + key2 = generate_key( + _make_weak_key_dict(dep_names=[["proj", "base.bst"]]) + ) + assert key1 != key2 + + def test_changes_when_plugin_name_changes(self): + """Different plugin type must produce a different key.""" + key1 = generate_key(_make_weak_key_dict(plugin_name="autotools")) + key2 = generate_key(_make_weak_key_dict(plugin_name="cmake")) + assert key1 != key2 + + def test_changes_when_sandbox_config_changes(self): + """Different sandbox configuration must change the key.""" + key1 = generate_key( + _make_weak_key_dict(sandbox={"build-os": "linux", "build-arch": "x86_64"}) + ) + key2 = generate_key( + _make_weak_key_dict(sandbox={"build-os": "linux", "build-arch": "aarch64"}) + ) + assert key1 != key2 + + def test_changes_when_environment_changes(self): + """Different environment must change the key.""" + key1 = generate_key( + _make_weak_key_dict( + sandbox={"build-os": "linux"}, + environment={"PATH": "/usr/bin"}, + ) + ) + key2 = generate_key( + _make_weak_key_dict( + sandbox={"build-os": "linux"}, + environment={"PATH": "/usr/bin", "CC": "gcc"}, + ) + ) + assert key1 != key2 + + def test_no_sandbox_vs_sandbox(self): + """Having sandbox config vs not having it must change the key.""" + key1 = generate_key(_make_weak_key_dict(sandbox=None)) + key2 = generate_key( + _make_weak_key_dict(sandbox={"build-os": "linux"}) + ) + assert key1 != key2 + + +class TestWeakKeyFormat: + """Verify key format properties.""" + + def test_key_is_hex_digest(self): + """Key should be a valid sha256 hex digest.""" + key = generate_key(_make_weak_key_dict()) + assert len(key) == 64 + assert all(c in "0123456789abcdef" for c in key)