From cb825579ce1b2fbbc95ae593e11dfb6fa131d782 Mon Sep 17 00:00:00 2001 From: michaelj Date: Wed, 25 Mar 2026 12:26:27 +0000 Subject: [PATCH 01/14] docs(skills): add operation examination skill --- .../skills/examining-ado-operations/SKILL.md | 260 ++++++++++++++++++ 1 file changed, 260 insertions(+) create mode 100644 .cursor/skills/examining-ado-operations/SKILL.md diff --git a/.cursor/skills/examining-ado-operations/SKILL.md b/.cursor/skills/examining-ado-operations/SKILL.md new file mode 100644 index 000000000..0428adb06 --- /dev/null +++ b/.cursor/skills/examining-ado-operations/SKILL.md @@ -0,0 +1,260 @@ +--- +name: examining-ado-operations +description: >- + End-to-end workflow to examine and summarize an ado operation — fetch + operation and space YAML, summarise configuration, export + entities/requests/results to CSV, perform simple analysis, and interpret + failures and data quality. Use when the user asks to summarize, analyse, + debug, or review an operation; wants insights from measurement data; or + provides an operation ID or asks to use --use-latest for the current + operation. +--- + +# Examining ADO operations + +Structured workflow for understanding what an operation did, which space it ran +on, and whether measurements and results look healthy. Run all commands from the +**repository root** with `uv run`. + +**Related skills**: + +- For CLI verification and command spelling, see + [using-ado-cli](../using-ado-cli/SKILL.md). +- For metastore filtering, schemas see + [query-ado-data](../query-ado-data/SKILL.md). + +## Context + +Operations are applied to discoveryspaces. There are different types of +operation. The General Workflow can be applied to all types of operation. + +In addition, the Explore/Search Workflow operations can be applied to +Explore/Search operations. + +- Read [operations](../../../website/docs/resources/operation.md) documentation + for details + +## Pre-requisites: The Operation Identifier + +To apply this skill you need either: + +(a) an operation id; (b) explicit instruction to examine the latest operation + +In the case of (b) (latest) get the actual operation identifier as follows + +```bash +uv run ado show related operation --use-latest +``` + +This will output the id of the latest operation along + +## General Workflow + +### Step 1: Get the operation YAML + +```bash +uv run ado get operation OPERATION_ID -o yaml +``` + +Extract and summarise: + +- Resource **identifier**, **operationType**, **operatorIdentifier** +- The identifiers of **input resources** to the operation: + - discovery spaces from the spaces field) + - actuatorconfigurations from the actuatorConfigurationIdentifiers field +- **config**: operator-specific parameters (what the run was asked to do) +- **status**: latest **event** (e.g. started / finished) and **exit_state** when + finished (success / fail / error) + +Note anything in config that influences what operation does (thresholds, +objectives, stopping rules, etc.). + +#### Identifying if an operation is still running + +An operation which does not report finished is usually still running. + +However it is possible it failed in a way that meant it could not record the +failure. In this case: + +1. Determine how long it has been running. +2. If it is many hours and the operationType is not search/explore flag that + there may be a problem +3. If it is many hours and the operationType is search/explore proceed use + specific techniques in Explore Operation Workflow to determine if its still + running + +### Step 2: Review the operator + +Each operation is run by an operator. The operators name is retrieved in +previous step (operatorIdentifier) + +Execute + +```commandline +uv run ado get operator --details $OPERATOR_IDENTIFIER +``` + +#### Understanding operator parameters + +To understand an operators parameters examine its schema: + +```commandline +uv run ado template operation --operator-name $OPERATOR_IDENTIFER --include-schema +``` + +This will create a file called operation*template*$UID_schema.yaml containing +the schema. + +### Step 3: Describe the space + +Using the space id from step 1 + +```bash +uv run ado get space SPACE_ID -o yaml +uv run ado describe space SPACE_ID [--use-latest] +``` + +Summarise the: **dimensions** (parameters), **experiments** (actuators, +experiment types), **entity space** structure, and notable **constraints** or +metadata. For deeper context, read operator and experiment documentation under +`website/docs/operators/` and actuator/experiment docs as needed (match +**operatorIdentifier** and experiment types from the space). + +### Step 4: Get the output resources of the operation + +Operations can create other resources. To identify these + +```bash +uv run ado show related operation $OPERATION_IDENTIFIER +``` + +This will output the identifiers of the input and output resources related to +the operation. + +From step 1 you know the input resource identifiers so you can work out the +output identifiers. + +### Step 5: Examine the output resources of the operation (if any) + +An operation can create the following resources + +- discovery spaces: In this case examine the space as in step 2 +- operations: In this case recursively examine the operations using this skill +- datacontainers: This contains non-ado resource outputs e.g. CSV data. + +To retrieve contents of data container + +```commandline +uv run ado get datacontainer -o yaml $DATACONTAINER_IDENTIFIER +``` + +For each output resource summarize what it is/contains. + +## Explore/Search Operation Workflow + +The following assumes the General Workflow has been applied. + +Explore/Search operations sample entities from a discovery space and make +measurements on them. + +Notes: + +- If the data for the measurements exists it can be memoized (depends on + operation parameters) +- The operation parameters will specify the number of entities to sample in some + way. + +Relevant Documentation + +- [sample process](../../../website/docs/core-concepts/discovery-spaces.md#sampling-and-measurement) +- [memoization](../../../website/docs/core-concepts/data-sharing.md#memoization) + +### Step 1: Get Details on what was Sampled and Measured + +```bash +uv run ado show details operation $OPERATION_ID +``` + +Compare this with the number of samples requested in the operator parameters. + +If the state is finished, exit status was successful and all requested samples +were completed there are no issues -> +[examine entities](#step-3-get-entities-and-measurements) + +If the state is not finished -> +[Diagnose if sampling operation running](#diagnose-if-operation-running) For all +other combination -> [Diagnose sampling issues](#diagnose-sampling-issues) + +### Step 2 (Optional): Diagnose issues + +First run these two commands to get the metadata on what was requested and +measured + +```bash +uv run ado show requests operation OPERATION_ID \ + --output-format csv +uv run ado show results operation OPERATION_ID \ + --output-format csv +``` + +- **requests**: This is metadata on what the sampling operation asked an + actuator to measure. It includes the timestamp of when the request was + created - at the moment the comp The requests contain the results of executing + the request +- **results**: This is metadata on an actual measurement triggered by a request + - ValidMeasurementResult: The experiment executed and return one or more + observed property values + - InvalidMeasurementResult: The experiment failed for some reason + +#### Diagnose if operation running + +- Check if the operation is submitting experiment in batches +- Confirm if the operation uses continuous batching (new experiment requested + once one has finished) or static batch (full batch finishes then next starts) +- For continuous batching + - Use the request time-series to determine the typical inter-request start + time after the first batch i.e. this tells you how often after the first + batch you should expect to see a new request +- For static batch + - Use the request time-series to determine the typical inter-batch time i.e. + how long between batches/how long a batch takes to execute on average +- Determine if the time since last recorded request is much greater than the + expected inter request time e.g. 5x more. This indicates there may have been + an issue. + +#### Diagnose sampling issues + +From the output of `show request` and `show results` + +Identify: **failed** or **invalid** rows, **reasons** for invalidity, and +anomalies in **timing** or **ordering** if those columns are present. + +### Step 3: Get entities and measurements + +To get the data on measurements + +```bash +uv run ado show entities operation OPERATION_ID [--use-latest] \ + --output-format csv +``` + +Perform quick analysis: distributions, outliers, correlations. + +## Producing a report + +Structure the report as: + +1. **Overview**: What the operation purpose was. Can be inferred from space and + operation chosen. Short and narrative. 1a. **Operation summary** – ID, + operator, parameters, status 2b. **Space summary** – dimensions, experiments, + entity count +2. **Measurement overview** – sampled vs requested, success vs failure counts +3. **Findings** – notable patterns, best/worst performers, anomalies +4. **Unusual behaviour** – failures, timeouts, invalid results, unexpected + distributions + +## Guidelines + +- Use **`uv run`** for every `ado` invocation from repo root. +- Confirm any unfamiliar subcommand or flag with `uv run ado ... --help` per + [using-ado-cli](../using-ado-cli/SKILL.md). From a9f63a77891ebdfeecf45a6b61420ff860b0c46c Mon Sep 17 00:00:00 2001 From: michaelj Date: Wed, 25 Mar 2026 14:09:53 +0000 Subject: [PATCH 02/14] docs(agents): add operation analyzer agent --- .cursor/agents/operation-analyzer.md | 43 ++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 .cursor/agents/operation-analyzer.md diff --git a/.cursor/agents/operation-analyzer.md b/.cursor/agents/operation-analyzer.md new file mode 100644 index 000000000..1e8402e67 --- /dev/null +++ b/.cursor/agents/operation-analyzer.md @@ -0,0 +1,43 @@ +--- +name: operation-analyzer +description: + Analyzes the results of ado operations on discovery spaces and plans next + research steps. Use proactively when the user wants to analyze an operation, + understand measurement outcomes, and decide what to do next +--- + +# Operation Examiner + +You are an Operation Analyzer for ado - a tool for computational +experimentation. You analyse operations on discovery spaces and produce +structured reports summarizing results, highlighting unusual behaviour, and plan +next steps. + +**Skills to apply**: + +The main skill is +[examining-ado-operations](../skills/examining-ado-operations). This will +provide the basis for the analysis. + +Also see using-ado-cli, query-ado-data, formulate-discovery-problem + +## Workflow + +- Use [examining-ado-operations](../skills/examining-ado-operations) to get + overview of the operation(s) in question +- Determine if further analysis is required +- If yes, before coding any analysis scripts + - Check available ado operators (ado get operators --details) to see if any + should be applied + - Check available ado operators if any should be extended with necessary + analysis + - Plan with user if they want to develop the necessary analysis as an operator + or just go ahead with inpdendent scripts +- Execute the analysis +- Produce a plan for next research steps using ado. + - Interact with user if input is required to refine plan + +## References + +- see [plugin-development](../rules/plugin-development.mdc) for details on + developing operator plugins From 69f031f365311b9c99253080e7289475f0e62d9a Mon Sep 17 00:00:00 2001 From: michaelj Date: Wed, 25 Mar 2026 15:27:08 +0000 Subject: [PATCH 03/14] docs(skills): updates --- .../skills/examining-ado-operations/SKILL.md | 74 +++++++++--------- .../formulate-discovery-problem/SKILL.md | 20 ++--- .../formulate-discovery-problem/examples.md | 6 +- .../formulate-discovery-problem/reference.md | 28 ++++--- .../reference-property-domain.yaml | 2 +- .cursor/skills/query-ado-data/SKILL.md | 75 ++++++++++--------- .cursor/skills/remote-execution/SKILL.md | 41 +++++----- .../skills/resource-yaml-creation/SKILL.md | 41 +++++----- .cursor/skills/using-ado-cli/SKILL.md | 33 ++++---- 9 files changed, 166 insertions(+), 154 deletions(-) diff --git a/.cursor/skills/examining-ado-operations/SKILL.md b/.cursor/skills/examining-ado-operations/SKILL.md index 0428adb06..be321d261 100644 --- a/.cursor/skills/examining-ado-operations/SKILL.md +++ b/.cursor/skills/examining-ado-operations/SKILL.md @@ -46,7 +46,7 @@ In the case of (b) (latest) get the actual operation identifier as follows uv run ado show related operation --use-latest ``` -This will output the id of the latest operation along +This will output the id of the latest operation. ## General Workflow @@ -90,7 +90,7 @@ previous step (operatorIdentifier) Execute -```commandline +```bash uv run ado get operator --details $OPERATOR_IDENTIFIER ``` @@ -98,8 +98,8 @@ uv run ado get operator --details $OPERATOR_IDENTIFIER To understand an operators parameters examine its schema: -```commandline -uv run ado template operation --operator-name $OPERATOR_IDENTIFER --include-schema +```bash +uv run ado template operation --operator-name $OPERATOR_IDENTIFIER --include-schema ``` This will create a file called operation*template*$UID_schema.yaml containing @@ -111,7 +111,7 @@ Using the space id from step 1 ```bash uv run ado get space SPACE_ID -o yaml -uv run ado describe space SPACE_ID [--use-latest] +uv run ado describe space SPACE_ID ``` Summarise the: **dimensions** (parameters), **experiments** (actuators, @@ -138,13 +138,13 @@ output identifiers. An operation can create the following resources -- discovery spaces: In this case examine the space as in step 2 +- discovery spaces: In this case examine the space as in step 3 - operations: In this case recursively examine the operations using this skill - datacontainers: This contains non-ado resource outputs e.g. CSV data. To retrieve contents of data container -```commandline +```bash uv run ado get datacontainer -o yaml $DATACONTAINER_IDENTIFIER ``` @@ -182,10 +182,11 @@ were completed there are no issues -> [examine entities](#step-3-get-entities-and-measurements) If the state is not finished -> -[Diagnose if sampling operation running](#diagnose-if-operation-running) For all -other combination -> [Diagnose sampling issues](#diagnose-sampling-issues) +[Use the diagnose if sampling operation running workflow](#diagnose-if-explore-or-search-operation-is-running-workflow). +For all other combinations -> +[Diagnose sampling issues](#step-2-optional-diagnose-sampling-issues) -### Step 2 (Optional): Diagnose issues +### Step 2 (Optional): Diagnose sampling issues First run these two commands to get the metadata on what was requested and measured @@ -199,31 +200,13 @@ uv run ado show results operation OPERATION_ID \ - **requests**: This is metadata on what the sampling operation asked an actuator to measure. It includes the timestamp of when the request was - created - at the moment the comp The requests contain the results of executing - the request + created - at the moment the completion time is not available. The requests + contain the results of executing the request - **results**: This is metadata on an actual measurement triggered by a request - ValidMeasurementResult: The experiment executed and return one or more observed property values - InvalidMeasurementResult: The experiment failed for some reason -#### Diagnose if operation running - -- Check if the operation is submitting experiment in batches -- Confirm if the operation uses continuous batching (new experiment requested - once one has finished) or static batch (full batch finishes then next starts) -- For continuous batching - - Use the request time-series to determine the typical inter-request start - time after the first batch i.e. this tells you how often after the first - batch you should expect to see a new request -- For static batch - - Use the request time-series to determine the typical inter-batch time i.e. - how long between batches/how long a batch takes to execute on average -- Determine if the time since last recorded request is much greater than the - expected inter request time e.g. 5x more. This indicates there may have been - an issue. - -#### Diagnose sampling issues - From the output of `show request` and `show results` Identify: **failed** or **invalid** rows, **reasons** for invalidity, and @@ -234,27 +217,38 @@ anomalies in **timing** or **ordering** if those columns are present. To get the data on measurements ```bash -uv run ado show entities operation OPERATION_ID [--use-latest] \ +uv run ado show entities operation OPERATION_ID \ --output-format csv ``` Perform quick analysis: distributions, outliers, correlations. +## Diagnose if Explore or Search Operation is Running Workflow + +- Check if the operation is submitting experiment in batches +- Confirm if the operation uses continuous batching (new experiment requested + once one has finished) or static batch (full batch finishes then next starts) +- Get the requests and results timeseries using ado show cli command +- For continuous batching + - Use the request time-series to determine the typical inter-request start + time after the first batch i.e. this tells you how often after the first + batch you should expect to see a new request +- For static batch + - Use the request time-series to determine the typical inter-batch time i.e. + how long between batches/how long a batch takes to execute on average +- Determine if the time since last recorded request is much greater than the + expected inter request time e.g. 5x more. This indicates there may have been + an issue. + ## Producing a report Structure the report as: 1. **Overview**: What the operation purpose was. Can be inferred from space and - operation chosen. Short and narrative. 1a. **Operation summary** – ID, - operator, parameters, status 2b. **Space summary** – dimensions, experiments, - entity count + operation chosen. Short and narrative. + - **Operation summary** – ID, operator, parameters, status + - **Space summary** – dimensions, experiments, entity count 2. **Measurement overview** – sampled vs requested, success vs failure counts 3. **Findings** – notable patterns, best/worst performers, anomalies 4. **Unusual behaviour** – failures, timeouts, invalid results, unexpected distributions - -## Guidelines - -- Use **`uv run`** for every `ado` invocation from repo root. -- Confirm any unfamiliar subcommand or flag with `uv run ado ... --help` per - [using-ado-cli](../using-ado-cli/SKILL.md). diff --git a/.cursor/skills/formulate-discovery-problem/SKILL.md b/.cursor/skills/formulate-discovery-problem/SKILL.md index 707ebdca4..350617eff 100644 --- a/.cursor/skills/formulate-discovery-problem/SKILL.md +++ b/.cursor/skills/formulate-discovery-problem/SKILL.md @@ -15,11 +15,11 @@ to formulate problems for execution with ado. ## Plan Mode -If in planning mode consider the following three phases. -In each phase consider getting the users input on the following -while building the plan. +If in planning mode consider the following three phases. In each phase consider +getting the users input on the following while building the plan. -1. Coding Phase: If custom experiments, actuators or operators need to be created +1. Coding Phase: If custom experiments, actuators or operators need to be + created - Should experiments be implemented as actuators or custom experiments - If certain parameters should be required or optional in a custom experiment @@ -31,7 +31,7 @@ See [plugin-development.mdc](../../rules/plugin-development.mdc) coding skills. - If an optional property of experiment should be added to the entity space - What exploration technique they want to use e.g. random search, space filling, -multi-objective optimization + multi-objective optimization - Values for actuator configuration 1. Execution Phase: If the user wants to execute an operation @@ -46,11 +46,11 @@ See [remote execution](../remote-execution/SKILL.md) for remote execution skills - Unless directed otherwise place all YAML and .md files created in a subdirectory of examples/ dedicated to the given problem. -- If you want to change the default value of an optional property use - experiment parameterization, rather than setting a single valued property in - the entity space -- Before creating a space or actuator configuration, check if one already - exists — see [resource-yaml-creation](../resource-yaml-creation/SKILL.md) +- If you want to change the default value of an optional property use experiment + parameterization, rather than setting a single valued property in the entity + space +- Before creating a space or actuator configuration, check if one already exists + — see [resource-yaml-creation](../resource-yaml-creation/SKILL.md) - Learn [ado CLI command-line construction and testing](../using-ado-cli/) - For metadata conventions, dynamic references (--use-latest, --with, --set), and resource-specific guidance, see diff --git a/.cursor/skills/formulate-discovery-problem/examples.md b/.cursor/skills/formulate-discovery-problem/examples.md index 67729fae9..2278280d9 100644 --- a/.cursor/skills/formulate-discovery-problem/examples.md +++ b/.cursor/skills/formulate-discovery-problem/examples.md @@ -4,7 +4,8 @@ Concrete examples of formulating problems for ado execution. ## Example 1: Simple Hyperparameter Optimization -**User Query:** "I want to optimize learning rate and batch size for model training" +**User Query:** "I want to optimize learning rate and batch size for model +training" ### Step 1: Choose Experiments @@ -174,7 +175,8 @@ uv run ado create space -f space.yaml --dry-run ### Step 2: Create Actuator Configuration -See [example6-actuator-config.yaml](yaml-examples/example6-actuator-config.yaml). +See +[example6-actuator-config.yaml](yaml-examples/example6-actuator-config.yaml). ```bash uv run ado create actuatorconfiguration -f gpu_config.yaml diff --git a/.cursor/skills/formulate-discovery-problem/reference.md b/.cursor/skills/formulate-discovery-problem/reference.md index 22851a233..b5082948a 100644 --- a/.cursor/skills/formulate-discovery-problem/reference.md +++ b/.cursor/skills/formulate-discovery-problem/reference.md @@ -31,13 +31,15 @@ Detailed reference information for formulating problems in ado. ### Experiment Reference Format -See [reference-experiment-format.yaml](yaml-examples/reference-experiment-format.yaml). +See +[reference-experiment-format.yaml](yaml-examples/reference-experiment-format.yaml). ## Entity Space Property Schema ### ConstitutiveProperty Structure -See [reference-property-domain.yaml](yaml-examples/reference-property-domain.yaml). +See +[reference-property-domain.yaml](yaml-examples/reference-property-domain.yaml). ### Variable Types @@ -67,7 +69,8 @@ See [reference-property-domain.yaml](yaml-examples/reference-property-domain.yam ### Core Structure -See [reference-operation-structure.yaml](yaml-examples/reference-operation-structure.yaml). +See +[reference-operation-structure.yaml](yaml-examples/reference-operation-structure.yaml). ### Operation Types @@ -114,8 +117,7 @@ property domain: - For discrete: All entity space values must be in experiment's domain values - For continuous: Entity space range must be within experiment's range -- For categorical: Entity space values must be subset of experiment's - values +- For categorical: Entity space values must be subset of experiment's values ### Validation @@ -123,27 +125,31 @@ ado validates: 1. All required constitutive properties are in entity space 2. Entity space domains are compatible (subdomain check) -3. No redundant dimensions (all entity space properties required by at least - one experiment) +3. No redundant dimensions (all entity space properties required by at least one + experiment) 4. Optional properties in entity space don't conflict with parameterization ## Common Patterns ### Pattern 1: Single Experiment, Simple Space -See [reference-pattern1-simple-space.yaml](yaml-examples/reference-pattern1-simple-space.yaml). +See +[reference-pattern1-simple-space.yaml](yaml-examples/reference-pattern1-simple-space.yaml). ### Pattern 2: Multiple Experiments with Dependencies -See [reference-pattern2-multiple-experiments.yaml](yaml-examples/reference-pattern2-multiple-experiments.yaml). +See +[reference-pattern2-multiple-experiments.yaml](yaml-examples/reference-pattern2-multiple-experiments.yaml). ### Pattern 3: Parameterized Experiment -See [reference-pattern3-parameterized.yaml](yaml-examples/reference-pattern3-parameterized.yaml). +See +[reference-pattern3-parameterized.yaml](yaml-examples/reference-pattern3-parameterized.yaml). ### Pattern 4: Optional Property in Entity Space -See [reference-pattern4-optional-property.yaml](yaml-examples/reference-pattern4-optional-property.yaml). +See +[reference-pattern4-optional-property.yaml](yaml-examples/reference-pattern4-optional-property.yaml). ## Validation Commands Reference diff --git a/.cursor/skills/formulate-discovery-problem/yaml-examples/reference-property-domain.yaml b/.cursor/skills/formulate-discovery-problem/yaml-examples/reference-property-domain.yaml index 6fe319f8d..f68632369 100644 --- a/.cursor/skills/formulate-discovery-problem/yaml-examples/reference-property-domain.yaml +++ b/.cursor/skills/formulate-discovery-problem/yaml-examples/reference-property-domain.yaml @@ -4,5 +4,5 @@ identifier: property_name # Required - unique identifier propertyDomain: # Required - domain definition variableType: DISCRETE_VARIABLE_TYPE | CONTINUOUS_VARIABLE_TYPE | CATEGORICAL_VARIABLE_TYPE | BINARY_VARIABLE_TYPE domainRange: [min, max] # For discrete/continuous numeric - values: [val1, val2, '...'] # For discrete/categorical + values: [val1, val2, "..."] # For discrete/categorical interval: step_size # Optional - for discrete numeric diff --git a/.cursor/skills/query-ado-data/SKILL.md b/.cursor/skills/query-ado-data/SKILL.md index 970f0c6b8..8f49cd036 100644 --- a/.cursor/skills/query-ado-data/SKILL.md +++ b/.cursor/skills/query-ado-data/SKILL.md @@ -1,6 +1,11 @@ --- name: query-ado-data -description: Query ado metadata and measurement data using CLI commands. Use when the user needs to find resources, filter by metadata, retrieve entities and measurements, or get resource schemas. Covers metastore queries (operations, discoveryspaces, samplestores, datacontainers, actuatorconfigurations) and samplestore queries (entities and measurements). +description: + Query ado metadata and measurement data using CLI commands. Use when the user + needs to find resources, filter by metadata, retrieve entities and + measurements, or get resource schemas. Covers metastore queries (operations, + discoveryspaces, samplestores, datacontainers, actuatorconfigurations) and + samplestore queries (entities and measurements). --- # Query ado Data @@ -8,7 +13,7 @@ description: Query ado metadata and measurement data using CLI commands. Use whe ado stores data in two places: 1. **Metastore**: Metadata about all resources (operations, discoveryspaces, -samplestores, datacontainers, actuatorconfigurations) + samplestores, datacontainers, actuatorconfigurations) 2. **Samplestores**: Entities and measurements made on them ## Guidelines @@ -23,28 +28,29 @@ samplestores, datacontainers, actuatorconfigurations) DOs: - IMPORTANT Before deciding on what to query check the resource schema to - confirm what is available in metadata - - ado template RESOURCETYPE --include-schema + confirm what is available in metadata - ado template RESOURCETYPE + --include-schema - Use Server side filtering - - prefer --query or --matching to fetching metadata and filtering on - client side + - prefer --query or --matching to fetching metadata and filtering on client + side - Fetch metadata over fetching data - - if a query can be answered via metadata it is much faster - - filter via metadata first if possible, before obtaining data - - IMPORTANT: ado show details space can be slow as it internally fetches - spaces data to calculate - prefer using metadata -- Consider writing a script directly using SQLResourceStore API if the CLI - is not expressive enough BEFORE fetching data - - you can make batch requests e.g. getResources - much faster than - one-by-one requests + - if a query can be answered via metadata it is much faster + - filter via metadata first if possible, before obtaining data + - IMPORTANT: ado show details space can be slow as it internally fetches + spaces data to calculate - prefer using metadata +- Consider writing a script directly using SQLResourceStore API if the CLI is + not expressive enough BEFORE fetching data + - you can make batch requests e.g. getResources - much faster than one-by-one + requests DONTs - Do not fetch discoveryspace or operation data for summary queries - - Do not use: ado show entities, ado show requests, ado show results, - ado show details) + - Do not use: ado show entities, ado show requests, ado show results, ado show + details) - Do not instantiating DiscoverySpace instances or SQLStore instance -- Only use these commands or classes when drilling down on a narrow set of resources +- Only use these commands or classes when drilling down on a narrow set of + resources ### Using Resource models @@ -52,13 +58,11 @@ Each resource has a pydantic model. If working in code you can use these models - discoveryspace, orchestrator/core/discoveryspace/resource.py: DiscoverySpaceResource -- samplestore, orchestrator/core/samplestore/resource.py: - SampleStoreResource +- samplestore, orchestrator/core/samplestore/resource.py: SampleStoreResource - datacontainer, orchestrator/core/datacontainer/resource.py: DataContainerResource - operation, orchestrator/core/operation/resource.py: OperationResource -- actuatorconfiguration, - orchestrator/core/actuatorconfiguration/resource.py: +- actuatorconfiguration, orchestrator/core/actuatorconfiguration/resource.py: ActuatorConfigurationResource ## Querying Metadata @@ -71,10 +75,12 @@ Get a general overview of what's present: uv run ado get $RESOURCETYPE --details ``` -Returns an age-sorted list (most recent last) of resources of the specified type. +Returns an age-sorted list (most recent last) of resources of the specified +type. **Resource types**: `operations` (`op`), `discoveryspaces` (`space`), -`samplestores` (`store`), `datacontainers` (`dcr`), `actuatorconfigurations` (`ac`) +`samplestores` (`store`), `datacontainers` (`dcr`), `actuatorconfigurations` +(`ac`) ### Filtering Resources @@ -84,7 +90,8 @@ Filter resources based on metadata fields using MySQL JSON Path queries: uv run ado get $RESOURCETYPE --query 'path=candidate' ``` -- Use single quotes around the candidate (required for strings, dictionaries, arrays) +- Use single quotes around the candidate (required for strings, dictionaries, + arrays) - Path is dot-separated (e.g., `config.metadata.labels`) - Candidate is a valid JSON value - Can specify `--query` multiple times (all filters must match) @@ -99,7 +106,7 @@ uv run ado get operations -q 'config.operation.module.moduleClass=RayTune' uv run ado get spaces -q 'config.experiments={"experiments":{"identifier":"finetune-lora-fsdp-r-4-a-16-tm-default-v2.0.0"}}' # Combine multiple filters -uv run ado get operations -q 'config.operation.parameters.batchSize=1' +uv run ado get operations -q 'config.operation.parameters.batchSize=1' -q 'status=[{"event": "finished", "exit_state": "success"}]' ``` @@ -134,8 +141,8 @@ uv run ado get space --matching-space-id space-abc123-456def uv run ado get space --matching-space space.yaml ``` -**Note**: `--matching-point`, `--matching-space`, and `--matching-space-id` -are exclusive to spaces and override `--query` and `--label`. +**Note**: `--matching-point`, `--matching-space`, and `--matching-space-id` are +exclusive to spaces and override `--query` and `--label`. ### Related Resources @@ -182,8 +189,8 @@ uv run ado show entities $RESOURCETYPE [RESOURCE_ID] [--use-latest] \ **Key options:** - `--include` (spaces only): `sampled`, `unsampled`, `matching`, `missing` -- `--property-format`: `observed` (one row per entity) or `target` -(one row per entity-experiment pair) +- `--property-format`: `observed` (one row per entity) or `target` (one row per + entity-experiment pair) - `--output-format`: `console`, `csv`, or `json` - `--property`: Filter specific properties (can specify multiple times) - `--aggregate`: Aggregate multiple values @@ -273,7 +280,7 @@ uv run ado get spaces -q 'config.entitySpace={"propertyDomain":{"values":["mistr ### Export operation entities to CSV ```bash -uv run ado show entities operation OPERATION_ID --output-format csv +uv run ado show entities operation OPERATION_ID --output-format csv ``` ### Get all resources related to a space @@ -284,13 +291,13 @@ uv run ado show related space SPACE_ID ## Advanced Filtering -The metastore class can provide more powerful querying via scripts. -See orchestrator/metastore/sqlstore.py +The metastore class can provide more powerful querying via scripts. See +orchestrator/metastore/sqlstore.py ## References When modifying or creating code while using this skill, follow: - [AGENTS.md](../../../AGENTS.md) -- [plugin-development.mdc](../../rules/plugin-development.mdc) (if working - with plugins) +- [plugin-development.mdc](../../rules/plugin-development.mdc) (if working with + plugins) diff --git a/.cursor/skills/remote-execution/SKILL.md b/.cursor/skills/remote-execution/SKILL.md index 2552d8868..f5d4d5b9e 100644 --- a/.cursor/skills/remote-execution/SKILL.md +++ b/.cursor/skills/remote-execution/SKILL.md @@ -1,10 +1,11 @@ --- name: remote-execution -description: Run ado operations on remote Ray clusters using --remote execution context - files. Use when the user wants to create an operation, asks about remote clusters, - wants to ship local plugins or data files to a cluster, or asks about execution context - YAML files. Also applies proactively when creating an operation if execution context - files are present in the workspace. +description: + Run ado operations on remote Ray clusters using --remote execution context + files. Use when the user wants to create an operation, asks about remote + clusters, wants to ship local plugins or data files to a cluster, or asks + about execution context YAML files. Also applies proactively when creating an + operation if execution context files are present in the workspace. --- # Remote Execution with ado @@ -31,9 +32,9 @@ See `website/docs/getting-started/remote_run.md` for full schema reference. ## Proactive prompt when creating operations When the user asks to create an operation, **check the repo root for -`*_execution.yaml` files**. If any exist, ask which (if any) they want to -use before proceeding. Do not assume remote by default — local execution is -still common. +`*_execution.yaml` files**. If any exist, ask which (if any) they want to use +before proceeding. Do not assume remote by default — local execution is still +common. Do **not** dispatch other ado commands (`get`, `show`, `create space`, etc.) remotely unless the user explicitly requests it. @@ -70,8 +71,8 @@ uv run ado --remote morrigan_execution.yaml create operation \ -f operation.yaml --use-latest space ``` -Only supply `-c context.yaml` when you need to target a different project -than the one currently active. +Only supply `-c context.yaml` when you need to target a different project than +the one currently active. --- @@ -106,21 +107,21 @@ Any file path appearing in a space, operation, or actuator configuration YAML (e.g. `mps_file`, a model checkpoint, a dataset path) must satisfy **both** conditions for the remote job to succeed: -- **File not present on cluster**: add the local path to `additionalFiles` - in the execution context YAML. +- **File not present on cluster**: add the local path to `additionalFiles` in + the execution context YAML. - **Path invalid on cluster**: use a bare filename in the YAML; ado symlinks - `additionalFiles` entries into the Ray working dir, so `my-file.gz` - resolves but `/Users/me/data/my-file.gz` does not. + `additionalFiles` entries into the Ray working dir, so `my-file.gz` resolves + but `/Users/me/data/my-file.gz` does not. Failing either condition produces a file-not-found error at experiment runtime, not at submission time, so the job starts successfully but measurements fail. #### Pattern to follow -To avoid, if the experiment references a file use a **bare -filename** (no path) in the space/operation YAML. Add the absolute local path -to `additionalFiles`; ado symlinks it into the Ray working directory so the -bare filename resolves on the cluster. +To avoid, if the experiment references a file use a **bare filename** (no path) +in the space/operation YAML. Add the absolute local path to `additionalFiles`; +ado symlinks it into the Ray working directory so the bare filename resolves on +the cluster. ```yaml # space.yaml @@ -129,7 +130,7 @@ entitySpace: propertyDomain: variableType: OPEN_CATEGORICAL_VARIABLE_TYPE values: - - pigeon-10.mps.gz # bare filename — resolves from working dir + - pigeon-10.mps.gz # bare filename — resolves from working dir ``` ```yaml @@ -150,6 +151,6 @@ If you see `Changing the ray version is not allowed`, pin the Ray version in ```yaml fromPyPI: - ado-core - - ray==2.52.1 # match the cluster's installed version + - ray==2.52.1 # match the cluster's installed version - ado-ray-tune ``` diff --git a/.cursor/skills/resource-yaml-creation/SKILL.md b/.cursor/skills/resource-yaml-creation/SKILL.md index f48a03dc6..78eeed130 100644 --- a/.cursor/skills/resource-yaml-creation/SKILL.md +++ b/.cursor/skills/resource-yaml-creation/SKILL.md @@ -10,22 +10,23 @@ description: | # Creating ado Resource YAML Files -For CLI command syntax, see [using-ado-cli](../using-ado-cli/SKILL.md). -For full problem formulation workflow, see +For CLI command syntax, see [using-ado-cli](../using-ado-cli/SKILL.md). For full +problem formulation workflow, see [formulate-discovery-problem](../formulate-discovery-problem/SKILL.md). ## Metadata Fields Every resource YAML should include a `metadata` block. The CLI uses these for -display (`ado get --details`) and filtering (`ado get --label`, `ado get --query`). +display (`ado get --details`) and filtering (`ado get --label`, +`ado get --query`). ```yaml metadata: - name: my_space # short human-readable identifier - description: | # longer explanation of purpose + name: my_space # short human-readable identifier + description: | # longer explanation of purpose Optimize learning rate and batch size for ResNet training. labels: - project: my_project # arbitrary key=value pairs for filtering + project: my_project # arbitrary key=value pairs for filtering team: ml_team ``` @@ -49,8 +50,8 @@ uv run ado create space -f space.yaml uv run ado create operation -f operation.yaml --use-latest space ``` -> **Context scoping**: `--use-latest` resolves relative to the *execution -> context*, not the project. On a laptop, it finds the last resource created +> **Context scoping**: `--use-latest` resolves relative to the _execution +> context_, not the project. On a laptop, it finds the last resource created > locally. When launched with `--remote`, it finds the last resource created on > the remote cluster. Keep this in mind when reusing resources across > environments. @@ -77,8 +78,8 @@ uv run ado create operation -f operation.yaml \ ### --set -Overrides individual fields in the YAML at creation time without editing -the file. Useful for environment-specific values or quick one-off changes. +Overrides individual fields in the YAML at creation time without editing the +file. Useful for environment-specific values or quick one-off changes. ```bash # Override the sample store identifier @@ -122,7 +123,8 @@ uv run ado template actuatorconfiguration --actuator-identifier my_actuator ### DiscoverySpace -**Before creating** (ado create space), check if a matching space already exists: +**Before creating** (ado create space), check if a matching space already +exists: ```bash # Match by space config (entity space + experiments) @@ -143,9 +145,8 @@ operation benefits from measurements already collected. - Declare a property as a constitutive property domain in the entity space when you want to **explore a range of values** for that property. - Use **experiment parameterization** when you want to change the default value - of an optional experiment property but keep it fixed across all entities. - Do not add a single-valued domain to the entity space just to override a - default. + of an optional experiment property but keep it fixed across all entities. Do + not add a single-valued domain to the entity space just to override a default. ```yaml # Correct: parameterization overrides experiment default, keeps it out of space @@ -155,14 +156,14 @@ experiments: parameterization: - property: identifier: optimizer - value: adam # overrides default "sgd" + value: adam # overrides default "sgd" # Incorrect: single-valued domain should be parameterization instead entitySpace: - identifier: optimizer propertyDomain: variableType: DISCRETE_VARIABLE_TYPE - values: [adam] # single value — use parameterization instead + values: [adam] # single value — use parameterization instead ``` **Creating a space with a fresh samplestore**: @@ -180,8 +181,8 @@ uv run ado get actuatorconfigurations --details uv run ado get actuatorconfigurations --label actuator=my_actuator ``` -Reuse an existing actuator configuration when appropriate rather than -creating duplicates. +Reuse an existing actuator configuration when appropriate rather than creating +duplicates. ### SampleStore @@ -198,6 +199,6 @@ uv run ado create samplestore -f samplestore.yaml ## Related Resources - [using-ado-cli](../using-ado-cli/SKILL.md) — CLI command syntax and shortcuts -- [formulate-discovery-problem](../formulate-discovery-problem/SKILL.md) — - full problem formulation workflow +- [formulate-discovery-problem](../formulate-discovery-problem/SKILL.md) — full + problem formulation workflow - [AGENTS.md](../../../AGENTS.md) — YAML testing and linting guidance diff --git a/.cursor/skills/using-ado-cli/SKILL.md b/.cursor/skills/using-ado-cli/SKILL.md index 635b0546e..7e18d1e21 100644 --- a/.cursor/skills/using-ado-cli/SKILL.md +++ b/.cursor/skills/using-ado-cli/SKILL.md @@ -33,23 +33,24 @@ uv run ado [COMMAND] [SUBCOMMAND1] [SUBCOMMAND2] --help These plausible-sounding commands do not exist in ado. Do not write them: -| ❌ Does not exist | ✅ Correct equivalent | -| --- | --- | -| `ado run` | `ado create operation -f op.yaml` | -| `ado start` | `ado create operation -f op.yaml` | -| `ado execute` | `ado create operation -f op.yaml` | -| `ado launch` | `ado create operation -f op.yaml` | -| `ado list` | `ado get spaces` / `ado get operations` | -| `ado status` | `ado show details space SPACE_ID` | - -**Key principle**: `ado create operation` both *defines* and *starts* the operation -in a single command. There is no separate "run" step. +| ❌ Does not exist | ✅ Correct equivalent | +| ----------------- | --------------------------------------- | +| `ado run` | `ado create operation -f op.yaml` | +| `ado start` | `ado create operation -f op.yaml` | +| `ado execute` | `ado create operation -f op.yaml` | +| `ado launch` | `ado create operation -f op.yaml` | +| `ado list` | `ado get spaces` / `ado get operations` | +| `ado status` | `ado show details space SPACE_ID` | + +**Key principle**: `ado create operation` both _defines_ and _starts_ the +operation in a single command. There is no separate "run" step. ## Point Testing with run_experiment `run_experiment` is a **separate CLI entry point** (not an `ado` subcommand) for running a single entity through an experiment locally, without creating a space -or operation. It is the correct tool for functional validation of custom experiments. +or operation. It is the correct tool for functional validation of custom +experiments. ```bash uv run run_experiment PATH_TO_POINT_YAML @@ -120,18 +121,18 @@ Outputs a human readable description ```bash # Output a description of a space -# Dimensions, values, experiments +# Dimensions, values, experiments uv run ado describe space SPACE_ID -#Output a description of an experiment +#Output a description of an experiment # (input params, output params etc.) uv run ado describe experiment EXPERIMENT_ID ``` ## Debugging -If commands are not given expected output use -the -l flag to activate different log levels +If commands are not given expected output use the -l flag to activate different +log levels e.g. for debug level logs From d89917277b9339e5eaf41a4428d99a61c8c5467a Mon Sep 17 00:00:00 2001 From: michaelj Date: Thu, 26 Mar 2026 10:24:46 +0000 Subject: [PATCH 04/14] docs(agents): updates --- .cursor/agents/operation-analyzer.md | 33 ++++++++++++++++------------ 1 file changed, 19 insertions(+), 14 deletions(-) diff --git a/.cursor/agents/operation-analyzer.md b/.cursor/agents/operation-analyzer.md index 1e8402e67..a816a27d9 100644 --- a/.cursor/agents/operation-analyzer.md +++ b/.cursor/agents/operation-analyzer.md @@ -19,23 +19,28 @@ The main skill is [examining-ado-operations](../skills/examining-ado-operations). This will provide the basis for the analysis. -Also see using-ado-cli, query-ado-data, formulate-discovery-problem +Also see + +- using-ado-cli +- query-ado-data +- resource-yaml-creation ## Workflow -- Use [examining-ado-operations](../skills/examining-ado-operations) to get - overview of the operation(s) in question -- Determine if further analysis is required -- If yes, before coding any analysis scripts - - Check available ado operators (ado get operators --details) to see if any - should be applied - - Check available ado operators if any should be extended with necessary - analysis - - Plan with user if they want to develop the necessary analysis as an operator - or just go ahead with inpdendent scripts -- Execute the analysis -- Produce a plan for next research steps using ado. - - Interact with user if input is required to refine plan +1. Use [examining-ado-operations](../skills/examining-ado-operations) to get + an overview of the operation(s) in question +2. Determine if further analysis is required + - IMPORTANT: If **yes**, before coding any analysis scripts + - Check available ado operators (ado get operators --details) to see if any + should be applied + - Check if ado operators can be extended to perform the analysis to it +3. Implement the analysis necessary - **preferring** operator extension if possible +4. Execute the analysis (scripts or ado operations) + - IMPORTANT: + - Report the analyses run to the user + - Report scripts created if any and what they do + - Report operators extended if any and how +5. Produce a plan for next research steps using ado. ## References From 3d1eae238dca29092ca87accaa5a05d9c5aec809 Mon Sep 17 00:00:00 2001 From: michaelj Date: Fri, 27 Mar 2026 15:29:12 +0000 Subject: [PATCH 05/14] docs(skills): updates --- .../skills/examining-ado-operations/SKILL.md | 35 ++-- .../examining-discovery-spaces/SKILL.md | 180 ++++++++++++++++++ .cursor/skills/using-ado-cli/SKILL.md | 2 +- 3 files changed, 205 insertions(+), 12 deletions(-) create mode 100644 .cursor/skills/examining-discovery-spaces/SKILL.md diff --git a/.cursor/skills/examining-ado-operations/SKILL.md b/.cursor/skills/examining-ado-operations/SKILL.md index be321d261..b09337a4b 100644 --- a/.cursor/skills/examining-ado-operations/SKILL.md +++ b/.cursor/skills/examining-ado-operations/SKILL.md @@ -48,8 +48,17 @@ uv run ado show related operation --use-latest This will output the id of the latest operation. +## Tips + +`ado get-o yaml` flag outputs YAML to console. It's often useful to redirect +this to a temporary file and work with that to avoid multiple `ado get` calls +for same YAML. + ## General Workflow +- Steps 2, 3 and 4 can be run in parallel +- Step 5 depends on Step 4 + ### Step 1: Get the operation YAML ```bash @@ -60,7 +69,7 @@ Extract and summarise: - Resource **identifier**, **operationType**, **operatorIdentifier** - The identifiers of **input resources** to the operation: - - discovery spaces from the spaces field) + - discovery spaces from the spaces field - actuatorconfigurations from the actuatorConfigurationIdentifiers field - **config**: operator-specific parameters (what the run was asked to do) - **status**: latest **event** (e.g. started / finished) and **exit_state** when @@ -73,7 +82,7 @@ objectives, stopping rules, etc.). An operation which does not report finished is usually still running. -However it is possible it failed in a way that meant it could not record the +However, it is possible it failed in a way that meant it could not record the failure. In this case: 1. Determine how long it has been running. @@ -182,7 +191,7 @@ were completed there are no issues -> [examine entities](#step-3-get-entities-and-measurements) If the state is not finished -> -[Use the diagnose if sampling operation running workflow](#diagnose-if-explore-or-search-operation-is-running-workflow). +[Use the diagnose if sampling operation running workflow](#diagnose-if-an-explore-or-search-operation-is-running-workflow). For all other combinations -> [Diagnose sampling issues](#step-2-optional-diagnose-sampling-issues) @@ -207,10 +216,9 @@ uv run ado show results operation OPERATION_ID \ observed property values - InvalidMeasurementResult: The experiment failed for some reason -From the output of `show request` and `show results` - -Identify: **failed** or **invalid** rows, **reasons** for invalidity, and -anomalies in **timing** or **ordering** if those columns are present. +From the output of `show requests` and `show results` identify **failed** or +**invalid** rows, **reasons** for invalidity, and anomalies in **timing** or +**ordering** if those columns are present. ### Step 3: Get entities and measurements @@ -221,14 +229,18 @@ uv run ado show entities operation OPERATION_ID \ --output-format csv ``` -Perform quick analysis: distributions, outliers, correlations. +### Step 4: Analyze the Measurement data + +Perform quick analysis of the measurements, checking e.g. distributions of +metrics, metric outliers, correlations. -## Diagnose if Explore or Search Operation is Running Workflow +## Diagnose if an Explore or Search Operation is Running Workflow - Check if the operation is submitting experiment in batches - Confirm if the operation uses continuous batching (new experiment requested once one has finished) or static batch (full batch finishes then next starts) -- Get the requests and results timeseries using ado show cli command +- Get the requests and results timeseries using `ado show requests` and + `ado show results` - For continuous batching - Use the request time-series to determine the typical inter-request start time after the first batch i.e. this tells you how often after the first @@ -251,4 +263,5 @@ Structure the report as: 2. **Measurement overview** – sampled vs requested, success vs failure counts 3. **Findings** – notable patterns, best/worst performers, anomalies 4. **Unusual behaviour** – failures, timeouts, invalid results, unexpected - distributions + distributions 5 **Next Steps**: A plan for the next research steps to take + using ado. diff --git a/.cursor/skills/examining-discovery-spaces/SKILL.md b/.cursor/skills/examining-discovery-spaces/SKILL.md new file mode 100644 index 000000000..8b8dc4cf5 --- /dev/null +++ b/.cursor/skills/examining-discovery-spaces/SKILL.md @@ -0,0 +1,180 @@ +--- +name: examining-discovery-spaces +description: >- + End-to-end workflow to examine and summarise an ado discoveryspace — fetch + space YAML, describe entity and measurement space structure, assess sampling + coverage, export measurement data, and find related resources. Use when the + user asks to inspect, summarise, debug, or analyse a discoveryspace; wants to + understand dimensions, experiments, or sampling coverage; provides a space ID + or asks to use --use-latest for the current space. +--- + +# Examining ado Discovery Spaces + +Structured workflow for understanding what a discoveryspace contains, how +covered its entity space is, and what data has been collected. Run all commands +from the **repository root** with `uv run`. + +**Related skills**: + +- For CLI verification and command spelling, see + [using-ado-cli](../using-ado-cli/SKILL.md). +- For metastore filtering and schemas, see + [query-ado-data](../query-ado-data/SKILL.md). +- For examining operations run on a space, see + [examining-ado-operations](../examining-ado-operations/SKILL.md). + +## Context + +### Operations and DiscoverySpaces + +- discoveryspaces (or spaces for short) define a set of points (entities) and + how to measure them. They also contain the results of the measurements +- operations operate on discovery spaces either selecting or measuring points or + analysing existing measurements + +### Terminology: Distinguishing Entities in a DiscoverySpace + +When working with the data from a discoveryspace the following distinctions are +important. + +- Measured: These entities have been measured by an operation on the space +- Unmeasured: These entities have not been measured by an operation on the space + +The samplestore used by a discovery space is shared. This means there may be +relevant measurement data in the samplestore for entities in the space but that +measurement has not been performed by an operation on the space (it was +performed on another). + +- Matching: Data in the spaces samplestore that matches the space definition - + it includes measured entities +- Missing: Entities that have no matching data in the samplestore. + +Why is it useful to work with matching data? + +1. Allows using the discoveryspace as a view to fetch particular data without + having to perform operations on it + - Concrete example: You create a discvoeryspace that is a sub-space of + another sampled spaced to analyze it. You can perform analysis on existing + data even though no operation has been run on the new discoveryspace. +2. Memoization: You can understand if their are memoization opportunities that + would speed up a operation on the space. + +## Pre-requisites: The Space Identifier + +To apply this skill you need either: + +(a) a space id; (b) explicit instruction to examine the latest space + +In the case of (b) get the actual identifier: + +```bash +uv run ado show related space --use-latest +``` + +## Tips + +`ado get-o yaml` flag outputs YAML to console. It's often useful to redirect +this to a temporary file and work with that to avoid multiple `ado get` calls +for same YAML. + +## Workflow + +Steps 2,3,4 can be run in parallel. Step 5 depends on step 2 + +### Step 1: Get Space YAML + +```bash +uv run ado get space SPACE_ID -o yaml +``` + +Extract and summarise: + +- Resource **identifier** and **metadata** (name, description, labels) +- **sampleStoreIdentifier**: the sample store backing this space +- **entitySpace**: dimensions — property names, types (categorical / discrete / + continuous), and their domains/values +- **experiments**: actuator and experiment identifiers that define what can be + measured, and which target properties each experiment produces + +### Step 2: Sampling coverage and related resources + +Execute + +```commandline +uv run ado show details space SPACE_ID +``` + +This outputs two sections: + +**DETAILS** — sampling coverage: + +- Total entities in the space +- How many have been measured +- How many have failed measurements +- How many are unmeasured +- How many are matching + +Compare measured vs total to understand exploration progress. Compare measured +vs matching to understand memoization opportunities. Also, a signal that other +overlapping spaces exist. + +**RELATED RESOURCES** — all operations and stores linked to this space. + +> **Performance note**: `ado show details space` is slow as it fetches and +> aggregates entity data. Use only when sampling coverage is needed. + +### Step 3: Find Similar spaces + +`ado get space --matching-space-id SPACE_ID --details` finds spaces with the +same entity structure. Use this to understand research progression and why this +space was created. + +```bash +uv run ado get space --matching-space-id SPACE_ID --details +``` + +### Step 4: Export Measurement Data + +```bash +uv run ado show entities space SPACE_ID \ + --include measured \ + --property-format target \ + --output-format csv +``` + +This writes the data to `SPACE_ID_description_measured_target.csv` +automatically. If you find `SPACE_ID_description_measured_target.csv` already +exists do not use it as data may be stale + +You can also get lists of all unmeasured or missing entities, though this is not +typically required unless you want to analyse the unsampled portion. + +Perform quick analysis: distributions, outliers, correlations between parameters +and target properties. + +### Step 5: Examine Related Operations + +For each related operation (output in step 2), use the +[examining-ado-operations](../examining-ado-operations/SKILL.md) skill to +understand what each operation did and what it produced. + +Note: Do not analyze the data in the operations, or do detailed diagnoses. Just +enough for summary. + +## Producing a Report + +Structure the report as: + +1. **Overview**: What the space represents. Infer from metadata, dimensions, and + experiments. Short and narrative. + - **Space summary** – ID, metadata, entity count, dimensions (parameters and + their types/values) + - **Measurement space** – experiments, target properties +2. **Related Spaces** (Optional): If there are related spaces, describe them and + how they relate. +3. **Sampling coverage** – sampled vs unsampled vs missing counts; progress + assessment +4. **Data summary** – distributions of measured properties, notable performers, + outliers, correlations +5. **Related operations** – which operations ran on this space and their status diff --git a/.cursor/skills/using-ado-cli/SKILL.md b/.cursor/skills/using-ado-cli/SKILL.md index 7e18d1e21..1dc57d897 100644 --- a/.cursor/skills/using-ado-cli/SKILL.md +++ b/.cursor/skills/using-ado-cli/SKILL.md @@ -75,7 +75,7 @@ needed beyond a local Ray instance (started automatically). ### ado get -Lists resources of a given and gets resource YAML +Lists resources of a given type and gets resource YAML ```bash #List all spaces From 18187ed2407e7edab162c8660d63ca88ed1d6422 Mon Sep 17 00:00:00 2001 From: michaelj Date: Fri, 27 Mar 2026 15:55:40 +0000 Subject: [PATCH 06/14] docs(agent): saving and using saved reports --- .cursor/agents/operation-analyzer.md | 9 +++--- .../skills/examining-ado-operations/SKILL.md | 29 ++++++++++++++----- .../examining-discovery-spaces/SKILL.md | 28 ++++++++++++++---- 3 files changed, 47 insertions(+), 19 deletions(-) diff --git a/.cursor/agents/operation-analyzer.md b/.cursor/agents/operation-analyzer.md index a816a27d9..e563b2ada 100644 --- a/.cursor/agents/operation-analyzer.md +++ b/.cursor/agents/operation-analyzer.md @@ -2,8 +2,8 @@ name: operation-analyzer description: Analyzes the results of ado operations on discovery spaces and plans next - research steps. Use proactively when the user wants to analyze an operation, - understand measurement outcomes, and decide what to do next + research steps. Delegate to instances of this agent when the user want + to analyse a large number of operations in parallel. --- # Operation Examiner @@ -27,8 +27,7 @@ Also see ## Workflow -1. Use [examining-ado-operations](../skills/examining-ado-operations) to get - an overview of the operation(s) in question +1. Examine the operation 2. Determine if further analysis is required - IMPORTANT: If **yes**, before coding any analysis scripts - Check available ado operators (ado get operators --details) to see if any @@ -40,7 +39,7 @@ Also see - Report the analyses run to the user - Report scripts created if any and what they do - Report operators extended if any and how -5. Produce a plan for next research steps using ado. +5. Produce a report with next-steps as outlined in examining-ado-operations ## References diff --git a/.cursor/skills/examining-ado-operations/SKILL.md b/.cursor/skills/examining-ado-operations/SKILL.md index b09337a4b..c9a916115 100644 --- a/.cursor/skills/examining-ado-operations/SKILL.md +++ b/.cursor/skills/examining-ado-operations/SKILL.md @@ -13,8 +13,11 @@ description: >- # Examining ADO operations Structured workflow for understanding what an operation did, which space it ran -on, and whether measurements and results look healthy. Run all commands from the -**repository root** with `uv run`. +on, and whether measurements and results look healthy. + +- Run all commands from the **repository root** with `uv run`. +- Write the report to a md file in the following directory (create if it does + not exist) `reports/$CONTEXTNAME/$OPERTATIONID_$DATE_report.md` **Related skills**: @@ -56,8 +59,8 @@ for same YAML. ## General Workflow -- Steps 2, 3 and 4 can be run in parallel -- Step 5 depends on Step 4 +- Steps 3, 4 and 5 can be run in parallel +- Step 6 depends on Step 5 ### Step 1: Get the operation YAML @@ -92,7 +95,17 @@ failure. In this case: specific techniques in Explore Operation Workflow to determine if its still running -### Step 2: Review the operator +### Step 2: Check for existing report + +If the operation is finished, + +- check if there is an existing report for this operation in + `reports/$CONTEXT_NAME` +- if yes, check if that report indicated the operation was finished + - if yes, ask the user if they want to replace it with a new report + - if no, continue with creating new report + +### Step 3: Review the operator Each operation is run by an operator. The operators name is retrieved in previous step (operatorIdentifier) @@ -114,7 +127,7 @@ uv run ado template operation --operator-name $OPERATOR_IDENTIFIER --include-sch This will create a file called operation*template*$UID_schema.yaml containing the schema. -### Step 3: Describe the space +### Step 4: Describe the space Using the space id from step 1 @@ -129,7 +142,7 @@ metadata. For deeper context, read operator and experiment documentation under `website/docs/operators/` and actuator/experiment docs as needed (match **operatorIdentifier** and experiment types from the space). -### Step 4: Get the output resources of the operation +### Step 5: Get the output resources of the operation Operations can create other resources. To identify these @@ -143,7 +156,7 @@ the operation. From step 1 you know the input resource identifiers so you can work out the output identifiers. -### Step 5: Examine the output resources of the operation (if any) +### Step 6: Examine the output resources of the operation (if any) An operation can create the following resources diff --git a/.cursor/skills/examining-discovery-spaces/SKILL.md b/.cursor/skills/examining-discovery-spaces/SKILL.md index 8b8dc4cf5..e3a95ea35 100644 --- a/.cursor/skills/examining-discovery-spaces/SKILL.md +++ b/.cursor/skills/examining-discovery-spaces/SKILL.md @@ -12,8 +12,11 @@ description: >- # Examining ado Discovery Spaces Structured workflow for understanding what a discoveryspace contains, how -covered its entity space is, and what data has been collected. Run all commands -from the **repository root** with `uv run`. +covered its entity space is, and what data has been collected. + +- Run all commands from the **repository root** with `uv run`. +- Write the report to a md file in the following directory (create if it does + not exist) `reports/$CONTEXTNAME/$SPACEID_$DATE_report.md` **Related skills**: @@ -80,7 +83,7 @@ for same YAML. ## Workflow -Steps 2,3,4 can be run in parallel. Step 5 depends on step 2 +Steps 4,5 and 6 can be run in parallel. ### Step 1: Get Space YAML @@ -97,6 +100,8 @@ Extract and summarise: - **experiments**: actuator and experiment identifiers that define what can be measured, and which target properties each experiment produces +Check if there is a pre-existing report + ### Step 2: Sampling coverage and related resources Execute @@ -124,7 +129,18 @@ overlapping spaces exist. > **Performance note**: `ado show details space` is slow as it fetches and > aggregates entity data. Use only when sampling coverage is needed. -### Step 3: Find Similar spaces +### Step 3: Check for existing report + +- check if there is an existing report for this space in `reports/$CONTEXT_NAME` +- if yes, check if either of the following are true + - New operations have been run on space since report + - The number of measured entities has increased +- If neither of above are true as the user if they want to write a new report or + use existing + - As nothing has changed, the only purpose of creating a new report is if a + different agent is being used + +### Step 4: Find Similar spaces `ado get space --matching-space-id SPACE_ID --details` finds spaces with the same entity structure. Use this to understand research progression and why this @@ -134,7 +150,7 @@ space was created. uv run ado get space --matching-space-id SPACE_ID --details ``` -### Step 4: Export Measurement Data +### Step 5: Export Measurement Data ```bash uv run ado show entities space SPACE_ID \ @@ -153,7 +169,7 @@ typically required unless you want to analyse the unsampled portion. Perform quick analysis: distributions, outliers, correlations between parameters and target properties. -### Step 5: Examine Related Operations +### Step 6: Examine Related Operations For each related operation (output in step 2), use the [examining-ado-operations](../examining-ado-operations/SKILL.md) skill to From c70ba30f78652ef745148ae21fd138a30222172c Mon Sep 17 00:00:00 2001 From: michaelj Date: Fri, 27 Mar 2026 16:14:25 +0000 Subject: [PATCH 07/14] docs(skills): fixes --- .cursor/agents/operation-analyzer.md | 15 +++++++-------- .cursor/skills/examining-ado-operations/SKILL.md | 9 +++++---- .../skills/examining-discovery-spaces/SKILL.md | 15 +++++++-------- .cursor/skills/using-ado-cli/SKILL.md | 16 ++++++++-------- 4 files changed, 27 insertions(+), 28 deletions(-) diff --git a/.cursor/agents/operation-analyzer.md b/.cursor/agents/operation-analyzer.md index e563b2ada..d7bb67805 100644 --- a/.cursor/agents/operation-analyzer.md +++ b/.cursor/agents/operation-analyzer.md @@ -2,11 +2,11 @@ name: operation-analyzer description: Analyzes the results of ado operations on discovery spaces and plans next - research steps. Delegate to instances of this agent when the user want + research steps. Delegate to instances of this agent when the user wants to analyse a large number of operations in parallel. --- -# Operation Examiner +# Operation Analyzer You are an Operation Analyzer for ado - a tool for computational experimentation. You analyse operations on discovery spaces and produce @@ -16,14 +16,13 @@ next steps. **Skills to apply**: The main skill is -[examining-ado-operations](../skills/examining-ado-operations). This will +[examining-ado-operations](../skills/examining-ado-operations/SKILL.md). This will provide the basis for the analysis. -Also see +Also see: -- using-ado-cli -- query-ado-data -- resource-yaml-creation +- [resource-yaml-creation](../skills/resource-yaml-creation) for how to write operation +resource YAML ## Workflow @@ -32,7 +31,7 @@ Also see - IMPORTANT: If **yes**, before coding any analysis scripts - Check available ado operators (ado get operators --details) to see if any should be applied - - Check if ado operators can be extended to perform the analysis to it + - Check if ado operators can be extended to perform the analysis 3. Implement the analysis necessary - **preferring** operator extension if possible 4. Execute the analysis (scripts or ado operations) - IMPORTANT: diff --git a/.cursor/skills/examining-ado-operations/SKILL.md b/.cursor/skills/examining-ado-operations/SKILL.md index c9a916115..67651cd5e 100644 --- a/.cursor/skills/examining-ado-operations/SKILL.md +++ b/.cursor/skills/examining-ado-operations/SKILL.md @@ -17,7 +17,7 @@ on, and whether measurements and results look healthy. - Run all commands from the **repository root** with `uv run`. - Write the report to a md file in the following directory (create if it does - not exist) `reports/$CONTEXTNAME/$OPERTATIONID_$DATE_report.md` + not exist) `reports/$CONTEXT_NAME/$OPERATIONID_$DATE_report.md` **Related skills**: @@ -53,7 +53,7 @@ This will output the id of the latest operation. ## Tips -`ado get-o yaml` flag outputs YAML to console. It's often useful to redirect +`ado get -o yaml` flag outputs YAML to console. It's often useful to redirect this to a temporary file and work with that to avoid multiple `ado get` calls for same YAML. @@ -124,7 +124,7 @@ To understand an operators parameters examine its schema: uv run ado template operation --operator-name $OPERATOR_IDENTIFIER --include-schema ``` -This will create a file called operation*template*$UID_schema.yaml containing +This will create a file called `operation_template_$UID_schema.yaml` containing the schema. ### Step 4: Describe the space @@ -276,5 +276,6 @@ Structure the report as: 2. **Measurement overview** – sampled vs requested, success vs failure counts 3. **Findings** – notable patterns, best/worst performers, anomalies 4. **Unusual behaviour** – failures, timeouts, invalid results, unexpected - distributions 5 **Next Steps**: A plan for the next research steps to take + distributions +5. 5 **Next Steps**: A plan for the next research steps to take using ado. diff --git a/.cursor/skills/examining-discovery-spaces/SKILL.md b/.cursor/skills/examining-discovery-spaces/SKILL.md index e3a95ea35..341d274b3 100644 --- a/.cursor/skills/examining-discovery-spaces/SKILL.md +++ b/.cursor/skills/examining-discovery-spaces/SKILL.md @@ -16,7 +16,7 @@ covered its entity space is, and what data has been collected. - Run all commands from the **repository root** with `uv run`. - Write the report to a md file in the following directory (create if it does - not exist) `reports/$CONTEXTNAME/$SPACEID_$DATE_report.md` + not exist) `reports/$CONTEXT_NAME/$SPACEID_$DATE_report.md` **Related skills**: @@ -57,10 +57,11 @@ Why is it useful to work with matching data? 1. Allows using the discoveryspace as a view to fetch particular data without having to perform operations on it - - Concrete example: You create a discvoeryspace that is a sub-space of + - Concrete example: You create a discoveryspace that is a subspace of another sampled spaced to analyze it. You can perform analysis on existing data even though no operation has been run on the new discoveryspace. -2. Memoization: You can understand if their are memoization opportunities that +2. Memoization: You can understand if there are + [memoization opportunities](website/docs/core-concepts/data-sharing.md) that would speed up a operation on the space. ## Pre-requisites: The Space Identifier @@ -100,13 +101,11 @@ Extract and summarise: - **experiments**: actuator and experiment identifiers that define what can be measured, and which target properties each experiment produces -Check if there is a pre-existing report - ### Step 2: Sampling coverage and related resources Execute -```commandline +```bash uv run ado show details space SPACE_ID ``` @@ -135,8 +134,8 @@ overlapping spaces exist. - if yes, check if either of the following are true - New operations have been run on space since report - The number of measured entities has increased -- If neither of above are true as the user if they want to write a new report or - use existing +- If neither of above are true, ask the user if they want to write a + new report or use existing - As nothing has changed, the only purpose of creating a new report is if a different agent is being used diff --git a/.cursor/skills/using-ado-cli/SKILL.md b/.cursor/skills/using-ado-cli/SKILL.md index 1dc57d897..b1b88e293 100644 --- a/.cursor/skills/using-ado-cli/SKILL.md +++ b/.cursor/skills/using-ado-cli/SKILL.md @@ -33,14 +33,14 @@ uv run ado [COMMAND] [SUBCOMMAND1] [SUBCOMMAND2] --help These plausible-sounding commands do not exist in ado. Do not write them: -| ❌ Does not exist | ✅ Correct equivalent | -| ----------------- | --------------------------------------- | -| `ado run` | `ado create operation -f op.yaml` | -| `ado start` | `ado create operation -f op.yaml` | -| `ado execute` | `ado create operation -f op.yaml` | -| `ado launch` | `ado create operation -f op.yaml` | -| `ado list` | `ado get spaces` / `ado get operations` | -| `ado status` | `ado show details space SPACE_ID` | +| ❌ Does not exist | ✅ Correct equivalent | +| --------- | -------------------- | +| `ado run` | `ado create operation -f op.yaml` | +| `ado start` | `ado create operation -f op.yaml` | +| `ado execute` | `ado create operation -f op.yaml` | +| `ado launch` | `ado create operation -f op.yaml` | +| `ado list` | `ado get spaces` / `ado get operations` | +| `ado status` | `ado show details space SPACE_ID` | **Key principle**: `ado create operation` both _defines_ and _starts_ the operation in a single command. There is no separate "run" step. From 6f034d0b60c4984c850d4b6fa886f16a89e1f350 Mon Sep 17 00:00:00 2001 From: michaelj Date: Fri, 27 Mar 2026 16:35:48 +0000 Subject: [PATCH 08/14] docs(skills): fixes --- .cursor/skills/examining-ado-operations/SKILL.md | 10 +++++++--- .cursor/skills/examining-discovery-spaces/SKILL.md | 13 +++++++++---- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/.cursor/skills/examining-ado-operations/SKILL.md b/.cursor/skills/examining-ado-operations/SKILL.md index 67651cd5e..85f334d58 100644 --- a/.cursor/skills/examining-ado-operations/SKILL.md +++ b/.cursor/skills/examining-ado-operations/SKILL.md @@ -16,8 +16,11 @@ Structured workflow for understanding what an operation did, which space it ran on, and whether measurements and results look healthy. - Run all commands from the **repository root** with `uv run`. -- Write the report to a md file in the following directory (create if it does - not exist) `reports/$CONTEXT_NAME/$OPERATIONID_$DATE_report.md` +- Write the report to `reports//` (create the +directory if needed) + - where `ado_context_name` is the + **active ado metastore context** (`uv run ado context`) +- Write the report as `__report.md` **Related skills**: @@ -59,6 +62,7 @@ for same YAML. ## General Workflow +- Run Steps 1 and 2 first - Steps 3, 4 and 5 can be run in parallel - Step 6 depends on Step 5 @@ -100,7 +104,7 @@ failure. In this case: If the operation is finished, - check if there is an existing report for this operation in - `reports/$CONTEXT_NAME` + `reports//` - if yes, check if that report indicated the operation was finished - if yes, ask the user if they want to replace it with a new report - if no, continue with creating new report diff --git a/.cursor/skills/examining-discovery-spaces/SKILL.md b/.cursor/skills/examining-discovery-spaces/SKILL.md index 341d274b3..b8f5dff9a 100644 --- a/.cursor/skills/examining-discovery-spaces/SKILL.md +++ b/.cursor/skills/examining-discovery-spaces/SKILL.md @@ -15,8 +15,11 @@ Structured workflow for understanding what a discoveryspace contains, how covered its entity space is, and what data has been collected. - Run all commands from the **repository root** with `uv run`. -- Write the report to a md file in the following directory (create if it does - not exist) `reports/$CONTEXT_NAME/$SPACEID_$DATE_report.md` +- Write the report to `reports//` (create the +directory if needed) + - where `ado_context_name` is the + **active ado metastore context** (`uv run ado context`) +- Write the report as `__report.md` **Related skills**: @@ -84,7 +87,8 @@ for same YAML. ## Workflow -Steps 4,5 and 6 can be run in parallel. +Run Step 2 and 3 first. +Then steps 4,5 and 6 can be run in parallel. ### Step 1: Get Space YAML @@ -130,7 +134,8 @@ overlapping spaces exist. ### Step 3: Check for existing report -- check if there is an existing report for this space in `reports/$CONTEXT_NAME` +- check if there is an existing report for this space in + `reports//` - if yes, check if either of the following are true - New operations have been run on space since report - The number of measured entities has increased From 5283291066ee34e2e69055c9bfeb6a3a612a5435 Mon Sep 17 00:00:00 2001 From: michaelj Date: Fri, 27 Mar 2026 16:38:03 +0000 Subject: [PATCH 09/14] chore(skills): add bob dir --- .bob | 1 + 1 file changed, 1 insertion(+) create mode 120000 .bob diff --git a/.bob b/.bob new file mode 120000 index 000000000..aa39edf78 --- /dev/null +++ b/.bob @@ -0,0 +1 @@ +.cursor \ No newline at end of file From 474c91000e759cd4841b09b4f27411f31373a3a4 Mon Sep 17 00:00:00 2001 From: michaelj Date: Fri, 27 Mar 2026 19:13:33 +0000 Subject: [PATCH 10/14] docs(skills): add troubleshooting section For edge cases when sampling. --- .../skills/examining-ado-operations/SKILL.md | 34 +++++++++++++++---- 1 file changed, 28 insertions(+), 6 deletions(-) diff --git a/.cursor/skills/examining-ado-operations/SKILL.md b/.cursor/skills/examining-ado-operations/SKILL.md index 85f334d58..564189707 100644 --- a/.cursor/skills/examining-ado-operations/SKILL.md +++ b/.cursor/skills/examining-ado-operations/SKILL.md @@ -16,10 +16,10 @@ Structured workflow for understanding what an operation did, which space it ran on, and whether measurements and results look healthy. - Run all commands from the **repository root** with `uv run`. -- Write the report to `reports//` (create the -directory if needed) - - where `ado_context_name` is the - **active ado metastore context** (`uv run ado context`) +- Write the report to `reports//` (create the directory if + needed) + - where `ado_context_name` is the **active ado metastore context** + (`uv run ado context`) - Write the report as `__report.md` **Related skills**: @@ -56,6 +56,8 @@ This will output the id of the latest operation. ## Tips +### Avoiding refetching YAML + `ado get -o yaml` flag outputs YAML to console. It's often useful to redirect this to a temporary file and work with that to avoid multiple `ado get` calls for same YAML. @@ -281,5 +283,25 @@ Structure the report as: 3. **Findings** – notable patterns, best/worst performers, anomalies 4. **Unusual behaviour** – failures, timeouts, invalid results, unexpected distributions -5. 5 **Next Steps**: A plan for the next research steps to take - using ado. +5. 5 **Next Steps**: A plan for the next research steps to take using ado. + +## Troubleshooting + +### Number points sampled is greater than the number of entities measured by operation + +Some samplers can sample the same entity twice. In this case you may see +conflicting statistics about how many entities are measured. For example if an +operation is configured to sample two points, and it samples same point twice, +the additional number entities with measurements after the operation is 1, but +the number of points sampled by operation is 2. + +Comparing the size of the set of entity identifiers to the timeseries length can +confirm this. + +### Memoization on, but Entities measured twice + +The requests which use memoized results for Entities are called "replayed +measurements". If the same entity is sampled twice in an operation the second +should be replayed. If it is not it means the sampling algorithm selected the +same point again before the first was stored to be reused. In this case it means +the same entity will be measured twice. From 9389b8b2b3ec0731f252698aa1f81995380c5aad Mon Sep 17 00:00:00 2001 From: Michael Johnston <66301584+michael-johnston@users.noreply.github.com> Date: Mon, 30 Mar 2026 14:19:53 +0100 Subject: [PATCH 11/14] Apply suggestions from code review Co-authored-by: Alessandro Pomponio <10339005+AlessandroPomponio@users.noreply.github.com> Signed-off-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com> --- .../skills/examining-ado-operations/SKILL.md | 20 +++++++++---------- .../examining-discovery-spaces/SKILL.md | 6 +++--- 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/.cursor/skills/examining-ado-operations/SKILL.md b/.cursor/skills/examining-ado-operations/SKILL.md index 564189707..d563d9889 100644 --- a/.cursor/skills/examining-ado-operations/SKILL.md +++ b/.cursor/skills/examining-ado-operations/SKILL.md @@ -105,15 +105,15 @@ failure. In this case: If the operation is finished, -- check if there is an existing report for this operation in +- Check if there is an existing report for this operation in `reports//` -- if yes, check if that report indicated the operation was finished - - if yes, ask the user if they want to replace it with a new report - - if no, continue with creating new report +- If yes, check if that report indicated the operation was finished + - If yes, ask the user if they want to replace it with a new report + - If no, continue with creating new report ### Step 3: Review the operator -Each operation is run by an operator. The operators name is retrieved in +Each operation is run by an operator. The operator's name is retrieved in previous step (operatorIdentifier) Execute @@ -124,7 +124,7 @@ uv run ado get operator --details $OPERATOR_IDENTIFIER #### Understanding operator parameters -To understand an operators parameters examine its schema: +To understand an operator's parameters, examine its schema: ```bash uv run ado template operation --operator-name $OPERATOR_IDENTIFIER --include-schema @@ -255,7 +255,7 @@ metrics, metric outliers, correlations. ## Diagnose if an Explore or Search Operation is Running Workflow -- Check if the operation is submitting experiment in batches +- Check if the operation is submitting experiments in batches - Confirm if the operation uses continuous batching (new experiment requested once one has finished) or static batch (full batch finishes then next starts) - Get the requests and results timeseries using `ado show requests` and @@ -283,7 +283,7 @@ Structure the report as: 3. **Findings** – notable patterns, best/worst performers, anomalies 4. **Unusual behaviour** – failures, timeouts, invalid results, unexpected distributions -5. 5 **Next Steps**: A plan for the next research steps to take using ado. +5. **Next Steps**: A plan for the next research steps to take using ado. ## Troubleshooting @@ -301,7 +301,7 @@ confirm this. ### Memoization on, but Entities measured twice The requests which use memoized results for Entities are called "replayed -measurements". If the same entity is sampled twice in an operation the second -should be replayed. If it is not it means the sampling algorithm selected the +measurements". If the same entity is sampled twice in an operation, the second +should be replayed. If it is not, it means the sampling algorithm selected the same point again before the first was stored to be reused. In this case it means the same entity will be measured twice. diff --git a/.cursor/skills/examining-discovery-spaces/SKILL.md b/.cursor/skills/examining-discovery-spaces/SKILL.md index b8f5dff9a..6eb7ba796 100644 --- a/.cursor/skills/examining-discovery-spaces/SKILL.md +++ b/.cursor/skills/examining-discovery-spaces/SKILL.md @@ -134,9 +134,9 @@ overlapping spaces exist. ### Step 3: Check for existing report -- check if there is an existing report for this space in +- Check if there is an existing report for this space in `reports//` -- if yes, check if either of the following are true +- If yes, check if either of the following are true: - New operations have been run on space since report - The number of measured entities has increased - If neither of above are true, ask the user if they want to write a @@ -165,7 +165,7 @@ uv run ado show entities space SPACE_ID \ This writes the data to `SPACE_ID_description_measured_target.csv` automatically. If you find `SPACE_ID_description_measured_target.csv` already -exists do not use it as data may be stale +exists do not use it, as data may be stale You can also get lists of all unmeasured or missing entities, though this is not typically required unless you want to analyse the unsampled portion. From 91fbd363e84d2a62af1d275af73946d2cec25fef Mon Sep 17 00:00:00 2001 From: michaelj Date: Mon, 30 Mar 2026 15:20:12 +0100 Subject: [PATCH 12/14] fix(skills): fix incorrect step Also clarify operatorIdentifier and operationIdentifier are different. --- .cursor/skills/examining-ado-operations/SKILL.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/.cursor/skills/examining-ado-operations/SKILL.md b/.cursor/skills/examining-ado-operations/SKILL.md index 564189707..be3b6eac2 100644 --- a/.cursor/skills/examining-ado-operations/SKILL.md +++ b/.cursor/skills/examining-ado-operations/SKILL.md @@ -113,8 +113,10 @@ If the operation is finished, ### Step 3: Review the operator -Each operation is run by an operator. The operators name is retrieved in -previous step (operatorIdentifier) +Each operation is run by an operator. The operators name is retrieved in step +one as the value of the field operatorIdentifier. + +Note: `operatorIdentifer` is not the same as `operationIdentifier` Execute From 2f17f0a654f43c47337edaebf3ef5a4c8a9b64b8 Mon Sep 17 00:00:00 2001 From: michaelj Date: Mon, 30 Mar 2026 15:22:35 +0100 Subject: [PATCH 13/14] docs(skills): guidelines on large results files. --- .../skills/examining-ado-operations/SKILL.md | 24 ++++++++++++++++--- .../examining-discovery-spaces/SKILL.md | 19 +++++++++++++++ 2 files changed, 40 insertions(+), 3 deletions(-) diff --git a/.cursor/skills/examining-ado-operations/SKILL.md b/.cursor/skills/examining-ado-operations/SKILL.md index be3b6eac2..cc97aa532 100644 --- a/.cursor/skills/examining-ado-operations/SKILL.md +++ b/.cursor/skills/examining-ado-operations/SKILL.md @@ -62,6 +62,23 @@ This will output the id of the latest operation. this to a temporary file and work with that to avoid multiple `ado get` calls for same YAML. +In particular "get datacontainer -o yaml|json" can be large and should be +redirected to a file and loaded with python. + +### Large output files + +The files created by '-o/--output-format' can be very large e.g. from "show +entities", "show requests" or "show results". + +When inspecting these files: + +- Use wc to count the file size first before using head/tail/cat etc. on it. +- Use head -n1 to get column headers, this will not be large +- Avoid head -n > 1 unless you have a specific need e.g. checking if file is + corrupted +- Avoid tail unless you have a specific need +- Prefer python e.g. pandas.read_csv for any detailed analysis on the file. + ## General Workflow - Run Steps 1 and 2 first @@ -175,7 +192,7 @@ An operation can create the following resources To retrieve contents of data container ```bash -uv run ado get datacontainer -o yaml $DATACONTAINER_IDENTIFIER +uv run ado get datacontainer -o yaml $DATACONTAINER_IDENTIFIER > datacontainer.yaml ``` For each output resource summarize what it is/contains. @@ -219,7 +236,7 @@ For all other combinations -> ### Step 2 (Optional): Diagnose sampling issues First run these two commands to get the metadata on what was requested and -measured +measured, noting the [guidelines on large files](#large-output-files): ```bash uv run ado show requests operation OPERATION_ID \ @@ -243,7 +260,8 @@ From the output of `show requests` and `show results` identify **failed** or ### Step 3: Get entities and measurements -To get the data on measurements +To get the data on measurements execute (noting the +[guidelines on large files](#large-output-files)): ```bash uv run ado show entities operation OPERATION_ID \ diff --git a/.cursor/skills/examining-discovery-spaces/SKILL.md b/.cursor/skills/examining-discovery-spaces/SKILL.md index b8f5dff9a..6de712a28 100644 --- a/.cursor/skills/examining-discovery-spaces/SKILL.md +++ b/.cursor/skills/examining-discovery-spaces/SKILL.md @@ -81,10 +81,26 @@ uv run ado show related space --use-latest ## Tips +### Avoiding refetching YAML + `ado get-o yaml` flag outputs YAML to console. It's often useful to redirect this to a temporary file and work with that to avoid multiple `ado get` calls for same YAML. +### Large output files + +The files created by '-o/--output-format' can be very large e.g. from "show +entities". + +When inspecting these files: + +- Use wc to count the file size first before using head/tail/cat etc. on it. +- Use head -n1 to get column headers, this will not be large +- Avoid head -n > 1 unless you have a specific need e.g. checking if file is + corrputed +- Avoid tail unless you have a specific need +- Prefer python e.g. pandas.read_csv for any detailed analysis on the file. + ## Workflow Run Step 2 and 3 first. @@ -156,6 +172,9 @@ uv run ado get space --matching-space-id SPACE_ID --details ### Step 5: Export Measurement Data +Note: Keep in mind the [guidelines on large output files](#large-output-files) +for the following. + ```bash uv run ado show entities space SPACE_ID \ --include measured \ From 78f1a23d11c7fbbf6652fb5b0c1d5620139e417c Mon Sep 17 00:00:00 2001 From: michaelj Date: Mon, 30 Mar 2026 15:55:25 +0100 Subject: [PATCH 14/14] docs(skills): refine analysis guidance. --- .cursor/skills/examining-ado-operations/SKILL.md | 6 ++++-- .cursor/skills/examining-discovery-spaces/SKILL.md | 6 ++++-- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/.cursor/skills/examining-ado-operations/SKILL.md b/.cursor/skills/examining-ado-operations/SKILL.md index 048e49310..d7724ddba 100644 --- a/.cursor/skills/examining-ado-operations/SKILL.md +++ b/.cursor/skills/examining-ado-operations/SKILL.md @@ -270,8 +270,10 @@ uv run ado show entities operation OPERATION_ID \ ### Step 4: Analyze the Measurement data -Perform quick analysis of the measurements, checking e.g. distributions of -metrics, metric outliers, correlations. +Perform an analysis of the measurements, checking e.g. distributions of +metrics, metric outliers, correlations between metrics. +Take into account the domain of the experiment and meaning of metrics +when looking for patterns. ## Diagnose if an Explore or Search Operation is Running Workflow diff --git a/.cursor/skills/examining-discovery-spaces/SKILL.md b/.cursor/skills/examining-discovery-spaces/SKILL.md index 7de08d1f6..fe5c5077c 100644 --- a/.cursor/skills/examining-discovery-spaces/SKILL.md +++ b/.cursor/skills/examining-discovery-spaces/SKILL.md @@ -189,8 +189,10 @@ exists do not use it, as data may be stale You can also get lists of all unmeasured or missing entities, though this is not typically required unless you want to analyse the unsampled portion. -Perform quick analysis: distributions, outliers, correlations between parameters -and target properties. +Perform an analysis of the measurements, checking e.g. distributions of +metrics, metric outliers, correlations between metrics. +Take into account the domain of the experiment and meaning of metrics +when looking for patterns. ### Step 6: Examine Related Operations