feat(cli): allow for extended legacy support for ado upgrade of resources

[AlessandroPomponio]

Issue

Let’s say you have an old sample store that’s referencing an old import path for the class (e.g., it is referencing SQLEntitySource instead of SQLSampleStore).
We had an upgrade validator that fixed that automatically for you. We removed it in version X.

We’re now at version Y, where Y >> X. Between version X and Y, we have made other changes to sample stores which also had temporary validators in place and a subset of them might also have been removed.

HOWEVER, you kept using ado without issues because you never needed that sample store anymore and kept using new sample stores for your new experiments. This meant that you never even knew anything had changed.

The situation now is:

1. You have never run ado upgrade store because you never needed it
2. You have old, unsupported stores in your DBs that cannot be upgraded anymore. To be upgraded you’d have to roll back to version X.
3. You have new sample stores, for which you need version Y.

This means we are in a state where no version can actually fetch all of the resources from the DB, and that’s the whole issue

---

Related to #620

[VassilisVassiliadis]

The root cause of the observed behaviour is that ado performs the upgrade at the granularity of a single sample store.

I believe this is a conscious decision for a several reasons. Some are:

• upgrading all resources every single time "there is a need to upgrade" can be time consuming
• we cannot reliably tell whether a change to pydantic models also necessitates an upgrade

One way to fix this problem, is to upgrade all ado resources in every ado context whenever the version of ado changes. This is likely cost prohibitive so we should try something else first.

An alternative, would be to record inside sample store metadata about the last version of ado that successfully stored data in it. Then, when performing an upgrade:

1. first check whether all resources in the sample store can be upgraded,
2. if they all can be upgraded then, persist the upgraded resources and update the sample store metadata to reflect the current version of ado
3. if at least 1 object cannot be upgraded, then do not persist the upgraded resources and tell the user they have to use the old version of ado that's recorded in the metadata of the sample store when interacting with that sample store.

Benefits:

• current upgrade protocol remains roughly the same
• a sample store never becomes unusable
• worst case scenario, you're stuck with using an old version of ado with that sample store.

Perhaps there's a way to export such a sample store in a JSON-like format using the ado version that's known to work for that sample-store. Then re-ingest the exported data but this time using with the new version of ado. But that's something we can think of down the line.

[AlessandroPomponio]

The root cause of the observed behaviour is that ado performs the upgrade at the granularity of a single sample store.

I believe this is a conscious decision for a several reasons. Some are:

• upgrading all resources every single time "there is a need to upgrade" can be time consuming

Yes and no - we load all resources of a type (which validates them), and then store them again. While the validation (and upgrade mechanism) is at the level of a single resource, the perceived effect is that all resources are upgraded at the same time.

• we cannot reliably tell whether a change to pydantic models also necessitates an upgrade

It depends - the way we suggest people to handle this case is to use our utility functions to warn people that a resource is being temporarily auto-upgraded and the ado upgrade command should be run. Of course there's no guarantee people will follow that protocol. In general, users could/would be aware - we would have to see if we can have something similar in a programmatic fashion.

An alternative, would be to record inside sample store metadata about the last version of ado that successfully stored data in it. Then, when performing an upgrade:

1. first check whether all resources in the sample store can be upgraded,
2. if they all can be upgraded then, persist the upgraded resources and update the sample store metadata to reflect the current version of ado
3. if at least 1 object cannot be upgraded, then do not persist the upgraded resources and tell the user they have to use the old version of ado that's recorded in the metadata of the sample store when interacting with that sample store.

There are a few things that need consideration with this approach. I'll use (VX) to refer to your numbered points and I'll just use (X) to refer to those in the body of the issue.

(V2) does not help give a lot of information to the user: it tells them "the last time you ran the upgrade it was on version A.B.C", which could also be never. This information isn't immediately actionable either: they now know that something changed between version A.B.C (old) and X.Y.Z (new), and they will have to try - possibly many - ado versions to find one where (a) the field was modified and (b) an upgrade validator was still present. A way to make it more actionable would be to update the metadata every time a newer version of ado validates the resource successfully (truncating dev versions, but that's less of an issue): this tells the user that the resource was readable - one way or another - up to version D.E.F. This approach, of course, is much more involved as there would need to be many more calls to the DB and there still wouldn't be any guarantee that this would be any kind of added information compared to (V2).

Let's say that with the above we have a solution for (2): the user goes back to a certain version A.B.C to read a certain resource R. At this point the user can read R but cannot upgrade it in the DB because the semantics from (V1) run into (3) if there are newer resources that ado A.B.C cannot read. The best way to handle this would be to optionally allow upgrading a subset of resources instead of only allowing all-or-nothing upgrades like in (V1). With something on the lines of ado upgrade RESOURCE --compatible-only, ado would fetch all resources whose ado version is less than the current version and upgrade just those. The user would then be able to go back to a newer version of ado and see if things work again. Note that if - for some reason - the user doesn't run ado upgrade again with the newer version, depending on what was decided in the previous paragraph, the metadata of the resource might still say only version A.B.C.

NOTE: If we decide to go ahead with this approach, we should also change ProjectContext's model configuration to not forbid extras anymore, or we risk breaking previous ado versions when changes happen there.

---

In general, I think this approach has the obvious strength of being pretty easy to implement, as it doesn't require any kind of specific intelligence on ado's side: the user has to downgrade the ado version, run the upgrade command, then upgrade again (and possibly repeat). The user experience, on the other hand, would in my opinion be pretty poor. This would also only solve future issues, not current ones.

[michael-johnston]

First a short clarification

The root cause of the observed behaviour is that ado performs the upgrade at the granularity of a single sample store.

It doesn't.

One way to fix this problem, is to upgrade all ado resources in every ado context whenever the version of ado changes.

ado upgrade

• upgrades all ado resources of a type
• in current context
• on demand

updating this to every resource, in every contexts, and whenever the version of ado changes, while reducing chances of encountering a problem would not solve the underlying issue.

[michael-johnston]

@AlessandroPomponio @VassilisVassiliadis I think your both saying something along these lines is what I think we should go with

• when we make a change that causes the resource schema to change we update the version (maybe the current version field maybe its a different one)
• ado upgrade on encountering a resource it can't load upgrade, emits a warning about it and keeps going (compatible-only - although I don't think we need a flag)
• then the version of ado doing an upgrade can know if a resource it could not read (a) too old - version older than resource version in ado doing the upgrade and no upgrade path existed to solve it b) too new - version is newer than the version of ado doing upgrade (c) its some kind of bug

[michael-johnston]

we could keep a file in ado cli with the resource upgrades and what versions are capable of upgrading them .
resource | version | supported until | upgradable until

with the very latest version of ado would also be able to read this file to tell a user how to upgrade any problematic resources.

[VassilisVassiliadis]

First a short clarification

The root cause of the observed behaviour is that ado performs the upgrade at the granularity of a single sample store.

It doesn't.

Is it at the granularity of contexts? What if I said this:

The root cause of the observed behaviour is that ado does not upgrade all Resources in all Sample Stores in all Contexts.

[VassilisVassiliadis]

when we make a change that causes the resource schema to change we update the version (maybe the current version field maybe its a different one)

I was suggesting something slightly different. Whenever a user creates a resource (i.e. actuatorConfiguration, space, Operation) we update the metadata of a Sample Store to record the version of ado.

However, I now realize that my suggestions were a bit naive so I'll take a step back!

[AlessandroPomponio]

What I think we should be doing is:

1. Extend the the removal time of validators. Usually we target 2 or 3 minor versions down the line, which doesn't seem like a lot of time, given that we keep on adding features, and those require bumping the minor version.
2. Keep the retired validators in the codebase, in a "legacy" package or something so that people can still fish them out if they need them down the line. This legacy package could be purged on major version bumps.
3. Have a mechanism for auto discovery of these validators
4. Catch pydantic validation errors and use the mechanism from (3) to suggest possible upgrades using legacy validators

So:

• when we make a change that causes the resource schema to change we update the version (maybe the current version field maybe its a different one)

I agree, but we need to see what the depth of this is: do we have a version field for each actuator configuration, for example? Let's say I change one parameter name in the vllm actuator's configuration. What version do I bump? Do we need to add a field to GenericActuatorConfiguration, for example? Should we use semver for these as well? e.g., new fields with defaults are only a minor version change; a change in a field/ a new field with no default is a major version change, etc.

• ado upgrade on encountering a resource it can't load upgrade, emits a warning about it and keeps going (compatible-only - although I don't think we need a flag)

Yes, I think we should keep the current behaviour

• then the version of ado doing an upgrade can know if a resource it could not read (a) too old - version older than resource version in ado doing the upgrade and no upgrade path existed to solve it b) too new - version is newer than the version of ado doing upgrade (c) its some kind of bug

I'd say yes but with the added intelligence from (3) and (4)