feat: feat: custom serialize to compress measurement result

[michael-johnston]

Is your feature request related to a problem? Please describe.

The serialised representation of a MeasurementResult contains a large amount of redundant information as every ObservedPropertyValue repeats the same ExperimentReference object (as ObservedProperty it need it to generate the id). This can be upwards of 3.5 times more than is actually required.

This applies to the data transferred by show entities, show operation, show results, show requests, show details space, and get measurementrequests

Example, for two measured values of solve_times and objective_values from a particular experiment you would get

  - property:
      experimentReference:
        actuatorIdentifier: custom_experiments
        experimentIdentifier: solve_mip
        parameterization:
        - property:
            identifier: mps_file
          value: sorrell3.mps.gz
          valueType: STRING_VALUE_TYPE
        - property:
            identifier: time_limit_s
          value: 3600
          valueType: NUMERIC_VALUE_TYPE
        - property:
            identifier: progress_interval_s
          value: 60
          valueType: NUMERIC_VALUE_TYPE
      targetProperty:
        identifier: solve_times
    value:
    - 2065.2863233109965
    - 3600.004925447007
    - 565.9625509519974
    - 2771.719036005001
    - 2362.510065069
    valueType: VECTOR_VALUE_TYPE
  - property:
      experimentReference:
        actuatorIdentifier: custom_experiments
        experimentIdentifier: solve_mip
        parameterization:
        - property:
            identifier: mps_file
          value: sorrell3.mps.gz
          valueType: STRING_VALUE_TYPE
        - property:
            identifier: time_limit_s
          value: 3600
          valueType: NUMERIC_VALUE_TYPE
        - property:
            identifier: progress_interval_s
          value: 60
          valueType: NUMERIC_VALUE_TYPE
      targetProperty:
        identifier: objective_values
    value:
    - -16.0
    - -16.0
    - -16.0
    - -16.000000000000007
    - -16.0
    valueType: VECTOR_VALUE_TYPE

of the ~700 characters required for each value ~560 are repeated. So for such values it currently takes 7000 chars but could take 1960 (72% compression, 3.5x less)

Describe the solution you'd like.

In general we need to change the serialized representation - either using a custom-serialiser+validator, or by changing the MeasurementResult model itself.

In general the serialization (which is a list of ObservedPropertyValues currently) should become, for example

experimentReference: 
  actuatorIdentifier: custom_experiments
        experimentIdentifier: solve_mip
        parameterization:
        - property:
            identifier: mps_file
          value: sorrell3.mps.gz
          valueType: STRING_VALUE_TYPE
        - property:
            identifier: time_limit_s
          value: 3600
          valueType: NUMERIC_VALUE_TYPE
        - property:
            identifier: progress_interval_s
          value: 60
          valueType: NUMERIC_VALUE_TYPE
values: #LIST OF PropertyValues
   - value: $VALUE
      valueType: $VALUE_TYPE
      property: 
           identifier: $IDENT

Additional context.

Should we upgrade sample-stores - do we have a way?

[michael-johnston]

@AlessandroPomponio FYI

[AlessandroPomponio]

We should think this one through, because we've already had an instance (DiscoverySpaceResource) where we had to introduce a "minimization" function to make the output easier for the users to understand.

I think the general case is this:

1. We want to have a rich internal representation of objects that allows us to perform complex validation and that makes it easy to perform common operations.
2. We want to have a slim external representation to either make something more user friendly or save storage space/tokens/whatever

The issue with what we want to do here, though, is that the models would often be incompatible, even though it should always be possible to go from one to the other.

Now, I'm not sure how we can best do this, but I feel like we should find what kind of pattern governs this (whether it's DTO or something else), and then use this as a use-case for applying this and then extend it to many other resources.

Anyway, in general I think we should have two models for each of these instances