StorageClientConfig.from_file ignores all options in implicit profile creation

[pavelgein]

Summary

Subject

Steps to Reproduce

1. Create a config file named msc_bug.yaml

profiles:
    s3-my-bucket:
        storage_provider:
            type: s3
            options:
                endpoint_url: https://my-endpoint.s3.com
                base_path: my-bucket

2. Create a script named msc_bug.py

import multistorageclient as msc


def main():
    client, _ = msc.resolve_storage_client("s3://my-bucket/my_file.txt")
    assert client._storage_provider._endpoint_url == "https://my-endpoint.s3.com", f"{client._storage_provider._endpoint_url=}"

if __name__ == "__main__":
    main()

3. Run

MSC_MSC_CONFIG=msc_bug.yaml python msc_bug.py 

Environment Details

• MultiStorageClient version
• Operating system and version
• Python version

Expected Behaviour

Url, as well as all other options, should be passed

Additional Context

Add any other context or screenshots about the bug here.

[pavelgein]

In this line https://github.com/NVIDIA/multi-storage-client/blob/main/multi-storage-client/src/multistorageclient/config.py#L77
only some of the options are passed

[commiterate]

Reproduction with MSC 0.44.0: https://github.com/NVIDIA/multi-storage-client/tree/reproduction/19/reproduction

This is currently intended behavior since multistorageclient.shortcuts.resolve_storage_client() (ref) doesn't know which profile to choose if the MSC config file looks like this:

profiles:
  s3-profile:
    storage_provider:
      type: s3
      options:
        endpoint_url: https://my-endpoint.s3.com
        base_path: my-bucket
  s3-emulator-profile:
    storage_provider:
      type: s3
      options:
        endpoint_url: http://localhost:7070
        base_path: my-bucket

From an s3://{bucket}/... URL, we only know the storage provider type (s3) and the bucket (which is used for the base_path). If we filtered profiles for that, we'd get back {"s3-profile", "s3-emulator-profile"}.

Since there currently isn't a tie breaking method, the implicit profiles feature instead creates a separate profile with minimum options that can be inferred from the URL (e.g. _s3-my-bucket with type: s3 + options.base_path: my-bucket set. This implicit profile naming scheme may change in the future).

Today users need to use msc://{profile name} instead to get an existing profile, so msc://s3-my-bucket/my_file.txt. The reproduction shows the differences in the resolved MSC storage clients.

[commiterate]

Note that there's an existing path mapping feature that lets users map s3:// URLs to msc:// URLs so code can still continue to use s3:// URLs.

https://nvidia.github.io/multi-storage-client/references/configuration.html#path-mapping

Using the reproduction as the base, this would work:

msc_bug.yaml

profiles:
  s3-my-bucket:
    storage_provider:
      type: s3
      options:
        endpoint_url: https://my-endpoint.s3.com
        base_path: my-bucket
path_mapping:
  s3://my-bucket/: msc://s3-my-bucket/

$ just run
# Prepare the virtual environment.
uv sync --all-extras --python python
Resolved 218 packages in 4ms
Audited 32 packages in 1ms
# Run the reproduction.
MSC_CONFIG=msc_bug.yaml uv run msc_bug.py
Resolving MSC client for msc://s3-my-bucket/my_file.txt...
s3
https://my-endpoint.s3.com
my-bucket

Resolving MSC client for s3://my-bucket/my_file.txt...
s3
https://my-endpoint.s3.com
my-bucket

A path mapping of s3://: msc://s3-no-base-path/, however, isn't supported yet.

[pavelgein]

Thanks, it works