End-to-end remote access foundation: UI, worker integration, URI/recent flow, and SSH challenge handling

[bnlawrence]

This PR establishes remote connectivity and browsing in xconv2, with worker-backed orchestration, and aligns the UI around practical open flows (Configure Remote, Open Remote, Open URI, and Recent).

What This PR Delivers

1. End-to-end remote access.
2. Remote protocol configuration and session setup for S3, SSH, and HTTPS.
3. Worker control-task plumbing for remote prepare/list/open/release.
4. Worker-side warm remote session reuse and cleanup.
5. Remote file navigator UI integrated with worker-side listing/open.
6. URI-driven opening flow and integration with recent files.
7. Settings persistence/migration for remote state.
8. Authentication preflight/prompt/retry handling for SSH, including ProxyJump bastion prompts.
9. Expanded test coverage across flow, auth, and integration behaviour.

Architecture and Flow

1. UI gathers remote config or URI input.
2. Main window resolves target protocol and constructs remote descriptor/spec.
3. Worker receives typed control tasks:
   • REMOTE_PREPARE
   • REMOTE_LIST
   • REMOTE_OPEN
   • REMOTE_RELEASE
4. Worker emits structured status/result messages (REMOTE_STATUS, REMOTE_LIST_RESULT, REMOTE_OPEN_RESULT) consumed by GUI.
5. Remote sessions are pooled and reused worker-side to reduce repeated setup latency.

Key UX Changes

1. Remote actions split into Configure Remote and Open Remote.
2. Open URI defaults from most recent remote URI.
3. Recent menu labels for remote entries now show filename (Host alias).
4. Recent tooltips and Open URI defaults for S3 use shareable host-form URIs.
5. Unknown-host URI handling routes users into configure-add-new flow.
6. SSH auth challenges are detected early and prompted before slow timeout paths.
7. Auth failure offers retry without restarting the app.
8. ProxyJump prompt includes explicit bastion context:

• Authenticating with bastion host [hostname] before proxyjump to [target host].

Protocol Capability (Current State)

1. S3: fully working end-to-end path in this branch (configure, browse, open, URI/recent integration).
2. SSH and HTTPS:

• configuration + navigation/auth scaffolding is implemented,
• due to upstream cf/cfdm read-path constraints, behavior beyond navigation/connectivity is intentionally conservative.

3. This PR is structured so upstream filesystem/read improvements can be adopted with minimal UI/flow churn.

Authentication Handling

1. SSH preflight probes advertised auth methods.
2. Password and keyboard-interactive challenge paths prompt users immediately.
3. Wrong/expired secrets are cleared and retried via guided prompt.
4. ProxyJump supports separate bastion secret collection and caching (session-memory only).
5. Secrets are transient in-process values only; no internal keychain persistence added.

Other URI Improvements

1. Remote URIs are preserved verbatim in settings (no path normalization corruption).
2. S3 resolver supports both:
   • bucket-first URI form,
   • endpoint-host shareable URI form.
3. Alias-aware recent rendering and open resolution keep endpoint/auth routing stable.

Known Limitations

1. S3 is the strongest validated end-to-end protocol path in this PR, but we will likely replace this path when we can use fsspec objects directly, not least to eventually support PyActiveStorage.

2. SSH/HTTPS beyond the delivered navigation/auth/connectivity scaffolding remains bounded by upstream cf/cfdm read-path support (i.e. the need to support handover of a file ssytem object).

3. Multi-step custom keyboard-interactive Q&A prompts beyond single-secret entry are not fully modelled yet.

[Copilot]

Pull request overview

This PR establishes a first end-to-end “remote open” foundation in xconv2 by adding worker-owned remote session orchestration (prepare/list/open/release), GUI flows for configuring/opening remotes and URIs, and expanded plotting/layout + options infrastructure (including lineplot options), with accompanying tests and documentation.

Changes:

• Add worker-side remote control tasks (REMOTE_PREPARE/LIST/OPEN/RELEASE) with session pooling and GUI IPC handling for remote navigation/open flows.
• Introduce a remote file navigator dialog and URI/recent integration (including recent URI alias rendering and HTTPS migration).
• Refactor/extend plotting UI/options (shared title/annotation sections, lineplot options dialog) and plot layout helpers.

Reviewed changes

Copilot reviewed 45 out of 54 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
xconv2/xconv_cf_interface.py	Switch field metadata payloads to structured dict rows; refactor layout helpers usage; cfplot save/backends tweaks.
xconv2/worker.py	Add typed control-task header parsing, remote session pool + REMOTE_* handlers, and auxiliary helper inlining for saved scripts.
xconv2/ui/settings_store.py	Persist new remote/URI-related settings, migrate HTTP→HTTPS keys, preserve remote URIs verbatim in recents.
xconv2/ui/selection_controller.py	Make num2date calendar optional; enable options for lineplot/contour selections.
xconv2/ui/remote_file_navigator.py	New remote filesystem spec/descriptor helpers, remote browsing dialog, filtering, ProxyJump handling, and logging dialog.
xconv2/ui/plot_options_shared.py	New shared UI builder for titles/annotations + margins used across plot option dialogs.
xconv2/ui/menu_controller.py	Add Configure Remote/Open Remote actions; improve recent labels/tooltips via host helpers.
xconv2/ui/lineplot_options_controller.py	New lineplot options dialog controller (legend, text sizes, shared sections).
xconv2/ui/field_metadata_controller.py	Enforce structured mapping-based field metadata and simplify properties parsing accordingly.
xconv2/ui/contour_options_controller.py	Adopt shared plot options sections; normalize property display; prevent default-button Enter behavior.
xconv2/ui/init.py	Export additional dialogs and remote navigator at package level.
xconv2/plot_layout_helpers.py	New module for shared matplotlib layout helpers (title/annotation padding).
xconv2/main_window.py	Implement worker-backed remote flows (prepare/list/open/release), URI open routing, SSH auth preflight/prompt/retry, and task timing.
xconv2/lineplot.py	Add legend toggles, text sizing, and page title/annotation rendering with layout padding logic.
xconv2/core_window.py	Wire new controllers, recent URI alias labeling/tooltips, and add stubs/bridges for remote actions in core window.
xconv2/cf_templates.py	Emit “Saved plot to …” status for lineplot saves in generated worker code.
xconv2/aaa/aaa_config.py	New helper to load MinIO-style S3 alias configuration (locations/servers).
xconv2/init.py	Bump version tag from pre-alpha to pre-beta.
tests/test_worker_remote_tasks.py	Unit tests for remote session reuse and remote read fallbacks.
tests/test_worker_remote_integration.py	Docker/MinIO-backed integration tests for remote S3 open/metadata and recent reopen.
tests/test_settings_store.py	Tests for verbatim remote URI recent handling and local path expansion.
tests/test_selection_controller.py	Tests for calendar-less time formatting and lineplot options enabling.
tests/test_remote_file_navigator.py	Unit tests for remote spec building, listing normalization, filters, ProxyJump parsing, zarr detection.
tests/test_remote_configuration_dialog.py	Tests for S3/SSH/HTTPS config persistence helpers and open dialog inputs.
tests/test_remote_auth_flow.py	Tests for SSH auth preflight/prompt/retry and ProxyJump bastion prompting logic.
tests/test_load_selected_file.py	Update tests for structured metadata rows; add tests for control task headers and recent/URI helpers.
tests/test_lineplot.py	Expand tests for legend toggles, page title/annotations, text sizes, and layout helper usage.
tests/test_coordinate_flow.py	Add tests for lineplot options dialog routing and remote status/task timing handling.
tests/test_contour_options_dialog.py	UI regression test: Enter in title fields should not trigger annotation chooser.
tests/test_contour.py	Add test for contour property text normalization.
tests/test_cf_interface.py	Update tests for dict-based field metadata and cfplot viewer=None behavior.
tests/conftest.py	Add Docker/MinIO fixtures and integration marker; silence noisy loggers.
pyproject.toml	Add runtime deps for remote navigation (fsspec/s3fs/paramiko) and test deps (docker/minio).
environment-integration.yml	Add optional conda env for integration tests (plus pip docker/minio).
docs/uml/remote_worker_warmup_sequence.puml	New sequence diagram for worker-only remote warmup/list/open flow.
docs/architecture/remote_navigation_and_worker_warmup.md	New architecture note describing remote navigation + IPC/session pooling design.
data/example_field_0.zarr3/zarr.json	Add Zarr v3 example dataset root metadata.
data/example_field_0.zarr3/time/zarr.json	Add Zarr v3 example dataset time metadata.
data/example_field_0.zarr3/time/c	Add Zarr v3 example dataset time chunk payload.
data/example_field_0.zarr3/q/zarr.json	Add Zarr v3 example dataset variable metadata.
data/example_field_0.zarr3/q/c/0/0	Add Zarr v3 example dataset variable chunk payload.
data/example_field_0.zarr3/lon_bnds/zarr.json	Add Zarr v3 example dataset bounds metadata.
data/example_field_0.zarr3/lon_bnds/c/0/0	Add Zarr v3 example dataset bounds chunk payload.
data/example_field_0.zarr3/lon/zarr.json	Add Zarr v3 example dataset lon metadata.
data/example_field_0.zarr3/lon/c/0	Add Zarr v3 example dataset lon chunk payload.
data/example_field_0.zarr3/lat_bnds/zarr.json	Add Zarr v3 example dataset bounds metadata.
data/example_field_0.zarr3/lat_bnds/c/0/0	Add Zarr v3 example dataset bounds chunk payload.
data/example_field_0.zarr3/lat/zarr.json	Add Zarr v3 example dataset lat metadata.
data/example_field_0.zarr3/lat/c/0	Add Zarr v3 example dataset lat chunk payload.
README.md	Document new integration conda environment + additional architecture docs.
.gitignore	Minor whitespace-only change.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[bnlawrence]

@valeriupredoi All the tests are failing here, but are not failing in my environment. This is what I see:

platform darwin -- Python 3.12.12, pytest-9.0.2, pluggy-1.6.0
rootdir: /Users/bnl28/Repositories/xconv2
configfile: pyproject.toml
testpaths: tests
plugins: zarr-3.1.5
collected 100 items                                                                                              

tests/test_cf_interface.py ..........                                                                      [ 10%]
tests/test_contour.py ......                                                                               [ 16%]
tests/test_contour_options_dialog.py .                                                                     [ 17%]
tests/test_coordinate_flow.py .............                                                                [ 30%]
tests/test_lineplot.py .........                                                                           [ 39%]
tests/test_load_selected_file.py ...............                                                           [ 54%]
tests/test_remote_auth_flow.py ..........                                                                  [ 64%]
tests/test_remote_configuration_dialog.py ..........                                                       [ 74%]
tests/test_remote_file_navigator.py .............                                                          [ 87%]
tests/test_selection_controller.py ..                                                                      [ 89%]
tests/test_settings_store.py ..                                                                            [ 91%]
tests/test_worker_remote_integration.py ..x                                                                [ 94%]
tests/test_worker_remote_tasks.py ...                                                                      [ 97%]
tests/test_worker_save_script.py ...                                                                       [100%]

================================================ warnings summary ================================================
../../miniforge3/envs/work26/lib/python3.12/site-packages/cfplot/cfplot.py:36
../../miniforge3/envs/work26/lib/python3.12/site-packages/cfplot/cfplot.py:36
  /Users/bnl28/miniforge3/envs/work26/lib/python3.12/site-packages/cfplot/cfplot.py:36: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    if StrictVersion(cf.__version__) < StrictVersion(cf_version_min):

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================================== 99 passed, 1 xfailed, 2 warnings in 4.77s ====================================

Copilot has a bunch of suggestions which are more in your area than mine.

The failure was caused by a "Fatal Python error: Aborted" during test execution, resulting in a process abort with exit code 134 (core dumped). This type of error (signal 6 / SIGABRT) is most often due to low-level issues, such as:

• Incompatible or corrupt Python C-extension binaries (especially numpy or numcodecs, referenced in your logs)
• Memory corruption or double free in native code (including in 3rd party Python packages)
• Incompatible binary wheels (due to conflicts in dependency versions or Python version mismatches)

It recommmended a bunch of stupid pins, but I guess some of the following could be applied in GHA?

Summary

• Pin versions for numpy and numcodecs for binary compatibility.
• Ensure every run uses a fresh environment.
• Run tests serially (-n 1).
• Clean caches before installing.

After making these changes, rerun your workflow. If the error persists, examine C-extension usage and try upgrading or downgrading problem dependencies progressively.

[valeriupredoi]

am looking at it right now @bnlawrence - issue is fully reproducible on GHA machine, will see if I can reproduce it locally

[valeriupredoi]

hey @bnlawrence this branch passes all tests for me both locally and on JASMIN (eg an env without Xwin), there must be something crooked in the GHA runtime env (dependency envs are all identical, including GHA) - so you're all in the clear, I'll have to see what the heck's up in the GHA runtime env and fix it. Please don't follow Copilot#s bs advice and start pinning - it's not xconv2's env that needs be fixed 🍺

[bnlawrence]

I was totally not going to do that, not least because it wanted Numpy from before Noah ... hence my "stupid pins".

Thanks. I can merge this, and will shortly.

[valeriupredoi]

@bnlawrence green green green 😃

Proves out there is a wee bit of an issue with QT being set up (or, not set up) on the GHA machine, solution from https://stackoverflow.com/questions/75497408/github-action-pytest-exit-code-134

V+Stackoverflow 1 - 0 Copilot 😁