Skip to content

List files from local cache when pending S3 upload#8

Open
Leonorus wants to merge 2 commits intosol1:mainfrom
Leonorus:terraform_upload
Open

List files from local cache when pending S3 upload#8
Leonorus wants to merge 2 commits intosol1:mainfrom
Leonorus:terraform_upload

Conversation

@Leonorus
Copy link
Copy Markdown

Issue #7: Terraform bpg/proxmox provider fails to read back files immediately after upload — failed to read file from "s3-iso:snippets/cloud-config-ubuntu.yaml".

Root cause: When PVE writes a file directly to the cache directory (e.g. via the Terraform provider), the file watcher needs 3+ seconds (debounce + stability check) before uploading to S3 and indexing metadata. The /v1/list endpoint only queries S3, so the file is invisible until the watcher completes the upload. The provider tries to read the file back immediately and fails because list_volumes() doesn't include it.

Fix: Modify handleList in internal/api/api.go to scan the local cache directory after querying S3, merging any files that exist locally but aren't yet in S3 into the response. Also improved the S3-unreachable fallback path to return locally cached files instead of an empty list.

Copy link
Copy Markdown
Member

@davekempe davekempe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall — clean approach, solid test coverage. One minor suggestion:

mergeLocalFiles should skip the .meta directory when recursing for images content. Currently it would treat .meta/ as a vmid subdirectory if it existed inside a content dir. Low risk since .meta lives at the cache root, but worth guarding:

if content == "images" {
    if entry.Name() == ".meta" {
        continue
    }
    // Recurse into VM ID subdirectories for images

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Leonorus
Copy link
Copy Markdown
Author

mergeLocalFiles now skips the .meta directory when recursing into subdirectories for images content

Copy link
Copy Markdown
Member

@davekempe davekempe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good — .meta guard added, tests pass, clean approach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants