artifact download URL is not resumable

[davepacheco]

From this buildomat run:
https://github.com/oxidecomputer/omicron/runs/38078608713

I wanted to download /work/repo.zip, which is:
https://buildomat.eng.oxide.computer/wg/0/artefact/01JND89XBVFYKYFJXD6JFDA8XR/i361qB3lfswvUlvUnk2I8TAIInnC6R0aVLrufUE3xiKZ7Jpm/01JND8A832Z6DMSNDE05HMFHVH/01JNDBQH91APJC86M538P066PN/repo.zip

That seems to reliably fail to download just after the 1 GiB mark (a separate bug):

$ curl -O https://buildomat.eng.oxide.computer/wg/0/artefact/01JND89XBVFYKYFJXD6JFDA8XR/i361qB3lfswvUlvUnk2I8TAIInnC6R0aVLrufUE3xiKZ7Jpm/01JND8A832Z6DMSNDE05HMFHVH/01JNDBQH91APJC86M538P066PN/repo.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 50 2039M   50 1030M    0     0  5613k      0  0:06:12  0:03:07  0:03:05 5476k
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

I tried to retry it but:

$ curl -C- -O https://buildomat.eng.oxide.computer/wg/0/artefact/01JND89XBVFYKYFJXD6JFDA8XR/i361qB3lfswvUlvUnk2I8TAIInnC6R0aVLrufUE3xiKZ7Jpm/01JND8A832Z6DMSNDE05HMFHVH/01JNDBQH91APJC86M538P066PN/repo.zip
** Resuming transfer from byte position 1080246028
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (33) HTTP server does not seem to support byte ranges. Cannot resume.

This looks similar to #31, but the fix for that looks like it was fixing a different endpoint (/public/file/...).

[iximeow]

oh, yeah, variety::basic::artefact doesn't get or pass along range headers which are presumably present and ignored in github/server/src/http.rs' async fn artefact. if that were plumbed through i think range requests would work right on this endpoint too and -C - would be happy.

[jclulow]

I have reworked this a bit, and have been able to get range requests to work for objects that have already been archived to the object store. I believe passing a range request into progenitor is still a work in progress (see oxidecomputer/progenitor#745 and oxidecomputer/progenitor#750) so range requests don't yet work before the range request is complete.

I've also reworked the github frontend (which is what you're downloading from here) so that it can request the file directly from S3, rather than having to ask the backend API server to proxy it from S3. My hope is that this will cut down on the reliability issues with the actual transfers.

[jclulow]

I've been able to grab the whole file a couple of times at the office, at least:

$ curl -O 'https://buildomat.eng.oxide.computer/wg/0/artefact/01JND89XBVFYKYFJXD6JFDA8XR/i361qB3lfswvUlvUnk2I8TAIInnC6R0aVLrufUE3xiKZ7Jpm/01JND8A832Z6DMSNDE05HMFHVH/01JNDBQH91APJC86M538P066PN/repo.zip'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2039M  100 2039M    0     0  53.4M      0  0:00:38  0:00:38 --:--:-- 63.7M

[ahl]

I have reworked this a bit, and have been able to get range requests to work for objects that have already been archived to the object store. I believe passing a range request into progenitor is still a work in progress (see oxidecomputer/progenitor#745 and oxidecomputer/progenitor#750) so range requests don't yet work before the range request is complete.

I've also reworked the github frontend (which is what you're downloading from here) so that it can request the file directly from S3, rather than having to ask the backend API server to proxy it from S3. My hope is that this will cut down on the reliability issues with the actual transfers.

Josh I had thought the ball was in your court for both the progenitor feature (i.e. doesn't that already work as intended?) and the PR (does it solve your problem?)