Race condition in test_wrapper unit tests

[jason-callaway]

If testobject doesn't already exist in the target bucket the test_download test will fail. After re-running the unit tests everything will work because that object will have already been copied up.

root@pkg-test-2004-3:~/client-side-encryption# python3 test_wrapper.py 
test_download (gcs_client_lib_test.TestGCSWrapper)
Test copy from cloud with local decryption. ... ERROR
test_upload (gcs_client_lib_test.TestGCSWrapper)
Test copy to cloud with local encryption. ... ok
test_download (google_cloud_storage_test.TestGCSWrapper)
Test copy from cloud with local decryption. ... ok
test_upload (google_cloud_storage_test.TestGCSWrapper)
Test copy to cloud with local encryption. ... ok
test_application_credentials (gsutil_test.TestGsutilWrapper)
Ensure our application credentials can copy and list files in GCS. ... ok
test_encryption (gsutil_test.TestGsutilWrapper)
Verify that files copied to GCS have been encrypted. ... gsutil is being wrapped. Standard gsutil available at: /snap/bin/gsutil
Copying file:///root/.gsutil-wrapper/igE4rM0n//testobject [Content-Type=application/octet-stream]...
/ [1 files][  156.0 B/  156.0 B]                                                
Operation completed over 1 objects/156.0 B.
Setting metadata on gs://aw-314374a4-9a5a-4090-a9db-663-storage/testobject...
/ [1 objects]
Operation completed over 1 objects.
ok
test_gsutil_wrapper (gsutil_test.TestGsutilWrapper)
Test copy to cloud and copy from cloud with local encryption. ... gsutil is being wrapped. Standard gsutil available at: /snap/bin/gsutil
Copying file:///root/.gsutil-wrapper/0RWlVsWp//testobject [Content-Type=application/octet-stream]...
/ [1 files][  156.0 B/  156.0 B]                                                
Operation completed over 1 objects/156.0 B.
Setting metadata on gs://aw-314374a4-9a5a-4090-a9db-663-storage/testobject...
/ [1 objects]
Operation completed over 1 objects.
gsutil is being wrapped. Standard gsutil available at: /snap/bin/gsutil
Copying gs://aw-314374a4-9a5a-4090-a9db-663-storage/testobject...
/ [1 files][  156.0 B/  156.0 B]                                                
Operation completed over 1 objects/156.0 B.
ok

======================================================================
ERROR: test_download (gcs_client_lib_test.TestGCSWrapper)
Test copy from cloud with local decryption.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/google/cloud/storage/blob.py", line 1040, in download_to_file
    self._do_download(
  File "/usr/local/lib/python3.8/dist-packages/google/cloud/storage/blob.py", line 899, in _do_download
    response = download.consume(transport, timeout=timeout)
  File "/usr/local/lib/python3.8/dist-packages/google/resumable_media/requests/download.py", line 168, in consume
    self._process_response(result)
  File "/usr/local/lib/python3.8/dist-packages/google/resumable_media/_download.py", line 185, in _process_response
    _helpers.require_status_code(
  File "/usr/local/lib/python3.8/dist-packages/google/resumable_media/_helpers.py", line 106, in require_status_code
    raise common.InvalidResponse(
google.resumable_media.common.InvalidResponse: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/client-side-encryption/tests/wrapper/gcs_client_lib_test.py", line 85, in test_download
    self.blob.download_to_filename(self.plaintext_path)
  File "/root/client-side-encryption/encryption_wrapper/storage.py", line 266, in download_to_filename
    super().download_to_filename(filename, client, start, end, raw_download,
  File "/usr/local/lib/python3.8/dist-packages/google/cloud/storage/blob.py", line 1134, in download_to_filename
    self.download_to_file(
  File "/usr/local/lib/python3.8/dist-packages/google/cloud/storage/blob.py", line 1052, in download_to_file
    _raise_from_invalid_response(exc)
  File "/usr/local/lib/python3.8/dist-packages/google/cloud/storage/blob.py", line 3659, in _raise_from_invalid_response
    raise exceptions.from_http_status(response.status_code, message, response=response)
google.api_core.exceptions.NotFound: 404 GET https://storage.googleapis.com/download/storage/v1/b/aw-314374a4-9a5a-4090-a9db-663-storage/o/testobject?alt=media: No such object: aw-314374a4-9a5a-4090-a9db-663-storage/testobject: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)

----------------------------------------------------------------------
Ran 7 tests in 15.500s

FAILED (errors=1)
root@pkg-test-2004-3:~/client-side-encryption# python3 test_wrapper.py 
test_download (gcs_client_lib_test.TestGCSWrapper)
Test copy from cloud with local decryption. ... ok
test_upload (gcs_client_lib_test.TestGCSWrapper)
Test copy to cloud with local encryption. ... ok
test_download (google_cloud_storage_test.TestGCSWrapper)
Test copy from cloud with local decryption. ... ok
test_upload (google_cloud_storage_test.TestGCSWrapper)
Test copy to cloud with local encryption. ... ok
test_application_credentials (gsutil_test.TestGsutilWrapper)
Ensure our application credentials can copy and list files in GCS. ... ok
test_encryption (gsutil_test.TestGsutilWrapper)
Verify that files copied to GCS have been encrypted. ... gsutil is being wrapped. Standard gsutil available at: /snap/bin/gsutil
Copying file:///root/.gsutil-wrapper/F7Ybxf8o//testobject [Content-Type=application/octet-stream]...
/ [1 files][  156.0 B/  156.0 B]                                                
Operation completed over 1 objects/156.0 B.
Setting metadata on gs://aw-314374a4-9a5a-4090-a9db-663-storage/testobject...
/ [1 objects]
Operation completed over 1 objects.
ok
test_gsutil_wrapper (gsutil_test.TestGsutilWrapper)
Test copy to cloud and copy from cloud with local encryption. ... gsutil is being wrapped. Standard gsutil available at: /snap/bin/gsutil
Copying file:///root/.gsutil-wrapper/y7104mQi//testobject [Content-Type=application/octet-stream]...
/ [1 files][  156.0 B/  156.0 B]                                                
Operation completed over 1 objects/156.0 B.
Setting metadata on gs://aw-314374a4-9a5a-4090-a9db-663-storage/testobject...
/ [1 objects]
Operation completed over 1 objects.
gsutil is being wrapped. Standard gsutil available at: /snap/bin/gsutil
Copying gs://aw-314374a4-9a5a-4090-a9db-663-storage/testobject...
/ [1 files][  156.0 B/  156.0 B]                                                
Operation completed over 1 objects/156.0 B.
ok

----------------------------------------------------------------------
Ran 7 tests in 13.964s

OK