EDI Data Package Staging Service

A web service for staging data packages for upload to PASTA.

Overview

The staging service provides a public API and web interface for uploading and managing data packages before final submission to PASTA. It eliminates the requirement for users to maintain their own publicly accessible hosting infrastructure.

Features

• REST API protected by JWT bearer token or API key
• Web UI with upload dashboard
• Asynchronous upload processing
• S3-backed storage
• Per-upload status reporting with checksums (SHA-1/MD5)
• Configurable data lifecycle management (garbage collection)
• Integration with the EDI IAM service for authentication

Architecture

• Runtime: Python 3.x
• Server: nginx + gunicorn + uvicorn
• ORM: SQLAlchemy
• Database: PostgreSQL
• Storage: Amazon S3 via boto3
• Auth: JWT bearer tokens / API keys via IAM service

API

Method	Endpoint	Description
POST	/upload?[key=|token=]	Stage a data package (returns 202)
GET	/upload?[key=|token=]	List all of the caller's uploads
GET	/upload/{id}/data?[key=|token=]	Download a staged file (302 redirect)
DELETE	/upload/{id}?[key=|token=]	Delete a staged upload

Authentication

All API endpoints require either a ?token=<jwt> or ?key=<api-key> query parameter. API keys are issued through the EDI IAM service and are automatically exchanged for a JWT before each request is processed.

Example

# Stage a file using an API key
curl -X POST "https://<host>/upload?key=<api-key>" \
  -F "file=@data.zip" \
  -F "label=my-dataset-2026"

# List your uploads
curl "https://<host>/upload?token=<jwt>"

# Download a file
curl -L "https://<host>/upload/<id>/data?token=<jwt>"

# Delete an upload
curl -X DELETE "https://<host>/upload/<id>?token=<jwt>"

Upload Report

Each entry in the upload list includes:

• Upload ID
• Label
• Upload status (pending / success / failure)
• SHA-1 and MD5 checksums (populated after successful upload)
• Data download URL (/upload/{id}/data)
• Expiry timestamp (time-to-live)

Data Lifecycle

Staged data is automatically removed after a configurable retention period (default: one month). The timer starts at the time of upload.

Update environment

pixi run freeze
git add requirements.txt
git commit

Preparing the database on first install

sudo -u postgres createuser --pwprompt staging
sudo -u postgres createdb -O staging staging

Then update config.py with the password you set for the staging Postgres user.

Export Postgres DB to another server

Export:

sudo -su postgres pg_dump -U staging -h localhost staging > /tmp/staging-dump.sql

Import:

sudo -su postgres
psql -U staging -h localhost -c 'drop database if exists staging;'
psql -U staging -h localhost -c 'create database staging;'
psql -U staging -h localhost -c 'alter database staging owner to staging;'
psql -U staging -h localhost staging < /tmp/staging-dump.sql

Development Setup

Prerequisites

• Python 3.11+
• PostgreSQL
• AWS credentials with S3 access
• Access to an EDI IAM service instance

Install

git clone <repo-url>
cd staging
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Configure

Copy the config.py template and fill in the required values:

cp webapp/config.py.template webapp/config.py

Run (development)

uvicorn app.main:app --reload

Run (production)

gunicorn app.main:app -k uvicorn.workers.UvicornWorker

Deployment

The service is deployed behind nginx. See deploy/ for configuration templates and the nginx site configuration.

Testing deployments target the web-x server.

EDI IAM service integration

The Staging service relies on the EDI IAM service for authentication and access control.

In order to use the staging service, users must have an account in the IAM service and obtain an API key or JWT token for authentication. The Staging service validates incoming requests by verifying the provided API key or JWT token with the IAM service.

In addition, users must have write permission on the staging_upload resource in the IAM service in order to upload new data entities to Staging. This resource should be set up by an IAM superuser as follows:

./cli/resource.py <superuser-key> create staging_uploads 'Staging service upload access' staging

The IAM UI can then be used by a superuser to grant write permission on the staging_uploads resource to individual users or groups as needed. The resource will be available in the IAM UI Permissions page when searching on resource type "staging".

License

See LICENSE.