Decide data flow process for scanning

[grossir]

We need to decide how:

1. the scanner will upload the scans to S3 or another system
   - if it is another system, how we will migrate that into S3
2. we will validate uploaded data
3. we will track completeness of each scanned reporter
4. we will prioritize the order of scanning

We have already decided the S3 path we want, we are missing decisions on storage classes and lifecycle

[flooie]

I think this is simple.

Upload to S3:
We find an app to sync from our locked-down Android devices directly to S3. I'm getting a test device tomorrow and will verify this works.

Validation:
Lambda function triggers on upload, does the processing we need, and generates an email report or some validation. Though I think we'll want the scanner to review the documents before submitting to catch obvious issues at the source.

Tracking/completeness:
Lambda can handle logging what's uploaded and flagging gaps. We have a good idea of how many volumes we need, but that list grows. I don't have a good way to track it yet. Open to ideas here.

Prioritization:
For now, let's generate a spreadsheet of all volumes. Start with the newest stuff — have them track ongoing arrivals first since those are most important. Then work backward from volumes present. We generally view newer
as more important.

Honestly, they may not even need a computer - just the locked-down Android device. Will verify tomorrow when I test out features, depending on when the device arrives and setting it up. Need to decide if we need a new google account to go with this device.

[mlissner]

Honestly, they may not even need a computer

I was thinking that too. We just need a little mobile site where they can upload stuff. Seems like it could be quite smooth.

[quevon24]

I think these are some possible options:

Option 1: Django Web App with login and upload form

Build a simple Django web application with a login page and a file upload form. Users open the URL from any device browser, log in with their Django credentials, pick the PDF exported from vFlat and submit it. Django receives the file on the server and pushes it to S3 directly using boto3. AWS credentials live only on the server and are never exposed to the browser or the user.

Since it's just a web app, no installation is required. It works on any phone browser, desktop, or tablet.

For the implementation we have two paths: build a new small standalone Django app just for this purpose, or reuse CourtListener directly by adding a new URL with a dedicated upload form. In either case, the form would use a special API key to authenticate upload requests and keep the endpoint secure, ensuring only authorized users can submit files.

• What needs to be built: a Django app with a login view, an upload view, and a minimal frontend or can reuse CourtListener's existing templates and auth system.

User opens URL on any browser (phone, desktop, tablet)
      ↓
Django login page → authenticates with Django auth
      ↓
User picks PDF from device → submits upload form
      ↓
Django receives file → boto3 → uploads directly to S3
      ↓
S3 stores file at custom path (need to define it)

Option 2: Mobile App + Django API

The user authenticates through a mobile app against a Django API using an API key. Once authenticated, the app sends the PDF file directly to the API. Django receives the file on the server and pushes it to S3 using boto3, so AWS credentials and S3 details are never exposed to the device or the user.

For the mobile app we have two paths: build a native Android app, or use Flutter which gives us Android and iOS support from the same codebase

• What needs to be built: a Django API endpoint in CourtListener or a new standalone Django API, and a mobile app either as a native Android app or a Flutter app with Android and iOS support.

Authenticates using Django endpoint to get token
      ↓
Android (native) or Flutter App (Android/iOS) → POST /api/upload/ (authenticated with API key or Token)
      ↓
Django API receives file → boto3 → uploads directly to S3
      ↓
S3 stores file at custom path (need to define it)

Option 3: Flutter Web App + AWS Amplify + AWS Cognito

The user accesses a web URL from any device (phone browser, desktop, etc.) and logs in through a form backed by AWS Cognito, which is Amazon's user identity service that handles signup, login, and password reset without needing a custom auth backend. After login, Cognito issues temporary, scoped AWS credentials directly to the browser session.
AWS Amplify is the SDK that connects the Flutter app to AWS services. It wraps Cognito for authentication and handles S3 uploads directly from the browser using the temporary credentials, so no backend API is needed at any point.

Since it is built with Flutter, the same codebase runs as a web app accessible from any phone browser and can also be compiled as a native Android/iOS app if needed in the future.

• What needs to be built: a Flutter web app with a login screen and a file picker/upload screen. No backend API needed.
• Security considerations: access is controlled via Cognito user accounts. Only invited users with a valid account can log in and upload. Credentials expire automatically and are scoped per user.

User opens web URL on phone browser
      ↓
Flutter Web App → Amplify SDK → Cognito (login)
      ↓
Cognito issues temporary AWS credentials (scoped to uploads/user_id/)
      ↓
Flutter App → Amplify SDK → uploads PDF directly to S3
      ↓
S3 stores file at custom path (need to define it)

Option 4: Google Drive shared folder (no app needed)

Users manually export the PDF from vFlat using its built-in "Export to Google Drive" feature and place it into a shared Drive folder. A backend process polls this folder periodically and picks up new files for processing. No custom development is required on the user side. The only manual step is dropping the exported PDF into the shared folder, which can be done natively from any phone.

Access is controlled by sharing the folder only with specific Gmail accounts. Those users can upload to that folder and nothing else in the infrastructure is exposed to them. This is the simplest access control model for a known, closed group of volunteers.

• What needs to be built: nothing on the user side. Backend polling logic in Lambda (This is one option).
• Security considerations: folder access is managed entirely through Google account sharing. Only invited Gmail accounts can upload.

vFlat → Export to Google Drive (built-in feature)
      ↓
User drops PDF into shared folder (path to define)
      ↓
Scheduled Lambda polls folder every 5 minutes
      ↓
Downloads new files
      ↓
Process files

---

After looking at all the options, here is why I think we should go with one of these two:

Option 1 (Django web app) is my favorite. We already know how to build Django apps and CourtListener is already Django, so we could just add a new URL with an upload form and call it a day. Minimal effort, no new tech, and we reuse what we already have. Or we could build a small standalone Django project just for this, which also would not be difficult given that we already know how to do it.

Option 2 (mobile app + Django API) is also a good option. The backend part is simple, just two endpoints, one for auth and one for the upload. For the mobile app we could use Flutter, which means the same codebase runs on both Android and iOS, and it would still talk to the same Django API. I already have some experience building Android apps and consuming Django APIs from Android so this is not uncharted territory for me. Using Flutter would just extend that to iOS as well without much extra effort.

Options 3 and 4 are not bad but Option 3 brings in Flutter, Amplify and Cognito which is a lot of new stuff for something we can solve with Django. Option 4 depends on Google Drive which adds an external dependency we do not control.
What does the team think?

[quevon24]

I think this is simple.

Upload to S3: We find an app to sync from our locked-down Android devices directly to S3. I'm getting a test device tomorrow and will verify this works.

Validation: Lambda function triggers on upload, does the processing we need, and generates an email report or some validation. Though I think we'll want the scanner to review the documents before submitting to catch obvious issues at the source.

Tracking/completeness: Lambda can handle logging what's uploaded and flagging gaps. We have a good idea of how many volumes we need, but that list grows. I don't have a good way to track it yet. Open to ideas here.

Prioritization: For now, let's generate a spreadsheet of all volumes. Start with the newest stuff — have them track ongoing arrivals first since those are most important. Then work backward from volumes present. We generally view newer as more important.

Honestly, they may not even need a computer - just the locked-down Android device. Will verify tomorrow when I test out features, depending on when the device arrives and setting it up. Need to decide if we need a new google account to go with this device.

For the validation part we can use Lambda and S3 notifications to automatically detect when a new file is uploaded (you already mentioned this), process it with blackletter and save the redacted output to a new S3 location. But before we think about the implementation of the review step I have some questions to better understand what you expect:

• How do you expect the scanner to review the redacted document?
• Should the scanner be able to compare the original and the redacted document side by side to verify what was removed?
• What should the email report contain?
• What kind of validation are you expecting at this step? Is it just a visual check by the scanner?
• How should the scanner mark a document as good to go?

[grossir]

I think a Django web app "Option 1" will also give us some flexibility to control actual filepath of the upload (which is important metadata). We could create a simple upload page with some choice fields so the scanner can select what he is uploading: name of reporter, volume number, number of pages, maybe even a picture of the book). From the normalized values, we create the proper S3 paths.

About the other options,

I don't think we should make a mobile app, unless someone has expertise on creating them. Even the dev environment is different.

Google Drive seems easy enough, but, how would we ensure files are named correctly by the scanner. That seems error prone.

[mlissner]

For the implementation we have two paths: build a new small standalone Django app just for this purpose, or reuse CourtListener directly

Legally, it's probably a good idea to keep these separate just so the raw data is siloed from CourtListener entirely.

Option 2: Mobile App + Django API

This sounds quite complex. Just dealing with the app stores feels like a pain. I'm not sure what we get from this that we don't get from option 1.

Google Drive seems easy enough, but, how would we ensure files are named correctly by the scanner. That seems error prone.

I had a similar thought, but I guess we could just ask AI to tell us what the scan is? But yeah:

We could create a simple upload page with some choice fields so the scanner can select what he is uploading: name of reporter, volume number, number of pages, maybe even a picture of the book).

This is a good point too.

---

I think if we do option 1 (django app), we could probably stand it up in a day or two, which is pretty great. I bet claude could single-shot it if you gave it a good enough spec!

About reviewing/processing, I think we should solve that once the content is flowing into S3, right?

[grossir]

Some updates to the data flow from what we have been talking this week.

Any suggestions, corrections, etc @flooie @mlissner @quevon24 ?

We may need some model additions to fulfill this

1. Scans using

• Who: scanners in LLoC
• How: manual process

2. Upload scans

• Who: scanners
• How: using the upload portal, from their phones. (Right now, in Google Drive). File metadata is controlled by the form input. Signed URLs discussion is relevant here

2.1 Automated checks

• 2.1.1 Django checks
  - Who: automated
  - How: right now, it only checks that the uploaded file is a PDF. Can we improve this?

• 2.1.2 Blackletter checks (needs sub issue)
  - Who: automated
  - How: Bill demoed this week: checks for missing or blurry pages, bleed through, etc. Reports back to the user. We need to settle on a return format

3. Corrections (needs sub issue)

• 3.1.1 Correction upload (needs sub issue)

  • Who: scanners
  • How: they take the reviewer or blackletter comments. They scan those and upload them.

• 3.1.2 Correction integration

  • Who: automated
  • When: we shouldn't integrate the PDF everytime they upload a single page. We need a way for them to mark that the corrections are complete
  • How: Code to insert or replace uploaded pages into the original PDF, creating a new PDF

After these steps, go back to automated checks

4. Rewiew by staff

• Who: We haven't decided yet. Could they review each other? Should they review each other? (PR related to this)
• How: Skim through the scan to look for any errors.

From here, we may jump to step 3 again if needed

5. Opinion extraction

After the file looks good, we run it against black letter for opinion separation, redaction, masking, etc.

6. Redaction checks

We still need to plan this process