Make Statcan scraping more reusable

[melkuo]

Most of our metrics comes from Statcan. Right now we scrape Statcan in the OutcomeTracker repository with Github Actions. We'd like to make it easier to grab datasets from Statcan and automatically update them using a background job on this repo. The end goal is to make it easy to grab a Statcan URL, and have the data available and updated for the front end to consume.

[jameslong]

Hey @xrendan, I can pick this one up if that's ok with you? If you'd prefer to do this yourself (or have someone else do it) no worries.

Proposed Approach

Data models:

• Dataset: One per Statcan URL, includes url, name, update schedule, and latest jsonb data

Services:

• StatcanScraper: Fetches JSON data given a URL

Jobs:

• StatcanDatasetSyncJob: Enqueued to fetch and update JSON for a given dataset
• StatcanSchedulerJob: Cron job that runs hourly to find datasets needing updates and enqueues sync jobs for each

Routes:

Named datasets:

• GET /datasets/demographic-incomes

Or plain IDs:

• GET /datasets/1

Questions/unknowns:

• Data history: Should we store historical data? If so, we could use separate Datasource and Dataset models (1 Datasource → many Datasets over time)
• Cron scheduling: Is an hourly cron job acceptable? Simple and robust to server restarts, but means ~1 hour latency for updates
• Service organization: Any preference between /services directory vs domain-based organization?
• Adding new datasets: Not familiar with Avo, but it looks like it could handle the admin interface for this
• Issue breakdown: Would you prefer this work to be broken down into multiple PRs?

Let me know your thoughts. 🙂