feat: enrich csv and download by joaquimds · Pull Request #372 · commonknowledge/ts-mapped

joaquimds · 2026-03-14T21:01:04Z

No description provided.

Copilot

Pull request overview

Adds “enriched CSV download” support for CSV-based data sources by generating enrichment values on-the-fly and streaming them back as a downloadable CSV from a new API route, with corresponding UI changes to expose the download action.

Changes:

Added a new API endpoint to stream a generated “enriched CSV” for CSV data sources.
Updated the enrichment dashboard UI to show a “Download enriched CSV” action for CSV sources instead of running an enrichment job.
Enabled enrichment feature flag for CSV data sources and added csv-stringify dependency.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/server/repositories/DataRecord.ts	Adds a streaming helper that orders records by numeric `external_id`.
src/features.ts	Enables enrichment feature flag for CSV data sources.
src/app/api/data-sources/[id]/enriched-csv/route.ts	New streaming download endpoint that enriches each record and emits CSV.
src/app/(private)/data-sources/[id]/DataSourceEnrichmentDashboard.tsx	UI changes to download enriched CSV for CSV sources; disables enrichment job flow.
package.json	Adds `csv-stringify` dependency.
package-lock.json	Locks `csv-stringify` dependency version.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

src/app/(private)/data-sources/[id]/DataSourceEnrichmentDashboard.tsx

+            <Button asChild disabled={dataSource.enrichments.length === 0}>
+              <a
+                href={`/api/data-sources/${dataSource.id}/enriched-csv`}
+                download
+              >
+                <Download />
+                Download enriched CSV
+              </a>
+            </Button>


src/app/api/data-sources/[id]/enriched-csv/route.ts

+          // Write header row on the first record
+          if (!headerWritten) {
+            const originalColumns = Object.keys(record.json);
+            allColumns = [...originalColumns, ...enrichmentColNames];
+            const headerCsv = stringify([allColumns]);
+            controller.enqueue(encoder.encode(headerCsv));
+            headerWritten = true;
+          }


src/app/api/data-sources/[id]/enriched-csv/route.ts

+        // Handle empty data source
+        if (!headerWritten) {
+          const enrichmentHeaders = enrichmentColNames;
+          const headerCsv = stringify([enrichmentHeaders]);
+          controller.enqueue(encoder.encode(headerCsv));
+        }


Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

…d into feat/enrich-csv

…ame includes the Mapped: prefix

Copilot

Pull request overview

Enables enrichment for CSV-backed data sources by switching to an on-demand “enriched CSV download” flow, while also standardizing enrichment column identification as an external-facing name (externalName) across adaptors/jobs.

Changes:

Standardize enrichment column defs to use def.externalName throughout server jobs, repositories, adaptors, and unit tests.
Enable enrichment for CSV data sources and add a new API route to stream an “enriched CSV” download.
Update the enrichment dashboard UX to show a download button for CSV sources (and disable background-enrichment UI).

Reviewed changes

Copilot reviewed 18 out of 19 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/unit/server/adaptors/mailchimp.test.ts	Update tests to use `def.externalName` for enriched column defs.
tests/unit/server/adaptors/googlesheets.test.ts	Update tests to use `def.externalName`.
tests/unit/server/adaptors/airtable.test.ts	Update tests to use `def.externalName`.
tests/unit/server/adaptors/actionnetwork.test.ts	Update tests to use `def.externalName` (including mapped prefix).
src/server/trpc/routers/dataSource.ts	Skip background cleanup job when deleting enrichment columns on CSV data sources.
src/server/repositories/DataRecord.ts	Add ordered streaming helper and write enrichment values keyed by `externalName`.
src/server/models/DataRecord.ts	Change `EnrichedRecord` column def shape to `{ externalName, type }`.
src/server/mapping/enrich.ts	Emit enriched column defs using `externalName` and adjust types for internal helpers.
src/server/jobs/enrichDataSource.ts	Collect enriched column defs keyed by `externalName`.
src/server/jobs/enrichDataRecords.ts	Collect enriched column defs keyed by `externalName`.
src/server/adaptors/mailchimp.ts	Use `externalName` when mapping merge fields/tags.
src/server/adaptors/googlesheets.ts	Use `externalName` for header detection and column indexing.
src/server/adaptors/airtable.ts	Use `externalName` for field creation and record updates.
src/server/adaptors/actionnetwork.ts	Use `externalName` for ActionNetwork custom field updates.
src/features.ts	Turn on `enrichment` for CSV data sources.
src/app/api/data-sources/[id]/enriched-csv/route.ts	New endpoint to stream an enriched CSV download.
src/app/(private)/data-sources/[id]/DataSourceEnrichmentDashboard.tsx	CSV-specific UI: download enriched CSV instead of running enrichment jobs.
package.json	Add `csv-stringify`.
package-lock.json	Lockfile update for `csv-stringify`.

Comments suppressed due to low confidence (1)

src/server/trpc/routers/dataSource.ts:500

This skips enqueueing removeEnrichmentColumns for CSV when deleting enrichment columns via this mutation, but updateConfig still enqueues removeEnrichmentColumns unconditionally when enrichments are removed (see earlier in this file). With CSV enrichment now enabled, that means removing enrichments via config update will still run the background cleanup job (and will attempt CSVAdaptor.deleteColumn, which always throws). Consider applying the same CSV guard to the updateConfig enrichment-removal path for consistent behavior.

      // Enqueue background job for expensive cleanup (external source + record JSON)
      // CSV data sources don't need background cleanup — just removing the config is enough
      if (
        input.externalColumnNames.length > 0 &&
        ctx.dataSource.config.type !== DataSourceType.CSV
      ) {
        await enqueue("removeEnrichmentColumns", ctx.dataSource.id, {
          dataSourceId: ctx.dataSource.id,
          externalColumnNames: input.externalColumnNames,
        });

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

src/app/api/data-sources/[id]/enriched-csv/route.ts

src/server/repositories/DataRecord.ts

src/app/(private)/data-sources/[id]/DataSourceEnrichmentDashboard.tsx

src/app/api/data-sources/[id]/enriched-csv/route.ts

+        while (row.value) {
+          const record: DataRecord = row.value;
+          const enrichedRecord = await enrichRecord(
+            { externalId: record.externalId, json: record.json },


Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

…d into feat/enrich-csv

Copilot

Pull request overview

Adds support for CSV data sources to use enrichments via an on-demand “enriched CSV” download, and migrates enrichment column identification from def.name to def.externalName across jobs/adaptors/tests.

Changes:

Switch EnrichedRecord.columns[].def to { externalName, type } and propagate through adaptors/jobs/repository writes.
Enable enrichment for DataSourceType.CSV and update the enrichment UI to offer an “Download enriched CSV” path for CSV sources.
Add a streaming API route that generates an enriched CSV export for CSV data sources.

Reviewed changes

Copilot reviewed 18 out of 19 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/unit/server/adaptors/mailchimp.test.ts	Updates enrichment column def usage to `externalName` in tests.
tests/unit/server/adaptors/googlesheets.test.ts	Updates enrichment column def usage to `externalName` in tests.
tests/unit/server/adaptors/airtable.test.ts	Updates enrichment column def usage to `externalName` in tests.
tests/unit/server/adaptors/actionnetwork.test.ts	Updates enrichment column def usage to `externalName` (but assertion likely needs update).
src/server/trpc/routers/dataSource.ts	Skips background enrichment-column cleanup job for CSV data sources.
src/server/repositories/DataRecord.ts	Adds ordered streaming helper; writes enrichment into JSON keyed by `externalName`.
src/server/models/DataRecord.ts	Redefines `EnrichedRecord` column def to `{ externalName, type }`.
src/server/mapping/enrich.ts	Uses `externalName` for enrichment output; adjusts internal types for `getEnrichedColumn`.
src/server/jobs/enrichDataSource.ts	Collects enriched column defs keyed by `externalName` and persists `name=externalName`.
src/server/jobs/enrichDataRecords.ts	Collects enriched column defs keyed by `externalName` and persists `name=externalName`.
src/server/adaptors/mailchimp.ts	Uses `def.externalName` when mapping merge fields/tags.
src/server/adaptors/googlesheets.ts	Uses `def.externalName` for header detection and updates.
src/server/adaptors/airtable.ts	Uses `def.externalName` when creating fields and updating records.
src/server/adaptors/actionnetwork.ts	Uses `def.externalName` when building `custom_fields` payload.
src/features.ts	Enables `enrichment` feature for CSV data sources.
src/app/api/data-sources/[id]/enriched-csv/route.ts	New streaming endpoint to download enriched CSV for CSV data sources.
src/app/(private)/data-sources/[id]/DataSourceEnrichmentDashboard.tsx	Adjusts UI behavior for CSV: download button + disables job status subscription.
package.json	Adds `csv-stringify` dependency.
package-lock.json	Locks `csv-stringify` dependency.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

tests/unit/server/adaptors/actionnetwork.test.ts

src/app/api/data-sources/[id]/enriched-csv/route.ts

+          // Write data row using column order from header
+          const values = allColumns.map((col) => {
+            const val = rowData[col];
+            if (val === null || val === undefined) return "";
+            if (typeof val === "object") return JSON.stringify(val);
+            return String(val);


src/app/api/data-sources/[id]/enriched-csv/route.ts

+          const enrichedValues: Record<string, unknown> = {};
+          for (const enrichment of dataSource.enrichments) {
+            const colName = enrichmentColumnName(enrichment.name);
+            const col = record.geocodeResult
+              ? await getEnrichedColumn(
+                  { externalId: record.externalId, json: record.json },
+                  record.geocodeResult,
+                  enrichment,
+                )
+              : null;
+            enrichedValues[colName] = col?.value ?? null;
+          }


src/server/repositories/DataRecord.ts

+  return db
+    .selectFrom("dataRecord")
+    .where("dataSourceId", "=", dataSourceId)
+    .selectAll()


src/server/trpc/routers/dataSource.ts

+      // CSV data sources don't need background cleanup — just removing the config is enough
+      if (
+        input.externalColumnNames.length > 0 &&
+        ctx.dataSource.config.type !== DataSourceType.CSV
+      ) {


Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

feat: enrich csv and download

e10e390

joaquimds requested a review from Copilot March 14, 2026 21:01

Copilot started reviewing on behalf of joaquimds March 14, 2026 21:01 View session

Copilot AI reviewed Mar 14, 2026

View reviewed changes

joaquimds and others added 4 commits March 14, 2026 22:08

Potential fix for pull request finding

a573b86

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

fix: small improvements to CSV enrichment

c96ec3a

Merge branch 'feat/enrich-csv' of github.com:commonknowledge/ts-mappe…

9296d14

…d into feat/enrich-csv

feat: refactor EnrichedRecord type to make it clear that the column n…

ceec4d7

…ame includes the Mapped: prefix

joaquimds requested a review from Copilot March 14, 2026 21:39

Copilot started reviewing on behalf of joaquimds March 14, 2026 21:39 View session

Copilot AI reviewed Mar 14, 2026

View reviewed changes

joaquimds and others added 3 commits March 14, 2026 22:52

Potential fix for pull request finding

7c20189

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

fix: improve CSV enrichment performance

0e31894

Merge branch 'feat/enrich-csv' of github.com:commonknowledge/ts-mappe…

c9d7cbf

…d into feat/enrich-csv

joaquimds requested a review from Copilot March 14, 2026 22:09

Copilot started reviewing on behalf of joaquimds March 14, 2026 22:10 View session

Copilot AI reviewed Mar 14, 2026

View reviewed changes

Potential fix for pull request finding

09af599

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

joaquimds merged commit ece64db into main Mar 14, 2026
1 check passed

joaquimds deleted the feat/enrich-csv branch March 14, 2026 22:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: enrich csv and download#372

feat: enrich csv and download#372
joaquimds merged 9 commits intomainfrom
feat/enrich-csv

joaquimds commented Mar 14, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

joaquimds commented Mar 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants