Use database cache for mg-context#1611
Merged
PaulAdamDavis merged 13 commits intomainfrom Mar 29, 2026
Merged
Conversation
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
PaulAdamDavis
force-pushed
the
dev-mg-context-db
branch
from
9852740 to
6fa9628
Compare
Contributor
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 2 minutes and 16 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (9)
WalkthroughThis PR adds SQLite-backed persistence to mg-context: a new database module, prepared statements, caches, transactional helpers, and DB lifecycle methods ( Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes 🚥 Pre-merge checks | ✅ 1 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (1 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 15
🧹 Nitpick comments (1)
packages/mg-context/src/test/tasks.test.ts (1)
50-63: Avoid double-saving insideforEachPost().
MigrateContext.forEachPost()inpackages/mg-context/src/lib/MigrateContext.ts:178-199already callspost.save(this.db)afterawait callback(post). Saving again here adds an extra write cycle and can become visible ifsave()ever stops being fully idempotent for relations.💡 Suggested simplification
await ctx.MGContext.forEachPost(async (post: any) => { post.addTag({ name: 'My Tag', slug: 'my-tag' }); post.addAuthor({ name: 'Jane', slug: 'jane', email: 'jane@example.com' }); - - await post.save(ctx.MGContext.db); });🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/mg-context/src/test/tasks.test.ts` around lines 50 - 63, The test is double-saving posts: remove the redundant await post.save(ctx.MGContext.db) in packages/mg-context/src/test/tasks.test.ts because MigrateContext.forEachPost() already calls post.save(this.db) after awaiting the callback; instead, keep all mutations inside the provided callback (the post.addTag/addAuthor calls) and rely on forEachPost to persist them, ensuring you do not introduce an extra write by calling post.save again.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/mg-context/README.md`:
- Around line 187-199: The README currently says "Export all tags/users" but
writeGhostTagsJson() and writeGhostUsersJson() only export referenced (used)
entities; update the documentation text and examples to clarify that these
functions export only tags/users that are referenced by posts (i.e., "used"
queries), not orphan/standalone tags or authors. Mention the filtering behavior
explicitly next to the examples and, if helpful, add a short note pointing to
the "used" query behavior or function names (writeGhostTagsJson,
writeGhostUsersJson) so readers know why some tags/users may be omitted.
- Around line 74-78: Update the README paragraph to state that setting HTML is
lazy: calling post.set('html', ...) only invalidates cached lexical/mobiledoc
and defers regeneration until conversion is required (e.g., when
convertContent(), getFinal(), or writeGhostJson() are invoked), rather than
performing immediate/eager conversion on set()/save(); mention that the default
contentFormat remains 'lexical' but conversion happens lazily on those explicit
operations.
In `@packages/mg-context/scripts/foreach-ghost-post.ts`:
- Line 5: The hardcoded machine-local DB path in the MigrateContext constructor
(const ctx = new MigrateContext(...)) makes the script non-portable; change it
to accept a configurable path (e.g., read from an environment variable like
process.env.DB_PATH or a CLI argument) and fall back to a repo-relative default
using path.join(process.cwd(), 'packages/mg-context/scripts/yolo.db') or
similar. Update the script’s argument parsing (or add a small env check) so
MigrateContext is instantiated with the resolved dbPath variable instead of the
literal '/Users/paul/...'.
- Around line 4-25: The code currently calls ctx.init() then awaits
ctx.forEachGhostPost(...) and finally ctx.close(), but if forEachGhostPost
throws the ctx.close() call is skipped; modify main so that after creating and
awaiting ctx.init() you wrap the work (the forEachGhostPost call and any
post-processing) in a try/finally block and move ctx.close() into the finally to
guarantee the MigrateContext.close() is always invoked; keep references to
MigrateContext, main, ctx.init(), ctx.forEachGhostPost, and ctx.close() to
locate where to apply the try/finally.
- Around line 10-18: The callback passed to ctx.forEachGhostPost currently
triggers lint errors: add a suppression for the unused parameter by inserting
"void post;" at the top of the async callback body, change the sleep Promise
from an implicit-return executor to a block executor (e.g., new Promise(r => {
setTimeout(r, 250); });) to avoid no-promise-executor-return, and add "//
eslint-disable-next-line no-console" immediately above each console.log to
silence no-console errors; these edits should be applied inside the async (json,
post) => { ... } callback used with ctx.forEachGhostPost.
In `@packages/mg-context/scripts/generate-large-export.ts`:
- Around line 55-56: The hard-coded absolute DB path passed to the
MigrateContext constructor makes the script non-portable; change the code that
creates new MigrateContext(...) (and the const ctx variable) to derive dbPath
from either a command-line flag or environment variable with a sensible
repo-relative default (e.g., path.join(__dirname, '..', 'scripts', 'yolo.db'))
so that if no flag/env is provided the script uses the repo-relative DB,
otherwise it uses the provided path; ensure the rest of the flow (ctx.init())
continues to use that dbPath variable.
- Around line 16-19: The script uses direct console logging inside the
process.on('exit', ...) handler and other spots (references: process.on('exit',
_maxMemoryConsumption, _dtOfMaxMemoryConsumption, and the other console
occurrences around lines noted) which violates eslint-plugin-ghost; either
replace these console.* calls with the approved logger used in this repo or, if
CLI/intentional, add targeted comments like "// eslint-disable-next-line
no-console" immediately above each console invocation (also apply the same
change to the other console uses you flagged at ranges 58-64, 95-99, 112-115,
119-143) so lint passes while keeping the intent and location of the logs
unchanged.
- Around line 47-139: The main function currently initializes a MigrateContext
(ctx) but only calls ctx.close() on the happy path, so exceptions leave the DB
open; wrap the lifecycle so ctx.close() always runs (use a try/finally around
the body after await ctx.init() or ensure ctx.close() is invoked in a finally
block), keeping the existing await ctx.init(), all loops/transactions (the for
loop with ctx.transaction, ctx.forEachPost, and the ctx.writeGhostJson call)
inside the try and calling ctx.close() in finally; reference MigrateContext,
main, ctx.init(), ctx.transaction, ctx.forEachPost, ctx.writeGhostJson and
ctx.close() when applying the change.
In `@packages/mg-context/src/lib/AuthorContext.ts`:
- Around line 53-60: The save() method currently returns early on an
author-cache hit which prevents the later upsert/update logic from running;
modify the slug-cache branch in save() (where db.authorCache is used) to still
set this.dbId and this.ghostId from the cached entry but do NOT return early —
let execution fall through so updateAuthorById (or the existing upsert path)
runs when this.dbId exists; in short: keep the dedupe cache lookup to populate
ids but remove the early return so save() performs the upsert/update
(updateAuthorById) to persist richer name/email/metadata when present.
In `@packages/mg-context/src/lib/database.ts`:
- Around line 139-152: The SELECTs for findUsedTags and findUsedAuthors lack
deterministic ordering, causing nondeterministic exports; update the prepared
statements for findUsedTags and findUsedAuthors to append an ORDER BY clause
(e.g., ORDER BY slug ASC or ORDER BY id ASC) so rows are returned in a stable,
deterministic order—locate the findUsedTags and findUsedAuthors prepared
statements in database.ts and modify their SQL to include the chosen ORDER BY.
In `@packages/mg-context/src/lib/db-helpers.ts`:
- Around line 4-17: withTransaction currently issues BEGIN/COMMIT/ROLLBACK
unconditionally causing nested BEGIN failures in SQLite; modify withTransaction
(in packages/mg-context/src/lib/db-helpers.ts) to detect db.inTransaction and
only call db.db.exec('BEGIN') / 'COMMIT' / 'ROLLBACK' when not already in a
transaction, and only toggle db.inTransaction for the outermost invocation
(i.e., set inTransaction = true before BEGIN when entering outer transaction and
restore it to false on outer completion in finally), while still running the
callback and propagating errors as before so nested callers (e.g.,
MigrateContext code that marks an outer transaction) reuse the existing
transaction.
In `@packages/mg-context/src/lib/MigrateContext.ts`:
- Around line 204-210: When handling a prebuilt PostContext instance in
MigrateContext.addPost (the branch where post instanceof PostContext), the code
only copies warnOnLookupKeyDuplicate and omits applying this.#contentFormat,
allowing the post to retain a different contentFormat; update that branch to
assign the enclosing context's content format to the instance (e.g., set
newPost.contentFormat = this.#contentFormat) before calling
newPost.save(this.db) so PostContext instances are consistent with the other
branch that constructs a new PostContext with contentFormat.
- Around line 90-93: The init() method currently overwrites this.#db by calling
createDatabase(storage) without handling prior connections; update init() to
guard against repeated calls by checking if this.#db already exists and either
return early (no-op) or throw to fail fast, or if reinitialization is required,
call this.#db.close() (or equivalent DatabaseSync close method) before assigning
a new connection; reference init(), this.#db, createDatabase(), DatabaseSync,
this.#ephemeral and this.#dbPath when making the change.
In `@packages/mg-context/src/lib/PostContext.ts`:
- Around line 313-394: Wrap the whole PostContext.save(db) body in a single DB
transaction: call BEGIN (or use a db.transaction helper) at the start of save
and COMMIT at the end, and ensure any early returns (e.g. duplicate-skip after
findPostByLookupKey) still COMMIT/ROLLBACK appropriately; on exception ROLLBACK
and rethrow. Specifically, start a transaction before calling
db.stmts.updatePost.run / db.stmts.insertPost.run and before the subsequent
deletePostTagsByPostId/insertPostTag and
deletePostAuthorsByPostId/insertPostAuthor calls, commit after all these
statements succeed, and rollback if any statement fails so partial writes cannot
persist. Use the existing db object (the same one passed to save) and target the
statements insertPost.run, updatePost.run, deletePostTagsByPostId.run,
insertPostTag.run, deletePostAuthorsByPostId.run, insertPostAuthor.run and the
findPostByLookupKey.get path when implementing the transaction boundaries.
In `@packages/mg-context/src/test/migrate-context.test.ts`:
- Around line 948-969: The test currently stubs console.warn with warnMock but
restores it only at the end, risking leaked stubs if an assertion throws; wrap
the test body so warnMock.mock.restore() always runs (use a try/finally) and
ensure instance.close() is also called in the finally block; apply the same
change to the similar test around the other block (lines noted in review) so
both tests always restore the console stub and close the MigrateContext instance
even on failure.
---
Nitpick comments:
In `@packages/mg-context/src/test/tasks.test.ts`:
- Around line 50-63: The test is double-saving posts: remove the redundant await
post.save(ctx.MGContext.db) in packages/mg-context/src/test/tasks.test.ts
because MigrateContext.forEachPost() already calls post.save(this.db) after
awaiting the callback; instead, keep all mutations inside the provided callback
(the post.addTag/addAuthor calls) and rely on forEachPost to persist them,
ensuring you do not introduce an extra write by calling post.save again.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: b815c2cb-5e0e-4e4b-a133-2dc9059ea816
📒 Files selected for processing (20)
.gitignorepackages/mg-context/README.mdpackages/mg-context/package.jsonpackages/mg-context/scripts/foreach-ghost-post.tspackages/mg-context/scripts/generate-large-export.tspackages/mg-context/src/index.tspackages/mg-context/src/lib/AuthorContext.tspackages/mg-context/src/lib/MigrateBase.tspackages/mg-context/src/lib/MigrateContext.tspackages/mg-context/src/lib/PostContext.tspackages/mg-context/src/lib/TagContext.tspackages/mg-context/src/lib/database.tspackages/mg-context/src/lib/db-helpers.tspackages/mg-context/src/test/migrate-context.test.tspackages/mg-context/src/test/post-context.test.tspackages/mg-context/src/test/tasks.test.tspackages/mg-json/lib/to-ghost-json/index.jspackages/mg-json/lib/to-ghost-json/meta-to-ghost.jspackages/mg-json/lib/to-ghost-json/process-post-relations.jspackages/mg-json/test/to-ghost-json.test.js
✅ Files skipped from review due to trivial changes (1)
- .gitignore
🚧 Files skipped from review as they are similar to previous changes (5)
- packages/mg-json/lib/to-ghost-json/meta-to-ghost.js
- packages/mg-context/package.json
- packages/mg-context/src/lib/MigrateBase.ts
- packages/mg-json/lib/to-ghost-json/index.js
- packages/mg-context/src/lib/TagContext.ts
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
♻️ Duplicate comments (2)
packages/mg-context/src/lib/MigrateContext.ts (1)
207-210:⚠️ Potential issue | 🟠 MajorKeep
addPost(postInstance)aligned with the other branches.This branch saves the passed
PostContextwithout normalizing itscontentFormat, while the object/no-arg branches always forcethis.#contentFormat. A prebuiltPostContextcan therefore persist/export in a different format than the enclosingMigrateContext.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/mg-context/src/lib/MigrateContext.ts` around lines 207 - 210, The addPost branch that handles an existing PostContext (in MigrateContext) must normalize its contentFormat before persisting; update the branch that detects post instanceof PostContext to set the PostContext's contentFormat to this.#contentFormat (and preserve warnOnLookupKeyDuplicate) prior to calling save(this.db) so it matches the object/no-arg branches' behavior and avoids persisting a mismatched contentFormat.packages/mg-context/src/lib/database.ts (1)
77-97:⚠️ Potential issue | 🟠 MajorMake tag/author slugs unique at the DB layer.
TagContext.save()andAuthorContext.save()both treatslugas the stable lookup key, butfindTagBySlug/findAuthorBySlughere read a single row from tables that only have non-unique indexes. Once duplicate slugs exist, saves and lookups become nondeterministic and can update the wrong record. Add unique constraints onTags.slugandAuthors.slug(nullable-safe if needed).Also applies to: 134-152
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/mg-context/src/lib/database.ts` around lines 77 - 97, Add a uniqueness constraint on the slug columns so TagContext.save/AuthorContext.save and the lookup helpers findTagBySlug/findAuthorBySlug are deterministic: modify the table DDL for Tags and Authors to make slug unique (e.g., specify slug TEXT UNIQUE or add a UNIQUE index like CREATE UNIQUE INDEX IF NOT EXISTS ux_tags_slug ON Tags(slug)), and do the same for Authors (ux_authors_slug); ensure the change is applied both where Tags/Authors are created (current block and the similar block at lines ~134-152) so nullable slugs remain allowed but duplicate non-null slugs are rejected.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/mg-context/src/lib/database.ts`:
- Around line 168-180: createDatabase currently constructs DatabaseSync(dbPath)
then runs PRAGMAs, executes SCHEMA and calls prepareStatements, but if any of
those steps throw the SQLite handle (db) is leaked; modify createDatabase to
wrap the initialization steps (PRAGMA calls, db.exec(SCHEMA),
prepareStatements(db)) in a try/catch, and in the catch call db.close() (or the
appropriate close on DatabaseSync) before rethrowing the error so the
DatabaseSync handle is always closed on failure; reference createDatabase,
DatabaseSync, SCHEMA, prepareStatements, and the returned DatabaseModels when
applying the change.
In `@packages/mg-context/src/lib/MigrateContext.ts`:
- Around line 181-200: forEachPost currently paginates the mutable filtered
query and then saves each post inside the loop, which allows the filter to
change between batches and causes skipped/duplicated processing; fix by
snapshotting a stable list of matching post IDs up front (use `#buildFilterWhere`
to build the filter and run a single query to collect all matching post IDs) and
then iterate those IDs in batches: for each id call find post by id (or
findPostsWhere filtered to id list), construct PostContext.fromRow, run the
callback, and save with withTransaction/post.save; ensure you no longer re-run
the original mutable filtered query between batches so changes to
tags/authors/created_at/published_at cannot shift pagination.
- Around line 105-117: The transaction method should short-circuit when a
transaction is already open: if this.db.inTransaction is true, simply await and
return callback() without calling BEGIN/COMMIT/ROLLBACK or changing the
inTransaction flag; otherwise run the existing BEGIN, set inTransaction=true,
await callback, COMMIT on success, ROLLBACK on error, and reset
inTransaction=false in finally. Update the transaction method (function name
transaction in MigrateContext / this.db.inTransaction usage) to branch on
existing inTransaction before executing any SQL or toggling the flag.
---
Duplicate comments:
In `@packages/mg-context/src/lib/database.ts`:
- Around line 77-97: Add a uniqueness constraint on the slug columns so
TagContext.save/AuthorContext.save and the lookup helpers
findTagBySlug/findAuthorBySlug are deterministic: modify the table DDL for Tags
and Authors to make slug unique (e.g., specify slug TEXT UNIQUE or add a UNIQUE
index like CREATE UNIQUE INDEX IF NOT EXISTS ux_tags_slug ON Tags(slug)), and do
the same for Authors (ux_authors_slug); ensure the change is applied both where
Tags/Authors are created (current block and the similar block at lines ~134-152)
so nullable slugs remain allowed but duplicate non-null slugs are rejected.
In `@packages/mg-context/src/lib/MigrateContext.ts`:
- Around line 207-210: The addPost branch that handles an existing PostContext
(in MigrateContext) must normalize its contentFormat before persisting; update
the branch that detects post instanceof PostContext to set the PostContext's
contentFormat to this.#contentFormat (and preserve warnOnLookupKeyDuplicate)
prior to calling save(this.db) so it matches the object/no-arg branches'
behavior and avoids persisting a mismatched contentFormat.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 570a2395-1d95-4e6d-96e6-6684f4bd0e48
📒 Files selected for processing (6)
packages/mg-context/README.mdpackages/mg-context/scripts/foreach-ghost-post.tspackages/mg-context/scripts/generate-large-export.tspackages/mg-context/src/lib/MigrateContext.tspackages/mg-context/src/lib/database.tspackages/mg-context/src/lib/db-helpers.ts
🚧 Files skipped from review as they are similar to previous changes (2)
- packages/mg-context/README.md
- packages/mg-context/src/lib/db-helpers.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP