Skip to content

Fixes for forkserver/spawn serialization and fix for LMDB upgrade issues#148

Open
christinaflo wants to merge 4 commits intomainfrom
feature/multiprocessing-lmdb-refactor
Open

Fixes for forkserver/spawn serialization and fix for LMDB upgrade issues#148
christinaflo wants to merge 4 commits intomainfrom
feature/multiprocessing-lmdb-refactor

Conversation

@christinaflo
Copy link
Copy Markdown
Collaborator

Summary
LMDB refactor to allow for forkserver/spawn serialization and resolve issues that required pinning to LMDB==1.6.2 for training.

Changes

  1. Exposes more dataloader options like persistent workers, prefetch factor, multiprocessing context
  2. Lazy load CCD obj to prevent issues with serialization
  3. Modify LMDB handling to allow for serialization. Close parent db connection prior to forking.

Related Issues
PR #143 defaults to forkserver. This PR adds additional fixes needed for training with forkserver.

…ng methods. LMDB refactor to allow for forkserver/spawn serialization + resolve issues that required pinning lmdb for training.
…ure connection is closed when dataset init finishes prior to forking
@christinaflo christinaflo requested review from jandom and jnwei March 26, 2026 02:32
@christinaflo christinaflo self-assigned this Mar 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants