Skip to content

Fix abi3 dataloader lifecycle handling#6

Merged
bobcao3 merged 7 commits intomainfrom
limited-abi-remediation
Mar 26, 2026
Merged

Fix abi3 dataloader lifecycle handling#6
bobcao3 merged 7 commits intomainfrom
limited-abi-remediation

Conversation

@bobcao3
Copy link
Collaborator

@bobcao3 bobcao3 commented Mar 24, 2026

Summary

  • replace direct Python object-header access with module-owned heap-type state and limited-ABI-safe allocation and teardown in the abi3 bindings
  • harden the native dataloader and Lua bridge around large-state initialization, file-handle transport, and error cleanup so constructor and teardown paths do not crash or leak
  • add subprocess lifecycle regressions plus document the reviewed remaining abi3 surface and deferred follow-up work

Test Plan

  • zig build test -Doptimize=Debug
  • zig build test
  • zig build python-bindings -Doptimize=ReleaseSafe
  • PYTHONPATH=python/src uv run --with pytest python -m pytest python/tests/test_limited_abi.py -q
  • PYTHONPATH=python/src uv run --with pytest python -m pytest python/tests/test_dataloader.py -q
  • uv run --with build --with setuptools --with wheel python -m build --wheel --no-isolation python/

bobcao3 and others added 7 commits March 24, 2026 16:44
Move FileHandle padding to high bits so idx/generation/checksum fit
within f64 exact integer range (48 bits), preventing silent handle
corruption when round-tripping through LuaJIT numbers. Replace
non-atomic worker_thread null-signaling with atomic worker_done flag
to eliminate data race between worker and join threads. Replace
std.debug.assert in open_file response handler with explicit error
return that survives ReleaseFast.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
errdefer was dead code — moduleExec returns c_int, not an error union,
so the deferred cleanup never fired. Replace with explicit moduleClear
calls on each failure path after data_loader_type is allocated.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@bobcao3 bobcao3 merged commit b6e8fc6 into main Mar 26, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant