[WIP]: Improve predict_tile memory use and windowed raster inference #1304
[WIP]: Improve predict_tile memory use and windowed raster inference #1304henrykironde wants to merge 7 commits intoweecology:mainfrom
Conversation
- Add helper to convert inputs to RGB CHW float32 in [0, 1] - Validate ndim/channel placement and reject grayscale early - Normalize based on dtype (uint8 -> /255), not max/min heuristics - Ensure contiguous arrays before torch conversion - Refactor SingleImage to keep full image in CHW without forcing full float32 copy; convert/normalize per-window crops in get_crop - Improve TiledRaster/window strategy: reuse single rasterio handle, warn when untiled, close datasets promptly after predict - Simplify predict_step return
- Warn when CUDA is available but trainer falls back to CPU - Add test_gpu_inference_uses_cuda: regression test for GPU inference - Add hpc_multi_gpu_train: DDP smoke test for multi-GPU training - Relax requires-python to allow 3.13
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1304 +/- ##
==========================================
- Coverage 87.16% 86.71% -0.45%
==========================================
Files 21 21
Lines 2851 2914 +63
==========================================
+ Hits 2485 2527 +42
- Misses 366 387 +21
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Copy only when chw may alias the input (np.shares_memory), then use in-place division. Preserves caller's data without memory spike from regular division creating temporary arrays. Co-authored-by: Cursor <cursoragent@cursor.com>
henrykironde
changed the title
henrykironde
force-pushed
the
fix-gpus
branch
from
ca0449a to
525ef03
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.
… crops Apply normalization to the entire image once before prediction. Co-authored-by: Cursor <cursoragent@cursor.com>
henrykironde
changed the title
|
@henrykironde how big of an image are you testing with here? I'll run some tests with memray on some NEON tiles if that'd help. |
|
@jveitchmichaelis I am using the orthos we created for Everglades. You can try with unprojected files in i.e |
henrykironde
changed the title
|
I need to test some edge case. |
3 participants
Description
Related Issue(s)
AI-Assisted Development
AI tools used (if applicable):
Note
Medium Risk
Touches core inference data loading/normalization and raster IO lifecycle, which can affect prediction correctness and performance on large tiles. Added guards/tests reduce risk but edge cases around raster windowing and multiprocessing still warrant review.
Overview
Improves
predict_tile/prediction datasets by making image preprocessing stricter and more consistent (shared RGB/CHW + dtype/range normalization) and preventing crop views from being mutated in-place (SingleImage.get_crop()now clones).Hardens windowed raster inference (
TiledRaster) by keeping a single raster handle open with explicitclose()/pickle-safe reopen, warning (not failing) on non-tiled rasters, filtering/guarding against out-of-bounds/zero-size windows, and ensuring the handle is always closed via atry/finallyinpredict_tile.Adds a warning when CUDA is available/requested but the Lightning
Trainerisn’t actually using a GPU accelerator, plus new GPU/HPC-only regression tests for CUDA usage and multi-GPU DDP training; also bumps supported Python to<3.14.Written by Cursor Bugbot for commit d847701. This will update automatically on new commits. Configure here.