WIP: Add spatial-temporal metadata embeddings to CropModel#1334
Open
WIP: Add spatial-temporal metadata embeddings to CropModel#1334
Conversation
Species distributions vary by location and season. This adds an optional late-fusion pathway that encodes (lat, lon, date) as a small sinusoidal embedding (~1.5% of image features) concatenated with ResNet-50 features before the classifier. Metadata is fully optional — the model gracefully degrades to image-only predictions when metadata is absent. Key changes: - SpatialTemporalEncoder with sinusoidal encoding + small MLP - CropModel.create_model splits into backbone + classifier when use_metadata=True - MetadataImageFolder wraps ImageFolder with CSV sidecar for training - BoundingBoxDataset supports per-crop metadata for inference - predict_tile accepts metadata dict (lat, lon, date) - Config: use_metadata, metadata_dim, metadata_dropout - 19 new tests covering encoder, model, datasets, and integration - Documentation for CropModel docs and configuration reference Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Collaborator
Author
|
I changed something small, I don't agree with bugbot. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Key changes
SpatialTemporalEncoder: Sinusoidal positional encoding + MLP (6 → configurable dim, default 32)CropModel: Splits into backbone + metadata encoder + classifier whenuse_metadata=TrueMetadataImageFolder: Wraps ImageFolder with a CSV sidecar (filename,lat,lon,date) for trainingBoundingBoxDataset: Supports per-crop metadata dict for inferencepredict_tile: Acceptsmetadata={"lat": ..., "lon": ..., "date": "YYYY-MM-DD"}use_metadata,metadata_dim,metadata_dropoutTest plan
pytest tests/test_metadata_cropmodel.py)pytest tests/test_crop_model.py)🤖 Generated with Claude Code
Note
Medium Risk
Touches core CropModel training/inference data flow and changes batch/forward signatures to support metadata; defaults remain unchanged, but incorrect metadata wiring or shape handling could break crop-model prediction/training in edge cases.
Overview
Adds optional spatial-temporal metadata (lat/lon/date) support to
CropModel, including a newSpatialTemporalEncoderand a backbone+metadata+classifier path whencropmodel.use_metadatais enabled, while keeping image-only behavior as the default.Training can now load per-image metadata from a sidecar CSV via
CropModel.load_from_disk(..., metadata_csv=...), and inference can pass image-level metadata throughdeepforest.predict_tile(..., metadata=...)which is expanded to per-crop tensors and threaded throughpredict/BoundingBoxDataset.Updates config/schema/docs to expose
use_metadata,metadata_dim, andmetadata_dropout, and adds a dedicated test suite covering encoder behavior, training/predict steps, dataset wrappers, and checkpoint save/load.Written by Cursor Bugbot for commit c763c17. This will update automatically on new commits. Configure here.