Skip to content

WIP: Add spatial-temporal metadata embeddings to CropModel#1334

Open
bw4sz wants to merge 7 commits intomainfrom
claude/friendly-beaver
Open

WIP: Add spatial-temporal metadata embeddings to CropModel#1334
bw4sz wants to merge 7 commits intomainfrom
claude/friendly-beaver

Conversation

@bw4sz
Copy link
Collaborator

@bw4sz bw4sz commented Mar 5, 2026

Summary

  • Adds optional late-fusion spatial-temporal metadata (lat, lon, date) to CropModel for improved species classification
  • Sinusoidal encoding of coordinates + day-of-year through a small MLP (~1.5% of image feature vector), concatenated with ResNet-50 features before the classifier
  • Fully backward compatible — metadata is optional at both training and inference time; model gracefully degrades to image-only predictions when metadata is absent

Key changes

  • SpatialTemporalEncoder: Sinusoidal positional encoding + MLP (6 → configurable dim, default 32)
  • CropModel: Splits into backbone + metadata encoder + classifier when use_metadata=True
  • MetadataImageFolder: Wraps ImageFolder with a CSV sidecar (filename,lat,lon,date) for training
  • BoundingBoxDataset: Supports per-crop metadata dict for inference
  • predict_tile: Accepts metadata={"lat": ..., "lon": ..., "date": "YYYY-MM-DD"}
  • Config: use_metadata, metadata_dim, metadata_dropout
  • 19 new tests covering encoder, model, datasets, checkpoint save/load, and full training integration
  • Documentation added to CropModel user guide and configuration reference

Test plan

  • All 19 new metadata tests pass (pytest tests/test_metadata_cropmodel.py)
  • All 8 existing crop model tests pass (pytest tests/test_crop_model.py)
  • Full test suite passes on CI
  • Manual smoke test with real field data

🤖 Generated with Claude Code


Note

Medium Risk
Touches core CropModel training/inference data flow and changes batch/forward signatures to support metadata; defaults remain unchanged, but incorrect metadata wiring or shape handling could break crop-model prediction/training in edge cases.

Overview
Adds optional spatial-temporal metadata (lat/lon/date) support to CropModel, including a new SpatialTemporalEncoder and a backbone+metadata+classifier path when cropmodel.use_metadata is enabled, while keeping image-only behavior as the default.

Training can now load per-image metadata from a sidecar CSV via CropModel.load_from_disk(..., metadata_csv=...), and inference can pass image-level metadata through deepforest.predict_tile(..., metadata=...) which is expanded to per-crop tensors and threaded through predict/BoundingBoxDataset.

Updates config/schema/docs to expose use_metadata, metadata_dim, and metadata_dropout, and adds a dedicated test suite covering encoder behavior, training/predict steps, dataset wrappers, and checkpoint save/load.

Written by Cursor Bugbot for commit c763c17. This will update automatically on new commits. Configure here.

Species distributions vary by location and season. This adds an optional
late-fusion pathway that encodes (lat, lon, date) as a small sinusoidal
embedding (~1.5% of image features) concatenated with ResNet-50 features
before the classifier. Metadata is fully optional — the model gracefully
degrades to image-only predictions when metadata is absent.

Key changes:
- SpatialTemporalEncoder with sinusoidal encoding + small MLP
- CropModel.create_model splits into backbone + classifier when use_metadata=True
- MetadataImageFolder wraps ImageFolder with CSV sidecar for training
- BoundingBoxDataset supports per-crop metadata for inference
- predict_tile accepts metadata dict (lat, lon, date)
- Config: use_metadata, metadata_dim, metadata_dropout
- 19 new tests covering encoder, model, datasets, and integration
- Documentation for CropModel docs and configuration reference

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bw4sz bw4sz added Feature Request New feature or request in progress Someone is currently working on this. Check with the assigned user before working on it. labels Mar 5, 2026
@bw4sz
Copy link
Collaborator Author

bw4sz commented Mar 8, 2026

I changed something small, I don't agree with bugbot.

Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Feature Request New feature or request in progress Someone is currently working on this. Check with the assigned user before working on it.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant