Skip to content

Fix AP/mAP degradation due to unsorted prediction confidence scores & Add IoU Zero-Division Safeguard#545

Open
shardulmahamulkar wants to merge 1 commit intoJdeRobot:masterfrom
shardulmahamulkar:fix-evaluation-metrics
Open

Fix AP/mAP degradation due to unsorted prediction confidence scores & Add IoU Zero-Division Safeguard#545
shardulmahamulkar wants to merge 1 commit intoJdeRobot:masterfrom
shardulmahamulkar:fix-evaluation-metrics

Conversation

@shardulmahamulkar
Copy link
Copy Markdown

This PR addresses two critical bugs in the perceptionmetrics detection evaluation pipeline that result in artificially degraded AP (Average Precision) scores and potential crashes:

Prediction Confidence Sorting (#Bug 3): Standard evaluation mapping protocols dictate that multiple boxes targeting a single ground truth must be resolved sequentially in descending order of their confidence scores. The _match_predictions() method was evaluating mapping criteria based on arbitrary array index order. This allowed low-scoring detections to lock in early as the True Positive target, delegating higher confidence detections as False Positives, unjustly penalizing high-threshold precision curves.

Degenerate Area Bug / IoU Zero-Division Safeguard (#Bug 1): compute_iou() did not protect against zero-area unions (i.e., evaluating degenerate predicted or ground-truth boxes), causing ZeroDivisionError crashes mid-evaluation. A safeguard was implemented to return 0.0 seamlessly in these cases without halting execution.

Changes Made

  • Added np.argsort() within perceptionmetrics/utils/detection_metrics.py to correctly map predictions traversing from highest confidence to lowest.
  • Protected Python float division in compute_iou() against division-by-zero crashes, utilizing if unionArea > 0 else 0.0.

How to Verify

  • Run detection evaluation against overlapping low-confidence/high-confidence objects—AP/mAP calculations will accurately penalize the duplicates and effectively reward the peak detection thresholds.
  • Execute compute_iou() on overlapping degenerate/empty boxes—the application correctly resolves an IoU of 0.0 instead of crashing.

Related Issues

Closes #544

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DetectionMetricsFactory AP/mAP computation evaluates predictions unsorted, causing precision degradation

1 participant