Skip to content

[FEAT]: Add per-field confidence scoring and requires_review flag to LLM extraction#368

Open
SHANKAR8983 wants to merge 3 commits intofireform-core:mainfrom
SHANKAR8983:feat/confidence-score-requires-review
Open

[FEAT]: Add per-field confidence scoring and requires_review flag to LLM extraction#368
SHANKAR8983 wants to merge 3 commits intofireform-core:mainfrom
SHANKAR8983:feat/confidence-score-requires-review

Conversation

@SHANKAR8983
Copy link
Copy Markdown

Summary

In production emergency form processing, blind trust in LLM output is
dangerous. This PR adds heuristic confidence scoring to each extracted
field and a top-level requires_review flag so downstream systems and
human operators know when to intervene.

Changes

  • src/llm.py: Added _compute_field_confidence() — scores each field 0.0–1.0
  • src/llm.py: Added build_extraction_result() — wraps all fields with scores + _meta block
  • src/llm.py: main_loop() now logs confidence-scored output
  • src/test/test_confidence.py: 15 unit tests, all passing ✅

Example Output

{
  "name": {"value": "John Smith", "confidence": 0.9},
  "date": {"value": null, "confidence": 0.0},
  "_meta": {
    "requires_review": true,
    "low_confidence_fields": ["date"],
    "overall_confidence": 0.45
  }
}

Why this matters

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant