BDMS-626: Improve validation, schema alignment, and well inventory handling

[ksmuczynski]

Why

This PR addresses the following problem / context:

• Database mapping was incomplete: public_availability_acknowledgement, monitoring_status, well/water notes, and water level observations were not being persisted correctly.
• Several schema fields were overly strict — requiring values that should be optional, using plain strings where lexicon enums were expected, and enforcing contact/water level fields more rigidly than necessary.
• WellInventoryRow schema now integrates directly with Lexicon-based enums to ensure database consistency.
• BDD tests suffered from primary key conflicts and outdated test data that didn't align with new validation rules.
• Previously, a single malformed row in a CSV would abort the entire import process. Users need "best-effort" logic where valid data is saved while invalid rows are flagged.
• Now, individual row failures (validation or database) no longer abort the entire ingestion.

How

Implementation summary - the following was changed / added / removed:

• Database mapping:
  • public_availability_acknowledgement now maps to Location.release_status (True → public, False → private, unset → draft).
  • monitoring_status is now written to the StatusHistory table.
  • well_notes and water_notes are now stored as polymorphic notes on the Thing.
  • Providing depth_to_water_ft now auto-creates a Sample and Observation linked to the field activity.
• WellInventoryRow schema:
  • Made site_name, elevation_ft, elevation_method, and measuring_point_height_ft optional.
  • Replaced plain str types with lexicon-based enums for elevation_method, depth_source, well_pump_type, monitoring_status, sample_method, and data_quality.
  • Added a flexible_lexicon_validator for case-insensitive, whitespace-tolerant matching.
  • Relaxed contact validation to require only one of contact_name or contact_organization, and made water_level_date_time only required when depth_to_water_ft is present.
• Best-effort logic:
  • Implemented row-level atomic savepoints so individual row failures no longer abort the full import.
  • Failed rows are logged with 1-based row numbers and well IDs in a validation_errors list.
  • Fixed an UnboundLocalError caused by auto-generated well IDs
  • Updated error reporting to surface database-level failures as "Database error" fields.
  • Added commit=False support to allow services/thing_helper.py and services/contact_helper.py o participate in the outer best-effort transaction without prematurely committing.
• BDD test suite:
  • Updated test CSVs in tests/features/data/ to use valid lexicon terms and properly quoted comma-containing values.
  • Added scenario-based unique ID suffixing in Given steps to isolate tests and prevent primary key conflicts.
  • Updated well-inventory-csv.feature to assert partial success (e.g., "1 well is imported") in negative scenarios. All 44 scenarios now pass.

Notes

Any special considerations, workarounds, or follow-up work to note?

• Further enhancements may be required for complete schema validation coverage.
• Complete schema validation coverage will be scoped in a separate PR, most likely a future PR scoped at testing the ingestion of real user-entered data
• Lexicon matching is intentionally lenient (case-insensitive, whitespace-stripped) to reduce friction for CSV authors, but values must still resolve to a known lexicon entry or the row will be skipped.
• Test isolation is achieved via ID suffixing at the Given step level; if the shared test database is ever replaced with per-scenario teardown, this workaround can be removed.

[Copilot]

Pull request overview

This PR updates the well-inventory CSV ingestion pipeline to better align schema validation with lexicon-backed enums, persist previously-missed fields (notes, monitoring/public availability, water level observations), and change the import behavior to “best-effort” so individual row failures don’t abort the entire upload.

Changes:

• Added row-level savepoints and improved error reporting so invalid rows are skipped while valid rows are persisted.
• Updated WellInventoryRow schema to relax/adjust requirements and introduce lexicon-backed enum parsing plus CSV field aliases.
• Updated BDD/unit tests and test CSV fixtures to reflect new validation rules and partial-success behavior.

Reviewed changes

Copilot reviewed 34 out of 34 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
services/well_inventory_csv.py	Best-effort import via nested savepoints; mapping updates for release status, notes, monitoring status, and water level observation persistence
schemas/well_inventory.py	Schema optionality updates, lexicon enum coercion, contact + water level validation adjustments, and alias handling
services/thing_helper.py	Adds monitoring status history writes; adjusts commit/rollback behavior for outer-transaction support
services/contact_helper.py	Adjusts commit/rollback behavior for outer-transaction support
cli/service_adapter.py	Improves exit code and stderr reporting for partial success and validation failures
tests/test_well_inventory.py	Adds schema alias tests and helper row builder
tests/features/well-inventory-csv.feature	Updates expected partial-success outcomes in negative scenarios
tests/features/steps/*.py	Updates step definitions for partial success and loosens validation-error matching
tests/features/data/*.csv	Refreshes fixture data to match new lexicon/validation expectations
core/lexicon.json	Adds a new note_type term

[ksmuczynski]

@jirhiker Recent updates to the CLI command tests are failing on my branch. Since these changes don't seem to impact the well inventory ingestion, should I resolve the test failures within this PR, or would you prefer I handle them in a separate one before re-opening for review?