Package project and harden local setup

[Sidharthwin]

fixes #366

Description

This PR improves the local development experience, resolves test execution issues, and introduces proper packaging for the project.

Previously, running the test suite in a clean environment failed due to module resolution errors (ModuleNotFoundError: api) and required manual PYTHONPATH configuration. Additionally, the setup was under-documented, dependencies were tightly coupled, and optional components caused unnecessary failures.

This PR addresses these issues by making the project installable, improving dependency management, and ensuring tests run reliably out of the box.

Fixes # (local setup / test execution issue)

---

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

---

Changes Made

Packaging and Setup

• Added pyproject.toml to make the project installable
• Enabled editable installs (pip install -e .)
• Defined Python version (>=3.11)

Dependency Management

• Split dependencies into:

  • requirements-dev.txt (dev + testing)
  • requirements-ai.txt (optional AI dependencies)

• Removed unnecessary heavy dependencies from core install

Test Reliability

• Fixed test discovery using pytest.ini (testpaths = tests)
• Updated test structure and imports
• Added proper fixtures for database and client setup
• Mocked optional dependencies (commonforms) in tests
• Ensured tests run without requiring optional AI dependencies

Environment Fixes

• Fixed incorrect PYTHONPATH usage in Docker and environment configs
• Updated Makefile test command to correct test directory

Code Improvements

• Converted hard dependency on commonforms into optional import with proper error handling
• Improved modularity and robustness of file handling logic

Documentation

• Updated README with:

  • Python version requirement
  • Virtual environment setup
  • Editable install instructions
  • Clear testing instructions

• Updated database setup documentation

---

How Has This Been Tested?

• Test suite runs successfully using:

  pytest -q

• Verified tests run in a clean virtual environment

• Verified tests pass without installing optional AI dependencies

• Verified Docker setup works with updated configuration

Test Configuration:

• Python version: 3.11
• OS: macOS / Linux (local testing)
• Docker: Used for container validation

---

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code where necessary
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective
• New and existing unit tests pass locally with my changes
• Any dependent changes have been handled

---

Note on Development Process

While working on this PR, I leveraged AI-assisted tools to explore potential solutions and accelerate debugging. However, I carefully reviewed, tested, and validated all changes to ensure correctness and alignment with the project's architecture.

This process also helped me deepen my understanding of Python packaging, dependency management, and test design.

---

Additional Notes

This PR significantly improves developer onboarding and reduces friction for contributors by ensuring the project works out of the box without manual configuration.

Future improvements could include further modularization of optional features and CI integration for automated testing.