-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Description
Summary
AI-enabled tests currently register themselves with AiExecutor, return placeholder results, and rely on GraderService to detect executors after grading to send a batch request. Results are then mutated in place, breaking the step-by-step pipeline model.
Scope
- Redesign AI execution so batch processing happens in a dedicated pipeline-aware component (e.g., separate step or executor strategy) instead of hidden inside test functions.
- Remove the
hasattr(test_func, "executor")check inGraderServiceand disallow in-place mutation ofTestResultobjects. - Ensure AI tests follow the same contract as other tests (execute → return result).
Impacted Files
- autograder/utils/executors/ai_executor.py
- autograder/services/grader_service.py
- Templates/tests using AI executors
Risks
- AI grading may regress if batch semantics change; add targeted integration tests.
Acceptance Criteria
- AI execution no longer hijacks test functions; results flow through the pipeline like other tests.
- GraderService has no executor-specific checks.
References
- docs/roadmaps/TECHNICAL_DEBT_ROADMAP.md (Item 21)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels