Phase 5: Recalibration Analytics & Monitoring

[smithrashell]

Phase 5: Recalibration Analytics & Monitoring

Overview

Implement analytics and monitoring for the recalibration system to track effectiveness and provide insights to users about their retention and recalibration patterns.

Background

Phases 1-4 of the recalibration system are complete:

• Phase 1: Passive background decay
• Phase 2: Welcome back modal with gap-based strategies
• Phase 3: Diagnostic session creation and processing
• Phase 4: Adaptive first session with performance-based adjustments

This phase adds visibility and metrics to help users and developers understand how the recalibration system is performing.

Goals

1. Track recalibration events and outcomes
2. Monitor retention accuracy and decay effectiveness
3. Provide user-facing analytics about their retention
4. Add developer-facing metrics for system optimization

Implementation Tasks

1. Analytics Data Collection (2-3 hours)

• Create RecalibrationAnalyticsService to collect metrics
• Track diagnostic session outcomes (accuracy, topics retained/forgotten)
• Track adaptive session outcomes (decay kept/reduced/reverted)
• Record recalibration approach choices (diagnostic vs adaptive)
• Store time-series data for trend analysis

2. Dashboard Widgets (3-4 hours)

• Add "Days Since Last Use" widget to overview page
• Create "Retention Summary" widget showing:
  • Overall retention rate after gaps
  • Topics with strong retention
  • Topics with decay
• Add "Recalibration History" widget:
  • Recent recalibration events
  • Approach used (diagnostic/adaptive)
  • Outcomes and adjustments

3. Statistics & Reports (2-3 hours)

• Calculate retention accuracy metrics:
  • Retention rate by topic
  • Retention rate by difficulty level
  • Retention vs gap duration
• Generate diagnostic success metrics:
  • Average diagnostic accuracy
  • Most/least retained topics
• Track adaptive session effectiveness:
  • Frequency of decay adjustments
  • Average performance improvement

4. User Insights (2-3 hours)

• Add insights to progress page:
  • "Your knowledge retention is X% after Y days"
  • Topic-specific retention feedback
• Create retention trend charts:
  • Retention over time
  • Gap duration impact
• Add recommendations:
  • Optimal practice frequency
  • Topics needing review

5. Developer Metrics (1-2 hours)

• Add console logging for recalibration events
• Track system-level statistics:
  • % of users experiencing decay
  • Average decay magnitude
  • Recalibration approach distribution
• Add performance monitoring:
  • Recalibration processing time
  • Database query performance

Technical Considerations

Data Storage

• Store analytics in IndexedDB settings store
• Keep time-series data for last 12 months
• Aggregate older data for long-term trends

Performance

• Calculate metrics asynchronously
• Cache computed statistics (15-minute TTL)
• Use web workers for heavy computation if needed

Privacy

• All analytics stored locally (no external tracking)
• User can clear analytics data
• No personally identifiable information

Success Criteria

• Users can see their retention statistics
• System tracks all recalibration events
• Analytics load in <500ms
• Insights are actionable and clear
• No performance impact on core features

Future Enhancements (Post-Phase 5)

• Export analytics to CSV/JSON
• Compare retention with other users (anonymized)
• Machine learning for personalized retention predictions
• Integration with goal-setting features

Estimated Time: 10-15 hours

Dependencies

• Phases 1-4 must be complete ✅
• Dashboard service architecture ✅
• Chart components (Recharts) ✅

Testing Requirements

• Unit tests for analytics calculations
• Integration tests for data collection
• UI tests for dashboard widgets
• Performance benchmarks for metric computation

Related Issues

• Handle large usage gaps with intelligent recalibration system #206 (Parent issue: Recalibration System)