An end-to-end mixed reality golf training application that bridges physical and virtual gameplay using Meta Quest 3's passthrough cameras and computer vision.
Watch the full demo on YouTube →
This system detects and tracks a physical golf ball in real-time, captures its launch velocity when struck, and spawns a physics-accurate virtual ball that continues the trajectory through a virtual golf course—all while maintaining sub-frame latency requirements for a seamless user experience.
Implemented HSV color segmentation and contour analysis using OpenCV to detect golf balls at 30+ FPS with adaptive region-of-interest optimization for performance.
Developed a camera-to-world coordinate transformation pipeline using Quest 3's camera intrinsics (focal length, principal point) and extrinsics to accurately map 2D detections to 3D world positions.
Built a multi-frame velocity estimation system with exponential smoothing, outlier rejection, and launch confirmation logic to capture accurate ball speed and direction.
Implemented translational and rotational head motion compensation to filter out apparent ball movement caused by user head movement, significantly reducing false positive launches.
Leveraged Meta's Mixed Reality Utility Kit (MRUK) for world-locking, spatial anchors, and scene understanding to align virtual content with the physical environment.
Developed hole-to-hole navigation with MRUK resynchronization to maintain tracking accuracy across virtual course locations.
| Category | Technologies |
|---|---|
| Game Engine & XR | Unity, Meta Quest SDK, Meta XR SDK, MRUK, OpenXR |
| Computer Vision | OpenCV, HSV Color Segmentation, Contour Detection, Camera Calibration, 3D Reconstruction |
| Machine Learning | YOLOv8, Unity Sentis (no longer implemented) |
| VR/AR/MR | Meta Quest 3, Passthrough, Spatial Anchors, World Locking, Hand Tracking |
| 3D Math & Graphics | Coordinate Transformations, Pinhole Camera Models, Quaternions, Physics Simulation, Shaders (HLSL/ShaderLab) |
| Tools | Git/GitHub, Android (APK Deployment), Unity Profiler |
| Languages | C#, C |
This project is for educational and portfolio purposes only.