Agentified personal in-home care for the elderly. A multimodal speech-to-speech agent system that handles patient charts, leading medical knowledge, medicine routines, and emergency contacts — all in a friendly, voice-first package.
"Hey Jarvis, when is my next appointment?"
Built at TreeHacks 2026 by Team .dot
Every year, millions of elderly patients struggle to navigate complex digital health portals like MyChart. They forget medication schedules, miss appointments, and can't easily access their own lab results — not because the information doesn't exist, but because the interfaces weren't designed for them.
We watched our own grandparents fumble through tablets and phone screens just to check a prescription refill, and we thought: what if your medical records could just talk to you?
.dot is an AI-powered healthcare companion that lets elderly patients interact with their full medical records through natural voice conversation. No screens, no logins, no menus. Just talk.
- Voice-First Medical Record Access — Patients ask about prescriptions, lab results, upcoming appointments, visit history, and their care team through real-time speech.
- Medication Barcode Scanning — Scan any medication bottle with a USB barcode reader. .dot matches it against prescriptions, confirms dosage, flags refill status, and checks for drug interactions.
- Medical Information Search — General health questions are answered using trusted sources (PubMed, NIH, FDA, Mayo Clinic, CDC) via the Perplexity API.
- Emergency Escalation — If .dot detects signs of a medical emergency (chest pain, stroke symptoms, difficulty breathing), it places a real phone call to the patient's emergency contact or provider via Twilio.
- LED Ambient Feedback — A physical LED strip provides visual cues: gentle breathing purple when idle, solid blue when listening, red/blue chase when thinking. No screen needed.
- Proactive Medication Reminders — .dot parses prescription schedules and proactively reminds patients when it's time to take their medication.
- Conversation Logging — Every interaction is logged with full transcripts and streamed via WebSocket to a mobile app for caregiver review.
.dot System Architecture
┌──────────────┐ ┌──────────────────┐ ┌──────────────────────────┐
│ Arduino │ │ Wake Word │ │ OpenAI Realtime API │
│ LED Strip │◄────│ Detection │────►│ (GPT-4o Speech-to- │
│ (WS2812B) │ │ ("Hey Jarvis") │ │ Speech) │
└──────────────┘ └──────────────────┘ └──────────┬───────────────┘
│ │
┌────────▼────────┐ ┌────────▼────────────┐
│ USB Barcode │ │ Function Calling │
│ Scanner │ │ ┌─ Perplexity API │
│ (NDC/UPC │ │ ├─ Twilio Calls │
│ Lookup) │ │ ├─ Drug Interaction│
└─────────────────┘ │ └─ SMS Alerts │
└────────┬───────────┘
│
┌──────────────┐ ┌──────────────────────────────┐ ┌▼────────────────────────────┐
│ Caregiver │ │ Patient Mobile App (PWA) │ │ Mock MyChart Portal │
│ Admin │◄────│ dot-mychart-mock-beta. │◄────│ (Next.js + Prisma) │
│ Dashboard │ │ vercel.app/patient-app │ │ dot-steel.vercel.app │
└──────────────┘ └──────────────▲───────────────┘ └──────────────┬───────────────┘
│ │
│ ┌──────────────────────────┐ │
│ │ Browserbase + Stagehand │ │
└────│ AI-Powered EHR Scraper │◄───┘
│ Extracts patient data │
└──────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ OpenEMR Pipeline: Synthea → Docker OpenEMR → Stagehand Scraper │
└─────────────────────────────────────────────────────────────────────┘
.dot/
├── voice_agent/ # Python voice agent — OpenAI Realtime API, Twilio, barcode scanner
├── mock-mychart/ # Next.js patient portal — prescriptions, labs, appointments
├── admin-dashboard/ # React caregiver console — patient roster, vitals, chat logs
├── mobile-app/ # React PWA patient app — deployed on Vercel
├── browserbase/ # Stagehand scraper — AI-powered MyChart data extraction
├── openemr-pipeline/ # Synthea + OpenEMR Docker pipeline — synthetic EHR generation
├── arduino/ # FastLED controller — WS2812B LED strip animations
├── listen.py # Main entry point — wake word + barcode + medication scheduler
├── led_test.py # LED strip serial test utility
└── train_dot.ipynb # Custom wake word model training notebook
The core of .dot. A Python application that connects to OpenAI's Realtime API over WebSocket, streaming bidirectional audio at 24kHz with server-side voice activity detection.
| Module | Purpose |
|---|---|
healthcare_voice_agent.py |
Main agent — streams audio, manages conversation, calls functions |
barcode_scanner.py |
NDC/UPC lookup via openFDA, prescription matching |
conversation_logger.py |
SQLite logging + WebSocket streaming (port 8765) |
led_controller.py |
Serial communication to Arduino LED strip |
medication_scheduler.py |
Parses Rx instructions, fires timed reminders |
twilio_call_bridge.py |
Bidirectional audio bridge for emergency phone calls |
Function calling tools available to the agent:
call_emergency_contact— Places a Twilio phone call to the patient's emergency contactsearch_drug_interactions— Checks for interactions between medicationsonline_search— Queries trusted medical sources via Perplexity API (sonar model)send_sms— Sends SMS notifications to caregivers
The patient's full medical chart is injected into the system prompt, giving the agent grounded context to answer questions without hallucinating.
A full-featured patient portal built with Next.js 16, React 19, Prisma ORM, and NextAuth 5.
- Patient dashboard — Prescriptions, lab results, appointments, visit history, care team
- Admin interface — Provider tools for managing patient records
- Authentication — Credentials-based login with role-based access (Patient, Provider, Admin)
- Database — PostgreSQL via Prisma with schema covering patients, providers, prescriptions, lab results, visits, and appointments
A React 19 + Vite caregiver monitoring console with SMART on FHIR integration.
- Patient roster with clinical summaries
- Real-time vitals monitoring with Recharts visualizations
- Voice conversation log review
- Alert management and escalation tracking
- Medication adherence analytics
- Device flee
Web_Dashboard.mp4
A Progressive Web App deployed on Vercel for patient self-service.
- Home dashboard with health summary
- Voice chat interface for .dot conversations
- Activity calendar for medication adherence tracking
- Health insights and trends
- Installable on mobile devices with offline support
Mobile_App_Demo.mp4
A TypeScript + Stagehand pipeline that automates data extraction from the mock-mychart portal using Browserbase's AI-powered browser automation.
- Logs into the patient portal using Stagehand
act() - Navigates to patient settings and extracts profile data
- Outputs structured JSON for the voice agent to consume
Browserbase_Session.mp4
End-to-end pipeline for generating and extracting synthetic patient data from a real EHR system.
make pipeline # Runs: Synthea → Docker OpenEMR → OAuth2 Import → Stagehand Scraper- Synthea generates FHIR R4 bundles for elderly patients (ages 65–85)
- Docker Compose runs OpenEMR 7.0.2 + MariaDB 10.11
- Import scripts handle OAuth2 client registration and FHIR bundle ingestion
- Stagehand scraper uses AI-powered browser automation for login, REST API for encounters, and direct DB queries for medications and diagnoses
- Outputs structured JSON: demographics, medications, diagnoses, past visits
OpenEMR.mp4
A FastLED-based WS2812B controller (32 LEDs) that provides ambient visual feedback.
| Serial Command | State | Animation |
|---|---|---|
'1' |
Idle | Soft breathing purple pulse |
'2' |
Thinking | Red LED chasing blue LED with random direction flips |
'3' |
Listening | Solid blue |
Custom OpenWakeWord model trained for "Hey Jarvis" using:
- Synthetic voice samples generated with Piper TTS
- Audio augmentation with environmental noise (ACAV100M, FMA, AudioSet)
- Room impulse responses (MIT RIRs) for acoustic realism
- Exported as ONNX model for on-device inference — no cloud dependency
- 92% accuracy, 87% recall
| Layer | Technologies |
|---|---|
| Voice AI | OpenAI Realtime API (GPT-4o), Perplexity API (sonar), OpenWakeWord, ONNX |
| Frontend | React 19, Vite 7.3, Tailwind CSS 4, Radix UI, PWA (Workbox) |
| Backend | Next.js 16, Prisma 5 (PostgreSQL), NextAuth 5, WebSockets |
| Healthcare | SMART on FHIR, FHIR R4, openFDA NDC API, Synthea, OpenEMR 7.0.2 |
| Browser Automation | Browserbase Stagehand v3, Playwright |
| Telephony | Twilio Voice API (bidirectional audio bridge) |
| Hardware | Arduino + FastLED (WS2812B), USB barcode scanner, PyAudio |
| Infrastructure | Docker Compose, Vercel, SQLite, serial (115200 baud) |
- Python 3.8+
- Node.js 22+
- Docker & Docker Compose
- Arduino IDE (for LED strip)
- USB barcode scanner (optional)
cd voice_agent
pip install -r requirements.txt
cp .env.example .env
# Add: OPENAI_API_KEY, PERPLEXITY_API_KEY, TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN
python ../listen.pycd mock-mychart
npm install
cp .env.example .env
# Add: DATABASE_URL, NEXTAUTH_SECRET
npx prisma migrate dev
npx tsx prisma/seed.ts
npm run devcd admin-dashboard
npm install
npm run dev
# Open http://localhost:5173/admincd mobile-app
npm install
npm run dev
# Open http://localhost:5174/patient-appcd openemr-pipeline
docker compose up -d
make pipeline
# Output: pipeline/output/scraped/stagehand-patients.json- Open
arduino/main.inoin Arduino IDE - Connect WS2812B strip to pin D6 (32 LEDs)
- Upload sketch
- The Python agent auto-detects the serial port
.dot is a four-layer system: a real-time voice agent, a full-stack mock patient portal, a caregiver mobile app, and a hardware feedback loop.
Voice Agent (Python) — Connects to OpenAI's Realtime API over WebSocket, streaming bidirectional audio at 24kHz. Server-side VAD detects when the patient starts and stops speaking. The patient's full medical chart is injected into the system prompt for grounded responses. Perplexity API handles medical search filtered to trusted domains. Twilio creates real-time bidirectional phone calls for emergencies. The barcode pipeline queries openFDA's NDC database and UPCItemDB to identify scanned medications.
Wake Word Detection — Custom model trained with OpenWakeWord using synthetic Piper TTS samples augmented with noise. Achieves 92% accuracy running locally via ONNX with zero cloud dependency.
Patient Portal (Next.js + PostgreSQL) — Full MyChart-style portal with NextAuth authentication, Prisma ORM, role-based access, and comprehensive schema covering patients, providers, prescriptions, labs, visits, and appointments. A Browserbase Stagehand pipeline scrapes patient data from the portal for the agent.
Hardware (Arduino + WS2812B LEDs) — 32-LED addressable RGB strip communicating over serial at 115200 baud. Single-character commands switch between breathing purple (idle), red/blue chase (thinking), and solid blue (listening).
Real-time audio reliability was our biggest hurdle. Streaming bidirectional audio over WebSocket with the OpenAI Realtime API required careful management of audio buffers, playback queues, and interruption handling.
WebSocket stability proved tricky — multiple iterations to get WSS working reliably for conversation logging, debugging SSL certificate issues and connection drops.
Wake word accuracy started with too many false positives. We iteratively improved by generating diverse synthetic training samples and applying audio augmentation, reaching 92% accuracy.
Barcode-to-prescription matching required stitching together multiple APIs. Medication barcodes don't use a single standard — we handle NDC codes, UPC codes, and Rx numbers, falling back through multiple lookup services.
Prompt engineering for safety was critical. The agent must be helpful and conversational while refusing to diagnose or prescribe. Balancing warmth with medical safety guardrails took extensive iteration.
- End-to-end voice interaction with real medical data — Ask about medications, scan a pill bottle, get grounded answers from actual records, no screen needed
- Emergency escalation that actually calls — Real Twilio phone calls to emergency contacts, not just alerts
- Custom wake word model — "Hey Jarvis" at 92% accuracy, running entirely on-device
- A complete functional patient portal — Full-stack with auth, RBAC, CRUD, and realistic seed data
- Physical hardware integration — LED strip transforms software into a tangible bedside presence
- Conversation logging and auditability — Every interaction recorded and streamable in real-time
- Real EHR integration — Connect to Epic MyChart and other systems via FHIR APIs
- Multi-language support — Spanish, Mandarin, Tagalog, and other languages common in home care
- Caregiver real-time dashboard — Live monitoring of conversations, alerts, and adherence patterns
- Clinical validation — Pilot with home health agencies and geriatric care providers
- HIPAA compliance — End-to-end encryption, secure credential storage, and audit logging
Built with care at TreeHacks 2026.