AI-Enhanced VR Venipuncture Training & Medical AI System
An integrated medical training platform combining immersive Unity VR, a custom medical language model, retrieval-grounded medical assistance, voice interaction, and performance reporting.
Inside the Simulation
Explore the VR procedure, assessment scenarios, and integrated medical AI assistant through five project demonstrations.
High-resolution recordings ranging from 46 MB to 407 MB. Click "Watch Demo" on any card to stream directly in-page.
Medical AI Assistant
Demonstration of the medical language-model component and AI-assisted medical question answering grounded in clinical standards.
- Custom MedicalTransformerLM inference
- Clinical question answering & knowledge grounding
- Zero hallucination verification against medical protocols
Medical VR Web Demo
Comprehensive walkthrough of the medical VR project, its virtual clinical room environment, and hands-on phlebotomy training experience.
- Full clinical healthcare suite in Unity 3D
- Realistic equipment grabbing & two-handed manipulation
- End-to-end procedural workflow demonstration
Venipuncture Test Scene with Console
Demonstration of the venipuncture test scene, including the live visible developer console for real-time inspection of the state machine workflow.
- Blind assessment mode without visual assist prompts
- Active developer console logging state transitions
- Procedural error detection & milestone checks
Venipuncture Test Scene
Demonstration of the unassisted venipuncture assessment scene, evaluating procedural accuracy, technique execution, and sequential workflow.
- Objective student evaluation protocol
- Insertion angle validation (15°–30° range)
- Sharps disposal and patient care checklist verification
Venipuncture Training Scene with Console & Local Services
Demonstration of the guided training environment, developer console output, and local-service integration for procedural progression and telemetry caching.
- Interactive visual step indicators & clinical guidance hints
- Local storage telemetry and FastAPI service synchronization
- Real-time equipment feedback & vein palpation confirmation
Venipuncture VR — System ArchitectureVerified Pipeline
Select any pipeline stage below to inspect verified engineering details and data flow.
Stage: Unity VR Application
Role: VR Client & SimulationEnforces the structured 16-step phlebotomy protocol, two-handed tool physics, 15°–30° insertion angle tracking, and flashback visual verification.
System Components & Technical Deep Dives
A comprehensive breakdown of the integrated clinical architecture combining Unity VR simulation, custom PyTorch language modeling, hybrid RAG retrieval, hands-free voice assistance, and automated clinical evaluation.
Unity VR Development
High-fidelity clinical virtual room with tactile physics and 16-step procedural progression
Engineered an authentic hospital clinical room in Unity featuring a complete 16-step procedural state machine. The simulator supports both guided Training Mode (with visual guidance cues and real-time needle angle meters) and Test Mode (unassisted scenario assessment with automated error classification).
Architecture Components & Subsystems
Key Procedural & Algorithmic Capabilities
Custom MedicalTransformerLM
Custom PyTorch autoregressive transformer paired with tiered multi-model clinical routing
Designed, pre-trained, and fine-tuned an approximately 110M-parameter autoregressive decoder transformer in PyTorch specialized for medical question answering and clinical procedure assistance. Operates in a hybrid routing pipeline alongside OpenBioLLM-8B and Llama 3.2 3B for complex queries.
Architecture Components & Subsystems
Key Procedural & Algorithmic Capabilities
Medical RAG Pipeline
Dense semantic embeddings combined with sparse BM25 retrieval and strict grounding verification
Engineered a resilient medical retrieval-augmented generation pipeline that pairs lexical BM25 matching with PostgreSQL pgvector dense similarity embeddings, ensuring all answers generated for learners are grounded in authoritative clinical literature.
Architecture Components & Subsystems
Key Procedural & Algorithmic Capabilities
Voice-Based Medical Assistant
Sterile field-preserving voice interaction pipeline delivering real-time clinical answers
Built an interactive voice pipeline enabling medical trainees to speak natural clinical questions inside the virtual environment without releasing equipment or breaking sterile procedure. Audio captured by the headset microphone is converted to text, answered by the medical AI, and spoken aloud.
Architecture Components & Subsystems
Key Procedural & Algorithmic Capabilities
Assessment and PDF Reporting
Automated objective grading with timestamped mistake detection and downloadable clinical PDF reports
Developed an objective clinical evaluation engine that records all trainee actions, calculates angle precision and procedural timing, detects clinical errors, and outputs a publication-quality training performance report in PDF format.
Architecture Components & Subsystems
Key Procedural & Algorithmic Capabilities
Backend Integration and Validation
High-throughput asynchronous web services coordinating AI models, speech, and telemetry
Engineered an asynchronous FastAPI backend service that connects the Unity VR client to the AI inference engines, speech processors, RAG vector database, and reporting system into one cohesive platform.
Architecture Components & Subsystems
Key Procedural & Algorithmic Capabilities
Technology Stack
An integrated engineering stack spanning real-time game engines, custom deep learning models, hybrid information retrieval, voice AI, and clinical-grade reporting services.
VR & Simulation Runtime
Medical AI & Language Models
Retrieval & Vector Storage
Voice AI & Audio Systems
Backend & Automated Reporting
Validation, DevOps & Quality
End-to-End Procedural Workflow & Mechanics
Unity VR Scene & 16-Step Procedural Protocol
Spawns the learner inside an authentic hospital simulation room, enforcing sequential clinical milestones from patient identification to sharps disposal.
Tactile Tool Interaction & 15°–30° Angle Tracking
Handles physics-based manipulation of vacutainers, needles, and tubes via VR Interaction Framework, with real-time insertion angle validation.
Hands-Free Voice Query & Whisper STT Streaming
Captures microphone audio directly from the VR headset and streams buffers to FastAPI for real-time Whisper speech-to-text transcription.
Hybrid BM25 + pgvector Clinical RAG Retrieval
Queries clinical guidelines (CLSI GP41 & WHO) via dense semantic vectors and sparse lexical search in PostgreSQL, enforcing citation verification.
Grounded LLM Inference & Spoken Audio Feedback
Synthesizes clinically accurate guidance through the custom 110M model or foundation LLMs, converting text to audio via macOS speech engine.
Test Scenario Scoring & Automated PDF Report Generation
Tracks procedural execution, logs mistake telemetry, and automatically compiles comprehensive clinical PDF training summary reports.
Engineering Challenges & Design Decisions
Problem Statement
Venipuncture is one of the most critical foundational clinical procedures, yet traditional medical education relies on static rubber manikins or high-stakes live practice. Learners lack an immersive, repeatable simulation platform where they can develop physical procedural muscle memory, ask real-time clinical questions without breaking the sterile field, and receive objective automated performance assessments with certified reporting.
Engineering Goals
- Unity VR Development: Build an immersive virtual clinical healthcare room in Unity with C# and the VR Interaction Framework, implementing an enforced 16-step procedural state machine across Training and Test modes.
- Custom MedicalTransformerLM: Design, train, and deploy an approximately 110M-parameter PyTorch autoregressive medical model alongside OpenBioLLM-8B and Llama 3.2 3B for low-latency clinical Q&A.
- Medical RAG Pipeline: Engineer a hybrid retrieval engine combining BM25 lexical precision with PostgreSQL pgvector dense embeddings for verified clinical guidance and strict grounding.
- Voice Assistant: Integrate hands-free Whisper speech-to-text and macOS speech engine text-to-speech so learners can ask procedure questions without leaving the virtual sterile field.
- Assessment & PDF Reporting: Build an automated evaluation engine tracking insertion angle variance (15°–30°), procedural errors, and generating downloadable ReportLab clinical PDF reports.
- Backend Integration: Couple Unity to an asynchronous FastAPI backend service with REST endpoints orchestrating models, speech, and training session lifecycles.
Decision: Custom 110M MedicalTransformerLM + Foundation Router over single cloud LLM
Rationale: Local 110M model delivers sub-150ms inference latency critical for real-time VR guidance on consumer hardware, while router dispatches complex medical questions to OpenBioLLM-8B.
Alternative Rejected: Relying exclusively on high-latency cloud API calls for every VR hint.
Decision: Hybrid PostgreSQL pgvector + BM25 over dense-only vector search
Rationale: Medical equipment codes, tube cap colors (lavender EDTA vs gold SST), and phlebotomy abbreviations require exact keyword precision combined with semantic reasoning.
Alternative Rejected: Dense vector-only embeddings without keyword indexing.
Decision: Decoupled Training vs Test mode scene architecture
Rationale: Ensures the assessment mode cannot be compromised or influenced by training hints or step assists, guaranteeing objective evaluations.
Alternative Rejected: Single monolithic scene with conditional UI flags.
Core Architectural Highlights
The platform unifies six interconnected subsystems into an integrated training pipeline: (1) An OpenXR-compliant Unity VR client running a 16-step clinical state machine; (2) An asynchronous FastAPI backend acting as the central coordination gateway; (3) A custom 110M PyTorch MedicalTransformerLM routed dynamically with OpenBioLLM-8B and Llama 3.2 3B; (4) A hybrid clinical RAG pipeline querying PostgreSQL pgvector and BM25 indices; (5) A hands-free Whisper STT and macOS TTS audio loop; and (6) An automated assessment engine compiling performance metrics into downloadable clinical PDF reports.
Testing & Empirical Evaluation
Benchmarked across five dedicated demonstration recordings covering the medical AI assistant, full VR web walkthrough, test scenes with live developer console, and training scenes with local storage.
Security & Reliability Defenses
Key Engineering Learnings
- •Pairing spatial VR procedural muscle memory with hands-free voice-grounded medical RAG creates an unprecedentedly effective clinical learning environment.
- •A multi-tier model router running a fast local 110M model alongside foundation LLMs delivers the optimal balance between VR real-time latency and clinical depth.
Verified Project Resources & Artifacts
Direct links to verified repositories, Google Drive multi-stream video recordings, model specifications, and clinical guidelines.
Source Code Repository
Access the complete Unity 3D project scripts, C# procedural state machine, FastAPI backend services, PyTorch training scripts, and automated ReportLab evaluation engine.
Integrated Simulation Suite
All five high-resolution recordings (Medical LLM demo, full VR web walkthrough, test scenes with console telemetry, and training scene) are embedded directly in this case study.
Clinical Standards & Guidelines Grounding
All procedural steps, error triggers, question-answering vectors, and evaluation scoring models strictly conform to published clinical phlebotomy protocols: