Designing the First Hand Menus for Spatial Computing

Problem: With HoloLens II, the introduction of fully articulated hand tracking opened a new frontier for spatial interaction. For the first time, users could interact directly with holograms using natural hand movements no controllers, no clickers. But this leap came with a challenge: accessing apps and core system actions became slow and inconsistent because the technology was so new that its limitations were still unknown. We needed a fast, reliable, and ergonomically sound way for users to trigger system level commands, while operating within the real constraints of early hand tracking far from the “Minority Report” fantasy everyone hoped for.

Solution: I led the interaction design for the first generation of hand menus on HoloLens II, creating prototypes that tested the limits of the hand tracking system, evaluated ergonomics, and identified optimal placement, gesture reliability, and activation thresholds. Through iterative prototyping and cross disciplinary collaboration, we uncovered the boundaries of the technology and transformed the most successful patterns into a direct activation menu system anchored to the user’s hands. This gave developers and users a fast, controller free way to access actions and tools, significantly improving usability and precision across mixed reality scenarios. The project established foundational principles for hand-based UI in spatial computing and helped shape one of the defining interaction models of HoloLens II.