VR Multi-Camera Control with Voice Commands and Autonomous View Selection for Robotic Nursing Assist

Teleoperation in robotics has significantly advanced with the integration of multi-camera systems, which enhance operator performance and reduce cognitive load compared to traditional single-camera interfaces. This project aims to implement an intuitive multi-camera control system for the Intelligent Robotic Nursing Assistant (IONA) to support teleoperation tasks in complex environments through a virtual reality (VR) interface. The system features three fixed workspace cameras within the nursing setup, static cameras mounted on the robot’s chest and arms, and an active camera that mirrors the operator’s head movements to provide an immersive experience.
To simplify operator control, the system incorporates voice commands for switching camera views and employs an object detection model running on the ZED Mini workspace camera to estimate the robot’s pose and automatically select the most relevant camera view. The effectiveness of this system will be evaluated through a comparative user study across three interaction modalities: (i) Manual Mode, where camera views are switched via manual interactions with the interface; (ii) Voice Mode, where views are changed using voice commands; and (iii) Automatic Mode, where camera views are selected autonomously based on the robot’s pose and teleoperation phase.
This study aims to assess the impact of voice control and automated camera selection on teleoperation efficiency and user experience when operating the nursing assistant robot.