UI for Medical AR/VR Applications

Problem Statement

The digitisation extends to all areas of life and offers many opportunities. The medical sector in particular can benefit from it. Modern technologies like augmented and virtual reality can support healthcare workers for example by displaying important medical information. An enormous challenge for medical professionals in their daily work with patients is to compare previous diseases and current symptoms with existing knowledge about possible illnesses and treatment options. This requires various skills from the medical professionals: On the one hand, there must be an up-to-date and well-founded knowledge background, and on the other hand, an unbiased, objective and calm judgement must be made, even when dealing with a large number of patients. Especially in highly frequented hospitals this is often not feasible.

Idea and Concept

The main aspect of this project is the research of new, standardised input methods for VR applications and the appropriate presentation of important medical information to support the physician. Based on current research, non-diegetic and spatial UI elements will be designed in a medical scenario, an eye specialist examination, then implemented using the Unreal Engine and finally evaluated with volunteers. The prototype follows a defined procedure, in which different stages of an examination at the eye specialist are passed through. Information is displayed to the volunteer in different ways. The subject experiences the scenario from the doctor’s perspective and is given additional information about the patient like underlying health conditions or personal data such as the age.

Implementation

The implementation of the project is realised as an AR simulation in virtual reality in order to create an immersive work environment for the physician. To implement the prototype, the software Sketch and the game engine Unreal Engine were utilised. Sketch was mainly used to design widgets that enabled the interaction with the user. In order to create the spatial representation, the UI elements remain fixed on their position and the user has to move accordingly to act or view them. To simplify this, a pointer was implemented so the user is able to interact with distant objects that are not within the user’s reach. To test the non-diegetic interface, the UI elements were implemented in such a way that they are dependent on the user’s field of vision. The UI elements are focused on the user and move accordingly when the user changes their viewing position.