Can Smart Patient care benefit from Mixed Reality and Gestural Interfaces?
My name is Adelle Lin; I am a candidate for a Masters of Science at the Tandon School of Engineering, focusing on Human Computer Interaction. For my thesis, I am developing a project exploring the potential of Mixed Reality(MR) interfaces within the patient care experience. MR technology refers to the ability to access digital overlays in real space with our various senses through holographic technology, camera vision. This adds additional context to our physical worlds and the ability to create new experiences. MR affords the use of 3D digital objects, allowing us to access different parts of the brain than text and flat graphics can access. Connecting MR to the physical reality through Internet of Things (IoT) wifi networks or the hospital database, offers up the potential to incorporate these interfaces and experience into the daily workflow.

Currently, the technology enabling MR is still in its nascent stages, but there are consumer-ready devices such as Microsoft’s Hololens, ODG's Smart Glasses, Meta 2, and Looking Glass Factory’s Holoplayer one. There are enterprises that have already begun piloting studies with this technology. For example, NASA and Microsoft have teamed up to develop software called OnSight, a new technology that will enable scientists to work virtually on Mars using Microsoft HoloLens. ODG are working with Telemedicine companies like Medweb to develop solutions for the military in terms of facilitating electronic acquisition, viewing, communication/transmission, publishing, and storage of medical data.

My project is exploring the potential of future mixed-reality technology in patient care through the use of drawing gestures and movement. The opportunities I see can broadly be categorized within these areas:
- Communication
- Agency
- Safety
- Escapism
The current annotation platform I am currently working with is called Chalktalk, which is the development of an iconographic language accessed through drawing recognition algorithms.

Future Mixed Reality
Hololens Galaxy Explorer
Holoplayer Heart Slicer
Chalktalk drawn interface
The Survey
This survey presents a few scenarios where I believe MR could be implemented in a patient care environment. There are opportunities for both patient and clinician, either to be experienced individually or as a shared platform. For the clinician, scenarios presented are mostly around charting, EMR and communication with other clinicians and the patient. Whereas for the patient, the scenarios presented are around agency, education and entertainment There are many overlaps between patient and clinician use, since there is a high volume of interaction in a hospital ward or room, and therefor at times, patient and clinician can share the same system but with different information available to each party.

The first part of the survey are questions with fixed scaled choices and the second part presents a few open ended questions, which will be an opportunity to present long form feedback.

Please select a number on the scale from 1-7 to describe how "useful" a scenario may be (1 = "not useful" and 7 = "extremely useful".

Please submit your answers by Nov 22, 2017. Thank you!

Patient Opportunities
Patient can change their environment such as lighting and temperature of the room through gesture and drawing. This would connect them to IoT devices such as the Phillips Hue lighting suite and Nest, an IoT temperature controller.
Patient can create and access virtual environments in MR, changing the space that they are in as a form of escapism and entertainment. There could also be healing benefits to changing a patient’s scenery often.
Patient can access their food menu in MR and make selections through the interface.
Patients could play virtual sports that encourage movement in their room, such as miniature golf, tennis or bowling.
Often, patients may have difficulty locating their call bell if the device has fallen off the bed or they are not in bed. With MR, I imagine the patient is able to activate their call bell from anywhere they are in the room through drawing a symbol such as an “X”, “cross” or “circle”.
Captionless Image
Clinician opportunities
In an emergency situation, the clinician is able to activate emergency protocol by drawing a symbol, which will connect them quickly to the operator. They would also be able to select which protocol they would like to activate through a selection interface.
Another input method to the scenario above, is having a MR button that hovers behind or high up above the clinician that can be activated by gazing at it and then calling the protocol.
Once they have activated the protocol, a visual and audio guide to the emergency protocol would begin, providing a visual aid to what they need to do and also allowing notes to be taken where the patient is. The system displays checkboxes for the activities that it requires filling for clinician to check off, with a virtual clock recording time when notes are taken.
Clinician can retrieve event information on patient to communicate to to another clinician during shift change or physician visit, either in the form of visual chart, imagery or animations recorded during their shift.
Patient and clinician shared opportunities
When discussing patient problems, clinicians can pull up a 3D human form (or even annotate the patient themselves), on which they can record issues described by the patient on the form. This can be a recording over time, which can be played back via animation for other clinicians or for the patient themselves. This could potentially be something provided to the patient with their release form.
Captionless Image
Similarly, patients that need to keep track of their problems can use 3D body forms to annotate, using shapes, color, likert scale annotation etc. This hopes to provide a more expressive way of communicating issues between patient and clinician.
Captionless Image
Clinician can help patient set up a digital range of motion trainer in the form of an animated character or a moving target to follow that guides them through physical exercises that can be difficult or tedious to keep up with.
Open ended questions
Do you have any feedback on scenarios listed above?
Your answer
Can you list any other scenarios where you think that MR could add to the patient-clinician experience?
Your answer
Can you list any other scenarios where you think that freehand drawing gestures could add to the patient-clinician experience?
Your answer
What are your current problems with EMR? Where are the key opportunities for improvement?
Your answer
What are your current pain points with emergency protocols?
Your answer
What is your profession? *
Your answer
Are you responding as a patient or clinician?
If patient, what type of health problems were you addressing at the hospital? Did this require hospital stay?
Your answer
Have you had any other previous roles in the medical or health industry? *
Your answer
Have you performed an emergency protocol?
Your answer
Never submit passwords through Google Forms.
This content is neither created nor endorsed by Google. Report Abuse - Terms of Service