1 of 1

Rachel Naidich

Stanford University Department of Computer Science

rnaidich@stanford.edu

  • Stroke is currently the leading cause of disability in the United States. A major cause of post-stroke disability is Hemiparesis (arm and/or leg weakness), which affects up to 66% of stroke survivors, is a major cause of post-stroke disability. Hemiparesis patients are typically given rehabilitation exercises that they should perform at home to improve functional outcomes.
  • Rehabilitation is limited by a lack of resources to offer long-term daily intensive therapy, diminishing motivation over time to continue rehabilitation therapy, and significant time cost for outpatient services.
  • Stroke rehabilitation exercises include a variety of different hand exercises. An augmented reality application provides the opportunity to improve functional outcomes by tracking hand movement, providing both auditory and visual feedback, and motivating users with a fun, immersive experience

Fig.2: A A screenshot from the HoloLens of the beginning of the hand exercise scene. When the user first starts the scene, the first set of instructions appears for the user to complete the gravity eliminated finger flexion exercise. The user must place the side of their hand on a flat surface, and in order for the Hololens device to detect the hand, the user must look at their hand. B A screenshot from the HoloLens of a closed fist. After the hololens detects the hand, the hand exercise game begins and instructs the user to bend their fingers. The screenshot shows that when the user bends their fingers all the way into a fist, the text shows a high flexion percentage. This can be used to motivate the user to bend as much as possible. Cubes begin to appear next to the hand that move towards the hand, and users must flatten their fingers back out to smash the cube. C A screenshot from the HoloLens of a flat hand. When the user flattens out their hand, the text shows a very low flexion percentage. If the user successfully touches a cube, then the cube explodes and disappears. A repetition only counts and gets displayed in the text if the user successfully smashes the cube. This visual makes this repetitive exercise more enjoyable and exciting.

Background

Fig.1: The software uses the MRTK hand tracking capabilities to overlay a cube objects over each joint of the user’s hand. This allows the user to see that their hand has been detected by the device and enables the user to physically interact with virtual objects that appear.

​

Auditory Feedback with Holographic Augmented Reality Displays to Improve Stroke Rehabilitation Outcomes

Previous Work

I developed an approach using the HoloLens to track body movements, track hand movements, present feedback, and gamify rehabilitation movements. There are audio and visual cues when a patient completes their required exercises correctly.

A

B

C

Current Gameplay

Design Rationale

Research

Evaluation

Fig.1: A screenshot from the HoloLens in the middle of the game. The red slime character moves up when users bend their hand, and it moves down when users straighten their hand. Notes that users don’t hit on time turn green and move past without playing the note out loud.

​

The current version of the gameplay incorporates music to provide auditory feedback as users conduct their required rehabilitation movements. The software converts a MIDI file of “Clair de Lune” into a stream of capsule shaped objects moving horizontally in front of the players. These capsules represent the notes in the melody of the song. Users can bend and straighten their hand to control a red slime character. The goal is to try to match the position of the character with the notes at the correct time.

Fig.2: A screenshot from the HoloLens of the red slime character hitting a note. When a user hits the note on time, the capsule explodes and the note is played out loud. The accuracy shown in the accuracy bar depends on the number of notes successfully hit.

​

Fig.3: A screenshot from the HoloLens of the end of the game. Users are shown their final accuracy, which they can use to track their progress over time.

​

Auditory feedback can improve motor actions

  • One study showed that learning with auditory feedback can lead to improved movement quality and movement timing 2

Auditory feedback can help stroke patients

  • A study that tested music-based augmented auditory feedback on stroke patients found that it had the potential to enhance reaching ability 3
  • A study that tested gait training with auditory feedback on hemiparetic stroke patients found that auditory feedback led to improved trunk control and muscle activation 6

Auditory feedback is most effective in combination with visual feedback

  • A scientific review found that multimodal feedback (audio + visual) seemed to be the most efficient way to offer feedback in learning motor tasks in both healthy and diseased populations 5
  • One study found that continuous task-related audio feedback provided in addition to visual feedback can be beneficial for upper limb exercises and significantly improve performance 1
  • Research shows that human brain plasticity allows for musical training to potentially help people with neurological impairments. Practicing motor actions in combination with specific sounds and visual patterns can strengthen connections between auditory and motor regions 4
  1. Rosati, G., Oscari, F., Spagnol, S., Avanzini, F., & Masiero, S. (2012). Effect of task-related continuous auditory feedback during learning of tracking motion exercises. Journal of neuroengineering and rehabilitation, 9(1), 1-13.
  2. van Vugt, F. T., & Tillmann, B. (2015). Auditory feedback in error-based learning of motor regularity. Brain research, 1606, 54-67.
  3. Chen, J. L., Fujii, S., & Schlaug, G. (2016). The use of augmented auditory feedback to improve arm reaching in stroke: a case series. Disability and rehabilitation, 38(11), 1115-1124.
  4. Schlaug, G. (2015). Musicians and music making as a model for the study of brain plasticity. Progress in brain research, 217, 37-55.
  5. Moinuddin, A., Goel, A., & Sethi, Y. (2021). The Role of Augmented Feedback on Motor Learning: A Systematic Review. Cureus, 13(11).
  6. Jung, K. S., Bang, H., In, T. S., & Cho, H. Y. (2020). Gait training with auditory feedback improves trunk control, muscle activation and dynamic balance in patients with hemiparetic stroke: a randomized controlled pilot study. Journal of Back and Musculoskeletal Rehabilitation, 33(1), 1-6.

Testing

  • Experimental group would play the game throughout rehabilitation period
  • Control group would complete traditional hand-exercise rehabilitation for an equal amount of time
  • Rehabilitation compliance will be tracked for both groups
  • Hand movement performance would be compared using the Motor Assessment Scale (MAS)

Hypothesis

  • Experimental group will demonstrate more compliance with prescribed rehabilitation upper extremity exercises compared to control group
  • Experimental group will score higher compared to control group on the hand movement section of the MAS after equal amount of rehabilitation time
  • Experimental group will experience faster increase in scores on the hand movement section of the MAS compared to control group
  • Immersive augmented reality game will increase rehabilitation compliance by engaging and motivating users
  • Auditory feedback in combination with visual feedback will promote improved rehabilitation progress
  • Choosing well known music will strengthen associations between motor actions and auditory feedback as well as make the experience more engaging
  • Compared to traditional rehabilitation, which only involves completing repetitions without any rhythm, mechanics that require users to have control over full range of motion and timing of their movements will be more effective in helping users regain motor ability for practical purposes (ex. pouring a glass of wine)

References