Computer scientist Misha Sra uses AI to make motor skills training more accessible

Designing for motor tasks isn’t easy, requiring quick adaptability to movement. That’s where extended reality (XR) technologies can play a role, enabling motor skill learners to maintain mobility during fitness training, physical therapy or even dancing. 

To design head-worn XR devices that facilitate motor skills training, computer scientist Misha Sra, an assistant professor at UC Santa Barbara, has received a National Science Foundation (NSF) Early CAREER Award. Sra’s interactive design engages embodied extended reality agents (ERAs) to replicate the effectiveness of learning with a human expert. The award includes $606,000 in funding over five years.

The project will provide foundational knowledge about interactions between humans and AI agents that can replicate the verbal and non-verbal behaviors of human expert trainers in the context of motor-skill learning. 

“This project focuses on harnessing AI and XR to enhance human physical abilities,” Sra explained. “It is aimed at bringing AI technology out from two-dimensional screens into the real world, offering much needed support and guidance for learning and recovering motor skills.”

Potential benefits of Sra’s research include increased access to motor-skill learning by making it more affordable at scale. Her research engages with a spectrum of technologies associated with XR, combining physical and virtual worlds to create immersive and interactive experiences. 

A key component of Sra’s system is multimodal guidance and feedback through visual, auditory and haptic channels that provide explanations and demonstrations. The ERAs are designed to recognize and respond to a user’s physical actions in or near real time and to appear and act human-like, including body language like nodding and other facial expressions.  

Sra considers a possible application such as assisting an individual recovering from an injury who needs physical therapy, but because of the distance to the closest physical therapist they only receive care every few months.  

“The person might not be sure what to do between appointments to help speed up their own recovery,” she said, “or, if they follow (often printed) instructions, they may not be sure if what they are doing is correct or will, perhaps, cause injury. Lacking the ability to track their own progress may also lead to a loss of motivation. 

“Now, imagine an AI system that can provide guidance and feedback in real time,” she continued. “Wouldn’t that be great? You’d be able to speed up your recovery considerably by making the most of your time between appointments with your clinician/therapist or knowing how to continue once your therapy is officially over but you are far from fully recovered.”

Sra’s research integrates AI with extended reality (XR), which refers to a spectrum of technologies that combine the physical and virtual worlds to create immersive and interactive experiences. XR includes virtual reality (VR), which creates a fully simulated digital environment; augmented reality (AR), which overlays digital content onto the real world; and mixed reality (MR), which blends virtual and real elements, allowing users to interact with both. AI, on the other hand, enables machines or computer programs to learn, reason and make decisions based on data and algorithms. 

Ceding that many knowledge gaps and challenges need to be addressed to make the real time feedback system a reality, Sra said she hopes that her lab can “address some of them in our work and address others through collaborations with researchers in areas such as computer vision, natural language processing and cognitive science. Continuous advancements in these areas will contribute to the development of AI agents that can be used in the real world with far-reaching societal benefits.”