3 of 20

BACKGROUND

Telemedicine

Dementia

Problem Statement

The intersection of technology and medicine has been a field of study that is not greatly explored but has great potential. Due to the COVID-19 outbreak, the benefits of technology have been highlighted, with telemedicine becoming more widely used across the world in order to limit human contact. Telemedicine specifically refers to the distribution of health-related services in terms of a virtual format. The doctor is typically not physically present with the patient or caregiver and all clinical interventions are done remotely. Benefits for telemedicine providers are that they typically enjoy higher revenue, better patient follow through, fewer missed appointments and cancellations, and improved office efficiency. As for patients, they incur lower travel expenses, less time used up for waiting or traveling, and most prominently no exposure to other potentially contagious patients. ^[1]Given these advantages, telemedicine is a safer and stronger alternative to traditional in-person visits and may become the norm in the future.

Our team wanted to focus on people with dementia because we shared a common interest in learning about neurological diseases and found that people with dementia are one of the most affected groups during COVID-19. According to the Alzheimer’s Association, dementia is a general term for loss of memory, language, problem solving and other thinking abilities that are severe enough to interfere with daily life. Lewy body, vascular, frontotemporal dementia, Alzheimer’s, Parkinson’s and Huntington’s disease are associated with dementia. ^[2] Alzheimer’s Disease is currently ranked the 6th leading cause of death according to the National Institute of Aging. ^[3] Without a cure, dementia remains a prominent medical condition that millions are battling with, but with proper treatment and therapy, the condition can be managed.

While telemedicine is essential during a pandemic, this technology fails to reach rural areas, address social isolation impact on people with dementia, resolve medication management issues, and ensure efficient standardized communication. Miscommunication accounts for 70% of medical errors which can often prove to be detrimental or fatal. Issues like non-adherence to prescriptions and overdose are results of errors from poor medication management. In addition, caregivers often are often stressed and do not receive the adequate help they need especially during COVID-19. In most cases, the responsibilities of the caregiver force them to sacrifice their own wellbeing in order to take care of the person with dementia. There are various web applications for telemedicine that can be used to fix these problems; however, they lack the standardization of payment, scheduling, and communication between the different stakeholders (person with dementia, caregiver, doctor). Various applications need to be used to have a successful appointment, specifically when receiving therapy and ensuring an informed PWD.

PWD = Person/People with Dementia

4 of 20

METHODS

HYPOTHESIS

The care for early-stage people with dementia (PWD) around the world can be simulated as close as possible to an in-person setting through the use of a versatile teleneurology mobile application capable of facilitating remote cognitive stimulation therapy, easy-to-use social interaction supported by means of a virtual assistant, and subsequent emotion monitoring of the PWD through artificial intelligence (AI), all of which are rooted in an efficient standardized platform.

(2) Cognitive Stimulation Plan - This feature allows therapies to be conducted virtually when in-person interventions are not possible especially during COVID-19. Specialized therapy sessions may be conducted with a therapist one-to-one while common therapies are done in groups. Such interventions will help stimulate cognitive function for PWD and relieve stress for the caregiver. Our research helped identify specific activities that keep PWD engaged and conditions them to retain information and memories.

(1) NeuroComms - Research has shown that in teleneurology, and in telehealth in general, the lack of follow ups and future consultations have been the major bottlenecks to promote the acceptance of such platforms. We researched current communication platforms that would be able to manage various aspects including scheduling appointments and payments. We also spoke to medical professionals and experienced caregivers about the technical challenges they may face when providing care through telemedicine. Our research helped identify the improvements that can be made in the standardization aspect of telehealth.

(3) Behavior Analysis - We conducted research on the different methodologies that are applied to train machine learning models and usage of natural language processing (NLP) to analyze and monitor the PWD’s emotions. Additionally, researching the field of effective computing to exploit facial recognition technology to track emotions and using that data to communicate the remittent emotional changes of the PWD to the caregiver and medical professional. We also plan to implement a virtual assistant that would respond according to the PWD's emotional state.

(4) Social Interaction - We researched the benefits of social interaction for both the caregivers and people with dementia. While conversing with several professionals and experienced caregivers, we were able to understand how the pandemic significantly reduced social interaction among PWD and caregivers. This lack of interaction greatly influences the PWD’s condition and even the caregiver as well. We explored ways to initiate a convenient method to promote social interaction remotely for both stakeholders involved.

PWD = Person/People with Dementia

5 of 20

FEEDBACK/RESULTS

Interview 1: Dr. Terra Hussar (Neurologist)

During our interview, Dr. Hussar specifically highlighted the behavioral changes such as sudden anxiety or aggression of PWD and methods of providing affirmation. PWD want to remain independent individuals and not become a burden to anyone else; consequently, they may often show aggression or refusal to receive care. Ways to manage this include words of encouragement or pleasant walks around the neighborhood. Additionally, Dr. Hussar emphasized the vitality and effectiveness of social exchanges for people living with dementia. Their behavior can be calmed and motivation can be raised with any type of social interaction ranging from watching TV together to stretching together. To maintain this during the current pandemic, she suggested that telemedicine in a large setting (perhaps in a dementia center) could possibly facilitate this much-needed interaction while following guidelines. Finally, we also learned about the practical challenges associated with telemedicine interventions for severe cases; this made us more aware of which population or severity of people living with dementia we should focus on.

Interview 2: Julie Fleming (Founder and CEO at Purple Sherpa)

Mrs. Fleming was the caregiver for her father who suffered from Alzheimer’s Disease and passed away in 2017. She now serves as the CEO of the Purple Sherpa, which is a non-profit organization dedicated to supporting the caregiver of a loved one going through dementia. She highlighted the stress that the caregivers go through and how in many scenarios, caregivers lose sight of themselves and end up making so many sacrifices for their loved ones. We brought up the idea of a social interface of family and friends for the person living with dementia to interact with; however, she suggested that it would likely be confusing to them. She did suggest the possibility of “a way to use AI to recognize those micro-expressions.”

Interview 3: Kevin Jameson (Founder and CEO at the Dementia Society)

Mr. Jameson was a caregiver to his wife who lived with dementia and now is retired but serves as the CEO of the 501(c)3 non-profit organization called Dementia Society. During our interview, Mr. Jameson emphasized the importance of streamlining communication between caregivers, doctors, and people living with dementia. Additionally, he mentioned that social interaction, specifically the ability to see one’s face, is vital to the wellbeing of the person living with dementia.

Interview 4: Vic Steele (Caregiver)

Mr. Steele emphasized the importance of communication between the caregiver and the doctor, because there are multiple actions that non-medical caregivers must do that they are not necessarily trained for. He mentioned that having a portal where information can easily be shared with the caregivers would be very beneficial. He greatly supported the idea of emotional analysis of the caregiver in addition to the individual living with dementia. Additionally, Mr. Steele was approved of the idea of having a virtual social AI, similar to Amazon’s Alexa or Google Home, that the individual with dementia can communicate with. He also stated that group social sessions are very beneficial to the wellbeing of individuals living with dementia, as long as they’re not classified as “therapy” sessions carrying a negative social connotation.

Interview 5: Charlene Durante (Caregiver)

Mrs. Durante was a caregiver for her husband who lived with dementia. During our interview, the primary focus of the discussion was the social activity of both the caregiver and the person living with dementia. Like other caregivers, she underlined the stressful nature of the job of a caregiver and the necessity for a support system. She recommended exploring the possibility of adding networks or a way for caregivers to connect with other caregivers who have experienced similar challenges allowing the platform to help them cope with stress and mitigate isolation. This network could also be used to help caregivers during times of need by allowing them to find others who could take care of the person with dementia for a given time period. This network would not only alleviate stress for the caregiver when they are unavailable but also allow them to feel more connected to society. Lastly, she also proposed to incorporate music into our app.

Interview 6: Dr. Michael Ross (Chief Medical Officer at Cureatr)

Cureatr is a telemedicine company focusing on medication management and the problems that arise in healthcare when patients transition from different hospitals and providers. The platform originally helped doctors become “aware when a patient was going in and out of the emergency room or in and out of the hospital, and then enabled them through secure text messaging background to communicate with each other.” We discussed the problems that arise when a patient is on several different medications and consulting multiple doctors. The main function of the app is programming to ensure that these different medications that a single person is on doesn’t result in hazardous compound combination effects which can lead to drug toxicity or death. Using datasets like HIE, Cureatr is able to visualize the details of patients’ different medications and helps physicians and pharmacists see what’s going on beyond their realm of involvement.

Interview 7: Dr. Peter Alperin (Vice President of Product at Doximity)

Dr. Alperin is a physician who has been a part of Doximity, the largest online healthcare network, since its beginning. In our interview Dr. Alperin emphasized the importance of simplicity and the ease of use of the application. He emphasized on the importance of efficacy of telemedicine, because it allows an individual to access a medical professional efficiently. He also brought up some issues surrounding telemedicine such as the inability for a physician to perform physical exams and the imposed sense of disconnection that comes with meeting virtual as opposed to physically.

Our main takeaways and Implementation:

From the interviews, we learned about the struggles faced by the caregivers and the necessity to provide support. This motivated us to create a discussion forum for caregivers to talk to one another and support each other. As for the telemedicine companies, we derived a lot of inspiration from them, nad learned about the importance of medication management and a streamlined and easy communication process. Our interview with mrs. Fleming was what sparked the idea for having a virtual assistant (Caren) that could also analyze long term behaviors and provide data to the caregivers and doctors.

PWD = Person/People with Dementia

6 of 20

SOLUTION OVERVIEW

Target: people with low to moderate forms of dementia, and caregivers with experience at all levels

Our solution consists of four primary features: NeuroComms, virtual cognitive therapy, social interaction, and behavioral analysis, all of which will be available on one platform: our mobile app, iMed. To accommodate the specific needs of our stakeholders, we created three different dashboards, one for the doctor, caregiver, and person with dementia. The NeuroComms aspect of our solution focuses on standardizing communication and includes medication management, faxing, dialing through a HIPAA-secure number, and the SBAR method (situation, background, assessment, background). The social interaction aspect of our solution uses a public discussion forum to improve support for caregivers, virtual social therapy to increase interaction for users, and a virtual assistant named Caren which converses with the user to keep them company. Caren also aids in the behavioral analysis feature of our solution. Through facial and voice recognition, Caren generates a summary report of mood swings and other emotional changes of the user which can then be used as data for the doctor to to implement an individualized care for the PWD. The final aspect of our solution is virtual cognitive therapy, which specifically targets PWD and aims at simulating in-person therapy in virtual means. This feature will both improve cognitive function for people with dementia and save stress from the caregiver.

Three Dashboards

Person with dementia

Caregiver

Doctor

7 of 20

COLOR SCHEME FOR MOBILE APP

A login component was added to the app in order to ensure security for all who use the app’s interface and to improve the overall user experience.

Caregiver

The caregiver would create an account using their email, phone number, and created password. Within their account they can include additional information like their health insurance number, address, etc. From the caregiver dashboard a new PWD profile can be created for their care recipient. When creating a PWD profile, the caregiver will have to input the PWD’s personal information (name, age, etc.).

Person with Dementia

The goal for the PWD login is for it to be as easy to use as possible. After the caregiver creates a PWD profile from the caregiver dashboard, the caregiver will be able to send a link to the PWD that will redirect them to the PWD dashboard. For security purposes, the caregiver also will have to approve the initial login via the caregiver dashboard. Additionally, the login information will be stored so that consecutive logins from the same device do not need constant validation from the caregiver’s dashboard (however the caregiver will be notified every time the PWD logs on via the app).

Doctor

A doctor would login to the app with their license ID number. For extra security, the doctor will submit their diploma so that it can be verified. Only once it is verified the doctor can use the app.

We chose blue as our primary color scheme for the application because research has shown that blue has a “restful and calming effect,” even helping to reduce blood pressure. Pastel colors especially have been shown to have a significant calming effect, allowing people to feel relaxed and comfortable ^[5]. Often times, many senior centers and other community spaces have a specific color theme that was implemented thoughtfully, and we wanted to translate that knowledge into our application so that our users feel relaxed too.^[6]

PWD = Person/People with Dementia

8 of 20

NEUROCOMMS

STREAMLINED COMMUNICATION

DIALER

FAX

Miscommunication is regularly the culprit of tragic medical mistakes, so efficient and accurate communication is important in telemedicine. Our solution is a form of communication that standardizes and streamlines information between the caregiver, doctor, and PWD. The SBAR method [7] is a popular method utilized in hospitals for structured communication and would be implemented into our NeuroComms intervention. SBAR is an acronym for situation (what’s the problem?), background (PWD’s historical medical conditions), assessment (doctor’s observations of the PWD), and recommendation (what does the doctor diagnose and what is the doctor’s next steps for treatment?). Because a PWD does not always have the same doctor, these four topics must be communicated between the doctors that may meet with the same PWD at different times in order to ensure there are no communication gaps when switching doctors. All doctors on the same case will be able to access each other’s SBARs on the NeuroComms intervention to view what others think about the PWD’s diagnosis. PWD will also have the SBAR feature where they can fill out a SBAR of their own, except their SBAR (called SBAR3) stands for story (what happened to them?), background, assessment (PWD asks questions), review (R1: review the doctor’s recommendation, R2: is the provider’s recommendation Right for them?, R3: repeat the treatment plan). This SBAR3 method^[8] has proven effective in in-person healthcare, allowing PWD to be heard and fully understand the plan of action for their treatment.

The primary function of the dialer is to protect the privacy of the stakeholders. The dialer for audio and video call, private messaging, faxing, and SBAR will all be on the caregiver and doctor dashboard. As for the PWD’s dashboard, it will be simpler in order to avoid confusion. Because of HIPAA laws and other privacy concerns, when communicating with other users for calling, the user’s number will be hidden and not show up on the caller ID. It would be similar to the function of *67 when dialing on the phone app. Essentially, your cell number will be kept private and you designate the call back number. What differentiates this dialer from the normal phone app is that stakeholders will be able to make and receive calls with convenience and without giving out their personal number. Also, to increase the ease of use, when joining appointments via our app, the user will be sent a link that they can join the meeting with in one tap. This link could be texted or emailed to them and after they click on it, it would take them to the video call page on our app.

Faxing has also been a hassle in traditional in-person visits, taking a very long time to send physical faxes of a PWD’s electronic health records when transitioning to different hospitals. With the iMed app, PWD, doctors, and caregivers will be able to send secure faxes virtually. Being able to quickly fax saves valuable time. This feature will also be HIPAA-secure.

PWD = Person/People with Dementia

9 of 20

NEUROCOMMS

MEDICATION MANAGEMENT

Keeping up with your prescriptions can be a daunting task, especially so for people with dementia. To make the process easier for both caregivers and PWD, the doctor and pharmacy will be connected to the PWD profile and all communication happens on a single Neurocomms platform. The doctor is responsible for updating the PWD’s log of medications, specifically when to take them (time of day, how often, etc.). After the log is filled out and completed by the doctor, NeuroComms will remind the PWD/caregiver by sending a notification every time there is an upcoming medication that needs to be taken. It can also be used to alert the user when the medications are ready for pickup at the pharmacy. This notification system will be similar to that of an online calendar, reminding the user a couple minutes or at the exact time of the event. This intervention can also be used for scheduling appointments and follow-ups. This feature will allow for both faster communication between all stakeholders. This will also allow for doctors to make sure PWD are taking their medications, as medication management is often a problem. If there is a case where the PWD didn’t pick up their medications, the doctor can reach out to the PWD to figure out what the problem is, whether the reason for the nonadherence is because of side effects, financial problems, scheduling, etc. Another feature of the medication management page is that the doctors will be able to see when there’s a duplication of medications. Often times when a PWD is seeing several doctors, a problem called inoperability occurs. This can be dangerous if several different medications that a person is on may end up being a hazardous drug combination resulting in severe side effects. If the doctor is able to see that these duplication warnings and hazardous drug combinations, then preventable medical deaths and errors can be easily avoided.

Frame 1. Duplication Warning: Alerts the doctor that the PWD is on two meds that are potentially dangerous together

Frame 2. Main Med Management Page: General overview of all the medications that a particular PWD is on

Frame 3. Medication Details: Lipitor’s image and other specifics regarding intake, pharmacy, prescriber, and supply.

Frames 1-3 are the doctor view

PWD = Person/People with Dementia

10 of 20

SOCIAL THERAPY

IMPORTANCE OF SOCIAL INTERACTION - PEOPLE W/ DEMENTIA

Due to the current COVID-19 impact, a major comfort factor of PWDs’ lives has been reduced significantly: social interaction. Prior to the pandemic, PWD in care centers or with personal caregivers would deeply enjoy meeting and conversing with their friends. Unfortunately, not being able to do so due to the safety guidelines has interfered with the means of getting PWD to think, converse, and exercise their cognitive functions.

Typically, PWD are able to learn from one another, perform activities with each other, and talk about past memories. Using personal pictures, old memories, music, news, and old TV shows can initiate conversations. PWD hold on to their cognitive abilities. The lack of being able to do so in-person, however, results in feelings of anxiety and isolation for these PWD, both of which alter their emotional and behavioral status.

IMPORTANCE OF SOCIAL INTERACTION - CAREGIVERS

Caregivers of PWD undertake the responsibility of ensuring (full-time) that their loved one or PWD can gradually recover and live comfortably. However, the consequence of devoting so much and effort is that caregivers can feel stressed or even burden when taking care of their loved one. Therefore, caregivers tend to isolate themselves beyond the lack of social interaction given the COVID-19 pandemic. The caregivers’ feelings of helplessness, depression, and exhaustion eventually^[17] leads to ineffective care and uncomfort in both the caregivers’ and PWDs’ lives.

Through several research studies relating to telehealth, caregivers highly ^[17] emphasized the need for convenient social interaction and discussion. Additionally, caregivers of these studies provided more insight on the benefits of this; they call for a means to exchange caring tips/techniques or feedback along with a means to simply converse and let off stress. For several of these caregivers, understanding and responding to the behavior changes in their PWD was difficult; having a forum to ask questions and gain insight is valuable for all caregivers.

Finally, a major contributor to the caregivers’ stress in their day-to-day life is all the tasks they must balance.^[17] Caregivers of various studies expressed the need for an easy way to manage both their personal life errands and care-related tasks. Losing track of tasks in either one of these categories can be detrimental for the caregivers’ and PWDs’ well beings.

^[18]33% of caregivers in the U.S. face difficulties maintaining their own health - Alzheimers.net

^[17]70% of caregivers between several telehealth studies call for more remote social interaction and task management

PWD = Person/People with Dementia

11 of 20

DASHBOARDS PT. 1

[social therapy]

Discussion Forum

The Caregiver Dashboard of the iMed app consists of an easy-to-use and reliable discussion forum for caregivers to use at will. Upon clicking on the forum in their dashboards, caregivers will be prompted to select which conversation channel they would like to post in:

General: In this channel of the forum, any caregiver with the app can engage in a conversation across a broad range of topics, whether it be care-related or life-related.

Questions & Answers: The Q&A platform would potentially provide answers to common concerns or queries that caregivers looking after PWD might have. This channel is to be used if a caregiver wants to learn about how to detect and/or respond to these emotional/behavioral changes.

Tips/Techniques: Caregivers of diverse backgrounds and experience levels will be present in the forum. The platform would provide an opportunity to share caring techniques or feedback.

Relax: More often than not, caregivers are stressed as they are tasked with managing countless things at once and therefore, need an appropriate outlet to relax and let out any feelings to other caregivers who may share similar opinions. Sharing those feelings will help clear their minds and relieve stress.

Speak to CAREN: Caregivers, similar to the PWD, will also be given the chance to speak with CAREN, the virtual assistance. But, this will only be for conversing purposes and not for emotional analysis, which is seen in the PWD dashboard.

It is important to note that in the ‘Questions & Answers’ channel, there will be iMed Care Team Navigators (CTNs) and/or doctors that are designated to answering questions and ensuring no caregiver is left without a response. The reliability of receiving a response and the very straight-forward purpose and categories of this messaging platform differentiates it from other conventional platforms.

Task Control Center

The Caregiver Dashboard is also designed with a efficient task managing intervention. In order to combat the caregivers’ difficulty of balancing both care-related and personal tasks, our control center allows caregivers to store and manage both categories of responsibilities in one localized area of the user interface. It has been observed that the overwhelming amount of tasks a caregiver may have on any given day is likely to lead to further stress and isolation. The Task Control Center serves as a feature to prevent caregivers from forgetting tasks and helps to increase the amount of time they have to socially interact with others. Additionally, all the tasks and the respective timings will be linked on the app’s master calendar.

For dementia care-related tasks, the caregiver will actually first be able to peruse through preset/common tasks such as picking up medications or scheduling a virtual meeting with the doctor. This makes adding regular tasks quite labor-free when those responsibilities may be very regular. The caregiver will also have the option to add different care-related to-do items and timings along with them.

For personal life related tasks, the caregiver will have have the freedom to add any errands for their daily life on the same page as the dementia care-related tasks. Throughout their day, they will be able to check off any items for better management and also add small notes if necessary. The ability to view both categories of tasks in one area saves time for the caregiver to self-care or interact with others. This also reduces the need to use a third-party app as it is in one application.

CAREGIVER DASHBOARD:

Upon setting up the iMed app as a caregiver, the user will be welcomed with a home page. The following two features out of the many are social therapy based.

PWD = Person/People with Dementia

12 of 20

DASHBOARDS PT. 2

[social therapy]

PERSON WITH DEMENTIA DASHBOARD:

Upon setting up the iMed app as a PWD, the user will be brought to the home page. This dashboard is designed to be extremely user-friendly and easy to navigate for the PWD. Its features allow for the PWD to socialize with one another and help stimulate cognitive function

One of the most important factors in managing PWD is getting them to think and practice their ability to retrieve memories, and do away with the feeling of isolation. Since in-person interaction is not feasible during these times, iMed’s intervention provides an extremely user-friendly mode of generating conversation topics.

The PWD dashboard consists of only five distinct features:

1- Calendar: Automatically updates as their respective caregiver assigns tasks or sets appointments

3- Personal Photo Gallery: Studies have shown that rekindling old memories through old photographs or any other digital media helps dementia patients retain more information. Our team’s interview with Dr. Hussar also informed us about a common strategy of using images of loved ones to stimulate PWD’s emotions and cognitive activity. This photo gallery will only have these types of pictures for the PWD to recall a story to tell their caregiver or virtual assistant Caren.

2- Music Library: A music library feature that contains old track records from the PWD’s childhood or early adulthood can assist in stimulating the PWD’s brain and recalling old memories.

4- News Articles: In order to ensure that PWD are still in touch with the outside world, a selection of regularly updated news articles are at their disposal to read. These resources may act as conversation starters between their caregiver or other PWD in group therapy sessions.

5- CAREN: If a caregiver is not present with the PWD during a given time, the iMed app can utilize the virtual assistant Caren to prompt the PWD to view one of the three features (music, photos and news). Caren will follow up with conversation starters and questions as well.

PWD = Person/People with Dementia

13 of 20

CAREN: VIRTUAL SOCIAL ASSISTANT

Exploiting Natural Language Processing (NLP) and Facial Recognition (FR), Caren can analyze both the language and the facial expressions of the user. The audio and visual input will be via a microphone (from a mobile device, laptop, etc.) and a webcam (phone camera, laptop camera, etc.) respectively.
Caren aims to utilize a Convolutional Neural Network (CNN) for classifying emotions in the inputted data and a Recurrent Neural Network (RNN) for NLP to understand what a user is saying when they talk to Caren. A CNN is effective for classifying images or sounds by extracting certain features from them, such as a smile, eyebrow raise, etc. A RNN, however, is more effective in analyzing temporal data, such as sequential information including sentences, which is essential for natural language processing and understanding the meaning of sentences.
The emotional data would then be visually represented to the caregiver via the iMed dashboard in an effective way that will illustrate the fluctuations of emotions by the PWD. In addition, a detailed report that can show the specific instances where the user expressed a certain emotion (ex. a spike frequency of joyful facial expressions when playing music).

PURPOSE

Social interaction is vital for the cognitive well-being of people with dementia; however, due to the COVID-19 pandemic, social interaction is severely undermined.
Since individuals living with dementia often are reticent in openly sharing their emotional feelings, being able to visually analyze their emotional fluctuations allows for the caregiver to better understand the person’s emotional changes and triggers ^[21].
Our solution, Caren, aims to address this issue. Caren is a virtual assistant (similar to a Google Assistant or an Alexa) that can be found on the patient dashboard and can interact with the user when prompted by a command or the click of a button.
The primary purpose of Caren is to aid the social needs of the user by conversing with them and performing various functions (such as presenting a slideshow of a photo album).
Additionally, Caren’s is also designed to analyze the emotions of the PWD when in use and subsequently display that data to caregivers and doctors, granting them a more in-depth analysis of the fluctuations of emotions by the person living with dementia, which may also be clinically relevant to monitor the progression of the disease.

PWD = Person/People with Dementia

DESIGN

14 of 20

CAREN: VIRTUAL SOCIAL ASSISTANT

Emotions exhibited from vocal analysis can be evaluated by the loudness, pitch of voice, and lexical analysis (words referring to things rather than grammatical purposes). Utilization of Mel Frequency Cepstral (MFC) as automatic speech recognition algorithms helps represent audio visually in a comparable way to how humans perceive sound^[8]. MFCs can turn audio data into a visual format, thus shifting the approach of emotional recognition from audio data into image classification.

Emotions from facial expression will be analyzed for a multitude of features. The first is engagement, which is measured by the amount of facial muscle activation (such as a brow raise, brow furrow, chin raise, etc.) that illustrate expressiveness^[9]. Secondly is valence, which is the measurement of the positive or negative nature of the user’s emotion. An example of a facial gesture showing positive valence would be a smile, and an example of a negative valence would be a brow furrow^[16]. Finally, the valence can be synthesized to “tell” the machine learning model to what degree of magnitude is a certain emotion expressed. For example, a smile would increase the likelihood of the emotion joy and decrease the likelihood of the emotion anger.

EMOTIONAL ANALYSIS

^[16]Valence Metrics

^[20]Heatmap of Mel Frequency Cepstral Example of Happy Male Voice

^[16]Emotional Classification Derived from Valence

Click on mindmap to expand

15 of 20

VIRTUAL COGNITIVE THERAPY

Prior to the COVID-19 pandemic, people living with dementia are often prescribed to cognitive therapy. According to the American Psychological Association (APA), cognitive behavioral therapy (CBT) is a form of psychological treatment that is effective against various mental health problems such as depression and eating disorders^[12]. For people living with dementia, it is important they receive cognitive therapy, as this can better manage their condition^[13]. These types of treatments are meant to stimulate cognitive function. However, COVID-19 prevents in-person cognitive treatments to take place, which causes additional stress to the PWD and caregiver.

To resolve this prevalent issue, we propose a virtual cognitive therapy plan. Our research suggests that the most common therapies that could be efficiently implemented into a telemedicine app are speech therapy, music therapy, virtual environments, and cognitive stimulating games, wh as well as providing information to the caregiver by an occupational therapy (OT) specialist, as most of the times, caregivers of people with dementia may not know the best route to take in taking care of their loved one ^[14][15].

Overview Video of Virtual Cognitive Therapy & prototypes

16 of 20

Person with Dementia Dashboard

Caregiver Dashboard

Doctor Dashboard

PROTOTYPE DEMONSTRATION

https://www.figma.com/proto/Rp9S1GaZTBzqJRD8GWkkuK/CAREGIVER-DASHBOARD?node-id=1%3A9&scaling=min-zoom

https://www.figma.com/proto/RnQwhCtRTU8GZ0ldMapnTH/Patient-Dashboard-iMed?node-id=273%3A161&scaling=min-zoom

https://www.figma.com/proto/UOwiV1Slt9x8S3SGDrR590/Doctor-Dashboard-iMed?node-id=243%3A2&scaling=min-zoom

If the interactive buttons don’t work in the prototype, try going full screen or removing toolbar and footer (go to options on the tap right and turn off footer and footer).

Disclaimer: all names used in the prototype are not based on real people. Do not follow the medications and their intake amount mentioned in the prototype.

YouTube link for more information about the Caregiver Dashboard: https://www.youtube.com/watch?v=uF3c8V3PfEA&feature=emb_logo

Click on the pictures to watch a video of our interactive prototypes.

17 of 20

DIGITAL

DIVIDE

Reducing App Size

Because it is beyond our realm to fix the digital divide across the world, we will ensure that our mobile app is available and easily accessible to those with some form of access to the Internet and necessary hardware (i.e. smartphone). We plan to ensure that our app can be accessed by those living in rural and underrepresented areas by keeping its battery usage and computational expense to a minimum to make sure that users can use it all day. Reducing the computational requirements so that it can efficiently function on the majority of smartphones even in areas with low bandwidth. There would also be various offline features on our app like the task force, calendar events, and medication management. Having some features of the app being functional offline will greatly benefit those who may not have consistently stable Internet connection. Similar to that of streaming platforms, iMed will also have features where users can download data so that they can access it when Internet or power is down. This is helpful in the case of emergencies and backing up important information. In regards to reaching out to areas with significant latency issues (such as low bandwidth), users can attend meetings via a phone call from the dashboard or transitioning from video conference to audio when the internet connection becomes too unstable, thus mitigating any technical disruptions.

Funding and Future Investments

We primarily plan to direct our funding and future investments towards the integration of computers and mobile devices in remote areas that are not currently able to access the internet. Organizations such as Smartriverside are pursuing this goal, in addition to educating new users on computer and technology literacy. By directly purchasing and donating technological devices to areas of interest, we maximise the reach and potential of our application.

Research and Development

Launching independent research and development initiatives will increase overall transparency of the general process. Certain initiatives may include (but are not limited to) maximizing exposure to our application, cheaper production of digital mobile devices, efficient remote internet connectivity, increasing technological literacy, etc. “The greater the investment in R&D, the broader the scope for innovation that can drive growth.“^[19]

Technological globalization accentuates the growing need for universal accessibility of technology; addressing the technology gap is a key component of our research, which we plan to resolve through the following initiatives.

18 of 20

FUTURE POTENTIAL

Currently CAREN is capable of analyzing the emotions of the PWD and visually displaying that data to the caregiver. Future iterations of this function could be used to measure the progression of the disease in the PWD and such knowledge could offer subsequent insight for the caregiver as they provide aid.
In regards to CAREN, adding a digital face to provide visual feedback (such as nodding or smiling) can create a more intimate social connection with the user. Additionally, iterating on the deep learning architecture of CAREN can give it the ability of carrying on conversations more naturally and integrating reinforcement learning will allow Caren to adapt to the personality of the user as they frequently converse.
CAREN currently consists of the ability to analyze the behavior and monitor that data for PWD only. The ability to analyze the emotional fluctuations of the caregiver can give a better insight on how the wellbeing of the caregiver themselves. Analyzing various emotional trends, such as sadness, and creating responses (such as offering resources to help with anxiety or depression) to help improve the emotional wellbeing of the individual.
Similar to group therapy sessions for patients, iMed can deploy a feature which allows caregivers to have face-to-face virtual group discussions to build stronger interpersonal connections.
Another element that could be incorporated into the app is PERS, which pilots emergency system responses for caregivers and the person with dementia, using technology such as GPS tracking and mobile alerts for caregivers^[16].
Improving infrastructure (such as a video conference platform) can optimize the accessibility of our application for individuals who have poor internet connection.

Firstly, with iMed we want to reach out to people with dementia not only in rural areas, but also in care centers and old-age homes.
Also, although the iMed app is designed specifically for people living with dementia and their caregivers, the caregiver and doctor dashboards can be used by patients who are not living with dementia and thus are able to navigate the app without confusion. The app would be widely available to anyone and if we were to take the project further, we could code the app and put it on the app store to be available on iOS and Android.
Deploying a web app would also be a next step for iMed so that users can easily interact with the app via a browser.

One implementation strategy is to standardize referral of telemedicine to patients and clarifying the duration of patient telemedicine use^[14].There are some patients where telemedicine is just not a feasible option and being able to filter out who can or cannot use telemedicine will help in its successful implementation.
Another strategy is to increase data sharing and access. This will allow for more transparency between all stakeholders, surpass the issue of interoperability, and ensure that patient records are regularly updated.
As for insurance, universal health coverage would be best for adoption of telemedicine because less people would have to worry about making sure that their healthcare is being paid for. Especially with the COVID pandemic, there’s been a high influx of patients who need care and not being able to receive that care is not only harmful to the patients but also to the adoption of telemedicine since less people will be interested to use it.

EXPANDING TO MORE NICHES

BROADENING CAPABILITIES

IMPLEMENTATION

PWD = Person/People with Dementia

19 of 20

Acknowledgements

Mentor, Shibu Antony

Professional Contributions

The New York Academy of Sciences

Our mentor was able to give us so much guidance and support as to what would be beneficial to focus on. Not only did he formulate different questions that would benefit our in-depth research, but also spent his valuable time to come to almost all of our meetings and give is very valuable feedback. Needless to say, his help was very integral to our team’s success.

We had the opportunity to interview many experienced professionals and gain their insights on different aspects of our solution. This was very useful for us, as it added various new dimensions to our project and helped assess the feasibility of our project in the real world, to actually bring about a change. We wanted to give a special shoutout to all the professionals who were kind enough to spend some of their time to be interviewed: Dr. Terra Hussar, Dr. Michael Ross, and Dr. Peter Alperin, CEO Julie Fleming, CEO Kevin Jameson, Caregiver Vic Steele, and Caregiver Charlene Durante,

Of course, we would not have been able to collaborate as a team and even get the chance to create our solution if not for the New York Academy of Sciences and their Launchpad for the Junior Academy. Hence, we wanted to sincerely thank the academy for providing us with this wonderful opportunity to create something that will actually be making a real world impact. Many thanks, once again!

20 of 20

REFERENCES

^[1]“What Is Telemedicine?: Telemedicine System.” Chiron Health, chironhealth.com/telemedicine/what-is-telemedicine/.

^[2]“What Is Dementia?” Alzheimer's Disease and Dementia, www.alz.org/alzheimers-dementia/what-is-dementia.

^[3]Alzheimer's Disease Fact Sheet. https://www.nia.nih.gov/health/alzheimers-disease-fact-sheet.

^[4] Nowak, A. http://designingfordementia.com/index.php/2017/06/12/how-to-utilize-paint-colors-in-dementia-care-communities/.

^[5] Boh, W. by P., & Boh, P. (2020, May 10). What Colors Do Dementia Patients Prefer? https://readementia.com/what-colors-do-dementia-patients-prefer/.

^[6] Dingley C, Daugherty K, Derieg MK, et al. Improving Patient Safety Through Provider Communication Strategy Enhancements. In: Henriksen K, Battles JB, Keyes MA, et al.,�editors. Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 3: Performance and Tools). Rockville (MD): Agency for Healthcare Research and�Quality (US); 2008 Aug. Available from: https://www.ncbi.nlm.nih.gov/books/NBK43663/�^[7] Clochesy, J. M., Dolansky, M. A., Hickman, R. L., J.R., & Gittner, L. S. (2015). ENHANCING COMMUNICATION BETWEEN PATIENTS AND HEALTHCARE�PROVIDERS: SBAR3. Journal of Health and Human Services Administration, 38(2), 237-252. Retrieved from http://login.ezproxy1.lib.asu.edu/login?url=https://www-proquest-com.ezproxy1.lib.asu.edu/scholarly-journals/enhancing-communication-between-patients/docview/1684664853/se-2?accountid=4485

^[8] Practical Cryptography. (n.d.). Practical Cryptography. Retrieved December 16, 2020, from http://practicalcryptography.com/miscellaneous/machine-learning/guide-mel-frequency-cepstral-coefficients-mfccs/

^[9] “Deep Learning - Affectiva : Affectiva.” Affectiva, Affectiva, https://www.affectiva.com/how/deep-learning-at-affectiva/. Accessed 16 Dec. 2020.

^[10] Cohen-Mansfield, J. (2018, April 5). The impact of group activities and their content on persons with dementia attending them. Alzheimer's research & therapy. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5887184/.

^[11] Carson, L. (2020, September 30). Cognitive Stimulation Therapy (CST) Offered Virtually. Occupational Therapy Services. https://otservices.wustl.edu/cognitive-stimulation-therapy-cst-offered-virtually/.

^[12] WebMD. (2020, March 31). Dementia Treatments: Medication, Therapy, Diet, and Exercise. WebMD. https://www.webmd.com/alzheimers/dementia-treatments-overview.

^[13] Gowans, G., Dye, R., Alm, N., Vaughan, P., Astell, A., & Ellis, M. Designing The Interface Between Dementia Patients, Caregivers and Computer-Based Intervention. http://staff.computing.dundee.ac.uk/nalm/CC_website_at_Dec_2016/gowans_et_al_2007.pdf.

^[14] Taylor, J., Coates, E., Wessels, B., Mountain, G., & Hawley, M. S. (2015, December 1). Implementing solutions to improve and expand telehealth adoption: participatory action research in four community healthcare settings. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4666096/.

^[15] Pugliese, L., Woodriff, M., Griffin, M., Chokshi, D. S. K., Lynch, K., Sampson, A., … Kachnowski, D. S. Designing Technology for Caregivers: Understanding What Works and What Doesn't. https://www.aarp.org/content/dam/aarp/research/surveys_statistics/ltc/2018/designing-tech-caregiving.doi.10.26419%252Fres.00191.001.pdf.

^[16] Affectiva. (n.d.). Emotion AI 101: All About Emotion Detection and Affectiva’s Emotion Metrics. Blog - Emotion AI and Tech Innovation - Affectiva. Retrieved December 16, 2020, from https://blog.affectiva.com/emotion-ai-101-all-about-emotion-detection-and-affectivas-emotion-metrics

^[17] Hsiao, Y., Chang, C., & Gean, P. (2018, January 11). Impact of social relationships on Alzheimer's memory impairment: Mechanistic studies. Retrieved December 18, 2020, from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5764000/

^[18] Why Alzheimer's and Dementia Caregivers Are So Stressed. (2019, July 24). Retrieved December 18, 2020, from https://www.alzheimers.net/the-reasons-why-alzheimers-and-dementia-caregivers-are-so-stressed

^[19] Closing the Technology Gap in Least Developed Countries. https://www.un.org/en/chronicle/article/closing-technology-gap-least-developed-countries.

^[20] Muhr, A. (2019, August 14). Building a Vocal Emotion Sensor with Deep Learning | by Alex Muhr | Towards Data Science. Medium; Towards Data Science. https://towardsdatascience.com/building-a-vocal-emotion-sensor-with-deep-learning-bedd3de8a4a9

^[21] Understanding and supporting a person with dementia | Alzheimer’s Society. (n.d.). Alzheimer’s Society. Retrieved December 18, 2020, from https://www.alzheimers.org.uk/get-support/help-dementia-care/understanding-supporting-person-dementia#content-start