老年智能手机助理
Assistify
Your AI audio companion that guides every smartphone interaction
Anthea Guo, Bonnie Wang, Dylan Chen
Problem
Cognitive overload
Too many icons, tiny labels, and confusing interface elements create overwhelming digital experiences.
Fear of consequences
Elderly users hesitate at every tap — they don't want to "break" anything or make costly mistakes.
Scam vulnerability
Phishing attempts and fake pop-ups specifically target the most vulnerable users.
Static tools fall short
Traditional accessibility features don't solve real-time, contextual confusion when it matters most.
A growing, underserved market
1.4B
People aged 60+
Worldwide population with >85% smartphone ownership in developed markets
80%
Struggle daily
Estimated elderly users who have difficulty with phone interfaces
Our Solution: Assistify
Multimodal Screen Understanding
ERNIE-4.5-VL analyzes your screen in real-time to understand text, icons, layouts, and contextual relationships.
Natural Voice Guidance
Speaks precise verbal instructions with no visual clutter: "At the bottom right, there's a screen icon. That activates screen sharing. Tap it once."
RAG-Powered Personalization
Vector database stores all interactions, retrieving top-5 similar past conversations to provide contextually-aware guidance that improves over time.
Example: First Time vs. Returning User
First Time: "I want to send this photo to my daughter"
Assistify: "I can see you're viewing a photo. Let's send it. What is your daughter’s name. First, tap the share icon at the bottom left—that's the small square with an arrow pointing up."
Returning User: "Send this to my daughter"
Assistify: "Sure! Let’s go through this again. Tap the share icon, then Messages, and select your daughter's contact."
01
Request help naturally
Elderly users simply speak their goal or question out loud
02
Continuous understanding
Assistify sees and comprehends exactly what's on screen in real-time
03
Step-by-step guidance
Clear spoken instructions for every tap, swipe, and interaction
04
Patient companion
Feels like a knowledgeable friend guiding users through any task
Assistify:
overlay-free, audio-first, multimodal assistance
Real-time screen understanding
ERNIE-4.5-VL anchors visual context to conversational guidance, connecting what users see with what they need to do next.
Overlay-free audio design
Verbal spatial guidance keeps the interface visually uncluttered using Apple ReplayKit no confusing overlays or additional elements to navigate.
Continuous verification
Capture → analyze → instruct → re-capture ensures accurate, adaptive guidance throughout every interaction.
RAG personalization
Learns the user's patterns and preferences over time for safer, faster, and more personalized assistance based on previous tasks.
What does Assistify do?
Example Screenshots
System Architecture
User Voice Input
Natural language request triggers screen capture
Background Screen Capture
1 FPS continuous monitoring, 10 evenly-sampled screenshots
ERNIE-4.5-VL Processing
OCR, icon recognition, layout analysis, state detection
RAG Context Retrieval
Query vector database, retrieve top-5 relevant past conversations
Personalized Instructions
Natural language generation with adaptive vocabulary and pacing
Screen Re-Capture & Verify
Analyze new state, compare against expected outcome, adapt next step
Beta Tester Independence
5 elderly testers (ages 68-82) completed tasks independently
Response Latency
End-to-end: speech input → screen analysis → speech output
Validated Use Cases
📸 Send photos to family
💳 Pay bills safely
🏥 Navigate health apps
⚠️ Detect scams in real-time
Technical Robustness
100%
5s
Testing and Performance
92%
Task Completion
Comprehensive Testing
Technical Achievements & Validation
Proven Performance Across 250+ Real-World Scenarios
94.2%
Text Understanding
150 test cases: messages, healthcare, banking, social media, scam detection
89.7%
UI Classification
200 screenshot samples: buttons, fields, toggles, navigation, modals
91.2%
Personalization
50 returning user sessions validating RAG database effectiveness
88.5%
RAG Retrieval
100 query-context pairs ensuring semantic similarity
Engineering + product at Berkeley
Dylan Chen
Applied Math + Data Science
Systems architecture and on-device integration expertise
Anthea Guo
EECS
Product and data engineering
Bonnie Wang
EECS
Core engine development and user testing
Our team has shipped open-source projects including Promptli and Accent Classification.
We've solved hard engineering challenges like rebuilding background capture and creating seamless voice UX — and we're ready to scale.
Live Demo!
Explore Assistify
Baidu AI Studio Project
OpenAtom Demo
GitHub Repository
Video Demonstration
Thank You!