Designing glanceable
Intelligence
Visual Overlays XR
GenUX Widget
+ Magic Leap Spotlight Apps
Jinu Yang jinu.w.yang@gmail.com
Selected work shared for interview discussion only. Please do not distribute.
Hi! I’m Jinu,
a product and visual interaction designer.
Click
Touch
Embodied
Tangible
Space
Click
Touch
Embodied
Tangible
Space
Extended Reality
Gemini Visual Overlays XR
Role, scope & contribution
Problem space
Discovery
Hypothesis
Prototype
System
Implementation
Outcome
1
2
3
4
5
6
7
8
Role, Scope & Contribution
Overview
Role
Senior Visual Interaction Designer embedded with the Android XR team
Collaboration
XR UX · PM/Program · Engineering · Research
Project Stage
Concept exploration → prototyping → validation → system definition → implementation / productization
My Scope
Contribution
What is Visual Overlays
01 Case Study / Visual Overlays XR
Enable Gemini in XR�to show, not just tell
“Give Gemini the Pen”
Lightweight visual feedback that enhances user understanding & creates a richer two-way conversation experience.�
Start with highlighting object (i.e. bounding box) and evolve visual language as expand on use cases.
Why it matters
In XR, real-time guidance becomes more immediate, contextual, and easier to act on.
Problem space
How do we bring Gemini into the �physical world through spatial interfaces that adapt naturally to the user?
Design strategy
Connected
Adaptive
Scalable
Contextual Anchoring
Forgiving Precision
Spatial constraints
Human factors
Tracking Fidelity
Level of Interaction
My process
Discovery
Hypothesis
Prototype
System Implementation
Discovery
Homework Tutor CUJ
Where does spatial guidance actually add value?
Visual overlay on Glasses, blackboard on Phone
Visual overlay & blackboard both on Headset
With potential to generate & display 3D diagrams
Hey Gemini, help me understand this.
Gemini Live
User Intent
Perception
Reasoning
Spatial Guidance
→
→
→
→
Interaction Flow
Discovery
Where does spatial guidance actually add value?
The Android Show: XR Edition, Dec 2025
Gaming Coach CUJ
Hey Gemini, where would be a good place to add a park?
Gemini Live
User Intent
Screen Context
Reasoning
Spatial Guidance
→
→
→
→
Interaction Flow
Discovery
Where does spatial guidance actually add value?
Expert Helper + Translation CUJ
Hey Gemini, can you help find the right setting for silk?
Gemini Live
User Intent
Perception
Translation
spatial guidance
→
→
→
→
Reasoning
→
Interaction Flow
Hypothesis
Defining Spatial Behavior
Point Center-axis dot
Base area Surface-aligned boundary
Surface Billboarded bounding box
Object Segmentation mask
What does Gemini attach to?
Lightweight & forgiving
Precise & integrated
vs
Hypothesis
Defining Spatial Behavior
How should Gemini highlights behave?
Expand establishes target
Converge focuses on target
Expand + Gemini 4C
Converge + Gemini 4C
Hypothesis
Defining Spatial Behavior
What happens at difficult viewing angles?
Switching sides logic
When the viewing angle falls below 20°, the highlight shifts to the nearest perpendicular face.
Prototype
Testing spatial behavior
How should highlights adapt to distance �and viewing angle?
Validating spatial behavior across distance and viewing angle
Defined rotation behavior for how highlights track and reshape across changing viewing angles.
Established thresholds for transitioning highlight as the user moves closer to or farther from target.
Distance
Viewing angle
Prototype
Testing spatial behavior
Prototyping revealed that surface-aligned highlights felt most natural—enclosing the content without rotating with the viewer.
Single-axis
Head-aligned
Surface-aligned — selected
Should highlights anchor to the surface on �digital content?
System
Turning Interaction Patterns Into a System
A three-state highlight system that adapts to object size and distance within the field of view.
Small Dot
For distant or visually small targets.
Bounding Box
For objects that can be clearly enclosed within the active viewing range.
Large Dot
For objects that become too large or too close to frame effectively.
Animated transitions provide clear visual consistency and helps users track as highlight moves between states.
Visual Overlays Framework
System
Turning Interaction Patterns Into a System
Implementation
Headset User Flow
Outcome
My Direct Contribution
Designed the spatial highlight language
Defined how digital and physical targets are identified, tracked, and visually prioritized in space
Systemized spatial guidance
Evolved 2D visual guidance into a lightweight, adaptive 3-state spatial system
Drove cross-functional iteration
Partnered with engineering to adapt tracking behavior across FOV, distance, viewing angle, and lighting constraints
Established the Visual Overlays blueprint
Documented a scalable spatial design language that enables consistent implementation and future multimodal overlays.
External validation
“... it's going to play a big part in how we use devices running Android XR because it allows the headset (or glasses) to understand better what you're looking at in order to provide more helpful contextual responses.”
— Engadget
Before�
Gemini primarily explains through language and flat, screen-based UI cards.
After�
Gemini brings glanceable, contextual intelligence into the user’s physical surroundings.
GenUX Widget
Role, Scope & contribution
Problem space
Design Strategy
Concept exploration
Outcome
1
2
3
4
5
Role, Scope & Contribution
Overview
Role
Senior Visual Interaction Designer embedded with the Android XR team
Collaboration
XR UX, PM/Program, Engineering, Research, cross form-factor Android teams
Project Stage
Early product vision → concept validation → framework definition → roadmap
My Scope
Contribution
What is GenUX
From
Static, pre-defined, one-size-fits-all interfaces
The opportunity
Move from one experience for many users toward experiences shaped for the individual.
To
Dynamic, AI-generated interfaces created on demand around user intent and context
Google I/O 2026 “Create My Widget”
Problem space
How can we design generative interfaces in a predictable, structured way-without cluttering the user’s real-world field of view?
Design Strategy
Glanceable
Contextual
Customizable
Highly selective
Low-attention
Contextual input
User intent
Personalized
Flexible layout
Cross-device context
Concept Exploration
Adaptable Layout
Different score structures
Different content density
Template flexibility
Information hierarchy
Show me the live score of the LAFC game.
When are the next matches for the Bulls?
Concept Exploration
Overcoming the “Blank Canvas” in natural language creation
Curated categories and sample prompts eliminate the blank canvas, helping users easily discover, modify and personalize their widgets
Concept Exploration
North Star Concept _ Multimodal contextual widget
Outcome
My Direct Contribution
Proposed
A scalable widget framework for handling varied and non-deterministic Gemini responses.
�Aligned
UX, Health/Fitness, Research, PM, Engineering, and partner teams around a shared product direction.
Defined
Critical user journeys that clarified where GenUX could add meaningful value for Glasses users.
�Designed
High-fidelity GenUX concepts across key use cases to explore how generated interfaces adapt to user needs and context.
�Established
A north-star use case demonstrating the potential of personalized, context-aware GenUX.
Before
GenUX was designed around traditional screen-based interactions
�After
GenUX extends to glasses through glanceable, flexible experiences designed that can adapt to individual needs and context.
Magic Leap Spotlight apps
Role, Scope & Outcomes
Overview
Role
Senior Visual Interaction Designer Magic Leap 2 Spatial Products
Collaboration
UX, Product, Engineering, Prototyping, Marketing
Project Stage
Product definition → design system → implementation → iterative releases
Workshop · Assist · Spectator
My Scope
Outcomes
UI design
Workshop
UI design
Assist
UI design
XR Gallery
Spatial interaction design
Assist Workspace
Snapping UI - wall /
Spatial interaction design
Snap to surface
Snapping UI - wall /
Spatial interaction design
Gaze interaction
Thank you!
Jinu Yang jinu.w.yang@gmail.com