Exploring Virtual Reality and Embodied Computational Reasoning
International Computing Education Research
August 14, 2021
Agenda
11:00-11:05 - Welcome
11:05-11:20 - Talks 1: Theories of Embodiment (talks in main room)
11:20-11:50 - Breakout 1 - Introductions and Discussions (breakout rooms)
11:50-12:00 - Reconvene the whole group. Reports from breakout sessions. (main room)
12:00-12:15 - Talks 2: Designing Embodied Coding Platforms (main room)
12:15-12:45 - Breakout 2 - Design Activities (breakout rooms)
12:45-01:00 - Reconvene with whole group. (whole group in main room)
Lenses of Embodiment
What is embodiment?
Simple definition:
“investment in or manifestation through a physical body; a bringing into or presentation in or through a form." [1]
In Embodied Cognition:
“embodied representations are shaped by sensorimotor interactions, and consequently by the physical constraints of the individual’s body. Thus, embodiment is a consequence of the filtering properties of our sensory and motor systems, but this input is already structured and shaped in accord with physical principles, and these provide the grounding of cognition.” [2]
In Embodied Design, Phenomenology, and Somatics:
“Embodiment is the participatory process of existing as a living body, awareness of the body, and awareness through the body, engagement with meaning, and agency in relationship to self and other.” [3]
What does embodiment mean to you?
[1] https://www.wordnik.com/words/embodiment.
[2] Pezzulo, Giovanni, et al. The mechanics of embodiment: a dialog on embodiment and computational modeling. 2011.
[3] Wood, T. R. Embodied Worldmaking. 2021.
Embodied Cognition
Perception and Action
Higher order abilities
Embodied Metaphor Theory
Antle, A. N., et al. Designing to Support Reasoned Imagination through Embodied Metaphor. 2009.
Examples of Embodied Metaphors
Metaphor mediates reasoning about abstract concepts
LIFE is a JOURNEY ARGUMENT is WAR
TIME is SPACE
Future is ahead
Past is behind
VALENCE is a VERTICAL AXIS
Positive is up
Negative is down
Evidence for Embodied Cognition in Language Processing
The ranger saw the eagle in the sky.
The ranger saw the eagle in the nest.
Zwaan, Stanfield, & Yaxley (2001)
Wu & Barsalou (2001)
Participants enumerated properties of whole vs half watermelon. Seeds and red were identified more frequently in response to the half watermelon.
Evidence for Embodied Cognition in Brain Function
Embodied Learning
MEteor Simulation
(Lindgren et al, 2014)
Embodied Design
Designing with and for the whole body, diverse and unique bodies with unique experiences.
Prioritizes lived first-person experiences and perspectives of individual bodies.
Body driven vs Brain driven
Somatics - somatics is the field which studies the soma or body as perceived from within. [1]
Somaesthetics - “by learning, improving and playfully engaging with movements involving our muscles, nervous system, and senses, (breath etc.) we extend on our experiences and create for better ways of being in the world” [2]
Somaesthetic Appreciation Design:
- Subtle guidance: directing attention inwards
- Making space: places for reflection
- Intimate correspondence: feedback that follows rhythm of the body
- Articulating experience: means to share experienced bodily sensations
[1] Eddy, M. A brief history of somatic practices and dance: Historical development of the field of somatic education and its relationship to dance. 2009
[2] Höök, K.,, et al. Somaesthetic appreciation design. 2016.
🤔 How do we harness embodied cognition for coding?
Breakout 1 - Discussions
This part is focused on how we use bodies in teaching,
and how we use bodies in learning environments.
Part 2: Designing Embodied Coding Platforms
Writing Code
Tandy 1000, from Radio Shack (ca. 1984)
Representations of Code
above left: textual code (javascript)
above: visual code (blockly)
Visual Programming Languages
Other Representations of Code
Boxer 1986
PYGMALION 1975
GRAIL 1969
Jupyter / IPython (2001)
Logo 1967
From Gallery of programmer interfaces by Jonathan M Edward (link)
Other Ways to Code
Johnny Mnemonic (1995)
From Visual to Embodied Code
Immersive technologies present new opportunities for code:
Engaging innate fluencies:
Concept Image for Embodied Coding Platform from
An Embodied, Augmented Reality Coding Platform for Pair Programming, NSF Award #2017042
Code Comments
/* Calculates the transform of an object
* relative to a marker.
* Given w_T_m, and w_T_o,
* calculate m_T_o = w_T_m^{-1} * w_T_o
* @param markerTransform (w_T_m) the transformation matrix
* of the marker in the world space
* @param objectTransform (w_T_o) the transformation matrix
* of the object in the world space
*/
function getRelativePose(markerTransform, objectTransform) {
...
}
/* | x
* | /
* | /
* | x
* |
* +-------------
*/
function getLineEquation(point1, point2) {
...
}
Code Comments
/* Calculates the transform of an object
* relative to a marker.
* Given w_T_m, and w_T_o,
* calculate m_T_o = w_T_m^{-1} * w_T_o
* @param markerTransform (w_T_m) the transformation matrix
* of the marker in the world space
* @param objectTransform (w_T_o) the transformation matrix
* of the object in the world space
*/
function getRelativePose(markerTransform, objectTransform) {
...
}
/* | x
* | /
* | /
* | x
* |
* +-------------
*/
function getLineEquation(point1, point2) {
...
}
Code Comments
/* Calculates the transform of an object
* relative to a marker.
* Given w_T_m, and w_T_o,
* calculate m_T_o = w_T_m^{-1} * w_T_o
* @param markerTransform (w_T_m) the transformation matrix
* of the marker in the world space
* @param objectTransform (w_T_o) the transformation matrix
* of the object in the world space
*/
function getRelativePose(markerTransform, objectTransform) {
...
}
/* | x
* | /
* | /
* | x
* |
* +-------------
*/
function getLineEquation(point1, point2) {
...
}
Code Comments
/* Calculates the transform of an object
* relative to a marker.
* Given w_T_m, and w_T_o,
* calculate m_T_o = w_T_m^{-1} * w_T_o
* @param markerTransform (w_T_m) the transformation matrix
* of the marker in the world space
* @param objectTransform (w_T_o) the transformation matrix
* of the object in the world space
*/
function getRelativePose(markerTransform, objectTransform) {
...
}
/* | x
* | /
* | /
* | x
* |
* +-------------
*/
function getLineEquation(point1, point2) {
...
}
Code Comments
/* Calculates the transform of an object
* relative to a marker.
* Given w_T_m, and w_T_o,
* calculate m_T_o = w_T_m^{-1} * w_T_o
* @param markerTransform (w_T_m) the transformation matrix
* of the marker in the world space
* @param objectTransform (w_T_o) the transformation matrix
* of the object in the world space
*/
function getRelativePose(markerTransform, objectTransform) {
...
}
/* | x
* | /
* | /
* | x
* |
* +-------------
*/
function getLineEquation(point1, point2) {
...
}
Abstracting Thought vs Abstracting Code
Related Work: VR Ideation in Post-Post-It
Related Work: Facebook Horizon’s VR Scripting
3D Assets, Modeling and Scripting in shared world.
Facebook Horizon https://www.oculus.com/facebook-horizon
Breakout 2 - Design Activities
This part is focused on brainstorming meaningful designs, metaphors, and gestures for embodied coding platforms.
Design Studies: Spring Break Research Experience
SBRE: XR Design Studies With Oculus Quest 2
Prototyping
above left: interactive 3d selection demo with handtracking (WebXR)
above: visual design studies in VR (Gravity Sketch)
Embodied Coding VR prototype environment
Embodied Coding VR prototype environment
Embodied Coding VR prototype environment
Thank You and Stay Tuned!
Links
Let us know if you want to collaborate further on VR design studies or research in this area!
Early Speculative Prototype for Virtual Keyboard (1995)