Loudoun County Public Schools
Assistive Technology
Gesture-based Learning Using the
Xbox 360 with Kinect
Authors
Christopher Bugaj, MA CCC-SLP
Dr. Tara Jeffs, Ph.D
Karen Dietrich, M.Ed
Mark Nichols, M.Ed
Overview/Summary of Implementation
In the Spring of 2012, a pilot study involving five Xbox 360 with Kinect were placed in autism classrooms in the school district (two elementary, two middle, and one high school) to explore the benefits of gesture-based learning. The primary objectives of the initiative were to explore the impact of whole-body gaming on goals outlined within individual student’s Individualized Education Program. Specific goals targeted were related to increasing collaboration, positive behaviors, social communication, pragmatic language skills, receptive language skills (ie, following directions), proprioceptive skills, and academic achievement. Implementation of the pilot ran from March 2012 until June of 2013.
Throughout the duration of the pilot students at all grade levels participated in a wide range of activities. Such activities involved using the units for side-by-side cooperative games that require whole-body movement to control the environment; gesture-based matching and vocabulary activities through avatar creation; gesture-based games to practice math facts; video monologues; and student-to-student interviews using Avatar Kinect (a closed, virtual space).
Implementation
Educators implemented activities using the units in a variety of ways. The following offers a brief description of different types of sessions:
The commercial games used during the pilot were Kinect Adventures, Disneyland Adventures, Kinectimals, Kinectimals (with Bears), Body and Brain, and Kinect Sports. Avatar Kinect, which requires an Xbox Live Gold membership, was used for the video monologues and student-to-student interviews. The avatar creation, video monologue, and student-to-student interview sessions were used less frequently than individualized gaming and cooperative, side-by-side gaming sessions.
Support & Training
Assistive technology trainers provided support for the program at each of the five sites. This included the creation and management of Xbox Live accounts, placement and initial setup, troubleshooting, and implementation strategies. During video monologuing and student-to-student interviews Assistive Technology Trainers were present to provide support. Educators also accessed their onsite technology resource teachers for immediate support.
Challenges, Solutions & Considerations
Initial and ongoing support was provided to help educators implement the units. Throughout the course of the pilot the support team and educators faced practical, technical, and training challenges. These challenges and their corresponding solutions are as follows:
Student Considerations for Student-to-Student Interviews
When selecting students to participate in video monologues and student-to-student interviews students need to demonstrate certain prerequisite abilities for successful participation. Specifically, students need to be able to express themselves verbally or with an augmentative/alternative communication device. An additional requirement would be for the student to be able to follow single step directions or prompts.
Student Considerations for Gesture-based Gaming
When participating in student-to-student interviews or video monologuing sessions a student does not need to be ambulatory. For some (most) commercial games using the Kinect sensor the participant needs to be ambulatory with full body control. One needs to be able to control and coordinate muscle movements, especially including their extremities.
Account Management
During the initial setup phase of the pilot the decision was made for every student participating in the pilot to have his or her own avatar along with a personalized Xbox Live account. E-mail addresses were necessary to create the Xbox Live accounts. Once e-mails were generated, the Xbox Live accounts were generated. Creating of the Xbox Live accounts also generated a Gamer Tag ID. In order to manage and maintain the e-mails accounts, Xbox Live accounts, and Gamer Tag IDs a shared online spreadsheet was created. These accounts were then downloaded to the Xbox units. Students and teachers then took turns participating in guided lessons where they used the gesture-based controls of the until to customize their individual avatars.
Ultimately, it was determined that every student did not require his or her own avatar and that the use of a three accounts per box model was adequate. The three accounts were one for the teacher, a female avatar, and a male avatar. In this way, students would control the classroom avatar when playing a game or interacting in a student-to-student interview. When playing cooperative games a second student would either control one of the remaining two avatars or a generic, system-generated avatar.
In order to connect in “Parties” which allow accounts to interact with one another (using either asynchronous text or audio messaging, synchronous audio, or synchronous gaming) the accounts need to “friend” one another. Navigating the system to establish these account connections was difficult to manage when maintaining a large number accounts. Instead of trying to initially establish all the friend connections a model of establishing connections immediately prior to a student-to-student interview session was employed. A few minutes prior to a student-to-student interview session assistive technology trainers would navigate the system to establish these connections for the educators. Although this decreased confusion about how to establish and maintain account “friendships” it also increased the duration of each session.
Technical Considerations
In order to operate the Xbox one must be connected to a power source, have speakers with an RCA connection, and be connected to a RGB video output source such as a television, standalone projector or a projector integrated into an interactive whiteboard.
In order to have the Internet-based portions of the technology work properly, such as connecting to the Xbox Live servers and connecting to others via Avatar Kinect) coordination with the Department of Technology Services was necessary to unblock communication ports. Wi-Fi keys were necessary to be placed on the units and, in some instances, hardwiring the units worked better than Wi-Fi. Despite these coordinated efforts there were still instances of connection difficulties.
During student-to-student interview sessions many different aspects of the technology were necessary to function properly all at once. Difficulties included the requirement for the room to be silent during audio calibration to limit feedback, managing unexpected updates to the Xbox firmware/system software, and balancing the lighting requirements for the system to adequately pickup movements without too much light being present to wash out the screen.
Spatial Considerations
Units were placed differently to meet the unique needs based on the classroom setup and school environment. In some classrooms, the Xbox 360 was placed in a stationary location within the room. A static play space was then defined and maintained for ongoing use. In other locations, the unit was placed in a stationary location but the play space needed to be created each time a session was going to occur by moving furniture out of the area. In other schools, the unit was placed on a mobile cart which could be moved from room to room to be used in a variety of settings. Again, a play space needed to be defined in order for students in interact safely.
Training & Practical Considerations
Placing a game into the XBox disk drive required the least amount of navigation and preparation. Educators and students required little support to access and play the commercial games. Communicating with other accounts in student-to-student interviews using Avatar Kinect required the most amount of navigation and presented the greatest challenge. Educators demonstrated difficulty differentiating between communicating in a party and establishing an Avatar Kinect session, navigating to Avatar Kinect (even once the application was pinned to a “My Pins” page per account), how to troubleshoot problems, how to initially save, share and edit a video monologue. Although some training materials could be found online that were created by personal users no established training materials exist. In order to train teachers how to navigate the system for different functions digital pictures were taken of sequences and placed in a step-by-step multimedia slideshow. Training materials created included step-by-step tutorials for how to navigate the audio calibration wizard, how to invite and accept friend requests, how to invite others into a party, and how to access Avatar Kinect. All of these tutorials are accessible from http://bit.ly/xboxhowtos.
Coordination of times to conduct student-to-student training sessions can be challenging especially when trying to connect two different schools and different grade levels. Variance in student and classroom schedules makes it difficult to establish consistent times to conduct student-to-student interview sessions.
Additionally, when the pilot started in March some educators who started with the pilot changed their roles at the beginning of the ensuing school year. When an educator exits the program a new educator steps in to take his or her place which means starting from scratch with regard to technical training. Hands on experience with navigating the system is a necessity and it takes time to develop the skills to support successful implementation. Teachers in the pilot were offered the option to keep the Xbox system for personal home use during the summer months (to gain a better understanding of the system software and games). Only one teacher chose to take advantage of this opportunity.
Cost
Beyond the costs related to time spent by the staff, the costs per implementation includes the cost of the Xbox 360 with Kinect sensor, any associated games, and an ongoing cost to provide Xbox Live Gold subscriptions per Xbox Live accounts provisioned to the students and educators.
Outcomes & Benefits
Throughout the pilot the educators were interviewed about ongoing progress toward student goals and their own professional development. Teachers reported all of the following outcomes and benefits experienced for the duration of the pilot:
Student Outcomes
Educator Outcomes
Additional Benefits
Recommendations
If an educational institution is considering the implementation of an Xbox 360 with Kinect sensor as a gesture-based learning initiative consider the following recommendations:
After the completion of the first year of this pilot it was evident that teachers needed and wanted activities that were easy to implement and required little preparation (plug and play mindset). They found that the commercial games provided this experience. It was also obvious that the Xbox was a motivator for students to become engaged in the learning activity regardless of the activity. This provided possible extended or alternative practice to build basic skills. Teachers observed students that seemed to be self-isolated or “in their own world” take an interest and participate in the Xbox activity sessions with their classmates. Positive things were happening in the classroom such as turn taking, coaching each other, and communication. Though there are many variables in a classroom that could increase these desired behaviors, the teachers in this pilot shared that the interaction with the Xbox in the classroom provided a natural stage to see them occur.
Students of all ages and ability were interested in the Xbox activities. Teachers were advised to preview the games and tasks involved so that they would know what prerequisite knowledge or skill sets may be needed by the students. Furthermore, previewing the content helped the teachers understand and plan for what scaffolding or other supports may be necessary to ensure student success.
Summary
The pilot was successful in exploring ways the Avatar Kinect and Xbox 360 Kinect can support students with Autism in their customary learning environments through kinesthetic engagement and fostering social communication skills. Such activities leveraged engagement of the students as well. The assistive technology team members involved in this pilot found that the Avatar Kinect and Xbox integrated favorably into the universal design framework by providing additional, flexible options to educators to represent information, allow students to express what they know, and to engage every learner.
Despite the challenges of making the student-to-student interviews a success, at the end of the first year, teachers in the pilot indicated that they were willing to continue to try to work out schedules to conduct student-to-student interview sessions. This illustrates that a progression took place and it is time to plan the next stages which, perhaps, could include action research or other avenues the teachers would like to pursue with their students.
Additional Resources