Loudoun County Public Schools

Assistive Technology

Gesture-based Learning Using the

Xbox 360 with Kinect

Authors

Christopher Bugaj, MA CCC-SLP

Dr. Tara Jeffs, Ph.D

Karen Dietrich, M.Ed

Mark Nichols, M.Ed

Overview/Summary of Implementation

In the Spring of 2012, a pilot study involving five Xbox 360 with Kinect were placed in autism classrooms in the school district (two elementary, two middle, and one high school) to explore the benefits of gesture-based learning. The primary objectives of the initiative were to explore the impact of whole-body gaming on goals outlined within individual student’s Individualized Education Program. Specific goals targeted were related to increasing collaboration, positive behaviors, social communication, pragmatic language skills, receptive language skills (ie, following directions), proprioceptive skills, and academic achievement. Implementation of the pilot ran from March 2012 until June of 2013.

Throughout the duration of the pilot students at all grade levels participated in a wide range of activities. Such activities involved using the units for side-by-side cooperative games that require whole-body movement to control the environment; gesture-based matching and vocabulary activities through avatar creation; gesture-based games to practice math facts; video monologues; and student-to-student interviews using Avatar Kinect (a closed, virtual space).

Implementation

Educators implemented activities using the units in a variety of ways. The following offers a brief description of different types of sessions:

  • Individual Access to Commercial Games & Cooperative, Side-by-Side Gaming - Some classrooms used the gaming sessions as an end of the week activity (ie, “Fun Friday”). Others used the games as an option as a reinforcer for positive behavior management that students could choose daily.

  • Avatar Creation - Students navigate the avatar store making selections pertaining to body parts, clothing options, and accessories. Students worked individually and in whole group activities to create a digital representation of the teacher and of themselves.

  • Video Monologues - Used to create video social stories of students talking about appropriate behaviors, end of the unit and end of the year summaries (ie, What have I learned), Mother’s Day video cards, and book summaries.

  • Student-to-Student Interviews - Students asked and answered questions while controlling and interfacing with an avatar from another school.

The commercial games used during the pilot were Kinect Adventures, Disneyland Adventures, Kinectimals, Kinectimals (with Bears), Body and Brain, and Kinect Sports. Avatar Kinect, which requires an Xbox Live Gold membership, was used for the video monologues and student-to-student interviews. The avatar creation, video monologue, and student-to-student interview sessions were used less frequently than individualized gaming and cooperative, side-by-side gaming sessions.

Support & Training

Assistive technology trainers provided support for the program at each of the five sites. This included the creation and management of Xbox Live accounts, placement and initial setup, troubleshooting, and implementation strategies. During video monologuing and student-to-student interviews Assistive Technology Trainers were present to provide support. Educators also accessed their onsite technology resource teachers for immediate support.

Challenges, Solutions & Considerations

Initial and ongoing support was provided to help educators implement the units. Throughout the course of the pilot the support team and educators faced practical, technical, and training challenges. These challenges and their corresponding solutions are as follows:

Student Considerations for Student-to-Student Interviews

When selecting students to participate in video monologues and student-to-student interviews students need to demonstrate certain prerequisite abilities for successful participation. Specifically, students need to be able to express themselves verbally or with an augmentative/alternative communication device. An additional requirement would be for the student to be able to follow single step directions or prompts.

Student Considerations for Gesture-based Gaming

        When participating in student-to-student interviews or video monologuing sessions a student does not need to be ambulatory. For some (most) commercial games using the Kinect sensor the participant needs to be ambulatory with full body control. One needs to be able to control and coordinate muscle movements, especially including their extremities.  

Account Management

During the initial setup phase of the pilot the decision was made for every student participating in the pilot to have his or her own avatar along with a personalized Xbox Live account. E-mail addresses were necessary to create the Xbox Live accounts. Once e-mails were generated, the Xbox Live accounts were generated. Creating of the Xbox Live accounts also generated a Gamer Tag ID. In order to manage and maintain the e-mails accounts, Xbox Live accounts, and Gamer Tag IDs a shared online spreadsheet was created. These accounts were then downloaded to the Xbox units. Students and teachers then took turns participating in guided lessons where they used the gesture-based controls of the until to customize their individual avatars.

Ultimately, it was determined that every student did not require his or her own avatar and that the use of a three accounts per box model was adequate. The three accounts were one for the teacher, a female avatar, and a male avatar. In this way, students would control the classroom avatar when playing a game or interacting in a student-to-student interview. When playing cooperative games a second student would either control one of the remaining two avatars or a generic, system-generated avatar.

In order to connect in “Parties” which allow accounts to interact with one another (using either asynchronous text or audio messaging, synchronous audio, or synchronous gaming) the accounts need to “friend” one another. Navigating the system to establish these account connections was difficult to manage when maintaining a large number accounts. Instead of trying to initially establish all the friend connections a model of establishing connections immediately prior to a student-to-student interview session was employed. A few minutes prior to a student-to-student interview session assistive technology trainers would navigate the system to establish these connections for the educators. Although this decreased confusion about how to establish and maintain account “friendships” it also increased the duration of each session.

        Technical Considerations

        In order to operate the Xbox one must be connected to a power source, have speakers with an RCA connection, and be connected to a RGB video output source such as a television, standalone projector or a projector integrated into an interactive whiteboard.

 In order to have the Internet-based portions of the technology work properly, such as connecting to the Xbox Live servers and connecting to others via Avatar Kinect) coordination with the Department of Technology Services was necessary to unblock communication ports. Wi-Fi keys were necessary to be placed on the units and, in some instances, hardwiring the units worked better than Wi-Fi. Despite these coordinated efforts there were still instances of connection difficulties.

        During student-to-student interview sessions many different aspects of the technology were necessary to function properly all at once. Difficulties included the requirement for the room to be silent during audio calibration to limit feedback, managing unexpected updates to the Xbox firmware/system software, and balancing the lighting requirements for the system to adequately pickup movements without too much light being present to wash out the screen.

Spatial Considerations

Units were placed differently to meet the unique needs based on the classroom setup and school environment. In some classrooms, the Xbox 360 was placed in a stationary location within the room. A static play space was then defined and maintained for ongoing use. In other locations, the unit was placed in a stationary location but the play space needed to be created each time a session was going to occur by moving furniture out of the area. In other schools, the unit was placed on a mobile cart which could be moved from room to room to be used in a variety of settings. Again, a play space needed to be defined in order for students in interact safely.

        Training & Practical Considerations

Placing a game into the XBox disk drive required the least amount of navigation and preparation. Educators and students required little support to access and play the commercial games. Communicating with other accounts in student-to-student interviews using Avatar Kinect required the most amount of navigation and presented the greatest challenge. Educators demonstrated difficulty differentiating between communicating in a party and establishing an Avatar Kinect session, navigating to Avatar Kinect (even once the application was pinned to a “My Pins” page per account), how to troubleshoot problems, how to initially save, share and edit a video monologue. Although some training materials could be found online that were created by personal users no established training materials exist. In order to train teachers how to navigate the system for different functions digital pictures were taken of sequences and placed in a step-by-step multimedia slideshow. Training materials created included step-by-step tutorials for how to navigate the audio calibration wizard, how to invite and accept friend requests, how to invite others into a party, and how to access Avatar Kinect. All of these tutorials are accessible from http://bit.ly/xboxhowtos.

Coordination of times to conduct student-to-student training sessions can be challenging especially when trying to connect two different schools and different grade levels. Variance in student and classroom schedules makes it difficult to establish consistent times to conduct student-to-student interview sessions.

Additionally, when the pilot started in March some educators who started with the pilot changed their roles at the beginning of the ensuing school year. When an educator exits the program a new educator steps in to take his or her place which means starting from scratch with regard to technical training. Hands on experience with navigating the system is a necessity and it takes time to develop the skills to support successful implementation.  Teachers in the pilot were offered the option to keep the Xbox system for personal home use during the summer months (to gain a better understanding of the system software and games).  Only one teacher chose to take advantage of this opportunity.  

Cost

Beyond the costs related to time spent by the staff, the costs per implementation includes the cost of the Xbox 360 with Kinect sensor, any associated games, and an ongoing cost to provide Xbox Live Gold subscriptions per Xbox Live accounts provisioned to the students and educators.

Outcomes & Benefits 

Throughout the pilot the educators were interviewed about ongoing progress toward student goals and their own professional development. Teachers reported all of the following outcomes and benefits experienced for the duration of the pilot:

Student Outcomes

  • Increases in student motivation & engagement
  • Increases student active participation
  • Increases in student socialization & collaboration with specific instances of showing interest in interacting with peers
  • Increases in peer-to-peer communication including verbal requests
  • Increases in the understanding and practice of turn-taking
  • Decreases in adverse behaviors when a gaming session was a reinforcement option
  • Decreases in adverse behaviors during a gaming session

Educator Outcomes

  • Increases in awareness of methodologies for representing content to students
  • Increases in awareness of methodologies for student expression to demonstrate the understanding of academic concepts
  • Increases in awareness of methodologies for engaging students in active learning

        Additional Benefits

  • During student-to-student interview sessions educators can provide prompts to students without the communication partner knowing that the prompts are being provided.
  • During video monologues students can create files which can be shared over the Internet with anonymity due to the fact that the only possible identifying factor is the student’s voice.

Recommendations 

If an educational institution is considering the implementation of an Xbox 360 with Kinect sensor as a gesture-based learning initiative consider the following recommendations:

  • During the initial implementation phase, limit the number of schools to 2-4 classrooms per support personnel. Depending on their other duties unrelated to the gesture-based learning initiative, the support person could allocate time to help with sessions, help coordinate student-to-student sessions, and generally provide necessary assistance.
  • When conducting student-to-student interview sessions, limit the interactions to one other classroom. Student participants could establish relationships with classmates in another building, grow those relationships, and then maintain those relationships without the confusion of interacting with multiple classrooms. When establishing these initial classroom connections, student abilities and interest should be considered along with similar schedules.
  • Consider the benefits of placing the unit in a shared space accessible to all students, such as the library or a special activity room that any student can access.
  • Provide students with a visually defined play space, such a mat, tape or other marker to outline the front, side, and back boundaries. Although most games provide some sort of text and visual prompt when moving out of the sensors range many students were too immersed in the game to take notice of the onscreen prompt.
  • With regard to the implementation of Avatar Kinect, consider the benefits of keeping the focus on student monologues at first. When connecting to other classrooms for student-to-student interviews the management of the technology may present difficulties and increase frustration. Practicing the use of video monologues will help users become experienced with the Avatar Kinect platform establishing foundational skills that are necessary for successful implementation of student-to-student interview sessions.
  • Consider the benefits of establishing a project management leader and liaison to help guide the initiative. This person might be responsible for scheduling progress update meetings, creation and maintenance of the account management system, and generating support materials.
  • When purchasing Xbox Live Gold subscriptions, in order to save money, only utilize one account at a time. Careful deliberation should occur prior to purchase to examine how often a classroom would be using Avatar Kinect within the given timeframe.
  • When creating video monologues, the implementation team should consider the benefits and/or necessity of gesture-based learning activities when compared to other similarly engaging tools. There are a number of additional tools available to create video monologues. Educators should be careful to examine both the pros and cons of each tool to determine which is best to complete the job. Student preferences and learner variability may be factors that are considered such that some students might choose to work within the Avatar Kinect environment while others might choose a different yet similar tool to accomplish the same task.

After the completion of the first year of this pilot it was evident that teachers needed and wanted activities that were easy to implement and required little preparation (plug and play mindset). They found that the commercial games provided this experience. It was also obvious  that the Xbox was a motivator for students to become engaged in the learning activity regardless of the activity. This provided possible extended or alternative practice to build basic skills. Teachers observed students that seemed to be self-isolated or “in their own world” take an interest and participate in the Xbox activity sessions with their classmates. Positive things were happening in the classroom such as turn taking, coaching each other, and communication. Though there are many variables in a classroom that could increase these desired behaviors, the teachers in this pilot shared that the interaction with the Xbox in the classroom provided a natural stage to see them occur.

Students of all ages and ability were interested in the Xbox activities. Teachers were advised to preview the games and tasks involved so that they would know what prerequisite knowledge or skill sets may be needed by the students. Furthermore, previewing the content helped the teachers understand and plan for what scaffolding or other supports may be necessary to ensure student success.

Summary 
         
The pilot was successful in exploring ways the Avatar Kinect and Xbox 360 Kinect can support students with Autism in their customary learning environments through kinesthetic engagement and fostering social communication skills. Such activities leveraged engagement of the students as well. The assistive technology team members involved in this pilot found that the Avatar Kinect and Xbox integrated favorably into the universal design framework by providing additional, flexible options to educators to represent information, allow students to express what they know, and to engage every learner.

Despite the challenges of making the student-to-student interviews a success, at the end of the first year, teachers in the pilot indicated that they were willing to continue to try to work out schedules to conduct student-to-student interview sessions. This illustrates that a progression took place and it is time to plan the next stages which, perhaps, could include action research or other avenues the teachers would like to pursue with their students.

Additional Resources

  • Presentations
  • Closing The Gap 2013 - Get Your Body Moving - Experiences with Gesture-based Learning (Xbox Kinect) - http://bit.ly/xboxctg13 
  • Assistive Technology Industry Association 2013 - Get Your Body Moving - Experiences with Gesture-based Learning (Xbox Kinect) – http://bit.ly/getmovingatia2013 
  • Virginia Society for Technology in Education 2012 - Get Your Body Moving - Experiences with Gesture-based Learning (Xbox Kinect) - http://bit.ly/vste12xbox 

  • Recorded Webinar

 

  • Social Media

  • How-To Guides Created by the LCPS Assistive Technology Team

  • List of Web-based Resources Curated by the LCPS Assistive Technology Team

  • Samples of Student Monologue Videos