Google Mail Training Program at Storm City International School (SCIS)

Final Evaluation Report

Submitted to the Head Teacher

and the Learning Technology Leader (Program Director)

on December 12, 2014

Lead Evaluator: Martin Mullan

Summary

At SCIS, the Learning Technologies team run a program of induction, lunchtime drop-ins, after school sessions and have a subscription to the Synergyse Google Chrome extension for contextual demonstrations. Staff are provided with multiple opportunities to enhance their skills in the use of the technologies used at SCIS

This report outlines the evaluation undertaken to determine how the training opportunities are perceived by the staff, in particular the lunchtime drop-ins, and the effectiveness of training overall in enhancing the skills of the staff specifically in Gmail.

A pre and post program survey/quiz was administered to staff to determine their skills in the use of Gmail. The pre-program survey was a self assessed measure based on a Likert scale. The post-program quiz was a multiple-choice and text input quiz that was then standardised against the pre-program survey and compared. A formative evaluation was carried out at the midpoint of the program to determine the staffs’ attitude toward the lunchtime drop-in sessions and as to whether they should continue.

The finding indicate that the staff find the videos and collegiate nature of their departments to be useful in acquiring the skills they need. The staff value the ‘lunchtime drop-ins’ but feel that the school should provided dedicated In Service Training (INSET) time and not make the acquisition of Core Skills such as Gmail something that has to be learned at lunchtime.

The program in its current form showed a significant increase in the skills of staff in the use of Gmail and the more advanced searching techniques needed to model digital literacy to students.

Description of the program evaluated

SCIS has become a Google Apps For Education (GAFE) school in the last 12 months. It is the vision of the school leaders to move to a 1to1 device to student program in September 2016 and they wish for the staff to be capable GAFE users by this time. Much of using Google requires good levels of digital literacy and the ability to search effectively; the concepts of boolean searching and using the advanced search options are skills required by all 21st Century learners.

The current training program focuses on the acquisition of skills in the use of Gmail in the context of SCIS. The program is aimed specifically at new teachers of which there are 18, however the incumbent staff members are also expected to have a desired skill level. Of a total of 81 staff members, 22% of the current faculty members are new to SCIS and are expected to acquire the necessary skills to allow them to effectively use and model the use of the technologies to the students.

The training program began with an induction session prior to the start of the first term and lasts until the end of the first term.

Program Objectives

The objectives of the Gmail Training Program are to give all faculty members the opportunity to access training at a time suitable to them and in a format that they prefer to learn from. The Gmail Training Program objectives are to provide faculty members with:

1. a baseline ability to use Gmail to a competent level,
2. multiple opportunities and media to learn from that they find valuable.

Program Components

The Gmail Training Program consists of four main elements: the New Staff Technology Induction day, scheduled after school sessions, lunchtime drop-in sessions, and a subscription to the Synergyse contextual training extension for Google Chrome.

The New Staff Technology Induction is a 135 minute training session on the basic requirements for new staff needed to get through their first day at SCIS. This session includes the basics of Gmail, SMART Notebook and SIMS (Student Information Management System). This session is run by the Learning Technology Officer with the support of a member of the IT Support team who is available to help with technical issues.

The after school sessions are a series of training sessions that run 3 times per week by the various Learning Technology Innovators (LTIs) in the school for the duration of the first term. The sessions are face-to-face and are repeats of each other so that faculty members have a choice of which to attend based on their schedules.

The lunchtime drop-ins are an addition to the program this year. The LTIs work together to man a drop-in center in the staff room every lunchtime. Staff members can speak to the LTIs regarding any element of technology use; these drop-ins, while aimed at new staff, give all members of staff an opportunity to come into contact with the skilled technology users in the school.

The Synergyse Extension for Google Chrome is a new addition to the technology training program at SCIS this year. Synergyse offers contextual video demonstrations that run with the Gmail interface.

Evaluation Method

Participants

All new members of staff at SCIS attend the Learning Technology Induction session before the start of the school year. Throughout the term, staff members may choose to attend the after school and lunchtime drop-in sessions. There is no stipulation at this time that staff members must attend sessions throughout the term; staff members are expected to assess their own needs in a professional manner and attend appropriate sessions as required. All members of staff will be expected to complete the Core Skills pre-program self-evaluation and post program quiz.

Lunchtime drop-in sessions and after school sessions are held by the Learning Technology Officer and the various LTIs at SCIS. LTIs perform this role voluntarily while the LTO has a specific responsibility for the management and organisation of the sessions. Communication with the staff regarding sessions is performed by the LTO who uses emailed announcements, targeted event invites, briefing announcements and notice board updates.

Procedures

During an interview with the Head Teacher and Learning Technology Leader, these program sponsors wanted to know if the new drop-in sessions were valued by the staff as a whole. The sponsors also wanted to find out if the training opportunities provided to the staff in their current format are meeting the objectives of enhancing the skills of using Gmail.  

A decision-making model was used half way through the program to formatively assess the value of the drop-in with the intention of continuing or abandoning them in response to their perceived value by the staff.

A goal-based model of comparing the skills of the staff in Gmail before and after the program was used to determine if the objective of enhancing the staff skills in Gmail had occurred or not. The sponsors were not interested to know which of the training types were most successful in generating a change in skills as they intended to maintain of the options in future if they were deemed hopeful to even one member of staff in their learning.

Data Sources

A survey given to all members of staff each year, new and current, was used to identify the self-assessed skills of the staff. While this survey covers all of the technologies used at SCIS, a section specifically on Gmail was used to identify the perceived skills of the staff in their knowledge of Gmail (Appendix A). The pre-program survey was designed to collect ordinal data on a Likert Scale: ‘Never Used’, ‘Need reminding’, ‘I’m good with this’ and ‘I could teach this’. Since it was unlikely that new staff will have come into contact with Gmail for a GAFE school, this format of self assessment seemed fair. The staff were asked to assess themselves against the following skills:

1. Send/Receive/Forward messages
2. Create labels, and move messages
3. Archive messages
4. Searching emails
5. Settings and configuring Gmail
6. Create and manage contacts
7. Create and manage tasks

The post-program analysis, at the request of the sponsors, was designed as a quiz in the contextual format presented by Google in their certification quizzes (Appendix B). This quiz is in the form of a 30 question, multiple choice and typed quiz that provides immediate feedback to staff. The Evaluator and Learning Technology Leader worked together to design the instrument in this fashion to model ‘Assessment As Learning’. While comparing the data from the pre and post tests may be considered invalid, the Evaluator believes that the analyses have value that the sponsors can use since the post program quiz has been review by the LTIs and equilibrated against their scores; this limitation was discussed with the sponsors and agreed. The sponsors agreed that self assessments on the pre-program survey would have the following equivalents on the post-program quiz:

1. 0-39% = ‘Never used’
2. 40-59% = ‘Need Reminding’
3. 60-79% = ‘I’m good with this’
4. 80-100% = ‘I could teach this’

A score of 60% was considered a pass on the pre-program quiz. Since the post quiz is specific to GAFE and SCIS, this lowering of the passing boundary brings the pre-program self-assessments into closer alignment with the post-program quiz.

A formative evaluation of the drop-in sessions was produced based on a survey instrument administered to all staff at the halfway point of the program (Appendix C). A combination of Likert Scale and multiple choice questions, along with free response text input questions were designed to give staff the opportunity to express their feelings toward the drop-in sessions at lunchtime.

Attendance data from the after school sessions was recorded in order to determine how many members of staff were making use of the sessions. The LTO used Google Calendar events to invite members of staff and record attendance at each session. It should be noted that the Program Director made an alteration to the program and the LTO stopped sending invites to the staff or advertising any of the training sessions 2 weeks into the second half term of the program.

Results

Objective 1, Gmail competency level:

Gmail skills were assessed using a pre-program survey and a post-program quiz and the data equilibrated as described above. Each member of staff’s responses on the pre-program survey were given a value and various levels of central tendency were calculated. It was found that there was little difference in the mean, median and mode of these values so the Evaluator has chosen to represent the staff’s self-assessed average as it is the most common and understood analysis (see Appendix B Results). The LTO used the staff’s individual responses to target individuals to be invited to the after school sessions; these too were optional to attend, but the staff received the reminder.

Table 1 below displays the average results of the whole staff’s (81 participants) self-assessed averages and their average performance on the post-program quiz :

Table 1

The graph above visualises the results from the table.

76% of staff self-assessed themselves to be competent users of Gmail prior to the the training program implementation, with 87.5% of staff reaching a passing score in the post-test quiz; an increase of 11.5%. 12.5% of staff failed to reach a passing score of 60% in the post-program quiz compared to a pre-program assessment of 23%. 50% of staff achieved a score of 80% or above in the post-program quiz compared to a self-assessed pre-program value of 21%, an increase of 138%.

Objective 2, value of ddrop-in lunchtime sessions:

The following graphs and data were captured from the drop-in survey (Appendix C) built in Google Forms. 50 of 81 staff members completed the survey. Another question was asked of those members of staff that did not drop-in; these comments can be found in Appendix C and provide information more pertinent to the overall training program for Learning technologies and will be considered in the discussion section.

1 = not at all valuable, 5 = very valuable.

Attendance data

Table 2

Table 2 and the Session Attendance graph show the attendance data for the after school sessions. After Week 3, the LTO stopped sending invites to the staff or making any effort to advertise the sessions to the staff on the instruction of the Program Director; the attendance drops to zero for all sessions.

Discussion

The purpose of this evaluation was to discern whether the current program of face-to-face lunchtime drop-in sessions, after school sessions, and video resources were adequate in providing the training needed for the SCIS staff to become skilled users of Gmail, and be able to model Gmail digital literacy skills to their students.

The results of the study will be used by the program sponsors to modify and enhance their provision and model of Learning Technologies training, not only for Gmail, but for the other technology skills required by the SCIS staff.

Does the current program provide staff with the baseline skills required to be effective users of Gmail and be able to model digital literacy skills? Yes, based on the pre and post-program assessments, there is an improvement from 76% pass rate to 87.5%. Upon examination of the questions in the post-program quiz, it is possible to reach a passing score of 60% without using any of the filter and searching skills; the significant increase in staff members achieving a score of 80% and above provides evidence that the program is providing the staff members with the skills to answer the advanced searching questions and make use of Boolean search techniques.

Considering the attendance at the after school sessions drops to zero, the staff are either using their faculty LTIs and colleagues to support their learning when they need it or they are using the Synergyse video training to improve their skills (a future review of the Synergyse access logs may reveal the extent of the video library use). This conclusion is supported by the comments made in the drop-in survey (Appendix C) where staff comment along these lines:

“My department have been great at going through processes with me”

“ My HOD is also an LTI and has been very helpful in giving me input when needed”

“I watched the videos which were great and I can watch them when I have time as well”

Are the drop-in sessions valued?

While only 7 of the 50 respondents made use of the drop-in facility, 54% felt the drop-ins were useful compared to 8% who thought they weren’t. 64% of staff felt the sessions should continue despite so few using the drop-ins and they felt that the sessions should be run again at the start of term one in the next academic year to support the new staff. It is clear that staff need to be reminded that there is a service available to them even if they choose not to use it. The setting up of the lunchtime drop-in sessions should be made early in the year - the first week.

“It was VERY useful, and makes me feel safer and more confident knowing that there are people available to help and support almost as soon as you need it. Yes, I can do what I wanted to now, and would love for these sessions to continue. I foresee other Qs coming up as we start using more new IT skills and software”

Staff at SCIS want to learn to use the technologies they need but feel their personal time is being taken up in order to acquire these skills. The free response comments provided by the drop-in survey indicate that staff feel that the school needs to organise dedicated time and training sessions to help staff acquire the skills they are expected to have.

26% of respondents stated that “Lunchtime is for eating”. 36% stated that they had other meetings to attend and many other staff commented on using lunchtime to talk with students or complete lesson preparation.

This evaluation has highlighted that the staff want to learn and value the efforts made by the LTO and LTIs. They use the video resources and the collegial nature of the staff has allowed skills to be spread to new staff despite their being little attendance at training sessions or in the absence of dedicated INSET.

While the program in its current form provides resources that have proven useful to staff e.g. the Synergyse videos, and valuable e.g. the lunchtime drop-ins, free responses from the staff indicate a need for SCIS to implement a more organised, whole school approach to technology skills that start before new staff members arrive at SCIS and are in place to support them from week one.

Project Cost

The following invoice indicates the service provided by the lead evaluator on site at SCIS. The Lead Evaluator interviewed the the program sponsors and developed the pre and post program survey/quizzes. Communication of the survey instruments to staff and the analyses of the data were performed by the Lead Evaluator.

Invoice

Staff:                         Lead Evaluator (5 days @ $500/day)                 $2,500.00

Travel                 Drive to SCIS (5x 15miles x $0.565/mi)                 $42.38

Lunch                        5 Days @ ~10$/day                                        $50.00

Total                                                                                 $2,592.38

Appendices

Appendix A: Core Skills Survey of Gmail section.

Appendix B: Gmail Quiz  - Linked.

Appendix C: drop-in survey

Why Didn’t you drop-in?

“I am in agreement that the support is needed to help staff update their skills. However, this is a busy school, and to expect staff to give up their lunch is pushing the limits. I do like the videos and have learnt from them. I can watch the videos at my leisure and at my pace. I think the videos are good.”

“Too busy! But it is nice to share good practice and get tips within department. We do this informally. I have also watched some of the videos. I'm not sure that we're all fully aware of what we should and shouldn't be able to do.”

“Duty on Monday lunchtime. Students on detentions at lunchtime Some meetings at lunchtime Preparing for lesson pm preferred after school”

“My department have been great at going through processes with me when doing something for the first time but if I didn't have them, I would have attended lunchtime drop in sessions.”

“This should be part of our directed meeting/training time. We should not be expected to give up our lunch breaks for what are essential skills we need to be trained in. The current system is not working and I feel staff morale will suffer as a result of unrealistic expectations regarding IT provision.”

“I think it's a great idea having this as a regular lunchtime thing, but this is an intense time of the term with organisation of the FOBISIA music festival, rehearsals and concert preparations all going on. “

“There's not enough time in the day to think about developing LT skills at the moment! :(“

“Have watched a couple of videos and lunchtimes disappear quite quickly!”

“All the training is carried out on PCs with no adaptation or consideration made for Mac users, which means essentially we are figuring it out for ourselves anyway, so it is easier for me to do that in my own time, rather than squeeze it into a lunch break. Also I really need breaks at lunch time, as the days the sessions are on I have really full timetable and ECAs that run until 5. My HOD is also an LTI and has been very helpful in giving me input when needed, but I am pretty confident and competent on a mac and with google, so feel comfortable working through it in my own time.”

“I love getting involved in these things but find lunch time is always really frantic reacting to things happening during the day especially as a new HOF I intended to attend to lots of sessions but always had something else to commit to which was a shame I watched the videos which where great and I can watch them when I have time as well”

“I don't feel that there is enough time in the day to eat and have some needed time-out if I attend lunchtime sessions”

“so busy, it's hard to find time”

“Though offering the drop in sessions is a fantastic idea and is really great of staff to give up their time to help. I feel proper planned training sessions given to new staff at the start of the year on training days or first few weeks would have been more useful as we had to learn on our feet as we went along. Also there should have been specific help for Macbook users as accessing Sims etc is more difficult in terms of reports and exporting information.”