Evaluation Report

 

 

The potential impact of a blended learning approach on learners in a Sixth Form College

 

Yvonne Moore
10 July 2008
Student No: 10624848

 



 

Contents                                                                                                                          Page Numbers

 

Executive Summary..............................3

Introduction..........................................3-4

Methodology........................................5-6

Results Analysis#..................................6-9

Limitations ..........................................9

Discussion & Recommendations............9-10

References .........................................11

Appendices. ........................................12-20

A - Observation checklist. ..........12

B - Student Survey ...................13-14

C – Results* .............................15-21

 

 

#Graphs

Fig 1.................. 7

Fig 2.................. 8

Fig 3.................. 8

 

 

*Tables

Table 1................15

Table 2................15

Table 3................15

Table 4................15

Table 5............... 15-16

Table 6................16

Table 7............... 17

Table 8................17

Table 9............... 18

Table 10..............18-19

Table 11............. 20-21

Table 12............. 21

 

 


 

 

 


Executive Summary

This evaluation project aimed to discover the impact of a blended learning approach on learners in a Sixth Form college environment. The evaluation is formative in nature as it provides an insight into the effectiveness of the introduction, and ongoing use of, a learning management system (LMS) for use in support of face-to-face teaching. The project followed the eclectic-mixed methods-pragmatic paradigm, aiming to pull together an appreciation of current practice to inform future practice. Utilising Stakes Responsive Evaluation method enabled an observation of current usage and a survey of student views.  Conclusions to the project reveal the extent to which blended learning has been implemented, leading to recommendations about its future use.

 


 

Introduction

 

Moodle is a LMS or virtual learning environment (VLE), which was introduced to College X (a Sixth Form college specialising in education for those aged 16-19 years) in September 2005. It is available to all teaching staff, administrators and students. It is used as a repository for college administration (e.g. such as policy documents) and training documentation. It is also used as a resource repository with space available for every subject. Moodle 'areas' are generated by subject course qualification (e.g. AS ICT, A2 ICT). Some teachers use the available areas to provide all course documentation, resources that supplement the face-to-face classroom learning and relevant activities and links. The level of use of these areas varies greatly from teacher to teacher.

Initially for the purposes, and small scale nature, of this project, two curriculum areas were chosen to be observed:

The BCI department was chosen as teachers in this area tend to naturally make use of computer based resources e.g. spreadsheets for accounting or for developing software skills.  The Computing & ICT subject areas have used interactive whiteboards and digital projectors to support the teaching of software skills for some time. The Business subject areas have used Internet research into company case studies for a number of years.


The MFL department was chosen has they had not traditionally used computer based resources much, although over the past two years a new Curriculum Manager has championed the use technology and lead his team to integrate computer resources into their schemes of work.  However, due to timetable and staffing issues at the time of this project, this department were unable to offer access to their lessons and students.

 

Instead, another subject teacher volunteered to participate. The subject is Health & Social Care from the curriculum area:

 

·         Humanities

 

The purpose of the evaluation was to formatively review Moodle to judge its impact on learning from the student perspective. Although Moodle is available to all subjects across college this project will focus on the curriculum areas outlined above.

 

The following two e-learning guidelines (taken from the eLearning Guidelines for New Zealand) guided the investigation to qualify any impact Moodle (and the resources accessed from within it) has made.

  1. ST7 Will the e-learning foster students’ curiosity and creativity?
  2. ST9 Do the technologies employed help students successfully meet the learning outcomes?

The following objectives were achieved:

1. Approval to conduct the evaluation by the Assistant Principal;

2. One lesson observation, in Humanities

3. Survey of students – paper-based questionnaire;

4. Collation and data analysis;

5. Write up of the results;

6. Submission of a draft evaluation report with recommendations made to blog/facilitator;

7. Revision of the evaluation report - recommendations and decisions

The evaluation has provided outcomes relevant to the following areas (taken from the eLearning Guidelines for New Zealand):


Methodology

This project followed the eclectic-mixed methods-pragmatic paradigm as "the focus is on practical problems rather than on issues of reality and theories of society" (Philips, Bain, McNaught, Rice & Tripp. (2000, Table 1.3). Evaluation of teaching and learning is realistically a practical activity as it seeks to reflect on good practice (which can be shared) as well as bad practice (for which solutions can be generated). This approach to evaluation seeks to take account of flaws in other paradigms by enabling an evaluator to rely on an open approach to finding answers to a given problem. The mixed methods element is particularly useful as it enables data, qualitative or quantitative, to be drawn from a variety of sources that have a relevance to the evaluation questions.

This project generated a formative evaluation of the use of Moodle. Moodle has been in place since September 2005 and up till now there has been no formal evaluation of its use, its benefits or student perceptions. As the academic year comes to an end it is a useful time to survey students who have been involved in their current courses for nine months and will have some, if variable, experience of Moodle. Troachim (2006, under Types of Evaluation heading) has said that "formative evaluations strengthen or improve the object being evaluated - they help form it by examining the delivery of the program or technology, the quality of its implementation, and the assessment of the organizational context, personnel, procedures, inputs, and so on". In this context, the project examined the quality of the implementation of blended learning - from the student perspective.

Stakes Responsive Evaluation method led the gathering of evaluation data. This approach is one that concentrates on the experiences and mood of those involved with the teaching programme and allocates less importance to the scientific analysis approach of more traditional methods. Whereas a traditional method "draws legitimacy from scientific rigor, responsive evaluation draws legitimacy from endorsements by a majority of important stakeholders" (Visser, Updated 2008, ¶6). The scope of this project is fairly small and therefore the only stakeholders being considered directly are the students and teachers. In further exploration of the issues, other stakeholders who could be included are:

 


The following multiple methods were attempted:

Lesson Observation

Student Survey

 

 

The curriculum managers will see the results of the evaluation so that they can discuss with their teaching teams the impact of using Moodle. They may decide to share the results, or follow up on issues raised, with their students.

 

The following instrumentation has been used:

Lesson Observation - a checklist was used to identify evidence of student learning, particularly evidence of 'curiosity and creativity'. An observation checklist can be seen in Appendix A. Observation will also be made of whether students appear to have achieved the lesson objectives identified by the teacher at the start of the lesson.

 

Student Survey - a series of closed, ranking and open questions were given to students in a paper format (due to technical issues with the online format initially planned). This elicited perceptions of learning by the students including whether they believed the blended approach is beneficial to them. A list of survey questions can be seen in Appendix B.

 

At a busy time in the academic year, the following sample was used:

·         Survey - whole class, ICT students, 6 students

·         Observation & survey - whole class, Health & Social Care students, 10 students

 

Time did not permit additional interviews with the students.


Results Analysis

Data was collected from surveys and one lesson observation and the results are recorded in Appendix C. There is evidence that students like using Moodle. The survey and observation revealed that students liked the freedom to work at their own pace within a structured environment which meant resources were generally easy to find.  However, it was clear that students liked the ability to use computers in general rather than Moodle specifically. The Internet offered students more information, in varying formats, than could be currently gleaned from Moodle alone. This seemed partly due to the fact that teachers were currently only using a small part of Moodle’s functionality. This was confirmed by the fact that no students at all had used the forums or personal journal features of the software. (See Fig 1)

Fig 1

Students identified few significant problems with Moodle itself, instead offering negative issues related to computer availability or the teacher’s organisation skills when storing resources in Moodle. Half the comments received in the survey identified no problems with Moodle which is promising considering that the majority had not used Moodle before the current academic year. (See Fig 2)

 

 

 

 

 

Fig 2

It appeared difficult for students to identify whether they could be creative when using Moodle with most students not understanding the question or seeming to view the LMS only as a repository for the teacher’s resources. There was some indication that using computer resources could support students in producing better work than would otherwise be possible. Students were observed creating diagrams and including web links when these were not stipulated elements, which suggests some element of creativity. Also it was clear that some students used Moodle in a self-directed way or at least away from the classroom environment.  Although this does not identify creativity per se, it suggests a confidence in the use of Moodle for completing tasks that could further lead to creativity. (See Fig 3)

Fig 3

Similarly students had difficulty in perceiving the impact of Moodle on their achievements.  Although a number of students appeared to view Moodle as a storage facility for resources, relating little to their achievement, there was some appreciation that these resources could help produce more effective work. The observation showed that only three students completed the task set by the teacher, which may suggest, superficially, that Moodle did not help students achieve. However, students were able to complete at least five questions and over 60% of students said they used Moodle outside of lessons to complete class work and over 40% said they would finish tasks at home.  Without a much more thorough investigation there is little evidence to support whether Moodle really aids achievement levels.

 
Limitations

The project was undertaken in a college that I no longer work in full time and as such I had to rely on the good will of volunteers to participate.  Unfortunately the MFL department could not fit in my request for a lesson observation at a time convenient to this project and as such I had to find a new volunteer at the last moment.  This was not ideal as the project was now focussing on two curriculum areas that openly admitted to not using Moodle regularly.  This limited the opportunity to collect perceptions from students exposed to significant use of Moodle.

The surveys were completed on paper (due to a technical problem accessing the Survey Monkey website).  The responses were later entered into Survey Monkey to obtain the summary provided in Appendix C, however, there appear to have been some glitches and these are commented on at the relevant point in the results data.

There appeared to be some deficiencies in the survey questions. The open-ended questions relating to creativity and achievement seemed to prove difficult for students to answer.  More care in wording the question in a way that learner’s could relate to could have helped elicit responses.  Also, it is apparent that although some triangulation was possible between the survey responses and the observations recorded, it was not always possible to substantiate the evidence to make definitive judgements.

Also, the sample is very small considering the size of the college student population (currently 1600 students) and it is therefore difficult to generalise conclusions. Nevertheless, as a starting point, a picture starts to appear of current practice.


Discussion & Recommendations

This project set out to discover the perceptions of students who were exposed to some level of blended learning in their studies. There was some evidence that the e-learning fostered “students’ curiosity and creativity” (ST7) although not to any great level. It did appear that students enjoyed using Moodle, although more specifically they enjoyed using computers and the Internet. This demonstrates a certain level of ‘student satisfaction and use of online resources’.

 

The project also tried to identify if the “ technologies employed help[ed] students successfully meet the learning outcomes” (ST9) and this was more difficult to judge. It was possible to observe that students could complete tasks using e-learning which demonstrates some attainment but again the scope of the project does not enable any further conclusions to be drawn.  Sharpe, R., Benfield, G., Lessner, E., & DeCicco, E in their Scoping study (2005) set out their approach “to investigate learner’s current experience of e-learning” (page 1) by identifying that a number of data collection methods are likely to yield effective results. Such methods could include learner diaries, individual observations and interviews whilst working with e-learning and tracking of login duration. Any further evaluation could benefit from this type of approach.

 

Finally, this evaluation project can make a judgement on the “suitability of Moodle as a platform for online resources in a blended learning environment”. ( ST7 & ST9).  Students did not seem to have any significant negative responses to using Moodle. In Winnard’s (n.d.) investigation into the development of online individual learning plans an evaluation was included of student perceptions of Moodle and highlights that “there were more positive comments than negative highlighting potential benefits to users.” (Page 32)  As in this project students generally have no problems, although here the limited functionality utilised by the teachers may be a significant reason for the lack of challenge for the students. Nevertheless, Moodle appears suitable for organising course content, even in its limited form.

 

Overall, this project has been of limited value although there is enough evidence to suggest that the student experience of blended learning is a developing one.  There does not appear to be any real negativity towards its use and there are some positive signs that students feel their work can be improved by the use of blended learning.

 

As a result of this evaluation the following recommendations are made:

1.      Teaching staff could benefit from training in the following areas:

A)     how to organise resources into a clear structure

B)     how to use additional features of the Moodle software

C)    how to incorporate those additional features in a blended learning lesson

§         online quizzes with instant grading

§         importing video

§         discussion forums

§         personal journals

2.      Students could benefit from being introduced to other aspects of Moodle that may be beneficial to them (features which might more easily result in learner creativity or clearer outcomes) such as:

a)      discussion forums – to enable peers to discuss work openly

b)      personal journals – to enable learners to record their progress (good or bad) and seek help

c)      instant grading – to enable students to see their progress as they work

 

3.      In addition, further evaluations with more curriculum areas could be beneficial to see if a pattern emerges across the college.



References

  1. Milne, J. & Dimock, E. (2006). eLearning Guidelines. Guidelines for the support of e-learning in New Zealand. Massey University. Retrieved from http://elg.massey.ac.nz/Guidelines-questions.pdf
  2. Philips, R., Bain, J., McNaught, C., Rice, M., Tripp, D. (2000). Handbook for Learning-Centred Evaluation of Computer Facilitated Learning Projects in Higher Education. Murdoch University. Retrieved from http://www.tlc.murdoch.edu.au/archive/cutsd99/handbook/handbook.html
  3. Sharpe, R., Benfield, G., Lessner, E., & DeCicco, E.(2005).S coping Study for the Pedagogy strand of the JISC e-Learning Programme. Final Report. JISC
  4. Trochim, W.M.K. (2006). Research Methods Knowledge Base. Web Center for Social Research Methods. Retrieved from http://www.socialresearchmethods.net/kb/intreval.php
  5. Visser, R.M.S. (Updated 2008).Trends in Program Evaluation Literature: The Emergence of Pragmatism, TCALL Occasional Research Paper No. 5. Texas Center for the Advancement of Literacy & Learning. Retrieved from http://www-tcall.tamu.edu/orp/orp5.htm
  6. Winnard, W. (n.d.) Implementing and Evaluating the Impact of a Course Management System on Personalised Learning. Development Project. The University of Huddersfield School of Education and Professional development.

Appendices

 

Appendix A

Observation checklist

Curiosity and creativity:

 

Achieving learning outcomes:

 


Appendix B


Using Moodle - Student Survey

1. Introduction

 

Thank you for participating in this survey.
This questionnaire is designed to find out your views about using Moodle in your learning.
Try to answer all the questions but please leave blank any you do not understand.
The results will be used to evaluate the impact of computer technology on learning and be helpful in developing future lessons.

2. About Moodle

1. Before starting your current courses, had you ever used Moodle?

tYes

tNo


2. Where do you use Moodle?

tIn class, when directed by a teacher

tIn class, self-directed to complete a task

tOutside class but in college, to complete/finish a task

tAt home, to complete/finish a task

 

Other (please specify)

 



3. When do you tend to use Moodle?

 

Very Often

Often

Average

Occasionally

Never

In lessons

Very Often

tOften

tAverage

tOccasionally

tNever

For coursework research

Very Often

tOften

tAverage

tOccasionally

tNever

For revision

Very Often

tOften

tAverage

tOccasionally

tNever

For homework

Very Often

tOften

tAverage

tOccasionally

tNever

For fun/interest

Very Often

tOften

tAverage

tOccasionally

Never

 

 

 

 

 

 


4. What kind of activities do you use in Moodle?

tAccessing and reading course notes (e.g. Word documents, PowerPoint presentations)

tInteracting with course materials (e.g. completing quizzes or drag and drop exercises)

tListening to audio files (e.g. podcasts)

tWatching video (e.g. vodcasts)

tMonitoring your progress (e.g. checking test results/scores)

tCreating files to send/show to your teacher (e.g. essays, presentations)

tPosting questions to discussion forums (e.g. asking for help)

tPosting answers to discussion forums (e.g. offering advice)

tPosting entries to a personal journal (e.g. keeping track of your learning)

Other (please specify)


3. Your Learning

1. What do you like about using Moodle in your lessons?

2. What sort of problems do you have using Moodle in your lessons?

3. How creative does Moodle allow you to be?

4. How does Moodle help you achieve your learning objectives?


Appendix C - Results

 

The survey was carried out as follows:

 

Class 1 – Applied ICT (level 3) group of 6 students

Class 2 – Applied Health & Social Care (level 3) group of 10 students

 

I was not present when the surveys were completed due to complications in arranging lesson observations.

 

Table 1

Response Summary

Total Started Survey: 

16

Total Completed Survey

16 (100%)

About Moodle

Table 2

1. Before starting your current courses, had you ever used Moodle?

 

Response
Percent

Response
Count

Yes

18.8%

3

No

81.3%

13

 

answered question

16

 

skipped question

0

 

Table 3

2. Where do you use Moodle?

 

Response
Percent

Response
Count

In class, when directed by a teacher

100.0%

16

In class, self-directed to complete a task

18.8%

3

Outside class but in college, to complete/finish a task

62.5%

10

At home, to complete/finish a task

43.8%

7

Other  (please specify)

0.0%

0

 

answered question

16

 

skipped question

0

 

Table 5

3. When do you tend to use Moodle?

 

Very Often

Often

Average

Occasionally

Never

Response
Count

In lessons

12.5% (2)

31.3% (5)

37.5% (6)

18.8% (3)

0.0% (0)

16

For coursework research

0.0% (0)

21.4% (3)

28.6% (4)

35.7% (5)

14.3% (2)

14

For revision

0.0% (0)

14.3% (2)

21.4% (3)

14.3% (2)

50.0% (7)

14

For homework

0.0% (0)

21.4% (3)

50.0% (7)

21.4% (3)

7.1% (1)

14

For fun/interest

0.0% (0)

7.1% (1)

14.3% (2)

35.7% (5)

42.9% (6)

14

 

 

 

 

answered question

16

 

 

 

 

skipped question

0

 

Table 6

4. What kind of activities do you use in Moodle?

 

Response
Percent

Response
Count

Accessing and reading course notes (e.g. Word documents, PowerPoint presentations)

93.8%

15

Interacting with course materials (e.g. completing quizzes or drag and drop exercises)

12.5%

2

Listening to audio files (e.g. podcasts)

18.8%

3

Watching video (e.g. vodcasts)

43.8%

7

Monitoring your progress (e.g. checking test results/scores)

25.0%

4

Creating files to send/show to your teacher (e.g. essays, presentations)

43.8%

7

Posting questions to discussion forums (e.g. asking for help)

0.0%

0

Posting answers to discussion forums (e.g. offering advice)

0.0%

0

Posting entries to a personal journal (e.g. keeping track of your learning)

0.0%

0

Other (please specify)

 

*Comment Text Response 

1.web links to sites about our units 

2.links to web sites  

3.links to the internet by our teacher  

4.Links to internet sites about the unit we are studying

5.Links to internet sites 

6.reading what we've been told to and answering questions 

 

25.0%

4

 

answered question

16

 

skipped question

0

* Six comments are reported and only four responses counted. This could be due to the fact that the student had to tick Other as well as type their comment and two may have omitted to tick the Other box.

 

Your Learning

 

Please note that the number of comments for each question that follows equals ten, even though a greater number of responses are counted.  I can only assume there is a restriction on the basic Survey Monkey account I used to create the surveys. Hence, some comments are lost.

 

Table 7

1. What do you like about using Moodle in your lessons?

 

Response
Count

1. It’s fun to use the computer where everythings put in the right order for us.

2.don't know

3.it gives us information about the unit we’re doing and its easy to find

4.everything is one place and i can get it from home as well

5. All the unit is organised for us. We can find information on the unit at any time.

6.using the computers makes it more fun but moodle is a bit pointless we have our units there with handouts and worksheets and some presentations but thats all

7. Not much, prefer just to use the internet. The teacher sometimes puts good links there and its organised by unit so that’s helpful if you miss a lesson.

8.its fun to use the computers sometimes, makes lessons less boring

9.nothing

10 We can write assignments and use material the teacher has given us accesss to. You can save your work and send it to the teacher for marking. When she sends it back, you can save it to use again later for revising. I can work at my own speed and I'm not slowed down by other students.

14

answered question

14

skipped question

2

 

Table 8

2. What sort of problems do you have using Moodle in your lessons?

 

Response
Count

1. Sometimes finding resources that the teacher has told us about when you're not in a lesson.

2.none

3.none

4.it takes time to log in and not everyone will have a computer

5. Not enough computers if the whole class turn up. We’ve got to go to other computer rooms and sometimes the teacher has to go between two classes and it can be hard to get help.

6.i forget what we were shown so i never look at it again and the internet is better anyway

7. Just finding what the teacher has put there as sometimes she puts it on the share drive and other times she puts it on moodle. it can get abit confusing.

8.none, sometimes its hard to find stuff you want.

9.none

10.None unless the computers crash.

14

answered question

14

skipped question

2

 

Table 9

3. How creative does Moodle allow you to be?

 

Response
Count

1. Don’t know about Moodle but using the computer can help you create better assignments that look profesional and you can find information on the internet about the topic you are studying.

2.don't know

3.don't know

4.don't know

5. I can write my assignments after using information from moodle but I get help from other places as well like the library and my teacher.

6.don't know internet is better

7. Not very creative. There's not much there apart from class notes and presentations for us to look at. mostly we use the internet to find information for our assignments and you don’t need moodle for that.

8.it doesn't allow me to be creative, its just got class resources for me to use in my work

9.don't know

10. I can collect information from all over like google and decide what to use in my work. It's easier than collecting information from text books, and college hasn't got as many text books as you can get on the internet.

14

answered question

14

skipped question

2

 

Table 10

4. How does Moodle help you achieve your learning objectives?

 

Response
Count

1. In some lessons it's better than others as some teachers know how to use it properly. You get more handouts and links to help you with your work which helps you achieve a good grade.

2.don't know

3.it gives us information to help write our units that we haver to pass to pass the course

4.it has information about the units so we can use that to help pass them

5.It helps me complete my assignmnets and get good grades

6.it doesn't

7.it doesn't

8.it doesn't really help me achieve as i said above its just resources, but it can help me learn about my course like a text book only on the computer

9.don't know

10. I can get better grades because there's more information to use.

14

answered question

14

skipped question

2

 


Observation

 

Lesson Observation: Applied Health & Social Care (Level 3)

Lesson duration: 1 hour 40 minutes

Number of students present: 10

Activity:

  1. Reviewing PowerPoint presentation (it was delivered by teacher in previous lesson)
  2. Answering questions related to the presentation content and an assignment brief. 
  3. Internet research undertaken with starting links provided in a Word document.

 

The resources above were all accessible from a subject link in Moodle.

Lesson Objective: Students to complete eight questions by the end of the lesson.  Questions to be presented in a Word document.  The teacher intends to discuss the responses in the following lesson.

The numbers recorded represent the number of times the checklist statement (in bold) was observed. Additional comments (in italics) are also included for clarification of the observation.

 

Curiosity and creativity:

Table 11

Total

Students make use of support – online

0

Students make use of support – face to face

Where to find suitable information within a web site, checking that content is relevant, asking for clarification on content, asking for a check on written answers, asking for help with search criteria

6

Students find interactive materials intuitively

Students find materials intuitively – all the students seemed to navigate to the right subject area within Moodle without prompting.

0

Students become distracted from task – unrelated to subject area

YouTube

3

Students become distracted from task – related to subject area

Found a link to an interactive game

1

Students incorporate other ICT skills into their learning task – without prompting from teacher

Search engine browsing, saving web sites as favourites, sharing answers by emailing each other, copy & paste, formatting

10

Students incorporate other ICT skills into their learning task – with prompting from teacher

Search engine browsing tips, page numbering, use of bullets

3

Students ask questions related to task/subject (not ICT related)

Mostly related to content of web sites but also clarification of purpose of task, checking of written answers

7

Students complete task and stop

Printed answers and handed work in

1

Students complete task and continue to explore

Unrelated content – YouTube, email

2

Students do not complete task

Six students completed five questions, one student completed seven.

7

Students provide support/help to other students (not ICT related)

Sharing web links with useful information.

4

Students ask questions to improve their use of the technology

Request on how to include a web link in the Word document.

1

Students show signs of frustration with the technology

Scrolling horizontally in a web site, problems setting up a table in Word, problems drawing a simple diagram in Word

 

3

 

Achieving learning outcomes:

 Table 12

Total

Students complete the task

Three students finished the task.

3

Students demonstrate learning through the task (i.e. they answer questions by the teacher or complete written/verbal responses during the task)

All students completed some written answers and made verbal responses to the teacher.

10

Students achieve a grade or score

Grading did not take place in this lesson

0

Students are pleased with their grade or score

Grading did not take place.  However, students made positive comments about their efforts (“look at how much I’ve done”, “I can use that in my assignment”)  and two that did not finish were making arrangements to use the library computers to complete the task before the next lesson.

0

 

Other observations:

After viewing the presentation and downloading the questions, students tended to close Moodle down and concentrate solely on the Internet Browser and their Word document.