Sample Evaluation Report

For

FDP Design Activity on

Peer Instruction

ANALYSIS OF ENGAGEMENT        3

A) What was the gross engagement in the course? [Macro level]        3

B) What was the engagement pattern of individual LeDs, LbDs, LxIs, and LxTs? [Micro Level]        4

C) What is the transition pattern of engagement of learners across LeD, LbD, LxI, and LxT?        6

ANALYSIS OF LEARNING        8

A) What is the gross learning in the course?[Macro]        8

B) What is the learning pattern across LbDs and Quizzes? [Micro]        9

C) What is the transition pattern of learning across LbDs and Quizzes? [Meso Level]        10

ANALYSIS OF PERCEPTION        10

What was the perception of learners about the course? [Macro]        10

REFLECTIONS ON ANALYSIS        15

What do gross data about learning, perception and engagement tell about the course?        15

What does the analysis tell about the course experience?        15

What needs improvement in the current course offering?        15

WORKLOAD DISTRIBUTION        16

1. ANALYSIS OF ENGAGEMENT

A) What was the gross engagement in the course? [Macro level]

The MOODLE course on peer instruction (PI) was originally designed as part of a blended course which was to be taken by participants of STTP (Short Term Training Program). There were around 30-35 participants who took the course as part of the STTP. Around 100 participants from FDP301x have also registered for the course which makes the total number of participants around 130. The course involved the following activities that were to be completed by participants during the duration of the course:

• LeDs on PI- 5 Nos
• LbDs on PI- 5 Nos
• Discussion on Peer Instruction
• Reflection Quiz (RQ)
• Advancing Your Learning (Lxt) - 3 Videos and Resources
• Assimilation Quiz (AQ)
• Knowledge Quiz (KQ)
• Activity- Designing your own PI
• Peer review of PIs
• Evaluation and Feedback

The activity completion report generated from the course Reports section shows the engagement of participants in the activities as shown in Table 1.^[1] 

Table 1: Activity Completion Summary

B) What was the engagement pattern of individual LeDs, LbDs, LxIs, and LxTs? [Micro Level]

Looking into the performance and engagement of individuals in the course activities, following is the engagement pattern of selected participants in the course. Three participants were selected for the analysis so that we have a sample groups showing high engagement, medium engagement and low engagement pattern. Table 2 shows the level of engagement of the three sampled participants in the course.

Table 2: Participation of individual learners^[2] in the course

C) What is the transition pattern of engagement of learners across LeD, LbD, LxI, and LxT?

^[3]

2. ANALYSIS OF LEARNING

A) What is the gross learning in the course?[Macro]

Analysis of the learning of the students is done by analysing the marks obtained by the participants in attempting various quizzes and activities. There were five types of quizzes and activities for the participants. The weighted marks for each of them is given below.

1. Learning by Doing (LbD) Activities- 5 Nos         [20 marks]
2. Reflection Quiz                                [15 marks]
3. Assimilation Quiz                                [15 marks]
4. Knowledge Quiz                                [25 marks]
5. Peer Instruction Design Activity                [25 marks]

                        Total                                        100 marks

Table 3: Number of participants who attempted each activity and the average, marks and percentage they scored.

B) What is the learning pattern across LbDs and Quizzes? [Micro]

Table 4: Number of participants who have attempted each LbDs. The total is calculated in 20 as it was the weightage given to the LbDs in the whole course scoring.

Table 6: Number of participants who submitted the peer instruction design activity, peer reviewed the activities and the marks they have been given. The total is calculated in 25 as it was the weightage given to the Design activity in the whole course scoring.

C) What is the transition pattern of learning across LbDs and Quizzes? [Meso^[4]]

3. ANALYSIS OF PERCEPTION

1. What was the perception of learners about the course? [Macro]

We can analyse the overall perception of learners through their feedback in the three surveys^[5]:

• Perception of Learning
• Perception of Engagement
• Perception of Usability

Perception of Learning

For each of the parameters to analyse perception of learning, we can see that the participants seem to have a good perception about the learning offered by the course.

Talking about the learning avenue offered by additional resources/reference materials, some of the learners are neutral about their learning potential. This suggests the need to have a fresh look at the materials and check if they really offer in-depth knowledge of the topic.

Fig 1la. Perception of Learning

Fig 1lb. Perception of Learning

Perception of Engagement

It is clearly evident that the perception of engagement has been low.

Observing each criteria, we can find that the participants are not easily able to get actively involved each time they access the course and don’t find the additional resources/reference materials engaging enough.

Fig 1ea. Perception of Engagement

Fig 1eb. Perception of Engagement

However one positive aspect are the quizzes/activities. From Fig 1eb it can be observed that although the opinion about engagement is diverse, a good number of learners find the quizzes/activities mildly engaging.

Fig 1ec. Perception of Engagement

Perception of Usability

Here’s a visual summary of the survey responses of participants.

Fig 1ub. Perception of Usability

Fig 1ub. Perception of Usability

Fig 1uc. Perception of Usability

From the graph, we can analyse each of the 9 categories used to evaluate the usability and further drill down into the response of each participant.

Going by the bar graphs, we can see that most of the participants are Mildly and Strongly agree about the usability of the course, except for the consistency in course material design (Fig 1ub) where the response is Neutral.

REFLECTIONS ON ANALYSIS

1. What do gross data about learning, perception and engagement tell about the course?

The gross data about engagement shows that the number of participants completing the activities/quizzes have decreased as the course progressed.

Also, from the data on scores of participants, we can see that the scores of the participants are gradually fluctuating, although the number of participants attempting the final Knowledge Quiz (KQ) has drastically fallen (6% of the total participants enrolled). This indicates that the perceived learning value of the course needs improvement.

From the participants' perception of the course, it can be observed that the pace of the course and consistency of course material design needs improvement.

2. What does the analysis tell about the course experience?

The analysis shows that the course wasn’t engaging enough and the levels of usability can be improved, although the levels of learning were not that bad.

3. What needs improvement in the current course offering?

The following sections need considerable improvement:
Engagement:
•        Engagement in Discussion forums
•        Activities and Quizzes
Usability
•        Consistency in course material design
•        Duration of the session
Learning
•        Reference materials/additional resources

4. WORKLOAD DISTRIBUTION

The team consisted of 4 participants. The workload is distributed among them as follows:^[6]

1. Member 1: (Team leader)

1. Preparing the workload distribution
2. LeD 1,2,5;
3. LbD 1,2,5;
4. Uploaded Led 1,2,5 and LbD 1,2,5 to moodle.
5. Survey report on Perception of Engagement, Learning and Usability
6. FDP design report: reviewing and compiling reflections by team members

2. Member 2

1. LeD 3,4;
2. LbD 3,5;
3. Uploaded  LeD 3,4, and LbD 3,4, RQ, AQ, KQ
4. Moodle Gradebook setup and configurations
5. FDP design analysis report: analysis and reflection of engagement

3. Member 3

1. LxTs,
2. Assimilation Quiz
3. Feedback form
4. Moodle administrator (setup)
5. FDP design analysis report: analysis and reflection of learning

4. Member 4

1. Course Design Layout
2. LxIs,
3. Reflection Quiz
4. Knowledge Quiz,
5. Activity Design
6. FDP design analysis report: analysis and reflection of perception