Technology Learning Fund EQAO Math Report 2017

EQAO-based Pre and Post Results

For the Waterloo Catholic District School Board

06/04/2017

By: the TLFM Implementation Team

Overview: The Technology Learning Fund – Mathematics (TLF-M)        1

Research Question        2

Grade 3-5 Pre and Post-test Participants:        3

Grade 6-7 Pre and Post-test Participants:        4

Mean Scores        4

Effect Sizes        6

Percent Change Scores        7

Students who scored low on pre-test: Students scoring 8/13 or lower        9

Interpretation and Limitation        11

Overview: The Technology Learning Fund – Mathematics (TLF-M)

In the 2017, the Technology Learning Fund – Mathematics (TLF-M) at the Waterloo Catholic District School Board (WCDSB) bridged math curriculum learning and 21st century competencies spanning the junior division. Mathematics content knowledge and big ideas focused on geometry and spatial sense through the use of math-tech enabled resources.

The goal of the cross-school network was to construct understanding of the curriculum expectations/big ideas and continuum of learning from grade 4 to grade 6.  The project aimed to build capacity of effective teaching and learning of transformational geometry through classroom embedded collaborative inquiry that incorporated different school contexts, 21st century competencies and technology.

Six elementary schools participated in the TLF-M network, with four junior teachers representing grades 4, 5 & 6, including combined grades and cross-divisional grades such as 3/4 and 6/7. A total of 23 that completed the professional development.

Through 8 full day sessions, teams of educators conducted in-class investigations of student work and lessons in classrooms, using a collaborative inquiry process (i.e. plan, act, observe, reflect). This process aimed to have participants identify and build knowledge about mathematics instruction in reference to the use of technology and the development and assessment of 21st century competencies; that work to improve student learning and achievement.

The WCDSB action plan for the Ministry's Technology Learning Fund included the purchase of licenses for 6 schools to receive training on the New Pedagogies for Deeper Learning (NPDL) framework. In February 2017, educators participated in two full day facilitated sessions with a NPDL consultant and participated in 6 more sessions from February to May of 2017.    

NPDL is a global partnership, created by Michael Fullan, which includes 1000 schools in 10 countries (14 school boards in Ontario). Deep learning is mobilized by four elements that combine to form the new pedagogies; learning partnerships, learning environment, pedagogical practices and leveraging digital. The Deep Learning Competencies, (collaboration, communication, creativity, critical thinking, citizenship, character) are the skill sets each and every student needs to achieve and excel in, in order to flourish in today’s complex world.

In partnership with the WCDSB, researchers from Wilfrid Laurier developed an evaluation framework to understand the impacts of the TLF-M on teachers and students.

Methods: TLF-M Student Pre and Post Tests

Research Question

One of the questions from the TLF-M Evaluation Project is:

What are the impacts of the TLF-M teacher professional development on student learning and achievement?

This report shares the results of one attempt at capturing the impact on students through use of pre and post tests.

Description of Pre and Post Tests

A WCDSB Math consultant developed a pre-test for participating teachers to have students complete using a Google Form prior to implementing any of the new pedagogies in the classroom. The tests included transformation geometry thinking/application (only application for primary) questions from 2006-2010 - Gr 3 EQAO (13 qs) for Grade 3-5s. For participating Grade 7-8 students, Grade 6 transformation geometry EQAO questions were used (13qs).

All pre-tests were completed in class prior to session 3. The pre tests were analysed and individual and class results were used as assessments of learning to inform educator’s learning plans beginning in session 3.  

Post tests were completed within 2 weeks of the last session on May 8, 2017.

Results

Participants

Grade 3-5 Pre and Post-test Participants:

• 284 students completed both a Grade 3 EQAO-based pre-test and a post-test

• 8 Grade 3s
• 87 Grade 4s
• 189 Grade 5s

Grade 6-7 Pre and Post-test Participants:

• 173 students completed both a Grade 6 EQAO-based pre-test and a post-test

• 142 Grade 6s
• 31 Grade 7s

Mean Scores

Chart 1. shows the mean pre and post scores for each of the grade groups.

Notes about Chart 1 data:

• Grade 3-5 completed the Grade 3 EQAO-based test and Grades 6-7 completed a Grade 6 EQAO based-test of selected transformation geometry questions
• there were only 8 Grade 3s and 31 grade 7s who completed both a pre and post test, reflecting a relatively small sample size, thus: interpret with caution
• the differences between the mean scores presented in Chart 1 are statistically different at the p>.001 level^[1], with the exception of the Grade 7 means, that are NOT statistically different

Charts 2 and 3 show the mean scores on the pre and post tests by school.

Effect Sizes

Table 1 shows the effect sizes (calculated using r for non-parametric tests) for the differences between the pre and post mean scores, by grade.  Effect size quantifies the size of the difference between two groups and usually gives an indication of whether something is practically significant, rather than merely statistically significant.

Table 1. Effect Sizes by Grade

Interpreting effect sizes

Cohen (1988) recommends interpreting effect sizes as:

Small effect:                 r = 0.10

Medium effect:        r = 0.30

Large effect:                 r = 0.50

Percent Change Scores

Table 2 shows the percentage change in scores. Percent change shows the difference between the post vs. pre scores using percentages.   Percent change is one means of demonstrating program impact or conveying the magnitude of a change.

Table 2. Change Scores by Grade

Interpretation

• there was a 58.6% increase in Grade 3 test scores on the post-test compared to pre-test
• there was a 19.3% increase in Grade 4 test scores on the post-test compared to pre-test
• there was a 16.4% increase in Grade 5 test scores on the post-test compared to pre-test
• there was a 31.9% increase in Grade 6 test scores on the post-test compared to pre-test
• there was a 7.7% increase in Grade 7 test scores on the post-test compared to pre-test

Table 3. Change Scores by School

Students who scored low on pre-test: Students scoring 8/13 or lower

Overall, results showed the tests devised for pre and post measurements skewed high and were not normally distributed. This is partially because a number of students were completing tests that were aimed at younger cohorts: the Grade 4s and 5s completed Grade 3 EQAO questions and Grade 7s completed Grade 6 EQAO questions. Given the skewed results, there are ceiling effects on the overall means (less variance in scores, since many scored at the higher end at the pre test).

Thus, another way to see pre and post test change is to only look at students who scored fairly low on the pre-test. In this case, we are looking just at students who scored 8/13 questions correct on the pre-test

Table 4 shows the changes in pre and post scores, effect sizes and change scores for students scoring low (8/13 or less) on the pre-test.

Table 4. Changes on Pre and Post Test Scores for Students Scoring Low on Pre-Test

Table 5. Changes on Pre and Post Test Scores for Students Scoring Low on Pre-Test by School

Interpretation and Limitation

The results of the pre-post test analyses reveal that there were significant increases between the pre and post test scores. Given the short period of intervention (less than 3 months between tests), these increases seem to suggest significant student learning occurred related to transformational geometry.

In particular, students in most schools who had scored low on the pre-test saw significant improvement. Also, Grade 3s (though very few) and Grade 6s, for whom the tests were most developmentally appropriate for, seemed to see the most improvements.

A limitation is the use of Grade 3 and Grade 6 EQAO questions for students who were older than the intended grade level (use of Grade 3 test questions for Grade 4s and 5s and Grade 6 test questions for Grade 7s). Given the availability of options, this test was the best available measure of knowledge of transformational geometry.