Part 1 to be found here. Link to this document here.
And how do we measure it?
So, we may not all agree upon what students ought to learn in school, what is more important, factual knowledge or learning how to learn? But putting that aside for a moment, lets look at what is actually meant by the enhancement of learning actually mean, and how one could measure it? Lets start with an illustration (see figure 1)
Here the point is that, given that we could define what we mean by learning - what the students should learn, and given that there is a way to actually measure what they learned - the enhancement of learning could be defined as either a direct measure of increased fulfillment of learning goals or grades in a direct experimental comparison between different groups, or we could use more indirect evidence. For example, the learning activities could, with the help of Technology, be designed in such a way as to support those cognitive processes that favours effective learning. An activity could be multimodal, it could be found to be fun by the students and it could be a collaborative effort. All of these factors could, in the right circumstances support the learning process. So, we could gather this evidence by making interviews, doing surveys or make recordings of the students during their learning. Still we would not have direct evidence that the students did learn more, or with a higher quality. For that, we would need to define how to measure that, in some kind of experimental setup.
Another even harder thing to measure, due to the longer time scale before it could be measurable, would be learning practices where the students at the same time get to use (and hence master) methods that would increase the students possibility to learn more effectively in their future studies (or in life). For example, by learning to search for releant information in a critical and effective manner in a specific course might not show measurable results directly. These kinds of effects might not show up until next years grades will be handed out, if not long after that. Should we not encourage these kinds of methods just because the outcome of that would not show up in the grade envelope of the course in question? Or because it is hard to get direct evidence of their efficiency? Here once again it would be important instead to try and make indirect measurements by looking at the general tendencies - what types of activities can we claim to enhance the students learning processes in the long term? And what type of evidence do we have for that?
Another important aspect of this indirect type of evidence is that it is hard to know if the learning actually using alternative methods is on the expense of the fulfilment of other learning goals? Or on the expense of other learning activities due to limitations of time? The proposed Technology Enhanced Learning activity might take longer time to finish compared to a more traditional educational settings. How do we account for that if we do not have the final learning outcome in print? In terms of grades, or perhaps even final grades? Or perhaps we would measure it in terms of mean income a few years after the students finished school.
So, what should we aim to measure - the direct effects on the learning outcome in terms of grades etc, or the indirect evidence in terms of the properties of the learning activities and how they are perceived by the students? The answer one chooses limits the type of learning activity one can experiment with and gather evidence around, and hence could influence the type of education to be chosen. Once again, the question is contaminated by human values - what we would like for our students to learn, and if we would like for them to learn for the next test, or for life? Do not misunderstand me - I am all in favour of experiments and direct evidence if possible, but I also know that doing that kind of research is quite often not feasible.
One dimension of variables to measure could be how the learning activities complies with the factors that according to John Hatties “Visible Learning” influences student achievements, and especially how the introduced Technological enhancement could be claimed to support those factors. But how do we do that in practice on a larger scale?
In an upcoming blog-post I will talk about the global maker movement, as well as review the book “Makers” by Chris Andersson. The idéas proposed in that book could be used as one part of the solution. If teachers could only be made to share their lesson activity planning documents, and these could be shared and collaborated around, this could indeed increase the quality of the product (the educational activities) as well as make it possible to gather data on how different methods and technologies work in real educational settings. It would obviously be even better if the grades for classes could also be shared openly, but I guess now I am just dreaming.
In the next text I will be talking about the maker movement, how it could be used in the educational system, what aspects of it the educational system could benefit from, and how it could be supported by evidence from eg. cognitive science. The general discussion outlined in this text will be continued later on, but feel free to comment in this document.
 See: http://visible-learning.org/hattie-ranking-influences-effect-sizes-learning-achievement/ for more details.