3 of 26

Please share your feedback

1, Was the course content easy to navigate and access on the website? Were there any technical issues that affected your learning experience?

2, How relevant was the course content to your policy-making context? Do you see clear opportunities to apply the concepts in your work? Would you recommend this course to other policy-makers in your country/networks?

3, Was the content easy to understand?Was there some content in the course that you already knew/had awareness of? And if so, how did the course enhance your understanding? Are there any other concepts you think should be covered in the course?

4, How long did it take you to complete the course?

5, What challenges, if any, do you anticipate in applying the course insights to your policy environment? What support or modifications would improve its practical use?

4 of 26

Research guides informed decisions throughout an EdTech venture's journey to impact.

Research impacts equity, scalability, and innovation in education.

Continuous research offers real-time insights into learners' evolving needs and the shifting demands of the educational landscape.

WHY RESEARCH IN EDTECH MATTERS?

So why is it so important to know the answer to the question if it works?

The questions, "Does it work?" or ‘Does this EdTech solution have a positive impact on learners?’ can only be answered by research. And by research, I mean a systematic, objective investigation, ideally by qualified independent researchers. Research goes far beyond product development and resource allocation in EdTech. Think of research as a compass that guides informed decisions throughout a venture’s journey to impact. It impacts equity, scalability, and innovation in education.

Imagine founding your team pouring months into developing an app without research. Their creation might need to be revised, wasting resources (time, money, energy) and potentially excluding specific learner needs. Research should indeed be an integral part of any EdTech venture, guiding informed decision-making throughout the product lifecycle.

Continual research offers real-time insights into end users' evolving needs and the shifting demands of the educational landscape. It empowers EdTech creators to remain agile, adaptive, and responsive to these dynamics.

Research also allows you to understand these diverse needs and backgrounds—the very foundation of an inclusive and accessible EdTech product. Education is not one-size-fits-all; learners come from diverse backgrounds with unique needs and challenges. Research can help you understand and address learner variability, ensuring that your Edtech solutions are inclusive and equitable. For example, by conducting research on the accessibility features of an educational app, developers can ensure that learners with disabilities have equal access to educational opportunities.

Effective communication goes beyond marketing messages. Research-backed evidence of an EdTech product's effectiveness becomes a powerful tool, attracting users and building trust within the educational ecosystem.

Scaling successful models and driving innovation requires understanding what works and why. Take Kahoot! Or Age of Learning, for example. Their research-based approach to personalised learning fueled their global reach, impacting millions. You can replicate these models and drive further innovation by understanding the "why" behind successful implementations of multi-million EdTech organisations.

5 of 26

What is evidence?

6 of 26

Evidence definitions

Obtained by rigorous, independent and objective research

By research we mean rigorous, independent and objective evaluation of the EdTech’s use on children’s learning
Research is conducted to find out when and how an EdTech works
For which children does it work best?
For which skills?
For which types of teaching?

Use can be by the child alone (independent learning) or in the classroom (collaborative learning)
Teachers’ implementation of EdTech in their classroom (their pedagogy) can be part of the evaluation study

7 of 26

Types of research-based evidence

Summative

Formative

Foundational

Surveys

Teachers’ product reviews and feedback

Literature review

Desktop research

Case study

Observational study

Usability study

Correlational study

Experiment

Randomised controlled trial

Figure adapted from: Zielezinski, M. (2019) Demystifying evidence in EdTech, presentation at EdSurge Fusion 2019, Available from SlideShare: https://www.slideshare.net/molly_bullock/demystifying-evidence-in-edtech

Interviews

Feasibility, usability and implementation studies

Impact evaluation and validation studies

Longitudinal intervention

8 of 26

Three stages of evidence-gathering

9 of 26

IMPLEMENTATION RESEARCH

PROBLEM ANALYSIS: IDENTIFYING CHALLENGES

FIDELITY OF IMPLEMENTATION:

USING THE SOLUTION AS INTENDED

EFFECTIVE COMMUNICATION: ENSURE CONSISTENCY

STAKEHOLDER ENGAGEMENT: GATHERING INSIGHTS

AND BUILDING OWNERSHIP

EdTech implementation isn't a straight line. It's a cyclical and iterative process. Established practices within educational contexts heavily influence how EdTech solutions are used. To ensure success, companies must understand these contextual variations and the factors that impact implementation.

Here's where research on Edtech interventions comes in. This research, typically involving qualitative and smaller-scale quantitative studies, enables you to identify barriers and enablers, as well as contextual variability. Reflect on the question: How do different educational contexts (e.g., school size, location, resources) influence the implementation of my product?

The Centre for Evidence and Implementation outlines four key stages for effective implementation, starting with:

Problem Analysis: Identifying Challenges

This analysis involves a deep dive into the educational context, organisational structure, and specific needs of educators and learners. Typically, it involves assessing technology infrastructure, accessibility, teacher training, student engagement, and much more. The problem analysis is crucial for developing targeted strategies and solutions that ensure effective and sustainable EdTech implementation, ultimately enhancing teaching and learning experiences.

For example, a systematic literature review helps you identify potential roadblocks to successful implementation in similar contexts. For example, it might reveal challenges like limited internet access or lack of teacher motivation to use new technology.

By synthesising existing research findings with our own problem analysis, you can gain a holistic understanding of the conditions, limitations, and potential solutions for successful implementation. This knowledge is then further refined through ongoing dialogue with key stakeholders involved in the EdTech solution.

Effective Communication: Ensuring Consistency

Clear and consistent communication is essential for successful implementation. It ensures all stakeholders adhere to established standards and procedures, critical for achieving "fidelity" in the intervention.

Fidelity of Implementation: Using the Solution as Intended

Fidelity refers to the extent to which the EdTech solution is implemented as designed and in accordance with best practices. Maintaining fidelity through ongoing dialogue between all stakeholders is crucial. It allows for meaningful comparisons across different contexts, maintains scientific rigour, and fosters shared commitment among educators and EdTech creators.

Stakeholder Engagement: Gathering Insights and Building Ownership

Effective stakeholder engagement involves understanding attitudes and perceptions through in-depth qualitative studies and interviews. It's about empowering stakeholders to feel motivated and take ownership of the implementation process. Gathering insights about the local context through user interviews, surveys, or official policy documents feeds into monitoring and evaluation impact metrics, allowing you to track progress and assess the overall effectiveness of the EdTech solution over time.

By following these steps and continuously adapting based on research and stakeholder feedback, EdTech solutions can have a positive and lasting impact on educational outcomes and more broadly, environments.

10 of 26

MONITORING AND EVALUATION:

THE 4 RS

RELEVANCE:

ALIGN WITH GOALS AND LEARNING OUTCOMES

REGULARITY:

CONSISTENT TRACKING OF PROGRESS

RELIABILITY:

TRUSTWORTHY AND RIGOROUSLY SCRUTINISED DATA

RESPONSIVENESS:

ADAPTABILITY TO

CHANGING NEEDS

And establishing robust mechanisms for monitoring and course correction is crucial in all contexts. This process relies on being data-driven, with clear indicators reported at specific frequencies and clearly identified data sources. This allows you to continuously assess the effectiveness of the EdTech solution and make adjustments as needed.

Now, let's examine the principles underpinning effective monitoring and evaluation processes. These might be referred to as the 4Rs: relevance, reliability, regularity, and responsiveness.

Firstly, relevance ensures that the data collected aligns with the goals of the organisation’s monitoring and evaluation practices. You should not be gathering data for the sake of it; you should be focusing on outcomes that genuinely matter, encompassing essential skills, knowledge areas, and even socio-emotional competencies crucial for further learning and growth.

Secondly, reliability is critical. Ensure that the data you report is trustworthy and has undergone rigorous scrutiny. Reliable metrics uphold the integrity of our evaluations and empower us to draw sound conclusions from the impact evaluations.

Next, regularity emphasises the importance of conducting data requests at routine intervals. By consistently tracking learning trajectories and patterns and evaluating the progress of both learners and educators, we gain valuable insights into learner development. We can promptly identify any learning gaps that may arise.

Finally, responsiveness embodies the essence of adaptability and improvement. Your monitoring system should not be static; it should evolve based on feedback and be responsive to the ever-changing needs of learners and educators. This proactive approach ensures that our monitoring efforts not only provide real-time information but also actively contribute to enhancing learning experiences.

11 of 26

VALIDATION:

CONFIRMING IMPACT

ROBUST EXPERIMENTAL DESIGN

ADVANCED STATISTICAL ANALYSIS TO MEASURE THE CHANGE

MULTIDISCIPLINARY RESEARCH EXPERTISE

ALIGNMENT WITH IMPLEMENTATION INSIGHTS AND CONTEXTS

Finally, once you confirm that the product works as anticipated and explore whether learners are using it appropriately and the context is appropriate, you arrive at the validation stage. Here, you can validate the change defined earlier in your Theory of Change and Logic Model.

The cyclical nature of EdTech implementation involves ensuring that educational technology leads to measurable, positive impacts on users. While we would recommend partnering up with qualified researchers for all stages, it is most pertinent at the validation stage. Researchers who can create an intervention plan, determine eligibility criteria for participants, establish an experimental protocol, and set up data collection processes. While these elements are consistent across interventions, their specifics vary based on their purpose and context. For instance, an intervention plan may specify participant eligibility criteria based on the theory of change established earlier, combined with usage patterns provided by the EdTech company's insights.

The methodological approach to interventions varies and must align closely with factors identified during the implementation and scaling phases. Researchers prioritise feasible and acceptable approaches within a context, considering envisioned impact measurements and synthesising insights from relevant literature. Balancing intended impact with feasible measurement is essential. Adjustments to outcome measures may be necessary to align with a specific context, and statisticians can advise on selecting appropriate approaches.

For example, in regions with severe teacher shortages, interventions may require EdTech-trained facilitators instead of traditional educators. While this adjustment may deviate from typical scenarios, it ensures consistent testing of student outcomes by certified professionals. Statisticians are also crucial in establishing links and patterns between EdTech utilisation and resulting outcomes, facilitating robust efficacy analysis. You see, a good research study requires several types of research expertise to be brought together, including EdTech researchers who are skilled at various methodologies, domain experts and in the case of efficacy studies, also statisticians.

12 of 26

Why is evidence necessary?

13 of 26

Current research shows limited evidence of EdTech’s positive impact on learning and teaching

See UNESCO GEM Report and the UNESCO slides

Examples of research studies:

14 of 26

EDUCATORS

COLLABORATIVE EDTECH ECOSYSTEM

EDTECH

POLICYMAKERS

INVESTORS &

VC FUNDS

EDTECH CREATORS

USERS/ LEARNERS

RESEARCHERS

EdTech quality can be raised if we work together as one ecosystem and in partnerships

Kucirkova, N. (2024). A Partnership Industry for Impactful Ed-Tech. Stanford Social Innovation Review. https://doi.org/10.48558/58NS-6760

�

Collaboration within the EdTech ecosystem is critical for EdTech solutions' development, evaluation, and success. Input from users, including teachers, students, and subject matter experts, is essential during the creation phase to ensure that the design effectively supports learning experiences. Teachers and students offer insights into their needs, preferences, and challenges, while subject matter experts contribute expertise in content development and pedagogical principles.

Moreover, collaboration with researchers and data scientists is crucial for navigating the intricacies of evaluation and assessment. Researchers can design rigorous studies to evaluate the impact of EdTech products on learning outcomes, while data scientists can analyse large datasets to uncover valuable insights. Their expertise ensures that EdTech products are on track to positively impact learning outcomes.

One thing I want to highlight is that collaboration with investors and policymakers is vital for establishing and ensuring quality standards in the EdTech space. Investors provide funding and resources to support the development and scaling of innovative solutions, while policymakers create regulations and guidelines to safeguard educational quality and equity. Their collaborative efforts contribute to transparency, trust, and positive change for learners by promoting the adoption of effective and ethically sound EdTech solutions. So, choose your investor wisely and check that their commitment to research aligns with yours.

By leveraging research-based approaches, EdTech solutions can drive meaningful improvements in education and empower learners worldwide. After many years of a fragmented EdTech landscape, it is time to embrace the transformative power of collaboration and evidence-based decision-making to shape the future of learning.

Finally, I invite you to take the next step in your journey towards research readiness. In the description of this video, you'll find a link to assess your organisation's level of research readiness, according to international criteria of efficacy and effectiveness. The tool was developed by researchers at WiKIT AS. Let's join forces and work together to enrich the EdTech ecosystem with knowledge and evidence, laying the foundations for innovative solutions that truly make a difference in education. Together, we can create a brighter future for learners everywhere.

15 of 26

Support for EdTech companies

16 of 26

https://www.foreduimpact.org/playbook

The EdTech Impact Playbook is a free set of tools, resources and templates that EdTech enterpreneurs at any stage (early startup or advanced product) can access at their own convenience. The Playbook was put together by WiKIT and and Owl Ventures, the world’s largest investor in educational technology.

17 of 26

https://www.foreduimpact.org/selfassessment

The Self-assessment tool is a free guided questionnaire, where companies can fill out their current practices, evidence progress and plans for research studies and get a customised report with recommendations on how to improve their impact and evidence.

18 of 26

Certifications

19 of 26

Why are certifications necessary?

Certifications provide a clear and simple signal to schools and decision-makers about which EdTech tools have been independently verified for quality

As long as they are voluntary, certifications support innovation by positively motivating companies to achieve higher standards

They can be national (for example Austria has developed their own system) or international (for example EduEvidence operates globally)

They can focus on pedagogical criteria and expert evaluations (for example EdTech Tulna) or on research-based criteria (for example ESSA certifications)

20 of 26

ICEIE is the only independent, international and research-based EdTech certification in the world.

ICEIE works with established certification providers worldwide by uniting them under one shared umbrella

ICEIE is funded by the Norwegian government through Innovation Norway

www.eduevidence.org

ICEIE is a not-for-profit organisation established to respond to the missing code of practice for certifying “what works” in educational technologies globally.

About the International Certification of Evidence of Impact in Education – EduEvidence

21 of 26

EduEvidence Certification principles

All officially recognised international certifications in one place

Two-step validation process to ensure independence

Fair and transparent process with fees proportional to the company size

Criteria based on weight of evidence of each certification

Evaluations performed by only qualified researchers

All evaluation rubrics available for companies to try and follow

A global catalogue of evidence-based edtech solutions

22 of 26

DATABASE of

certified products

Congratulations !

Certified companies get a badge and certificate

23 of 26

https://certifications.eduevidence.org/

Three levels based on weight of evidence (Gold, Silver, Bronze)

Total score aggregating performance across all five dimensions

Global Catalogue of Evidence-Based Resources

24 of 26

Efficacy:

What is the EdTech’s impact on

learning, social, and economic

outcomes?

Effectiveness:

To what extent does the EdTech

address cost, local and

infrastructural issues?

Equity:

How much does the EdTech promote

social justice, equitable use across genders

and historic inequalities?

Ethics:

Is the EdTech’s approach to engaging with communities and partners within the ecosystem ethical?

Environment:

To what extent does the EdTech company evidence

its care for life on land, under water and climate change?

The EduEvidence evaluation is based on the scientifically validated 5Es evaluation framework

25 of 26

UNESCO, UNICEF, ITU and GPE (2024). Education Digital Transformation: Framework for System Transformation with Education Technology. Paris, France.

5Es Framework	Education Digital Transformation: Framework*
Efficacy	Coordination (monitoring & evaluation and Evidence-Informed Decision Making
Effectiveness	Cost Content & curriculum
Ethics	Connectivity & infrastructure (Cyber-security and Learning spaces)
Equity	Capacity & culture Sustainability (Ecosystem development)
Environment	Sustainability (Eco-friendly solutions)

UNICEF The EdTech for Good Framework

5Es Framework	UNICEF: The EdTech for Good Framework
Efficacy	Impact on learning (Theory, research, engagement)
Effectiveness	Designed for children (Pedagogical design, user experience) Ready to reach all (cost and affordability)
Ethics	Safety first (Data protection and privacy)
Equity	Leave no one behind Ready to reach all (localisation)
Environment	-

The 5Es Framework aligns with the global agencies’ approach

26 of 26

ICEIE

Org no. 932 595 036

https://www.linkedin.com/company/eduevidence/

Contact: Saila Juuti (Manager)

Saila.juuti@foreduimpact.org

EduEvidence is funded by the Norwegian government.

EduEvidence is happy to collaborate with EdTech governing bodies, national and international bodies as well as EdTech providers and alliances. EduEvidence can

Provide a research-backed framework, fully vetted by experts according to international standards and ready for local adaptation
Offer training for schools or government officials on selecting evidence-based tools and assessing impact in EdTech according to the certification criteria
Provide certification and offer preferential rates for certifying a cohort of companies or tools from your country or portfolio of companies.
Expand the catalogue by adding tools prioritized by national governments

Contact