�Formative Evaluation for Intelligent Quality Management in an Education Program.�Case Study�

​

Dimitris Tsipianitis, Georgios Mandellos

dtsipianitis@ece.upatras.gr mandello@upatras.gr�Electrical & Computer Engineering Department, �University of Patras, Greece

​

​

Hellenic Society for Systemic Studies, 15th HSSS National & International Conference Systemics and Business Intelligence

The Hellenic Society for Systemic Studies (HSSS)

�15th HSSS National & International Conference�Systemics and Business Intelligence��Department of Informatics �University of Piraeus�29-30 November 2019��

Presentation Overview

• Theoretical Frame
• Evaluation Importance
• Literature Review
• Our Case Study & The Evaluated Program
• The Evaluated Program
• Evaluation Axes & Stages
• Methodology / Carrying Out the Evaluation
• Evaluation Indicators
• Data Collection, Questionnaire & Observation
• The Structural Model
• Discussion & Conclusion

​

​

​

​

​

​

​

​

Theoretical Frame

• What Evaluation is?

Evaluation is a process that compare the results with the goal, an initial situation with the final one. “Evaluate” means making an assessment based on some measure or information and referring to some criteria.

• What is Formative Evaluation?

The Formative evaluation is a process used by trainers and trainees in teaching that provides feedback on updating current teaching and learning methods in order to improve trainer’s achievement in relation of teaching goals.

​

Evaluation Importance

There is plenty of literature reviews on the purposes and functionality of evaluation but the research on the instructor’s beliefs and particularly on trainee’s beliefs is limited, (Struyven, Dochy & Janssens, 2005).

However, the study of trainee’s perceptions about the evaluation has particular importance because they have significant impact on the quality of a learning program.

Literature Review

Numerous definitions of Formative Evaluation are identified in the literature.

• Is a method that reflects all training practices that carried out in a training program by the trainer or/and trainees. Both of them are involved in this process. (Black & William,1998).
• According to Van De Walle, Karp & Bay-Williams (2013), a teaching-based evaluation process that controls and regulates who learns and who does not and how the trainer could helped for his new program.
• In order to be the Evaluation Formative must includes schedules for future actions. (William, 2007)
• According to Sadler (1998), formative evaluation refers to the evaluation with main target to provide feedback about trainess' performance, in order to improve and speed up their learning capability.

Our Case Study

In this case study a research was carried out using the technic of Formative Evaluation in order to evaluate two thematic areas with the cooperation of the program’s leaders.

The aim of the study was to extract conclusions and opinions about the program’s value, the achievement goals and the organization implementation requirements.

Our results will been feed backed to program managers and policy makers with the appropriate information which are necessary to adopt appropriate corrective interventions for programs quality improvement.

The Evaluated Program

The evaluated program is a large-scale and high-quality training project for employers, self-employees, employees of medium small and very small enterprises (SME’s) commercial and craft businesses from six cities in Greece.

As part of this project 106 training seminars took place, and 1300 trainees will be educated.

The training activities concerned the development of small business entrepreneurship and the knowledge of IT applications.

The project was part of Operational Program “Education and Initial Vocational Training”.

Evaluation Axes

• The preparation of the Program and the Feasibility Study.
• Publicity and dissemination actions.
• The trainers’ readiness, the training teaching material and the way of implementing the training program.
• The effectiveness of the used educational techniques.
• Assessing the adequacy of the factors which involved in the project.
• If the trainees were satisfied with their participation to the program and their believes that they earned benefits for their work performance.

Evaluation Stages

• Design the Evaluation
• Investigation of Needs
• Objectives Defining
• Selecting the type of Evaluation
• Schedule and Budget
• Currying out the Evaluation
• Identifying the axes & Structures
• Selection of methods & techniques & data collection & analysis
• Interpretation of results & suggestions
• Publication of Results
• Selection of recipients of the evaluation results
• Writing an evaluation report
• Communication of results to recipients

Methodology

• The data collection based on field observations during the implementation of the programs
• Sample interviews were conducted with the trainers and with others who involved in the program
• Analysis of project implementation documents
• Completion of questionnaires by trainees

Carrying out the Evaluation (I)

1. Evaluating of Project Implementation and Management Guide

This is the key for the implementation and quality assurance of the project. Also examines the targets and the design according to the needs of the trainees.

2. Evaluating the Project dissemination

How the program Promote and Disseminate itself and what actions have been taken in order to inform the target group and how the program promoted.

3. Evaluating the Trainers

Τhe trainers readiness, did the use the appropriate training techniques? What about the educational material?

​

​

Carrying out the Evaluation (II)

4. Training Technics

How the training technics used? Were they the appropriates? What about their effectiveness and how they promoted the active involvement of trainees?

5. Human Resource

Quality and evaluation of administrative support, management, communication and organization. The infrastructure and the selection process of trainees.

6. Satisfaction of trainees

Were the trainees satisfied with their participation to the program? Could this training program had a significant benefit in their job performance?

​

​

Evaluation Indicators

• The key indicators for evaluating the project implementation and management guide as well as participants' needs are indicators for relevance and completeness.
• About the teaching material evaluation the key indicator is about the adequacy of the material. If the books meet the levels of understanding and consolidation.
• Satisfaction indicators were used through written comments from both trainees and trainers on apology or inadequacy about teaching material.
• Participation Indicators were used in order to evaluate the trainers, the training technics that they used, the right choices and the variety of them.
• Familiarity and Honestly Indicators between trainers and trainees and how the first encourage the second for participation.

Data Collection

The data collection was based on quantitative and qualitative techniques.

​

• Field observation during the implementation of the programs using the technique of sample interviewing of trainers, administrators, managers and other involved in the program.
• Analysis of project implementation documents as well as filling in questionnaires by trainees.

​

The purpose was to identify differences from the original design, in order to suggest proposals for achieving the original objectives and alternatives solutions.

Questionnaire & Observation

• The questionnaire to trainees and trainers examines issues such as the teaching material as well as the beliefs of the trainers that the gained knowledge will be useful to their Jobs.
• The observation technic applied in order to evaluated the trainers teaching efficiency and their CV’s as well.

​

Our structural model of Formative Evaluation

Discussion

Conclusion

Discussion

The project's strengths and innovations highlighted, as well as malfunctions and weaknesses.

Proposals have been formulated in order these conclusions to be used for better and effectively implement future actions.

The design and implementation highlights the know-how of program executives and their partners and the long-term strategy for developing teaching programs in the field of small and medium enterprises.

​

Communicating the Results

• The results, the conclusions and the proposals of program evaluation must be communicated to all who involved to it.
• The dissemination is absolutely necessary and demonstrates the seriousness of the Project Implementing Entity.
• The publication of the results of the program also includes the publication of the findings for the future actions.
• Finally itt should be clearly stated and accompanied by an additional effort to motivate the awareness of the people of small and medium enterprises to participate in future educational activities.

Conclusion

The systematically study of trainers and trainees beliefs helps to have a complete picture about the programs through the formative evaluation and how to use all this data in order to design appropriate models for development indicative practices for effective application of formative evaluation in teaching and learning.

Finally … and Future Actions

Through the study of trainers beliefs combined with this of trainers as it described, make more possible the change of trainers norms and practices as well as trainees about the formative evaluation because all these relationships (trainees, trainees and teaching technics) are dynamics and two ways. (Fernandez, 1997).

​

Future research and similar actions must focused both to trainers and trainees beliefs in order to have a complete opinion about the use of Formative Evaluation and how it helps the training process. These beliefs could be used for designing or managing intelligent and quality models for educational programs for the effective implementation of Formative Evaluation.

​

Thank you…

…for your Attention

​

Any questions??

Please e-mail me …

dtsipianitis@ece.upatras.gr

​