George Mandellos, Dimitris Tsipianitis

mandello@upatras.gr, dtsipianitis@ece.upatras.gr ��Electrical & Computer Engineering Department, �University of Patras,

Greece �

A Business Intelligent (BI) Connection

between

Quality and Evaluation methods.

Case Study

Hellenic Society for Systemic Studies, 15th HSSS National & International Conference Systemics and Business Intelligence

The Hellenic Society for Systemic Studies (HSSS)

�15th HSSS National & International Conference�Systemics and Business Intelligence��Department of Informatics �University of Piraeus�29-30 November 2019��

Presentation Overview

• Introduction
• Surveys as BI tools
• Surveys
• Questions that need answers
• Involved factors depending on survey administration
• The experiment
• Discussion on experiment results
• Future research
• Conclusions

​

​

​

Introduction 1/4

• The past decades the technology has noted an exploded advance.
• Technology is involved on each everyday action.
• Internet usage and computer-mediated communication has also made incredible advance and has changed the way people communicate.
• The internet technology, has influenced the way professionals conducts the customers.

Introduction 2/4

An area which internet technology has influenced in large scale is the Business Intelligence (BI).

​

BI is a philosophy that leverages software and services to transform data into usable knowledge that inform an organization’s strategic and tactical business decisions.

Introduction 3/4

BI tools access and analyze data sets and present analytical findings in reports, dashboards, graphs and maps to provide users with detailed intelligence about the state of the business.

​

The term BI also refers to a wide range of tools that provide quick, easy-to-digest access to insights about an organization's current state, based on available data. 

​

Introduction 4/4

A helpful tool to collect useful data relative with the BI purpose is the questionnaires.

Nowadays, the survey conduction consists of:

• the traditional paper surveys (PS) that have lost a large percentage, and
• the online surveys (OS) that have enhanced their popularity.

BUT…

• Which type is more efficient?
• Are there pros and cons?
• Is the replacement of one method with the other achievable?

​

Surveys as a BI tool

In this paper:

• At first, we will investigate how each survey method achieves its goal and try to find if one is better against the other as a BI tool.
• The subject of this study is the Educational process.
• Secondly, we will present a research have done between students, using the same target group and content, asking them to respond on different survey types in order to study their behavior.
• As a result we will show the survey suitability as a BI tool, and how the used method affects users behavior/opinion and the quantity/quality.

​

Surveys

… are the greatest way to influence the decisions are made almost about everything.

• before a new product’s presentation, is used to modify its characteristics in order to be widely accepted.
• products or services can be evaluated using the appropriate survey.
• influence people's behavior. Before an election runoff, a survey gives the predicted results. Also, the presented results are often able to change the voter's intention to vote.

… can be distinguished in many categories, dependent on

• the content
• the target group,
• the geographic coverage,
• the distribution type.

​

Questions that need answers.

• What is the most accurate way to do research?
• Are all surveys able to give reliable results?
• How many time is required for the investigation and the data analysis?
• What about costs?

Involved factors depending on survey administration

• 2.1 Cost
• 2.2 Impact – Population – Target Group
• 2.3 Speed of distribution – time to complete – response rates
• 2.4 Reliability – Confidentiality
• 2.5 Flexibility
• 2.6 Presentation – Appearance

​

2.1 Cost

includes all stages from design, diffusion (publishing, sending, receiving), data analysis (collection, processing), to announcement of results.

• Paper surveys have high costs including the cost of paper, printing, postage, data handling and processing.
• Online surveys have lower costs using automated steps, from e-mail sending asking users for the participation, until the result extraction.

2.2 Impact – Population – Target Group

• Paper surveys,
• usually, have limited area coverage.
• can include participants from all age ranges.
• include all educational states and don’t require any filling expertise (can be text or picture oriented).
• can be filled everywhere (in a city or on a mountain), with only a pencil.
• Online surveys
• have unlimited geographically distributed participants.
• one limitation is the user’s ability to operate computers, smartphones or other electronic devices.
• a data network must be available in the place of examination.

2.3 Speed of distribution – time to complete – response rates

• paper surveys
• have longer postage/distribution times.
• there is a significant delay on data acquisition and processing.
• the time to complete is usually short. Respondents have to check only a few pages.
• the response rate is usually high.
• online surveys
• the email invitation introduces significant delay on participant’s response (postpones or forgets the completion).
• have a limited deadline, collecting only a part of responses, with negative consistence on the people’s behavior.
• the number of different screens and the scrolling inside screens require more time to be filled.
• the data processing is automated - the researcher is able to have sub results each time.
• the technology itself, can have negative effects on survey participation and finally outcomes.
• A slow internet-connection can discourage people from taking a survey,
• missing plugins (such as a Flash® plugin),
• enabled popup blockers, limit the participation.

2.4 Reliability – Confidentiality

Surveys are valuable only if answers are honest. Honesty and anonymity have a close relation.

​

• paper surveys
• respondents fill surveys without personal data, so forms are anonymous.

​

• online surveys
• users have to follow an e-mailed link, or
• initiate the surveys through a web-page
• survey system knows if a respondent has already done a survey.
• none knows if system keeps a relation between responses and respondents.

2.5 Flexibility

• paper surveys
• the respondents, usually are placed in the same public area, are asked to complete printed questionnaires in a stable format at the same time.
• online surveys
• participants have to complete the surveys at any time at any place individually and in private.
• although technology gives this flexibility, there are many surveys that require specific equipment (pc or laptop, or smartphone or any combination), in order to be completed, influencing the number of participants.

2.6 Presentation – Appearance 1/2

• paper surveys
• the questionnaires has a stable format.
• offers better resolution and a more attractive appearance.
• use a few pages.
• participant can calculate the amount of needed time.
• there is no filling error detection.
• free text fields, depending the user’s handwriting can be unreadable.
• the physiological state (health problems), the age, the educational status or the affective disposition of respondents, makes users find the paper surveys more convenient.

2.6 Presentation – Appearance 2/2

• online surveys
• technology itself affects enough OS surveys.
• displaying differences across devices, screen-sizes, resolution and operating-systems, influence how participants interpret questions.
• questionnaires consist of many sequential screens, with few questions per screen.
• users are asked to give their responses using radio buttons (only one answer is permitted), check boxes (multiple answers), drop down lists (pre-entered answers), text boxes (when a customized response is required). An error completion algorithm advices user to correct errors before continues to the next screen.
• implementations with automated screen forwarding when an answer is completed introduce errors after an accidental click.
• participants can’t calculate the filling time and effort this type of survey needs.

THE EXPERIMENT

The aforementioned factors have shown that the used survey method leads to different behavior. Is it true? Does it affects the usefulness of questionnaires as a BI tool?

So, we designed a comparative study between 360 university students, in order to investigate their behavior and the quality of their answers depended on the survey method.

We compared response rates, data completeness, honesty, stringency on answers. The experiment included two scenarios.

• Paper questionnaires filled in a classroom
• Online questionnaires filled in a laboratory room

General Survey Information

General questions

Respondents are more strict on Paper Filled surveys, based on their anonymity.

Teacher specific

Respondents are more strict on Paper Filled surveys, based on their anonymity.

This section contains questions that can affect the Teacher-Student relation in case of a non honest answer.

Lesson Specific

Respondents are more strict on Paper Filled surveys, based on their anonymity.

​

Infrastructure

Respondents give relative answers on different types of surveys, because the questions relate with teacher independent objects.

​

Students responsibility

Respondents give relative answers on different types of surveys.

​

Discussion on experiment results 1/4

The comparison between the traditional paper-based surveys and the web based on-line surveys highlights the differences on people’s behavior and opinion.

These differences can lead to the catastrophic evaluation of the project’s quality.

Using surveys as a BI tool, needs a carefully designed scenario taking in mind the following parameters...

​

Discussion on experiment results 2/4

Mentioning the consistency…

• Paper Filled questionnaires noticed the best rates, the respondents feel forced to do this job because the collector easily checks if the questionnaires are filled .
• Online Filled questionnaires had the worst score, some respondents avoid to complete the forms.

…seems that the online surveys have a minus.

Discussion on experiment results 3/4

Mentioning the correctness…

• Considering filling errors and blank answers, the technology helped users to avoid errors.

​

… seems that the online surveys have an advantage.

​

Discussion on experiment results 4/4

Mentioning the sincerity and honesty…

• Paper Filled noticed more strict evaluation , because of the safety of anonymity.
• Online Filled were more favorable as a result of the unguaranteed anonymity.

​

… online surveys makes respondents suspicious with the anonymity.

Future research

After the differences found during the aforementioned experiment a new research is a necessity, trying to find answers at questions as:

• Why each respondent’s behavior is different between paper and online survey?
• Only the anonymity affects their opinion or there are some other reasons?
• How each one understands the anonymity?
• What type of questionnaire is more familiar depending on the content / construction / implementation?
• The questionnaire’s content have a negative effect on the answers?

​

Conclusion

This paper tried to intelligently investigate how the questionnaire methods can affect the real project’s quality and which type of questionnaire administration gives more reliable results.

The majority of differences found on questions related with personal criticism (sections where the respondent’s anonymity plays a significant role).

As a result, the quality check of a project using questionnaires, needs attentive design, trying to avoid points that introduce uncertainties and are able to spoil the real opinion of the respondents.

​

THANK YOU . . .

. . . FOR YOUR ATTENTION

​

​

Have you any questions?

Email me at: mandello@upatras.gr