1 of 16

Quality

Explanatory meeting on Chapter 18

12-13 February 2019

​

Claudia Junker

Head of Task Force "Peer reviews and quality", Eurostat

​

​

Eurostat

2 of 16

Overview

  • What is quality and quality in statistics?
  • ESS quality framework – legal framework and quality standards
  • Standards on Quality reporting in the ESS
  • Implementation of the ESS quality Framework

2

Eurostat

3 of 16

What is quality?

  • Fitness for purpose/use
  • Relative, multi-dimensional
  • Different definitions and approaches
    • ISO 9000: "Degree to which a set of inherent characteristics fulfils requirements"
    • ISO 8402:1986: "Totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs"

3

Eurostat

4 of 16

What is quality in statistics?

  • How well statistical processes and outputs fulfil stakeholder expectations in:
    • meeting their needs (statistical products and services)
    • addressing respondent concerns (burden, confidentiality)
    • following cost-effective procedures and sound methodology
    • ensuring the professional independence, impartiality and objectivity of statistical systems and their products and services

4

Eurostat

5 of 16

The quality framework of the ESS

Legal framework

    • Article 338 of TFEU
  • The production of Union statistics shall conform to impartiality, reliability, objectivity, scientific independence, cost-effectiveness and statistical confidentiality; it shall not entail excessive burdens on economic operators.
    • Regulation (EC) No 223/2009, amended by Regulation(EU) 2015/759

https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:02009R0223-20150608

Article 2 on statistical principles (professional independence, impartiality, objectivity, reliability, statistical confidentiality, cost effectiveness with reference to CoP)

Article 11 – European statistics Code of Practice (aims to ensure public trust, reviewed and updated as necessary, Commitments on confidence

Article 12 – Statistical quality (defines 7 quality criteria, legislation on quality reporting, quality reporting)

​

5

Eurostat

6 of 16

The quality framework of the ESS

Legal framework

​

Sectoral legislation including articles on quality reporting

https://ec.europa.eu/eurostat/web/quality/quality-reporting

'Self-regulatory' common quality framework

    • European statistics Code of Practice (Code, CoP)
    • Quality declaration
    • Quality Assurance Framework of the ESS (QAF)
    • General quality management principles

6

Eurostat

7 of 16

ES Code of Practice

  • Aim: to ensure public trust in European statistics by establishing how to develop, produce and disseminate European statistics in line with the statistical principles set out in Regulation (EC) No 223/2009 and best international statistical practice
  • Self-regulatory, based on voluntary commitment but referred to in regulation on European Statistics;
  • Applies to Eurostat, to National Statistical Institutes of the ESS and to other national authorities producing European statistics
  • Revised in 2017, 16 Principles and 84 indicators; includes the Quality Declaration of the ESS

http://ec.europa.eu/eurostat/web/quality/european-statistics-code-of-practice

​

7

Eurostat

8 of 16

Level 1 – Code Principles

  • Institutional environment
    • Professional independence
    • Coordination and cooperation�(NEW since Nov 2017)
    • Mandate for data collection
    • Adequacy of resources
    • Commitment to quality
    • Statistical confidentiality
    • Impartiality and objectivity

​

+ Level 2: 84 Indicators

​

  • Statistical processes
    • Sound methodology
    • Appropriate statistical procedures
    • Non-excessive burden on respondents
    • Cost effectiveness
  • Statistical output
    • Relevance
    • Accuracy and reliability
    • Timeliness and punctuality
    • Coherence and comparability
    • Accessibility and clarity

8

Eurostat

9 of 16

Quality declaration

  • Part of the ESCoP
  • Declares partnership
  • ESS operates under a strict legal regime
  • Robust, world-class and self-regulatory quality framework
  • Quality seen as competitive advantage
  • Commitment to statistical excellence

​

9

Eurostat

10 of 16

Managing and assuring quality

  • Quality management�The set of systems, frameworks and activities which are in place within an organisation to manage the quality of its products and processes – a broad range of activities
  • Quality assurance�Planned and systematic actions necessary to provide adequate confidence for an organisation that the products or services it offers meet the accepted quality standards - part of quality management

​

10

Eurostat

11 of 16

Quality assurance in the ESS

  • Level 1 = Code of Practice: Principles�(standards)
  • Level 2 = Code of Practice: Indicators�(how to implement and assess the standards)

11

  • Level 3 = Quality Assurance Framework - QAF�(collection of methods, tools, good practices which can be used to implement the principles and indicators)
  • Level 4 = Process-specific quality assurance�(adapted to process needs, e.g. EDP, GNI)

Eurostat

12 of 16

Level 3 -Quality Assurance Framework of the ESS (QAF)

  • Provides methods to implement CoP principles at institutional and process level
  • Currently being adapted to the revised Code, by May 2019
  • Will contain methods, tools and good practices for all principles

​

​

12

Eurostat

13 of 16

Quality assurance in practice�Levels 1-3

CoP Principle 8: Appropriate statistical processes

Appropriate statistical procedures, implemented throughout the statistical processes, underpin quality statistics

Indicator 8.2

In the case of statistical surveys, questionnaires are systematically tested prior to the data collection

​

ESS QAF Method 2

Testing of questionnaires.�Prior to data collection, survey questionnaires are tested by appropriate methods (questionnaire pretest, pilot in real situation, in depth - interviews, focus groups, interviewer support, etc).The response time (the interview length) is estimated at this stage, if necessary.

13

Eurostat

14 of 16

Standards on Quality reporting

  • Constitutes a quality standard
  • Data need metadata
  • Describe and quantify the quality of the statistics produced, based on defined quality criteria
  • Implementation in the ESS Metadata Handler according to:
    • Single Integrated Metadata Structure (SIMS) v2.0
    • Euro SDMX Metadata Structure (ESMS): user-oriented
    • ESS Standard for Quality Reports Structure (ESQRS): producer-oriented
  • ESS handbook for quality reports (new edition to come in mid-2019)
  • http://ec.europa.eu/eurostat/web/quality/quality-reporting

14

Eurostat

15 of 16

Implementation of the ESS Quality Framework - examples

  • ESS Peer Reviews - to assess how the Code is implemented (2006-2008, 2013-2015, 2020-2022)
  • Reports and improvement actions per country and Eurostat, followed by annual progress reports;

http://ec.europa.eu/eurostat/web/quality/peer-reviews

    • Quality and metadata reporting
    • In each MS: quality audits/reviews/assessments; quality guidelines, quality policy

15

Eurostat

16 of 16

More information at:

http://ec.europa.eu/eurostat/web/quality/overview

​

Thank you for your attention!

​

16

Eurostat