WHO Global Benchmarking Tool

Training course for regulators

Dr.S.Eswara Reddy, Joint Drugs Controller (I), CDSCO. 10^th April 2023

​

Outline

​

Background

​

​

​

Objectives

​

​

Expected Outcomes

​

WHA 67.20

​

​

​

Role of WHO

​

​

Global Benchmarking Tool

​

GBT-GRP

​

​

GBT- Emergency Preparedness

Background of the training

• Drugs plays imp role in Public Healthcare
• Drug Regulator to ensure Safety, Efficacy and Quality
• Pharma Industry- Indian public health and Global health
• All countries are looking at India ( High Quality/affordability of drugs)
• Quality of drugs exported (Pharmacy of the world)
• Enforcement Activities
• Signed MoU with USFDA, MHRA, PMDA, Russia etc (Sharing of information, Inspection reports….)

​

3

Functions…..Assessment

Functions:

• Marketing authorization
• Inspections
• License
• Vigilance
• Testing

Assessment?

• Science based decision making
• Consistency/Predictability/Transparency
• Compliance and enforcement

​

4

Objectives of training

• To make aware about WHO Global Benchmarking Tool used for evaluation of National Regulatory System

​

• To know significant additions made to GBT in the current version

​

• To keep knowledge abreast of the current material of evidence

​

​

​

​

​

5

WHO developed Global Benchmarking Tool

Role of WHO

WHA Resolution 67.20 �What WHO should do

7

To continue to support Member States upon their request in the area of regulatory system strengthening, including, as appropriate, by continuing to:

Benchmarking

THE AIM MAKES THE DIFFERENCE

BEST PRACTICES are never fully defined and will always evolve

• Comparative evaluation of systems and processes in individual agencies, against a set of indicators

• It is an opportunity to identify strengths and best practices as well as any area for improvement

Benchmarking

• Evaluation of a body of scientific or technical elements

to develop a judgment about something

• Designed to determine the non-compliance against mandatory

and specified regulatory requirements

• Usually associated to external evaluations
• Outcomes may result in sanctions

• Systematic, independent and documented process for checking against procedures, plans as well as regulatory requirements)
• Usually associated to internal evaluations

Inspection

Assessment

Audit

• Stable, well functioning and integrated regulatory system
• Eligibility for vaccine PQ
• WHO listed authorities (WLA)

1

2

3

4

5

WHO NRA 5 step capacity building model

Benchmarking process

Initial Institutional Development Plan

Institutional Development Plan

Application of Risk Based Approach

Performance evaluation of some or all regulatory functions

Global context: different assessment tools collecting information from Regulatory Authorities and affiliated institutions

• Within WHO
• PAHO

​

• PAHO Abrev.

​

• PAHO Medical device

​

• Vaccines

​

• Medicines

​

• Blood

​

• PQ Labs

​

• Medical device

​

• GGP

​

• Outside WHO
• BEMA

​

• IMDRF

​

• RAST

​

• RAPS

​

• PIC/s

​

• USP Rapid

​

• OMCL/ EDQM

​

Strategic direction

Development of the WHO Global Benchmarking Tool (GBT)

Objectives of the unification exercise

• Align the policy, scope, objective and methodology of benchmarking programmes
• Ensure coherence and robustness
• Greater consistency in standards and approach
• Improved outcomes and impact
• Reduce the unnecessary burden of assessments, costs and duplications for Member States, WHO and the other partners involved in regulatory system strengthening.
• Less burdensome interventions for the National Regulatory Authorities
• Contribute to better resource mobilization

Development of the Global Tool

Promoting reliance and recognition across regulatory

functions

National regulatory system

(RS)

Registration and Marketing Authorization

(MA)

Vigilance (VL)

Market surveillance & Control (MC)

Licensing establishmen ts (LI)

Regulatory Inspections (RI)

Laboratory

Testing (LT)

Clinical Trial

oversight (CT)

NRA Lot

Release (LR)

7

WHO RECOMMENDED REGULATORY FUNCTIONS BASED ON PRODUCT LIFECYCLE

Pre-marketing Phase

Post-marketing Phase

National Regulatory System

Vigilance

Market surveillance and control

Marketing authorization

Licensing

Laboratory Testing

Clinical Trial oversight

Regulatory Inspection

NRA Lot release

WHO Global Benchmarking Tool

Structure/Hierarchy

WHO Global Benchmarking Tool

Structure/Hierarchy

National Regulatory System (NRS) and Functions (NRF)

Indicators’ categories

CROSS-CUTTING TOPICS

Organization and governance

Human resources

Procedures

Monitoring performance and

output

Transparency, accountability and

communication

Policy and strategic planning

Leadership and crisis

management

QMS

Resources other than human

Legal provisions, regulations

and guidelines

RS MA

VL MC LI RI LT CT LR

GBT: Fact sheet template

15

WHO GBT revision VI – Sub-Indicators

21

Minimal capacity

Advanced/reference NRAs

Rating Scale of each sub indicator

• T
• s
• could be an option for some (but never all) sub-indicators

Not Applicable (NA)

he relevant sub-indicator does not apply hould be supported with a justification

8

Entail implementation of some but not all components of

the concerned sub-indicator

Ongoing Implementation (OI)

No evidence is provided to demonstrate any degree of implementation of the sub-indicator

Not Implemented (NI)

Some requirements /activities are showing that the concerned sub- indicator is implemented however is recent or relatively new with little cumulative data for consistent implementation.

Partially Implemented (PI)

No data is provided.

It is an option which can exist only prior to the self-benchmarking

Not Available

Requirements /activities are implemented in consistence manner over a period of time.

Implemented (I)

Rating Scale ��Mathematical meaning

duced

NA is the t

by 1 (i.e. the denominator is reduced by 1)

Not Applicable (NA)

removing the requirement for the sub-indicator:

otal number of sub-indicators required to be met is re

9

The sub-indicator is scored as 0.25 out of 1 (i.e. percentagewise = 25%)

Ongoing Implementation (OI)

The sub-indicator is scored as zero out of 1 (i.e. percentagewise = 0%)

Not Implemented (NI)

The sub-indicator is scored as 0.75 out of 1 (i.e. percentagewise = 75%)

Partially Implemented (PI)

The sub-indicator is scored as zero out of 1 (i.e. percentagewise = 0%)

Not Available

The sub-indicator is scored as 1 out of 1 (i.e. percentagewise = 100%)

Implemented (I)

WHO GBT Performance Maturity Levels

No formal approach

Reactive approach

Stable formal system approach

Continual improvement emphasized

ISO 9004

Regulatory system operating at advanced level of performance and continuous improvement

Evolving national regulatory system

that partially performs essential regulatory functions

Stable, well-functioning and integrated regulatory system

Some elements of regulatory system exist

Can ensure the quality of products if rely on ML 3/ ML 4 regulatory systems

Target of WHA

Resolution 67.20

Advanced and well resourced regulatory systems

WHO GBT

24

57

Countries

39

Countries

98

Countries

Allocation of each sub indicator to specific ML

• Legal provisions

• Regulations

Stepwise approach for implementation of related IDPs

​

• Guidelines
• Processes
• Resources

• Continues improvement (e.g. KPIs)
• Computerized procedures

Maturity Level of WHO Member States

Mar 2023

Current levels of maturity of NRS

WHO GBT (for medicines and vaccines: as of Mar 2023)

• Vaccines produced in countries with ML 3/ML 4 are eligible for EUL or prequalification,

GOAL of WHA Resolution 67.20

ML: (regulatory system) maturity level

WHO Regulatory System Strengthening Programme�Global status of benchmarking of regulatory systems (2016 – Mar 2023)

95

74%

Member-states

World population

=

Number of Member States benchmarked by GBT by year

Cumulative bar chart

GBT and GRP

21

Benefit of GBT

22

1. Promotes the development of Good Regulatory Practices
2. Enhances the implementation of Quality Management System
3. Fosters the evolution of plans to deal with emergencies
4. Requests responsiveness to prevent use and circulation of substandard and falsified medical products

GRP main principles

1. Legality

Regulatory systems and the decisions that flow from them must have a sound legal basis

• Authority, scope and flexibility to safeguard and promote health
• Delegation of power and responsibilities
• support and empower international cooperation
• Possibility to review regulatory decisions and sanctions
• Scope and lines of authority of involved institutions
• Accountable

MA01.01: There are legal provisions that require the receipt of a registration or marketing authorization (MA) before placing the product on the market.

​

MA02.01: There is a defined structure with clear responsibilities to conduct registration or MA activities

​

RS09.01: The NRA participates in regional and/or global networks to promote convergence and harmonization efforts and expand its collaboration in the regulatory field.

​

RS01.09: A guideline on complaints and appeals against regulatory decisions is available to the public.

Key elements:

GBT:

GRP main principles

2. Consistency

Regulatory oversight of medical products should be consistent with existing government policies and legislation and be applied in a consistent and predictable manner

• Fit coherently into the national legal and policy framework

​

• Complementary and not conflicting

​

• Consistent implementation and enforcement

MA04.04: The same criteria apply for assessing applications regardless of the origin of or destination for the medical products (e.g., domestic, foreign, public sector, or private sector)

​

RS01.04: All regulatory entities (central and decentralized ones) follow non- contradictory regulations, standards, guidelines and procedures.

​

MA01.10: There are guidelines on the format and content for submission of MA applications that are consistent with the WHO or other internationally accepted standards

Key elements:

GBT:

GRP main principles

3. Independence

Institutions responsible for regulation of medical products should be independent

• Operate in an independent and authoritative manner

​

• Discharging its duties independently from politicians, government and regulated entities

​

• Improper and undue influence of stakeholders on Regulatory activities and decisions

​

• Appropriate and clear funding

​

• The independence of the leadership

RS02.04: Independence of NRA from researchers, manufacturers, distributors and wholesalers, as well as from the procurement system

​

RS07.04: The NRA has authority to manage the funds allocated and/or generated internally.

​

Key elements:

GBT:

GRP main principles

4. Impartiality

All regulated parties should be treated equitably, fairly and free from bias

• No conflicts of interest or unfounded bias

​

• Impartial operations

​

• No engagement in the activities that must be regulated

​

• Science and evidence based decision-making

MA04.04: The same criteria apply for assessing applications regardless of the origin of or destination for the medical products (e.g., domestic, foreign, public sector, or private sector)

​

MA04.05: An advisory or scientific committee, including external experts is involved in the review of MA applications (as needed)

​

RS09.07: A code of conduct, which includes management of conflicts of interest, is published and enforced for internal and external staff, including members of the advisory committees.

​

Key elements:

GBT:

GRP main principles

5. Proportionality

Regulatory oversight and regulatory decisions should be proportional to the risk and to the regulator’s capacity to implement and enforce the decisions.

• Adequate to achieve the objectives without being excessive

​

• Proportionate to the risk

​

• Do not exceed the national capacity to implement and enforce

​

• Benefit–risk evaluation and continuous monitoring

MA01.12: There are established guidelines that cover circumstances under which the routine MA procedures may not be followed (e.g., for public- health interest)

​

RS04.05: Written criteria to cover circumstances in which the routine regulatory processes may not have to be followed in relation to crises and emergencies linked to a risk management plan.

Key elements:

GBT:

GRP main principles

6. Flexibility

Regulatory oversight should be flexible in order to respond to a changing environment and unforeseen circumstances.

• Sufficient flexibility to reflect or respond to changes

​

• Timely responses to urgent situations

​

• Performance based

​

• Alternative approaches

​

• Flexibility for applying good judgement

MA01.06: There are legal provisions to cover circumstances under which the routine MA procedures may not be followed (e.g., for public health interest)

​

​

MA04.07: There are documented mechanisms to handle non-routine registration or MA requirements in special situations (e.g., public-health interest)

​

RS07.03: There are provisions relating to reduction or exemption of dues, taxes, tariffs or fees in defined situations for public health interest.

Key elements:

GBT:

GRP main principles

7. Clarity

Regulatory requirements should be accessible to and understood by users

• Understandable language

​

• Terminology consistent with international norms

​

• Consultation, education and training

​

• Proper interpretation of regulations

​

• Clear process and basis for taking regulatory decisions and enforcement

​

MA01.09: Specific guidelines on the quality, nonclinical and clinical aspects are established and implemented

​

MA01.11: There are guidelines for MA holders that define the types and scope of variations, the format and content to be used for documenting the variations, and the identification of those variations that require prior approval or notification.

Key elements:

GBT:

GRP main principles

8. Efficiency

Regulatory systems should achieve the intended results within the required time and

at reasonable effort and cost

• Achieve the public health goals
• Effective use of resources and information from other authorities
• Most efficient and least burdensome means of achieving regulatory purposes
• Evaluation of the total burden and resources
• Explore ways of improving efficiency
• Alignment of regulatory requirements
• Contribution of regulated entities
• Performance-based indicators

MA04.06: Timelines for the assessment of the applications are defined and an internal tracking system has been established to monitor adherence to the targeted time frames

​

MA01.08: Legal provisions or regulations allow the NRA to recognize and/or rely on MA-relevant decisions, reports or information from other NRAs or regional and international bodies

​

MA06.02: Performance indicators for registration and MA activities are established and implemented

​

Key elements:

GBT:

International collaboration

GRP main principles

9. Transparency

Regulatory systems should be transparent; requirements and decisions should be made known, and input should be sought on regulatory proposals.

• Investment and a culture of openness, supported by government policy, commitment and action

​

• Stakeholders should be consulted

​

• Access to regulations and decisions

​

• disclosure should be consistent with national laws on access to information.

​

RS01.06: Legal provisions and regulations define requirements of transparency and dissemination of information to the public and relevant stakeholders.

​

RS09.04: Information on marketed medical products, authorized companies and licensed facilities is publicly available.

​

MA05.02: Updated list of all medical products granted MA is regularly published and publicly available

​

RS09.04: Information on marketed medical products, authorized companies and licensed facilities is publicly available.

Key elements:

GBT:

GBT and Emergency Preparedness

37

Emergency preparedness

Examples in the GBT

39

RS04:

Regulatory system is supported with leadership and crisis management plans

RS04.05:

Written criteria to cover circumstances in which the routine regulatory processes may not have to be followed in relation to crises and emergencies linked to a risk management plan

RS07.03:

There are provisions relating to reduction or exemption of dues, taxes, tariffs or fees in defined situations for public health interest.

​

MA01.06:

There are legal provisions to cover circumstances under which the routine MA procedures may not be followed (e.g., for public health interest)

​

MA01.12:

There are established guidelines that cover circumstances under which the routine MA procedures may not be followed (e.g., for

public‐ health interest)

​

CT01.05:

There are legal provisions or regulations covering circumstances in which the routine CT evaluation procedures may not be

followed (e.g. for public‐health interests)

GBT and SF medical products

40

Combating SF medical products

41

• The tool is able to benchmark regulatory system capacities and performance in the field of Substandard/Falsified (SF) medical products.
• Related indicators are not limited to one function. Rather, many related

indicators are distributed among different regulatory functions:

• National Regulatory System (RS) for systematic issues
• Market surveillance and Control (MC) for operative issues
• Laboratory Access and Testing (LAT) for quality testing issues
• Other functions (e.g. Regulatory Inspection, Marketing Authorization and Registration)

​

• Ten (10) sub-indicators are directly referring to SF medical products
• Approximately 85 sub-indicators are indirectly referring to SF medical products

Examples in the GBT

43

RS01.05: Legal provisions and relevant regulations to take actions on recall, suspension, withdrawal and/or destruction of substandard and falsified (SF) medical products.

​

RS04.02: A rapid alert system to for managing the threats by SF medical products and for recalling these products from the market.

dealing with substandard or falsified (SF)

MC01.03: Legal provisions and regulations address the role of NRA in medical products.

MC01.07: Guidelines exist on the recall, storage and disposal of SF medical products.

​

​

MC04.05: Documented and implemented procedures exist to enable the public to report suspected SF medical products.

​

​

MC04.07: Documented and implemented procedures and mechanisms exist to prevent, detect and respond to SF medical products

​

MC04.08: Documented and implemented procedures exist to ensure safe storage and disposal of detected SF medical products.

WHO Share Point

WHO Share Point

23

A web-based collaborative platform to share contents

Each member State has a dedicated page in the Share Point to rapidly and effectively exchange information and documentation related to the benchmarking visit

Administrative documents

​

ToR

​

Agenda of the visit

​

Travel information

​

NRA contacts

Team members info: Contact details, DoI and CA CVs

Technical documents

Previous benchmarking reports

IDPs

Quantitative tool

Self-assessment

Inputs from observed audit/filed visit

Background documents per each function

Criteria for Evaluation

Criteria for Assessment

​

​

50

51

Updates on WHO Listed Authorities (WLA)

WLA Framework

23

WLA

Performance Evaluation Process

GBT VI

(Jan 2019)

WHA67.20

(May 2014)

and other

Regional Resolutions

WLA

Operational guidance

SOPs

WLA Policy document (June 2021)

Developed

Under development and public consultation

Not yet developed

Target WLA framework

operational from 2022

WLA operating principles

ELIGIBILITY

01

02

03

04

Voluntary process initiated by a request from a Member State

National Regulatory Authorities (NRAs) and Regional Regulatory Systems (RRSs) are eligible

3

Regulatory bodies must meet ML 3

requirements to be eligible

All available evidence considered to determine compliance with WLA requirements

A combined approach to achieve listing

WLA builds upon GBT

GBT

PEP

WLA

ML

delinked from ML

The WLA status is reached when a regulatory authority or a regional regulatory system has been documented to comply with all the relevant indicators and requirements specified by WHO for

the requested scope of listing based on

​

an established benchmarking (GBT) AND a performance evaluation process (PEP)

Performance Evaluation Process

COMPONENTS

Mandatory GBT ML4

PE

indicators

PE

tools

Performance Evaluation Process

GBT ML3

fully met

for each function

If any of these 4 components fails, WHO listing is not reached