1 of 27

Quality Risk Management, ICH Q9(R1)

Training Slides

Subjectivity in Quality Risk Management

PART I: How to Identify and Control Subjectivity

International Council for Harmonisation of Technical Requirements

for Pharmaceuticals for Human Use

2 of 27

Legal Notice

This presentation is protected by copyright and may, with the exception of the ICH logo, be used, reproduced, incorporated into other works, adapted, modified, translated or distributed under a public license provided that ICH's copyright in the presentation is acknowledged at all times. In case of any adaption, modification or translation of the presentation, reasonable steps must be taken to clearly label, demarcate or otherwise identify that changes were made to or based on the original presentation. Any impression that the adaption, modification or translation of the original presentation is endorsed or sponsored by the ICH must be avoided.

The presentation is provided "as is" without warranty of any kind. In no event shall the ICH or the authors of the original presentation be liable for any claim, damages or other liability arising from the use of the presentation.

The above-mentioned permissions do not apply to content supplied by third parties. Therefore, for documents where the copyright vests in a third party, permission for reproduction must be obtained from this copyright holder.

3 of 27

Agenda

This training material on Subjectivity in Quality Risk Management (QRM) is divided into two different presentations:

Part I: how to Identify and Control Subjectivity

Key points from ICH Q9(R1) regarding Subjectivity in QRM
Why Subjectivity in QRM can be a problem
The Benefits of Controlling Subjectivity
Practical Tips & Examples
Controlling and minimizing subjectivity across the four elements of QRM

Risk Assessment
Risk Control
Risk Communication
Risk Review

Subjectivity Case Study
Key take-home points

Part II: what Subjectivity in Quality Risk Management is – Background and Theory

The role of Uncertainty, Heuristics and Bias in relation to Subjectivity
Risk Perception factors
Risk Tolerance considerations
How Subjectivity can influence the activities that make up the QRM process
Subjectivity in Risk Scoring Scales/Models
Key take-home points

4 of 27

Subjectivity in QRM – Key Points from ICH Q9(R1)�

Question:

The original (2005) version of ICH Q9 did not specifically address subjectivity in QRM. Why was the topic of subjectivity identified for elaboration in ICH Q9(R1)?

Answer:

Highly subjective risk scoring methods and differences in how risks are assessed and how hazards, risks, and harms are perceived by different stakeholders can lead to varying levels of effectiveness in the management of risks. (Ref. ICH Q9(R1) Concept Paper, November 2019)

Subjectivity can directly impact the effectiveness of risk management activities and the decisions made. Therefore, it is important that subjectivity is managed and minimized. (See ICH Q9(R1), Introduction)

5 of 27

Key Points from ICH Q9(R1), cont’d��

Question:

What are the responsibilities of decision-makers in QRM with regard to subjectivity?

Answer:

Decision-makers should assure that subjectivity in quality risk management activities is managed and minimized, to facilitate scientifically robust risk-based decision-making. (See ICH Q9(R1), Section 4.1, Responsibilities)

6 of 27

Key Points from ICH Q9(R1), cont’d��

Question:

What stages of QRM are impacted by subjectivity and how is it introduced into QRM?

Answer:

Subjectivity can impact every stage of a quality risk management process, especially the identification of hazards and the estimation of probability of occurrence and severity of harm. It can also impact the estimation of risk reduction and the effectiveness of decisions made from quality risk management activities. (See ICH Q9 (R1), Section 5.3, Managing and Minimizing Subjectivity)

Subjectivity can be introduced in quality risk management through differences in how risks are assessed and in how hazards, harms and risks are perceived by different stakeholders, (e.g., bias). Subjectivity can also be introduced when risk questions are inadequately defined, and when tools have poorly designed risk scoring scales. (See ICH Q9 (R1), Section 5.3, Managing and Minimizing Subjectivity)

7 of 27

Key Points from ICH Q9(R1), cont’d��

Question:

Can subjectivity be eliminated from QRM? What needs to be done?

Answer:

While subjectivity cannot be completely eliminated from quality risk management activities, it may be controlled by addressing bias and assumptions, the proper use of quality risk management tools and maximizing the use of relevant data and sources of knowledge. (See ICH Q10, Section 1.6.1)

All participants involved with quality risk management activities should acknowledge, anticipate, and address the potential for subjectivity.

(See ICH Q9(R1), Section 5.3. Managing and Minimizing Subjectivity)

8 of 27

Why Subjectivity in QRM can be a problem

Hazards may not be identified appropriately, and as a result, the risks associate with certain hazards may go unmanaged.
When risks presented by hazards are being assessed, the probabilities of occurrence and the severities of harm that are arrived at may be underestimated or overestimated.
In addition, subjectivity in assumptions about the value or effectiveness of certain risk controls can result in an over estimation of the extent of risk reduction that they deliver – this can lead to a ‘false sense of security’ in those controls.

Subjectivity in QRM can lead to biased risk assessment outcomes, and these can give way to sub-optimal risk controls, poor risk-based decisions, and ineffective QRM outcomes generally.

9 of 27

The benefits of controlling subjectivity

Less subjective Risk Assessments and QRM outputs in general have the potential to yield many benefits. In this regard, the November 2020 Concept Paper for ICH Q9(R1) states the following:

Less subjective risk assessments to support manufacturing processes should lead to fewer quality defects that could present risks to patients.
Less subjective risk assessments should also lead to more science-based manufacturing operations, control strategies, and validation activities, resulting in the potential for reduced costs and the possibility to free up resources for other necessary activities.
ICH Q8, Q10, and Q11 expect science- and risk-based applications, and revising ICH Q9 to address subjectivity in QRM more explicitly will help enable/accelerate the continued implementation of Q8, Q10, Q11 (and Q12), because of the foundational relevance of QRM.

Ultimately, better Risk-based Decision making should be the result of less subjective Risk Assessments and QRM activities

10 of 27

Controlling Subjectivity in QRM

Have people on QRM teams with a high level of knowledge on the factors that introduce Subjectivity into QRM activities.

Competency in this area can help assure that the adverse effects of uncertainty, heuristics and bias are minimized during risk assessments and decision making.
It may lead to more robust brainstorming and other activities when identifying hazards and assessing risks.
It may particularly help address the factors that can influence probability of occurrence estimates.
It may help deal with the factors related to risk perception, and in doing so, risk communications can be designed that counteract risk perception problems.

✔

Ref. Haddad, G., Greene, A., Quality risk management competency model - identification of standard individual QRM practitioner roles, PDA J Pharm Sci Technology. Jan-Feb 2020; 74(1):58.doi:10.5731

11 of 27

Controlling Subjectivity, cont’d

Have people on QRM teams who have a high level of knowledge of QRM Tools.

This can help assure the proper use of such tools, so that the QRM activity delivers objective results and effective risk controls.
It may also lead to new or customized QRM tools being designed that will a) deliver more scientific and less subjective risk assessments, b) focus on preventative controls, and c) lead to demonstrated risk reduction. This can lead to several benefits, such as:

Risk assessments are informed by the controls that may (or may not) be in place, and by the true effectiveness of those controls.
Probability of Occurrence ratings are informed by an assessment of the preventative controls that may be in place.
Severity ratings are based on an assessment of the controls that may reduce the severity of a negative event, should it occur.
Detectability ratings are based on an assessment of the detection controls that may be in place.
The GMP controls that are important in risk control are assessed for their Qualification or Validation requirements.

✔

Ref. O’Donnell, K., Zwitkovits, M., Greene, A., Calnan, N., Quality Risk Management – Putting GMP Controls First, PDA J. Pharm. Sci. Technology., May/June 2012, Vol. 66, No. 3. pp 243-261

12 of 27

Controlling Subjectivity, cont’d

This person can bring structure to the risk assessment and guide the team participants on how certain biases may be counteracted during their teamwork.
The facilitator can also ensure that every team member has their say. They can also ensure that the appropriate expertise is present on the QRM team. (QRM team-work can be spread over more than one meeting.)
The facilitator should be very familiar with the risk assessment methodology that will be used; they should also possess effective leadership, communication and time management skills.
They should also be well versed in the literature on biases and human heuristics.
An effective QRM facilitator can help clarify important QRM concepts as the risk assessment progresses, such as how hazards and risks are not the same, how risk scoring should occur, etc.
The facilitator can drive the team to look for strength of evidence when risk ratings are being proposed and discussed.
They can help drive the risk assessment to completion.

✔

Use a QRM facilitator to lead and direct the risk assessment.

Ref. Haddad, G., Greene, A., Quality risk management competency model - identification of standard individual QRM practitioner roles, PDA J Pharm Sci Technology. Jan-Feb 2020; 74(1):58.doi:10.5731

13 of 27

Controlling Subjectivity, cont’d

This can help ensure that the risk assessment is not focused solely on the knowledge and experiences of one department or function.

The literature generally shows that subject matter experts are less impacted by the influences of biases and human heuristics than non-subject matter experts.

✔

Have a cross-functional team with expertise in the process or item under study.

14 of 27

Controlling Subjectivity, cont’d

Ensure that the team is briefed in advance on the risk question, on the goals and details of the risk assessment, and that it is provided with background and other information on the process or area that is to be risk-assessed.

Such information can include:

why the risk assessment is being performed
the risk questions that will be addressed
the risk assessment methodology that will be used
a map of the process
comprehensive data on the process or area that is to be risk-assessed, including an outline of the current control strategy that is in place and any performance data that is available on the process or area.

The team should also receive information about the risk assessment methodology that will be used.

✔

This helps ensure that the team members are aligned on what is expected of them, that they are all familiar with the process or area to be risk assessed, and that they are focussed on the objectives of the risk assessment and on the tasks at hand.

15 of 27

Controlling Subjectivity, cont’d

Ensure that, during a risk assessment team meeting, the QRM facilitator (or if there isn’t a facilitator in place, the team lead) reminds the team members of key risk concepts and terms.

The facilitator or team lead should also provide an overview of how the risk assessment methodology works at a practical level,

The QRM facilitator or team lead can explain the various biases, human heuristics and other sources of subjectivity that may come into play when the teams convene to perform Risk Assessments.
The facilitator can also outline the strategies that will be used during the Risk Assessment to control and minimise subjectivity.

✔

16 of 27

Controlling Subjectivity, cont’d

Allow for Creative Thinking

This is especially useful in relation to hazard identification, and the identification of options in risk-based decision-making.
While it is important to control for bias and subjectivity, thinking about what might go wrong can benefit from some level of creative thinking – e.g. imagine if the operator did this instead of that. (Please see the ICH Q9(R1) training material on Hazard Identification for more information.)
One way to support creative thinking is to design the risk assessment meetings so that the team members have an opportunity to work separately, as well as together:

This avoids being overly influenced by the ideas of loud or senior people.
The team members come together again after a period of working alone - this allows them to align and make knowledge-driven choices and decisions.
This approach is sometimes called ‘Working Together, Alone’.

It can sometimes also be useful if ideas and opinions are not attributed to specific individuals on the team – they can be anonymised, treated as hypotheses and discussed for their merits and evidence-basis.
Anonymous voting can also be used – it provides a safe space for people to provide their personal perspectives, without judgement.

✔

Ref. Based on materials and figures submitted to the ICH Q9(R1) EWG by Era Sciences Ltd., Ireland

17 of 27

Controlling Subjectivity, cont’d

In relation to Creative Thinking, below is an outline of how ‘Working Together, Alone’ might work in practice during a Risk Assessment

After an initial introductory meeting, the team members diverge and work alone for a period to identify the hazards they perceive to be relevant to the risk question of concern. This empowers them to apply their experience and insights, without the interferences of any biases associated with group-work.
Then, the team members converge and discuss their identified hazards. Trends are identified, hazards discussed and decisions made on which hazards should be further assessed.
The team members diverge again and, working alone, they consider the current controls that are in place in relation to each hazard. Then, taking that information into account, they make estimates of the likelihood of occurrence and the potential severity of each hazard.
The team converges again and works collaboratively to assess the potential risks presented by each selected hazard. When doing this, the team members present their thoughts on the likelihood of occurrence and severity estimates for each hazard, taking into account current controls as well as the data that supports their estimates.
When it comes to risk control, the team collaborates to identify what risk controls are required, but it can diverge / converge again, if needed, to align on those risk controls and to identify any additional risk mitigation that may be required.

Ref. Based on materials and figures submitted to the ICH Q9(R1) EWG by Era Sciences Ltd., Ireland, and also https://www.thesprintbook.com

18 of 27

Tips for Controlling Subjectivity across the 4 elements of QRM

1. Risk Assessment

Sources of Subjectivity	How to Control Subjectivity:
QRM team members do not bring the right level of expertise. Some QRM team members are external consultants not familiar with internal processes	When teams are formed, they should include experts from the appropriate areas (e.g., quality unit, product development, business development, engineering, regulatory affairs, production operations, sales and marketing, supply chain, legal, statistics and clinical) in addition to individuals who are knowledgeable about the quality risk management process. Use interdisciplinary teams to cover all aspects. Prepare for the assessment by gathering applicable pre-existing historical data. Use appropriately qualified consultants who can fill in knowledge gaps and provide outside perspectives. Note: Experts may be subject to biases and other sources of subjectivity just as non-experts may be, and they may also perceive hazards differently.
QRM team members are not familiar with QRM principles or tools	Ensure QRM team members are familiar with the QRM principles and tools. Consider use of a QRM Facilitator with experience and knowledge of the QRM process.
Team composition: strong personalities dominate risk assessment outcome	Consider use of a QRM Facilitator to moderate the discussions and ensure less vocal/less dominant QRM team members are heard. The QRM team can agree certain ground-rules for their Risk Assessment meetings to ensure that everyone has a say.
Misinterpretation of the scope of the risk assessment	Pre-define and agree the risk question and scope of the assessment with internal stakeholders (e.g., QRM team members and other relevant colleagues). Identify what is in scope vs. out of scope.

19 of 27

Tips for Controlling Subjectivity, cont’d

Sources of Subjectivity	How to Control Subjectivity:
Uncertainty is not adequately considered in risk rating scores	Discuss and address, where possible, any sources of uncertainty, assumptions, or gaps in knowledge during the risk scoring process (e.g. unknown root causes, sources of uncertainty, pendong scoentif data, etc.). Assign a higher risk rating where the level of uncertainty is significant. Seek to obtain additional data from all relevant sources, and review risk once data are available.
Qualitative, non- standardized or undefined scoring criteria	Spend time developing risk rating scales with detailed and meaningful criteria for each score. Use severity rating scales with clear descriptions of potential impact on product quality, availability and/or the patient. For occurrence ratings, consider scales, where possible, with detailed quantitative criteria using actual, tangible incidences where a given score would be the appropriate choice. If risk thresholds / cut-off levels are used to determine risk acceptability, they should be justified and have a clear scientific basis.
Lack of rationale for the risk rating / scores	Consider documenting the rationale for each selected score, with references to the supporting data (e.g. scientific studies, trend data, etc.) Develop a risk analysis worksheet that includes a space to document the rationale for the risk ratings/scores.
Bias is not recognized or controlled	Ensure QRM team members are aware about the impact of subjectivity, bias and heuristics at a practical level, this could be achieved by having guidance in place that helps the team minimize the impact of biases and heuristics.

1. Risk Assessment, cont’d

20 of 27

Tips for Controlling Subjectivity, cont’d

Sources of Subjectivity	How to Control Subjectivity:
Interpretation of the likely effectiveness of the proposed risk controls Over-reliance on controls whose effectiveness may be excessively subjective	Carefully consider the likely effectiveness of each proposed risk control, as well as any validation requirements for such controls. Do not overly rely on controls whose effectiveness is highly subjective or highly uncertain. Clarify that prevention/design controls are always preferred over detrection controls, as they address the root cause(s) of the failure mode. Do not overly rely on controls that are after-the-fact (e.g. 100% final inspection); also assure robust design and preventive controls to prevent defects are in place. Document sources of uncertainty, assumptions and knowledge gaps so that these issues can be addressed during Risk Review.
Over-reliance on Risk Priority Numbers (RPNs)	Clarify that the RPN concept is intended to inform risk prioritization. Ensure that RPNs are not used in isolation to accept risks - use a second set of considerations in concert with the RPN values (e.g. any risk with the high severity scores should be reduced as low as possible via risk control) Do not over-rely on the RPN to make risk control decisions – this is especially important when the RPN is close to the threshold / cut-off number. Consider expressing risks in other ways instead of using just RPNs.
Over-relying on detection at the end of the process, and controls in earlier process steps are overlooked	Do not take credit for controls that are not well established. Ensure that up-stream controls are included so that quality is in-built. QRM team members should include personnel who are directly involved in the process.

2. Risk Control

21 of 27

Tips for Controlling Subjectivity, cont’d

Sources of Subjectivity	How to Control Subjectivity:
When the wrong decision-makers are engaged in risk communications Where risk communications do not correct decision-makers’ perceptions, biases and incomplete prior knowledge When the communication contains biased information for the decision-makers When the cost-benefit of the proposed risk controls is not well communicated When communications are incomplete, and uncertainty is not addressed When the wrong stakeholders are informed about particular risks	Appropriate decision-makers are selected, and these determine the content of the communications. QRM team should understand the factors that can influence how risks are perceived and take appropriate countermeasures during risk communication. QRM team should communicate risks early, to promote risk awareness among stakeholders. Risk communications should be performed by experts who are trustworthy, balanced and open, and not influenced by bias. Significant uncertainties should be addressed. Assess the level of information that the different stakeholder groups will likely need to receive. QRM team to assess which stakeholders need to be informed of which particular risks.

3. Risk Communication

22 of 27

Tips for Controlling Subjectivity, cont’d

Sources of Subjectivity	How to Control Subjectivity:
When the impetus (need) for the Risk Review is not well defined When the data used for Risk Review is insufficient or non-objective When the Risk Review team does not have the right expertise to identify if new risks have arisen When qualitative, non standardized or undefined criteria are used during Risk Review When there are insufficient data to support the effectiveness of risk controls, or when there is a bias towards inaction	Have a clearly defined objective for Risk Reviews. Ensure that the data used for Risk Review is objective and sufficient (proper use of relevant data, SOPs, Std Tools, etc.). Ensure the Risk Review team has the correct expertise. Ensure that objective criteria are applied when making Risk Review decisions. Consider using a QRM Facilitator during Risk Review.

4. Risk Review

23 of 27

Case Study

There is one case study that supports this presentation

See next slide for details

24 of 27

Case Study

This discusses subjectivity in a risk assessment regarding critical data integrity findings, and the consequent impact on remediation effectiveness

It deals with a remediation approach following critical data integrity inspection findings and it demonstrates how subjectivity can impact every stage of a QRM process.
It discusses how subjectivity can impact effective risk assessment, including the depth and scope of data reviews.
It demonstrates that appropriate scope and severity determinations relating to data integrity lead to effective QRM outcomes (Risk Assessment, Risk Control, Risk Review, Risk Communication) and the satisfactory resolution of health authority concerns.

Click here to access the Case Study

25 of 27

Take-home Points/Conclusions

Subjectivity can be observed across all parts of the QRM process, from Risk Assessment through to Risk Review.
There are many strategies that can be adopted to control and minimize subjectivity. A number of these are outlined in this presentation.
Part II of this training material, together with the Case Study referred to in this presentation, illustrates the sources of subjectivity and things to consider when working to control it. These relate to the following:

Taking steps that reduce the effects of heuristics and bias

Having fit-for-purpose risk scoring methods

Making sure that QRM tools are used appropriately and as intended

Maximising the use of relevant data and sources of knowledge

1 of 27

2 of 27

3 of 27

4 of 27

5 of 27

6 of 27

7 of 27

8 of 27

9 of 27

10 of 27

11 of 27

12 of 27

13 of 27

14 of 27

15 of 27

16 of 27

17 of 27

18 of 27

19 of 27

20 of 27

21 of 27

22 of 27

23 of 27

24 of 27

25 of 27

26 of 27

27 of 27