Quality Assurance (QA) - information session

Jointly prepared by PCU, PPU and D&D unit 

29 October 2024

​

​

Objectives & content 

Objectives

• Refresher on the QA process
• Flag core aspects for Initiatives, Platforms, SGPs

​

Content

• QA process overview
• 2024 enhancements and instructions to flag:
• Evidence requirements 
• Focus on outcomes
• Guidance on Other Outputs and Other Outcomes
• AI integration
• Presentation of the QA Module
• Support documents 
• Q&A

www.cgiar.org

1. Overview of the 2024 QA process�

What is Quality Assurance?

Process that aims to:

• improve quality of reported data
• Consistency across the system
• Reliability of reported results
• Inform a solid learning process 
• improve overall CGIAR accountability and transparency.

​

QA includes:

• Checking reported result data against criteria
• Helping to improve the quality of reported data (pre-QA)
• Improving the QA process (streamlining) so that value > cost

​

​

www.cgiar.org

Reporting process: the place of QA

• It is important to increase early submission for QA batch 1 to avoid the January rush – deadline Nov 15.
• It is especially important to submit and QA Innovation Development results that will be used to report Innovation packages (IPSR pathways only) in 2024.
• Reporting of 2024 results can be done until 31 January 2025.

www.cgiar.org

The use of QA’d data

​

​

QA’d data

Featured in the Results dashboard

​

Integrated in all 2024 Technical Reporting products (Type 1, Type 2, Type 3 and Portfolio Narrative)

​

Integrated in the overall CGIAR Annual Report 

www.cgiar.org

QA process: Result Indicators and fields to QA

5 indicator categories to assess

+ 1 Other category for each reporting level

Outcome

(WP and E-o-I)

Output

1. Priority/core fields for QA

(the 3rd party will decide on disagreements)

​

2. Regular fields 

(Assessors provide an assessment, but Initiatives hold the responsibility of reported data)

​

3. Data not QAed 

(context only, e.g. Contributing partners)

​

​

​

WHICH RESULTS ARE NOT QA'd: 

• KP that are not PR papers or MELIA studies

Full table of priority/non priority fields is included in the Assessors Guidance.

www.cgiar.org

QA process: main steps and timeline

Anonymous assessors

Agree/disagree and make adjustments

Two assessments are made

Initiatives/Platforms/SGPs

Lead assessors

Final decision is taken on unresolved disagreements

Highlight unresolved disagreements and

 unimplemented changes

3^rd party 

PPU/PCU

ROUND 1: for all fields

ROUND 2: for priority/core fields

Implementation of changes in the PRMS

Data submission

Initiatives/Platforms/SGPs

The lead assessor makes sure to leave only one comment per field.

Initiatives/Platforms/SGPs have final responsibility for the data on regular fields – there is no further exchange/discussion with assessors on those fields

Data submission deadline are intended EoB time

Two full weeks are granted to Initiatives/Impact Platforms/SGPs to tackle assessors’ comments

www.cgiar.org

The 3^rd party mechanism

Aim:

Resolve differences on a sub-set of core/priority data, when Initiatives and Assessors do not agree.

​

What:

In case of disagreement, the third party will broker an agreement between assessors and Initiatives, with the support of subject matter experts as required. 

The 3rd party cannot ask for major changes (e.g. providing missing essential evidence). 

• If results cannot meet the quality criteria, the 3rd party will decide for removing the result. 
• Submitters will be informed about any change instructed by the 3rd party.

​

Who:

The third-party decision maker (1 per AA) was identified by Action Area Science Group Directors and are the AA MELIA focal points.

​

​

​

​

www.cgiar.org

2. 2024 enhancements and Instructions to flag

Evidence field requirements

• ALERT: > 500 results with lack of access to evidence in 2023

​

• CGSpace will store the great majority of evidence for all results​ to make it as open as possible. Otherwise, upload evidence files in the PRMS repository.

​

• Confidential evidence especially relevant for early stage Innovation dev and use, and Policy change.

​

• Evidence will not go to the dashboard unless it is a known open link (CGSpace open access- or YouTube).

​

• Up to 6 pieces of evidence per item and optional description.

​

• IA score 2 claims are supported by evidence (one of the 6 pieces above).

​

• Option to tag evidence to indicate that it supports the Innovation readiness level.

​

​

​

​

​

​

​

​

www.cgiar.org

QA Focus on outcomes

​

​

​

​

​

​

Prioritize Quality over Quantity: Shift focus from reporting large quantities of results to ensuring the quality and impact of outcomes and partnerships (2022-2023 CGIAR Learning process).

​

Provide strong evidence:  The evidence should demonstrate an actual change in behavior or "Knowledge, Attitude, Practices" (KAP) progress. For Innovation use, it should provide the number of real users, not projections or expected figures.

​

Monthly quality reviews: MELIA Science Group focal points review outcomes monthly and flag necessary changes before the QA process to speed things up.

​

Addressing comments (Batch 2): If comments remain unresolved (by Feb 21), PPU/PCU will adopt the assessor's feedback and proceed with necessary changes or removal of outcomes.

www.cgiar.org

��Guidance on Other Outputs & Other Outcomes

New instructions with examples: Guidance on the Other Output and Other Outcomes categories

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Boundaries between OT and OO

• Report an event (e.g., a conference panel or participation in conferences) as an output and report an outcome only if there is evidence of behaviour change or “Knowledge, Attitude, Practices” (KAP) progress.

www.cgiar.org

Other instructions to flag

​

• Be sure that acronyms are spelled out. 

​

• Ensure your results are informative and understandable to non-specialist readers.
• The title should include:
• What the result is about – “what it is”, “what it does”. 
• The societal/environmental relevance – “for what/whom” and “from/by whom”. 
• The description should contain complementary information and not repeat the title. Only limited QA will be provided.

​

• Avoid duplicates, i.e. the same result reported twice in different result categories.

​

• Providing comments when you “Accept” a QA suggestion is optional.

​

• For KP:
• If information is updated in CGspace during the QA assessors' windows (Batch 1: from 15 – 22  November), it is possible that assessors will assess obsolete data and their comment will no longer require a change from your side. In these cases, we ask that you “Accept” their comments and add in the optional comment box: "comment no longer relevant".
• ISI and Open Access fields are mandatory (although not mandatory in CGSpace).

​

​

​

​

​

www.cgiar.org

Enhancing QA with AI and other tools

​

​

​

Objective:

Improve the effectiveness and efficiency of CGIAR’s QA process for technical reporting using AI and other tools.

​

Rationale:

• AI and automation could enhance accuracy and consistency, addressing limited time, varied submitters, and a growing number of results.
• It saves time for both assessors and submitters, particularly in QA follow-up when guidelines aren’t followed.

​

2024 Priorities:�Focus on Impact Area tagging, innovation readiness levels, and evidence links.

​

​

​

​

www.cgiar.org

Current status

PHASE I

For QA batch 1, Nov 2024

​

QA Batch 1 is a testing phase where AI will assist by identifying mismatches, but human assessors will still review and decide.

​

Objectives:

• Assessment of IA scores for all �IAs and all result types and levels
• Assessment of readiness level at �output level (innovation dev)

​

​

​

​

​

​

​

PHASE II

For QA batch 2, Feb 2025

​

Objectives:

• Assessment of readiness level at outcome level (innovation packages)
• Improved accuracy (against experts and assessors' assessments)
• Interface for assessors in the QA Platform

Work to date: ​

• Several versions of AI helper created for testing
• Validation of testing results by IAP and IPSR experts
• Guidance & process revisions being proposed for approval

www.cgiar.org

Assessing Impact Area tags

• Why Tagging is Important: Accurate tagging aligns reported results with funders' priorities, demonstrating activity relevance and effectiveness. It helps secure support while ensuring transparency and accountability.

​

• All 5 Impact areas are tagged: 
• i) nutrition, food and health security; 
• ii) poverty reduction, livelihoods and health; 
• iii) gender equality, youth and social inclusion; 
• iv) climate adaptation and mitigation; 
• v) environmental health and biodiversity.

​

• New tagging requirements:
• Maximum of 2 IAs can have a score of 2.
• At least one IA needs to have a score of 1 or 2; no result can have a score of 0 for all 5 IAs.

​

• Evidence for tag 2 is required for all Indicators types – for all submitters-.

​

• For gender tag, note that:
• A specific enhanced explanation of the tagging is provided.
• Evidence where results are only disaggregated by gender is not sufficient for a tag 1. 
• Look beyond using only the number of M/F participants in the research as indicators of gender relevance. Consider qualitative aspects like roles, responsibilities, decision-making influence, and resource distribution.

​

​

​

​

​

​

​

​

​

​

​

​

​

​

www.cgiar.org

3. Tour of the Platform�

Quality Assessment Platform

18

User manual for Initiatives, Platforms, SGPs

QA Users Manual 2023 (IDTs).pptx

19

How to access the QA Module in the PRMS Reporting Tool

CGIAR QA PLATFORM  Initiatives

USER MANUAL

20

CGIAR QA PLATFORM  Initiatives

USER MANUAL

1. HOME

1. Side menu

This is the main menu in the QA Platform. It allows Initiatives to navigate between the different indicators and the home page.

​

2. Indicator dashboard

This provides graphical representations of the QA process, organized by indicator.

​

3. New comments report

This new functionality allows users to download the full list of comments for each Initiative, SGP, or Platform, with an option to filter by indicator category if desired.

​

21

CGIAR QA PLATFORM  Initiatives

USER MANUAL

2. INDICATOR DASHBOARD

1. Status by indicator

This shows the overall progress of the assessment for each specific indicator. Comments for each item are classified as either: answered / no action needed or pending.

​

3. Number of pending responses

This highlights the number of comments to which an Initiative has not yet responded. 

2. Total comments

This displays the total number of comments, categorized by response type to assessors' comments (Pending, Accepted, Accepted with comment, and Disagreed).

​

4. Number of 3rd party broker decisions

Indicates how many 3rd party instructions to be implemented by PPU/PCU are pending. This will be used only in Round 2 and not by Initiatives.

​

​

Note: Each indicator has an independant dashboard. On the homepage, a vertical list of each indicator dashboard can be found.

1

4

22

3. INDICATOR LIST

CGIAR QA PLATFORM  Initiatives

USER MANUAL

1

3

4

2

Peer-reviewed Papers example. The columns of the table vary according to the indicator.

1. Tech reporting guidance

This links directly to the Technical Reporting guidance.

3. Status bar

This displays the status of each item for the selected indicator. There are 2 possibilities.

4.  Additional columns

This enables and disables columns in the table, depending on the user's interest. Note that the columns "Highlighted comments", “Third party broker instructions", and "Implemented decisions" will be used only in Round 2 and not by Initiatives.

​

2. Download comments button

This downloads all comments for this indicator and the actual Initiative session in an Excel file.

5

5.  Search filter

This search bar filters the list of items by common words in the title.

23

CGIAR QA PLATFORM  Initiatives

USER MANUAL

1. Detailed route

This clarifies what item an Initiative is currently

working on by detailing the path (Initiative/Indicator/Indicator ID).

3. Download comments button

By clicking on the “download” icon, the Initiative user can access all comments made for this item in an Excel file.

4. INDICATOR DETAILS

5. Assessment with comments

This item was assessed and there are comments that need a reply from the Initiative.

1

3

2

5

2. Response status

This indicates whether the comments provided by the assessors have already been answered or not.

4. Approved without comments

This indicates that an item is approved. No more action is required.

4

24

CGIAR QA PLATFORM  Initiatives

USER MANUAL

1. “Accept” button

By clicking on “accept”, an Initiative approves the assessors' suggestion of change. 

1

2

3

5. INDICATOR COMMENTS SECTION

2. “Disagree” button

Clicking on “Disagree” opens a new text area for an Initiative to explain the reasons for disagreement with an assessors’ comments.

3. Reply text area

This is the text area to submit an explanation of disagreement (1000 words max.). Users click the ADD button to validate the comment.

Note: Changes made in the PRMS Reporting Tool are not implemented in real time within the QA Platform. Changes could take a few hours to appear in the QA Platform.

4

4. Accept comment pop-up

This window opens when an Initiative clicks on the “Accept” button. The pop-up window allows users to confirm their acceptance and to commit to implementing the change in the PRMS Reporting Tool.

4. Instructions recap and next steps

Additional Resources

Webinars

• Live Virtual Tour of the QA Platform with real data�To be scheduled week of 18 November

​

Contact

• General QA support email: PRMSTechSupport@cgiar.org
• For general enquiries on the QA process: performanceandresults@cgiar.org the PPU and the technical team will reply.

​

Support Materials

• CGIAR Quality Assurance Process for Type 1 Technical Reports
• Type 1 Technical Reporting Guidance
• Quality Assurance Assessor Guidance
• Guidance on the Other Output and Other Outcomes categories
• QA Platform Users Manual
• Brief report on 2023 Quality Assurance�

Please see P&R for further updates included webinar recording, PPT slides, Guidance documents and FAQs.

​

​

​

​

www.cgiar.org

What it means for Initiatives: recap Batch 1

You hold the final word on regular data fields

The 3rd party holds the final word on core data fields

www.cgiar.org

Thank You!