1 of 31

��Applications-Oriented Benchmarks for Quantum Computing��

LNE, THALES, EVIDEN, CEA, CNRS, TERATEC

Presented by Frédéric BARBARESCO (THALES)

May 4^th 2023

MetriQs-France

National Program on�Measurement, Standards and Evaluation�of Quantum Technologies�

www.thalesgroup.com

1

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

2 of 31

MetriQs-France Project « BACQ »�Benchmarks Applicatifs des Calculateurs Quantiques

BACQ PROJECT

Part of LNE MetriQs-France Program supported by the French National Quantum Strategy
Global Budget: 4 M€ including 2.5 M€ France 2030 funding
Objective:

R&D Project, funded by MetriQs-France (France 2030), to elaborate full benchmarks integrated in a « testbed », to be deployed by quantum computers providers and users
Promotion at international level

Partners

Coordinator: THALES
Partners: CEA (LIST, IPhT, PHELIQS), EVIDEN, CNRS, TERATEC, LNE

Duration: 36 months
Outputs

Fast Track on Q-SCORE metrics
Prototype of applications-oriented benchmarks
Multi-criteria model of quantum computers notation

2

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

3 of 31

MetriQs-France & BACQ Phases

Main Goal

With the goal of developing a reliable, objective and long-lasting measurement instrument for the performance of quantum computers in terms of applications, BACQ aims to develop all the benchmarks constituting the corresponding "testbed", to exploit and promote it internationally. The project is part of the National program on measurement, standards and evaluation of quantum technologies MetriQs-France.

2 Phases of deployment

A first phase consists of identifying and developing the first elements (software bricks) of the first candidate demonstrations (at TRL 3) which will then make it possible to specify the testbed. This first step is the main part of the project.
A second phase will then be necessary to develop the testbed and make it usable by the community (TRL >5).

3

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

4 of 31

BACQ: Applications-Oriented Benchmark & Multi-criteria Approach

Drawbacks of existing benchmarks

A number of initiatives exist to benchmark the performance of quantum computers. Examples include Quantum VOLUME and CLOPS from IBM, SupermarQ from Super-Tech or Quantum LINPACK from Berkeley Lab and QED-C Benchmarks. The metrics used in these approaches are relatively technical and require some knowledge of the underlying technologies. They often do not provide operational indicators of the performance of the different families of algorithms executed on the different existing quantum platforms.

Main advantages of BACQ: End-Users-Oriented Multi-Criteria Benchmarks

We will therefore seek to go back to higher-level metrics, meaningful for industrial users. The proposed approach therefore aims to initiate a multi-criteria analysis approach in order to help such an evaluation of the different quantum solutions. THALES jointly provides a methodology and a “MYRIAD” tool for this aggregation of low-level technical metrics towards the qualities of services of interest to the user.

4 types of application problems will be studied to develop benchmarks:

Quantum physics model simulation, linear systems, optimization and factorization.

4

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

5 of 31

BACQ Re-Use by LNE MetriQs-France Program

BACQ testbed operation under the supervision of LNE will aim to:

Enable the users to perform assessment of quantum computers by their own and analysis of the results with explicable, transparent and impartial protocols.
Use the results obtained on the tested machines to refine the definition of the testbed.
At medium term, enable to establish and maintain a performance list over time with consent of the participants : access to the results and right to examine them before any publication.
Develop communication and presentation material for the project (manifestos, brochures, publications, website, seminars, etc.) to encourage adhesion and participation.
Set up European and international dialogue on the subject of benchmarking quantum computers to build common practices.
Promote the developed benchmarks in international standards, with regard to the specifications of quantum machines and their evaluation methods.

5

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

6 of 31

BACQ Work Break Structure

WP0: Project Management

WP2: « Benchmarks » tests definition

WP1: Consultations, Disseminations & Exploitations

WP4: Implementation / Coding of Tests

WP5: Analysis & Consolidation of results

WP3: Energetic Criteria on Quantum Gates Computer

Fast Track Q-SCORE on QPUs

TERATEC

Benchmark Working Group

Norms & Standards

(AFNOR, CEN/CENELEC, ETSI, UIT-T, ISO, IEC)

End-Users + Research Teams

QPU Providers + other benchmarking Initiatives

Publications

Benchmarks

Protocols

METRICS

Energetic METRICS

MULTI-CRITERIA MODEL

CODES &

PROTOCOLS

TGCC

Emulators: QLM, Perceval, Pulser
QPU: PASQAL,…

Other QPUs

CONSENSUS

6

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

7 of 31

BACQ Workpackages Objectives

WP0 Project Management: This WP is in charge of managing the project on a daily basis but also of interacting with the WG in France to ensure the consistency of the work carried out with the national ambitions and strategy.
WP1 Consultations, Dissemination & Exploitation: This WP aims to ensure the visibility of the project at the national and international level, to promote its results and to share its ambitions with other studies in progress at the global level.
WP2 Benchmarks Tests Definition: The objective of this WP is to identify the different algorithmic approaches adapted to “benchmark” tests. For this, the WP partners will help identify the criteria and divide them into different classes (criteria linked to the problem addressed by a benchmark, criteria linked to the method or to the different calculation methods and criteria linked to the machines). And finally, identify the generic multi-criteria approach to build an aggregate score
WP3 Energetic Criteria: The objective of the WP is to propose, simulate and test energy efficiencies for useful algorithms studied in this project. The lot will study in detail the case of a gate-based machine performing a simple algorithm such as solving a linear system. This is about creating models and optimizations using tools and data from multiple levels of description.
WP4 Implementation/Coding of Tests: This lot of tasks is at the heart of the project insofar as the objective here is to implement (code) and develop on real quantum machines (possibly in parallel with the use of emulators ) the test protocols that will have been proposed for all the tasks of WP 2.
WP5 Analysis & Consolidation of results: This WP consists of collecting the codes and the first experimental results carried out in WP 4. Then it will carry out a critical and statistical analysis of the results. These results will be consolidated through successive refinements with feedback from QPU vendors on how to best utilize QPUs.

7

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

8 of 31

Planning

8

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

9 of 31

BACQ Applications-Oriented Problems for End-Users

Simulation of Quantum Physics models

Optimization

Linear Systems Solving

Factorization

(Hamiltonian version)

9

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

10 of 31

Initial Metrics provided by partners

Simulation of Quantum Physical Models

Q-Score Many-Body (Eviden)
Many-Body Fidelity (CEA-DRF)

Optimization

Q-Score Max-Cut (Eviden)
Accuracy indexed by noise level (THALES)
Compilation-dependant criteria (THALES)
Probability of obtaining the optimum(CEA-LIST)
Min case/Max case Gap wrt Optimum (CEA-LIST)
Size of problem in number of variables, number of Qbits to solve the problem, Class of problem addressed (CEA-LIST).

Linear systems Solving

Accuracy indexed by noise level (THALES)
Compilation-dependant criteria (THALES)
Probability of solving the Problem, Number of variables and precision in Qbits.(CEA-LIST)
Algorithm-dependant Energetic criteria (CNRS)

Factorization

Probability to factorize(CEA-LIST)
Size in Qubit of the calculated number (CEA-LIST)

Generic

Computation Time
Latency
Throughput

10

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

11 of 31

Defining resource efficiency

11

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

12 of 31

Benchmark Metrics for QPU and Quantum Algorithms

Drawback of existing Quantum �Benchmarking Rules/Tools

Very technical metrics

SupermarQ Suite

(Super-Tech)

Quantum LINPACK

(Berkeley Lab)

CLOPS

(IBM)

How to aggregate all these metrics to compare several quantum solutions?

How to obtain

an overall assessment interpretable for end-users?

For the 3 QPUs

Error impact / accuracy
Latency
Throughput
Accessibility :

For algorithms for dedicated resources

Advantage
Input
Noise
Scaling

Programming
Supported applications
Power Consumption

Accuracy
Non-linear
Output
Flexibility

Metrics

Use of Multi-Criteria Decision Aiding

MYRIAD = methodology + Tool
Already used with French MoD DGA and NATO

QED-C

(USA)

12

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

13 of 31

MYRIAD approach at a glance

STEP 1: Problem formalization and construction of the evaluation function

Aggregation layer

Criteria layer

Metrics

Structuration phase

Identify relevant metrics
Organize them in a tree

Criteria = normalisation

Elicitation: 2 references + Intensities of pref

Aggregation of normalized scores

Representation of complex decisions

Elicitation: Learn model from examples of solution comparisons

13

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

14 of 31

MYRIAD approach at a glance

STEP 2: Application of the evaluation function on quantum solutions

Model intrinsically interpretable

Hierarchical where each node makes sense
All nodes have the same semantics (good 🡺 bad)
Interpretation of aggregation nodes

Explainability

Local explanation

Graphical �explanation of�aggregation

Global explanation

Feature attribution
Worth to improve
Sensitivity analysis

14

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

15 of 31

QPUs Benchmarks jointly on all problems or for one problem

OpenQASM

(JsonAwsQuantumCircuitDescription)

Script Modèle Ising

(QUBO)

Physics Simukation

Optimization

Linear System

Factorization

Problème X

QPU 1

QPU 2

QPU 3

QPU 4

15

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

16 of 31

��Partners Contribution��

LNE, THALES, EVIDEN, CEA, CNRS, TERATEC

MetriQs-France

National Program on�Measurement, Standards and Evaluation�of Quantum Technologies�

www.thalesgroup.com

16

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

17 of 31

THALES contribution in BACQ: KPI aggregation with MYRIAD

MYRIAD at a glance

MYRIAD added-values

Aggregate all KPIs into a unique quality score …

KPI identification

KPI normalization

KPI

aggregation

Solutions

assessment

Solutions

explanation

Problem Formalization

Performance assessment

The model is fully interpretable

Explanable of the evaluations

The model can be shared among stakeholder

… making it possible to rank solutions

Representation of interacting criteria

Model constructed from preferences of experts

17

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

18 of 31

THALES contribution to BACQ: Metrics Elaboration

Focus on two applications: Linear System Solving (Finite Elements EM simulations) & optimization problems
Focus on NISQ & FTQC QPUs
Definition of two metrics following 3 phases:

1^st phase:

Evaluation of physical characteristics of a selected set of QPUs to better understand noise origins (with the help of QPU providers) & noise modelling.

2^nd phase:

Study of the impact of physical errors on a set of logic algorithm bricks.
Identification of error correction codes supported by each QPU.
Estimation of the induced overhead on the algorithm bricks.
Evaluation of the impact of error propagation on real QPU machines.
Definition of a performance metrics related to the noise level & computing accuracy.

3^rd phase:

Study of the impact of these observations on more complex algorithm compilation (according to connectivity constraints).
Test and measures on real QPU machines.
Definition of an overhead metrics due to the compilation.

Deterministic methods (Finite Element)

18

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

19 of 31

CNRS Contribution to BACQ: Energetic Benchmarks for Gate-Based

Goal: propose and simulate energetic benchmarks

Energy is critical for competitivity and soveraignty
Currently no energetic benchmark for quantum computing => France can take a leadership
Challenges: methodologic (interdisciplinarity), strategic (acceptation of the benchmark)

MNR (metrics noise resource)

Under study: a simple gate-based quantum machine
Build up of a systemic hardware-agnostic approach

D3.1 List of control parameters and algorithmic resources for a gate-based quantum machine performing a simple task. Design and exploitation of an Algorithmic Resource Estimator for the studied problem (M18)

D3.2 Optimization of the algorithmic efficiency for a fixed noise and effective relation to the energetic performance of the machine (M36)

Working group Quantum Energy Initiative @ IEEE

D3.3 (M12, M24, M36): Proposition of frameworks and contribution of the validation of the standard

19

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

20 of 31

Gate-based and analogue simulations of condensed-matter inspired quantum many-body models CEA/DRF: IPhT & IRIG/Pheliqs

Idea: develop metrics to quantify the capacity of a quantum machine to simulate difficult and physically relevant quantum problems (example: quantum phase transitions).
Criteria: many-body fidelity of state preparation (target non-trivial N-qubit entangled states), accuracy of dynamics simulation, accuracy of extracting physical quantities of interest.

Gate-based and analogue application for optimization, linear system solving and (Hamiltonian-based) integer factorization CEA/DRT : LIST/DSCIN

Idea: develop metrics to evaluate quantum computers on several classes of optimization problems (from easy-sparse, to difficult-dense problems), linear systems (at different sizes), and integer factorizations (with an increasing complexity)
Criteria: probability of finding the optimum or distance to optimum (optimization), size, complexity and sparsity of the associated problem;

CEA contribution to BACQ: Metrics Elaboration

20

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

21 of 31

Eviden activities within BACQ project

On going work

Leading Fast-Track (March 2023 - December 2023)

Future work with BACQ collaborators

Define a new metric (Q-score manybody) �and collect the metrics of other collaborators
Make the partners QPU accessible to run tests on them
Gather and analyse results of the runs�

Eviden gains

QLM is designed for interoperability and this project is an opportunity to prove it on an extensive set of various technologies of QPUs
Extending the Q-score methodology to another metric (manybody) is an opportunity to remind its qualities and promote its adoption.

QPU type 1

QPU type 2

QPU type 3

QPU type 4

21

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

22 of 31

QUESTIONS

��Applications-Oriented Benchmarks for Quantum Computers��

frederic.barbaresco@thalesgroup.com

felicien.schopfer@lne.fr

22

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

23 of 31

BACQ�FAST-TRACK Q-SCORE��

LNE, THALES, EVIDEN, CEA, CNRS, TERATEC

Presented by Anne-Lise Guilmin (EVIDEN)

MetriQs-France

National Program on�Measurement, Standards and Evaluation�of Quantum Technologies�

www.thalesgroup.com

23

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

24 of 31

Fast Track phase of BACQ initiative

Opportunity to introduce BACQ to potential partners
Gather early partners to join a preliminary phase test on an existing metric

An application oriented metric for quantum computation is already defined: Q-score maxcut.
It has been designed to be generic.
As part of Fast-Track, we want to run it on various technologies and gain rapid experience on the Q-score evolutions required to universality and the challenges along the BACQ project.
Q-score is a metric. It defines a way to score a QPU for a specific purpose. It does not specify how the QPU is supposed to achieve this purpose. In order to run Q-score, we first need to find an implementation which fits each machine we test.

24

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

25 of 31

What is the Q-score maxcut metric?

Q-score maxcut addresses a complex problem (maxcut) �for which a quantum advantage is expected.

The maxcut problem only has a meaning for a size of graph >= 5 vertices

Methodology to assess if a QPU can solve a given size of graphs

Classically compute the average solution for a family of graphs of this size
Randomly choose n graphs among this family of graphs
Execute it on a quantum machine
Compute the quantum average and its standard deviation to the classical average

Q-score is the maximal size of graphs that the quantum machine can handle�
Q-score is scalable because the reference used to validate a success is an average on a random sample of graphs which is easily computed classically; despite individual graphs being complex to solve classically.

25

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

26 of 31

The Q-score protocol in practice

Running Q-score on various technologies requires to write an implementation of the maxcut problem for each qubit technology and each programming model (quantum gate, analog, annealing).�
New questions arrived:

Due to topology, some QPUs are able to run some graphs but not any graphs of a given size. How to rate them?�
Today, QPUs are not able to solve bigger size of graphs than classical machines. How to assess the part of classical computing in a hybrid computation?

26

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

27 of 31

Partners contacted to participate in Fast-Track

Sorted by qubit technology

Photonic

Psiquantum (Palo Alto, US)
Quandela (Massy, France)
QuiX (Enschede, The Netherlands) via TNO
Xanadu (Toronto, Canada)

Superconductors

Alice & Bob (Paris, France)
Google (Santa Barbara County, US)
IBM (Cambridge, US) via TNO
IQM (Espoo, Finland)
OQC (Oxford, UK)

Trapped ions

AQT (Innsbruck, Austria)
Quantinuum (Cambridge, UK)

Spin qubits

C12 (Paris, France)
Siquance (Grenoble, France)

Neutral atoms

Pasqal (Massy, France)
QuEra (Boston, US)

Annealing

D-Wave (Burnaby, Canada) via TNO and FZJ
Fujitsu (Kanagawa, Japan) via Riken via CEA
NEC (Tamagawa, Japan) via Riken via CEA

NV centers in diamond

Quantum Brilliance (Acton, Australia)

* Indirect access via collaborators to be discussed

* Collaboration under discussion

27

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

28 of 31

Partners contacted to participate in Fast-Track �and interest points for future collaborations with BACQ initiative

Image by rawpixel.com on Freepik

28

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

29 of 31

Done and on-going work

Past

Q-score definition, by Atos [DOI: 10.1109/TQE.2021.3090207]
Q-score maxcut on D-Wave, by TNO [DOI: 10.1109/QSW55613.2022.00017]
Q-score maxcut on Pasqal emulator, by Pasqal [https://arxiv.org/abs/2207.13030]�

Fast-Track achievement so far

Q-score maxcut implementation on Quandela emulator

On going activities:

Q-score maxcut implementation on IQM and Quantinuum emulators
Q-score maxcut run on Quandela and Pasqal hardware
Q-score maxcut run on a photonic hardware (when the machine is available in TGCC )
Brainstorming with A&B around other possible Q-score metrics
On-going discussions to formalize engagement with partners (MoU)

29

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

30 of 31

Invitation to participate

BACQ is a 3-year project.
Partners are welcome to test Q-score and other metrics to come.

If interested, please contact Félicien Schopfer and Frédéric Barbaresco

felicien.schopfer@lne.fr

frederic.barbaresco@thalesgroup.com

THANK YOU!

30

SBP 2022-2026 – multi-GBU Key Question – Boost quantique

31 of 31

QUESTIONS

TQCI (Teratec Quantum Computing Initiative) Seminar

May 11^th, THALES Research & Technology, Palaiseau
Overview of upcoming application-oriented benchmarks for quantum computing in France and abroad

31

SBP 2022-2026 – multi-GBU Key Question – Boost quantique