1 of 30

ESR12: Accelerated Anomaly Detection

​

Pratik Jawahar

Supervisors:�Caterina Doglioni, Jiri Masik, Alex Oh,�Maurizio Pierini

1

Overview:

  • Qualification Task - Heterogenous Tracking
  • ML based Data Compression - Baler
  • Misc. Activities

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

2 of 30

About me:

  • BS: Mechanical Engineering (2019)
  • MS: Robotics Engineering (2022)
  • PhD: Robo.. *sike* Particle Physics
  • Experience:
    • Summer student (CERN, 2018)
      • Control algorithms for GEM detectors (CMS)
    • Technical student (CERN, 2021), DIANA HEP Fellowship (2020)
      • ML based anomaly detection for new physics searches
  • Website: https://www.pratikjawahar.com/

2

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

3 of 30

Qualification Task: Heterogenous Track Reconstruction

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

4 of 30

Track Reconstruction:

  • Reconstruct trajectories of charged particles as they pass through different layers of the detector, subject to a magnetic field

4

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

5 of 30

Track Reconstruction:

  • Step 1: Detect a charged particle when it passes through a layer
    • Ionization in semiconductors

5

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

6 of 30

Track Reconstruction:

  • Step 2: Clusterization
    • Depends on how you treat readout, detector design etc.
    • Connected Components Algo (CCA) iterated on edges

6

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

7 of 30

Track Reconstruction:

  • Step 3: Spacepoint Formation
    • Move from local sub-detector frame to global detector frame

7

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

8 of 30

Track Reconstruction:

  • Step 4: Seeding
    • Find triplets of space points that could potentially belong to the same track
      • Conformal maps => Hough Transform

8

x

y

𝛳

r

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

9 of 30

Track Reconstruction:

  • Step 5: Track finding and fitting
    • Start from a seed
      • Kalman formalism (eg. Combinatorial KF) => Smoothing (eg. global 𝛘2 fit or walk back with Kalman fit) => Ambiguity resolution

9

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

10 of 30

Problem(s)!

  • No. of tracks per event for HL-LHC, expected to increase 2.5x
  • Current R&D: Use GPUs for speedup via parallel computation
  • However, sequential algorithms like CKF do much better on CPUs than GPUs!

Computing term for specific purpose architectures (eg. GPU, TPU, IPU etc.)

10

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

11 of 30

Track Reconstruction:

  • Heterogeneous Track Reconstruction:
    • Step 1: Profiling CPU and GPU code to identify speed-up in inference time
      • Ideally without drop in tracking efficiency
    • Step 2: Identify bottlenecks
      • Points where one architecture outperforms the other
    • Step 3: Calculate data-transfer latencies at bottlenecks
      • Data transfer latencies between host (CPU) and device (GPU) eat up speed-up

11

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

12 of 30

CUDA Profiling

12

D2H Calculations - CPU vs D2H CPU

D2H Calculations - CPU vs CUDA

Step 1: Ensure GPU computations do not decrease physics performance

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

13 of 30

CUDA Profiling - NSight Compute

13

Step 2: Profile code in as close-to-deployment conditions as possible

CUDA tools: NSight Compute, NSight Systems

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

14 of 30

CUDA Profiling - NSight Compute

14

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

15 of 30

CUDA Profiling

15

CPU

​

​

​

GPU

​

​

​

Parent process

Duration [mu-sec]

​

​

Parent process

Duration [mu-sec]

​

​

​

​

​

​

​

​

Container Instantiation

8

​

​

Container Instantiation

6

​

File reading

1,720,207

​

​

File reading

NA

​

Clusterization

40,197

​

​

Clusterization

NA

​

Spacepoint Formation

6,078

​

​

Spacepoint Formation

NA

​

Clusterization + Spacepoints

46,275

​

​

Clusterization + Spacepoints

40,197

​

Seeding

1,075,583

​

​

Seeding

80,072

​

Track param est

27,909

​

​

Track param est

2,651

CPU

​

​

​

GPU

​

​

​

Parent process

Duration [mu-sec]

​

​

Parent process

Duration [mu-sec]

​

​

​

​

​

​

​

​

Container Instantiation

8

​

​

Container Instantiation

6

​

File reading

1,288,893

​

​

File reading

NA

​

Clusterization

36,070

​

​

Clusterization

NA

​

Spacepoint Formation

3,131

​

​

Spacepoint Formation

NA

​

Clusterization + Spacepoints

39,201

​

​

Clusterization + Spacepoints

2,433

​

Seeding

792,729

​

​

Seeding

11,686

​

Track param est

9,316

​

​

Track param est

411

Data File

Detector Geometry

No. of Events

tml_full/ttbar_mu300

tml_detector/trackml-detector

1

Old Version

New Version - with FastSV

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

16 of 30

CUDA Profiling - NSight Compute

16

Data File

Detector Geometry

No. of Events

tml_full/ttbar_mu300

tml_detector/trackml-detector

1

Old Version

New Version - with FastSV

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

17 of 30

CUDA Profiling - NSight Systems

17

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

18 of 30

CUDA Profiling

  • POC feasibility example:
    • Clusterization, Spacepoint formation, Seeding are significantly faster on GPU
    • Considering Host-Device and Device-Host wall-time overheads,
      • there is still a speedup of ~O(800 msec)
  • *Note* This is only an example for one event

18

CPU

​

​

​

GPU

​

​

​

Parent process

Duration [mu-sec]

​

​

Parent process

Duration [mu-sec]

​

​

​

​

​

​

​

​

Container Instantiation

15

​

​

Container Instantiation

5

​

File reading

1,341,172

​

​

File reading

NA

​

Clusterization

36,070

​

​

Clusterization

NA

​

Spacepoint Formation

3,131

​

​

Spacepoint Formation

NA

​

Clusterization + Spacepoints

39,201

​

​

Clusterization + Spacepoints

2,433

​

Seeding

814,789

​

​

Seeding

11,686

​

Track param est

9,316

​

​

Track param est

411

Data File

Detector Geometry

No. of Events

tml_full/ttbar_mu300

tml_detector/trackml-detector

1

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

19 of 30

CUDA Profiling

  • POC feasibility example:
    • Clusterization, Spacepoint formation, Seeding are significantly faster on GPU
    • Considering Host-Device and Device-Host wall-time overheads,
      • there is still a speedup of ~O(800 msec)
  • If CKF on GPU is considerably slower, it may still be feasible to run the first part of the chain on Device and move data to Host for CKF
    • Potential solution for net speedup in the tracking chain
  • *Note* This is only an example for one event

19

Data File

Detector Geometry

No. of Events

tml_full/ttbar_mu300

tml_detector/trackml-detector

1

CPU

​

​

GPU

​

​

Parent process

Duration [mu-sec]

​

​

Parent process

Duration [mu-sec]

​

​

​

​

​

​

Host to Device [Cells]

3035

​

​

Device to Host [Cells]

NA

Host to Device [Spacepoints]

2,703

​

​

Device to Host [Spacepoints]

1087

Host to Device [Seeds]

781

​

​

Device to Host [Seeds]

349

Host to Device [Track params]

1,655

​

​

Device to Host [Track params]

1,161

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

20 of 30

Next Steps

  • Extend studies to larger number of events with more realistic data (eg. mu=60, 140, 200 etc)
  • Use seeds generated by traccc in ACTS CKF function to understand Device-Host costs
  • Repeat overhead studies wrt other parameters such as Memory, GPU Occupancy etc.

20

CPU

​

​

GPU

​

​

Parent process

Duration [mu-sec]

​

​

Parent process

Duration [mu-sec]

​

​

​

​

​

​

Host to Device [Cells]

3035

​

​

Device to Host [Cells]

NA

Host to Device [Spacepoints]

2,703

​

​

Device to Host [Spacepoints]

1087

Host to Device [Seeds]

781

​

​

Device to Host [Seeds]

349

Host to Device [Track params]

1,655

​

​

Device to Host [Track params]

1,161

Data File

Detector Geometry

No. of Events

tml_full/ttbar_mu300

tml_detector/trackml-detector

1

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

21 of 30

Baler: ML based Data Compression

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

22 of 30

Baler: ML based Data Compression

22

Paper: https://arxiv.org/abs/2305.02283

Code: https://github.com/baler-collaboration/baler

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

23 of 30

Misc Activities

Side Quests:

  • Helped lift new wire bonder into the Lund Uni clean room
  • Working on DDMTD corrections for mu-CTP trigger chip for temperature variance with Kalman Filters

Hackathons:

  • DL in HEP - MIT
  • Baler (Lund)

Workshops/Conferences:

  • IOP APP&HEPP UK National - Talk
  • FastML - Talk
  • Hammers and Nails - Poster

Schools:

  • tCSC Split
  • HASCO particle physics
  • STFC UK theory summer school

Teaching/Outreach:

  • Jupyter Notebooks Intro - Lund
  • Advanced C++ - Manchester

23

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

24 of 30

Conclusions & Future Work

  • Track reconstruction may not scale to HL-LHC
    • R&D required to speed up track reconstruction
      • without loss in tracking efficiency
      • GPUs could be a viable option
        • GPUs may not be able to speed up the full chain due to sequential algo (eg. CKF)
    • Heterogeneous computing solutions may be feasible
      • POC studies show:
        • First half of the chain on GPUs, followed by offloading to CPUs for CKF, could show order of 100msec speedup per event
  • Work on novel anomaly detection solutions?
  • Contribute to the HLS4ML project?
  • Analysis: Semi-visible jet signatures for dark sector searches - ML based signal-background discrimination?
  • TBD!

24

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

25 of 30

Thank you!

Questions?

People are hungry, just putting it out there :)

Feel free to chat later as well! :)

25

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

26 of 30

BACKUP

26

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

27 of 30

27

CPU

​

​

​

​

​

​

​

​

​

​

​

​

​

CUDA

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

28 of 30

28

CPU

​

​

​

​

​

​

​

​

​

​

​

​

​

CUDA

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

29 of 30

CUDA Profiling

29

Old Version

  • Seeding - Synchronous kernel launches
    • fill_prefix_sum used for synchroniszation

New Version - with FastSV

  • Seeding - Asynchronous kernel launches
    • Increases paralellization

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

30 of 30

CUDA Profiling- NEW

30

Data File

Detector Geometry

No. of Events

tml_full/ttbar_mu300

tml_detector/trackml-detector

1

Old Version

count_doublets()

New Version - with FastSV

count_doublets()

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086

ESR12: Pratik Jawahar

SMARTHEP is funded by the European Union’s Horizon 2020 research and innovation programme, call H2020-MSCA-ITN-2020, under Grant Agreement n. 956086