1 of 20

Fast Machine Learning for Science Workshop 2022

Southern Methodist University

10/04/2022

Quantized Distilled Autoencoder Model on FPGA for Real-Time Crystal Structure Detection in 4D STEM

Credit: Dr. Josh Agar

1

2 of 20

Background - 4D STEM

  • Electron particle beam transmitted through material
  • 2D diffraction image captured at each pixel position (4D)
    • 4D data package generated as a result (256, 256, 120, 120)
  • We can reconstruct crystal image from resulting diffraction patterns
    • Contains a lot of information about the material structure, properties, defects

256

256

120x120

single diffraction image

Four-Dimensional Scanning Transmission Electron Microscopy (4D-STEM): From Scanning Nanodiffraction to Ptychography and Beyond - Scientific Figure on ResearchGate. Available from: https://www.researchgate.net/figure/Experimental-4D-STEM-measurement-of-a-dichalcogenide-2D-material-Atomic-map-is-inferred_fig1_333090886 [accessed 17 Aug, 2022]

Gatan. (2017). 4D Stem operating principle. YouTube. Retrieved August 17, 2022, from https://www.youtube.com/watch?v=tV4hCOZm41Y.

2

3 of 20

Real-Time Detection of Crystal Structure, Rotation, and Strain in 4D STEM

Credit: Shuyu Qin and Dr. Josh Agar

Real-time data reduction codesign for the extreme edge - ProjectCheckin-Apr2022

3

Predicted Transformation (6 value output)

  • Rotation (2)
  • Scale (2)
  • Shear (2)

Credit: Dr. Josh Agar

4 of 20

Original PyTorch Architecture

_______________________________________________________________

Layer (type) Output Shape Param #

================================================================

Conv2d-1 [-1, 128, 120, 120] 1,280

Conv2d-2 [-1, 128, 120, 120] 147,584

ReLU-3 [-1, 128, 120, 120] 0

Conv2d-4 [-1, 128, 120, 120] 147,584

ReLU-5 [-1, 128, 120, 120] 0

Conv2d-6 [-1, 128, 120, 120] 147,584

LayerNorm-7 [-1, 128, 120, 120] 28,800

ReLU-8 [-1, 128, 120, 120] 0

conv_block-9 [-1, 128, 120, 120] 0

Conv2d-10 [-1, 128, 120, 120] 147,584

LayerNorm-11 [-1, 128, 120, 120] 28,800

ReLU-12 [-1, 128, 120, 120] 0

identity_block-13 [-1, 128, 120, 120] 0

MaxPool2d-14 [-1, 128, 30, 30] 0

Conv2d-15 [-1, 128, 30, 30] 147,584

ReLU-16 [-1, 128, 30, 30] 0

Conv2d-17 [-1, 128, 30, 30] 147,584

ReLU-18 [-1, 128, 30, 30] 0

Conv2d-19 [-1, 128, 30, 30] 147,584

LayerNorm-20 [-1, 128, 30, 30] 1,800

ReLU-21 [-1, 128, 30, 30] 0

conv_block-22 [-1, 128, 30, 30] 0

Conv2d-23 [-1, 128, 30, 30] 147,584

LayerNorm-24 [-1, 128, 30, 30] 1,800

ReLU-25 [-1, 128, 30, 30] 0

identity_block-26 [-1, 128, 30, 30] 0

MaxPool2d-27 [-1, 128, 10, 10] 0

Conv2d-28 [-1, 128, 10, 10] 147,584

ReLU-29 [-1, 128, 10, 10] 0

Conv2d-30 [-1, 128, 10, 10] 147,584

ReLU-31 [-1, 128, 10, 10] 0

Conv2d-32 [-1, 128, 10, 10] 147,584

LayerNorm-33 [-1, 128, 10, 10] 200

ReLU-34 [-1, 128, 10, 10] 0

conv_block-35 [-1, 128, 10, 10] 0

Conv2d-36 [-1, 128, 10, 10] 147,584

LayerNorm-37 [-1, 128, 10, 10] 200

ReLU-38 [-1, 128, 10, 10] 0

identity_block-39 [-1, 128, 10, 10] 0

MaxPool2d-40 [-1, 128, 5, 5] 0

Conv2d-41 [-1, 1, 5, 5] 1,153

Linear-42 [-1, 20] 520

Linear-43 [-1, 2] 42

LayerNorm-44 [-1, 2] 4

Softmax-45 [-1, 2] 0

Linear-46 [-1, 3] 69

================================================================

Total params: 1,835,676

Trainable params: 1,835,676

Non-trainable params: 0

-----------------------------------------

​

Original PyTorch Encoder Model

Credit: Shuyu Qin

4

5 of 20

Model Distillation Architectures

_________________________________________________________________

Layer (type) Output Shape Param #

=================================================================

input_1 (InputLayer) [(None, 120, 120, 1)] 0

average_pooling2d (AverageP (None, 60, 60, 1) 0

ooling2D)

flatten (Flatten) (None, 3600) 0

q_dense (QDense) (None, 128) 460928

batch_normalization (BatchN (None, 128) 512

ormalization)

q_activation (QActivation) (None, 128) 0

q_dense_1 (QDense) (None, 64) 8256

batch_normalization_1 (Batc (None, 64) 256

hNormalization)

q_activation_1 (QActivation (None, 64) 0

)

q_dense_2 (QDense) (None, 64) 4160

batch_normalization_2 (Batc (None, 64) 256

hNormalization)

q_activation_2 (QActivation (None, 64) 0

)

q_dense_3 (QDense) (None, 32) 2080

batch_normalization_3 (Batc (None, 32) 128

hNormalization)

q_activation_3 (QActivation (None, 32) 0

)

q_dense_4 (QDense) (None, 6) 198

=================================================================

Total params: 476,774

Trainable params: 476,198

Non-trainable params: 576

-----------------------------------------

Large Distilled MLP Model (quantized)

_________________________________________________________________

Layer (type) Output Shape Param #

=================================================================

input_1 (InputLayer) [(None, 120, 120, 1)] 0

average_pooling2d (AverageP (None, 30, 30, 1) 0

ooling2D)

flatten_1 (Flatten) (None, 900) 0

q_dense_1 (QDense) (None, 64) 57664

batch_normalization_3 (Batc (None, 64) 256

hNormalization)

q_activation_3 (QActivation (None, 64) 0

)

q_dense_2 (QDense) (None, 32) 2080

batch_normalization_4 (Batc (None, 32) 128

hNormalization)

q_activation_4 (QActivation (None, 32) 0

)

q_dense_3 (QDense) (None, 16) 528

batch_normalization_5 (Batc (None, 16) 64

hNormalization)

q_activation_5 (QActivation (None, 16) 0

)

q_dense_4 (QDense) (None, 6) 102

=================================================================

Total params: 60,822

Trainable params: 60,598

Non-trainable params: 224

-----------------------------------------

​

Smaller Distilled MLP Model (quantized)

5

_________________________________________________________________

Layer (type) Output Shape Param #

=================================================================

input_1 (InputLayer) [(None, 120, 120, 1)] 0

average_pooling2d (AverageP (None, 30, 30, 1) 0

ooling2D)

flatten (Flatten) (None, 900) 0

q_dense (QDense) (None, 64) 57664

batch_normalization (BatchN (None, 64) 256

ormalization)

q_activation (QActivation) (None, 64) 0

q_dense_1 (QDense) (None, 64) 4160

batch_normalization_1 (Batc (None, 64) 256

hNormalization)

q_activation_1 (QActivation (None, 64) 0

)

q_dense_2 (QDense) (None, 64) 4160

batch_normalization_2 (Batc (None, 64) 256

hNormalization)

q_activation_2 (QActivation (None, 64) 0

)

q_dense_3 (QDense) (None, 64) 4160

batch_normalization_3 (Batc (None, 64) 256

hNormalization)

q_activation_3 (QActivation (None, 64) 0

)

q_dense_4 (QDense) (None, 6) 390

=================================================================

Total params: 71,558

Trainable params: 71,046

Non-trainable params: 512

-----------------------------------------

​

Small Distilled MLP Model (quantized)

6 of 20

Model Distillation Results

1.8M Param PyTorch Encoder

60k Param Smaller Distilled MLP Model (quantized)

476k Param Large Distilled MLP Model (quantized)

Credit: Shuyu Qin

6

71k Param Small Distilled MLP Model (quantized)

7 of 20

MLP Avg. Pooling (Quantized 12-bit) HLS model

Model

ReuseFactor: 64

Strategy: Resource

Precision: ap_fixed<16,6,AP_RND>

LayerName

input_1

Precision

result: ap_fixed<16,6>

Trace: True

ReuseFactor: 64

average_pooling2d

Precision: ap_fixed<16,6>

Trace: True

ReuseFactor: 64

q_dense

Precision

weight: ap_fixed<12,4,AP_RND>

bias: ap_fixed<12,4,AP_RND>

ReuseFactor: 64

Trace: True

q_dense_linear

Precision: ap_fixed<16,6>

ReuseFactor: 64

table_size: 1024

table_t: ap_fixed<18,8>

Trace: True

batch_normalization

Precision

scale: ap_fixed<16,6>

bias: ap_fixed<16,6>

ReuseFactor: 64

Trace: True

q_activation

Precision

result: ap_ufixed<12,3>

ReuseFactor: 64

Trace: True

q_activation_quantized_relu

Precision

result: ap_ufixed<12,3>

ReuseFactor: 64

Trace: True

q_dense_1

Precision

weight: ap_fixed<12,4,AP_RND>

bias: ap_fixed<12,4,AP_RND>

ReuseFactor: 64

Trace: True

q_dense_1_linear

Precision: ap_fixed<16,6>

ReuseFactor: 64

table_size: 1024

table_t: ap_fixed<18,8>

Trace: True

batch_normalization_1

Precision

scale: ap_fixed<16,6>

bias: ap_fixed<16,6>

ReuseFactor: 64

Trace: True

q_activation_1

Precision

result: ap_ufixed<12,3>

ReuseFactor: 64

Trace: True

q_activation_1_quantized_relu

Precision

result: ap_ufixed<12,3>

ReuseFactor: 64

Trace: True

q_dense_2

Precision

weight: ap_fixed<12,4,AP_RND>

bias: ap_fixed<12,4,AP_RND>

ReuseFactor: 64

Trace: True

QKeras

HLS (new)

Model: "model"

_________________________________________________________________

Layer (type) Output Shape Param #

=================================================================

input_1 (InputLayer) [(None, 120, 120, 1)] 0

average_pooling2d (AverageP (None, 30, 30, 1) 0

ooling2D)

flatten (Flatten) (None, 900) 0

q_dense (QDense) (None, 64) 57664

batch_normalization (BatchN (None, 64) 256

ormalization)

q_activation (QActivation) (None, 64) 0

q_dense_1 (QDense) (None, 64) 4160

batch_normalization_1 (Batc (None, 64) 256

hNormalization)

q_activation_1 (QActivation (None, 64) 0

)

q_dense_2 (QDense) (None, 64) 4160

batch_normalization_2 (Batc (None, 64) 256

hNormalization)

q_activation_2 (QActivation (None, 64) 0

)

q_dense_3 (QDense) (None, 64) 4160

batch_normalization_3 (Batc (None, 64) 256

hNormalization)

q_activation_3 (QActivation (None, 64) 0

)

q_dense_4 (QDense) (None, 6) 390

=================================================================

Total params: 71,558

Trainable params: 71,046

Non-trainable params: 512

_________________________________________________________________

​

7

8 of 20

MLP Model Results on Alveo FPGA (5ns clock)

Latency Summary

+---------+---------+-----------+-----------+-------+-------+----------+

| Latency (cycles) | Latency (absolute) | Interval | Pipeline |

| min | max | min | max | min | max | Type |

+---------+---------+-----------+-----------+-------+-------+----------+

| 14467| 14468| 72.335 us | 72.340 us | 14405| 14405| dataflow |

+---------+---------+-----------+-----------+-------+-------+----------+

​

​

Latency by Layer

+------------------------------------------------------+---------+---------+-----------+-----------+-------+-------+----------+

| | Latency (cycles) | Latency (absolute) | Interval | Pipeline |

| Module | min | max | min | max | min | max | Type |

+------------------------------------------------------+---------+---------+-----------+-----------+-------+-------+----------+

|dense_array_array_ap_fixed_16_6_0_3_0_64u_config4_s | 964| 965| 4.820 us | 4.825 us | 964| 965| none |

|dense_array_array_ap_fixed_16_6_0_3_0_64u_config8_s | 66| 67| 0.330 us | 0.335 us | 66| 67| none |

|dense_array_array_ap_fixed_16_6_0_3_0_64u_config12_s | 66| 67| 0.330 us | 0.335 us | 66| 67| none |

|dense_array_array_ap_fixed_16_6_0_3_0_64u_config16_s | 66| 67| 0.330 us | 0.335 us | 66| 67| none |

|normalize_array_array_ap_fixed_64u_config6_s | 59| 59| 0.295 us | 0.295 us | 60| 60| function |

|normalize_array_array_ap_fixed_64u_config14_s | 61| 61| 0.305 us | 0.305 us | 62| 62| function |

|normalize_array_array_ap_fixed_64u_config10_s | 53| 53| 0.265 us | 0.265 us | 54| 54| function |

|normalize_array_array_ap_fixed_64u_config18_s | 47| 47| 0.235 us | 0.235 us | 48| 48| function |

|dense_array_array_ap_fixed_16_6_0_3_0_6u_config20_s | 66| 67| 0.330 us | 0.335 us | 66| 67| none |

|relu_array_array_ap_ufixed_64u_relu_config7_s | 0| 0| 0 ns | 0 ns | 1| 1| function |

|relu_array_array_ap_ufixed_64u_relu_config11_s | 0| 0| 0 ns | 0 ns | 1| 1| function |

|relu_array_array_ap_ufixed_64u_relu_config15_s | 0| 0| 0 ns | 0 ns | 1| 1| function |

|relu_array_array_ap_ufixed_64u_relu_config19_s | 0| 0| 0 ns | 0 ns | 1| 1| function |

|pooling2d_cl_array_array_ap_fixed_1u_config2_s | 14404| 14404| 72.020 us | 72.020 us | 14404| 14404| none |

+------------------------------------------------------+---------+---------+-----------+-----------+-------+-------+----------+

​

​

MLP Quantized (12-bit) Synthesis Report

Resource Utilization Summary

+---------------------+---------+-------+---------+---------+------+

| Name | BRAM_18K| DSP48E| FF | LUT | URAM |

+---------------------+---------+-------+---------+---------+------+

|DSP | -| -| -| -| -|

|Expression | -| -| 0| 2| -|

|FIFO | 1| -| 5199| 27893| -|

|Instance | 388| 1162| 87077| 150099| -|

|Memory | -| -| -| -| -|

|Multiplexer | -| -| -| 9| -|

|Register | -| -| 1| -| -|

+---------------------+---------+-------+---------+---------+------+

|Total | 389| 1162| 92277| 178003| 0|

+---------------------+---------+-------+---------+---------+------+

|Available SLR | 1344| 3072| 864000| 432000| 320|

+---------------------+---------+-------+---------+---------+------+

|Utilization SLR (%) | 28| 37| 10| 41| 0|

+---------------------+---------+-------+---------+---------+------+

|Available | 5376| 12288| 3456000| 1728000| 1280|

+---------------------+---------+-------+---------+---------+------+

|Utilization (%) | 7| 9| 2| 10| 0|

+---------------------+---------+-------+---------+---------+------+

​

​

➢ Params: 71k

➢ Precision: 12 bits

➢ Reuse: 16

➢ DSP Usage: 58.61%

  • MSE: 0.0121738293632542
  • Classified 65536 samples in 17.442984 seconds (3757.798165137614 inferences / s)
  • Or 266.1132812572639 us / inference

​

+---------------------+-------------------+------------------+-------------------+----------------+----------------+-----------------+

| Name | LUT | LUTAsMem | REG | BRAM | URAM | DSP |

+---------------------+-------------------+------------------+-------------------+----------------+----------------+-----------------+

| Platform | 142024 [ 8.22%] | 12528 [ 1.58%] | 217267 [ 6.29%] | 227 [ 8.44%] | 0 [ 0.00%] | 7 [ 0.06%] |

| User Budget | 1585016 [100.00%] | 778032 [100.00%] | 3236813 [100.00%] | 2461 [100.00%] | 1280 [100.00%] | 12281 [100.00%] |

| Used Resources | 56566 [ 3.57%] | 192 [ 0.02%] | 66830 [ 2.06%] | 105 [ 4.27%] | 0 [ 0.00%] | 1110 [ 9.04%] |

| Unused Resources | 1528450 [ 96.43%] | 777840 [ 99.98%] | 3169983 [ 97.94%] | 2356 [ 95.73%] | 1280 [100.00%] | 11171 [ 90.96%] |

| krnl_rtl | 56566 [ 3.57%] | 192 [ 0.02%] | 66830 [ 2.06%] | 105 [ 4.27%] | 0 [ 0.00%] | 1110 [ 9.04%] |

| krnl_rtl_1 | 56566 [ 3.57%] | 192 [ 0.02%] | 66830 [ 2.06%] | 105 [ 4.27%] | 0 [ 0.00%] | 1110 [ 9.04%] |

+---------------------+-------------------+------------------+-------------------+----------------+----------------+-----------------+

​

MLP Quantized (12-bit) on FPGA

  • Average Pooling issue: 72us latency (of total 72.3us)

Final Logic Synthesis Routing Report

+---------------------+-------------------+------------------+-------------------+----------------+----------------+-----------------+

| Name | LUT | LUTAsMem | REG | BRAM | URAM | DSP |

+---------------------+-------------------+------------------+-------------------+----------------+----------------+-----------------+

| Platform | 142021 [ 8.22%] | 12528 [ 1.58%] | 217277 [ 6.29%] | 227 [ 8.44%] | 0 [ 0.00%] | 7 [ 0.06%] |

| User Budget | 1585019 [100.00%] | 778032 [100.00%] | 3236803 [100.00%] | 2461 [100.00%] | 1280 [100.00%] | 12281 [100.00%] |

| Used Resources | 81598 [ 5.15%] | 6240 [ 0.80%] | 81488 [ 2.52%] | 113 [ 4.59%] | 0 [ 0.00%] | 1110 [ 9.04%] |

| Unused Resources | 1503421 [ 94.85%] | 771792 [ 99.20%] | 3155315 [ 97.48%] | 2348 [ 95.41%] | 1280 [100.00%] | 11171 [ 90.96%] |

| krnl_rtl | 81598 [ 5.15%] | 6240 [ 0.80%] | 81488 [ 2.52%] | 113 [ 4.59%] | 0 [ 0.00%] | 1110 [ 9.04%] |

| krnl_rtl_1 | 81598 [ 5.15%] | 6240 [ 0.80%] | 81488 [ 2.52%] | 113 [ 4.59%] | 0 [ 0.00%] | 1110 [ 9.04%] |

+---------------------+-------------------+------------------+-------------------+----------------+----------------+-----------------+

​

8

Resource Utilization by Layer

+-------------------------------------------------------+------------------------------------------------------+---------+-------+-------+-------+-----+

| Instance | Module | BRAM_18K| DSP48E| FF | LUT | URAM|

+-------------------------------------------------------+------------------------------------------------------+---------+-------+-------+-------+-----+

|dense_array_array_ap_fixed_16_6_0_3_0_64u_config12_U0 |dense_array_array_ap_fixed_16_6_0_3_0_64u_config12_s | 22| 64| 5411| 7405| 0|

|dense_array_array_ap_fixed_16_6_0_3_0_64u_config16_U0 |dense_array_array_ap_fixed_16_6_0_3_0_64u_config16_s | 22| 64| 5411| 7405| 0|

|dense_array_array_ap_fixed_16_6_0_3_0_64u_config4_U0 |dense_array_array_ap_fixed_16_6_0_3_0_64u_config4_s | 320| 960| 59761| 84649| 0|

|dense_array_array_ap_fixed_16_6_0_3_0_64u_config8_U0 |dense_array_array_ap_fixed_16_6_0_3_0_64u_config8_s | 22| 64| 5411| 7405| 0|

|dense_array_array_ap_fixed_16_6_0_3_0_6u_config20_U0 |dense_array_array_ap_fixed_16_6_0_3_0_6u_config20_s | 2| 6| 2627| 3635| 0|

|normalize_array_array_ap_fixed_64u_config10_U0 |normalize_array_array_ap_fixed_64u_config10_s | 0| 1| 1892| 5629| 0|

|normalize_array_array_ap_fixed_64u_config14_U0 |normalize_array_array_ap_fixed_64u_config14_s | 0| 1| 2043| 5569| 0|

|normalize_array_array_ap_fixed_64u_config18_U0 |normalize_array_array_ap_fixed_64u_config18_s | 0| 1| 1701| 5521| 0|

|normalize_array_array_ap_fixed_64u_config6_U0 |normalize_array_array_ap_fixed_64u_config6_s | 0| 1| 2030| 5637| 0|

|pooling2d_cl_array_array_ap_fixed_1u_config2_U0 |pooling2d_cl_array_array_ap_fixed_1u_config2_s | 0| 0| 764| 1313| 0|

|relu_array_array_ap_ufixed_64u_relu_config11_U0 |relu_array_array_ap_ufixed_64u_relu_config11_s | 0| 0| 3| 2776| 0|

|relu_array_array_ap_ufixed_64u_relu_config15_U0 |relu_array_array_ap_ufixed_64u_relu_config15_s | 0| 0| 3| 2776| 0|

|relu_array_array_ap_ufixed_64u_relu_config19_U0 |relu_array_array_ap_ufixed_64u_relu_config19_s | 0| 0| 3| 2776| 0|

|relu_array_array_ap_ufixed_64u_relu_config7_U0 |relu_array_array_ap_ufixed_64u_relu_config7_s | 0| 0| 3| 2776| 0|

+-------------------------------------------------------+------------------------------------------------------+---------+-------+-------+-------+-----+

|Total | | 388| 1162| 87077| 150099| 0|

+-------------------------------------------------------+------------------------------------------------------+---------+-------+-------+-------+-----+

​

​

​

9 of 20

MLP Model Results on Alveo FPGA (5ns clock)

MLP Quantized (12-bit) Synthesis Report

  • Overall Latency: 72.34us
  • Resource Utilization
    • BRAM: 7%
    • DSP: 9%
    • FF: 2%
    • LUT: 10%

​

  • Parameters: 71k
  • Precision: 12 bits
  • Reuse: 16
  • Average Pooling issue: 72us latency (of total 72.3us)

+---------------------+-------------------+------------------+-------------------+----------------+----------------+-----------------+

| Name | LUT | LUTAsMem | REG | BRAM | URAM | DSP |

+---------------------+-------------------+------------------+-------------------+----------------+----------------+-----------------+

| Platform | 142021 [ 8.22%] | 12528 [ 1.58%] | 217277 [ 6.29%] | 227 [ 8.44%] | 0 [ 0.00%] | 7 [ 0.06%] |

| User Budget | 1585019 [100.00%] | 778032 [100.00%] | 3236803 [100.00%] | 2461 [100.00%] | 1280 [100.00%] | 12281 [100.00%] |

| Used Resources | 81598 [ 5.15%] | 6240 [ 0.80%] | 81488 [ 2.52%] | 113 [ 4.59%] | 0 [ 0.00%] | 1110 [ 9.04%] |

| Unused Resources | 1503421 [ 94.85%] | 771792 [ 99.20%] | 3155315 [ 97.48%] | 2348 [ 95.41%] | 1280 [100.00%] | 11171 [ 90.96%] |

| krnl_rtl | 81598 [ 5.15%] | 6240 [ 0.80%] | 81488 [ 2.52%] | 113 [ 4.59%] | 0 [ 0.00%] | 1110 [ 9.04%] |

| krnl_rtl_1 | 81598 [ 5.15%] | 6240 [ 0.80%] | 81488 [ 2.52%] | 113 [ 4.59%] | 0 [ 0.00%] | 1110 [ 9.04%] |

+---------------------+-------------------+------------------+-------------------+----------------+----------------+-----------------+

​

  • MSE: 0.0121738293632542
  • Classified 65536 samples in 17.442984 seconds (3757.798165137614 inferences / s)
  • Or 266.1132812572639 us / inference

​

+---------------------+-------------------+------------------+-------------------+----------------+----------------+-----------------+

| Name | LUT | LUTAsMem | REG | BRAM | URAM | DSP |

+---------------------+-------------------+------------------+-------------------+----------------+----------------+-----------------+

| Platform | 142024 [ 8.22%] | 12528 [ 1.58%] | 217267 [ 6.29%] | 227 [ 8.44%] | 0 [ 0.00%] | 7 [ 0.06%] |

| User Budget | 1585016 [100.00%] | 778032 [100.00%] | 3236813 [100.00%] | 2461 [100.00%] | 1280 [100.00%] | 12281 [100.00%] |

| Used Resources | 56566 [ 3.57%] | 192 [ 0.02%] | 66830 [ 2.06%] | 105 [ 4.27%] | 0 [ 0.00%] | 1110 [ 9.04%] |

| Unused Resources | 1528450 [ 96.43%] | 777840 [ 99.98%] | 3169983 [ 97.94%] | 2356 [ 95.73%] | 1280 [100.00%] | 11171 [ 90.96%] |

| krnl_rtl | 56566 [ 3.57%] | 192 [ 0.02%] | 66830 [ 2.06%] | 105 [ 4.27%] | 0 [ 0.00%] | 1110 [ 9.04%] |

| krnl_rtl_1 | 56566 [ 3.57%] | 192 [ 0.02%] | 66830 [ 2.06%] | 105 [ 4.27%] | 0 [ 0.00%] | 1110 [ 9.04%] |

+---------------------+-------------------+------------------+-------------------+----------------+----------------+-----------------+

​

MLP Quantized (12-bit) on FPGA

Final Logic Synthesis Routing Report

9

10 of 20

Average Pooling Issue

  • Using io_stream model implementation
  • io_parallel synthesis failed
  • 72us report latency due to average pooling
    • 120 * 120 input diffraction image * 5ns clock = 72us

10

11 of 20

Multiple Input Stream & Unroll

  • Read input split across 30, 15, 10, 6, 5, 3, 2, or 1 stream
  • Read sample from each stream, shift line buffer, compute output pixel if possible, write to all output streams in parallel
  • Merge to 1 stream for succeeding dense layer

e.g.

nnet_pooling_stream.h

​

// Input: 14400

// Streams: 30

// Pixels/Input Stream: 480

// Unroll Factor: 15

// Pixels/Output Stream: 30

​

for(i < 480)

//kernel array variables, partitioning, etc.

for(j < 30)

#pragma HLS PIPELINE II=1

#pragma HLS unroll factor=15

//read pixel from stream_j

//shift line_buffer_j

//compute if possible kernel_data_j[16]

//write result to result_stream_j

​

​

Results (All correct outputs)

Streams: 15

Unroll Factor: 3

Synth Report: 24.34 us

Alveo: 272.4 us

Pipelining: Yes

Axi Input Opt: No

Streams: 15

Unroll Factor: 5

Synth Report: 14.74 us

Alveo: 252.94 us

Pipelining: Yes

Axi Input Opt: No

Streams: 15

Unroll Factor: 1 (no unroll)

Synth Report: 72.34 us

Alveo: 301.98 us

Pipelining: Yes

Axi Input Opt: No

Streams: 15

Unroll Factor: 15 (full)

Synth Report: 149.2 us

Alveo: 331.01 us

Pipelining: No (due to full loop unroll)

Axi Input Opt: No

Streams: 15

Unroll Factor: 10

Synth Report: 174.4 us

Alveo: 502.18 us

Pipelining: Yes

Axi Input Opt: No

Streams: 15

Unroll Factor: 14

Synth Report: 233.8 us

Alveo: 701.6

Pipelining: Yes

Axi Input Opt: No

Streams: 30

Unroll Factor: 30 (full)

Synth Report: 219.9 us

Alveo: 384.2 us

Pipelining: No (due to full loop unroll)

Axi Input Opt: No

Streams: 30

Unroll Factor: 15

Synth Report: 5.15 us

Alveo: 257.6 us

Pipelining: Yes

Axi Input Opt: No

io_stream Pooling Parallelization Method 1

11

12 of 20

Streams: 30

Unroll Factor: 1 (no unroll)

Synth Report: 72.35 us

Alveo: 307.9 us

Pipelining: Yes

Axi Input Opt: No

Streams: 30

Unroll Factor: 29

Synth Report: 176.2 us

Alveo: 257.0 us

Pipelining: Yes

Axi Input Opt: No

Streams: 30

Unroll Factor: 15

Synth Report: 5.15 us

Alveo: 203.4 us

Pipelining: Yes

Axi Input Opt: Yes

Streams: 15

Unroll Factor: 5

Synth Report: 14.74 us

Alveo: 228.4 us

Pipelining: Yes

Axi Input Opt: Yes

io_stream Pooling Parallelization Method 1

Results (All correct outputs) cont.

12

13 of 20

Multiple Input Stream & Multiple Parallel Poolings

  • Read input split across 30, 15, 10, 6, 5, 3, 2, or 1 (N) streams
  • Instantiate N pooling2d_cl layers
  • Use id template class parameter or dataflow to execute in parallel
  • Merge to 1 stream for succeeding dense layer

Streams: 30

Synth Report: 4.84 us

Alveo: 200.17 us

Id: No

Dataflow: Yes

Axi Input Opt: Yes

Comments: unrolled input stream writes, pipelined

​

Streams: 15

Synth Report: 15.66 us

Alveo: 263.6 us

Id: Yes

Dataflow: No

Axi Input Opt: No

Comments: Id not considered for rpt?

​

​

​

Streams: 15

Synth Report: 5.14 us

Alveo: 208.6 us

Id: No

Dataflow: Yes

Axi Input Opt: Yes

Comments: unrolled input stream writes, pipelined

​

Results (All correct outputs)

Streams: 15

Synth Report: 5.14 us

Alveo: 263.7 us

Id: No

Dataflow: Yes

Axi Input Opt: No

​

Streams: 15

Synth Report: 5.14 us

Alveo: 262.18 us

Id: Yes

Dataflow: Yes

Axi Input Opt: No

Streams: 15

Synth Report: 15.66 us

Alveo: 297.1 us

Id: No

Dataflow: No

Axi Input Opt: No

e.g.

myproject.cpp

​

// Input: 14400

// Streams: 15

// Pixels/Input Stream: 960

// Pixels/Output Stream: 60

​

nnet::pooling2d_cl<1, input_t, layer2_t, config2>(input_1_1, layer2_part1);

nnet::pooling2d_cl<2, input_t, layer2_t, config2>(input_1_2, layer2_part2);

nnet::pooling2d_cl<3, input_t, layer2_t, config2>(input_1_3, layer2_part3);

...

nnet::pooling2d_cl<13, input_t, layer2_t, config2>(input_1_13, layer2_part13);

nnet::pooling2d_cl<14, input_t, layer2_t, config2>(input_1_14, layer2_part14);

nnet::pooling2d_cl<15, input_t, layer2_t, config2>(input_1_15, layer2_part15);

​

​

​

​

​

​

io_stream Pooling Parallelization Method 2

Idea for Method Credit: Dr. Phil Harris

13

14 of 20

Read to 2D Array & Unroll

  • Read input split across data subarrays (size 30, 15, 10, 6, 5, 3, 2, or 1)
  • Unrolled computation as much as possible (vivado hls excessive runtime errors)
  • Merge to 1 stream for succeeding dense layer

e.g.

nnet_pooling_stream.h

​

// Input: 14400

// Data Subarrays: 30

// Pixels/Subarray: 480

// Unroll Factor: 15

// Pixels/Output Stream: 30

​

for( i < 30)

#pragma HLS unroll factor=30

for(j < 30)

#pragma HLS PIPELINE II=1

res[i][j][0] =

(data[i][0+(j*4)][0] + data[i][1+(j*4)][0] + data[i][2+(j*4)][0] + data[i][3+(j*4)][0] + � data[i][120+(j*4)][0] + data[i][121+(j*4)][0] + data[i][122+(j*4)][0] + data[i][123+(j*4)][0] +

data[i][240+(j*4)][0] + data[i][241+(j*4)][0] + data[i][242+(j*4)][0] + data[i][243+(j*4)][0] +

data[i][360+(j*4)][0] + data[i][361+(j*4)][0] + data[i][362+(j*4)][0] + data[i][363+(j*4)][0])/16;

​

​

​

Results (All correct outputs)

Subarrays: 30

Synth Report: 49.80 us

Alveo: 251.9 us

Outer Unroll Factor: 1 (no unroll)

Outer Pipeline: No

Inner Unroll Factor: 1 (no unroll)

Inner Pipeline: No

Axi Input Opt: No

Subarrays: 30

Synth Report: 40.96 us

Alveo: 259.2 us

Outer Unroll Factor: 30

Outer Pipeline: No

Inner Unroll Factor: 1 (no unroll)

Inner Pipeline: Yes

Axi Input Opt: No

Subarrays: 30

Synth Report: 6.03 us

Alveo: 322.2 us

Outer Unroll Factor: 15

Outer Pipeline: Yes

Inner Unroll Factor: 1 (no unroll)

Inner Pipeline: Yes (due to outer pipeline)

Axi Input Opt: No

Subarrays: 30

Synth Report: 6.03 us

Alveo: 290.9 us

Outer Unroll Factor: 30

Outer Pipeline: Yes

Inner Unroll Factor: 30

Inner Pipeline: Yes (due to outer pipeline)

Axi Input Opt: No

Subarrays: 30

Synth Report: 54.01 us

Alveo: 269.6 us

Outer Unroll Factor: 30

Outer Pipeline: No

Inner Unroll Factor: 1 (no unroll)

Inner Pipeline: No

Axi Input Opt: No

Subarrays: 30

Synth Report: 40.81 us

Alveo: 264.7 us

Outer Unroll Factor: 1 (no unroll)

Outer Pipeline: No

Inner Unroll Factor: 30

Inner Pipeline: No

Axi Input Opt: No

​

Subarrays: 30

Synth Report: 5.73 us

Alveo: 265.14 us

Outer Unroll Factor: 15

Outer Pipeline: Yes

Inner Unroll Factor: 1 (no unroll)

Inner Pipeline: Yes (due to outer pipeline)

Axi Input Opt: No

Comments: Moved array -> stream write outside pooling

io_stream Pooling Parallelization Method 3

14

15 of 20

io_stream Pooling Parallelization Summary

  • Method 1 Best: 203.4us (5.15us report)
    • Scalable for n streams

​

  • Method 2 Best: 200.17us (4.84us report)
    • Scalable for n streams
    • Easier to add feature to hls4ml than method 1
    • Largest latency improvement

​

  • Method 3 Best: 251.9us (5.73us alt. report)
    • Lowest resource consumption

15

16 of 20

Test 1

- All layers removed (all

computation eliminated)

- 14400 pixels read into input_1

- 6 element output array struct

written to output

​

Alveo Latency: 187.0 us

​

​

myproject_axi.cpp

for(unsigned i = 0; i < 14400; ++i)

ctype = in[i].data;

in_local.write(ctype);

​

​

myproject.cpp

nnet::array<ap_fixed<16,6>, 6> result;

nnet::array<ap_fixed<16,6>, 1> pixel;

for(unsigned i=0; i < 14400; i++)

pixel = input_1.read();

if(i<6)

result[i] = pixel[0];

​

output.write(result);

​

​

​

Test 2

- All layers removed (all

computation eliminated)

- 6 pixels read into input_1

- 6 element output array struct

written to output

​

Alveo Latency: 42.3 us

​

​

myproject_axi.cpp

for(unsigned i = 0; i < 6; ++i)

ctype = in[i].data;

in_local.write(ctype);

​

​

myproject.cpp

nnet::array<ap_fixed<16,6>, 6> result;

nnet::array<ap_fixed<16,6>, 1> pixel;

for(unsigned i=0; i < 6; i++)

pixel = input_1.read();

result[i] = pixel[0];

​

output.write(result);

​

​

Test 1

  • All 14400 elements written and read from input stream
  • No computation performed to isolate latency due to streaming and overhead

​

Test 2

  • Only 6 elements written and read from input stream and no computation performed to isolate latency due to overhead

​

Latency Breakdown

  • Full model Alveo Latency (parallelized): 200.17us
  • Full model Alveo Latency (not parallelized): 266.1us
  • Theoretical Latency due to stream write and read = 120 * 120 * 2 * 5ns = 144us
  • Experimental latency due to stream write and read = 187us - 42.3us = 144.7us
  • Latency due to overhead/IO = 42.3us
  • Theoretical model computation latency: 200.17us - (120 * 120 * 2 * 5ns + 42.3us) = 13.87us
  • Experimental model computation latency (parallelized): 200.17us - 187us = 13.17us
    • Synthesis report latency: 4.84us
  • Experimental model computation latency (not parallelized): 266.1us - 187us = 79.1us
    • Synthesis report latency: 72.3us

io_stream Timing Tests (5ns Target Clock)

​

​

16

17 of 20

Target Clock 2ns (3.33ns/300MHz achieved on 71k param model)

  • Maintained 42.3us baseline Alveo latency due to IO/overhead (no streaming, no computation)
  • Overall Alveo Latency: 141.21us
  • Latency due to stream write and read = 120 * 120 * 2 * 3.33ns = 96us
  • Model computation latency: 141.21us - (120 * 120 * 2 * 3.33ns + 42.3us) = 2.996us
  • Synthesis Report: 2.810us

Standard clocks provide by platform

- ap_clk (300MHz default for U250, 400MHz

through mixed-mode clock manager PLL

- ap_clk_2 (500MHz default for U250)

https://china.xilinx.com/content/dam/xilinx/publications/presentations/c_04_rtl_kernel_Vitis_Tutorial_webinar.pdf

Target Clock 2ns (2.785ns/359MHz achieved on 60k param model)

  • Set kernel data clock to 400MHz (359MHz achieved) through MMCM
  • Maintained 42.3us baseline Alveo latency due to IO/overhead (no streaming, no computation)
  • Overall Alveo Latency: 125.2us
  • Latency due to stream write and read = 120 * 120 * 2 * 2.785ns = 80.2us
  • Model computation latency: 125.2us - (120 * 120 * 2 * 2.785ns + 42.3us) = 2.692us
  • Synthesis Report: 2.708us

io_stream Timing Tests w/ Scalable Clock Changed

Note: Target clock was set to 3.33ns and 2.785ns (instead of 2ns)

to obtain accurate synthesis report latencies for the final achieved clock speeds

17

18 of 20

io_stream Timing Tests

​

266.1us

200.17us

141.21us

125.2us

18

19 of 20

Final Resource Usage Parallelized

+---------------------+-------------------+------------------+-------------------+----------------+----------------+-----------------+

| Name | LUT | LUTAsMem | REG | BRAM | URAM | DSP |

+---------------------+-------------------+------------------+-------------------+----------------+----------------+-----------------+

| Platform | 142024 [ 8.22%] | 12528 [ 1.58%] | 217316 [ 6.29%] | 227 [ 8.44%] | 0 [ 0.00%] | 7 [ 0.06%] |

| User Budget | 1585016 [100.00%] | 778032 [100.00%] | 3236764 [100.00%] | 2461 [100.00%] | 1280 [100.00%] | 12281 [100.00%] |

| Used Resources | 107711 [ 6.80%] | 6245 [ 0.80%] | 175747 [ 5.43%] | 130 [ 5.28%] | 0 [ 0.00%] | 1161 [ 9.45%] |

| Unused Resources | 1477305 [ 93.20%] | 771787 [ 99.20%] | 3061017 [ 94.57%] | 2331 [ 94.72%] | 1280 [100.00%] | 11120 [ 90.55%] |

| krnl_rtl | 107711 [ 6.80%] | 6245 [ 0.80%] | 175747 [ 5.43%] | 130 [ 5.28%] | 0 [ 0.00%] | 1161 [ 9.45%] |

| krnl_rtl_1 | 107711 [ 6.80%] | 6245 [ 0.80%] | 175747 [ 5.43%] | 130 [ 5.28%] | 0 [ 0.00%] | 1161 [ 9.45%] |

+---------------------+-------------------+------------------+-------------------+----------------+----------------+-----------------+

In Summary

​

​

  • 71k param MLP average pooling distilled model from original pytorch encoder model
  • 100us latency goal (200us max)
  • FPGA Overall Latency (71k params): 141.21us @ 3.33ns clk (200.1us from 266.1us at 5ns clk)
    • FPGA Computation Latency: 2.996us
    • Synthesis Report: 2.810us (from 72.3us)
  • FPGA Overall Latency (60k params): 125.2us @ 2ns clk (200.1us from 265.1us at 5ns clk)
    • FPGA Computation Latency: 2.692us
    • Synthesis Report: 2.708us (from 72.3us)
  • Low resource consumption

MLP Quantized (12-bit) on FPGA

PyTorch Encoder (32-bit) on GPU

Future Work

  • Finish creating small distilled HLS model from Shuyu’s PyTorch model for simulation data
  • Continue testing with improved 71k param distilled model
  • Concatenate 32-bit inputs to 512-bit inputs and utilize multiple SLRs
  • Add pooling parallelization option to hls4ml, specified in config likely as “reuse” factor

The Team: Dr. Nhan Tran, Dr. Javier Duarte, Dr. Josh Agar, Shuyu Qin, Jules Muhizi, Javier Campos

Final Resource Usage No Parallelization

+---------------------+-------------------+------------------+-------------------+----------------+----------------+-----------------+

| Name | LUT | LUTAsMem | REG | BRAM | URAM | DSP |

+---------------------+-------------------+------------------+-------------------+----------------+----------------+-----------------+

| Platform | 142024 [ 8.22%] | 12528 [ 1.58%] | 217267 [ 6.29%] | 227 [ 8.44%] | 0 [ 0.00%] | 7 [ 0.06%] |

| User Budget | 1585016 [100.00%] | 778032 [100.00%] | 3236813 [100.00%] | 2461 [100.00%] | 1280 [100.00%] | 12281 [100.00%] |

| Used Resources | 56566 [ 3.57%] | 192 [ 0.02%] | 66830 [ 2.06%] | 105 [ 4.27%] | 0 [ 0.00%] | 1110 [ 9.04%] |

| Unused Resources | 1528450 [ 96.43%] | 777840 [ 99.98%] | 3169983 [ 97.94%] | 2356 [ 95.73%] | 1280 [100.00%] | 11171 [ 90.96%] |

| krnl_rtl | 56566 [ 3.57%] | 192 [ 0.02%] | 66830 [ 2.06%] | 105 [ 4.27%] | 0 [ 0.00%] | 1110 [ 9.04%] |

| krnl_rtl_1 | 56566 [ 3.57%] | 192 [ 0.02%] | 66830 [ 2.06%] | 105 [ 4.27%] | 0 [ 0.00%] | 1110 [ 9.04%] |

+---------------------+-------------------+------------------+-------------------+----------------+----------------+-----------------+

19

20 of 20

Simulated Data Model (Rotations)

20

PyTorch Encoder

173k Param Model

Model: "model"

_________________________________________________________________

Layer (type) Output Shape Param #

=================================================================

input_1 (InputLayer) [(None, 200, 200, 1)] 0

average_pooling2d (AverageP (None, 50, 50, 1) 0

ooling2D)

flatten (Flatten) (None, 2500) 0

q_dense (QDense) (None, 64) 160064

batch_normalization (BatchN (None, 64) 256

ormalization)

q_activation (QActivation) (None, 64) 0

q_dense_1 (QDense) (None, 64) 4160

batch_normalization_1 (Batc (None, 64) 256

hNormalization)

q_activation_1 (QActivation (None, 64) 0

)

q_dense_2 (QDense) (None, 64) 4160

batch_normalization_2 (Batc (None, 64) 256

hNormalization)

q_activation_2 (QActivation (None, 64) 0

)

q_dense_3 (QDense) (None, 64) 4160

batch_normalization_3 (Batc (None, 64) 256

hNormalization)

q_activation_3 (QActivation (None, 64) 0

)

q_dense_4 (QDense) (None, 6) 390

=================================================================

Total params: 173,958

Trainable params: 173,446

Non-trainable params: 512

_________________________________________________________________

​

  • Learn affine transformations on generated simulated and real data