1 of 43

A Uniform Latency Model for DNN Accelerators�with Diverse Architectures and Dataflows

Linyan Mei∗†, Huichu Liu∗, Tony Wu∗, H. Ekin Sumbul∗,

Marian Verhelst†, Edith Beigne∗

∗Meta Reality Labs, †MICAS-ESAT, KU Leuven

January 2022

2 of 43

Outline

Fast Latency Estimation for DNN Accelerator
A Uniform Intra-Layer Analytical Latency Model
Model Validation
Three Case Studies
Conclusion and Future Work

2

3 of 43

Outline

Fast Latency Estimation for DNN Accelerator

Necessity + Feasibility + Challenge

A Uniform Intra-Layer Analytical Latency Model
Model Validation
Three Case Studies
Conclusion and Future Work

3

4 of 43

Fast Latency Estimation - Necessity

4

HW Architect: Design high performance HW

Compiler Designer: Map each DNN layer to HW efficiently

Algorithm Developer: Develop HW-friendly DNNs

HUGE architecture design space�(E.g., PE array size, interconnection, memory hierarchy)
Traditional latency estimation methods: time-consuming, not flexible�(E.g., RTL simulation, cycle-accurate simulator)

HUGE mapping design space�(E.g., Loop tiling, loop ordering, loop unrolling)
Mapping impact

DNN layer type/shape/size impact

Can we estimate the latency of a given (NN layer, HW arch, mapping) quickly?

Yes, with analytical model!

For HW architects who would like to design high performance accelerator, it is not an easy task to pinpoint one best architecture given the huge architecture design space exists. In the early design phase, it would be ideal to be able to estimate the HW performance so as to make wise design choice. However, the traditional latency estimation methods are time-consuming and not flexible.

For Compiler designers who want to map each DNN layer to HW efficiently, it is also challenging due to a great number of mapping possibilities, and different mappings can lead to distinctive HW performance.

For Algorithm developers who aim at developing HW-friendly neural network, it is necessary to gain insight of how different algorithm parameters such like DNN layer type and size impact HW performance.

For all of them, a fast latency estimation model is very much helpful and necessary.

So can we estimate the latency quickly?

The answer is: yes, with an analytical model!

5 of 43

Analytical Latency Modeling - Feasibility

5

HW: Structured

Algorithm: Regular

Mapping: Deterministic

6 of 43

Analytical Latency Modeling – Challenge NO. 1

Guarantee Generality – “One for All”

6

One model that suits all different (NN layer, HW arch, mapping) combinations

NN layer

HW arch

Mapping

7 of 43

Analytical Latency Modeling – Challenge NO. 2

Capture Concurrency – “All in One”

7

Spatially underutilized

Temporally underutilized

PE array

Spatial mapping

❌Mismatch

❌ Insufficient memory BW and port

❌ Suboptimal temporal mapping

Timeline

…

HW components work concurrently�(parallel / series, independent / interference)

Spatial

underutilization

Easy ~

Temporal stall

Hard !

The second challenge is to capture the concurrency of different HW components.

In the real scenarios, PE array often isn’t 100% utilized due to different non-idealities.

When a PE array is spatially under-utilized, it means that certain PEs are always idle. It usually comes from the mismatch between the physical array dimension and the spatial mapping.

When a PE array is temporally under-utilized, it means that at certain moment along the timeline, there are some stalling cycle in which the array is waiting for data to come or to go. This is often because of the Insufficient memory BW and suboptimal temporal mapping.

Estimating the spatial underutilization is easy since we can calculate directly from spatial mapping the active PE count. While estimating the temporal stall is hard, due to that HW components usually work concurrently and can interfere with each other.

8 of 43

Analytical Latency Modeling – Challenges and SotAs

Guarantee Generality – “One for All”

8

State of the Arts	Guarantee Generality	Capture Concurrency
[MAESTRO]	No, with a pre-defined HW template
[dMazeRunner]	No, with a pre-defined HW template
[Interstellar]		No, only consider spatial underutilization
[Timeloop]		No, only consider spatial underutilization

❌ Insufficient mem BW and port

❌ Suboptimal temporal mapping

Capture Concurrency – “All in One”

One model that suits all different (NN layer, HW arch, mapping) combinations

9 of 43

Outline

Fast Latency Estimation for DNN Accelerator
A Uniform Intra-Layer Analytical Latency Model

“One-for-All”: A Uniform Dataflow Representation
“All-in-One”: Divide-Combine-Integrate 3-Step Method

Model Validation
Three Case Studies
Conclusion and Future Work

9

10 of 43

A Uniform Dataflow Representation – Prior Art

Loop Relevancy Principle

10

An uneven mapping example

r loop 🡪 data size

ir loop 🡪 data reuse

pr loop 🡪 r/ir loops

L. Mei, P. Houshmand, V. Jain, S. Giraldo and M. Verhelst, "ZigZag: Enlarging Joint Architecture-Mapping Design Space Exploration for DNN Accelerators," in IEEE TC, 2021

Uneven Mapping Format

W-Reg

I-Reg

O-Reg

Local Buffer (W/I/O)

The generality of the model is guaranteed by the uniform dataflow representation inherited from our previous work, ZigZag. There are two important concepts used in this talk.

One is the loop relevancy principle. Based on whether a loop dimension is related to an operand data dimension, we divide all 7 loop dimensions into r, ir, and pr loops for each operand. R loops are relevant loops, which define operand’s data size. ir loops are irrelevant loops, which define operand’s data reuse, pr loops can be treated as partially r loop and partially ir loop.

Another is the uneven mapping format, in which we can define for each operand at each architectural level freely what are the spatial or temporal loops we want to unroll.

With these two basic concepts in mind, let’s have a look of the model overview.

11 of 43

Latency Model Overview

Input to the model (NN layer, HW arch, mapping):

11

Weight Input Output

for B in [0, 4) GB GB GB

for K’ in [0, 2) GB LB GB

for OY in [0, 2) LB LB LB

for C’ in [0, 2) LB LB LB

for OX in [0, 2) W-Reg LB LB

for K in [0, 3) W-Reg I-Reg O-Reg

for C in [0,2) W-Reg I-Reg O-Reg

O[B][3K’+K][OY][OX] += W[K][2C’+C] × I[B][2C’+C][OY][OX]

Output of the model:

Total latency + latency breakdown
Each HW component’s working state
System latency bottleneck

for B in [0, 4)

for K in [0, 6)

for C in [0, 4)

for OX in [0, 2)

for OY in [0, 2)

O[B][K][OY][OX] +=

W[K][C] × I[B][C][OY][OX]

W-Reg

I-Reg

O-Reg

Local Buffer (W/I/O)

Global Buffer (W/I/O)

12 of 43

Modelling Methodology – The 3 Steps

12

Step 1: “Divide”

Step 2: “Combine”

Step 3: “Integrate”

🡪 Temporal stall

13 of 43

Step 1: Divide

Divide into Unit Mems & Identify DTLs & Compute key attributes 🡪 next slides

13

Weight Input Output

for B in [0, 4) GB GB GB

for K’ in [0, 2) GB LB GB

for OY in [0, 2) LB LB LB

for C’ in [0, 2) LB LB LB

for OX in [0, 2) W-Reg LB LB

for K in [0, 3) W-Reg I-Reg O-Reg

for C in [0,2) W-Reg I-Reg O-Reg

O[B][3K’+K][OY][OX] += W[K][2C’+C] × I[B][2C’+C][OY][OX]

W-Reg

I-Reg

O-Reg

Local Buffer (W/I/O)

Global Buffer (W/I/O)

Unit Mem: Memory that only holds a single operand (W/I/O).

DTL: Data transfer link between Unit Mem across memory levels (read/write).

Key attributes: Mem_DATA , Mem_CC , ReqBW_u , MUW_u , SS_u.

Mem7

Mem8

Mem9

Mem1

Mem2

Mem3

Mem4

Mem5

Mem6

14 of 43

Step 1: Divide - Compute key attributes 1

Attributes of Unit Mem

Mem_DATA : Data size held in the Unit Mem

The product of all the r loops' size (temporal & spatial) at current and lower memory levels.

Mem_CC : Updating period of Unit Mem

The product of all the temporal loop sizes at current and lower memory levels.

14

Weight Input Output

for B in [0, 4) GB GB GB

for K’ in [0, 2) GB LB GB

for OY in [0, 2) LB LB LB

for C’ in [0, 2) LB LB LB

for OX in [0, 2) W-Reg LB LB

for K in [0, 3) W-Reg I-Reg O-Reg

for C in [0,2) W-Reg I-Reg O-Reg

O[B][3K’+K][OY][OX] += W[K][2C’+C] × I[B][2C’+C][OY][OX]

Mem7

Mem8

Mem9

Mem1

Mem2

Mem3

Mem4

Mem5

Mem6

	B	K	C	OY	OX
W	ir	r	r	ir	ir
I	r	ir	r	r	r
O	r	r	ir	r	r

r

ir

r

ir

r

ir

Mem_DATA = 6

Mem_CC = 12

Mem_DATA = 12

Mem_CC = 48

Mem_DATA = 24

Mem_CC 384

15 of 43

Step 1: Divide - Compute key attributes 2

15

Attributes of Data Transfer Link (DTL)

ReqBW_u : Required BW to prevent computation stall

DTL double-buffered memory level

ReqBW_u = Mem_DATA / Mem_CC

DTL non-double-buffered memory level

ReqBW_u = (Mem_DATA / Mem_CC) × top-ir loop size

E.g.

Weight

If Mem 7 is double-buffered :

ReqBW_u = 6 data / 12 cycle = 0.5 data / cycle

If Mem 7 is non-double-buffered :

ReqBW_u = 6 data / 12 cycle × 2 = 1 data / cycle

(* spatial unrolling not considered)

Buffer A

Buffer B

Buffer A

r

ir

r

ir

Buffer A

Buffer B

Buffer A

W/I

O

ReqBW_u

Then, using the attributes of unit Mem, we compute the attributes of DTL.

The first attributes is the required memory BW of Unit Mem. If the DTL serves double-buffered memory level, the required memory BW equals to the Mem-DATA divided by the Mem-CC.

If the DTL serves non-double-buffered memory level, the required memory BW scales up by a factor of the top-ir loop size. It’s because in the non-double buffered case, each data element cannot be overwritten until it is fully reused, and the top ir loop exactly measures the top-level data reuse at that memory level.

Let’s have a look of the same weight mapping example. If Mem7 is double-buffered, the required BW is 0.5 data per cycle, while if Mem 7 is non-double buffered, the required BW becomes 1 data per cycle. For simplicity, spatial unrolling is not considered.

16 of 43

Step 1: Divide - Compute key attributes 3

16

Attributes of Data Transfer Link (DTL)

MUW_u : Total allowed memory updating time window
SS_u : Total Stall (+) or Slack (-)

Non-double-buffered

memory level

Double-buffered

memory level

“M” : Memory data updating

“C” : data Consuming

“n” in Mn/Cn : identify data producer-consumer pair, e.g., M1 serves C1

There are another two important DTL’s attributes. One is the MUW, the total allowed memory updating window, another one is SSu, the total stall and slack on a DTL.

For DTLs that serve double-buffered memory levels, there are three types of memory access patterns.

In the first case, the memory updating duration and the data consuming duration exactly match, thus there is no stall or slack, SSu equals 0. In the second case, the memory updating duration is smaller than the data consuming duration, thus there are slacks, SSu is negative. In the third case, the memory updating duration is longer than the data consuming duration, there are stalls, SSu is positive.

For DTLs that serve non-double-buffered memory levels, there are also three cases. The only difference is that a mem update keep-out zone is inserted due to the data overwriting issue we have discussed

Now, let’s use these figures to illustrate several attributes. Firstly, the Mem_CC is the length of the computation period.

17 of 43

Step 1: Divide - Compute key attributes 3

17

Attributes of Data Transfer Link (DTL)

MUW_u : Total allowed memory updating time window
SS_u : Total Stall (+) or Slack (-)

Non-double-buffered

memory level

Double-buffered

memory level

18 of 43

Step 1: Divide - Compute key attributes 3

18

Attributes of Data Transfer Link (DTL)

MUW_u : Total allowed memory updating time window
SS_u : Total Stall (+) or Slack (-)

Non-double-buffered

memory level

Z = 1

Z = 2

Z = 3

Z = 4

= X_REQ × Z

Double-buffered

memory level

19 of 43

Step 1: Divide - Compute key attributes 3

19

Attributes of Data Transfer Link (DTL)

MUW_u : Total allowed memory updating time window
SS_u : Total Stall (+) or Slack (-)

Double-buffered

memory level

Non-double-buffered

memory level

Z = 1

Z = 2

Z = 3

Z = 4

Each DTL:

For every Mem_CC ,

can be busy for X_REQ ,

actually is busy for X_REAL.

In total goes through Z Mem_CC.

= X_REQ × Z

= (X_REAL−X_REQ) × Z

20 of 43

Step 2: Combine

Consider memory port sharing, and combine unit attributes

20

Combine ReqBW_u @same port: ReqBW_comb(1,6) = ReqBW_u① + ReqBW_u⑥
Combine MUW_u @same port: MUW_comb(1,6) = MUW_u① ∪ MUW_u⑥
Combine SS_u @same port: SS_comb(1,6) = Combine(SS_u①, SS_u⑥) 🡪 next slides
Combine SS_u @same served mem: SS_comb((1,6),(2,7)) = max(SS_comb(1,6) , SS_comb(₂_,7))

ReqBW_u

MUW_u

SS_u

Unit Mems DTLs

21 of 43

Step 2: Combine - SS_u @same port

Combine Unit Stall/Slack (SS_u) @same port

21

Mem 1

Mem 2

Mem 4

Mem n

Mem 3

…

Mem

DTL(1)

DTL(2)

DTL(3)

DTL(4)

DTL(n)

W

I

O

W

I

Assume�one physical memory

22 of 43

Step 2: Combine - SS_u @same port

Combine Unit Stall/Slack (SS_u) @same port

22

n DTLs share 1 memory port

Assume

Mem ★ only has 1 port

Each DTL:

For every Mem_CC ,

can be busy for X_REQ ,

actually is busy for X_REAL.

In total goes through Z Mem_CC.

MUW_u = X_REQ × Z

SS_u = (X_REAL−X_REQ) × Z

Mem ★

Mem 1

Mem 2

Mem 4

Mem n

Mem 3

…

Known: SS_u(1), SS_u(2) , … , SS_u(n) ;

MUW_u(1), MUW_u(2) , … , MUW_u(n)

Want to know:

SS_comb(1,2,…,n)

Value & Position

Value

DTL(1)

DTL(2)

DTL(3)

DTL(4)

DTL(n)

SS_u(1)<0

SS_u(2)>0

SS_u(3)<0

SS_u(4)=0

SS_u(n)>0

23 of 43

Step 2: Combine - SS_u @same port

Combine Unit Stall/Slack (SS_u) @same port

23

Actual working cycle within each MUW

Assume n DTLs share 1 memory port

Reorder DTLs based on SS_u(+/-).

Assume:

Sum of positive stalls

24 of 43

Step 2: Combine - SS_u @same port

Combine Unit Stall/Slack (SS_u) @same port

24

Actual working cycle within each MUW

Sum of positive stalls

25 of 43

Step 2: Combine - SS_u @same port E.g. 1

Combine Unit Stall/Slack (SS_u) @same port

25

Mem ★

Mem 1

Mem 2

DTL(1)

DTL(2)

1 port

DTL(1)

DTL(2)

For every Mem_CC ,

can be busy for X_REQ ,

actually is busy for X_REAL.

In total goes through Z Mem_CC.

MUW_u = X_REQ × Z

SS_u = (X_REAL−X_REQ) × Z

			X_REAL	Z	MUW_u	SS_u	MUW_comb	SS_comb
Legend
DTL(1)	8	3	3	4	12	0	24	12
DTL(2)	8	6	6	4	24	0	24	12

Timeline

(cycle)

0

32

Each DTL

8

2

5

Ideal data transfer waveform

(Ctrl+1) hide me

Let’s have a look of this example. Mem star talks to Mem1 and Mem2 with 1 memory port.

Each DTL’s ideal periodic data transfer waveform is depicted, and the key attributes are marked out in the figure and listed in the table. Following the equation of the 3^rd case, we can calculate out the SS_comb of this two DTLs to be12 cycles. Although any one of them doesn’t introduce stall as both of their SSu equal to 0, still together them will make stalls due to HW resource’s contention.

The next 5 slides are 5 extra examples. Due to time limit, I won’t discuss them in this talk but just play them. Audiences who are interested are encouraged to pause the video for further comprehension.

=================

I have prepared 6 examples to illustrate the SS_Comb equation under n=2 case, due to the time limit, I will only play them.

26 of 43

Step 2: Combine - SS_u @same port E.g. 2

Combine Unit Stall/Slack (SS_u) @same port

26

Mem ★

Mem 1

Mem 2

DTL(1)

DTL(2)

1 port

DTL(1)

DTL(2)

For every Mem_CC ,

can be busy for X_REQ ,

actually is busy for X_REAL.

In total goes through Z Mem_CC.

MUW_u = X_REQ × Z

SS_u = (X_REAL−X_REQ) × Z

			X_REAL	Z	MUW_u	SS_u	MUW_comb	SS_comb
Legend
DTL(1)	8	3	3	4	12	0	24	12
DTL(2)	4	3	3	8	24	0	24	12

Timeline

(cycle)

0

32

Each DTL

8

5

1

4

Ideal data transfer waveform

27 of 43

Step 2: Combine - SS_u @same port E.g. 3

Combine Unit Stall/Slack (SS_u) @same port

27

Mem ★

Mem 1

Mem 2

DTL(1)

DTL(2)

1 port

DTL(1)

DTL(2)

For every Mem_CC ,

can be busy for X_REQ ,

actually is busy for X_REAL.

In total goes through Z Mem_CC.

MUW_u = X_REQ × Z

SS_u = (X_REAL−X_REQ) × Z

			X_REAL	Z	MUW_u	SS_u	MUW_comb	SS_comb
Legend
DTL(1)	8	3	3	4	12	0	24	0
DTL(2)	8	6	3	4	24	-12	24	0

Timeline

(cycle)

0

32

Each DTL

8

5

2

Ideal data transfer waveform

28 of 43

Step 2: Combine - SS_u @same port E.g. 4

Combine Unit Stall/Slack (SS_u) @same port

28

Mem ★

Mem 1

Mem 2

DTL(1)

DTL(2)

1 port

DTL(1)

DTL(2)

For every Mem_CC ,

can be busy for X_REQ ,

actually is busy for X_REAL.

In total goes through Z Mem_CC.

MUW_u = X_REQ × Z

SS_u = (X_REAL−X_REQ) × Z

			X_REAL	Z	MUW_u	SS_u	MUW_comb	SS_comb
Legend
DTL(1)	8	3	5	4	12	8	24	8
DTL(2)	8	6	3	4	24	-12	24	8

Timeline

(cycle)

0

32

Each DTL

8

5

2

10

Ideal data transfer waveform

29 of 43

Step 2: Combine - SS_u @same port E.g. 5

Combine Unit Stall/Slack (SS_u) @same port

29

Mem ★

Mem 1

Mem 2

DTL(1)

DTL(2)

1 port

DTL(1)

DTL(2)

For every Mem_CC ,

can be busy for X_REQ ,

actually is busy for X_REAL.

In total goes through Z Mem_CC.

MUW_u = X_REQ × Z

SS_u = (X_REAL−X_REQ) × Z

			X_REAL	Z	MUW_u	SS_u	MUW_comb	SS_comb
Legend
DTL(1)	8	3	5	4	12	8	24	20
DTL(2)	8	6	6	4	24	0	24	20

Timeline

(cycle)

0

32

Each DTL

8

5

2

10

Ideal data transfer waveform

30 of 43

Step 2: Combine - SS_u @same port E.g. 6

Combine Unit Stall/Slack (SS_u) @same port

30

Mem ★

Mem 1

Mem 2

DTL(1)

DTL(2)

1 port

DTL(1)

DTL(2)

For every Mem_CC ,

can be busy for X_REQ ,

actually is busy for X_REAL.

In total goes through Z Mem_CC.

MUW_u = X_REQ × Z

SS_u = (X_REAL−X_REQ) × Z

			X_REAL	Z	MUW_u	SS_u	MUW_comb	SS_comb
Legend
DTL(1)	8	3	5	4	12	8	24	36
DTL(2)	8	6	10	4	24	16	24	36

Timeline

(cycle)

0

32

Each DTL

8

5

2

10

12

Ideal data transfer waveform

31 of 43

Step 3: Integrate

31

Consider actual design constrains, integrate combined attributes

memory operations �that can be overlapped �(stalls hided under the longest one)

memory operations

that cannot be overlapped

Parallel 🡪 Max

Series 🡪 Sum

32 of 43

Outline

Fast Latency Estimation for DNN Accelerator
A Uniform Intra-Layer Analytical Latency Model
Model Validation
Three Case Studies
Conclusion and Future Work

32

33 of 43

Model Validation

33

Limited Global Buffer BW
Evaluated NN layers across a large range
Average latency estimation accuracy: 94.3%

Against an in-house DNN accelerator

34 of 43

Outline

Fast Latency Estimation for DNN Accelerator
A Uniform Intra-Layer Analytical Latency Model
Model Validation
Three Case Studies

Mapping v.s. latency
Workload v.s. latency
HW architecture v.s. latency

Conclusion and Future Work

34

35 of 43

Case Study 1: Mapping v.s. Latency 1

35

Hardware

Workload

Layer dimensions:

B: 64, K: 160, C: 960

Precision:

W/I: 8 bits, O: 24 bits

Temporal Loops

W

I

O

B: Latency-optimized

Temporal Loops

W

I

O

Mapping A: Energy-optimized

Data reuse (ir loops)

Mapping A

Mapping B

I (Inputs)

O (Outputs & Partial Sums)

Less O-Reg/GB traffic

Less I-LB/GB traffic

Mem Access
Energy
Latency
BW Analysis

In case study one, we fixed the workload and the HW, as well as the spatial unrolling, checking how different temporal mappings can impact the overall latency, and what is the tradeoff between optimizing a mapping for energy and for latency.

Mapping A is an energy-optimized mapping, while mapping B is the latency-optimized one. The biggest difference between them is whether the C loop is split. To visualize better, I also marked out the memory levels and the data reuse loops for each operand.

By checking the data reuse factor, we see that for weight, there is not difference between mapping A and B, since the weight’s data reuse factor are all at local buffer level. So, let’s ignore it.

For Input and Output operands, there is a clear difference regarding data reuse loop allocation. For Input, in mapping A, all the data reuse loops are at local buffer level, while in mapping B, both the global buffer and the local buffer levels have certain input data reuse. For output, things are opposite, mapping A has output data reuse loop at both Global buffer and the Register level, while mapping B is fully-output stationary at register level.

Knowing their difference, let’s compare these two mappings in detail. Firstly, regarding memory access count, due to their different data reuse distribution, mapping A has less Input access from global buffer to local buffer; while mapping B has less Output traffic between register and global buffer.

36 of 43

Case Study 1: Mapping v.s. Latency 2

36

Workload

Layer dimensions:

B: 64, K: 160, C: 960

Precision:

W/I: 8 bits, O: 24 bits

Hardware

Temporal Loops

W

I

O

B: Latency-optimized

Temporal Loops

W

I

O

A: Energy-optimized

Data reuse (ir loops)

+5%

[pJ]

Total

Partial energy breakdown

Mem Access
Energy
Latency
BW Analysis

37 of 43

Case Study 1: Mapping v.s. Latency 3

37

Workload

Layer dimensions:

B: 64, K: 160, C: 960

Precision:

W/I: 8 bits, O: 24 bits

Hardware

Temporal Loops

W

I

O

B: Latency-optimized

Temporal Loops

W

I

O

A: Energy-optimized

Data reuse (ir loops)

Temporal stall [cycle]

Ideal computation [cycle]

MAC Array Utilization [%]

-30%

Mem Access
Energy
Latency
BW Analysis

38 of 43

Case Study 1: Mapping v.s. Latency 4

38

Workload

Layer dimensions:

B: 64, K: 160, C: 960

Precision:

W/I: 8 bits, O: 24 bits

Hardware

Temporal Loops

W

I

O

B: Latency-optimized

Temporal Loops

W

I

O

A: Energy-optimized

Data reuse (ir loops)

Mem Access
Energy
Latency
BW Analysis

Real_BW [bit/cc]

Req_BW (Mapping A) [bit/cc]

Req_BW (Mapping B) [bit/cc]

Req_BW largely exceeds Real_BW + Frequent GB access

induce large stall.

39 of 43

Case Study 1: Mapping v.s. Latency 5

Conclusion:

39

Matching required memory BW (mapping-dependent) with �real memory BW (HW-dependent);

Reducing the frequent memory access of the low-BW link.

A good latency model is critical to help DNN accelerator mapper to minimize system temporal stall:

40 of 43

Case Study 2: Workload v.s. Latency 1

40

Workload Pool

Layer dimensions:

B/K/C: {8,32,128,512}

Hardware

Mapping

Latency-optimized

Temporal stall can play important role in total latency.
Latency breakdown helps to analyse trend, identify the system bottleneck.
Latency estimation time is always short, doesn’t scale up with workload size (<1ms).

In case study two, we fixed the HW and always take the latency-optimal mapping, checking how different neural network layer size can impact latency.

We take B/K/C loop dimension from 8 to 512, to construct the workload pool. This figure visualizes different operands’ data size ratio, the total MAC operation. and the total data size, of different layers. We have ordered the layers such that from left to right the total MAC Operation count is increasing.

The corresponding latency behavior is shown in the below figure, in which the left y axis measures the latency breakdown ratio and the right y axis measures the absolute latency cycle.

This case study tells us, Temporal stall can play important role in total latency, as shown by the purple bars

Latency breakdown helps to analyse trend and identify system bottleneck.

Most importantly, the latency estimation time is always short since it is analytical based, and it doesn’t scale up with the workload size compared to the

41 of 43

Case Study 3: HW Architecture v.s. Latency

41

Hardware

Mapping

Latency-optimized

Workload

Layer dimensions:

B: 64, K: 160, C: 960

4,176 HW Arch

Area-latency trade-off sensitivity towards memory size is high @ low GB BW.

MAC array size preference changes for different memory BWs.

BW-awareness is important for HW parameter optimization on latency.

In case study three, we fixed the workload and always take the latency-optimal mapping, checking how different HW characteristics can impact latency.

In total we have constructed more than 4000 architectures, each can vary in PE array size, local buffer and register capacity, and the global buffer read/write BW.

In these 3 figures, the y axis measures the total latency, the x axis measures the area, and all of them share the same x and y axis scale. Different color identify different PE array size, dots that sharing the same color vary in memory capacity, and the three sub-figure vary in Global buffer BW.

From these figures, we see that Area-latency trade-off sensitivity towards memory size is high @ low GB BW. It means for a memory BW limited system, carefully selecting memory capacity in the hierarchy is important for system performance.

We also see that with different GB BW, the preferred MAC array size changes.

This case study also clearly shows that the BW-awareness is important for HW parameter optimization on latency.

42 of 43

Outline

Fast Latency Estimation for DNN Accelerator

Necessity + Feasibility + Challenge

A Uniform Intra-Layer Analytical Latency Model

“One-for-All”: A Uniform Dataflow Representation
“All-in-One”: Divide-Combine-Integrate 3-Step Method

Model Validation
Three Case Studies

Mapping v.s. latency
Workload v.s. latency
HW architecture v.s. latency

Conclusion and Future Work

42

43 of 43

Conclusion and Future Work

A unified analytical intra-layer latency model for DNN accelerators is proposed. It can provide latency breakdown and identify performance bottlenecks.
Following a divide-combine-integrate 3-step approach, the model overcomes prior challenges by capturing the periodic operation of hardware components.

43

Conclusion

Future Work

Recent technology such as 3D IC with fine-pitch SRAM-on-logic stacking can offer high BW interconnects. The proposed BW-aware latency model can help explore the new design space.
This intra-layer latency model also builds a solid foundation for modeling latency in cross-layer and multi-core DNN mapping scenarios.