1 of 64

Mehran S. Moghadam^♣⁺, Sercan Aygun^♠⁺, Faeze S. Banitaba^♠, and M. Hassan Najafi^♣

^♣Case School of Engineering, Case Western Reserve University

^♠ School of Computing & Informatics, University of Louisiana at Lafayette

sercan.aygun@louisiana.edu

All You Need is Unary:

End-to-End Bit-Stream Processing in Hyperdimensional Computing

�

International Symposium on Low Power Electronics and Design (ISLPED 2024)

August 5-7, 2024

Newport Beach, CA, USA

⁺ Equal Contribution

2 of 64

2

Outline

ISLPED

2024

① Introduction, Background, and Motivations

- Hyperdimensional Computing (HDC)

- Stream Generators

- Pseudo-Randomness vs. Quasi-Randomness

② Novel Encoding Methods

- Single-source Position Hypervector (Position HV)

- Unary-based Level Hypervector (Level HV)

③ Results

- Hardware Efficiency

- Medical MNIST Performance

④ Conclusions

⑥

3 of 64

3

Introduction

3

ISLPED

2024

Emerging Computing

II. Hyperdimensional Computing (HDC)

IV.

Approximate Computing (AC)

V.

Quantum Computing (QC)

I.

Unary Bit-stream Computing

III.

Stochastic Computing (SC)

...

4 of 64

4

Introduction

4

ISLPED

2024

+

Emerging Computing

II. Hyperdimensional Computing (HDC)

IV.

Approximate Computing (AC)

V.

Quantum Computing (QC)

I.

Unary Bit-stream Computing

End-to-End

Bit-Stream Processing in Hyperdimensional Computing

Our Proposal

Exploiting the Unary Computing in HDC

Migrating from Random Process to Deterministic Computing

Target

III.

Stochastic Computing (SC)

...

5 of 64

5

Introduction

5

ISLPED

2024

☒

☑

Conventional Neural Networks

❶

6 of 64

6

Introduction

6

ISLPED

2024

☒

☑

iterations

Conventional Neural Networks

❶

❷

7 of 64

7

Introduction

7

ISLPED

2024

☒

☑

iterations

Conventional Neural Networks

❶

❷

>

8 of 64

8

Introduction

8

ISLPED

2024

☒

☑

ε

Error Calculation

iterations

Conventional Neural Networks

❶

❷

>

9 of 64

9

Introduction

9

ISLPED

2024

☒

☑

ε

Error Calculation

iterations

Conventional Neural Networks

❶

❷

Backprop.

>

10 of 64

10

Introduction

10

ISLPED

2024

☒

☑

ε

Error Calculation

iterations

Conventional Neural Networks

❶

❷

❸

Backprop.

>

11 of 64

11

Introduction

11

ISLPED

2024

☒

☑

ε

Error Calculation

iterations

Conventional Neural Networks

❶

❷

❸

❹

Backprop.

>

12 of 64

12

Introduction

12

ISLPED

2024

Single-pass learning

❷

No error check & backprop.

Light model�(model compression)

❸

Direct data processing

❹

HDC

❶

13 of 64

13

Introduction

13

ISLPED

2024

Single-pass learning

❷

No error check & backprop.

Light model�(model compression)

❸

Direct data processing

❹

HDC

❶

: Hypervector

(atomic data)

+1

...

-1

14 of 64

Encoding

Training Data

Training and Inference in HDC

14

ISLPED

2024

❶

15 of 64

Encoding

Training

Training Data

+1

-1

...

-1

Training and Inference in HDC

15

ISLPED

2024

❶

16 of 64

Encoding

Training

Training Data

+1

-1

...

-1

...

Class

C_0D

C₀₂

C₀₁

Training and Inference in HDC

16

ISLPED

2024

❶

❷

17 of 64

Encoding

Training

Training Data

+1

-1

...

-1

...

Class

C_0D

C₀₂

C₀₁

Encoding

Testing Data

Training and Inference in HDC

17

ISLPED

2024

❷

❶

18 of 64

Encoding

Training

Training Data

+1

-1

...

-1

...

Class

C_0D

C₀₂

C₀₁

Encoding

Testing Data

...

h_1D

h₁₂

h₁₁

Training and Inference in HDC

18

ISLPED

2024

❷

❶

19 of 64

Encoding

Training

Training Data

+1

-1

...

-1

...

Class

C_0D

C₀₂

C₀₁

Encoding

Testing Data

...

h_1D

h₁₂

h₁₁

query

Training and Inference in HDC

19

ISLPED

2024

❷

❶

20 of 64

Encoding

Training

Training Data

Item Memory

Assoc.�Memory

+1

-1

...

-1

...

Class

C_0D

C₀₂

C₀₁

Encoding

Testing Data

...

h_1D

h₁₂

h₁₁

query

Training and Inference in HDC

20

ISLPED

2024

❸

❷

❶

21 of 64

Encoding

Training

Training Data

Item Memory

Assoc.�Memory

+1

-1

...

-1

...

Class

C_0D

C₀₂

C₀₁

Encoding

Testing Data

...

h_1D

h₁₂

h₁₁

Similarity

query

Training and Inference in HDC

21

ISLPED

2024

❹

22 of 64

Encoding

Training

Training Data

Item Memory

Assoc.�Memory

+1

-1

...

-1

...

Class

C_0D

C₀₂

C₀₁

Encoding

Testing Data

...

h_1D

h₁₂

h₁₁

Similarity

query

Applications

22

ISLPED

2024

EEG, iEEG, ECG,

EMG, GSR, ECoG

DNA

MNIST COCO CIFAR

RNA

Voice

Letter

23 of 64

Encoding

Training and Inference in HDC

23

ISLPED

2024

Binding

Bundling

Permutation

Hypervector Generation

Hypervector Mapping

POP++

-1

1

…

24 of 64

① Introduction, Background, and Motivations

- Hyperdimensional Computing (HDC)

- Stream Generators

- Pseudo-Randomness vs. Quasi-Randomness

② Novel Encoding methods

- Single-source Position Hypervector (Position HV)

- Unary-based Level Hypervector (Level HV)

③ Results

- Hardware Efficiency

- Medical MNIST Performance

④ Conclusions

⑥

24

ISLPED

2024

25 of 64

Stream Generators

25

ISLPED

2024

26 of 64

Stream Generators

26

ISLPED

2024

27 of 64

Stream Generators

27

ISLPED

2024

28 of 64

Stream Generators

28

ISLPED

2024

29 of 64

Energy

Efficient

👍

Stream Generators

29

ISLPED

2024

30 of 64

① Introduction, Background, and Motivations

- Hyperdimensional Computing (HDC)

- Stream Generators

- Pseudo-Randomness vs. Quasi-Randomness

② Novel Encoding methods

- Single-source Position Hypervector (Position HV)

- Unary-based Level Hypervector (Level HV)

③ Results

- Hardware Efficiency

- Medical MNIST Performance

④ Conclusions

⑥

Stream Generators

30

ISLPED

2024

31 of 64

Pseudo-Randomness vs. Quasi-Randomness

31

ISLPED

2024

Pseudo-Randomness

Quasi-Randomness

Scattering

High-Discrepancy

Low-Discrepancy

Sequence-1

Sequence-2

Sequence-1

Sequence-2

32 of 64

Pseudo-Randomness vs. Quasi-Randomness

32

ISLPED

2024

Pseudo-Randomness

Quasi-Randomness

Scattering

High-Discrepancy

Distribution

Low-Discrepancy

Sequence-1

Sequence-2

Sequence-1

Sequence-2

Non-Uniform

Uniform

Probability

Population

High-Discrepancy

Low-Discrepancy

Probability

Population

33 of 64

Pseudo-Randomness vs. Quasi-Randomness

33

ISLPED

2024

Pseudo-Randomness

Quasi-Randomness

Scattering

High-Discrepancy

Distribution

Low-Discrepancy

Sequence-1

Sequence-2

Sequence-1

Sequence-2

Non-Uniform

Uniform

Probability

Population

High-Discrepancy

Low-Discrepancy

Probability

Population

Orthogonality

Vector-1

Vector-2

Vector-1

Vector-2

Weaker

Orthogonality

Strong

Orthogonality

34 of 64

Weaker

Orthogonality

👍

Strong

Orthogonality

👍

Pseudo-Randomness vs. Quasi-Randomness

34

ISLPED

2024

Vector Symbolic Representation

Orthogonality

35 of 64

Weaker

Orthogonality

👍

Strong

Orthogonality

👍

Pseudo-Randomness vs. Quasi-Randomness

35

ISLPED

2024

Vector Symbolic Representation

1

-1

1

…

D

Symbol-1

Orthogonality

36 of 64

Weaker

Orthogonality

👍

Strong

Orthogonality

👍

Pseudo-Randomness vs. Quasi-Randomness

36

ISLPED

2024

Vector Symbolic Representation

1

-1

1

…

D

Symbol-1

Symbol-2

-1

1

…

Symbol-n

1

-1

…

Orthogonal

Orthogonality

37 of 64

Weaker

Orthogonality

👍

Strong

Orthogonality

👍

Pseudo-Randomness vs. Quasi-Randomness

37

ISLPED

2024

Vector Symbolic Representation

1

-1

1

…

D

Symbol-1

Orthogonality

Symbol-2

-1

1

…

Symbol-n

1

-1

…

Orthogonal

38 of 64

Weaker

Orthogonality

👍

Strong

Orthogonality

👍

Pseudo-Randomness vs. Quasi-Randomness

38

ISLPED

2024

Vector Symbolic Representation

1

-1

1

…

D

Symbol-1

Orthogonality

Symbol-2

-1

1

…

Symbol-n

1

-1

…

Orthogonal

n different LFSRs!!

39 of 64

Weaker

Orthogonality

👍

Strong

Orthogonality

👍

Pseudo-Randomness vs. Quasi-Randomness

39

ISLPED

2024

Vector Symbolic Representation

1

-1

1

…

D

Symbol-1

Orthogonality

Symbol-2

-1

1

…

Symbol-n

1

-1

…

Orthogonal

Can be a single source possible?

n different LFSRs!!

Van Der Corput (VDC) sequences
Single counter & Hardwiring

40 of 64

Unbinding randomness from HV generation (Position & Level HVs)

Utilizing quasi-random/deterministic sequences for HV generation

Targets

40

ISLPED

2024

41 of 64

Outline

① Introduction, Background, and Motivations

- Hyperdimensional Computing (HDC)

- Stream Generators

- Pseudo-Randomness vs. Quasi-Randomness

② Novel Encoding Methods

- Single-source Position Hypervector (Position HV)

- Unary-based Level Hypervector (Level HV)

③ Results

- Hardware Efficiency

- Medical MNIST Performance

④ Conclusions

⑥

ISLPED

2024

41

42 of 64

VDC-2

Q

T

Q

T

Q

T

Q

T

Q

T

Q

T

Q

T

Q

T

CLK

V_cc

Tff7

Tff6

Tff5

Tff4

Tff3

Tff2

Tff1

Tff0

MSB

LSB

V7

V6

V5

V4

V3

V2

V1

V0

Single-Source Position Generator

Position Hypervector Generation

Strong

Orthogonality

👍

Proposed

Novel Encoding Methods

42

ISLPED

2024

43 of 64

VDC-2

Q

T

Q

T

Q

T

Q

T

Q

T

Q

T

Q

T

Q

T

CLK

V_cc

Tff7

Tff6

Tff5

Tff4

Tff3

Tff2

Tff1

Tff0

MSB

LSB

V7

V6

V5

V4

V3

V2

V1

V0

Position Hypervector Generation

Novel Encoding Methods

43

ISLPED

2024

🔍

M. S. Moghadam, S. Aygun, M. R. Alam and M. H. Najafi, "P2LSG: Powers-of-2 Low-Discrepancy Sequence Generator for Stochastic Computing," 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC), Incheon, Korea

44 of 64

VDC-2

Q

T

Q

T

Q

T

Q

T

Q

T

Q

T

Q

T

Q

T

CLK

V_cc

Tff7

Tff6

Tff5

Tff4

Tff3

Tff2

Tff1

Tff0

MSB

LSB

V7

V6

V5

V4

V3

V2

V1

V0

Position Hypervector Generation

Novel Encoding Methods

44

ISLPED

2024

🔍

M. S. Moghadam, S. Aygun, M. R. Alam and M. H. Najafi, "P2LSG: Powers-of-2 Low-Discrepancy Sequence Generator for Stochastic Computing," 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC), Incheon, Korea

45 of 64

VDC-2

Q

T

Q

T

Q

T

Q

T

Q

T

Q

T

Q

T

Q

T

CLK

V_cc

Tff7

Tff6

Tff5

Tff4

Tff3

Tff2

Tff1

Tff0

MSB

LSB

V7

V6

V5

V4

V3

V2

V1

V0

CMP

Single-Source Position Generator

Position Hypervector Generation

Strong

Orthogonality

👍

Proposed

Novel Encoding Methods

45

ISLPED

2024

46 of 64

T

Q

T-FF

VDC-2

Q

T

Q

T

Q

T

Q

T

Q

T

Q

T

Q

T

Q

T

CLK

V_cc

Tff7

Tff6

Tff5

Tff4

Tff3

Tff2

Tff1

Tff0

MSB

LSB

V7

V6

V5

V4

V3

V2

V1

V0

XOR

CMP

CLK

Single-Source Position Generator

Position Hypervector Generation

Strong

Orthogonality

👍

Proposed

Novel Encoding Methods

46

ISLPED

2024

47 of 64

T

Q

T-FF

VDC-2

Q

T

Q

T

Q

T

Q

T

Q

T

Q

T

Q

T

Q

T

CLK

V_cc

Tff7

Tff6

Tff5

Tff4

Tff3

Tff2

Tff1

Tff0

MSB

LSB

V7

V6

V5

V4

V3

V2

V1

V0

XOR

CMP

🔍

e.g.,110101…1010

CLK

Single-Source Position Generator

Orthogonal and equidistributed HV
Efficient and lightweight hardware
Improved accuracy with a single run
Optimized power up to 25%

Position Hypervector Generation

Strong

Orthogonality

👍

Proposed

Novel Encoding Methods

47

ISLPED

2024

48 of 64

Outline

① Introduction, Background, and Motivations

- Hyperdimensional Computing (HDC)

- Stream Generators

- Pseudo-Randomness vs. Quasi-Randomness

② Novel Encoding Methods

- Single-source Position Hypervector (Position HV)

- Unary-based Level Hypervector (Level HV)

③ Results

- Hardware Efficiency

- Medical MNIST Performance

④ Conclusions

⑥

48

ISLPED

2024

Outline

49 of 64

Proposed

Novel Encoding Methods

49

ISLPED

2024

MNIST Sample Data

(Number 7)

Pixel value

8-bit

✻

Unary-based Level Generator

Level Hypervector Generation

50 of 64

Proposed

Novel Encoding Methods

50

ISLPED

2024

MNIST Sample Data

(Number 7)

CNT

D

Q

W^th

. .

D

Q

1^th

CLK

Q

W-bit

Pixel value

8-bit

✻

CMP

Unary-based Level Generator

Level Hypervector Generation

51 of 64

Proposed

Novel Encoding Methods

51

ISLPED

2024

MNIST Sample Data

(Number 7)

CNT

D

Q

W^th

. .

D

Q

1^th

CLK

Q

W-bit

Left Shifter

Pixel value

8-bit

✻

# of shifts

c

✻

CMP

Unary-based Level Generator

Level Hypervector Generation

52 of 64

Proposed

Novel Encoding Methods

52

ISLPED

2024

MNIST Sample Data

(Number 7)

CNT

D

Q

W^th

. .

D

Q

1^th

CLK

111..11

000..00

e.g., pixel value = 75

111..11

1

00..00

Q

W-bit

Left Shifter

Pixel value

8-bit

✻

# of shifts

c

✻

e.g., pixel value = 76

consecutive Pixel values

🔍

CMP

Unary-based Level Generator

Level Hypervector Generation

Unary-like Level HVs
Streamlined & unique hardware design
Improved performance
Energy efficient design

53 of 64

End-to-End Design

53

ISLPED

2024

54 of 64

Outline

① Introduction, Background, and Motivations

- Hyperdimensional Computing (HDC)

- Stream Generators

- Pseudo-Randomness vs. Quasi-Randomness

② Novel Encoding methods

- Single-source Position Hypervector (Position HV)

- Unary-based Level Hypervector (Level HV)

③ Results

- Hardware Efficiency

- Medical MNIST Performance

④ Conclusions

⑥

ISLPED

2024

54

55 of 64

Position Hypervector Generation

Power consumption reduction by 98%

Energy efficiency improvement by 15%

Considering MNIST images || CPL: Critical Path Latency

Results

55

ISLPED

2024

56 of 64

Level Hypervector Generation

Considering 8-bit gray-scale image pixels within the [0,255] interval

Results

56

ISLPED

2024

Power consumption reduction ≈ 68×

Area × Delay improvement ≈ 39×

57 of 64

① Introduction, Background, and Motivations

- Hyperdimensional Computing (HDC)

- Stream Generators

- Pseudo-Randomness vs. Quasi-Randomness

② Novel Encoding Methods

- Single-source Position Hypervector (Position HV)

- Unary-based Level Hypervector (Level HV)

③ Results

- Hardware Efficiency

- Medical MNIST Performance

④ Conclusions

⑥

Outline

57

ISLPED

2024

58 of 64

Medical MNIST Performance

Ref: medmnist.com

Results

58

ISLPED

2024

59 of 64

Medical MNIST Performance

Ref: medmnist.com

Results

59

ISLPED

2024

60 of 64

Medical MNIST Performance

Ref: medmnist.com

Results

60

ISLPED

2024

61 of 64

Medical MNIST Performance

Ref: medmnist.com

Results

61

ISLPED

2024

62 of 64

Outline

① Introduction, Background, and Motivations

- Hyperdimensional Computing (HDC)

- Stream Generators

- Pseudo-Randomness vs. Quasi-Randomness

② Novel Encoding Methods

- Single-source Position Hypervector (Position HV)

- Unary-based Level Hypervector (Level HV)

③ Results

- Hardware Efficiency

- Medical MNIST Performance

④ Conclusions

⑥

62

ISLPED

2024

63 of 64

Conclusions

Discussion on Pseudo-Randomness vs. Quasi-Randomness for hypervector generation

Incorporating single-source, streamlined, and efficient Position Hypervector generator design

Leveraging Unary Computing for Level Hypervector design

Significant improvements in power consumption, energy efficiency, and area-delay product

Achieving higher scores on performance metrics (Sensitivity, Precision, F1-score, …) for Medical datasets

63

ISLPED

2024

64 of 64

THANK YOU

sercan.aygun@louisiana.edu

ISLPED

2024

64