Challenges and Opportunities for HLS Design Interpretability

Adrian Alupoaei

STAR Lab, Oregon State University

Hardware for Sparse Tensors & Need For Interpretability

​

• HLS used to implement sparse tensor accelerators
• HLS tools are not generally designed for sparse algorithms
• HLS tools do not guarantee optimal solutions (some optimal circuits are NP-complete)
• Higher chance of requiring edits
• Revising the generated RTL requires interpreting generated designs

The RTL-to-NL task

Translating RTL designs & related artifacts into natural language

Differs from traditional code-to-text tasks because:

• RTL describes hardware structure and behaviors
• Unique syntax. e.g. @always blocks
• Requires knowledge of digital logic design

Examples:

• Generating documentation for undocumented designs
• Asking “what does this module do?”

module or2(input a, b, output y);

assign y = a | b;

endmodule

RTL

This module performs the OR operation on 2 bits

NL

Impact of Interpreting Design

• Legacy code & IP Integration
• Good documentation speeds up integration
• Verification
• 70% of design cycle, interpretability can speed up the process
• 44% of the time is spent debugging (1/3rd of entire cycle)
• Made up of RTL-NL tasks for debugging: reading code, simulator logs, and other artifacts
• EDA
• Automatically generated RTL is hard to read or modify.
• LLMs can aid with understanding

LLMs for the RTL-NL Task

• Traditional approaches such as static analysis tools fail to generalize

​

• Deep Learning & LLMs are strong generalizers and can be used to automate the RTL-NL tasks

Needed infrastructure

To train robust AI models, you need:

• Datasets to train models
• Benchmarks to evaluate models
• Model Development Tools

Challenges: Curating Datasets

​

• Noticeable absence of datasets
• ~93x less .v and .sv files on github than .java
• Competitive nature in industry, far less open source projects

​

Challenges: Evaluation Metrics

​

• Minor errors or omissions can be critical to RTL-NL Tasks
• Traditional metric such as BLEU & ROUGE inaccurate since they only target lexical similarity / the overlap of words between references
• Embedding vectors & cosine similarity fail to penalize vague or incorrect descriptions
• Using LLMs to “grade” is difficult to scale and problematic

Challenges: Computation & Training

• Computation & long context lengths
• RTL code requires more lines of code, which translates into more tokens for LLMs to process
• Larger context lengths leads to
• Degradation in accuracy
• Hallucinations
• Performance hits
• The output of some HLS tools can exceed the context window of commercial LLMs
• Training
• Models pre trained on software code doesn’t help with RTL since RTL describes hardware structure
• Software code is sequential, RTL can be concurrent

Research Opportunities

• New approaches for dataset construction
• Synthetic datasets
• Evaluation
• Learned Metrics & BLEURT
• Models for RTL Interpretation
• Fine tuned LLaMa model
• Long context architecture / optimization techniques
• LLM-Aware EDA tools

Evaluation: BLEURT-Verilog-QA

• BLEU & ROUGE rely on n-gram overlap for similarity, fail to capture variations
• BLEURT is a pre trained & fine tuned BERT model specifically for NLG tasks
• Compares the similarity of a reference and candidate at a richer level
• Based on transformer architecture

Image Source: Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805.

Evaluation: BLEURT-Verilog-QA - Dataset

• Base dataset contains ~2k question answer pairs
• Pairs contain a question about a snippet of verilog code and a correct human written reference answer
• Gemini used to “perturbate” the reference answers to become incorrect and generates a number (0-1) indicating how correct it is
• Increases distribution of incorrect/correct answers, allows model to learn
• Brings total pairs up to 9k

Reference QA - Score: 1

Model

Generated Answer - Score: 0.7

Generated Answer - Score: 0.3

Generated Answer - Score: 0.2

Generated Answer - Score: 0.0

Evaluation: BLEURT-Verilog-QA

• BLEURT-Verilog-QA fine tuned on the pre-trained BLEURT model
• Richer than BLEU / ROUGE / vector embeddings and more performant than LLMs
• Fine tuned on 9k verilog question-answer pairs
• AUC 0.65 → 0.81

Fine-tuned LLaMa Model for Verilog tasks

• MG Verilog Dataset
• Includes instruction, description, code
• Description contains detailed global summary, high level summary, and block level summaries
• Input = code + instruction
• Output = description
• Model learns to generate structured summaries of Verilog

Fine-tuned LLaMa Model for Verilog tasks

• LLaMa 3.1-8B
• Small open source model with 8 billion parameters
• Fine tuned on the modified MG-Verilog dataset
• Trained for 2 epoch on the data

Future Steps

• Combining the BLEURT-Verilog-QA metric and the fine tuned LLaMA to grade LLaMa’s answers to measure accuracy or to assist with fine tuning.

​

• Integrating the fine tuned model and the evaluation metrics into tools for the chip design cycle

​

• Tackling the large context problem for large HLS generated designs