1 of 1

Introduced generative and classification-based LLMs to extract formal dependency relationships from 3GPP protocols, enabling formal verification for 5G and next-generation security.
Designed CAL with cross-attention to capture local and global information, processing complex patterns and extensive connections.
Implemented a dynamic cross-entropy loss function in CAL to address label imbalance.
Augmented the generative REBEL model (Aug-REBEL) with a context-based data augmentation technique to improve accuracy with limited training data.
Compared CAL (classification-based) and Aug-REBEL (generative-based), revealing insights into their performance, with CAL demonstrating robustness for unseen protocols.

Goal: Automate formal verification of 5G protocols to ensure security and correctness.
Approach: Introduced the Cross-Attention Large Language Model (CAL) for extracting formal properties and dependencies.
Comparison: Evaluated CAL against the REBEL model for relation extraction from protocol documents.
Results: Both models reduced manual modeling efforts by over 95% on datasets like DocRED and 3GPP 5G protocols.
Impact: Demonstrates the feasibility of automating formal verification, enhancing 5G protocol trustworthiness and robustness.

Project Abstract

Contributions:

Conclusion

We propose the RC model CAL and utilize the state-of-the-art RE model REBEL to extract formal properties from the 3GPP 5G RRC protocol.
We propose a novel data augmentation technique that enables fine-tuning the model with just four protocol sessions to achieve few-shot training on an uncommon formal dataset.
Our proposed CAL model achieves 92.57% accuracy and Aug-REBEL achieves 99.79% accuracy, which significantly reduces the labor load.
CAL maintains excellent performance on datasets with unseen terminologies.

Jingda Yang

Project Advisor: Ying Wang, Ph.D.

Stevens Institute of Technology

Trustworthy Formal Verification in 5G Protocols: Evaluating Generative and Classification Models for Relation Extraction

Model Structure:

Generative model: Augmented cross-attention large language model (Aug-REBEL) comprises a self-attention encoder to extract information and a cross-attention decoder to generate the sequence of tokens with specially formatted triplets.
Classification model: Cross-attention large language model (CAL) utilizes self-attention encoder to extract information and self-attention layers to classify the relationship between entity pairs.

Loss function (designed to address the problem of skewed class distribution in classification models):

Balances skewed class distribution by amplifying underrepresented classes using an inverse class ratio and a tunable focusing parameter (λ).
In this equation, N signifies the total number of cases, yi represents the ground truth, p(yi) is the predicted probability, n(yi) denotes the proportion of class yi within all cases and λ means the tunable focusing parameter. The inverse class ratio amplifies the impact of the minority class while reducing the dominance of the majority, helping to balance skewed data.

Methodology:

Acknowledgement

TABLE II: Comparison of Aug-REBEL and CAL result on 5G RRC protocol (technic protocol)

TABLE I: Comparison of Aug-REBEL and CAL result on DocRED (general nature language)

Fig. 3: Comparison of Aug-REBEL and CAL result on unseen data.

Result:

Generative model (Aug-REBEL) has better performance than classification model, no matter on specific technology dataset or general nature language.
CAL can achieve a 100% recall when λ is increased to 5. This exceptional performance significantly reduces over 95% of the manual labor load, which cannot be achieved by adjusting the parameter in RE models.
compared to the robustness of the classification model (CAL), the generative model (Aug-REBEL) experiences a significant drop in performance when dealing with context-partial-aware (unseen) data.

This effort was sponsored by the Defense Advanced Research Project Agency (DARPA) under grant no. D22AP00144. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of DARPA or the U.S. Government.

Fig. 2: Comparison of loss weight. Our novel data-distributed weighted loss applies (1 − ratio)λ as loss weight to the cross-entropy criterion. As the λ increases, loss will focus more on cases with lower distribution.

Fig. 1: Model architectures of Aug-REBEL and CAL.

www.PosterPresentations.com

QUICK START GUIDE�(THIS SIDEBAR WILL NOT PRINT)
This PowerPoint template produces a 36"x48" presentation poster. You can use it to create your research poster by placing your title, subtitle, text, tables, charts and photos. We provide a series of online tutorials that will guide you through the poster design process and answer your poster production questions. For complete template tutorials, go online to PosterPresentations.com and click on the HELP DESK tab. To print your poster using our same-day professional printing service, go online to PosterPresentations.com and click on "Order your poster".
This is a template for a presentation poster�36 inches tall�by�48 inches wide�	Important: Check the template size�Before you start working on your poster and to avoid printing problems check that you have downloaded and that you are using the correct size template for your poster presentation.�This template can also be printed at the following sizes without distortion and without any additional formatting:�30 tall x 40 wide�42 tall x 56 wide�48 tall x 64 wide
	How to Zoom in and out Use the PowerPoint zoom tool to adjust the screen magnification to view comfortably. PowerPoint provides 2 ways to zoom: �1. On the top menu bar click on the VIEW tab and then click on ZOOM. Choose the zoom percentage that works best for you. �2. For better zoom flexibility, use the zoom slider at the bottom right of the window.
Ruler and Guides�The dotted lines on his poster template are guides. The horizontal and vertical guides will help you align your poster elements accurately. Text boxes and other elements will ”snap” to the guides and stay within the boundaries of the columns. To hide the guides go to VIEW and uncheck the Guides box.
	Headers and text containers�Included in this template are commonly used section headers such as Abstract, Objectives, Methods, Results, etc. �- Click inside a section header to add its text. �- To add another header, click on edge of the section box so that it is outlined. Copy and paste it. �- To increase its size, click on the white circles and expand to the the desired size.
Adding content to the poster Start by adding your text to each section without spending too much time with formatting. Use the default font size even if your text extends beyond the bottom of the poster. Continue until you have added all your content including text, graphics, photos, etc. Once you finish adding your content you can go back and format your text as needed. If you run out of room, try to reduce the size of your fonts and/or the size of your graphics. If there is a lot of empty space try to increase your font sizes and the size of your graphics. The font used for references can be smaller.
Photos You can add photos by dragging and dropping from your desktop, copy and paste, or by going to INSERT > PICTURES. Resize images proportionally by holding down the SHIFT key and dragging one of the white corner handles (dots). For a professional-looking poster, do not distort your images by stretching them disproportionally.

Quality check your graphics Zoom in and look at your images at 100%-200% magnification. If they look clear, they will print well.

QUICK START GUIDE�(THIS SIDEBAR WILL NOT PRINT)
How to change the template colors You can change the overall template color theme by clicking on the COLORS dropdown menu under the DESIGN tab. You can see a tutorial here: https://www.posterpresentations.com/how-to-change-the-research-poster-template-colors.html You can also manually change the color of individual elements by going to VIEW > SLIDE MASTER. On the left side of your screen select the background master where you can change the template background, column sizes, etc. After you finish working on the SLIDE MASTER, it is important that you go to VIEW > NORMAL to continue working on your poster.
How to change the column layout configuration You can manually change the configuration on the columns by going to VIEW > SLIDE MASTER. You can delete columns, resize them or modify them as needed for your layout. You can see a tutorial here: https://www.posterpresentations.com/how-to-change-the-column-configuration.html
		How to hide the QUICK START GUIDE bars from the sides of the template The Quick Start Guides are outside the template’s printable area and they will not be on the printed poster. If you create a PDF file from your template, the guides will not be included. To hide the guides click on the Home tab (top of the screen) and then click on the Layout button below to see the available layouts. Choose the Without Guides layout.

How to preview your poster prior to printing You can preview your poster at any time by pressing the F5 key on your keyboard. You will see on the screen what's on your poster and how it should look when printed. Press the ESC key to exit Preview.		F5
How to print your poster When you are ready to have your poster printed go online to PosterPresentations.com and click on the "Order Your Poster" button. You can have your poster printed on professional papers, fabric for easy traveling and a variety of other materials. If you submit a PowerPoint document, you will be receiving a PDF proof for your approval prior to printing. If your order is placed and paid for before noon (Pacific time) Monday-Friday, your order will ship out that same day. FedEx Next day, Second day, Third day, and Free Ground services are offered. �Go to PosterPresentations.com for more information.

© 2019 PosterPresentations.com�2117 Fourth Street , STE C Berkeley CA 94710 USA	For complete tutorials visit: https://www.posterpresentations.com/helpdesk.html