1 of 4

AI to Prevent Phishing, Scams, and Online Hate

Co-Leads: Hongxin Hu (University at Buffalo, SUNY) and �Shirin Nilizadeh (University of Texas at Arlington)�

Scribe: Nishant Vishwamitra (University of Texas at San Antonio)�

NSF SaTC PI Meeting 2024 �Breakout 2

Breakout sessions are held with the goal of identifying important new challenges and trends in securing cyberspace, new directions for research, and areas in which the SaTC community can contribute to an improved future society. The breakout leads will organize and lead the discussion on Sept 4th (Day 1) and then do a very brief presentation on Sept 5th (Day 2) afternoon. The breakout group discussion is a good opportunity for our community to connect with each other, identify key challenges, and share ideas. Detailed goals for each of the breakout topics may include answering the following questions:

What is the topic? Why is it important to society? to a secure and trustworthy cyberspace? In other ways?
Is there an existing body of research and/or practice? What are some highlights or pointers to it?
What are the important challenges that remain? Are there new challenges that have arisen based on new models, new knowledge, new technologies, new uses, etc?
Are there promising directions to addressing them? What kinds of expertise and collaboration are needed (disciplines and subdisciplines)?

You can organize the breakout session in the way you think is most conducive to elicit answers to the above questions by the community. One suggestion is to give a short presentation introducing the topic area and some of your initial thoughts about it, and then divide participants into teams to answer the questions above, and finally ask the teams to report the main conclusions from their discussion. We also recommend breakout leads find volunteers to scribe the session so they can summarize the discussion.

For reference, examples of the reports and the slides from the last SaTC PI meeting can be found here: https://cps-vo.org/group/satc-pimtg22/breakouts

2 of 4

Problems

How is GenAI transforming the landscape of phishing, scams, and online hate?
How is GenAI reshaping the detection and mitigation of phishing scams, and online hate?

3 of 4

Challenges

Measuring GenAI’s impact on phishing and online hate is challenging.
GenAI adds sophistication to the attacks

in various languages and modalities
personalized
targeted

GenAI may introduce unseen phishing and implicit online hate, deceiving advanced detection models
Data collection and annotation have become more challenging in the post-GenAI era.

4 of 4

Opportunities

Use GenAI to detect and mitigate GenAI-generated attacks
The detection/mitigation methods should involve humans in the loop
Use GenAI capabilities in intend and situation detection, summarization, etc.
Adapt GenAI to provide explainable feedback
Designing chatbots to educate users