Before Ablation ��Groundwork for Lexical Alignment Tests

Research group members:

Presentation speakers:

• Tommie Juzek
• Xiaoyang Ming

RCC specialist:

• Jose Hernandez

​

​

07 Nov 2025

@SC-AI Seminar

Before Ablation ��Groundwork for Lexical Alignment Tests

Three core takeaways:

• Measuring AI language: Document-based frequency + windows allows for automation
• FSU’s RCC has great infrastructure
• Lambda is pretty neat

Background

Background

• LLMs overuse certain words

5

Background

• Assumption: �Exposure → influence (Zajonc, 1968; Hasher et al., 1977)
• LLM-authored messaging might shift attitudes (Bai et al., 2025).

6

Background

• LLMs overuse certain words
• Relevance:
• As hundreds of millions�of people are exposed�we want good alignment

7

language alignment & general alignment

Background

• It is thought that Human Preference Learning contributes to this by these fine-tuning methods:
• Learning from Human Feedback (LHF)
• Reinforcement Learning from Human Feedback (RLHF)
• Direct Preference Optimisation (DPO)

8

Background

9

Typical LHF procedures

Background

10

Typical LHF procedures

© NYT & WaPo

Background

• We know:
• LHF: small preference differences → bigger model behaviour differences
• Shown for formatting (Zhang et al. 2024/5)
• But needs to be shown for lexical usage

11

Background

• Thus, loosely speaking, the thinking is:
• relatively few ‘delves’ in the LHF datasets cause rather big changes in model behaviour downstream

12

Background

13

Background

• How to show this?
• Ablation study
• experimental procedure in which components of a system are systematically altered or removed to identify their causal contribution to the system’s overall behaviour

14

Background

• Thus:
• dosage of AI-buzzword usage during training�→
• and we check model behaviour post training

15

Background

Background

• We want two things:
• How do we measure “buzzwordiness”? �→ need a measure
• One of the criticisms of Zhang et al. (2024/5)’s formatting study was that it used synthetic data �→ need real data

17

Big Plan

Big plan

• 1) score buzzwordiness
• 2) get real data from experiment
• 3) run ablation

19

Big plan

• 1) score buzzwordiness ← been working on this
• 2) get real data from experiment
• 3) run ablation

20

Big plan

• 1) score buzzwordiness

→ issue: literature so far always has a moment of manual curation

21

Big plan

• 1) score buzzwordiness
• Mingmeng et al. → some 8 words
• Kobak et al. → ominous manual check (??”omicron”)
• JW25 → manual filtering (“radar”) (EXPLAIN)

22

Big plan

• Approach
• document frequency
• windows

23

Another procedure

​

New-New_Final copy (1).docx

LAS Score

XM

Lexical Alignment Score

Core idea:

• Obtain the lexical frequency discrepancies between human-authored and multiple models’ generated texts.
• Transform this difference as a metric to quantify and assess the alignment shift from human expectations.

26

Lexical Alignment Score

Implementation procedures:

Our research work investigated six public model families:

Lexical Alignment Score

Implementation procedures:

• Collected 42,000 scientific PubMed article abstracts published between 2012 and 2021, prior to the widespread application of LLMs.
• Following the previous approach (Juzek and Ward, 2025), divided each PubMed abstract into two halves of equal length. First half was used as the prompts for each model’s generated continuation and second one as the human standard for subsequent computation.

Lexical Alignment Score

Estimation:

• For each type S from model generation and human text:Compute the frequency of each lemmatised token (unique word or punctuation mark) w using Windowed Document Prevalence.

​

• Here comes the example of Windowed Document Prevalence.

29

Lexical Alignment Score

Continuation S:�In this study, we formulate a clear hypothesis regarding the correlation between social media use and attention span, proposing that frequent exposure to short-form content reduces sustained focus over time.

​

Lemmatized continuation S:�in, this, study, we, formulate, a, clear, hypothesis, regard, the, correlation, between, social, media, use, and, attention, span, propose, that, frequent, exposure, to, short, form, content, reduce, sustain, focus, over, time

postag

Lexical Alignment Score

Lemmatized continuation S using windowed document prevalence(window size=10, window index = 1):�in, this, study, we, formulate, a, clear, hypothesis, regard, the, correlation, between, social, media, use, and, attention, span, propose, that, frequent, exposure, to, short, form, content, reduce, sustain, focus, over, time

Lexical Alignment Score

For a lemma type

w and a model M ∈ {B, I} we define the lemma type-level LAS wLAS:

​

Lexical Alignment Score

Scoring:

Following the estimation way, we define per-lemmatised-token contributions to lemmatised token t:

​

33

Then we defined the each token’s Lexical Alignment Score (LAS):

Lexical Alignment Score

​

By utilizing each token's LAS score as the fundamental unit, we could generate several levels of LAS scores. This was achieved by summing the L2 mean of the basic LAS scores to quantify the alignment shift at the sentence, document, and corpus (model) levels.

Technical Demo

Demo

1. Continuations generation
2. Part-of-speech tag the continuations to get lemmatized tokens
3. Corpus level LAS scores calculation

Acknowledgement to RCC specialist Jose Hernandez for the assistance!

36

Outlook on distributed multiple GPUs computing

• Transformer models demand substantial memory and computing power for both training and inference.
• Our previous work basically required model inference, which is less consuming. However to explore future ablation study, fine-tuning instruction model is inevitable.
• Then, even the partial fine-tuning a 7B LLM in numerical format BF16 precision necessitates around 14-24 GB of VRAM.

37

Outlook on distributed multiple GPUs computing

38

We are notified that torch-based library accelerate can be supported by RCC.��This library is rarely being used by RCC users. Therefore, a series of trials will be necessary to verify its compatibility and functionality.

​

​

More Demo

TJ

Programming with .zips

• projects with zips
• show

40

Continuations with Lambda

• show

41

Analysis

and

Results

Analysis and Results

• analysis of document frequencies �in AI vs human
• show a few lists
• nice thing: by model

43

Lists

44

Interpretation

• All the function words → A good deal of AI language happens on the syntactic level

45

Validation

• “test” set: same results
• convergence with literature
• varying parameters

46

Next Steps

Next steps

Next steps:

• Ablation study
• AI language across the world’s languages

48

Thank You

Contact information

​

RCC

RCC has great infrastructure:

• Running on RCC is very doable
• Outlook for GPUs parallelisation
• And growing: LEEP grant for H100s submitted
• Currently, vanilla access: up to 4hrs
• A100 takes special permission

50

Technical Demo

1. Continuations generation
2. Part-of-speech tag the continuations to get lemmatized tokens
3. Corpus level LAS scores calculation��Acknowledgement to RCC specialist Jose Hernandez for the assistance!

Outlook on distributed multiple GPUs computing

Transformer models demand substantial memory and computing power for both training and inference.

Our previous work basically required model inference, which is less consuming. However to explore future ablation study, fine-tuning instruction model is inevitable.

Then, even the partial fine-tuning a 7B LLM in numerical format BF16 precision necessitates around 14-24 GB of VRAM.