The Blue Sky Research in LLM and Alignment

​

Soujanya Poria

On behalf of DeCLaRe Lab

POS

HMM/CR

NER

MT

Tag

NLI

TC

Finance

Search/IR

Social Media

Chatbots/

Assistants

Anomaly

Medical

2023 <

2023

2024

Fundamental

Applications

Chatbots

Maths,

Coding

Prompting

Search/IR

LLMs

PPO

DPO

CoT

LLMs

Merge

Distil

k-BitsQ

Reason

Align

LLMs

Navigation

Agents

AutoEval

Search/IR

LLMs

Why Alignment is Needed?

• Consider the current-day risks/harms of today’s AI systems
• misinformation
• fairness/biases
• privacy
• One thing these items have in common is that the risks directly relate to the things the systems were designed to do.

​

Preference Alignment: RLHF

Alignment is Often Defined as

Helpfulness – all types of capabilities required to help the user.

• Good at reasoning.
• Good at perception.
• RAG etc.

Harmlessness – all types resistance against harmful intent and affecting users with harmful information

- Trustworthiness.

- Safety.

- Privacy.

Understanding Reasoning Bottlenecks

Hong et al. Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions.

Are LLMs Math Marvels?

1. There are lot of existing Mathematical Reasoning datasets:
1. GSM8k (92%, five shot)
2. MMLU math (87.5%)
3. MATH (50.4%)

1. Does GPT-4 already perform well on arithmetic reasoning?
2. Is the performance drop due to complexity or diversity of the underlying context? Or something else?

Testing LLMs’ Math and Coding Competency

You perhaps heard about GSM-Symbolic but did not hear about this work 😓�This work came 10 months before GSM-Symbolic

Research Question

1. How robust is the arithmetic reasoning capability of LLMs?
1. Through Perturbations. (Great performance drop)
2. Succeed because of data contamination issues. (Almost certain)

​

• Under which conditions or across which dimensions do these models show limitations in reasoning?
• Can GPT4 perform only limited ways of reasoning? (Yes)
• Can LLMs understand unusual mathematical questions? (No)

​

We choose to study those questions from the easiest data - GSM8k.

If we make it harder, we succeed!

Question Decomposition

Ontology

Picked 5 Random Questions

​

Curation

1. GPT4 generation

For variability

​

​

• GPT4 filtering

​

​

​

• Manual Filtering (⅓ correct, ⅓ minor modification, ⅓ Fails)

- manual effort is needed when creating a high quality dataset! GPT4 cannot evaluate itself.

Examples - Logic Alteration

Original:

​

​

​

​

Variable Relationship:

John has X boxes. Each box is Y inches by 6 inches by 4 inches. The walls are 1 inch thick. If the total inner volume of all the boxes is 72 cubic inches, then find the equation that relates X and Y?

​

Examples - Concept Analysis

Original:

​

​

​

​

Step Necessity:

John has 3 boxes. Each box is 5 inches by 6 inches by 4 inches. The walls are 1 inch thick. Suppose we want to find out the total inner volume of all 3 boxes. To solve this math question, is there a way to determine the total inner volume of all 3 boxes without calculating the inner volume of one box?

Examples - Format Change

Original:

​

​

​

​

Question:

John has X boxes. Each box is Y inches by 6 inches by 4 inches. The walls are 1

inch thick. If the total inner volume of all the boxes is 72 cubic inches, then find the equation

that relates X and Y?

​

Results - General

​

1. Pronounced Drop in every model
2. Concept analysis is the most difficult category as it requires high level understanding of math
3. Closed sourced models are affected more by Format Change.

Multimodal bottlenecks

Chia et al. PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns. ACL Findings 2024.

PuzzleVQA Ontology

The Struggle of LLMs on PuzzleVQA

How about Algorithmic Puzzles?

The Story does not Change….

How about Planning?

Can Do Dataset

Planning Bottlenecks

Learning to Reason

Chia et al. Learning to Reason and Explore From Diverse Paths. EMNLP 2024

Motivations

1. Prompting techniques are not robust.
2. Upon identifying the reasoning bottlenecks, can we tune LLMs by making them powerful to address those bottlenecks?
3. What other techniques are possible?
1. Improved tokenization.
1. Left to right prediction may not work well for math.

Reasoning Paths Optimization:�A Framework For Exploring And Learning From Diverse Reasoning Paths

• Motivation: Reasoning in language models may easily diverse into errors

Framework: Reasoning Paths Optimization

• 1. Generation
• Leverage CoT to obtain reference paths
• 2. Exploration
• From each step in the path, expand with favorable and unfavorable branches
• 3. Optimization
• Provide contrastive feedback to enhance LLM reasoning

Main Results

• We observe consistent benefits across math and science reasoning tasks

Analysis

• Further experiments show that RPO scales well to longer reasoning solutions

Takeaways

• Reasoning in language models can easily diverse into errors
• Our approach addresses this with contrastive feedback over the favorable and unfavorable branches
• Unlike previous works, we do not require human-annotated reasoning paths
• Thus, we believe this is a scalable and effective method to improve reasoning

Improving Helpfulness with Verification

​

​

Yu et al. Reward Steering with Evolutionary Heuristics for Inference-time Alignment. Arxiv 2024.

Not All Votes Count!

Programs as Verifiers Improve Self-Consistency of Language Models for Math Reasoning

Motivations

1. LLMs often make arithmetic errors.
2. Majority-voted answers can still be incorrect.

​

Key Idea

• Use translated programs, derived from natural language solutions, as a verification mechanism to identify and filter out incorrect reasoning paths before aggregating final answers.

Framework

1. Generating plan and solution
1. Using Plan and Solve prompting.
2. Translation
1. Convert plan and solution to Python program.
3. Verification
1. Verify solution using program output.
4. Selection
2. Select final answer using majority voting over all verified answers.

Main Results

• PROVE consistently outperforms baselines across all model sizes and datasets, achieving improvements of up to 18% on GSM8K and 8% on MATH-500.

Analysis

• The improvement over baselines remains consistent as the number of samples increases.

Analysis

• PROVE reduces calculation errors.

Takeaways

• We demonstrate that using translated programs for verification can effectively filter out low-quality reasoning paths.
• Our approach is model-agnostic and does not require fine-tuning or few-shot exemplars for prompting.
• PROVE consistently outperforms baseline methods across 13 LLMs and eight mathematical reasoning datasets.

Inference-time Alignment

​

​

Yu et al. Reward Steering with Evolutionary Heuristics for Inference-time Alignment. Arxiv 2024.

Problem and Challenge

​

Problem : Ensuring LLMs operate in a way that aligns with human intended goals. Involves guiding the model's behavior to be safe, reliable, and aligned with the desired outcomes of its users, avoiding harmful or biased outputs.

​

​

​

Challenges: Current preference optimization methods interferes with model prior LLM training and risking adherence to evolving user expectations.

​

​

​

Inference-Time Alignment: Aligning models without explicit weight updates to LLMs through modifying in decoding method of LLM.

Reward Steering with Evolutionary Heuristics for Inference-time Alignment

​

Darwin approaches inference-Time Alignment problem

as a reward guided tree search problem.

​

✔ Decouples exploration and exploitation of tree search

✔ Able to use on top of preference tuning method

✔ Uses an off-the-shelf reward model

​

✔ Outperform strong baselines on Alpacaeval2 and MT-Bench

​

42

Exploration and Exploitation

​

​

​

​

​

Darwin Workflow

Darwin Improves LLM Performance during Inference

Darwin Improves SIMPO and DPO

Motivations

• Prompting techniques are not robust.
• Upon identifying the reasoning bottlenecks, can we tune LLMs by making them powerful to address those bottlenecks?
• What other techniques are possible?
• Improved tokenization.
• Left to right prediction may not work well for math.

Model Merging

DELLA-Merging:

Reducing Interference in Model Merging through Magnitude-Based Sampling

Model Merging:

• Computationally efficient compared to training multi-task models
• Enables model to use information from relevant tasks to improve task performance
• Improves out of distribution generalisation
• Reduces biases arising from single task training

Problem:

Maintaining separate fine-tuned models for different tasks presents several limitations eg memory footprint, cost, and leverage transfer learning.

DELLA-Merging

• Step 1 Drop:
• Assigns drop probability p_i to delta parameters inversely proportional to their magnitudes.
• Step 2 Elect:
• Reduces interference by electing parameters with dominant sign.
• Step 3 Fuse:
• Perform weighted averaging of the retained delta parameters.

Drop

Elect

Fuse

DELLA-Merging: MagPrune (Drop Step)

• Assigns a drop probability, p_i to all parameters and rescale retained parameters by 1/(1-p_i).

​

• Probabilities are assigned based on magnitude. As such, Parameters with higher magnitudes have a lower drop rate.

Results

• Della outperforms SOTA methods in 3/4 merge combinations of Instruct, Math and Code models.

• Della shows 2.4% improvement over SOTA merging methods.

Deep, Pala Tej, Rishabh Bhardwaj, and Soujanya Poria. "DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling." arXiv preprint arXiv:2406.11617 (2024).

Multimodal RAG

Overview

• Motivation
• Understanding documents can involve multimodal content such as texts, figures, and tables
• But questions answering over documents can involve hundreds of pages and detailed analysis
• While leveraging retrieval (RAG) can improve efficiency, models may still be distracted by irrelevant content
• Contributions
• M-LongDoc: A benchmark and automatic evaluation framework on multimodal long documents
• Retrieval-aware tuning framework for multimodal document understanding
• Findings
• Most models struggle with figure and table-based questions compared to text-based questions, revealing their multimodal bias
• Experiments show that our tuning approach achieves a relative improvement of 4.6%

Data Example

• In M-LongDoc, we focus on questions with longer explanations or analysis, rather than extracting short answers or counting objects

Data Overview

• M-LongDoc covers diverse topics in academic, financial, and product domains, with much longer documents

Data Construction

• Given each {text, table, figure}, we generate and verify open-ended questions

Evaluation Framework

• To assess open-ended answers, we leverage a detailed evaluation guide with multi-judge scoring

Preliminary Study

• Current models tend to be weaker in processing visual contents (figures and tables) than texts
• Increasing the retrieval context length is expensive and may hurt performance
• Multimodal models may be easily distracted by irrelevant content in the retrieved context

Training Framework

• To effectively leverage retrieval, the model is must distinguish between irrelevant and relevant multimodal content

Results

• Through our retrieval-aware tuning, we improve Qwen2-VL by 4.6% (3.84 -> 4.02)

Vision Language Action Models

LLM for Robotics

Sun, Qi, Pengfei Hong, Tej Deep Pala, Vernon Toh, U. Tan, Deepanway Ghosal, and Soujanya Poria. "Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning." arXiv preprint arXiv:2412.11974 (2024).

Meet Emma-X: An Embodied Multimodal Action Model

• Existing models have ”muscle memory”
• Not capable of reasoning
• Not generalizable
• Hallucinates

​

Meet Emma-X: An Embodied Multimodal Action Model

• Emma-X employs grounded chain of thought and look-ahead spatial reasoning
• Outperforms SOTA by 25%
• On out-of-domain tasks, performance improved by 30%
• On spatial reasoning tasks, performance improved by 40%

​

​

Overview of Emma-X

Overview of Data Construction

Training and Inference with Emma-X

• Trained with visually grounded chain-of-thought
• Knows how to reason spatially
• Easy human intervention

​

Emma-X is the new SOTA

Emma-X in Action

Open the microwave

Pick up an object that is a kind of vegetable

NORA

VLA Trained from Scratch

• VLM → VLA

NORA

Demo (Kitchen)

With object distraction: Put the pink toy in pot

With human distraction: Put the carrot in pot

With human + object distraction: Put the pink toy in pot

NORA

Community Reception

🚀 4K+ downloads in just two weeks

Tango Model Family

Text to Audio Generation

​

Majumder et al. Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization. ACM MM 2024.

Background of Tango

• LDM with UNet backbone
• Flan-T5 encoder as text encoder
• 63 times fewer training samples than AudioLDM

Observations on Tango Outputs

• Missing acoustic event
• Temporally misaligned acoustic events

Alignment to the Rescue

Alignment Dataset

Strategy 1: prompt → four audio samples; vary the denoising steps​

Alignment Dataset

Strategy 1: prompt → four audio samples; vary the denoising steps​

​

Strategy 2: prompt → perturbed prompts → audio samples​

​

Strategy 3: prompt → temporally perturbed prompts → audio samples

Perturbed Prompts

Alignment Dataset

Audio Alpaca Stats

Results

TangoFlux

• Powered by rectified flow matching.
• SOTA results thanks to online iterative DPO training.

Hung, Chia-Yu, et al. "TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization." arXiv preprint arXiv:2412.21037 (2024).

Online Iterative Training

Results

Community Response

Youtube video: https://www.youtube.com/watch?v=bzrRhnD23v4&t=376s&ab_channel=AISearch

Multimodal Representation Learning

Why Multimodal? — Human Communication is Multimodal

Introduction

• Each modality provides complimentary information
• Key applications in behavior understanding
• Analyse interviews.
• Deception detection.
• Application in legal.
• Tele-medication for mental health
• Brain-computer interface
• Multi-sensory inputs

Major Challenge in Multimodal Analysis

• Develop techniques to fuse multiple modalities.
• Modalities are heterogenous making it hard to fuse.
• Handling large data.

Introduction

Blueprint of Multimodal Fusion

Unimodal

Representations

Joint

Multimodal

Representation

Intermediate

Representations

Audio

Visual

Text

• Improving intermediate representations
• Lead to better joint multimodal representations.
• Apply inductive bias using constraints.

Introduction

Hazarika, Devamanyu, Roger Zimmermann, and Soujanya Poria. "Misa: Modality-invariant and-specific representations for multimodal sentiment analysis." Proceedings of the 28th ACM international conference on multimedia. 2020.

MISA vs Rest

Introduction

Why Disentangle Features?

• Modality-invariant features may not be helpful as modalities do not agree w.r.t. label

• An inductive bias to be faithful to the input modality composition

Introduction

Introduction

Task Setup

Overall Framework

Method

Combining Modality-invariant

and -specific Features

Method

Distributional Similarity for Invariant Features

Method

Modeling Orthogonal Modality-specific Features

Method

Preventing to Learn Trivial Representations

Method

Combining Modality-invariant and -specific Features

Method

One of the first few works showing attention can be used for multimodal fusion

The Overall Loss Function

Method

Datasets

Experiments

Baselines

Temporal Fusion:

​

Attention Transformer:

​

Graph-based:

​

Tensor-Fusion:

​

Common Representations:

​

Inter-utterance Joint Models:

MFN, MARN, MV-LSTM, RMFN

​

RAVEN, MulT

​

Graph-MFN

​

TFN, LMF, LMFN, HFFN

​

MCTN, ARGF, MFM

​

BC-LSTM, CH-FUSION, CIA, CIM-MTL, DFF-ATMF

Experiments

State of the Art

Interaction Canonical Correlation Network

Sun, Z. et al., Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis. AAAI 2020

Contextual Memory Fusion Network

Hasan et al. UR-FUNNY: A Multimodal Language Dataset for Understanding Humor. EMNLP-IJCNLP 2019

Experiments

Low-level Features

Experiments

Results

CMU-MOSI

MAE — Lower is better

• Consistent improvement over state-of-the-art
• Model improves more when the task is difficult
• Sentiment/emotion intensity prediction.
• 7-way sentiment classification.
• Indicates the importance of modality invariant and specific features.

Results

CMU-MOSEI

UR_FUNNY

Similar Trend of Results on CMU-MOSEI and UR_Funny

Results

Ablations

• Every modality is important.
• Language modality is the most crucial.
• Due to clean transcriptions.
• All three losses are vital.
• is the least important.
• Non trivial representations learned by modality encoders.

Analysis

t-SNE Projections

Final Loss:

• Presence of
• Invariant features are grouped together.

Analysis

Contribution of Learned Vectors

• Plot of penultimate multihead attention scores.
• Text modality contributes the most.
• High scores for both modality-specific and invariant.

Analysis

Learning Curve

• Plot indicates all three different losses decrease on the validation data during the training.
• Model is learning as per our hypothesis.

Improvements

Multimodal-Infomax: Overall Idea

Improvements

Multimodal-Infomax: Overall Idea

Modalities are correlated

Capture modality correlation in representations

Maximize Mutual Information

Between pair of modalities

Help learning better intermediate representations

Between modalities and fused representation

Improvements

Multimodal-Infomax: Results

Improvements

Multimodal-Infomax: Results

CMU-MOSEI

CMU-MOSI

Trustworthiness

Song et al. Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse. Arxiv 2024.

LLMs Hallucinate

Do LLMs Know what they Know?

Jokes Apart: The Problem is Really Critical

Problem

Problem

Problem

1. Realistic but incorrect information - hallucination!
2. How to verify information? Source?

Retrieval-Augmented Generation (RAG) as a Solution to Hallucination

• Want LLM to rely only on the external documents
• Need LLM to ground response in the documents rather than rely on parametric knowledge

https://github.com/princeton-nlp/ALCE

Retrieval-Augmented Generation (RAG) as a Solution to Hallucination

• Generate more factually reliable answers when provided external documents
• Easy verification of statements due to citations

https://github.com/princeton-nlp/ALCE

LLM Groundedness

Grounded response:

• ✅ Refuse to answer questions it does not have adequate information for
• ✅ Correctly answers question using only information from the documents
• ✅ Inline citations to the documents to support generated answers

Previous Works

Evaluation

• Current RAG evaluations focus (ALCE, ALiiCE) on the overall system performance, conflating the effects of retriever quality and LLM performance in the metric scores.
• Need for new ways to measure LLM effectiveness in RAG systems without the influence of the retriever.
• Nomiracl analyzes the refusal capabilities of LLMs in a RAG context but lacks holistic evaluation, as it does not account for both response and citation groundedness.

Mitigation

• AGREE, CaLM, FRONT propose frameworks to improve LLM response groundedness but overlook refusal behaviors in their metrics.
• Ignoring refusal behaviors, retriever influence, citation and answer groundedness weakens the ability of current metrics to effectively measure LLM performance in RAG.

Key Contributions

• TRUST-SCORE comprehensively evaluates LLM performance, including refusal, citation, and answer groundedness
• TRUST-ALIGN creates a corresponding alignment dataset, making the metric and approach more unique and holistic for LLM evaluations and alignment in RAG

​

TRUST-SCORE

Assesses an LLM across multiple dimensions:

1) Grounded Refusals: is the model able to discern which questions can be answered or refused based on the provided documents?

2) Exact Match scores: For the answerable questions, is the response correct?

3) Citation recall: Are the generated statements supported by the corresponding citations?

4) Citation precision: Are the citations to the statements relevant?

TRUST-SCORE

TRUST-ALIGN

• Propose alignment dataset consisting of 19K questions, documents, positive (preferred) responses, and negative (unpreferred) responses to enhance groundedness of LLMs
• Dataset covers a range of five LLM hallucination types—Inaccurate Answer, Over-Responsiveness, Excessive Refusal, Over-Citation, and Improper Citation

TRUST-ALIGN

Collecting Quality Questions

Collecting D’s

Augmenting (q,D) Set

Answerability Labelling

Details on Claim Document Mapping

Augmenting (q,D) Set

Obtaining r+ and r−

Obtaining r+ and r−

Effectiveness of Our Data Construction Approach

TRUST-ALIGN Boosts Trustworthiness of Models

TRUST-ALIGN Improves Models’ Refusal Capability

TRUST-ALIGN Enhances Models’ Citation Quality

Mixed Results on Exact Match Recall due to Models’ Usage of Parametric Knowledge

Models Aligned with DPO Generally Outperform those Trained with SFT

TRUST-ALIGN Generalizes across Model Families and Sizes

Importance Of Refusal Samples In Trust-Align

Improvements Generalizes on Out-of-Domain Data

Studying Parametric Knowledge Access

Quantify how many unanswerable questions were answered correctly

Revised Metrics Are Less Biased

Reduction in performance gap

Revised Metrics Are Less Biased

Revealing our model’s stronger performance as compared to baseline

Key Findings

• TRUST-ALIGN boosts trustworthiness of models.
• TRUST-ALIGN improves models’ refusal capability
• TRUST-ALIGN enhances models’ citation quality.
• Models aligned with DPO generally outperform those trained with SFT
• TRUST-ALIGN generalizes across model families and sizes.

Paper:

Codebase:

Introduction: Knowledge-Intensive Tasks

Introduction: Chain-of-Thought

Introduction: Retrieval-Augmented LLMs

Introduction: Synergizing Reasoning, Retrieval, Correction

Overview

• Motivation
• Reduce LLM hallucination through knowledge retrieval
• Diverse knowledge sources, both general and domain-specific, structured and unstructured
• Error propagation of factual mistakes is common in multi-hop questions
• Proposal
• Chain-of-knowledge (CoK): Framework to augment LLMs with knowledge sources
• Adaptive query generator: Module to help LLMs retrieve from diverse sources
• CoK employs iterative retrieval and correction to mitigate error propagation
• Findings
• Outperforms previous methods in multiple domains (factual, medical, physical etc)

Framework: Reasoning Generation & Domain Selection

Relevant domains: Factual (Wikidata, Wikipedia)

Framework: Iterative Retrieval and Correction

Framework: Adaptive Query Generation

Diverse Query Examples

Main Results

Analysis: Effect of Multiple Knowledge Sources

Analysis: Factuality of Rationales

Takeaways

• Heterogenous knowledge retrieval
• Is promising to improve factuality in multiple domains
• Requires query generators for specialized domains
• Chain-of-knowledge
• Synergizes reasoning, retrieval, and correction
• Employs iterative correction to mitigate error propagation
• Future work
• Can integrate new modalities such as images
• Can further improve the effectiveness of domain-specific retrieval

Safety

​

Safety issues with LLMs

1. Aligned LLMs can be red-teamed through just prompting.
2. Training LLMs on just 100 samples make them vulnerable.

Ferret: Motivation

• Can we automatically test how vulnerable LLMs are?
• Can prompts be diverse?
• Can we make automated red-teaming faster?
• Can we adapt to mixture of adversarial styles?

Ferret: Methodology

Pala et al. Ferret: Faster and Effective Automated Red Teaming with Reward-Based Scoring Technique. Arxiv 2024

Ferret: Main Results

• All Ferret variant outperform Baselines in Llama Guard 2 ASR.
• Reward Model Scoring Function Shows Consistent Performance Across Risk Categories.
• Reward Model Scoring Function Shows Greater Alignment with Llama Guard 2 and GPT-4 ASR.

Ferret: Analysis

• Ferret (RM) achieves ASR threshold faster than Rainbow Teaming (+CF)

Language Models are Homer Simpson!

​

The solution is quite simple!

Bhardwaj et al. Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic. ACL 2024.

Summary of the results

Side-effects

More Generalized Version

Safety Arithmetic

Hazra et al. Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations. EMNLP 2024.

Understanding Safe Align

Solution to this equation is the PCA of

Add ICV to latent states

Start with a few exemplars

Take the latent vectors toward safe from unsafe

Simple yet Effective Solution!

Base

SFT

WM for WizardMath, LM for LlamaMath, and EC for EvolCodeAlpaca

Lower is better

WalledEval

A Comprehensive Safety Evaluation Toolkit for Large Language Models

Safety vs Refusal

(exag-safety)

Multilingual Safety

LLM Benchmarking: Numbers on the left for the first four datasets indicate the percentage of safe responses to unsafe prompts, referred to as harmful behavior (Judge: LlamaGuard 2). Nmbers on the right represent the percentage of instances where the LLM correctly chooses to refuse (for unsafe prompts) or accept (for safe prompts), referred to as refusal behavior (Judge: MCQJudge). Green, yellow, and red colors denote the highest, second highest, and lowest scores in the columns, respectively. \textbf{XSTest} (Mutated) refers to XSTestm}.

Judge Benchmarking

Thank you!