1 of 26

UMSI Data4Good

Updates & Demo

D4G Team

July 25, 2024

2 of 26

Agenda

Introductions (4 mins)
Products (2 mins)
Hangul Design (12 mins)
Hangul Summary Generation (10 mins)
Other Algorithm Improvements (2 mins)
Hangul Demo (16 mins)
Q&A (12 mins)
Next Steps (2 mins)

3 of 26

Introductions

4 of 26

Intro

The Data4Good center is working on projects that are at the center of the UMSI mission: “We create and share knowledge so that people will use information — with technology — to build a better world.”

Our History:

The UMSI Data4Good (D4G) center brings together data from nonprofit organizations into larger datasets for benchmarking and trend analysis. By empowering nonprofits with comprehensive data, we contribute to a broader understanding of development and relief programs, ultimately enhancing decision-making in nonprofit work.

Our Values:

Transparency and equal access to information.
Utilizing data analysis to positively impact lives through nonprofit organizations.

Thank you for joining us. We look forward to your feedback and insights.

5 of 26

Team Members

6 of 26

Products

7 of 26

Data4Good Products

Chetah: a search engine for nonprofits reports published on the web. It locates reports with a state of the art deep learning algorithm.

Hangul: An NLP-based assistant designed for digital curators at ReliefWeb to handle a larger volume of documents.

8 of 26

Hangul Design

9 of 26

Hangul Timeline

W2023

Demo of Hangul 1.0 to ReliefWeb team (Mar. 7)
UMSI Poster Presentation (Apr. 17)
Added Hangul to the D4G website

S2023

Hangul 2.0 work
Added models for Theme, Language, Disaster Type, Locations, Improved Summary, Markdown text
Added the front-end and back-end architecture

F2023

Hangul 2.0 tuning
Presentation to systems advisor, Intelligems CTO (Oct. 18)
Summary experiments

S2024

Hangul 2.1 presentation
Demo to ReliefWeb team (July 25)

W2024

Hangul 2.1
Changed back-end structure; process PDF in stages
PythonAnywhere memory constraint
Algorithm testing, especially for the Summary

10 of 26

Hangul 2.1: What Changed?

Result Item	Hangul 1.0	Hangul 2.0
Filename	✖	✔
Report Title	✔	✔+
Report Author	✔	✔
Document Creation Date	✔	✔
Document Modified Date	✔	✔
Report Type	✔	✔
Number of Pages	✔	✔

Result Item	Hangul 1.0	Hangul 2.0
Document Language	✔	✔
Document Locations	✔	✔+
Disaster Type	✔	✔
Themes	✖	✔
Markdown	✖	✔
Keywords	✔	✔
Summary	✖	✔

Result Item	Hangul 2.1
Memory Re Architecture	✔

11 of 26

Hangul 2.1: Application Development

Overall Updates

Implemented new theme

Updated mobile-view

UI Updates

Cleaner interface

Feedback to user on backend progress

Front End Updates

Improved system robustness�
Admin metric alerts

Szymon; Q: What changed in design?

Cleaner interface and reduced color palette

> Collapsible results

> Display top 3 locations

> One click copy button

> Hangul 1.0 and 2.0 toggle

Feedback to user on backend progress

> More loading information presented to the user like the progress bar and feedback messages

Improved robustness�> Merged both Hangul versions to reduce Server Side Rendering (SSR)

> Redirecting errors to prevent “infinite” loading screen

> Applied exponential backoff + timeout to API calls

Admin metric alerts

> Provide email alerts for each Hangul use. Success or fail I get a notification with what was provided and if it failed at what stage it did

Implemented new theme

> Updated all pages to new theme

> Updated home page design

> Updated past projects from stubbed links

Updated mobile-view

> Updated mobile-view

��

12 of 26

Hangul 2.1 Data Structure

Objective:

Enhance the quality of metadata features and generate new ones.

Workflow:

Retrieve report data using the ReliefWeb API.
Parse stored reports and extract various related metadata features.
Analyze and process data using NLP techniques to create a dataset suitable for training and testing models.

13 of 26

Hangul API

Memory optimization

Earlier in the year we encounter memory limitations, that required us to run Hangul under 3 GB.

We overcame this limitation by reengineering our API for a two-stage document processing.

14 of 26

Hangul 2.1: Improved API Handling

Hangul

Python Anywhere

HTTP Request

Bad response?

Less than 3 tries?

2^n + rand delay

YES

NO

HTTP Response

YES

Szymon; Q:How did we restructure the front-end to optimize execution performance? Szymon: Added to presentation notes��Due to the 3GB server limit, we found that a lot of analyses were failing due to that limit. However, the same PDF did not result in a failure each time it was used. Therefore, we implemented the Applied Exponential Backoff methodology as shown here.��Applied exponential backoff means we attempt to call the API multiple times if there is a failed call. Each call is made with an increasing delays as well as a random offset. This helps to avoid cases in which some situation synchronizes many clients, and all retry at once, sending requests in synchronized waves.

From a user perspective, you will see the results as they populate and if there is a re-try necessary this will all be done in the background without user interference necessary

15 of 26

Hangul Summary

16 of 26

Summary: Generation

17 of 26

Summary: Quality Metrics

BERTScore is an evaluation metric that leverages BERT embeddings to measure the semantic similarity between generated and reference texts. Instead of relying on exact word matches, BERTScore captures the contextual and nuanced meanings of words in their specific contexts.

Example:

Reference: "The cat sat on the mat."

Generated: "The feline rested on the rug."

Example Scores

18 of 26

Summary Generation Time

Leo; Q:What are the factors that drive execution time for the summary and what did we find?

So one of the feature we have for hangul is that we provide an estimation to analyze the metadata for each report. Right now the factor that we used to generate the estimation is through the amount of pages for each report, because that’s the most instinctual factor that we think will impact the time to analyze the metadata, so that’s what we use for now. However, it’s not as accurate as we have thought. Depending on the report type there’s a huge difference in the time to analyze the metadata per page.

On avg how long it takes to analyze each pdf for reliefweb? On avg around 2 min for a 25 page pdf for us. Would that be a sufficient interest for you on how much time we help you save?

19 of 26

Summary Generation Time

20 of 26

Summary Generation Time

21 of 26

Other Algorithm Improvements

22 of 26

Hangul 2.1: Success Metrics

23 of 26

Hangul Demo

Presenter: Leo

We selected pdf that’s been analyzed by reliefweb. The first two are ones that have a great match. The last pdf is non-relief web
Talk about estimated time, it’s an estimation.Earlier than estimation, it will reach 0 straight away. Once it reached 0, it doesn’t mean it failed.
Page number or size doesn’t relate to time analysis to complete.
We should have a better metric to do time estimation, we’re still investigating the metric.
The green tick let us knw we’re able to gather the data.
Location shows the top 3 locations
Copy feature
Characteristics of the pdf. Where it’s from? Type of report? How big is it?
So right now our success rate around 75% with around 70 different files among different testers
Do you have any questions about our demo so far?

Side by side comparison between 2.0 and relief web

Generated summary not extracted summary

Monitor: Szymon/Xabier

Include successful demo and error handling.

Chetah 2.0 dataset changes:
Integrated the modifications present in existing metadata elements of Hangul, by incorporating all alterations and updates to Chetah 2.0 dataset.
Developed a python script for identifying and extracting new metadata items from the provided ReliefWeb PDF documents and then integrated those into the Chetah 2.0 dataset.
Crafted a presentation to assess and comprehend the quality of the metadata items present in Chetah 2.0 dataset.

24 of 26

Questions

Your Questions For Us

Our Questions For You

On average, how long does it take an editor to complete a PDF
Given the context of assistant-AI, what would be the obstacles to adoption?
Given what we have tried, what would you do next?
How can we make this better for the end-user?
As a user what would you expect to see and be able to do on the D4G website?
If you were to present this to a nonprofit leader, what would you present?
What constitutes “World” for location?

25 of 26

Next Steps

Continued summary improvement
Determine the factors driving the time-to-analyze
Share APIs between Hangul and Chetah
Have Hangul results feed the Chetah dataset
Develop an MVP for ReliefChat

26 of 26

Thank you!