Multi-Agentic Framework for ADA
Avik Dutta @ PROSE team
Microsoft Research
Contents
Motivation
With the introduction of code-interpreter by OpenAI, chat assistants are capable of solving data analytics tasks when a query and a data file is provided.
Our intention is to create a chat assistant, with a better capability to parse the data file, plan the sub-tasks better, be more conversational, and provide accurate answers to DA (Data Analysis) tasks.
Workflow (single agent arch.)
Multi-Agent Interaction
Tool response
Main conversation thread
User response
Tool response
File Management System
Arranged for an FMS which allowed syncing the versions of a data file saved in the local dir with the one with changes made in the OpenAI server. ��This setup is useful in allowing others to plug in their custom function tools to the assistant’s API, and make changes to a file available directly in their local machines instead of relying on permission issues.
Examples
Data File Preview
Performance comparison
ChatGPT ADA
Our Model
How many 911 calls are received during the weekends
I have read and understood the data. There seems to be difference in delimiters used in the file. I have handled this variation and have detected the following columns in the data – …�For finding out the number of 911 calls received during the weekends, I find the format in the ‘Date’ column quite inconsistent. Can you please confirm if I can proceed with a common format of DDMMYYYY for all elements in the column?
This example portrays the skill of the external function tool we use from the PROSE library for data wrangling, which is able to read data in inconsistent format. ChatGPT ADA fails to do so.
Contd…
How many plane crashes happen during the night?
To determine the number of plane crashed that happened during the night, we’ll first need to define what constitutes “night” in terms of hours.
Here’s what we need to clarify:
The assumption that I am making based on the feedback is that "night" is defined between 6:00 pm (18:00) and 6:00 am (06:00), and we should proceed with counting crashes within those hours. Ambiguous or missing entries in the time' column will be ignored for this analysis unless you direct otherwise. Let me know if these assumptions are correct, or provide the specific details you have in mind.
Yes for this case your assumptions are correct.
According to the given criteria for night time (between 6:00 pm and 6:00 am), there have been 1,262 plane crashes that occurred during the night, based on the data in the dataset.
How does the answer change when you consider night to be a duration of 8 hrs ending at 6:00 AM
Considering the revised definition of night as an 8-hour duration ending at 6:00 am (which means night starts at 10:00 pm or 22:00), there have been 649 plane crashes that occurred during this time frame.
✅
❌
✅
❌
ChatGPT ADA
Our Model
The Reviewer Agent checks for inconsistent or unclean data format, which is cleaned and then processed for the final answer. ChatGPT ADA fails to do this check! This demonstrates the usefulness of a multi-agent framework.
Contd…
ChatGPT ADA
Our Model
How many passengers died during plane crash that happened in Germany?
A total of 968 passengers died in plane crashes that occurred in Germany, according to the dataset.
To clarify the process I followed:
- I filtered the dataset for records where the 'location' contained the word "Germany".
- I used a regular expression to extract the number of passenger fatalities from the 'fatalities' column, which has a format like "5 (passengers:0 crew:5)".
- I then summed up these extracted numbers to reach the total count of passenger fatalities.
These steps ensured accurate calculation despite the 'fatalities' column containing complex string data that needed parsing.
Do you detect some pattern between the country and the plane crash
Detecting patterns between countries and plane crash occurrences in the dataset will involve analyzing the 'location' column to extract country information and then summarizing crash data by country. However, to enable proper analysis, we need to establish the granularity of the pattern detection you are interested in. For instance:
- Are we interested in the frequency of crashes per country?
- Do you want to see if certain types of planes crash more often in certain countries?
- Are we looking at a specific time period or considering the entire dataset?
- Are we considering other factors like operator or aircraft type along with the country?
…
For open-ended questions, our model keeps the user on the loop and asks for clarification without blatantly returning an answer like most agents do. This helps understand the user’s intention and get things right most of the time.
This demonstrates the quality and engaging conversation which our model is able to maintain.
Improvements over ChatGPT ADA Summarized
Thank You!