NL-to-SQL Agent

Presentation @ Blue Yonder by Morgan

​

Brief in the speaker notes

Why?

• A lot of companies have a lot of data
• Between this data and business people, there is an engineering wall
• This usually leads to:
• Very few people in the organization are using the data
• The data team complaining about being inundated with requests from business teams

Welcome Natural Language to SQL (NL-to-SQL)!

• The promise: Chat with your data

2 use cases

1. SQL co-pilots — that deliver a first draft SQL to a data scientist who can modify it
2. Natural language querying — where there is no human in the loop, the user does not see the intermediate SQL nor the underlying schema

In this presentation, we will be looking at NL querying.

SQL co-pilots is the current working use case, it increases the data team scalability but does not fully solve the problem.

NL-to-SQL in practice

The features built on text-to-SQL are mediocre at best and annoying at worst.

It is at a point where some are questioning if users even want these features at all -> “AI washing” (https://techcrunch.com/2024/08/20/is-your-company-ai-washing-rippling-founder-parker-conrad-thinks-it-might-be/ )

There are few if any solutions that actually work…

Current problems

• Non-deterministic — similar prompts can create different answers, as shown in the example below.
• Hard to train — there is no direct correlation between time invested in adding context and the resulting accuracy. It is a trial and error approach.
• Inaccurate — as a result of the above, real-world scenarios have accuracy of 60% (as I heard)

Elephant in the room: After only one failure, a user might doubt the technology or ignore it altogether

Current solutions

1. Connect the tool to your existing relational database to a subset of your schemas, tables, columns, or views
2. Add “context” by providing verified samples for few shot prompting or fine-tuning. Add more context by chunking and adding metadata from unstructured sources like business documentation.
3. Build an LLM agent to build a prompt with the most relevant semantic context, execute it against the DB, and iterate to recover from errors

Demo time

http://qualid.morgangiraud.com:5000/

This is an example of a basic direct use of NL-to-SQL using an Open Source LLM

Remarks

• There is no fancy tricks so far
• It works surprisingly well for simple queries!
• It starts to be broken as soon as something exotic exist in the DB
• It can’t seem to correct itself

Improving LLM performance

Generalized Chain-of-Thought

• LLM always improved their confidence in an answer when they can first add some reasoning about the question
• Making a tree of reasoning and voting for the best query

Structured output

• After the reasoning steps, We can ensure that the LLM outputs theoretically valid SQL statements for a given DB (CFG + schema aware)

LLM-modulo framework

• LLM should and must be getting feedback from software 1.0 (validator, optimizer)

Agent framework:

• ReAct: https://arxiv.org/abs/2210.03629
• Reflexion: https://proceedings.neurips.cc/paper_files/paper/2023/hash/1b44b878bb782e6954cd888628510e90-Abstract-Conference.html

​

Second demo

http://qualid.morgangiraud.com:5000/

This is an example of a React agent using an Open Source LLM

Remarks

• The agent is more complex than direct prompt engineering
• It still works surprisingly well for simple queries!
• Yet, it does not seem to be able to correct itself for exotic DB properties

NL-to-SQL Agent

Presentation @ Blue Yonder by Morgan

​

​

Thanks for listening