learning information onto external tools: Using ART for long document arXiv article summarization

Towards AGI Project

Kyle Roth, Georges Belanger Albarran, Alex Fulleringer

 Overview: What is ART ?

1. Art generates automatic multi-step decompositions for new tasks by selecting decompositions of related tasks in the task library

​

• Then Select and use tools in the tool library alongside LLM generation

​

• Human can optionally edit decompositions to improve performance

​

​

​

• For the evaluation, the authors constructed a task library for 15 diverse Big Bench tasks and evaluate ART on 19 unseen test tasks 

​

• These tasks are taken from Big Bench, 6 MMLU tasks and various tasks used by related work on tool use (SQUAD, TriviaQA, SVAMP, MAEPS)

Related work

There was a couple of work related such as :

​

• Scaled finetuning for low-resource adaptation 

​

• Prompting with intermediate reasoning steps 

​

• AuToCoT

​

• Toolformer

Program Grammar

Task: “Answer this high-school Physics question.    

Input: " Hector yanks...”

​

Followed by a sequence of sub-task nodes, represented as a (query answer) pair: 

 “Qi : ..., #i : ...”

​

The sub-task query Q_i has a sub-task name and sub-task input :

“Q1: [search] What is the formula...”

​

​

​

​

The input node contains the task name, a simple instruction describing the task and the input for an instance of the task :

The sub-task answer #i is simply the output of the sub-task ​:

“#1: The horizontal component (Fx) can be calculated...”​

​

The program ends with a dummy sub tasks ​:

“Q3: [EOQ]”​

​

Followed a final answer node ​:

“Ans: 59N”​

Tool Library

Whenever a sub-task query name matches a tool name in the task library e.g. “Q_i [search]”, generation is stopped and resumed after the tool is called and its output is incorporated into the partially completed program. The following tools are used:

Search:  "What is the formula for the horizontal component of tension force?"

    Output:  "…horizontal component (Fx) can be calculated as Ftens*cosine(theta)… "   

​

Code generation:  "Qi : [generate python code]"

    Then, it will be prompted to codex in a multi-line comment in python:

            "Use the formula Fx = Ftens*cosine(theta ) to solve ..." 

​

    Which will generate : 

              "T = 72.0 , theta = 35.0, … , Fx = T*math.cos(radians)"

​

​

Code execution:

•  Executed on a Python environment with arithmetic, symbolic, and scientific computing packages pre-installed 

​

•  The argument to code execution is the previous sub-tasks answer sequence :

               "# (i -1): ..."

​

ART

A run-through of ART on a new task, Physics QA

​

A) Programs of related tasks like anachronisms and Math QA provide few-shot supervision to the LLM where related sub-steps and tools in these programs can be used by the LLM for cross-task generalization (shown in purple)

​

B) Tool use: Search is used to find the appropriate physics formula, and code generation and execution are used to substitute given values and compute the answer (shown in orange)

ART

<Prompt Input Text>

{Frozen LLM}

<LLM Output>

Tool Interface

Tool Shed

<Tool output>

<New Prompt Text>

Tool Call detected!

Tool call

+

+

=

Summary:

1) A chunk is used a prompt

2) Tool command is generated

3) LLM is paused as the tool generates it output

4) LLM continues its generation with the new information

5) Repeats until the 'eoq' tool is called or a maximum number of attempts is reached

"LLM Output Text"

"[Tool name] (tool inputs)"

FROZEN

ART

Summary:

1) A chunk is used a prompt

2) Tool command is generated

3) LLM is paused as the tool generates it output

4) LLM continues its generation with the new information

5) Repeats until the 'eoq' tool is called or a maximum number of attempts is reached

<Prompt Input Text>

{Frozen LLM}

<LLM Output>

Tool Interface

Tool Shed

FROZEN

<Tool output>

Tool Call detected!

Tool call

<New Prompt Text>

<New Prompt Text>

+

+

=

The New Tools

IDEA: Keep running summaries as a dict such that the model has easy access to topical saved information via the dict keys.

​

[List_Keys] <No Input>

{key1, key2, …}

Returns a string with all the summary dictionary keys delineated by commas.

​

[Read] <key>

{ <<Information stored at summary_dictionary[key]>>}

Returns the summary stored under that key.

​

[Write] <(key) section summary>

{}

Saves section summary to the save dict, does not return any text.

A Sample Prompt

• galaxy clusters , as the largest peaks in the cosmic density field , play an important role in astrophysics and cosmology ( e.g. * ? ? ?  * ; * ? ? ?  structure formation theory , realized in large scale n - body simulations , makes robust predictions for cluster space density and clustering within various cosmological models . since clusters are also the most observationally accessible features of large scale structure , they provide an opportunity to place strong constraints on both cosmological parameters and the growth of structure .  the great challenge of cluster cosmology lies in confidently relating the dark matter halos we can robustly predict to the baryonic structures we observe .  substantial work is being done to close the gap between theory and observations from both sides . 
• [write] (galaxy clusters) Understanding galaxy clusters is the most accessible way of improving our cosmological theories and help close the gap between theory and observation.

​

A Sample Prompt

•  on the theory side , numerical simulations of ever increasing complexity and resolution provide new insights into the evolution of baryons within clusters , and to elucidate the connection between cluster galaxies and dark matter substructure @xcite .  observationally , much work is being done to assemble large samples of clusters detected and observed in a wide variety of ways .  this is possible because galaxy clusters provide a rich suite of observables .  optical light is emitted by individual cluster galaxies as well as intracluster stars .  x - rays are emitted by both the hot intracluster medium ( icm ) and agn within cluster galaxies . 
• [write] (numerical simulations and data collection) numerical simulations help us understand baryon evolution within clusters, and through that connections between cluster galaxies and dark matter. Galaxy clusters provide a lot of data for this, including light and x-rays.

​

A Sample Prompt

• this same hot icm scatters microwave background photons passing through the cluster , distorting their spectrum .  finally , the total projected mass distribution of the cluster produces weak and sometimes strong lensing distortions in the images of background galaxies .  each of these observables presents an opportunity to detect clusters and measure their properties . when combined , they allow us to cross - check our understanding of cluster physics in a variety of ways .  clusters were first detected as early as the 18th century as anomalous groupings of similarly bright galaxies @xcite .  optical surveys , which are relatively inexpensive , have long provided the largest cluster catalogs , primarily because of their ability to detect objects with relatively low mass thresholds . while successful in identifying thousands of clusters @xcite and providing the first evidence for dark matter @xcite , early optical detection was plagued by projection of galaxies along the line of sight @xcite . 
• [list-keys] 
• Galaxy clusters , Numerical simulations and data collection
• [read] galaxy clusters
• Understanding galaxy clusters is the most accessible way of improving our cosmological theories and help close the gap between theory and observation.
• [write] (galaxy clusters) Understanding galaxy clusters is the most accessible way of improving our cosmological theories and help close the gap between theory and observation. Early galaxy cluster detection happened in the 18th century but was flawed. It provided early evidence of dark matter

​

Experimental setup and baselines

• Text Summarization=(NLP) task that involves condensing a lengthy text document into a shorter, more compact version while still retaining the most important information and meaning. The goal is to produce a summary that accurately represents the content of the original text in a concise form.
• ArXiv dataset for Long document summarization�
• LLM: gpt-3.5-turbo (~250B) through the OpenAI API
• Evaluation metrics: (ROUGE)
• ROUGE 1: unigram (1-gram) based scoring
• ROUGE 2: bigram (2-gram) based scoring:
• ROUGE L: longest common subsequence-based scoring
• RougeLSum: splits text using “\n”

​

​

https://huggingface.co/spaces/evaluate-metric/rouge

https://paperswithcode.com/task/text-summarization

Experiments

Comparison on 10 examples between:

​

Base: original prompt

Bullets: prompted to produce shorter, bullet-point style intermediate summaries

Running summary: continually stores a running summary in the same key, loading and adding to it each time

​

Qualitative evaluation: 'rouge1': 0.35, 'rouge2': 0.12�Current summary having issues abstracting (stays on details)

GROUND TRUTH ABSTRACT: 'we study the phase behavior of a nematic liquid crystal confined between a flat substrate with strong anchoring and a patterned substrate whose structure and local anchoring strength we vary . by first evaluating an effective surface free energy function characterizing the patterned substrate we derive an expression for the effective free energy of the confined nematic liquid crystal . �

then we determine phase diagrams involving a homogeneous state in which the nematic director is almost uniform and a hybrid aligned nematic state in which the orientation of the director varies through the cell .

​

direct minimization of the free energy functional were performed in order to test the predictions of the effective free energy method .

we find remarkably good agreement between the phase boundaries calculated from the two approaches .�

in addition the effective energy method allows one to determine the energy barriers between two states in a bistable nematic device .'

{'current-summary': "The passage describes a study on the phase behavior of a nematic liquid crystal confined between a chemically uniform, asymmetrically grooved substrate with locally homeotropic anchoring and a flat substrate with strong homeotropic anchoring. The asymmetry of the grating substrate leads to a decrease in the groove depth at which a first-order transition between the h and han phases occurs. The energy barrier between the two coexisting states decreases with increasing asymmetry of the grating surface, but it is well above a typical nematic liquid crystal cell's energy barrier. The average director orientation at the grating surface in two bistable states has been calculated, and the difference between the two bistable states vanishes with increasing substrate asymmetry, which negatively affects the optical properties of a zenithally bistable nematic device."}

Challenges

Dirty code base:

ART authors didn’t take advantage of code reusability and versioning

​

Open AI API:

Monetary cost 

Slow inference time

Rate limit

Prompt size limits

​

Parallelization of the pipeline

​

​

Conclusions

​

​

​

• We create a system which reuses the ART program grammar and tool structure for Long Document Summarization

​

• Our prompts don't inspire GPT-3.5 to be creative with tool use
• It seems to copy details without reasoning about relevance to the summary
• It often calls tools unnecessarily (e.g. reading the text it just wrote)
• Doing better prompt engineering is hard without more space in the context length

​

• Using a local model (i.e. Llama or others) could help solve issues related to the cost and API

​

• Potential for continual learning on a frozen language model

​

​

​