Agentic Apps

Lessons Learnt From Creating Agentic Applications

By:

Michal Polanowski and Srikrishna Iyer (Krishna)

Group Technology Office

GTO AI.DA: What We Do?

We incorporate the latest Gen AI research into various POCs and MVPs.

Focus on the Agentic Approaches.

Primarily work on TRL 3-7 projects.

Have projects that vary from a single to as many as 5 agents. Upcoming projects are expected to incorporate 20+ different agents, agents in simulation environments and robotic agents.

Source

AI Agents Are LLMs With Additional Abilities

AI agents can encompass a wide range of functionalities beyond natural language processing including decision-making, problem-solving, interacting with external environments and executing actions.��AI agents work in a self-directing loop, setting tasks for AI, determining priorities, and re-prioritizing tasks until the overall goal is achieved.

Source Source2

Agentic Project Features

Information hub: can answer queries about the users' current situation, responsibilities, and expectations.

Empathetic listening: a safe space for users to express their thoughts, concerns, and experiences.

Memory: learns about the user and offers individual and personalised interactions with the recognition and recollection of past interactions.

Survey: incorporates evidence-based surveys to help monitor user well-being.

Adaptive support: offers personalized guidance and coping strategies to help users navigate challenges they may face.

Reliable Agentic Approach Is Difficult

Here are some challenges that we will discuss:

• System Design View
• How to design the system/states/transitions?
• How do we consistently direct the traffic to the right module and for how long?
• Prompt Design View
• Which prompt format to use?
• How to account for the reliable transitions between states?

Transition Slide

Popular Agentic Frameworks Did Not Work For Us

The frameworks are either too fragile or too slow to work in a “normal” conversation mode.

Also, there is no need for an agent to go through a plan at every interaction with a user.

Lastly, these solutions become very expensive due to loops and unnecessary actions.

Our Agentic Approach Needs To Be a State Machine

• A finite-state machine (FSM) or simply a state machine (SM), is an abstract mathematical model of computation that can be in exactly one of a finite number of states at any given time (Mealy, 1955; Moore et al., 1956)
• A few simple terminologies :
• State: a description of the status of a system that is waiting to execute a transition
• Transition: a set of actions to be executed when a condition is fulfilled or when an event is received.
• Entry: performed when entering the state.
• Exit: performed when exiting the state.
• Few examples include vending machines, elevators, traffic lights, combination locks and many more.

SMs Exert More Precise Control Over Agentic Systems

• Agentic LLM systems use LLMs to determine the sequence of steps to complete a task/query.
• However, several studies [Amazon,ASU,Google] have shown that LLMs lack deductive reasoning capabilities (for now) i.e. their capacity to follow instructions in reasoning tasks.
• To alleviate this issue, we restrict the LLMs to a pre-defined search space i.e. use specific states for the LLM planner to choose from.
• Reliability vs Generalization trade-off : However, it increases reliability and reduces generalizability across tasks, which is an agreeable trade-off since most application-specific LLMs have predefined states.

Sweet spot?

The above paper is a great overview of the solution we have created.��“A state represents the status of a running process. The transitions between states are controlled by heuristic rules or decisions made by the LLM, allowing for a dynamic and adaptive progression.”

​

States Determine the Behaviour of the System

States have three levels:

L0 Temporal: Pre | During | Post (eg. Pre-Course, During-Course, Post-Course)

L1 Chatbot Module: Empathetic, RAG (eg. General course queries), Counsellor RAG (eg. Tutor), Survey (eg. Assessment)

L2 Chatbot State: Enter, Execute, End

​

The challenge is to properly and reliably design the states and transitions.

[Design 1] Agentic Only Approach (Fail)

​

• Langgraph implements state machines using directed acyclic graphs (DAG).
• However, with the modification (iteration check) - not strictly a DAG anymore!
• Limitations :
• Every query needs to be passed through the supervisor, increasing latency, reducing state transition success rate.
• Increased LLM calls and costs
• Lack of state persistence
• Lack of transparency - eg. why did the agent transition not work ?

Source : https://blog.langchain.dev/langgraph/

[Design 2]: Semantic only routing (Fail)

​

• Uses predefined “utterances” to decide the route.
• Use semantic_router by aurelio-labs that selects the route measuring cosine-similarity between encoded query and encoded utterance.
• Limitations
• A giant if-then is not scalable and not easily maintainable.
• Poor reliability and consistency issues due to ambiguity in routing.
• Not generalizable to edge cases
• Lack of transparency
• Very sensitive to slight variations in prompt
• Cannot handle multi-step queries.

Transitions: Semantic, Hybrid, Agentic (Supervisor), LLMs

We can control the transitions using different approaches and at different States.

Static (If-Then)

Dynamic (LLM)

System Level

Conversation Level

Hybrid Approach

Semantic Routing

If-Then Rules

Conversation LLM

Agentic Supervisor

L0

L1

L1

L2

[Final] Hybrid Approach: Hierarchical LLM state machine

L1 Chatbot Modules:

Empathetic, Counsellor, Survey, RAG

Temporal router

RAG

L0 Temporal:

Pre, During, Post

L2 Chatbot State:

Enter, Execute, End

Pre

During

Post

L1 Chatbot Module:

Empathetic, RAG, Counsellor RAG, Survey

Evaluation results

​

• Dataset generated using Claude 3.5 Sonnet.
• 3 main tasks :
• Single : Single query
• Double : Two sequential queries
• Triple : Three sequential queries
• Each task divided into 3 levels of difficulty (easy,medium,hard) each with 10 queries.

​

Note : Excludes the survey state

Comparison of Success rates

(proportion of correctly classified states)

LLM Slide

How to structure the prompt?

There are many ways to design a prompt to make LLMs behave consistently.

There are huge differences between a Chat prompt vs a Production prompt.

We Got Lucky With A Claude 3.5 Artifact Prompt Leak

XML tags are very useful and generate consistent code behaviour.

Can be used for:

• Structured output
• Function calling
• Internal thinking
• Model Suggestions
• External data injections
• and more…

Our LLMs Handle L2 State Transitions (Execute + End)

Conclusions

Figuring out these elements took very long as we could not find any online references on how to do it well.

Start with understanding how your app will�behave (LLMs, transitions, functionality, etc).

Figure out what is static (if-then) and what should�be dynamic.

Prompt XML tags are your best friend for any�function calling or a dynamic chat information �Retrieval.

It is a lot of system engineering :(

About The GenAI Team

Srikrishna Iyer (Krishna) (Senior AI Engineer). He focuses on applied research, system design and development of generative AI (LLMs and multi-agent AI) solutions. Previously worked as a data analyst for ASMPT pte. Ltd. on data driven semiconductor process optimisation as well as a research assistant in SingTel Cognitive and AI lab (SCALE@NTU) for enterprises (Connect).

Michal Polanowski (Head of the Generative AI). Spearheads efforts to translate Gen AI research into commercial solutions. Has over a decade of experience leading DS, ML, and AI teams for local and international firms like Delhaize, Disney, Lazada, and GoTo (Connect).

Ryzal Kamis (MLOps Lead) Focuses on system administration, DevOps, and development. He works with on AI-enabled solutions, ranging from conventional ML applications to Gen AI solutions. In the past, he managed an MLOps team and initiatives for an AI-centric national program (AI Singapore), was a teaching assistant at NUS, and a research analyst for an MNC (Connect).