Unsupervised cycle detection in agentic applications
Agentic Trajectories
What is an Agentic Trajectory?
A detailed, structured log of all operations (spans) an agent system performs to handle a single request.
Each operation (e.g., a tool call, an LLM inference, an action) is a "span."
All spans for one request are linked by a common trace_id, forming the complete trajectory.
How Are They Created?
As the agent works, each step is automatically recorded.
The parent-child relationships between these steps naturally form a tree or a Directed Acyclic Graph (DAG).
This trajectory is crucial for debugging, performance analysis, and ensuring the agent behaves as expected.
Scenarios
Cycle detection
Drift detection
Healthy
Anomaly
Soft Anomaly
Anomaly
Anomaly
Healthy
Soft Anomaly
Cycles in Agentic Trajectories
What is a Cycle?
A common error where an agent gets stuck in a repetitive loop, calling the same tools or actions without making progress.
Why Do Cycles Happen?
Primarily due to the model's failure to analyze its current state effectively, leading to an infinite loop.
(a) It may not recognize a previous action failed or
(b) that it's repeating steps,
Example Cycle Scenario:
Agent -> Search Tool (fails) -> Agent -> Search Tool (fails) -> ... and so on.
In this work we are not focused on the correctness of the final output. We focus on the structural details of the graph emerging from agentic trajectories and similarity between outputs of sibling agent.
Problem: Tool Cycles
Start
Agent
Tool1
Tool 1
Tool 2
End
Input
Output
O1
O1
O2
Example: Tool Cycle Scenario
A trajectory illustrating presence of a bad cycle
Start
Agent
Tool1
Tool 1
Tool 2
End
Input
Output
O1
O1`
O2
A trajectory illustrating presence of a good cycle
Problem: Recursion Tool Error
Start
Agent 1
Tool1
Input
Error
O1
A1
End
Repeated N times and hit recursion limit
Example: Agent Exit With Error Scenario
A trajectory illustrating error scenario
Problem Formulation
Related Work
Related work | Short comings | How our approach address this |
Doesn’t provide methods to detecting step-repetition and drift. | We focus on drift and cycle detection approaches. | |
Supervised approach which needs large labelled dataset covering all scenarios. Uses LLMs in decision process which is costly, and user cannot quantify the hallucination in the response. | Unsupervised approach. Uses statistical approaches which are cost efficient augmented with semantic similarity check and improves over time. | |
Doesn’t provide any methods in anomaly detection. | Differentiates between good and bad cycles. | |
Doesn’t provide any methods in anomaly detection. | We focus on drift and cycle detection approaches, and we Differentiates between good and bad cycles. | |
Supervised cycle detection using BFA. Matches graph with known attack patterns. Doesn’t focus on drift and cycle detection. Doesn’t improve over time (Requires retraining). | Unsupervised cycle and drift detection which evolves over time. | |
Semi-supervised anomaly detection using autoencoder. Doesn’t focus on cycle detection. Requires examples covering all healthy scenarios to build autoencoder framework. Doesn’t improve over time (Requires retraining). | Unsupervised cycle and drift detection which evolves over time. |
Dataset
525 Unique Prompts
System Prompts
ReAct Prompt
Strict Prompt
Bad Prompt
LLM
GPT 4o
1575 Labelled Trajectories
Trajectories
Trajectory Visualisation
Time
Trajectory Representation: DAG
DAG
Trajectory Representation: Call Stack
Call Stack
We try to identify the structural anomalies when it is represented as:
Structural Algorithm: Solution Outline
Structural Algorithm: CDDAG
Build DAG using the spans of the trajectory with edge weights as the number of times an edge is invoked in a trajectory.
Structural Algorithm: CDCS
Build Call Stack using the spans of the trajectory.
Semantic Algorithm: Solution Outline
Purely structural representation is not sufficient for accurate cycle identification. Graph comparison or sequence matching with known cyclic patterns doesn’t capture semantic nature a cycle can exhibit .
Build DAG using the spans and we define a subgraph of the DAG as exhibiting a bad cycle if there exists a node 𝑣𝑖 with a sibling node 𝑣𝑗 (i.e., both share the same parent) such that their cosine similarity exceeds a predefined threshold
𝜙 ∈ (0, 1] : cos(v𝑖 , v𝑗 ) > 𝜙 then trajectory is labeled as cycle
Start
Agent
Tool1
Tool 2
End
Input
Output
O1
O2
How do we measure repetition of information in Tool2 with respect to Tool1?
To justify all the nodes invoked by an agent, we ensure the similarity between the nodes should be low
Semantic Algorithm: CDSA
Build DAG using the spans and restrict similarity checks to sibling nodes in the DAG
Information flows upward from leaves to parents. A subgraph is flagged as a bad cycle if a node’s content exceeds similarity threshold 𝑠 with its sibling since highly similar leaf nodes likely yield similar parents and ancestors.
Semantic Algorithm: CDSA
Hybrid Approach
We utilize the CDCS and CDSA in conjunction in a hybrid multi-stage approach to label a trajectory as follows:
Call Stack Analysis: Examines the sequence of function calls within the agent's execution to identify potential cyclic patterns, leveraging call stack structure.
Semantic Similarity Confirmation: Upon detection of a potential cycle through call stack analysis, the semantic similarity between trajectory spans is computed to confirm the presence of repetitive content.
Combines call stack-based cycle detection with semantic similarity analysis, provides a computationally efficient unsupervised method to detect cycles in agentic trajectories.
Results: CDDAG
Results: CDCS
CDCS
Results: CDSA
Results
CDCS is computationally inexpensive. It also achieves high specificity. Both qualities helps it to be an ideal primary filter before applying expensive CDSA to reduce false positives further while maintaining the specificity.
Q&A
Question: How do you avoid flagging loops where the agent is actually making progress?�Answer: Our semantic filter (CDSA) specifically looks for information redundancy. Productive cycles yield varied outputs; futile ones do not.
Question: Is this too slow to run during execution?�Answer: The Hybrid Approach uses CDCS as a "guardrail" first. Semantic analysis is only invoked when structural patterns trigger, minimizing latency.
Question: Where does a purely semantic approach fail?�Answer: Comparing structured data (e.g., Google vs. Microsoft stock). The raw data differs, but formats cause high embedding similarity. Our Hybrid CDCS filter prevents this by ensuring a structural cycle exists first.