TripCraft: A Benchmark for Spatio-Temporally Fine Grained Travel Planning�

ACL 2025

1

Preliminary Problem Statement

Given some details about a trip (places + duration + constraints),

we want to get an optimal itinerary.

Why use LLMs over Lookups?

More Interactive

power

Understanding

Context

Proximity is not

always the key !

Comparison of Travel Planning Datasets

ACL 2025

4

TripCraft’s Contributions

1. Fine-grained travel planning dataset: TripCraft introduces a rich dataset that includes detailed attraction types, persona-based preferences, event categories, and public transit information, surpassing prior benchmarks.

​

• Highly detailed itinerary generation: The approach enhances spatio-temporal granularity, ensuring comprehensive and contextually coherent travel itineraries in the form of a sequence of Point of Interests.

​

• Novel continuous evaluation metrics: Introduces the first-ever continuous evaluation framework for LLM-generated travel plans, moving beyond binary constraints for nuanced itinerary assessment.

ACL 2025

5

Overview

ACL 2025

6

���How does TripCraft’s output differ from TravelPlanner’s?

ACL 2025

7

TripCraft’s Response for a particular day

TravelPlanner’s Response for a particular day

Dataset details

ACL 2025

8

Dataset Construction Pipeline

​

• Data Collection & Cleaning: Web scraping and open-source tools (e.g., OSM, GTFS) ensure up-to-date, consistent data with missing entries carefully handled.
• Persona & Query Construction: Queries generated using GPT-4o in a few shot setting from structured inputs (json). Persona details are sampled from the 4 components.
• Annotation & Refinement: Graduate annotators and domain experts iteratively refine plans, ensuring high-quality, explainable, and feasible itineraries. (approx. 30 mins/query)

ACL 2025

9

Proposed Evaluation Metrics

ACL 2025

10

Persona Score

Ordering Score

Spatial Score

Temporal Meal Score

Temporal Attraction Score

Experimental Settings

• w/o Parameter Information: Modified prompts to include event details, PoI lists, and a refined one-shot example.
• with Parameter Information: Extends the prompt with natural language descriptions of parameterized distributions.

​

ACL 2025

11

Results

• Trade-off b/w objective metrics and constraint adherence.
• Incorporating parameter information improves objective metrics, especially in meal scheduling and spatial efficiency.

ACL 2025

12

Results

• However, this leads to more constraint violations, as seen in lower commonsense pass rates.
• The trade-off suggests that while parameter guidance helps structure itineraries, it may introduce rigid assumptions that conflict with real-world feasibility.

ACL 2025

13

Benefit of using our proposed metrics

• Traditional percentage-based evaluations fail to differentiate travel plans with similar feasibility under given constraints.
• However, our proposed continuous scoring metric goes beyond binary pass/fail outcomes, offering a more interpretable assessment of LLM-generated plans.

ACL 2025

14

Limitations in LLM-generated itineraries

• Temporal Inconsistencies
• PoI visits extend beyond departure times.
• Meal schedules misalign with natural dining hours.
• Activity timestamps are misordered, disrupting itinerary flow.

​

• Poor Adaptation to Traveler Personas
• Adventure-seekers: Under-scheduled activities.
• Laidback travelers: Excessive transit between locations.

​

• Weak Spatial Reasoning
• Impractical transit assignments.
• Accommodations and attractions placed far from transit stops.

​

ACL 2025

15

Future Direction of Work

ACL 2025

16

• Extending dataset to Europe & Asia – Expanding data collection to capture regional travel preferences and cultural nuances for better global generalization.
• Enhancing LLM performance – Improving reasoning, structured query handling, and real-time travel assistance through:
• CoT/ToT reasoning – Implementing multi-step logical reasoning to enhance response coherence and decision-making.
• Tool-usage – Similar to Two-stage method where 1^st step is to pinpoint what tool should be used and 2^nd step is to fetch the information using that tool.

​

​

Acknowledgement

• This work was partially supported by the Microsoft Academic Partnership Grant (MAPG) 2024.

​

• We sincerely thank the annotators for their diligent and meticulous efforts in curating and labelling the data, which were essential for the creation of the dataset.

​

• We also thank the anonymous reviewers for their valuable feedback and constructive suggestions, which helped enhance the quality and clarity of the paper.

Thank you !