UTP-Bench: Uncertainty-aware Travel Planning Benchmark

Etcharla Revanth Rao¹, Priyanshu Karmakar¹, Shubhojit Mallick²

Manish Gupta², Shreya Ghosh¹, Abhik Jana¹

​

​

​

​

¹IIT Bhubaneswar, India ²Microsoft, India

1

a24cs08008@iitbbs.ac.in, 24cs06010@iitbbs.ac.in, abhikjana@iitbbs.ac.in, shreya@iitbbs.ac.in

shubhojit.mallick@microsoft.com, gmanish@microsoft.com

Why is uncertainty-aware travel planning important?

• LLMs generate personalized multi-day itineraries
• TravelPlanner, TripCraft, ChinaTravel
• Reasoning over diverse travel constraints such as attraction preferences, transportation connectivity, and scheduling requirements.
• Existing benchmarks assume deterministic environments
• Fixed transport schedules, visit durations, transit conditions
• Real travel is uncertain
• Transit delays, congestion, crowd peaks, cascading disruptions
• Feasible plans become impractical
• Binary constraint checks miss robustness
• Does the itinerary allocate sufficient transition time to absorb unexpected delays and variability in attraction visits?
• Can transportation schedules tolerate realistic disruptions without causing cascading failures across subsequent activities?
• Does the plan avoid highly congested periods, or does it schedule visits during peak crowd hours that may significantly affect the overall experience?
• Does the itinerary adapt its pacing according to different traveller risk preferences, balancing exploration with reliability?

Etcharla Revanth Rao, Priyanshu Karmakar, Shubhojit Mallick, Manish Gupta, Shreya Ghosh, Abhik Jana. UTP-Bench: Uncertainty-aware Travel Planning Benchmark. EMNLP 2026.

2

What is UTP-Bench?

Etcharla Revanth Rao, Priyanshu Karmakar, Shubhojit Mallick, Manish Gupta, Shreya Ghosh, Abhik Jana. UTP-Bench: Uncertainty-aware Travel Planning Benchmark. EMNLP 2026.

• 1000 real-world queries: 3/5/7-day itineraries
• Human-annotated gold-standard plans
• 504 Indian cities, 33 states/UTs
• Empirical delay stats + crowd-density patterns
• 3 new uncertainty-aware metrics: BAS, CATS, TDAS

3

How was UTP-Bench curated?

• Web scraping + OpenStreetMap
• Transport, attractions, restaurants, accommodations, events
• Uncertainty signals from 20 major cities
• Historical delays: flights, trains, buses, cabs
• Hourly crowd density: attractions, restaurants
• Query construction with GPT-5 (few-shot)
• Sampling of departure city, destination, travel dates, budget, and persona profile.
• 3-day: 1 city; 5-day: 2 cities; 7-day: 3 cities in a state
• Hard constraints + persona + risk profile
• Easy / Medium / Hard

Etcharla Revanth Rao, Priyanshu Karmakar, Shubhojit Mallick, Manish Gupta, Shreya Ghosh, Abhik Jana. UTP-Bench: Uncertainty-aware Travel Planning Benchmark. EMNLP 2026.

4

Constraints, persona and annotation

• Constraints from TripCraft/TravelPlanner
• Commonsense + hard (budget, room rules, ...)
• Events cannot repeat across days
• Meals must maintain appropriate temporal gaps
• PoIs must follow a valid chronological order.
• New: uncertainty constraints
• Persona
• Traveler type: laid-back and adventure-oriented
• Purpose: relaxation, adventure, or cultural exploration
• Spending
• Location: beaches, mountains, heritage cities, wildlife destinations, or religious sites
• Risk profile
• Risk-Tolerant, Risk-Optimized, Risk-Averse
• 14 trained annotators
• Create gold std itinerary
• Helper scripts surface delays, crowds, visit durations
• Written rationale per itinerary: key planning decisions, the constraint-satisfaction logic, and the uncertainty-buffer choices

Etcharla Revanth Rao, Priyanshu Karmakar, Shubhojit Mallick, Manish Gupta, Shreya Ghosh, Abhik Jana. UTP-Bench: Uncertainty-aware Travel Planning Benchmark. EMNLP 2026.

• Multi-stage quality control
• Iterative refinement with domain experts
• Final manual review
• Automated structural validity checks

5

Uncertainty distribution

• Avg delay: trains 40.8–46.7 min, flights 40.59 min, road 52.2 min
• Mean > median: skewed, occasional large disruptions

Etcharla Revanth Rao, Priyanshu Karmakar, Shubhojit Mallick, Manish Gupta, Shreya Ghosh, Abhik Jana. UTP-Bench: Uncertainty-aware Travel Planning Benchmark. EMNLP 2026.

• Attractions: crowded during daytime/afternoon
• Restaurants: evening concentration

6

Evaluation Metrics

Etcharla Revanth Rao, Priyanshu Karmakar, Shubhojit Mallick, Manish Gupta, Shreya Ghosh, Abhik Jana. UTP-Bench: Uncertainty-aware Travel Planning Benchmark. EMNLP 2026.

7

Results

Etcharla Revanth Rao, Priyanshu Karmakar, Shubhojit Mallick, Manish Gupta, Shreya Ghosh, Abhik Jana. UTP-Bench: Uncertainty-aware Travel Planning Benchmark. EMNLP 2026.

8

Case study: Crowd-aware scheduling

• Good CATS score: the planner consistently schedules attraction visits during morning quiet windows, deliberately avoiding known peak crowd hours.
• LLM failure modes
• Temporal inconsistencies: PoI visits past departure, odd meal times, misordered timestamps
• Poor buffer adaptation across risk profiles
• Optimistic transit assumptions in multi-city trips
• fragile itineraries under realistic delays
• Harder with longer horizons
• Human plans also drop 91% → 63% (3 → 7-day)
• LLMs remain far below humans

Etcharla Revanth Rao, Priyanshu Karmakar, Shubhojit Mallick, Manish Gupta, Shreya Ghosh, Abhik Jana. UTP-Bench: Uncertainty-aware Travel Planning Benchmark. EMNLP 2026.

9

Summary

Etcharla Revanth Rao, Priyanshu Karmakar, Shubhojit Mallick, Manish Gupta, Shreya Ghosh, Abhik Jana. UTP-Bench: Uncertainty-aware Travel Planning Benchmark. EMNLP 2026.

10