TREC-ToT 2024
Session
Jaime Arguello
UNC
Sam Bhargav
UvA
Fernando Diaz
CMU
Bhaskar Mitra�Microsoft
Evangelos Kanoulas
UvA
Danny To Eun Kim
CMU
Yifan He
CMU
Today’s Schedule
2
Task Overview
Tip-of-the-Tongue (ToT) Known-Item Retrieval
4
Tip-of-the-Tongue (ToT) Known-Item Retrieval
1990
time
?
2024
5
MOTIVATION
Tip-of-the-Tongue (ToT) Known-Item Retrieval
6
TREC 2023: MOVIE IDENTIFICATION
Tip-of-the-Tongue (ToT) Known-Item Retrieval
1990
time
?
2024
8
…
question
Q&A, potential answers
confirmation!
correct answer
…
…
input
output
System
TREC 2024
Tip-of-the-Tongue (ToT) Known-Item Retrieval
movies
celebrities
landmarks
11
What do ToT queries look like?
What do ToT Queries Look Like?
(1) Verbose, but also…
Ok so I don’t really remember anything but one scene, so I will try to give as much detail as I can. I saw this movie when I was very young and only remember this scene because (from what I remember) I don’t think I actually watched most of the movie because I found it pretty scary, so it is probably not a children/family movie. I think I watched it in 2006 (mid-end of the year probably) and it was on a tv in someone’s house, so it was old enough to be released on tv/dvd (not still in cinemas). It was in English and colour I’m pretty sure. So this scene: There was a (or multiple) giant robot-like things and I think they were sort of sphere shaped. It was destroying a city and going around picking people up (possible killing them?) in giant net-like things i think. There was a father and daughter (i think it was a daughter) and they ended up getting separated because the father got picked up by the robot thing. When the robot thing was picking people up there was a lot of red liquid stuff (quite possibly blood, but maybe something else?). In the end I think the father made it back to the daughter. And that is all I remember, sorry if it is vague, but really hope someone can help.
13
What do ToT Queries Look Like?
(2) Social niceties
Ok so I don’t really remember anything but one scene, so I will try to give as much detail as I can. I saw this movie when I was very young and only remember this scene because (from what I remember) I don’t think I actually watched most of the movie because I found it pretty scary, so it is probably not a children/family movie. I think I watched it in 2006 (mid-end of the year probably) and it was on a tv in someone’s house, so it was old enough to be released on tv/dvd (not still in cinemas). It was in English and colour I’m pretty sure. So this scene: There was a (or multiple) giant robot-like things and I think they were sort of sphere shaped. It was destroying a city and going around picking people up (possible killing them?) in giant net-like things i think. There was a father and daughter (i think it was a daughter) and they ended up getting separated because the father got picked up by the robot thing. When the robot thing was picking people up there was a lot of red liquid stuff (quite possibly blood, but maybe something else?). In the end I think the father made it back to the daughter. And that is all I remember, sorry if it is vague, but really hope someone can help.
14
What do ToT Queries Look Like?
(3) Semantic memories about: metadata
Ok so I don’t really remember anything but one scene, so I will try to give as much detail as I can. I saw this movie when I was very young and only remember this scene because (from what I remember) I don’t think I actually watched most of the movie because I found it pretty scary, so it is probably not a children/family movie. I think I watched it in 2006 (mid-end of the year probably) and it was on a tv in someone’s house, so it was old enough to be released on tv/dvd (not still in cinemas). It was in English and colour I’m pretty sure. So this scene: There was a (or multiple) giant robot-like things and I think they were sort of sphere shaped. It was destroying a city and going around picking people up (possible killing them?) in giant net-like things i think. There was a father and daughter (i think it was a daughter) and they ended up getting separated because the father got picked up by the robot thing. When the robot thing was picking people up there was a lot of red liquid stuff (quite possibly blood, but maybe something else?). In the end I think the father made it back to the daughter. And that is all I remember, sorry if it is vague, but really hope someone can help.
15
What do ToT Queries Look Like?
(4) Semantic memories about: scenes
Ok so I don’t really remember anything but one scene, so I will try to give as much detail as I can. I saw this movie when I was very young and only remember this scene because (from what I remember) I don’t think I actually watched most of the movie because I found it pretty scary, so it is probably not a children/family movie. I think I watched it in 2006 (mid-end of the year probably) and it was on a tv in someone’s house, so it was old enough to be released on tv/dvd (not still in cinemas). It was in English and colour I’m pretty sure. So this scene: There was a (or multiple) giant robot-like things and I think they were sort of sphere shaped. It was destroying a city and going around picking people up (possible killing them?) in giant net-like things i think. There was a father and daughter (i think it was a daughter) and they ended up getting separated because the father got picked up by the robot thing. When the robot thing was picking people up there was a lot of red liquid stuff (quite possibly blood, but maybe something else?). In the end I think the father made it back to the daughter. And that is all I remember, sorry if it is vague, but really hope someone can help.
16
What do ToT Queries Look Like?
(5) Semantic memories about: plot
Ok so I don’t really remember anything but one scene, so I will try to give as much detail as I can. I saw this movie when I was very young and only remember this scene because (from what I remember) I don’t think I actually watched most of the movie because I found it pretty scary, so it is probably not a children/family movie. I think I watched it in 2006 (mid-end of the year probably) and it was on a tv in someone’s house, so it was old enough to be released on tv/dvd (not still in cinemas). It was in English and colour I’m pretty sure. So this scene: There was a (or multiple) giant robot-like things and I think they were sort of sphere shaped. It was destroying a city and going around picking people up (possible killing them?) in giant net-like things i think. There was a father and daughter (i think it was a daughter) and they ended up getting separated because the father got picked up by the robot thing. When the robot thing was picking people up there was a lot of red liquid stuff (quite possibly blood, but maybe something else?). In the end I think the father made it back to the daughter. And that is all I remember, sorry if it is vague, but really hope someone can help.
17
What do ToT Queries Look Like?
(6) Episodic (i.e., contextual) memories
Ok so I don’t really remember anything but one scene, so I will try to give as much detail as I can. I saw this movie when I was very young and only remember this scene because (from what I remember) I don’t think I actually watched most of the movie because I found it pretty scary, so it is probably not a children/family movie. I think I watched it in 2006 (mid-end of the year probably) and it was on a tv in someone’s house, so it was old enough to be released on tv/dvd (not still in cinemas). It was in English and colour I’m pretty sure. So this scene: There was a (or multiple) giant robot-like things and I think they were sort of sphere shaped. It was destroying a city and going around picking people up (possible killing them?) in giant net-like things i think. There was a father and daughter (i think it was a daughter) and they ended up getting separated because the father got picked up by the robot thing. When the robot thing was picking people up there was a lot of red liquid stuff (quite possibly blood, but maybe something else?). In the end I think the father made it back to the daughter. And that is all I remember, sorry if it is vague, but really hope someone can help.
18
What do ToT Queries Look Like?
(7) Mentions of (un)certainty
Ok so I don’t really remember anything but one scene, so I will try to give as much detail as I can. I saw this movie when I was very young and only remember this scene because (from what I remember) I don’t think I actually watched most of the movie because I found it pretty scary, so it is probably not a children/family movie. I think I watched it in 2006 (mid-end of the year probably) and it was on a tv in someone’s house, so it was old enough to be released on tv/dvd (not still in cinemas). It was in English and colour I’m pretty sure. So this scene: There was a (or multiple) giant robot-like things and I think they were sort of sphere shaped. It was destroying a city and going around picking people up (possible killing them?) in giant net-like things i think. There was a father and daughter (i think it was a daughter) and they ended up getting separated because the father got picked up by the robot thing. When the robot thing was picking people up there was a lot of red liquid stuff (quite possibly blood, but maybe something else?). In the end I think the father made it back to the daughter. And that is all I remember, sorry if it is vague, but really hope someone can help.
19
What do ToT Queries Look Like?
(8) Exclusion criteria (i.e., previous search attempts)
A girl is kidnapped. Her somewhat senile grandfather writes to various “adventurers” from books like “Mack Bolan” and “The Executioner” thinking them to be real. Surprisingly, one responds and comes to search for the girl; consisting of the rugged adventurer and his partner who will later write a book on the adventure. The girl’s sister goes along on the search. Comedy ensues. Think this was back in the 90s. I thought it was Bill Pullman but nothing like this is listed in his credentials.
20
What do ToT Queries Look Like?
(9) Relative comparisons (multi-hop reasoning)
This is a movie I barely remember watching when I was visiting my dad. I believe it was a movie made for TV. I was about 7 or 8 years old at the time. I think it was made in the 90’s and it was in color. Even though the movie took place in India or some place like that, it was in English. The only scene I remember vividly is a scene where some guy wearing a turban says a prayer to a gold statue of a giant cobra. The statue comes to life and says something intimidating. The turban guy’s bodyguard (a bald guy wearing an eye patch) shoots at the gold cobra with a rifle, but gets eaten. The Cobra then turns the turban guy into a pig. The other thing I remember about this is the main character. He looked similar to Tarzan, but he wore pants and had some kind of weapon strapped across his back.
21
What do ToT Queries Look Like?
(10) False memories
There was this movie, and unfortunately I remember very little about it. I remember seeing it at around 2001-2003-ish, so it should be a film that was released before that time. It was definitely in English; at least I think so. Also, it was in colour, if that helps. On the VCR cover, all that I remember is a guys face, and blue background, probably the ocean? The only things that I remember are that there was something to do with water, and by this, I mean like an ocean, or maybe it was something like a submarine or an oil rig, and I think that the main character was a guy, who was either a ghost, or a silvery liquid metally alien/liquidy thing, that looks a bit like the T-1000 in Terminator 2. Sorry for the lack of detail, and please help me remember this!
22
Task
TASK DEFINITION & TRAINING MATERIALS
23
Task
EVALUATION
24
Synthetic Query Generation
25
What do (Synthetic) ToT Queries Look Like?
MOVIES
I'm trying to recall this one film I watched ages ago. It was set in this vast, open landscape, probably somewhere in the American frontier. The main character was this soldier guy who ends up at a really rundown fort. He was supposed to be there alone, I think, and he starts fixing the place up. There was this really cool part where he befriends a wolf. I remember the wolf because it was such a touching moment, and it felt like the wolf was his only friend for a while. The soldier guy also starts interacting with a nearby Native American tribe. At first, there's a lot of tension, but over time, he learns their language and customs. There's this wise elder and a fierce warrior who stand out in my memory. I also recall a romantic subplot with a woman who was somehow connected to the tribe, maybe adopted or something. The whole movie had this epic feel, with sweeping shots of the plains and intense moments like a buffalo hunt. Does this ring any bells for anyone? It's been bugging me for days!
26
What do (Synthetic) ToT Queries Look Like?
CELEBRITIES
There's this actor I keep thinking about, and it's driving me nuts that I can't remember his name. He was pretty big in the 80s and 90s, often playing these quirky, everyman roles. I think he was in some teen movies back in the day, you know, those classic high school flicks where the guy is kind of an underdog but super charming in his own awkward way. One of those movies had him holding a boombox over his head in a really iconic scene, if that rings any bells. He also did some more serious stuff later on, like this one film where he was a hitman going back to his high school reunion. That one was a mix of dark comedy and action, and it was pretty unique for its time. I remember he was also in a movie with a lot of famous faces, something about a plane full of convicts, and he played a good guy trying to sort out the mess. His family is kind of a big deal too, I think. He has a sister who's also an actress, and they might have even been in a movie together at some point. Their last name is pretty distinctive, but it's just not coming to me right now. He's also known for being quite vocal about his political views. Does anyone know who I'm talking about?
27
What do (Synthetic) ToT Queries Look Like?
LANDMARKS
So, there's this place in New York City that I can't quite put my finger on. It's one of those grand, old buildings that feels like it's been around forever, you know? I remember going there a few years ago for a concert, and the whole experience was just magical. The venue itself is this massive, elegant structure with a kind of old-world charm. It's got these intricate designs on the walls and ceilings, almost like something you'd see in an old European opera house. The main auditorium is huge, with multiple levels of seating that seem to go on forever. I think it was five stories high, but my memory might be playing tricks on me. The acoustics were incredible; you could hear every note perfectly, no matter where you were sitting. I remember the seats being this rich, deep red color, adding to the overall luxurious feel of the place. There are also a couple of smaller halls within the same building. One of them is more intimate, with fewer seats, and I think it's used for smaller performances or recitals. I was with a group of friends, and we spent some time exploring the area around the venue before the concert. It's in a pretty bustling part of the city, with lots of restaurants and shops nearby. We grabbed a bite to eat at a cozy little cafe just a block or two away, which made the whole evening feel even more special. Does anyone know the name of this place? It's driving me crazy trying to remember! Any help would be much appreciated.
28
Baselines
29
Highlights: Results
Results
This year: 16 runs from 5 groups
Last Year: 30 runs from 10 groups
31
Results
Groups presenting today
32
Results
BM25: relevant item ranked in top-1000 only half the time.
33
System Performance Across Query Types
NDCG@1000 was slightly higher for synthetic queries
34
System Performance Across Query Types
System rankings (mostly) similar across query types
Kendall’s Tau between .74 & .91
Lowest: MS-ToT movies vs. Synthetic Landmarks
Highest: Synthetic Celebrities vs. Synthetic Movies
35
Summary
overview paper
36
TREC-ToT 2025
Planning Session
Jaime Arguello
UNC
Fernando Diaz
CMU
Bhaskar Mitra�Microsoft
Danny To Eun Kim
CMU
Plan for TREC-ToT 2025
38
New NIST ToT Queries
39
NIST ToT Query Elicitation
Recognize Image?
Start
Know Name?
Produce ToT Query
Is this the Wikipedia page?
Valid ToT Query
No
Yes
No
Yes
No
Yes
Yarmey, A.D. I recognize your face but I can’t remember your name: Further evidence on the tip-of-the-tongue phenomenon. Memory & Cognition 1, 287–290 (1973).
40
NIST ToT Query Examples
Recognize Image?
Start
Know Name?
Produce ToT Query
Is this the Wikipedia page?
Valid ToT Query
No
Yes
No
Yes
No
Yes
This is the movie where Burt Reynolds and a few of his friends go into the wood for a good time only they are encounter by a group of backwoods people that hold them hostage. The men are abused but at the end of the movie they get loose of the backwoods people. I think at least one of them dies.
41
NIST ToT Query Examples
Recognize Image?
Start
Know Name?
Produce ToT Query
Is this the Wikipedia page?
Valid ToT Query
No
Yes
No
Yes
No
Yes
I remember this person was a star basketball player. During his career he and another player named Larry Byrd were always in heated competition for the spotlight. His career was cut short by a serious illness. However, he became very successful in the business world including minority owner of at least one sports team. He was born in the late 50s and [is] married with children.
42
NIST ToT Query Examples
Recognize Image?
Start
Know Name?
Produce ToT Query
Is this the Wikipedia page?
Valid ToT Query
No
Yes
No
Yes
No
Yes
I believe this landmark is in New York City. I don't specifically know the name but I believe I saw it when we had a family trip in February 2023. We were walking to see the Trade towers and saw it on our way there while walking in the street. I remember getting impressed by how tall the buildings in New York City are, which are unlike the buildings in Virginia where I reside. It was winter time and walking in the streets of New York City on a sunny day was amazing. I think there is a glass building close to that one.
43
NIST ToT (Misremembered) Query Examples
Recognize Image?
Start
Know Name?
Produce ToT Query
Is this the Wikipedia page?
Valid ToT Query
No
Yes
No
Yes
No
Yes
Animated movie where multiple animals escape a zoo i think and then head off on their own for an escapade. its light hearted and funny but has a message, which i don't remember. the title of the movie may have zoo in it or something suggesting where they lived or that they made a breakaway escape
44
NIST ToT (Misremembered) Query Examples
Recognize Image?
Start
Know Name?
Produce ToT Query
Is this the Wikipedia page?
No
Yes
No
Yes
No
Yes
This bass guitar player may be the man who had a young son who died when the son climbed or fell over a balcony. It may be eric clapton. he is famous for his skilled smooth style. he should be around 70 i think and in this picture he doesn't look quite that old. so while he looks familiar, i may have the wrong guy in mind.
Valid ToT Query
Yes
45
NIST ToT (Misremembered) Query Examples
Recognize Image?
Start
Know Name?
Produce ToT Query
Is this the Wikipedia page?
No
Yes
No
Yes
No
Yes
I think this is a statue somewhere in Sicily. It is about the ships that come in the ports of Sicily. It represents the folklore of how the mermaids helped the ships navigate around. I think this is up north of the island of Sicily that was one of the areas where the ships would stop on their way to Italy.
Valid ToT Query
Yes
46
NIST ToT Query Examples
This celebrity is an American media personality, businesswoman, socialite actress. Her first claim to fame was through a reality show called "The Simple Life". As her name indicates she is a member of the Hilton hotel group. Her parents sent her to a boarding school in Utah where she allegedly claimed she endured abuse. She released a documentary detailing her claims.
47
NIST ToT Query Examples
I am thinking of an african american actor who played in a tv show in the 1970's maybe with a group of other guys referred to as the A Team. he always wore bling, especially a big gold toned necklace hanging to his waist it seemed. this actor was very bulked up, muscular from workouts no doubt.
48
NIST ToT Query Examples
This was a gangster movie starring John Travolta and Samuel Jackson. The movie possibly was made in the 1990s. I believe Uma Thurman was also in it.
49
Plan for TREC-ToT 2025
OPEN DISCUSSION
50