1 of 15

A System for Triggering �Sports Instant Answers on Search Engines�

Ankith Karat�ankar@microsoft.com

Atishay Tibrewal

atishay.tibrewal@microsoft.com

​

Aditya Mehndiratta

adityam0309@gmail.com

​

Ravindra Valluri

raravin@microsoft.com

​

Ankit Kumar

kumarank@microsoft.com

​

Nishka Kotian

nishkakotian@microsoft.com

​

Antony Ravi Teja Marineni

amarineni@microsoft.com

​

Manan Dang

mandang@microsoft.com

Rhea Sundaresan

rsundaresan@microsoft.com

Sunil Shah

shah.sunil@microsoft.com

​

Subhasis Panigrahi

supanigr@microsoft.com

​

Chandra Bhushan

chabh@microsoft.com

​

Sarthak Sahni

sarthak.sahni@microsoft.com

Arun D Poondi

arun.poondi@microsoft.com

Manish Gupta

gmanish@microsoft.com

​

2 of 15

Sports Instant Answers on

3 of 15

Soccer

​

​

Football

​

​

Basketball

​

​

Baseball

​

​

Ice Hockey

​

​

Rugby

​

​

Cricket

​

​

Motorsports

​

​

Golf

​

​

Tennis

​

Team Sports

Individual Sports

Sports Instant Answers on

4 of 15

• Scores • Schedule • Standings

• Players • Stats • Rankings • Game Stats • Highlights

Sports Instant Answers on

5 of 15

Challenges In Sports (Beyond live & updated scores)

1. Intent changes based on DateTime for the query

    • Super bowl, March madness, F1 race​
    • Wimbledon​

​

​

​

​

2. Intent changes based on Season of Sport and Location

      • Giants score New York Giants (NFL) San Francisco Giants (MLB)
      • Kings match Sacramento Kings (NBA) Los Angelos Kings (NHL)
    • Baseball MLB (US) NPB (Japan)
    • Ind vs Wi Indiana Hosiers vs Wisconsin Badgers (College Basketball-US) India vs West Indies (AUS, ENG, India etc - Cricket) 

​

​

3. Sports Fans have a separate Vocab - lot of Short / Nick names (ALTs)

    • Red Devil Manchester United (UK) Belgian national football team (BEL)
    • The King LeBron James (Basketball) Jerry Lawler (Wrestling)
    • Black Mamba Kobe Bryant (Basketball)
    • The Phenomenon Ronaldo (Brazilian-Soccer)

​

4. Sponsor names that change by season

    • Fedex cup , Sky Bet championship, G League

Wimbledon

(Jul 1 – July 14)

Wiki,

News Stories, Tickets etc

Live scores,

Live Streaming

News, Wiki, Stories etc

- 2 weeks

+ 2 days

Schedule, Fixtures

Game Results,

Video Highlights

6 of 15

Sports Triggering Stack��Principles�- Consolidate knowledge�- Leverage signal throughout stack�- Invest in Ranking�- “Plug and play” modular design�

7 of 15

Sports Triggering Stack��Principles�- Consolidate knowledge�- Leverage signal throughout stack�- Invest in Ranking�- “Plug and play” modular design�

Preweb Ranking

Region based

Popularity Based

Schedule Based

Postweb Ranking

Sentence Ranker

Template Ranker

Query Understanding

Block / Allow

Graph Stack

ML Stack

Bing KC

NRT Stack

Answer Lookup

Arena Workflow 1

Arena Workflow 2

Arena Workflow 3

Answer Selector

Triggering Stack

Lookup Stack

Sports Graph

NRT Correction pipelines

8 of 15

Sports Triggering Stack��Principles�- Consolidate knowledge�- Leverage signal throughout stack�- Invest in Ranking�- “Plug and play” modular design�

Preweb Ranking

Region based

Popularity Based

Schedule Based

Postweb Ranking

Sentence Ranker

Template Ranker

Query Understanding

Block / Allow

Graph Stack

ML Stack

Satori

NRT Stack

Answer Lookup

Arena Workflow 1

Arena Workflow 2

Arena Workflow 3

Answer Selector

9 of 15

Sports Triggering Stack��Principles�- Consolidate knowledge�- Leverage signal throughout stack�- Invest in Ranking�- “Plug and play” modular design�

Preweb Ranking

Region based

Popularity Based

Schedule Based

Postweb Ranking

Sentence Ranker

Template Ranker

Query Understanding

Block / Allow

Graph Stack

ML Stack

Bing KC

NRT Stack

Answer Lookup

Arena Workflow 1

Arena Workflow 2

Arena Workflow 3

Answer Selector

Triggering Stack

Sports Graph

NRT Correction pipelines

Ind vs wi

Ind – cricket.team, basketball.team, hockey.team, soccer.team

vs – vs intent

wi – cricket.team, basketball.team

Ind – cricket.team, basketball.team,

vs – vs intent

wi – cricket.team, basketball.team

Cricket :

Matchups : 09/2023, 06/2024

Basketball

Matchups : 09/2024

Ongoing

​

User/Location Popularity

IN : Cricket

Personalization : None

1. Ind (cricket.team) vs wi (cricket.team)

2. Ind (basketball.team) vs wi (basketball.team)

1. Cricket answer

2. Basketball answer

1. Cricket answer

2. Basketball answer (low score)

Cricket answer

Lookup Stack

10 of 15

Sports Triggering Stack��Principles�- Consolidate knowledge�- Leverage signal throughout stack�- Invest in Ranking�- “Plug and play” modular design�

  • Alternative names, Relations, Popularity signals etc
  • Online stack refreshes daily

Preweb Ranking

Region based

Popularity Based

Schedule Based

Postweb Ranking

Sentence Ranker

Template Ranker

Query Understanding

Block / Allow

Graph Stack

ML Stack

Bing KC

NRT Stack

Answer Lookup

Arena Workflow 1

Arena Workflow 2

Arena Workflow 3

Answer Selector

Triggering Stack

Sports Graph

NRT Correction pipelines

Corrections in Precision & Recall improvements automatically

Schedule based answer curation

Region & Popularity based ranking

Graph & ML based understanding

Cohesive with Web results

Lookup Stack

11 of 15

Sports Triggering Stack��Principles�- Consolidate knowledge�- Leverage signal throughout stack�- Invest in Ranking�- “Plug and play” modular design�

Preweb Ranking

Region based

Popularity Based

Schedule Based

Postweb Ranking

Sentence Ranker

Template Ranker

Query Understanding

Block / Allow

Graph Stack

ML Stack

Bing KC

NRT Stack

Answer Lookup

Arena Workflow 1

Arena Workflow 2

Arena Workflow 3

Answer Selector

Triggering Stack

Sports Graph

NRT Correction pipelines

Lookup Stack

12 of 15

Sports Triggering Stack

Preweb Ranking

Region based

Popularity Based

Schedule Based

Postweb Ranking

Sentence Ranker

Template Ranker

Query Understanding

Block / Allow

Graph Tagger

ML

Tagger

PreWeb Bing KC

NRT Stack

Answer Lookup

Arena Workflow 1

Arena Workflow 2

Arena Workflow 3

Answer Selector

Triggering Stack

Sports Graph

NRT Correction pipelines

Slot Resolution

Slot Resolution

Sid to

Tid

Graph Intent

ML

Intent

ML

Intent

Graph Stack

ML Stack

Satori

NRT

Block/Allow

Tagger

Resolution

Intent

Lookup Stack

13 of 15

Sports Triggering Stack

Preweb Ranking

Region based

Popularity Based

Schedule Based

Postweb Ranking

Sentence Ranker

Template Ranker

Query Understanding

Block / Allow

Graph Tagger

ML

Tagger

PreWeb Bing KC

NRT Stack

Answer Lookup

Arena Workflow 1

Arena Workflow 2

Arena Workflow 3

Answer Selector

Triggering Stack

Sports Graph

NRT Correction pipelines

Slot Resolution

Slot Resolution

Sid to

Tid

Graph Intent

ML

Intent

ML

Intent

Graph Stack

ML Stack

Satori

NRT

Block/Allow

Tagger

Resolution

Intent

Lookup Stack

* ML Models

14 of 15

  1. Evaluation Set
    • 80,000 Bing queries from May 2024
    • Distributed across sports, languages, regions, intents and traffic buckets (head/torso/tail).

​

​

​

​

  • Recall Set (FRESH PAST SET)
    • 10,000 queries from rolling 7 days
    • Mix of Bing + Competition queries
    • Daily set refresh & measurement
    • Algorithm in place to ensure representation across sports, intents and languages

​

​

B: 66%

Comp: 70%

Metrics

 Component

Precision

Recall

Domain Classifier

80.63

99.15

NER

77.55

68.5

Intent Classifier

78.55

73.39

Allowlist-based QU

100

0.02

Answers lookup

77.72

72.12

Post-web suppression and reranking

88.25

92.93

15 of 15

Thank you