xG Review | CBJHAC '20

2

Overview

xG Review | CBJHAC '20

3

Expected Goals: What is it?

What:

• Binary Classification problem
• Predict the probability that a shot will become a goal
• NHL’s RTSS (play-by-play) data

​

Use:

• Player & Team evaluation
• Predictive ability
• Skater defense (remove the goalie)
• Shooting ability
• Bedrock for numerous systems for player evaluation

xG Review | CBJHAC '20

4

A Brief History

Shot Quality Models

• 2004: Alan Ryder (Shot Quality)
• 2009-10: Ken Krzywicki (Observer bias, Shot angles and rebounds)
• 2011: Michael Schuckers (Goalie evaluation, Shot Quality)

​

​

​

Focus:

• Historic goal expectancy
• Evaluation of Shooting % from locations
• RTSS bias / adjustments
• Early team and player evaluation

Expected Goals

• 2012: Brian Macdonald (Expected Goals)
• 2015: Dawson Sprigings & Asmae Toumi
• 2016: Emmanuel Perry (Corsica)
• 2016: Peter Tanner (Money Puck)
• 2016+: Cole Anderson, Matt Barlowe
• 2018: Harry Shomer
• 2018: EvolvingWild

​

Focus:

• Statistical Modeling
• Machine learning / Optimization
• Feature selection
• Shooting/Goalie work
• Team/Player evaluation

​

xG Review | CBJHAC '20

5

The Data

NHL’s Real Time Scoring System (RTSS) Data

• All data acquired using the Evolving-Hockey scraper: github.com/evolvingwild/evolving-hockey
• Fenwick Shots (Goals, Missed Shots, Saved Shots)
• Shot location not available for blocked shots
• Other events:
• Blocked Shots, Takeaways, Giveaways, Faceoffs, Hits
• Shot attributes:
• Shot Type (Wrist, Slap, Snap, Wrap-Around, Tip-In, Deflection)

​

​

Cleaning/Modifying the Data

• Shots without coordinates removed
• Distance (feet from center of goal line) / Angle (degrees perpendicular to center Y line)
• Assume all shots taken below center line
• Recalculate if recorded zone is the defensive zone

xG Review | CBJHAC '20

6

Available Predictor Variables

From RTSS:

• Shot distance
• Shot angle
• Shot type (deflected/tip/slap/backhand/snap/wrap/wrist)
• Score state (score down / up around 0)
• Strength state (if applicable)
• Game time (period / seconds)
• Home/Away team
• Seconds since last event
• Distance from last event
• Prior event same team (all event types)
• Prior event opposing team (all event types)
• Coordinates (if applicable)
• Prior coordinates (if applicable)

Created:

• Rebound
• Any shot that occurred 2 seconds after a prior shot
• Rush
• Any shot that occurred 6 seconds after either:
• any event in the defensive zone or
• any giveaway or takeaway

xG Review | CBJHAC '20

7

Modeling

xG Review | CBJHAC '20

8

The Models

Baseline (Model 0)

• Logistic regression
• Shot distance only

​

Alan Ryder (Model 1)

• Logistic regression
• Shot distance
• Score state (down 3 through up 3)
• Game period / Game Seconds
• Shot type

​

glm Base (Model 2)

• Logistic regression
• [Prior features]
• Polynomial transformation for distance and angle
• Rush
• Rebound

glmnet (Model 3)

• Regularized logistic regression
• [Prior features]
• Prior events
• Prior teams

​

XGBoost (Model 4)

• Gradient boosting model
• [Prior features]
• Removed rush/rebound features
• Coordinates
• Prior coordinates

​

​

xG Review | CBJHAC '20

9

Evaluation

Timeframe

• 3 seasons, regular season only
• ‘15-16 / ’16-17 / ‘17-18
• 5v5
• 247,373 total Fenwick shots

​

Metrics

• Area Under Curve (ROC)
• Higher, better (0 - 1)
• Log loss
• Lower, better (0 - ∞)

​

Methods:

• In-sample Cross Validation
• Future seasons (‘18-19 / ‘19-20)

​

​

xG Review | CBJHAC '20

10

Shot Distance

AUC: 0.7294 | LL: 0.1954

Ryder Model

AUC: 0.7409 | LL: 0.1943

xG Review | CBJHAC '20

11

glm Base

AUC: 0.7669 | LL: 0.1878

glmnet

AUC: 0.7675 | LL: 0.1882

xG Review | CBJHAC '20

12

XGBoost 3 year

AUC: 0.778 | LL: 0.1847

CV Model Comparison

Future Season Comparison (‘18-20)

xG Review | CBJHAC '20

13

xG Review | CBJHAC '20

14

Comparison

xG Review | CBJHAC '20

15

Correlation Matrix: 5v5 xG Comparison, ‘18-20

xG Review | CBJHAC '20

16

Correlation Matrix - Skater On-Ice Measure:

xGF per 60 / xGA per 60, ‘18-20, Min. 200 Minutes

xG Review | CBJHAC '20

17

Correlation Matrix - Skater On-Ice Measure:

xGF per 100FF / xGA per 100FA, ‘18-20, Min. 200 Minutes

xG Review | CBJHAC '20

18

Team xGF%: ‘18-19, 5v5

xG Review | CBJHAC '20

19

Team xG +/- per Fenwick Shot: ‘18-19, 5v5

xG Review | CBJHAC '20

20

Magnitude: All Models

xG Review | CBJHAC '20

21

Calibration and Heatmaps

xG Review | CBJHAC '20

22

xG Review | CBJHAC '20

23

Models 0-1: Shot Distance / Ryder

xG Review | CBJHAC '20

24

Models 2-3: glm base / glmnet

xG Review | CBJHAC '20

25

Models 4+: XGBoost

xG Review | CBJHAC '20

26

Wrapping Up

xG Review | CBJHAC '20

27

Biases

1. Seconds since

2. Distance / Rink bias

3. Shots from behind the goal line

xG Review | CBJHAC '20

28

Takeaways

Objectives

• What is the purpose of your model?
• Interpretability / performance
• Predictive vs. Inferential
• Strength states

​

​

​

Data

• Timeframe
• Rolling / Multiple xG models
• Have you identified all biases?

Time / Performance Tradeoff

• Do you have a month?
• More complicated model is generally ”better”
• Less complicated model is still pretty good

​

​

​

​

Future

• Rink bias adjustment
• Player tracking

​

​

​

xG Review | CBJHAC '20

29

References

• “Shot Quality” - Alan Ryder (2004)

http://hockeyanalytics.com/Research_files/Shot_Quality.pdf

• “Removing Observer Bias from Shot Distance” - Ken Krzywicki (2009)

http://hockeyanalytics.com/Research_files/SQ-DistAdj-RS0809-Krzywicki.pdf

• “NHL Shot Quality 2009-10” - Ken Krzywicki (2010)

http://hockeyanalytics.com/Research_files/SQ-RS0910-Krzywicki.pdf

• “DIGR: A Defense Independent Rating of NHL Goaltenders using Spatially Smoothed Save Percentage Maps - Michael Schuckers (2011)

http://www.hockeyanalytics.com/Research_files/DIGR_Schuckers.pdf

• “A Look at Shot Quality” – Michael Schuckers (2011)

https://www.arcticicehockey.com/2011/6/15/2224920/a-look-at-shot-quality

• “An Expected Goals Model for Evaluating NHL Teams and Players” – Brian Macdonald (2012)

http://www.sloansportsconference.com/wp-content/uploads/2012/02/NHL-Expected-Goals-Brian-Macdonald.pdf

• “Expected Goals are a better predictor of future scoring than Corsi, Goal” – Dawson Sprigings & Asmae Toumi (2015)

https://hockey-graphs.com/2015/10/01/expected-goals-are-a-better-predictor-of-future-scoring-than-corsi-goals/

• “About” – Moneypuck/Peter Tanner

http://moneypuck.com/about.htm

• “NHL Expected Goals Model” – Matt Barlowe (2017)

https://rstudio-pubs-static.s3.amazonaws.com/311470_f6e88d4842da46e9941cc6547405a051.html

• “Expected Goals (xG), Uncertainty, and Bayesian Goalies” – Cole Anderson (2017)

http://www.crowdscoutsports.com/game-theory/expected-goal-xg-model/

• “Expected Goals Model” – Harry Shomer (2018)

http://fooledbygrittiness.blogspot.com/2018/01/expected-goals-model.html

• “Evaluating my Shooter xG model” – Harry Shomer (2018)

http://fooledbygrittiness.blogspot.com/2018/03/evaluating-my-shooter-xg-model.html

• “A New Expected Goal Model for Predicting Goals in the NHL” – Josh & Luke Younggren (2018)

https://rpubs.com/evolvingwild/395136/

​

Thank You!