1 of 31

Going Deep:

Models for Continuous-Time Within-Play Valuation of Game Outcomes in American Football with Tracking Data

Ronald Yurko¹, Francesca Matano¹, Lee F. Richardson¹, Nicholas Granered²,

Taylor Pospisil¹, Konstantinos Pelechrinis³, Samuel L. Ventura¹

September 28, 2019

¹Department of Statistics & Data Science, Carnegie Mellon University

²Department of Statistics, University of Pittsburgh

³School of Computing and Information, University of Pittsburgh

2 of 31

NESSIS 2017:

“Recent work in football analytics is not easily reproducible”

NESSIS 2019???

3 of 31

Thanks Max!

4 of 31

Play-by-play evaluation with nflscrapR

Expected points (EP): how many points have teams scored in similar situations?

Multinomial logistic regression

Win probability (WP): have teams in similar situations won the game?

Generalized additive model (GAM)�

Between play value: expected points added (EPA) / win probability added (WPA)

What about continuous, within play value?

5 of 31

Continuous-time valuation with player-tracking data

Cervone et. al. (2014, 2016), two-level Markov chain approach

Soccer extensions: Link et. al. (2016), Fernandez et. al (2019)

6 of 31

Enter the Big Data Bowl...

NFL collects tracking data at 10Hz with RFID chips in shoulder pads and ball

December 2018: NFL (Mike Lopez) released data from weeks 1-6 of 2017 season

Competition entries focus on receivers:

Chu et. al. (2019) - mixture models for identifying/characterizing routes
Deshpande and Evans (2019) - expected hypothetical completion probability

And more great entries! https://operations.nfl.com/the-game/big-data-bowl/

7 of 31

What does the data look like?

On-field (x, y), speed, and angle for each player (and ball) is recorded at rate of ten frames per second - 1,075,720 unique frames across 14,167 plays

NFL provides event annotations within plays (e.g. handoff, first contact, etc)

Example play: Cordarrelle Patterson’s 47 yard jet sweep TD run

8 of 31

Patterson’s 47 yard jet sweep TD run

10 of 31

Continuous-time play value framework

GOAL: For each play , model the end-of-play yard line

: time between start (i.e. snap) and end of play
: filtration of ball and player locations, trajectories, etc. until
: expected end-of-play yard line at time

General framework for any play?

11 of 31

Start

Run

Ball-carrier model

QB Decision Model

End

Pass

Global Catch Prob Model

Target Prob Model

Individual Catch Prob Model

Start, end, or play type

Model

Model outcome

Scramble or sack

Throw away

Catch (includes INT)

No Catch

Dropback

Predict

12 of 31

Ball-carrier model

We model the yards gained from the current position on the field at :

Use the fact that [player’s current yard line]

Then by linearity of expectations,

[player’s current yard line]

13 of 31

Ball-carrier model features

Split players into 3 groups: ball-carrier, offense, and defense:

Use (x, y), speed, direction, and distance traveled from previous frame

Order offense/defense players using distance to ball-carrier

e.g. defense2_x gives x coordinate for second closest defender

Voronoi tessellations summary:

Ball carrier’s area
x-coordinate of closest and farthest points from target endzone
Indicator if surrounded by teammates

14 of 31

Ball-carrier model choices and qualities

High-dimensions
Non-linearity
Interactions
Time

Intercept-only	LASSO	XGBoost	FNN	LSTM
	✔️	✔️	✔️	✔️
		✔️	✔️	✔️
		✔️	✔️	✔️
				✔️

Gradient boosted trees (XGBoost library)

Feedforward neural network (FNN) and long short-term memory (LSTM) network

15 of 31

Model validation

Leave-one-week-out (LOWO) cross-validation

i.e. train on weeks 1-5, get frame predictions for week 6

Criteria for hold-out predictions:

Overall holdout root mean-squared error (RMSE)
Error across ball-carrier sequence

16 of 31

LSTM displays best LOWO CV results

RMSE = 7.67

RMSE = 6.24

RMSE = 5.86

RMSE = 5.52

RMSE = 6.11

17 of 31

LSTM makes the smallest long-term errors

18 of 31

Example play: Patterson’s 47 yard jet sweep TD run

19 of 31

Can see clear impact of covariates on predicted end-of-play yard line

20 of 31

Can evaluate teams and players with respect to expectation at key moments in play

22 of 31

Compute continuous-time play value

Given our prediction for the end-of-play yard line we proceed to update:

, , etc. as input for EP and WP models

Generate point estimate for using nflscrapR multinomial logit model

Similarly for with GAM

(for now we use observed change in )

23 of 31

Generate point-estimates for both EP and WP using nflscrapR models

Evaluate player movements using EP/WP within a continuous framework

24 of 31

A note of caution...

This is �NOT the expectation of EP or WP!

26 of 31

Conditional density estimation (CDE) can be used for estimating the density curve for

Pospisil and Lee (2018) developed flexible methodology for CDE, e.g. RFCDE above

27 of 31

Recap and future work

General modular framework for continuous-time valuation in football

Use existing approaches: DeepQB (Burke, 2019), EHCP (Deshpande and Evans, 2019)

LSTM model of expected end-of-play yard line for running plays

Conditional density estimation for continuous expectation of EP/WP

Use state-of-the-art approaches: RFCDE, DeepCDE (Pospisil and Lee, 2018)

Implement the full framework and explore better features!
Player evaluation at the continuous-time level

28 of 31

Thank you Mike Lopez

More data please?

Photo credit: Gregory Matthews

29 of 31

Thank you and join us at #CMSAC19 Nov 1st-2nd!

Francesca Matano

Lee Richardson

Nick Granered

Taylor Pospisil

Kostas Pelechrinis

Sam Ventura

30 of 31

References I

Yurko, R., Ventura, S. & Horowitz, M. (2019). nflWAR: a reproducible method for offensive player evaluation in football. Journal of Quantitative Analysis in Sports, 15(3), pp. 163-183.

Pospisil, T. and Lee, A. (2018). RFCDE: Random forests for conditional density estimation. URL: https://arxiv.org/abs/1804.05753.

Cervone, D., D’Amour, A., Bornn, L. & Goldsberry, K. (2016): A multiresolution stochastic process model for predicting basketball possession outcomes. Journal of the American Statistical Association, 111, pp. 585–599

31 of 31

References II

Cervone, D., D’Amour, A., Bornn, L. & Goldsberry, K. (2014). Point-wise: Predicting points and valuing decisions in real time with nba optical tracking data. MIT Sloan Sports Analytics Conference.

Link, D., Lang, S. & Seidenschwarz, P. (2016). Real time quantification of dangerousity in football using spatiotemporal tracking data. PLoS ONE, 11

Fernandez, J., Bornn, L. & Cervone, D. (2019). Decomposing the immeasurable sport: A deep learning expected possession value frame-work for soccer. MIT Sloan Sports Analytics Conference.