​

Less Is More: Making Better Picks with Less Scouting Data and Better Analytics

Brian Maher (he/they)

Mentor, FRC 333 & FRC 7407

Outline

• Before scouting: know how to win
• Scouting best practices
• How to decide what data to collect
• How to visualize/analyze your data

What is scouting?

• Collecting data at competitions to use for making strategic decisions
• Collected by:
• Watching matches
• Talking to teams/looking at robots in pits
• Useful for:
• Preparing for alliance selection
• Planning match strategy
• Need tools for collecting, storing, and visualizing data

Have a mental model of how to win

• A good mental model of how to win is the foundation of any effective scouting system
• Take time to understand the game and how to win
• You should already be doing this to build your robot
• Develop understanding of
• How to win matches
• What winning alliances might look like
• What your ideal playoff alliance will look like

Building your mental model

• Read rules inside and out
• Guesstimate how much good robots/alliances will be able to score
• Good rule of thumb, in a typical year a ___ team can do ____ full-field cycles
• Average, 2-4
• Good, 5-6
• Elite, 7-8
• Consider impact of non-scoring actions like defense
• Human games
• Watch week 0/1 events
• See what people are saying online (Chief Delphi/Discord)

​

Three Rs of good scouting systems

• Relevant
• Reliable
• Reasonable

Reliable

• Every match you intend to scout is actually scouted
• There are no technical failures that cause missing/lost data
• You never know which matches will end up being important data points
• Few things worse than picking a team that is broken and not knowing they’re broken
• Might miss the diamond-in-the-rough team you need to win
• Teach your scouts how to use your scouting sheet/system BEFORE the competition
• Practice makes perfect

Relevant

• Data collected should be useful for at least one of
• Planning for qualification matches
• Planning for alliance selection
• Making a picklist
• Preparing to accept/decline captains
• Planning for playoff matches

Reasonable

• The data you collect during matches needs to be reasonable for one person to accurately collect
• The more someone has to keep track of, the harder it will be for them to keep accurate track
• Working memory can only process 3-4 pieces of new information at a time
• Test out your system before the competition, use it yourself with match videos
• If the person making it can’t use it easily, others won’t be able to either!

Pareto Principle (“80/20 Rule”)

“For many outcomes, roughly 80% of consequences come from 20% of causes” - Wikipedia

What does this mean for scouting?

• Diminishing returns are very real
• Some of the easiest data to collect is the most useful
• Collecting more data than that can make your data less accurate
• Focus on collecting easy data and doing the best possible analysis with it

What data to collect

• Ask yourself, how does collecting this data align with my mental model of winning?
• CART
• Compatibility
• Ability
• Reliability
• Trend

What data to collect

• Since your goal is to score more than your opponents, the most important data to collect is generally how much robots score
• Count each time a scoring objective occurs
• Very easy most years (except for games with many game pieces, eg 2017 fuel)
• Easy to analyze!
• Counting misses puts scoring in context
• If a team makes 50% of their shots, if they can fix their accuracy, they could potentially score more

Other stuff can be useful too

• Stay focused on what’s important (CART)
• Reliability: tracking robots breaking/dying can be useful, but leave the details for notes
• Compatibility: e.g. if you have a specific autonomous mode that only works from a certain position, keeping track of starting positions might be useful
• In a defense heavy game, defense skill ratings can help

Notes

• A notes section provides a good a space for scouts to fill in the gaps
• Make sure you look at them when analyzing
• Ask yourself, do I need this data in numeric form or can it just be a note?
• Do you need to be able to chart/plot this data? Or sort teams by it? Or is seeing it in notes enough?
• Do you need an actual number, or are vibes enough?

​

What did this look like in 2022?

• Auto
• High goals made/missed
• Low goals made/missed
• If you have a 3+ ball auto: side of field
• Teleop
• High goals made/missed
• Low goals made/missed
• Endgame
• Highest climb attempted
• Highest climb succeeded
• Other
• Notes section
• Did they break? Yes/No or Briefly/Some of match/Most of match
• If you’re willing to put in work to teach scouts: defense skill rating

Good notes

• Good notes add new information that can’t be gathered from numerical data
• Good things to write about
• Driver skill
• Defense skill
• Tippiness
• Difficulties intaking/scoring when X
• Bad things to write about
• “Likes shooting high goals”
• Will be obvious from numbers
• Take the time to talk to your team about what makes good vs bad notes
• Great chance for scouts to add their intuition and creativity

​

Pit scouting

• Easier than match scouting
• Good for information that is hard to collect from watching matches
• Easy to go overboard
• Stick to information that you’ll actually use
• I’ve tried pit scouting lots of things, 90% of the time I just use drivetrain info

Goals for data analytics

• Form
• Be easy to use
• Make it easy to see numbers
• Provide useful graphs/visuals for understanding
• Function
• Discoverability: be able to find teams that meeting your goals
• Be able to do a deep dive into an individual team’s performance
• Be able to compare teams
• Be able to prepare for matches

Tools for analysis

• Averages
• Good for understanding reliability/consistency
• DON’T RELY ONLY ON AVERAGES!!
• Maximums
• Good for understanding ceiling
• Line graphs
• Good for understanding trends
• Tables
• Good for showing lots of information compactly
• Conditional formatting
• Good for making trends more obvious in numerical data

Team list

• List all the teams at the event, and various stats
• Allows sorting by different stats
• Good for answering questions “who is good at X?”

​

Team lookup

• Provides a detailed look at a team’s performance
• Meant to be able to answer any question about a team’s data

Comparison

• Useful for making picklist decisions
• e.g. would I rather pick 1796 or 694?
• I usually do this as two condensed team lookup pages

Match prep

• Show information about all 6 teams in the match
• Good for getting high-level sense of match
• How conservative should we be with our strategy?
• Can be used to simulate potential playoff matchups

How to use all this for actually picking

1. Agree on priorities for alliance/picks
2. Sort the team list by something reflects your priorities
1. If your highest priority is autonomous scoring, sort by average auto points
2. If you just need as many points as possible, sort by average points
3. If you need a high-risk, high-reward pick, you can use max instead of average
3. Take top 2 teams, use comparison/lookup views to order them, this is the start of the picklist
4. Go down the sorted team list, compare with picklist teams, and add them one by one
5. Sort team list by other criteria to see if you missed anyone
• If you focused on an average, maybe sort by max or vice versa
• ​

Conclusions

• Focus on the most important data
• Don’t collect data you don’t need
• Take the time to build a good system for analyzing your data
• Easier and more useful than collecting more data

Thank you for attending!

​

Questions?

​

Additional scouting resources: https://linktree.com/gettinpicky