Note: This was originally given as a presentation at the Science + Football conference in May 2016. After a great many requests, it has been lazily converted into the article below.
In the beginning… Cavemen scouts watched cavemen footballers live and in the flesh.
Then… Cavemen invented television, and cavemen scouts could watch cavemen footballers on video.
Now… Cavemen have invented computers. And spreadsheets. And air conditioning. New, useful tools in evaluating cavemen footballers in a search for the best, brightest, and undervalued.
What Did Video Bring To Scouting?
- It’s cheaper. Look at a year of video provider service (like WyScout or Instat) compared to a year of travel budget to put scouts at live matches.
- Scouts now have easy access to more players and leagues than ever before.
- Finally, and most importantly… it’s all on demand. Whenever your scouts need to watch footage of players, they can. No more of this waiting for actual football to be played in front of you while you sit in cold, rainy stands on a Tuesday in Stoke nonsense.
What Does Data Bring to Scouting?
- Instead of evaluating hundreds of players a year, you can profile tens of thousands.
- Of those players, you get every minute played and every event they were involved in.
- Once the infrastructure is created, costs are whatever your data costs are.
- DIFFERENT, OBJECTIVE DATA than traditional scouting reports.
Plenty of people talk about the “data revolution” in sports. For me, it’s just another step and another set of tools to use in the evolving recruitment landscape.
How Does It Work Inside a Club?
With a small recruitment team of 2 stats + 6 part-time scouts, we evaluated over 1000 players in a year for the first teams of Brentford and Midtjylland.
Yes, but were you successful?
This is the most important factor, and obviously it depends on how you look at it. After a disastrous start in the first 9 games due to a poor manager choice, Brentford earned points at nearly a playoff pace, despite awful injuries in the first half of the season. The team also lead the league in goals scored and avoided an FFP-related transfer embargo.
And most importantly, they did it with one of the lowest wage budgets in the league and a £10-11m transfer fee surplus* in the year we were involved in recruitment. I’m going to notch that up as success, while admitting that at the start of the season, I was hoping for promotion just like the owner and every other Brentford fan out there.
*My estimate, not gospel truth.
Not Scouting Players Live is Ridiculous!
I get this sentiment a lot, both from fans and even from smart people who work inside of football. My perspective is like this
- By not spending time and money sending scouts to watch (many) live matches, we are able to watch a much greater volume of matches and more players. Travel time is a resource cost and it can be significant.
- Also by not sending scouts to watch live matches, we cut out an enormous source of cognitive bias.
Take this quote for example: “I can learn more about a player in 20 minutes in the stands than I can by watching him in hours of video footage.”
For this to be true, you either have to be some magical savant of player evaluation or you are full of shit. Conservatively, I would say 99% of people who feel this way have to be the latter.
This is a HUGE problem when it comes to scouting. If 4-5 games worth of video scouting work can be wiped away by sending one guy to the stadium, your process has a huge flaw, and likely so does your scout. It’s a correctable flaw, but if this type of thing is what decides whether you sign a player or not, good luck to you.
Yes, you need information about what the player is like with their coaches and teammates. That’s what personality profiles and background checks are for. Thinking you can get all of that information based on seeing a player once or twice in person is a myth the scouting world has sold itself.
Do I think there is potential value in watching certain positions live? Yes.
And some leagues have no useable video, so old-fashioned, in-person scouting is the only way to get the job done. In general though, I would do as much video work as possible in any club that I work in for the reasons explained above.
The Story of Radamel Falcao
And all of this was above and beyond the ACL injury, which for a 28 year old striker is a real concern. Loan fee: Unknown (Rumored to be 5-10M a season) Wages: Massive (Rumoured to be 13M+ a season) Outcome: 4 goals and 4 Assists at Manchester United. 1 Goal, 0 assists at Chelsea.
Lesson: Even if the ONLY THING adding data to your recruitment does is save you from a couple of bad decisions every year, they still pay for themselves. (In this case by saving something like 20M in wasted money on Falcao.)
Nate Silver and PECOTA
- PECOTA = Statistical scouting system for baseball.
- It was a very good system for evaluating Major League talent at the time.
- In 2006, Silver started applying it to minor league prospects. At that time, the only competition was from the scouts.
- In 2011, he compared PECOTA projections to the scouts…
- Scout forecasts were 15% better than PECOTA’s predictions, resulting in $+336M worth of extra wins during that period.
Reasoning behind this was two-fold. First, the scouts started to use basic numbers more to inform their scouting. Second, scouts have far more than just statistics at their fingers to evaluate prospects. They know whether guys are smart. Or overweight. Or hard working. Or any number of other things PECOTA did not know.
Nate’s conclusion: “The only way a purely stat-based list should be able to beat a hybrid list is if the biases introduced by the [scouting] process are so strong that they overwhelm the benefit.”
Building a Hybrid Process
- Names come in from any source – agents, coaches, players, scouts, numbers… whatever
- Initial check on Age/Passport/Realism/Need
- Stats check in leagues where you have stats – initial quick video scout in leagues where you don’t
- If players pass that, they go into the proper scouting queue. The player also triggers a detailed stat check.
- Scouting report comes back. Combined recruitment group ranks players for positions based on clear guidelines from head coach.
- List with executive summaries + video delivered to coaches.
Scouting Biases
What follow are actual quotes from scouts I have collated by networking with other football people for the last couple of years.
- “Six months in the devs and maybe he’ll be good enough.” This player went on to a big 5 move after being voted best player in his league
- “Poor tackler.” This player lead his league in tackling.
- “Couldn’t play in a midfield 2.” This player was the best DM in his league the year before. Then he was the best AM in his league the next season.
- “Plays in a shit league.” Quants have solid algorithms that tell which leagues are good/bad. This league was actually solid.
- “Few Shots. Snatches at the ball. Not a finisher.” Finished with an NPG90 of .9 for the season, and shots90 above 4. Whatever he was doing seemed to put the ball in the back of the net a surprising amount.
- “Not enough heart for a [REDACTED] player.” Sadly, we have no algorithm that directly measures heart. Also, I’m not sure how you argue back about this in a recruitment meeting without being extremely sarcastic. Thankfully, I wasn't involved.
So if this is the general environment you are walking into, what do you do?
Profile Your Scouts
- Scouting reports are DATA! Data is meant to be analysed.
- Go back through past work and review it.
- If using numbers, what are the average scores for each scout? For each position? What if some scouts never find a centerback or fullback that is deemed good enough to play in your team? You need to know that and adjust for it.
Are they biased:
- Toward young players?
- Against foreigners?
- Just don’t understand the football in certain leagues? If the style of football in certain leagues differs a lot from what a scout is used to, they can have problems evaluating players in those leagues.
Data analysis is complicated and hard. Scouting is also complicated and hard, but uses an entirely different set of skills. One club I know of has scouts in 37 different countries, speaking more than 10 different native languages. Standarizing output across that organization simply based on linguistic differences is nearly impossible.
Standardize Your Scouts
- Scouts are not just plug and play. Scouting is hard work and scouts from different clubs and backgrounds might be looking for totally different things in players than your club requires.
- Scouts need training, and clear guidelines on what to look for when evaluating players.
- Scouts may need training across multiple leagues and styles of football.
- Youth scouts need even MORE guidance, since there are a ton of pitfalls regarding relative age and growth spurts they need to work around.
Lesson: You need to standardize traditional scouting output as much as possible to make better decisions and so that even your scouting output is consistent.
Conclusions
- Increased use of stats and data is just a part of the long-term evolution of the recruitment landscape.
- Talented scouts are hard to find and will remain extremely valuable members of any club.
- Hybrid processes – both stats and traditional scouting - are mandatory as best practice.
- On-demand video and data allow teams to use these valuable scouting resources more effectively.
- You MUST be aware of scouting biases in order to get the most out of these valuable employees.
Ted Knutson @mixedknuts mixedknuts@gmail.com