Wednesday, July 22, 2009

Shooting Percentage and False Perception

A lot of the time gut feel works just fine in assessing a player, and a guy's ability to finish is a big part of that. There is no doubt that Heatley has more finish than Dvorak, the memories of Dany's highlight reel goals and Dvorak's blown chances as an Oiler ... these colour our opinions, and their history of shooting percentage confirms them.

Now playing on the powerplay helps a guy's shooting percentage a lot, especially if he is the trigger man and if he gets a lot of 5 on 3 powerplay time as well. Getting a few empty net goals really helps this number as well. So here we'll just look at even strength goals that happened with both team's goalies on the ice (EVshooting%).

Now everyone knows that even the league's premier goal scorers will have cold stretches this season, almost all of them will have a month (20 to 30 EV shots) with a brutal EVshooting%. And while some fans will get excited about it and start searching for reasons, most will accept that it's just the Hockey Gods in action. If Iginla has a stretch of five weeks with just one EV goal (say 1 goal on 30 shots, 3.3% EVshooting%) most around here will chalk it up to random chance. And they're probably right. mc79hockey.com has shown that the pattern of EV shooting% is very nearly identical to that expected by chance alone, granted only four players were studied. Still, it's impressive considering that line mates, injuries and psychological elements are surely factors. So while those things are likely all in play, they are extremely difficult to detect through the noise of luck. Plus I've never read or met anyone who has been able to predict future shooting% trends.

So we're good at sensing the level of luck involved with the small samples, where the human mind let's us down is over a larger number of games and shots. It feels like a season, or certainly two seasons, should be enough to give us a good gauge of a player's true finishing ability at even strength NHL hockey. My own gut feel would be "within one or two percent" after two seasons, and I would be wrong. And going by the things I read on the internet, I suspect that most people would mentally put a narrower error band on it than me.

Below is the confidence interval for a mythical player who we know to have a natural 10% EVshooting% ability. If Tyler is right, or largely right, then this is the range of results that we should expect to see from the player 95% of the time. And if we looked at 100 identical players, after any given interval 5 of them would be outside this range of shooting%.

A top six player will probably get about 130 EV shots in a season. Less than that if he is playing on a weak team, more than that if he is playing on a territorially dominant team. And a 10% EVshooting% ability is about what you'd expect from a top-six-ice-time type of forward. You'd want more than that if the player brought little else to the table, and could live with less than that if he had a wider range of skills. But that's a reasonable midpoint.

So the chart above reflects a total of about four seasons on a good team, and about five seasons on a weak team.

After 60 even strength shots, about half a season, our man's 95% confidence interval ranges from 2.2% to 17.4%. That's a hell of a swing. That means that in 100 parallel universes, our man would be expected to have an EVshooting% in that range 95 times. And the universe requires him to be higher or lower than this 5 times on average. It's cruel in a way, if our man doesn't have a track record in the NHL, then some of the parallel universe versions are being rewarded with rich NHL contracts, while in other parallel universes our man is being buried in the minors or shuttled off to Europe. All at the caprice of the hockey gods.

Speeds has an excellent post up on the most recent Oiler first round draft pick. Every scout seems to have seen him good, but the offensive numbers aren't where we would want them to be, it appears that he hasn't scored enough to justify the draft position. Speeds points out that he had a poor shooting%, and wisely suggests that it could be either because he doesn't have much finishing ability or that he was simply unlucky. And that we have no way of knowing which is true until time reveals more information.

And even for a guy like Cogliano, who has shot the lights out so far, he has only fired about 200 EV shots thus far in his career. So all we really know is that he is very likely (about 95% likely) to have a true EVshooting% within 5% of his rate so far.

So how good is Cogliano at finishing his chances? The skeptics and determinists can send their poets out to battle each other all day and night, but the fact of the matter is that we just don't know. We do know something close to the true probabilities, but it's still such a wide band that very few people would be foolish enough to bet the rent on him maintaining his current level.

The same intervals apply to on-ice EVshooting% for all players as well, though the larger sample (i.e. number of EVshots) is typically about four times higher. The same also applies to goalies and EV save percentage of course. So while we are judging goalies largely by EVsave%, which is sensible, we always have to remember just how volatile these can be in the short term. And while we can predict the behavior of the population, we can't predict which ones will throw more than their expected number of bullseyes.

This summer, teams that paid for players based on results that came from a year or two of good percentages ... they'll most likely regret it. Teams that have paid for players with good track records but a recent stretch of poor percentages, those were probably good bets.

27 Comments:

Blogger sunnymehta.com said...

vic, what SD did you use in the confidence interval calcs?

7/22/2009 3:03 pm  
Blogger sunnymehta.com said...

also, can you tell me what the average EV S% is for a defenseman and forward? my intuition tells me we should treat them as two different positions (rather than lump all non-goalies together) because certainly we see a difference in the populations, no? (i.e. - forwards as a group have a higher EV S% than d-men).

7/22/2009 3:11 pm  
Blogger Vic Ferrari said...

I didn't Sunny. I ran the numbers using a binomial probability algorithm.

you could use:
plus or minus 1.96*sqrt(.1*.9/n)

where 'n' is the number of shots would be mathematically simpler I suppose.

But it would bring in inaccuracy especially in the lower ranges, if this was 20 years ago that's what I would have done, but computers do the work in a second or two now, so what the hell, no need to cut corners.

7/22/2009 3:32 pm  
Blogger Vic Ferrari said...

I've never summed them up, Sunny. I think that the average for forwards was 9.3%. I don't know if that was the average of all forwards with over X shots on goal, or totalFWDgoals/totalFWDshots.

7/22/2009 3:34 pm  
Blogger sunnymehta.com said...

why isn't the SD just sqrt(.1*.9*n) ???

i.e. sqrt(npq)

7/22/2009 6:18 pm  
Blogger Vic Ferrari said...

I was talking confidence interval, Sunny, not SD.

7/22/2009 6:20 pm  
Blogger sunnymehta.com said...

ok, gotcha. thx. :)

Here's why I was asking...

I was curious how the S% studs actually compared to the luck distribution over the course of a single season.

I went straight to an uber-stud: Ilya Kovalchuk. Dude posted a 16.5 S% in 163 shots last season.

Turns out that an average 9.3% shooter would have 99% confidence intervals of 3.44 to 15.16 over 163 shots, so Kovalchuk is outside of that range. We can fairly confidently say that he is an above-average finisher. I'd be confident enough to bet on it. (As a side note though, this is what I was talking about with the induction vs. deduction thing. Technically, we don't know for SURE that he is above average. It's still possible he got really REALLY lucky - just not very likely.)

Back on point: 3.44 to 15.16!!! Wowzers, that's a huge freakin' range. One season really ain't squat of a time frame to be sure of a whole lot.

7/22/2009 6:57 pm  
Blogger Vic Ferrari said...

Yeah, that's for sure.

And with the same thinking you can look at a free agent and make a decent guess as to his EV shooting ability and effect on on-ice shooting percentage, but only ever in terms of probabilities.

And of course players abilities change with age and injury. The fact is that we will never know how good of a finisher Kovalchuk is, we'll only ever be able to narrow down the range of probabilities.

We can give a probabalistic answer to the question: what are the chances of Khabibulin/Druin-Deslauriers stopping more pucks in a season that LaBarbera/Anderson? And bet with the odds.

At even odds, for EV save%, where would you want your money riding for next season in the NHL?

I know where my cash would be.

7/22/2009 7:14 pm  
Blogger JavaGeek said...

I have 4 seasons of data and now Kovalchuk is 6 standard deviations away from what would be expected (for all shots though, not just EV). Which is about as unlikely as well being struck by lightening twice in one year.

7/23/2009 9:04 am  
Blogger Olivier said...

Say you throw missed and blocked shots; would that give you a more significant result set? Or are blocked and missed shots simply adding to the noise?

7/23/2009 11:37 am  
Blogger Vic Ferrari said...

Olivier,

I don't know. A lot of the European leagues count missed shots in the shooting% equation, I see that Gabe does the same on his site. I've never checked though.

The first step would be to check for real effects, I think. See if adding in missed or missed+blocked shots has a stronger association, sample to sample, than shots alone.

Compare results from season to season or half season to half season, etc. You could do that using timeonice.com, but my scripts are so slow that it would take ages to pull down the data. Sooner or later I'll get around to putting that information in a MySQL database to speed it up.

7/23/2009 2:38 pm  
Blogger Kent W. said...

Great stuff Vic. Those spreads are just crazy.

7/23/2009 11:40 pm  
Blogger Vic Ferrari said...

Yeah Kent, and it's the same for any statistic in any sport. Though those presented as percentages are more easily plotted out.

In life, in sport, we're soaking in randomness. And in almost every case, it doesn't 'settle out with time' at anything close to the rate our intuitions tell us.

7/24/2009 10:10 am  
Blogger mc79hockey said...

I have 4 seasons of data and now Kovalchuk is 6 standard deviations away from what would be expected (for all shots though, not just EV). Which is about as unlikely as well being struck by lightening twice in one year.

The obvious suggestion is that, in his case, he's better than the league norm at finishing.

In any event, given that we know that Kovalchuk plays far more than normal on the PP and probably has a much higher percentage of his shots there than anything, does your comment really tell us anything? I'm just asking, not snarking, but it seems to me that it doesn't.

In life, in sport, we're soaking in randomness. And in almost every case, it doesn't 'settle out with time' at anything close to the rate our intuitions tell us.

Yeah, this strikes me as being the most important thing I've learned over the past six years or so. Funny though, there's a guy at LT's making a brutal argument about how 20 games in 2007-08 was just too much for luck. Although he seems to think that "luck" is a binomial 50/50 proposition, which I don't really get either...

Also: word verification is "sledia" which sounds like an awesome winter theme park.

7/24/2009 10:58 am  
Blogger mc79hockey said...

@Sunny: http://www.mc79hockey.com/?p=2933

7/24/2009 11:01 am  
Blogger mc79hockey said...

Question for the guys who are better at math than I am. League average S% of 10% for fowards say. I shoot 14% on 200 shots. Is it equally likely that I'm an 18% shooter as it is that I'm a 10% shooter? I've always understood the answer to that to be yes, although that doesn't make sense to me intuitively.

7/24/2009 11:12 am  
Blogger sunnymehta.com said...

mc,

Classical statistics say yes, you are equally likely. This is the issue with using classical statistics like straight confidence intervals. The confidence interval just sets up a range, and essentially you are either "in" the range or you are "out" of the range. No specification is given on how likely different numbers are within that range.

However, this is where Bayesian statistics come in (i.e. - conditional probability), and this seems to be all the rage these days. Bayesian concepts say "Hey but you don't have NO OTHER INFORMATION. You have tons of information on what the general population does. Why not use it?"

So essentially, you take the range of the player in question and "combine" it with the range of an average player. Now you can make assumptions about exactly how probable different shooting percentages are for the player in question. And you can imagine, everything heavily regresses to the mean.

So a player with a smallish sample of 200 shots at 14 percent wouldn't be THAT much more likely to be an above average shooter than a totally average shooter would be. But, as he continues to pile up shots at a 14% clip and his sample size starts to grow, our confidence in saying how probable it is that he's above average starts to grow.

That make sense?

Others can probably add to/edit what i said as well.

7/24/2009 12:43 pm  
Blogger Olivier said...

Lovely discussion, again.

I think I'm still missing something of the gist of the argument, because that same exact question keeps popping in my head: how many shots before we begin to see some separation between players; what's the magic number? 2000 shots? 4000 shots? That's why I was enquiring about missed and blocked shots; if we add those, can we get faster to the magic number?

The stats we have available are quite crude; I wonder if some day the NHL will implement something akin to the pitch f/x system, ie fixed cameras at set spot in every arenas allowing for real location of shots and whatnot...

7/24/2009 1:13 pm  
Blogger Vic Ferrari said...

sunny:

As I've done it here, we KNOW the ability of the player, so the confidence intervals work.

Of course in the real world, sports or otherwise, that's rarely the case. There we have a bunch of observations, and that's all. Working backwards to find the actual ability is harder, obviously.

If we consider the player in isolation (which is what is almost always done in online baseball and hockey analysis) ... then he is far more likely to be exactly an 18% natural shooter than 10% shooter.

http://www.timeonice.com/binomial.html?n=200&k=28&p=.18

and

http://www.timeonice.com/binomial.html?n=200&k=28&p=.10

He would be about 60% more likely to be an .18 natural than a .10 natural.

If he played in a league in a parallel universe were there only ever were .10 and .18 players in existence, nothing better or worse or in between ... and if there were 10 times as many of the lesser players in the league, then the paradigm shifts, and the same guy is now about 6 times more likely to be a .10 natural than a .18 natural based on that one season's data.

Makes sense, no?

In the next parallel universe along maybe there is a third category, .08 natural shooters stealing about half of the .10 guys share. I'll leave you do do your own arithmetic, but again you have probabilities of each category of player that are simple to compute.

If the next parallel univers has a fourth category ... do the same.

Keep rolling through the universes until you get back to this one that we are in, several thousand parallel universes away. And the math has gotten more complicate, but the thinking is precisely the same.

And if you want to categorize this type of thinking, it is Bayesian, as sunny points out. I agree fully with all of sunny's comments except the first paragraph, btw. Which may well mean that we're both wrong :D

7/24/2009 1:18 pm  
Blogger sunnymehta.com said...

Hm, perhaps my first paragraph is applicable to standard confidence intervals, whereas we seem to be dealing with Binomial Proportion Confidence Intervals in this thread (i.e. - we're assuming a binomial distribution)? Is that the discrepancy, Vic?

So how would we do all this if we were dealing with a stat that wasn't a proportion, i.e. something we couldn't test binomially? For example... Pitcher ERA.

I might be missing something.

7/24/2009 5:57 pm  
Blogger sunnymehta.com said...

Yeah, Vic, you are into Bayesian realms in your last post with those two links you posted, no? Because, you are saying what are the odds of y if we know x. That's different than saying, Player A showed a S% of 14 percent over 200 shots, so let's assume that to be his true mean and set up confidence intervals from there. (which would be the "classical" method, and suboptimal imo.)

7/24/2009 6:06 pm  
Blogger Vic Ferrari said...

I'm not sure about any of the classifications.

The model for the chart above assumes all shots have an equal chance of going in, we know this isn't true, but if you nudge up some shots as great opportunities, and add many more to the 'slim chance' pile, at the end of the day you end up with expected results that are nearly spot on the simple model anyways. And you've made a lot of work for yourself for no reason. Believe me, I've done it, though feel free to invest a bit of time to do it yourself if that seems intuitively wrong. Plus, as referenced above, Tyler's work on streaks reinforces confidence in the model as well.

Tyler's question is well asked, though. And if you look at the tree in isolation, then those two links I provided work. Staying true to the original model.

But if you consider the forest, as Tyler implied with the 'league average 10% shooting' ... well that changes everything.

Of course that isn't enough information about the forest to really do it properly. But if we gave Tyler a bag of quarters and 20 cups, each with a shooting% written on it with a Sharpee.... then had him scroll go through the NHL rosters and chuck a quarter into the appropriate cup for each forward in the league (e.g. he sees Hemsky's name, thinks that at EV Hemmer is probably about a 13% shooter, and chucks a quarter into the cup that has "13%" written on it). That would give us a an assumed ability distribution.

He's kicked this stuff around enough that he'd probably get it close, on the whole. So now we have a distribution of the population, general info about the forest, so to speak.

Then we could answer his question accurately using the method from the parallel hockey universe as described above. And if we did that same math for every forward in the league, then we'd end up with the same distribution as he created for us by putting the quarters in the cup, but ONLY if he got it right in the first place.

If it wasn't quite the same, let's say that we realize that the forest doesn't look to have nearly as many truly puny trees (terrible shooters) as Tyler had sensed when he put the quarters in the cups the first go-round.

So, armed with this new insight, we dump the quarters out of the cups and let Tyler have another go. But first we give him a shot of bourbon (my Johnny Fever theory).

Rinse and repeat, until it's not making a material difference to results. And voila! You've got your ability distribution for the population. And then we'd really be cooking with gas.

Of course if you're a 'tree man', you're never going to give a toss about the forest, the whole notion just won't make sense to you. Which is fair enough, it's just sport after all.

And of course the key to predictive value is understanding the forest. And predictive value is very compelling as well.

I mean around the internet slipper and Bruce have argued for years. And I suspect that intuitively, the staggering majority of sphere readers sided with Bruce's point of view in the individual arguments.

I also suspect that this changed over time as people started to realize that, had been been betting real money on each of these arguments, slipper would surely own all of Bruce's wordly possessions. (Not to pick on Bruce or flatter slipper, they are just the most vocal of the tree and forest people out here in the Oiler blogiverse)

7/25/2009 9:16 am  
Blogger Vic Ferrari said...

that should read:
he sees Hemsky's name, thinks that at EV Hemmer is probably about a 13% shooter by natural ability, not necessarily by results to date in his career

7/25/2009 9:18 am  
Blogger Showerhead said...

Aha, yet another fantastic post and to this point in my afternoon I've only hit IOF and BDHS. Bless the mighty blogger, he of zero credibility, he of his parents' basement, he of the checkered boxers, holed socks, and beered belly. This is why the internet is cool.

Nothing to add to the math discussion here (obviously). It really makes you wonder about GM's though - they're under a lot of pressure to make smart decisions and being human, they probably rationalize the hell out of the decisions they make. Betting on a guy who just careered in terms of % is justified because he had a breakout year, he had chemistry with his linemates, he listened to the coach's advice, he got a new stick, whatever. 95% of it is bullshit, the language with which we attempt to add meaning and explanation to everything we observe, even when it's arbitrary, even when it's incredibly random.

And several hundred million dollars ride on this thought process every year in the NHL.

7/28/2009 4:34 pm  
Blogger oilswell said...

How confident is everyone with the notion that a player is an "N% shooter"? I can't completely wrap my head around the idea, though that's probably because the cranky part of my brain is so rudely fixated on shot context that it wants to treat shooting % as a proxy for aggregate scoring chance quality.

7/28/2009 10:58 pm  
Blogger Vic Ferrari said...

oilswell,

Good to see you around. Be sure to criticize my math, I'm sure that there is plenty wrong with it. In fact in some cases I know there is. The detractors never seem to guess right, though.

And personally, I'm pretty confident with that notion, the N% shooter at evens, though not enough to defend the idea with my normal level of selfrighteousness.

Have you looked at Dennis' scoring chance stuff, oilswell? Scott has aggregated it, I've posted it online but possibly forgotten to link to it (I think I put it up at timeonice, meant to ask Scott's permission to link to it, then got busy with other things and possibly forgot about it).

In any case, the whole "high event player" notion is taking a shitkicking to my mind. Admittedly I coined the term. It appears that I was wrong.

You can deduce the scoring chances (either + or -) but using shots-at-net totals, for and against summed, in lieu of icetime, and then the corsi ratio. And with surprising accuracy.

Goals are such rare birds that I wouldn't want to make any conclusions about it after one team and one season. But it looks, for the moment, that whether it's done beautifully or in ugly fashion ... territorial advantage leads to scoring chances advantage in proportional measure, and from there on out it's all down to Mike Keenan's belaboured "historical ability to finish".

Seems that way to me at the moment, anyways.

7/30/2009 1:55 pm  
Blogger Vic Ferrari said...

To add to above: guys who started more shifts in non-neutral territory get a wee bump in scoring chance rate, for obvious reasons.

7/30/2009 1:57 pm  

Post a Comment

<< Home