Friday, July 24, 2009

The 'Shot Quality' Fantasy

You can't swing a cat on the internet without hitting a goaltender apologist. Every time that a goaltender has a bad year in terms of save percentage, frightening and unrecognizable excuses come squirting out of every orifice of the hockey internet. A defensible rationalization would be "He was just unlucky, the shooters made their shots this year. Relax." But you never hear that one. It is always the defensemen, coaches, or perceived tactical changes that take the internet beatdown.

The question is: How much of these changes were required to happen by the existence of chance in the universe, and how many were the fault of the coaches and skaters?

The Contrarian Goaltender had a very cool idea a while ago, he looked at goalies that changed teams. The thinking was that if teams were the ones impacting the save percentage, it would reveal itself when the goalies switched squads. I'm giving the same idea a more rigorous test here. This is the list of every goalie's stats over the past ten NHL seasons. I'd advise that you calculate your own percentages if you are using these numbers to make a point, beyond that they seem correct.

The varying number of 5 on 3's team to team, that skews the PP save%'s a bit, so we'll just use even strength save% here (EVsave%). The first group of players we'll look at are the goalies that played for the same team two seasons in a row. lists the player's team as the one he played for when the season ended, if he started the season in Edmonton and ended up i Pittsburgh he'll just show up as a Penguin. So rather than look up all the trade information manually, I picked seasons for goalies that qualified as playing the previous two seasons for the same team. They also had to have faced 300 EV shots in both of the seasons. 196 goalie seasons qualify.

Next, I built a model, using the players total 1999-2009 EVsave% as the weighting of his coin, and the number of shots from each season as the number of flips. And I ran it 100 times.

And it turns out that the universe requires an average change, from season to season, to be .0103 for these guys. Of course it varies a bit from virtual season to virtual season because, as shown in the post below, randomness takes an awfully long time to truly settle out, especially when we're looking at such fine detail.

The actual change for these 196 goalie seasons was .0108. Even though that increase over virtual could be just chance, the distribution of the actual is also a whisker wider than the model, so I'll concede that the combined factors of coaching personnel and philosophy changes, defender changes, tactical shifts, the goaltender's own rebound control and puck handling abilities, and the 27 other notions swimming around in Kelly Hrudey's head ... they add together to create a change in 'expected goals against', from season to season, for the same goalie on the same team, of almost 1 goal. And I'm clearly being generous with that.

Now for changes in EVsave% for players who played a complete season for one team and then the complete next season for another team. To do this I had to manually go through the database and make sure that the goalies weren't traded during the hockey season. I ended up with a list of 66 guys, the same minimum 300 EV shots each season cutoff applies here also.

And the universe requires that the EVsave% change as the goalie moves from team to team, if shot quality doesn't exist at all, well it will average .0122. With these 66 goalies it averages .0133. There is nowhere near enough veracity to even declare that a difference in shot quality exists at all, but again the distribution is a smidge wider with the real than it is with the random, so I'll give the benefit of the doubt and say that, on average, the difference in shot quality from any two teams selected at random ... the expected difference will be 2 goals on the season.

And I haven't even broached the issue of whether or not goalie sellers outperform goalie buyers on the whole (they do by the way, persistently but just by a touch) because that would carve a big chunk out of the tiny bit of evidence for shot quality that exists. I'm erring well on the side of caution here.

In short: The universe thinks that the NHL 'Shot Quality' metrics are a complete crock of shit, but I like most of the people that create them.


Blogger Vic Ferrari said...

I should add that I think the shot quality metrics are usually measuring something besides scorer bias and randomness.

Most of them are skewing the right direction because they are adding extra value to PP shots, especially 5 on 3 shots. So ANA gets a bump over N.J ost years because they take a lot more penalties and usually end up killing a lot of 5v3s as well.

Desjardins 5v5 stuff seems like the most honest, granted it is built from dubious data. I think he is capturing quite a bit of the luck in the game though. So a slap shot from the point into traffic:
1. If it finds it's way through to the goalie and hits him in the chest it's a low quality shot.
2. If it gets tipped in front it's a high quality shot.
3. If it caroms off a leg into the high slot but on the shooter's backhand side, and he fires it on net ... medium quality shot.

I suspect that the elements that associate with EVsave% are fleeting, but have powerful correlations in the short term. And that as the sample grows larger, the association of ShotQual to EVsave% deteriorates in a pattern like that shown in the previous post.

So when Gabe says that Khabibulin faced high shot quality in his bad years, it tells me that he was probably just a touch unlucky that year. He's probably still a decent goalie. Which is better than finding out it was because he was riddled with chronic injuries.

He should be a good bet to give them something close to .920 EVsave%, which would have been pretty impressive 8 or 10 years ago, but gets less impressive with every passing year as more and more excellent young goalies come into the league.

When I was a kid nobody wanted to play goal. Now the kids fight over who gets to play net. On the whole, there is just a much better of quality of athlete playing the position now. I mean does Luongo play goal growing up if Patrick Roy isn't the biggest hockey star in Quebec?

7/25/2009 9:37 am  
Blogger Jonathan Willis said...

Amazing work Vic. Thanks for this.

One question - what about Minnesota? Is it a 5v3 thing, or is Backstrom really that good?

7/25/2009 12:10 pm  
Blogger Vic Ferrari said...

I think he's just a good goalie. .927 EVsave% for his NHL career, playoffs included. That's very good. Certainly not Roy or Hasek country, and it's just been three seasons. But going by Minnesota's history with goaltending, and knowing that they signed him to a big deal ... he may well be even better than that number.

Of course because Minny takes so few penalties, and almost never have to kill a 5v3,his overall save% gets a bump, so he gets extra love from the media.

Roloson is a career .927 EVsave% guy, btw. The Oilers let a very good, albeit old, goalie walk away.

He was .9276 in Minny and .9274 in Edmonton. Freakishly similar, granted that's just a coincidence. I think he probably has two or three more good years in him. He's an underrated guy, one of the better goalies in the league over the last decade. And he still plays like he has something to prove. Good pickup by Snow.

All the smart money will be on Roloson havng a better EVsave% next year than Khabibulin does. Or more rightly, make 20 wagers like that and you'll win about 15 of them.

7/25/2009 1:32 pm  
Blogger JLikens said...

Nice post.

I used to be a pretty big proponent of the usefulness of shot quality as a metric, but have become somewhat less enthused over the last year or so.

A lot of that has to do with the fact that there just isn't that much inter-team variance in shot quality.

Also, shot quality can only account for a fraction of the observed team-to-team variation in shooting and save percentage (R^2=~0.10-0.20).

I think that shot quality is only relevant at even strength in terms of the leading/trailing effect.

As both you and I have demonstrated, EV shooting percentage at the team level when the score is tied is entirely random. Even still, I think that some the overall variance in even strength shooting (and save) percentage is due to the fact that some teams will necessarily play more with the lead than others over the course of a season. In fact, I was just looking at data from the 2007-08 season and the teams that tended to play more with the lead tended to do better with the percentages (The Wings and San Jose were outliers).

7/25/2009 4:20 pm  
Blogger Kent W. said...

You are just on fire this summer Vic.

I love this stuff because I don't naturally think this way and can't come up with the math or models myself - I can follow it, kinda like someone who is familiar enough with a foreign language to listen and understand, but can't actually speak it fluently.

This is an issue I've been battling as a Flames commenter for awhile now. Since Kipper's descent into Sucksville, the collective cognitive dissonance of the local masses has swelled to near unprecedented heights. Go to any local Flames mesasgeboard and you'll find scores of posters who insist he's still a top 5 goalie and it was simply Keenan, fatigue, the defense and a myriad of other factors causing the decline. Hell, I read one comment recently that said "the forwards just need to score when it matters"(?). Why this is comforting is beyond me - I mean, were it true, it just means the rest of the team is terrible, right?


Anyways, thanks again for the work.

7/25/2009 6:16 pm  
Blogger Vic Ferrari said...


I don't doubt that you are right with the score effect thing. Thing is, it's fine brushstrokes, when you even consider the 5th worst EV team and the 5th best, depending on how things went on special teams that year for them, it's going to be pretty tight I think.

It's a valid point though, because we don't want to throw the baby out with the bathwater, and though I don't know for certain, I suspect that you point has real merit.

Then again, Bob Stauffer believes the whole world revolves around this phenomenon, which is flat out madness.

7/25/2009 7:24 pm  
Blogger Vic Ferrari said...

You know, Kent, the thing that's kind of been lurking in the back of my mind ... maybe Kiprusoff really isn't as bad as he seems now, and maybe he wasn't really ever as good as he seemed when he was stopping everything.

Seriously. If I'd watched more Flames games I'd have a better sense of it. OTOH if I'd watched more Flames games that would mean I was a Flames fan, and I'd be emotionally invested, and therefore as biased as hell.

If you were the bomb at sports trivia, and answered at a 93% clip for a couple of pub-trivia seasons in a row, winning the adoration of other thursday drinkers about (2000 Qs per season) ... Are we justified in qualifying you as a dumbass for answering at a 91% clip over the next two years?

I don't know.

7/25/2009 7:31 pm  
Blogger Statman said...

If 93% = 95 percentile, & 91% = 55 percentile... then maybe, yeah.

7/26/2009 8:41 pm  
Blogger kris said...


Nice work.

I'm worried about some assumptions, though. For one thing, even when you look at the same team from year to year, there's a lot of change in the lineup. So, you write that "The actual change for these 196 goalie seasons was .0108." But couldn't a large portion of that 0.108 be due to changes in shot quality a goalie faces from season to season, even if he isn't traded?

Or has I no understand?

7/27/2009 12:09 am  
Blogger The Contrarian Goaltender said...

Great work as always Vic.

I tried to defend EV shot quality and just couldn't find much there at all, so can't do anything other than agree with your position. It always seemed so intuitive to me that shot quality must be important that I never really took the time to test it. I agree with JLikens that there are probably a few things in shot quality that might be worth a closer look, like score effects, but there's almost nothing there in the big picture.

I have to admit that I'm not sure that my acceptance of the theory has filtered down yet to my subjective evaluation of Minnesota goalies. I'll be interested in how Backstrom does the next couple of seasons.

"When I was a kid nobody wanted to play goal. Now the kids fight over who gets to play net. On the whole, there is just a much better of quality of athlete playing the position now."

As the supply of good goalies continues to growing beyond the number of jobs available, it looks like the margins at the NHL level will continue to shrink. If there are more goalies than spots right now, what's going to happen if there are 50 starting calibre goalies in 2020? It'll probably be almost pure luck that determines who sticks at the NHL level, although I agree with you that Kelly Hrudey will probably have some theory or another to explain it.

7/27/2009 1:20 am  
Blogger Vic Ferrari said...


That' where the random model comes in. By chance alone we'd expect a goalie's EVsave% in this model to change season by season on the same team, by .0103, it in fact changes by .0108.

So there is likely some sort of change in a team's ability to effect shot quality from season to season. Just that, on the whole, it is tiny.

7/27/2009 8:27 am  
Blogger Vic Ferrari said...


Yeah, I agree with all of that. And just because the phenomenon of 'shot quality effect' is tiny in the general population, that doesn't mean it doesn't exist for some specific teams. But if it does, outliers are going to be rare.

And your last paragraph is the one that really should matter to GMs. In fact that day may already be here. Ken Holland gets that. That doesn't mean that he'll win the bet next year, randomness has the last word on that, just that he'll probably spend more wisely on goalies over the next ten years than most of his fellow GMs.

7/27/2009 8:37 am  
Blogger Matt N said...

Would I be correct in restating this as .....

"The only difference between good defensive teams and poor defensive teams is shot quantity."

7/27/2009 12:24 pm  
Blogger Vic Ferrari said...

Matt N:

I'd add in "extremely likely" and "significant difference".

Personally, more than anything I think it is wise to think of hockey as a game of flow, as opposed to trying to break it down into defense and offense.

I think that the fairest thing to say is: The best way to reduce goals against at EV is to play more of the game in the other team's end of the rink.

7/27/2009 12:40 pm  
Blogger Brian said...

It always seemed so intuitive to me that shot quality must be important that I never really took the time to test it.

CG, I think I know where your assumption erred. You recognized, correctly, that some shots are more dangerous than others. You further recognized, also correctly, that defenses can employ strategies which make it harder to get dangerous shots. What you failed to recognize is that the offense wants to take shots from the danger areas, just as the defense wants to prevent them from doing so, and the fact that the defense allows the offense to take less dangerous shots, doesn’t mean they actually will.

Take playing the pass rather than the shot on a 2-on-1. The strategy allows the offense to get a shot on net every time, if it wants. However, it also allows them to try and force a pass through the defender, and risk having it intercepted. There is no inherent reason, then, for the tactic to result in lower quality chances, rather than fewer chances of the same approximate quality.

The finding that defenses do not appear to alter SV% indicates that the competing desires of offense and defense tend to achieve equilibrium.

7/27/2009 2:24 pm  
Blogger Matt N said...

I am just working off of observation and experience and don't have data to back this up.

There are systems in hockey that dictate the type of chances that a team will typically give up.

Examples would be the current Bruce Boudreau coached Capitals that activate defensemen (Mike Green) and try for the stretch pass for streaking forwards (AO, Semin). They tend to give up lots of odd numbered rushes and break aways that are high quality chances. This tends to artificially lower the counting numbers for their goaltenders.

On the other end would be the '95-'96 Doug Mclean coached Panthers that played a trapping system that preached stick positioning and funnelling opposition breakouts towards the outside of the circles. This creates lots of poor angled shots from a distance that would artificially inflate the goaltenders counting numbers.

Another example would be the difference between teams that play a man vs. a zone defence in their own end. Man defensive schemes tend to create more turnovers because of pressure, however they also tend to give up better quality chances when the defender loses his check. A zone system preaches keeping players on the outside and collapsing towards the front of the net to block shots and controll rebounds.

In my opinion it just seems intuitive that different teams will give different quality chances.

7/27/2009 3:15 pm  
Blogger Jibblescribbits said...

I find a bit of a problem with the approach here.
I would expect the average change to be on the same order of season-to-season variances by goalies. In testing this theory some goalies would go to better defensive teams, and some worse.

Because of this using Average delta-ESsave% seems almost worthless, IMO.

Where I might expect to see a real change would be in the standard deviation.

7/28/2009 11:53 am  
Blogger Vic Ferrari said...


No, these are absolute values used for save% change. Otherwise they would have gravitated towards 0.0000, not towards .0103.

The data is all available at and model is easily duplicated.

7/28/2009 1:37 pm  
Blogger The Contrarian Goaltender said...

"Roloson is a career .927 EVsave% guy, btw. The Oilers let a very good, albeit old, goalie walk away.

He was .9276 in Minny and .9274 in Edmonton."

Your numbers for Roli don't match what I have. I have Roloson at .916 in Edmonton (1365/1500 in '07, 845/928 in 2008, 1390/1501 in 2009). The Oiler and Wild numbers are combined for 2005-06, but if that's included in the Edmonton numbers as well then Roloson drops to .915.

I agree he's a good goalie, but I don't think he did quite as well in Edmonton as he did in Minnesota, which is why I'm still fairly open to the possibility of the Wild being a shot quality outlier.

7/28/2009 2:29 pm  
Blogger Vic Ferrari said...

CG: I included playoffs games as well, just to up the sample sizes.

I did the same for Khabibulin and Biron as well and make them to be almost spot on the EVsave% since 99/00, though I may well have made a mistake doing the sums in either or both cases, as I did them manually.

What are your thoughts on goalie aging, then, CG? Do you have any thoughts on the Khabibulin contract?

7/28/2009 3:25 pm  
Blogger Showerhead said...

First off, wonderful post. Just love it. Seems there have been many excellent posts over the last week or two and I'm only starting to catch up :)

A lot of that has to do with the fact that there just isn't that much inter-team variance in shot quality.

This is how I see it as well. It would be correct to suggest that not all shots are created equal, and of course some are more dangerous than others depending on where they're coming from. We can quantify it, I also expect, and I'm sure some people probably have.

That said, at what, 25 shots against per game and 82 games a season, you're looking at more than 2000 shots against over the course of the year. Surely, if there were strong proof that a variance DID exist between teams, Vic would have identified it here. The difference, IMO, may exist from team to team or even from strategy to strategy on one night to another but over the course of a full NHL season I'd say 99% of that difference gets washed out. Teams impose themselves on their opponents roughly as often as they are imposed upon.

And with that, I have more excellent posts to catch up on :)

7/28/2009 4:23 pm  
Blogger The Contrarian Goaltender said...

Vic, even with playoffs included something is still off with your Roloson numbers. He's only a career .920 EV goalie (same as Khabibulin), and he did better in Minnesota than in Edmonton, although given his age that isn't unexpected.

2002: 818/897, 0/0
2003: 1004/1080, 190/206
2004: 1012/1077, 0/0
2006: 509/559, 0/0
TOT: 3533/3819, .925

2006: 348/382, 408/438
2007: 1365/1500, 0/0
2008: 845/928, 0/0
2009: 1390/1501, 0/0
TOT: 4356/4749, .917

Re: Khabibulin's contract, I'm not a fan. Too many years for a 36 year old, the Oilers failed to take advantage of the glut in the goalie market, and they bought him high coming off a season that he's unlikely to duplicate.

7/29/2009 2:46 am  
Blogger Jibblescribbits said...


I see now. I'm not sure I agree, I only wish I had more time to analyze this still, I may end up e-mailing you later on down the road, mainly because I am in the process of buying a house, and Shot Quality analysis isn't really high on my priorities list ATM.

It's interesting to say the least, but I'm not convinced yet. I'll bookmark it and analyze it further.

7/29/2009 7:51 am  
Blogger Vic Ferrari said...


Yeah, I made an error in arithmetic somewhere with Roloson.

And your thoughts on the Khabibulin deal are about the same as the vast majority all of the bloggers and commenters in this part of the net.

"Pretty good player, contract is a bit fat". You could say that about most of the Oiler roster.

7/29/2009 11:25 am  
Blogger Jibblescribbits said...

I think I have figured out why I don't like it, after a full day of ignoring the work I get paid to do.

I like what you did in finding the average EVsave% and then using that average to find the the variance due to the plain randomness of life.

But I don't like the comparison to the actual. When you average out the EVsave% variance over the 196 actual seasons you have eliminated the shot quality variable by essentially averaging it out.

In fact I think you got entirely expected results, whether opponents shot quality plays a role or not. I would expect, as more seasons accumulate that the shot quality against reaches equilibrium and that variation between the actual & predicted approaches equilibrium.

What I wouldn't expect is two-fold.

I would expect that the average variation for goalies who switch teams be measurably higher than the one's who don't. You have shown this is the case, by a decent margin. You have shown .0108 vs .0133 (.0103 vs .0122) That doesn't seem like a lot, but that's almost a 20% difference in variation just because of changing teams. A large portion of that could be the difference in shot quality against (And if you could eliminate the shot quality differences from non-randomness in players on the same team, such as coaching changes, personnel, etc. I would expect that gap to be even bigger.)

I also think the standard deviation would be significantly higher for the actual season's data when compared to the model seasons. That is data you haven't presented.

In summary; I think your analysis has averaged out shot quality out of the final equation, and you are getting an expected answer. I think a more meaningful number would be the Standard Deviation of EVSave%, however I think there are a lot of flaws with this as well.

Even this statement perplexes me: the difference in shot quality from any two teams selected at random ... the expected difference will be 2 goals on the season

But is that 2 goals ± 20 or 2 ± 1? So yeah the expected difference is 2 goals, but could it be anywhere between 2 and 20? I have no idea.

7/29/2009 12:41 pm  
Blogger Vic Ferrari said...


Again, absolute values are used.

The ONLY reason that the random model expects more change from the "switched teams" sample is because they, by and large, played fewer games and faced fewer shots than the goalies who tended to not get traded as much.

Go back to the model, it really is simple and repeatable, and the data is there for you in the link above.

BTW: For the 66 goalies that switched teams in the offseason, the std dev of the change in save% was .0170.

In the random sample, run 200 times, the std dev averaged .0155. Though in 46 of the 200 random trials the std. dev was actually wider in the model.

For that matter, in 38 of the 200 random trials the shift in EVsave% was greater than .0133.

So we can't say with any confidence at all that there is any shot quality difference between the teams in the sample whatsoever. But the fact that both the std dev and difference in save% are in the vicinity of the 80th percentile ... that suggests that it may exist. I was pretty generous in my original post in this regard.

To convince yourself: If you use the same basic model (66 goalies each with the same number of before and after EV shots as actually occured). Then assume that one of the various EV shot quality metrics is correct in it's ditribution of "shot quality effect" in the NHL.


Run that model and you'd be consistently expecting some absolutely madass swings in EVsave% (ABSOLUTE VALUES). And an enormous std dev most samples as well.

But it flat out didn't happen that way in the real world.

7/29/2009 2:50 pm  
Blogger Jibblescribbits said...

Again, absolute values are used.

I'm obviously not explaining myself very well, because i understand that and was accounting for that. I will try my hand at explaining my point of view later if i can get some free time.

7/29/2009 3:09 pm  
Blogger Vic Ferrari said...

Fair enough, I may well be misunderstanding.

7/29/2009 3:24 pm  
Blogger Zane Wooder said...

Can you imagine the pressure of being a goalie. Like what if you mess up badly and let a goal in. Your whole team is going to be riding you for making a stupid mistake. At least as a player your team mates are partially responsible for poor performance.

-Zane of ontario honey

9/30/2012 5:09 pm  
Blogger Jim Philips said...

It is part of their job to keep it up. So they have to have the best excuse. I really like the price per head free demo. It is all on you without excuses.

5/30/2013 10:58 pm  

Post a Comment

<< Home