The Misuse of Metacritic

There’s this teaching that I remember from my youth. A knife is just a knife, it has no inherent good or bad to it. In the hands of a surgeon or chef, it can be used for immense good. In the hands of a killer or thief, it can be used for bad. This pretty much sums up my opinion of Metacritic. It’s a tool. No, not that kind of tool - the other kind. There is a misunderstanding of what Metacritic is and how it can should be used, and a lot of the problems surrounding Metacritic stems from its misuse.

Firstly, let’s deal with how a score is determined there. Metacritic is a “weighted” average. That means that they place more emphasis on the opinions of some reviewers as opposed to others. If IGN gives a game 6/10 and XGN gives the same game an 8/10, the average score is NOT 7/10 (unless both publications have the exact same weight). The weight is determined by “quality and overall stature” of the publication. In this case, depending on the opinion of Metacritic, a 6 and an 8 can average to any number between 6 and 8. IGN has the greater stature and hence, the average of 6 and 8 can turn out to be 6.5. Of course, nobody knows which sites are ranked where, because Metacritic does not share that data. I love XGN by the way.

How does Metacritic decide which scores to use? Good question. According to their FAQ, they “work to identify publications that (1) are well-regarded in the industry and are known for quality reviews; (2) actually seem to produce quality reviews (or, if not, are so influential in the industry that they have to be included); and (3) have published a good quantity of reviews”. Read that second one again. Read what is in the parentheses. That’s right folks, poor quality is not a dealbreaker for Metacritic. The size of your following is a(the?) major factor.

Another factor in the production of a Metacritic score is scale conversion. Metacritic gives a score out of 100. Many sites do not follow that. Some operate on a scale of 4 and others a scale of 5. If you give a game 2/4, Metacritic equates that to 50/100. Not everyone agrees with that. Most importantly, the critics who produce the scores do not agree with that approach. If you want proof, here’s a video of Adam Sessler complaining about it. https://www.youtube.com/wat...

So, with all this, is Metacritic useless? Not really. Metacritic can be used as a broad indicator of game quality. For example, if I see any game with a score over 65, then I think it’s worth looking into. Everyone has their own cutoff point. I do not have the time to research every game made (and neither should you). My next step is either to buy the game or investigate further via reviews or videos. It depends on the actual score and genre involved. If I see 9.5 and RPG then I’m sold. Again, you don’t have to agree with my approach, I’m just giving you an idea as to how I use it to align with my tastes. I am very happy using this approach – you may not be. The key point I’m trying to make is that it is a good ESTIMATION of quality as determined by the herd. Even their color scheme alludes to that. Any game with a score above 75 is given a green highlight. The implication is that they all meet some common notion of what a good quality game is. You know what? I pretty much agree. With a few exceptions, most worthwhile games (for me) do get a score above 75, even though experience has taught me to lower the bar a bit.

For me, that’s it. That is pretty much the extent to which Metacritic proves itself useful. However, not everyone agrees with that. There are those that view Metacritic scores as some sort of holy divining rod to deem which game is best, to determine which platform has the best games or the best exclusives. They represent a good example as to what happens when you give a toddler a knife – (spoiler) nothing good happens.

So many people like to use Metacritic as a basis to decide whether game A is better than game B. Is this logical? – well, yes and no. If game A has a score of 95 and game B has a score of 45, then, despite the shortcomings that I previously outlined, it’s a safe bet that A is better. These are not the arguments you see though. The arguments usually go along the lines of “Game A has a 98 and game B has a 96. Game A is obviously better”. No. This has no logic to it. Let me explain why.

Firstly, two points on a 100 point scale is not a statistically significant difference. Second, Metacritic’s sample size is small, just a handful of reviewers – there’s not enough data to reliably establish significance. Third, the reviewers for each game are different. If you take a Nintendo game and a Sony exclusive and compare the contributors for Metacritic, you will realize that they represent two entirely different groups of people. The websites used are very different and even when the same website is found on both lists, the people involved are different. For example, IGN has one group of people who report on Nintendo games and another tasked to Playstation games. There is very little overlap. I could go on, but I really don’t see the point. Also, my statistics knowledge is way rusty – it’s been a while and I’m feeling lazy.

But what if you still want to compare two games? Could it be done? To objectively measure something, you first have to define it. How do you define quality? This was one of the questions explored by Robert M Pirsig in his critically acclaimed book “Zen and the Art of Motorcycle Maintenance: an Inquiry into Values”. The author, with an IQ of 170, chose to ponder this question as a young man. It drove him insane and years later, after electroconvulsive therapy to “zap” him back to normal, he wrote the book which outlined this journey. The synopsis is that quality cannot be defined in a rational way, it can only noticed when it happens. Quality is the "knife-edge" of experience, found only in the present, known or at least potentially accessible to all of "us". Confused? Me too, and based on what happened to Pirsig, I’m happy to maintain my ignorance.

Still not Happy? No problem. While we cannot measure quality objectively, we can establish consensus. Here’s an approximation of what can be done.
1. You identify the critics you want to use. The more the merrier (and the more accurate).

2. You give each of these critics the same group of games to score.

3. Then, you rank the scores. Eg if they were given 10 games, rank 1 describes the best in the lot, rank 10 describes the worse.

4. Once you’ve done that, you test for correlation between ranks of reviewers. For example, you can use the Spearman’s Rank Order Correlation

The purpose of all this is to get a group of critics with statistically similar tastes. Then you give them the two games in question. But wait, what if the games were produced years apart? You see, quality is temporally affected and is determined by associations and analogs. Ocarina of time got really high scores because it was so much better than the other games produced at this time. So, before reviewing Ocarina of time, the reviewers will have to play other games produced at that time in order to get a fair assessment of its relative quality.

Even that procedure that I just outlined is filled to the brim with shortcomings. For example, critics can all have the same bad or biased tastes. My purpose is not to give you an exact methodology. I want you to understand how much work and effort is involved in responsibly comparing 2 games and how far Metacritic (and other critical aggregators) fall short is fulfilling these requirements. Metacritic was never designed for this.

Your obsession with Metacritic has not gone unnoticed. There is now the industry practice of rewarding developers based on Metacritic scores. One famous example is where Obsidian missed out on royalties because their game got a Metacritic score of 84. They needed an 85. One point on a bullshit scale cost a studio and its hard-working employees money. More recently, it was shown that Bungie had a similar clause in their contract with Destiny. This is what bothers me the most. Should a poorly understood and poorly explained critical scale really have that kind of power? Are we seriously going to incorporate this into the way we do business in the industry? So now, compensation for my 5 years of work on a game comes down to what 50 people on the internet think and how some smoke and mirrors scale divines a score using an algorithm that nobody has seen/verified. No wonder people laugh at the gaming industry - at times we deserve ridicule. Nobody outside of Metacritic knows exactly how a Metascore is calculated. It reads like a recipe for disaster as far as I'm concerned. It’s not the fault of the knife. Should we ban knives? No – they have their uses. What we must do is educate and train the users and it is my hope that I achieved this in some small way today. Now I’m off to write some letters to publishers. Feel free to join me.