The Real Problem with Metacritic - Sessler views

[-HP-]

Really good points on Metacritic.

[Nysuatro]

yes, very very frustrating topic

[KoKo5oVaR]

You could also add that metacritic seems to use a coefficient of worthiness depending from where the review is coming from which is not visible anywhere. And that game journalists opinion, especially for those doing websites only, can be easily bought by editors. It's hard to believe in any objectivity at all

[Mazy]

Indeed. Another aspect that is somewhat disturbing is if it dictates greenlight meetings early on by getting 3rd party companies to rate a game, and base major decisions on that. The entire industry is so mesmerised by metascores at this point that it doesn't even stop up to realize what the hell it's doing.

I'm all for ways of measuring games in terms of informing whether it's worth your time or not, but this number crunching is ridiculous. I'm not the biggest fan of Kotaku, but that's actually something I really like about them. Their score is simply whether you should get a game or not. I wish more sites would favor that approach.

[Taylor]

Most games I’ve worked on have obsessed over the metascore, which is doubly unfortunate given they’re all pretty low. One even had a fake metacritic chart where the bigwigs would play through the latest build and come up a plus or minus on our running, imagined metascore. Boy am I glad they decided to drop that individual metascore nonsense.

I've got nothing against metacritic itself as a concept, but it's not a force for good while publishers obsess over it.

Edited April 9, 2013 by Taylor

[FMPONE]

this is not new. if you listen to howard stern (reminiscing about the days of terrestrial radio) he bitches about the radio "rating" system, similar to neilsen. it apparently benefits the big players in media to use obscure rubrics, probably because higher-up execs can cloud the system for their bosses, keeping their jobs despite not doing particularly much

Edited April 9, 2013 by FMPONE

[2d-chris]

Yup, nothing new, all media have respective rating systems that gets in the way of development in one way or another.

Publishers and developers have obsessed since the begging of time about what makes other games successful, vast amounts of time, energy and money are spent on looking into the competition and trying to simply replicate the formula. This sounds great until you realize in a 3 year development cycle times have moved on and you're now "old hat"

The games that are rated highest simply execute the concept the best or are relatively fresh compared to recent trends. Metacritic is simply a score system not a way to guarantee high sales and strong reception from the press.

[Furyo]

Furthermore (as I agree with every point made so far), a study last year showed that Metacritic score only ever impacts sales when that score is 90+. And there again, you could make the argument that the score only reflects a universally acclaimed title that sells so well precisely because it's universally acclaimed and NOT because suddenly your neighbor and his goldfish are browsing Metacritic (Chicken and egg argument).

When the future of your game/studio hangs in the balance of Metacritic (never happened to me, but I know it has elsewhere); something is seriously fucked up.

We should really wake up and smell the ashes now, with the state of the industry as a whole. There is no 100% fail safe recipe for determining what will sell. End of story.

[Izuno]

I deal with this issue all the time with both developers and publishers.

If you look at average sales of console games in most genres and compare all games with metacritic scores 90+ against all games with 80 to 89 scores to all games with 70 to 79 scores etc...then yeah, the average sales in the 90+ range are highest, followed by average sales of games in the 80s and so forth.

But then the correlations of review scores to unit sales (again, talking console games where we have reasonably good data) rarely are above 60% and usually much lower, but the genres where it's highest (like T & M rated RPGs on 360/PS3) it gets a little higher, but that's because you have megahits like Skyrim stretching out the scatter plot and making the correlation increase.

But beyond that, in meetings with suits where they talk about metacritic scores of past games as a guide for what kind of game to greenlight I have to constantly bring up examples of games that scored well but sold poorly or scored poorly and sold well to try to get them to understand it is so much more complicated than a linear regression and a number.

Drives me crazy sometimes, but then again I often get paid to explain this to suits so they can sit down, shut up and think through WITH their developer about the game rather than cite some bullshit regression model.

[Kedhrin]

God i hate Metacritic.

[blackdog]

I deal with this issue all the time with both developers and publishers.

[...]

in meetings with suits where they talk about metacritic scores of past games as a guide for what kind of game to greenlight I have to constantly bring up examples of games that scored well but sold poorly or scored poorly and sold well to try to get them to understand it is so much more complicated than a linear regression and a number.

All this looks schizophrenic.

I mean: they want to refer to Metacritic to greenlight a production, then they ("supposedly") manipulate the press to get higher scores and (arguably) sell better... so they know Metacritic cannot be a trustful source of data.

Also, looks like they sometime manipulate the press to get lower scores and not pay bonuses?

It's nice there's someone that tries to educate them btw.

Edited April 11, 2013 by blackdog

[Sentura]

http://kotaku.com/me...games-472462218

There’s the story of the mocked mock reviewer, for example. Some background: game publishers and developers often hire consultants or game critics to come into their offices, play early copies of games, and write up mock reviews that predict how those games will perform on Metacritic. Often, if possible, publishers and developers will make changes to their games based on what those mock reviews say. Mock reviewers are then ethically prohibited from writing consumer reviews of that game, as they have taken money from the publisher.

One developer–a high-ranking studio employee who we’ll call Ed–told me he hired someone to write a mock review, then threw that review in the shredder. Ed didn’t care what was inside. He just wanted to make sure the reviewer–a notoriously fickle scorer–couldn’t review his studio’s game. Ed knew that by eliminating at least that one potentially-negative review score from contention, he could skew the Metascore higher. Checkmate.

Edited April 11, 2013 by Sentura

[blackdog]

Stumbled on this, thought I could share:

95+ : The expression “95+” comes from the fact that all our games are ultimately reviewed and judged by critics and gamers. The Metacritic Score is an aggregation of reviews, measured from 1-100, where 100 is perfection. If our games would review around 60 or 70 it would be a disaster, but luckily we have never been in those regions. It is our DNA to aim for 90 or higher, thus, we talk about “95+” as a concept. In order to deliver 95+ games, you need to provide a 95+ office, and as much as possible, your responsibility as a Leader is to prepare a 95+ canvas for those that create the game. It means that you have to be relentless in your pursuit of Quality. There is no such thing as Quality Overkill!

— at Massive office

From this album, this picture.

[Sentura]

deer god....

[Taylor]