​Editor's Opinion: The Search for Honest Game Reviews

Settle in and grab a cuppa

Back when I first started out in this industry – which somewhat tragically, was in the last century – I quickly began to realise that I didn’t agree with a lot of reviews that I read. A lot of them just didn’t speak to me. Top titles with big advertising budgets were being given generous four-page reviews and glowing praise that glossed over their obvious issues, whilst smaller releases (or games that were targeted toward smaller niches) were given a perfunctory half or quarter-page summary that wasn’t any use to anyone who had interest. Failing that, they were discussed mockingly. Just look at any late-90s British game magazine review of a wrestling title and try to find one that didn’t describe the action as “advanced cuddling” or make “hilarious” references to men rolling around in spandex. Oh, my aching sides. It didn’t matter that the games were some of the biggest selling in the industry and that the writers were alienating that audience. All that mattered was that their in-jokes landed.

The reason I began writing is that I thought I could do a better job, frankly. Without the sniping and the snark.

Into the Psycho Circus

So, fast forward a year or two and I was writing about games on all platforms for various publications, focusing mainly on the Dreamcast. As I ploughed through the waves of average and dire titles that accompanied gaming’s first push into the mainstream on the back of the PlayStation’s success, I started to notice a trend. Bad games weren’t being called out for being bad games. Again, grab any one of those magazines and take a look through. You’ll find that across a dozen or so reviews each month, not many ended up scoring less than five out of ten (or the equivalent) when there were many, many titles during that era that deserved to be lambasted. Hell, take the rose-tinted spectacles off and out of the 200 or so Dreamcast titles that were released in this country, way less than half of them warranted a score of 5/10 or more. Unless you really like shlock like KISS: Psycho Circus, that is.

When I switched to solely focusing on online work, the problem became more obvious. Titles like Gravity Games: Street. Vert. Dirt. (which is widely considered to be one of the worst PlayStation 2 games released by a bigger publisher) would pick up a few outlier scores – including at least one 9/10 – when the average across all outlets was a 2 or 3. Surely a game that’s bad enough to warrant that low an average shouldn’t be finding scores anywhere that put it on a par with Ico, Shenmue, and Halo? Surely, regardless of the reviewer’s feelings about the gameplay, to be picking up such low scores, it must have massive flaws that everyone should be able to see?

It happened frequently. I’d be playing games that were categorically bad and seeing them pick up average-to-good-to-great scores. Only, the sites giving the scores weren’t large enough to be seeing the rewards of say, a larger advertising buy from the game’s publisher, and neither were they big enough to really affect the industry’s overall perception of a game. But yet they’d give high scores to titles that only the most generous or dewy-eyed player would dare to go as far as to even brand as “average.”

I couldn’t work it out.

That Gerstmann Feeling

Skip on to 2007 and Jeff Gerstmann is dismissed as GameSpot’s editorial director over the “low” score that he gave Kane Lynch: Dead Men (a score that was entirely justified, for my money) at a time when the game’s publisher was paying heavy amounts of advertising cash to Gamespot. The resulting fallout meant that the man in the street finally started to cotton on to the fact that review scores could be used as tools by game publishers, outside of traditional advertising routes. Their ears were pricked up and they were seeking out corruption.

Another chronological step forward brings us to GamerGate, which is a can of worms that I’m not even about to go near, let alone open. But since that whole saga kicked off, the witch-hunt has ratcheted up even further. Reviewers at bigger sites are instantly assumed to be corrupt as soon as they dare to say one thing that someone disagrees with. People are calling for ethics in video games journalism and demanding “honest” reviews. That’s absolutely understandable. A review should be an honest opinion of the product that’s being covered, without outside interference. The problem is that no one review will be suitable for all people. I, for example, like the sport of horse racing and I always have. No, I can’t explain it. It’s something to do with all the numbers and stats involved, I think. But, the chances are that I will like a horse racing game more than 99% of people out there, so when a reviewer rates the latest iteration of the G1 Jockey series at 6/10, I’ll probably disagree one way or the other. What seems to happen with a lot of people who would be in my position though, is that they’ll run to the comments section to call out the reviewer for being corrupt. They’ll say that the writer has been paid by EA in order to “keep the Metacritic score down” so that FIFA is still the top sports game, or in the case of slightly more extreme score differentials, they’ll fire off a shot that says the review is “clickbait.”

As a writer, that last term offends me. People spend all day mashing links that say “You’ll never guess how this woman in Insert Your Town Name Here earns $2,938 a day!” and “Which 80′s TV Character Are You?” as if they’re hammering a button in order to get food, but the second I write “Assassin’s Creed Release Date Revealed” rather than “Assassin’s Creed comes out on September 28th” as a headline, I’m pushing clickbait. Heaven forbid I should want anyone to click on a link to read the article that I’ve spent time writing for the site, as opposed to wanting them to continue to pump money into Facebook’s coffers by reading all the details there, where I and the site get absolutely nothing for it. I know. I’m scum, right? Sorry you had to click and that everything I had to say wasn’t able to be stated in emojis and SMS notification sounds. I’ll do better.

Quantum Broke

Again, Jeff Gerstmann gets a mention, since clickbait – and other accusations – were levelled at him when he gave Quantum Break a score of 2/5 stars over at GiantBomb. He was obviously paid by Sony. He obviously just hates the Xbox One. He obviously scored the game that way so that it would appear lowest on Metacritic and people would click on it to see what the fuss is about. He obviously doesn’t know what he’s talking about. He’s obviously being “too reckless” since the developers’ jobs are on the line and they’ll all be fired if that Metacritic isn’t at least 80/100.

What an absolute crock. Especially the last one. If your job as a developer is on the line, make a better game. I don’t expect to retain clients if my work is subpar. Why should anyone else? Business isn’t romantic. Also, blame the publisher if that’s the case. Who in their right mind says “This game made us $25million in profit but the Metacritic was 78/100 so you’re all fired”? Answer: Very few people that you’d want to actually work for.

Anyway, Gerstmann gave HIS opinion of the game and that’s absolutely fine. It isn’t an opinion that will line up with everyone else’s. Our very own Dave Letcavage gave the game an 8 on this site. I respect Dave’s opinion a great deal, as I know many of our readers do. After playing the game though, I probably would have been thinking of a 7 as a score, possibly a 6 at times. People differ. A friend of mine gave it a spin on my console and absolutely hated the way it controlled, so would probably agree with Gerstmann’s lower rating. Two people on my Xbox friends list have booked over 50 hours of play with the game, so I’d imagine that they would give it a 9 or maybe even a 10. Some people like the TV show segments more than others, so they’ll get more from the video sequences than someone who doesn’t. Some may like the gunplay, when others favour the action in a different game. I said it before, but a review cannot be all things to all people. All it can be is one person’s opinion. All it SHOULD be is one person’s opinion.

Objection, Your Honour

At this point, someone will undoubtedly chime in with comments about objectivity and personal bias, and I’ll shoot them all down in the same way that I did the PR person who, in 2004, insisted that I was unprofessional for giving the game they were promoting a 4/10 as opposed to the average it was getting, which was a 6. They refused to provide me with any more review code ever (well, until they went bust, at least) because I refused to change my score, since I “clearly” had an agenda and didn’t know how to review objectively. I’ll say right here and now that they were right. I did have an agenda. My agenda was and is that I don’t like playing games that are crap when I could be playing games that are good. Freakin’ sue me.

There’s this feeling that people shouldn’t inject their own opinion into reviews, but that’s what a review is. It’s an opinion and yes, I’m gonna keep hammering that point. It isn’t the solid defining last word and it isn’t the gospel. It’s someone telling you their opinion of a product. There are limits to how far the personal leanings should go of course and part of the skill of reviewing – and there is a skill to it, believe me – is in managing those biases, but just imagine if reviews contained no opinion whatsoever. Every game would get an average score on every site. Reviews – as some see them now anyway – would be entirely pointless.

So, I’m going to assume that we’re agreed that people who want reviews should get honest ones. However, what we can’t do as a community, is demand honest reviews and then slap anyone in the face who dares to vary from the average by so much as a point or two. “I want you to be brutally honest about games because if you aren’t then you’re obviously corrupt, but you also have to really like the ones I like or otherwise you’re also obviously corrupt” isn’t something that you should be able to say without realizing what an absolute toolbag you’re being.

Look Both Ways

The killer in all of this is that those seeking corruption spend most of their time only looking at large sites with their accusation glasses on. Our pals at Nintendo Life – who are a big (if not the biggest) independent Nintendo site – are often mooted on the likes of NeoGAF as being “overly generous” in reviews, when the actuality of the situation is that across 2,248 titles, they’ve rated on average 12.1/100 points lower than other sites. (Metacritic, 4th May, 2016)

Just think about that. On average, when other sites give an 8, they give a 7 or a 6. When other sites give a 5, they give a 4 or a 3. Yet somehow they’re biased and give everything high scores to get people to buy Wii Us and to further the Nintendo cause. Let’s not let the publicly available and easily accessible facts get in the way, though.

We’re looking up, when the biggest problem isn’t coming from the biggest sites. I can tell you from experience that in general, the least honest game reviewers are the ones that write for smaller sites. The ones who claim to have sites made “by gamers, for gamers” and so on. They don’t have advertising – outside of Google AdSense, which they have no control over – so therefore aren’t in a position to see themselves having conflicts between marketing and editorial. But in a lot of cases – and again, just like with review scores, this statement doesn’t apply to everyone – oftentimes, the reason they started their publication in the first place is to get early access to games, to get review copies of games, and to generally be “in the know.” They want “swag” and recognition from developers, sometimes as a form of validation from the industry that they’re trying to squeeze a path into. A game-branded pack of playing cards quietly slipped in with a review disc is enough to add a point or two when it comes to some of the less honest writers, and why would a developer publicise a review you’ve sent them directly if you’ve only given the game a four or five out of ten? Best give it a big ol’ eight, eh? There’s your bait, kids. Rather, it’s “retweet bait” as opposed to clickbait.

So in the case of Gerstmann and Quantum Break, we have two camps. The massively experienced and highly visible reviewer who owns and writes for a site that gets millions of visits per day, that has a super-successful podcast (or two) and a subscription model that allows them to run heavily reduced advertising across the entire site is in one. He gets paid by his regular readers who subscribe, so lying to them is probably not his best shout. In the other camp is the guy that just started up a new site which he swears will only contain “honest” game reviews and who just got his first review code from an indie developer that has seven more games due to drop in the next 12 months. Also moving into his camp is the Twitch streamer or YouTube content creator that focuses on reviews or new game content and who is desperate to throw up a “Sponsored by” or “In association with” logo as if it proves legitimacy. If you can believe it, there’s even at least one (probably more) chap that publishes reviews via the medium of a single tweet. Just a screenshot and then “I give this game 8 out of 10″ and that’s enough to get some indie devs on the hook, given that they’re desperate for coverage in an ever-more saturated marketplace. And no, I’m not even remotely kidding.

Who out of those camps do you think is the easiest to corrupt? The answer is after the subheading.

Imagine Gameshow Music Here

If you haven’t worked it out, I’ll tell you. It’s the smaller site, or the YouTube creator, or the Twitch streamer. Every. Single. Time. Why would these guys be more likely to review a game honestly, potentially cutting off their review code stream, when their advertising revenue (if they have any) doesn’t cover the cost of a new game every six months, let alone one every week, and their entire business model requires them to have new games to play on a very regular basis? You can only make so many videos of a two-hour game before people get bored, after all. Don’t you think it would be easier for them to make a video called “THIS IS THE BETS INDIE GAEM YOUR MISSING OUT ON” and post it to Reddit so the folks there can upvote it and garner them more views?

In the last year, I have provided advice to one new site (I didn’t offer, they asked and I was feeling generous – must have been a good coffee day) that wanted to be taken seriously and who gave the first six games they had been provided to review a score of 9/10. All six. 9/10. Why? Well, they didn’t want to upset the people that gave them the code and couldn’t see the slightest thing wrong with that. Yes, the mind boggles. It’s not even blatant corruption. It’s that they had a genuine misunderstanding about the purpose that a review should serve. Fortunately, they aren’t listed on Metacritic and have since appeared to mend their ways, anyway.

Again, I’ll clarify and say that I’m not talking about every smaller site (we’re a smaller site, technically…well, not technically…we’re just a smaller site) and indeed not every YouTuber or Twitch streamer. But my point – yes, after 3,000 words I actually do have one – is that these are the people that are being pushed as the preferred alternative to the big boys who – without any sort of proof other than “I thought this game I’ve only seen a trailer of and haven’t played myself would get a 9 and it got a 6” – are so blatantly and clearly corrupt in a lot of people’s eyes. That means that rather than game journalism (reviews in particular) evolving along with the industry and feeding people’s need for legitimate and honest information, we’re pushing it backwards. We’re shouting and hollering and accusing good writers, driving readers away from honest opinion and pushing them towards the very people that are now at the beck and call of even the smallest indie developer, let alone the big publishers. That’s scary.

Don’t believe me? Go and look at the forums of some industry sites, where the proprietors of the smaller outlets are offering to “trade” PR contacts, because all it (apparently) takes is the right email address, and you’ve got a game to review. Yay! Free things! Go and look on Twitter, where people with Tumblrs and one-page WordPress blogs and YouTube channels with 10 subscribers are demanding review copies from independent developers and appear to be willing to do whatever it takes to get their hands on that good stuff. Reviews aren’t work. Review code isn’t a tool to allow you to do your job. You say nice things and you get free games, right?

Then go and check any number of game forums, or Reddit. Despite there being stacks of reviews to choose from on Metacritic, it won’t take long to find someone asking if a game is any good. They get a response from a good-natured person asking “Why don’t you read a review?” and subsequently, the answer that comes back is that the person wondering about the game’s quality doesn’t trust reviews from big sites, and is looking for an “honest personal opinion” from a smaller outlet, or YouTuber, or Twitch streamer.

Kind of ironic, if you think about it.

Just as clarification as to what we do here at Pure Xbox in terms of providing honest reviews, writers (with the exception of myself and Dave) aren’t generally in contact with publishers, developers, or PR folks. If someone related to the game asks for our review score before the review is live, I ignore the email. They can read it when everybody else can, since there’s usually only one reason they want the score before then. We don’t tweet our reviews to developers or publishers. We refuse to attend “closed” review sessions. We don’t review multiplayer games on “press-only” game servers. We will no longer hold reviews until “day one” patches are out, unless that patch is available in good time for our review to be written. There appears to be a lovely tactic to get around embargoes that’s growing in popularity, where reviewers are asked not to review a game until a patch is available, then the patch only drops a day before launch, making a launch day review all but impossible. Clever, huh? Finally, we will – when we can – post embargoed reviews at the very minute that the embargo expires, rather than waiting for our busiest traffic period. If that damages our reader numbers, so be it. I’d rather be poor and honest than a rich man with a dirty conscience.

If any developers or publishers have issues with any of this, they can feel free to email me – ken@purexbox.com – where the outcome will be that I’ll explain the situation and they can choose to either deal with it, or walk away, leaving me to buy their game and review it fairly anyway, whilst clearly stating why the review was late in the text. Sounds fair to me.

This entry passed through the Full-Text RSS service – if this is your content and you’re reading it on someone else’s site, please read the FAQ at fivefilters.org/content-only/faq.php#publishers.

Leave a Reply

Your email address will not be published. Required fields are marked *

Curated By Logo