Editor's note: Tadhg Kelly is a game designer with 20 years experience. He is the creator of leading game design blog What Games Are, and consults for many companies on game design and development. You can follow him on Twitter here.
In the age of better metrics for games, some developers take it too far and let the numbers make their creative decisions. This generally leads to trouble. However one of the better aspects of metrics is the idea that you can assess the overall performance of your game against someone else's. You can know how well you're really doing, or at least you should be able to.
In the screen trade they have done this for a long time through Nielsen ratings and BARB, to the point that audience numbers are a regular part of the critical conversation. Such information is valuable because it shows what seems to work in any given market. Every company in the television market knows where they stand relative to everyone else, and they can segment their efforts accordingly.
In the games industry such objectivity has long been absent, and this is because games is a multi-platform business. It's not just about game developers or publishers fighting it out, but of whole systems striving to be the dominant host of all games. The games business is what the television business would look like if Disney, ABC, NBC, CBS and Fox all sold their own TV sets in addition to their own programming. As a result of platforms trying to outdo one another, performance data is often hidden.
Sony does not necessarily want to lift the lid on just how poorly (or well) PSHome is doing on a day-to-day basis, for example, because that gives ammunition to Microsoft and Nintendo. Apple does not necessarily want the daily active user count of Rage of Bahamut to be query-able, preferring instead to use relative chart positioning. They don't want Google to then turn around and show that the game is played more on Android. And vice versa. Rumour may have it that Halo 4 is attracting a lot less concurrent play than previous Halo games but nobody external knows that for sure, and Microsoft isn't keen to tell us.
In the retail space there are some companies like NPD and GFK Chart-track that conduct market research and publish charts. This information often varies per country and sampling method, but it's seen as trustworthy enough to act as a rough guide to the state of the industry. In the online space, however, there's almost nothing to compare.
There are plenty of companies who provide metric solutions, but these tend to be for internal use only. Just like hardware platforms do (and sometimes for legal reasons as part of the terms-of-service of those platforms), the tendency of publishers is to keep their true numbers secret and only expose their vanity metrics. It's all too easy to claim you have a million users if you stretch the definition to a million registered email addresses, for example, or how many Facebook likes your page got.
Such numbers are generally useless for any kind of analysis because they cannot be validated, and do not conform to any standardised methods for collection. Their only value tends to be as sales points for VC types who might read Techcrunch, or for fuelling Reddit threads. But overall we may as well be using divination sticks for all the good they actually do.
This means that our efforts to understand how well the industry is doing only come from incomplete sources like the iOS charts. The Top Grossing chart is particularly popular among game makers, especially those who are fans of free-to-play games. It helps make the argument for certain kinds of games over others, but the data is all relative. Clash of Clans may sit atop the chart, but we don't know to what extent. Supercell may kindly tell us that the game coins half a million dollars a day, but we don't know if that performance has sustained. We don't know on what size of user base. We can't really verify it.
Ideally what is wanted is objective measures of daily active users. Up until very recently there was one platform that provided that information: Facebook. Through services like Appdata it was possible to track the MAU and DAU of games as well as their chart positions, and this was very valuable information. In the early days these metrics were key to wowing potential investors as to the size of the market and helped formulate many a business model. They also provided a positive story. You could see where CityVille got over 100 million users, and that information was reasonably objective, which was powerful.
Now Facebook has instituted a change that only provides banded information about games (whether they are in a 500K+ MAU game or a 10 million+ game, etc.) and charts. The reasons Facebook cites for doing this are to bring its reporting into line with other platforms like Google Play or Amazon, but it is a deeply disappointing change. Rather than knowing, we are back to guessing and supposing based on our historical understanding of the platform, and over time that will become inaccurate. This means that soon vanity metrics will follow, with studios announcing unverifiable numbers to grab headlines.
It feels like a gigantic step backward. Everybody in the industry knows that this sort of thing goes on, and how difficult it makes market research, but nobody wants to fix it for fear of losing face. If one publisher were to start to reveal its numbers then it would be vulnerable to attack from others who had no compunction to do likewise. Nobody wants to be the one who leads with their chin, even if they know that that's near-sighted thinking.
This is why the industry needs someone to solve the problem for it. It needs its own version of Nielsen or BARB, across all platforms. The question is how to do it: Behind closed doors at every major publisher there are accountants and forecasters constantly analysing their own performance and that of their competitors to derive a picture of what's going on. What will it take to bring them together?
Nielsen did it by polling. Samples of customers were asked to record their viewing activity, and this data was then projected and weighted to determine the market size for radio, and then television. Later this technique evolved into distributing meters to a sample size of homes across the United States. For tracking games a similar approach may be needed.
Perhaps a company like Raptr would be well-placed to create an app that regularly asks players to submit what they are playing. It already has the back-end for this in place, but instead of trying to create a social network of gamers (which has never really taken off), it could be used as a Nielsen equivalent. Or perhaps one of the metric companies could do it. Ideally the place to be would be to convince the industry as a whole to embed some black box code in games to report usage stats directly, but I suspect that's more of a pipe dream.
It's crazy that in an age where we have the tools to know more about our audience, the industry actively supports deception and hoodwinking. As an industry we should be able to compare the performances of games on tablets versus those in browser, or console, or Steam to further open an honest debate. Otherwise we are just kidding ourselves and making up stories about our games that we want to hear.
Better playing through data yo.
No hay comentarios:
Publicar un comentario