[Editor's note: This is a guest post by Brice Morrison, former CrowdStar designer and editor of The Game Prodigy, a new site for game designers.]

“No no, you don’t understand. Most players don’t even get to level 22. Most of them drop out before then. We need to release the feature earlier.”

“That’s not true! A lot of our players are at that level, which is why we need to focus on more gameplay at that stage.”

I can’t tell you how many design discussions I’ve been in during my career. Discussions over what players want, what is the best way to spend our engineering and art resources, and what the spirit of our game is. When working on consoles and retail PC titles, we only had only limited feedback available to make decisions. Playtests, surveys, and playing the game ourselves before it was on the shelf were really the only ways to back up claims. And once the discs were out the door, then it was all said and done.

With social games however, it is a much different world. On Facebook or MySpace, the launch of a game title is just the beginning. Today the amount of data that developers are able to mine and manipulate is massive. How many players are at each level today. How much money players spend on what items. How long they play, when they play, and what they all do when they’re playing. Whereas other platforms require developers to adopt a “set it and forget it” attitude towards their games, social games spew mountains of feedback to their developers from launch day, allowing them to tweak and improve their social games based on player behavior.

Developers who understand and know how to make use of this data will have incredible perception into the behavior of their players and a leg up over their competitors. Let’s look at some of the advantages that data mining can provide.

Better Researched Decisions

For years, “classically trained” game designers in the industry have relied on their gut instincts to make decisions as to what players want. Much of this was based on intuition, imagining that yes, of course players would enjoy attacking other spaceships more than building and upgrading their own. Or a whiteboard discussion revolving around how long an average session is – a few minutes, for an hour, or for hours at a time. Or maybe an argument about player’s play styles – do they prefer to level up in order to improve stats, or do they like to spend money on items?

In the past, discussions and arguments like this are usually resolved by whichever designer can make the best points and steer the conversation towards their personal conclusion. While this approach is often effective if the development team is a talented one, it is often faulty and can produce decisions that don’t reflect player’s actual behavior.

By being able to pull live data from a game, arguments like this can be resolved almost instantly. In the middle of a shouting match on whether or not players like to upgrade their buildings every time they log in, someone can say, “Hey, guys, I looked it up, and yes, actually players level 1-30 upgrade 4 buildings on average every time they log in.” Everyone nods their head, makes the decision, and moves on. Or in a heated discussion of whether or not players need to be given more money in the game, someone can say, “Hey everyone, here is a graph showing the amounts of soft currency that people have right now. You can see that actually, most players have three times too much.” Income is slashed, expensive items are put on the market, and the job is done.

Sometimes knowing a key stat is just what the doctor ordered. By filling in the blanks as to what players are actually doing (not just what it seems like they’re doing), developers can learn to back up their decisions with data.

Experimenting

One of the best capabilities of data and instant user feedback is the ability it gives social game designers to experiment. There are often many options for the next feature that could be implemented in a game, and the right choice as to which will lift DAU, increase monetization, or improve retention to the greatest degree is not always clear.

By being able to get feedback quickly, developers can see how players react and change, tweak, or can the feature in no time flat. Releasing a new upgrade system that will give an advantage to paying users? No problem, let’s try it out on a few, watch them, and see how it affects their games. Wondering whether the unicorn fountain would sell better as a constructable viral item or a Facebook Credit purchase? Do an A/B test and look at the results tomorrow. Designing an experiment, putting it out into the wild, and then looking at the results is the best way to rapidly develop a hit title.

Experimenting is especially important in social games, where many of the designs that are now standard in the industry didn’t even exist several years ago. As Facebook and other social platforms have exploded in popularity, new designs needed to be created, tested, and discarded in darwinian fashion in order for companies to succeed. Without being able to look at the numbers quickly and decide what is and isn’t working, it would have been unlikely that the industry would have matured as quickly as it has.

Optimizing Where It Counts

Few game platforms have the luxury of being able to tweak their game to perfection. Sometimes just a few simple tweaks, a few numbers changed, or a few tuning values moved is all it takes to go from an also-ran to a blockbuster hit. There are so many numbers in social games that it can be overwhelming: the amount of XP to get to level 5. The amount of daily income that players receive at level 3. The cost of the in-game car. The exchange rate of 200 coins to Facebook Credits. It can be intimidating to realize that setting those numbers to 120, 17, 700 and 10 could result in $100 in revenue, while setting them to 115, 12, 900 and 12 could result in 10x that amount for your company.

Luckily, by being able to tweak different numbers endlessly and monitor the results, social game developers are able to optimize the tuning of their game until it’s nearly perfect. If players are making too little money, then the developer can just tick up their income a bit and see the results. If players are going too slowly, then they can increase the XP given or give out a bonus to see the effects. Watching these metrics and measuring them against the lifetime value indicators for players is one of the primary uses of massive data.

While these are all wonderful advantages to being able to mine massive data from players in real time, there are also many dangerous pitfalls. Teams who are suddenly blessed with being able to look at their players activity in detail without the experience of how to handle it can run into trouble. Let’s look at some of the common missteps that data can cause for social game teams.

Data Paralysis

Data is great for helping social game developers to make optimizations. However, game development isn’t just about making small optimizations. Sometimes for games that are doing mediocre or downright poorly, what’s really needed is a large new feature, a fundamental design change, or a reboot that better captures the essence of the game.

For example, let’s say we know that players consistently log in to get money from their city buildings twice a day, but we also see that the city building gameplay is causing long term problems, such as players becoming fatigued, racking up huge amounts of currency, or dropping out. This puts us in a tough spot: the right thing to do is to remove the feature from the game and try something else. But then what if they just stop logging in altogether? We have the data, and the data is telling us that is a bad idea. The downside is clear; the upside is fuzzy.

Changes like this aren’t easily quantified. But if DAU or ARPU are dropping like a rock, then something needs to be done. In cases like these, dependency on data can lead to paralysis, with designers not knowing what to do either way. They are able to easily quantify the problem but not the solution.

In cases like these the remedy is to use other traditional tools of the game development trade. It’s important to have someone on the team with a strong vision, an understanding of what players want backed up by playtests, surveys, and focus groups, and the leadership capabilities to lead the team into unknown territory. By not being completely dependent on data to make decisions, experienced developers will understand that sometimes the numbers aren’t there to help make the decision, but a decision will still need to be made. Having a robust system beyond data for making the best decision can help keep the team going.

Vanity Metrics

In social games, there is such a thing as too much information. When the numbers are something that you care about, then everyone likes looking at a graph. Any graph. A graph showing how many cultural exchanges players lost versus the week before. A graph showing the trend of raspberries farmed on average by level 55 players. A graph that shows the percentage increase of 25+ day users who built a chop shop week over week.

Ok, that sounds interesting, but what can we do with that? Does any of that actually matter? Does knowing it actually increase our bottom line, our DAU, or our players’ lifetime value?

Graphs are pretty, they make you feel like you know something, and they give you a concrete number to hold onto, giving the impression that you know and understand your game very well. However, not all numbers, graphs, or tables are actionable. I like to call these “vanity metrics”. They’re cool to have, and developers are willing to spend enormous amounts of time mining, producing, and debating them, but they don’t provide truly useful action items.

A good test for whether something is a vanity metric is for developers to ask themselves this question: if this number was radically different, would we act any differently? If the answer is no, then the data is useless and focus should be shifted to more important matters.

Vanity metrics are dangerous for two reasons. One, by spending time analyzing data that is actually useless, it takes away from time that could be spent developing features or improving the business. Second, by constantly sinking time into numbers that don’t matter, teams may begin to lose faith in data all together, eventually missing out on the metrics that do matter.

So when going to mine massive amounts of data, teams need to make sure that they know what they would do with the data first. Drowning in numbers is no way to improve your decision making.

Another Tool in the Toolbelt

Being able to pull instant data from games is a wonderful tool for developers to have in their belts. Flying blind may have worked in the past, but today the bar is higher than ever before, and if a game is designed or tuned poorly, there are plenty of high-quality competitors to flee to, competitors who have studied player behavior and chiseled the experience for maximum enjoyment. It’s important to remember that massive data on player behavior is a supplement to good design practices, not a replacement. But when used well, data can give invaluable insights into a company’s players and game.

A game designer who has worked at EA and CrowdStar, Brice Morrison is the editor of game design website The Game Prodigy and has been with teams for major titles like The Sims 3, Happy Aquarium, and It Girl. You can find more resources on game design from Brice and other industry designers at http://thegameprodigy.com/.