Playtesting and iterative development are the secret weapons for game developers. By understanding what players currently think or do, we can make changes to improve games pre-launch. This leads to more successful launches, and a game that people love.
In my career, I’ve spent over 25,000 hours with players and seen the value of playtesting first- hand. Big game studios employ user researchers, or hire outside agencies to run playtests throughout development because they know it works. Many studios don’t have the resources to dedicate a full-time role to organising playtests, though, and are forced to do it themselves.
I’ve been fascinated in understanding this space, and have spent the last year interviewing people who organise playtests in addition to their ‘main’ role – including solo developers, designers, producers, community managers, QA, UX designers, and more. I asked them about how they ran playtests and what made it difficult – hoping that we could bring some of the experience of user research to make playtesting easier for everyone.
“What this game needs is a shotgun”. Remember: not all suggestions need to be implemented. We’d be intrigued to play Animal Crossing with shotguns, though, don’t get us wrong.
These interviews revealed some easy-to-implement changes to how some teams currently run playtests, that incorporate best practices from UX and user research. This includes how to find appropriate playtesters, how to interpret playtest data, and how to find and answer the most important questions throughout development. I heard repeatedly that finding money is difficult, so I’m going to try and keep the cost of a playtest below £50, but describe options for when more budget is available.
RUN PLAYTESTS EARLY
It’s really tempting to put off playtesting until later. Sentiments such as “It’ll be better when the final graphics are in” and “What’s the point if we don’t have the tutorial ready” can be used to delay playtesting indefinitely. It’s also scary to put your ideas out into the world and expose them to critique, so it’s an understandable human reaction to hesitate.
This is extremely dangerous for game development, however, where the best ideas are exposed and refined through iteration. Reducing playtesting takes away opportunities for iteration and leads to a disappointing experience at launch (and hasty post-launch changes!). Instead, it’s best practice to tie your playtesting into decision-making. Consider what decisions your team is making currently – across all the design disciplines. That might include traversal, combat, level design, UI, etc. Then rank them by risk – which are most crucial to the success of the game, and which would derail the game if they weren’t experienced by players as expected.
Then test those risky things. Now. While someone is still thinking about designing and implementing it. This sometimes requires creativity – mocking up tutorials that aren’t in yet, telling players to ignore some known bugs, or dropping players in half-finished, grey box experiences. But it does mean you get immediate feedback on what players don’t understand, aren’t able to do, or aren’t experiencing in the right way – while still in the position to react.
Live playtesting can be as simple as putting your phone in someone’s hands.
These playtests don’t need to be huge – watching five people go through a five-minute grey box experience might be enough. Planning regular playtesting linked with design decisions, rather than ‘one big playtest’ at beta, leads to more reactive game development, and ultimately more successful games.
GET THE RIGHT KIND OF PLAYERS
During my interviews with game developers, I heard repeatedly how difficult it is to get playtesters. Teams rely on friends, family members, or a network of fans to playtest their game. Not because they are the right kind of players, but because they’re the only people around.
Unfortunately, this can seriously compromise the value of playtesting. They’re not like your ‘typical’ players, and what you learn won’t be applicable to the audience who’ll buy your game. Some won’t play this kind of game and so might behave strangely in it. Many will have pre-existing knowledge about the game that a ‘real’ player wouldn’t have, which will influence their behaviour in the game. Others will know you personally and not feel comfortable giving critical feedback.
Start by defining your players. Ignore demographics and focus on their behaviour – what other games will people who buy your game usually play? What games have they bought recently? How often have they bought a game this year? Asking these sort of questions allows you to narrow down a profile of what kind of person would buy your game.
Now we need to find these people. This is the most impactful place to spend our £50 to make playtesting better.
A great place to find people is Reddit. Ignore the dedicated playtesting forums, as the participants there are largely devs themselves. Instead, look to the more general help forums, such as r/forhire or (the unfortunately named) r/slavelabour. Use your player definition to create a ‘screening’ questionnaire, to weed out the wrong type of players. Offer everyone who passes a £10 Amazon voucher to take part in playtesting.
If you’re doing your playtesting in real life (rather than online), coffee shops, university campuses, and game shops are great places to catch people with some free time – but remember to use your screener to check they’re the right kind of player. If you have more budget available to you, professional research participant recruitment companies (like research-i in the UK) can take away all of the burden – you tell them who you want, when you want them to turn up, and then they do all the hard work to make it happen.
Video conferencing tools allow players to take part in live playtests wherever they are.
MAKE THE MOST OF PLAYER TIME
As we’ve heard, it’s difficult to find the right players to take part in playtests. Making the most of the ones we do find is sensible. Many studios are most comfortable with a survey or free-form chat – asking players to fill out a questionnaire after playing the game, or writing their comments in a Discord channel. This is because it’s easiest to administer, and feels like it takes the least time.
This does miss a lot of the potential from playtesting, though. Surveys are best set up to measure numbers (how do players rate this level, how many times did players attempt it), but a lot of what we want to learn isn’t a number. Understanding why these numbers occurred is a lot more useful for inspiring decisions – such as why players give that rating, or why it took them 20 attempts to complete the level. For most stages of development, there’s infinitely more useful, actionable information from studying a few players in depth than getting shallow insights from a large group. To do this, we want to watch people actually play the game.
This can be done live by putting them in front of the game, or giving them your phone with a build installed, and watching them play. Recording the session (and what the player says) will allow you to analyse it in detail later. In person this can be done easily, or you can do it remotely screen sharing over Google Meet or Zoom (which also records the screen and audio for you).
Failing to playtest your game early can lead to a disappointing launch and all kinds of post-release fixes. On an unrelated note, here’s Anthem.
Live sessions aren’t always possible, so we’ve also had success with asking players to record themselves playing and talk out loud about their thoughts. OBS is free screen-recording software that can be used to ask players to record their screen and voice while playing the game – although it can be a bit technical to set up, so players will need instructions. If you have budget available, websites like PlaytestCloud or Antidote can handle this for you – but they do have a fee.
DEALING WITH PLAYER DATA
Playtests generate a lot of data. There’s behavioural data (what people did when they played) and opinion data (what people said about it, or how they rated it). Teams can find large quantities of uncategorised data overwhelming and difficult to deal with. To tackle this, treat the data differently based on what it is.
Behavioural data is generated by watching people play, or analytics in the game. This is the safest data to take action on. Hopefully we’ve designed a playtest where we can watch players do something in the game, and understand why they did it. For example, seeing a player fail to find where to spend their in-game currency because they didn’t recognise the right menu option, or players wandering the wrong way on a level because they didn’t see a door.
This behavioural data is easiest to interpret. Having understood why the issue occurred (the player didn’t see the button), it’s possible to decide what an appropriate fix could be (put a label on the button). Then understanding what impact it had on the player (they got lost), you can decide how important it is to fix it (it meant they never finished the level). When we know how hard a fix would be, and how big a deal the problem is, we can make an informed decision about the priority of fixing it and schedule that in the backlog of tasks.
Opinion data is trickier, and is where I see teams go wrong most often. Players will tell you what is wrong with the game, or what should be different. “This enemy is annoying as it’s hard to beat” or “This game would be better if you had a shotgun”. The worst thing you can do at that time is to immediately go and implement their suggestion. Instead, we need to step back and add some context.
A thorough analysis of the cause and impact of issues will make it easier to decide what to do.
This requires first understanding why they gave that feedback. What happened in their gameplay that caused them to describe the enemy as annoying, or say that they needed a shotgun. This can be uncovered by asking them questions, or watching recordings of their gameplay. Then compare that experience to the experience you intended to make. Perhaps the enemy was meant to be annoying, or the horde was supposed to be overwhelming, and the player was receiving the emotional experience you intended. In that case, no action is necessary.
If their experience isn’t the one you intend, by understanding how it differed, you can generate potential ways to change the player experience. Remember, you’re the professional game designer – not them. By pushing their proposed solutions back into the ‘problem space’, you can hopefully come up with a wider range of options for fixing it, and create a more successful experience.
TOOLS FOR PLAYTESTING
I’ve made some free templates to help with the logistics of playtesting: a checklist for planning the study, a non-disclosure agreement, and some tools for defining and finding your players. You can download them all for free from playtestkit.com. They’re part of a growing suite of tools I’m making to help make playtesting easier. If you’d like to read more about playtesting, also see Jesse Schell’s The Art Of Game Design which has a chapter on advice about playtesting.
