⁠Steve Bromley⁠ is a highly regarded figure in the UX research community, renowned for his expertise in game user research and playtesting. He has authored numerous books aimed at developing and guiding researchers. In this session, we will delve into the distinctive aspects of conducting playtests, explore his insights gained from both the gaming and software development industries, and discuss the future of UX research.

Listen to the episode

Also available on all major podcast platforms and Youtube.

Transcript

The following transcript is automated and raw; thus will have numerous mistakes.

Vy: Hey, welcome back. Welcome to another episode of Experience Design Podcast. And in this session, I’m going to talk to Steve Bromley. And Steve is one of the prominent names in playtesting and user research world. He has worked with a lot of known gaming titles. But also has established quite a few research focused teams.

And today we’re going to talk about, I guess, play testing and to end it. What does it mean? How does that compare with, uh, you know, uh, regular software development, user testing, the intricacies, the parallels, what we can learn from it, what we can actually do better. And the second part of the session, we’re going to focus entirely on the future.

UX as a capability, but also user research, how that might evolve. And also how far away are we from that full on automation of user research and design. So do enjoy this episode as per usual. If you like it, make sure to share it, subscribe to wherever you’re listening or watching this. And on that note.[00:01:00] 

Do enjoy. Almost start with a deep dive. What has been your journey like to get to user research and also so kind of like a diverse user research and play test, you know, for gaming experiences, but also like outside it, what was your journey 

Steve Bromley: like? So I’ve been lucky enough to work in and around the games industry for a decade, about 11 or 12 years.

I started my career as a junior user researcher at PlayStation. They have a user research team based in London, Europe, who are responsible for a lot of the European games that PlayStation publish. That’s a really nice place, or at least I found that a really nice place to learn how to be a user researcher and what user research looks like in the games industry.

Because as a publisher, they get exposed to a whole bunch of different studios and different titles of different sizes. Some of those big AAA games when they’ve got thousands of people working on it for many years. Some of those through their smaller publishing sites might be small teams of five people making a tiny game.

And again, there’s [00:02:00] different types of research challenges and really interesting how the approach you might take to user research or the impacts that you can have with a user research study can change within different scopes. I was at the PlayStation for about five years. Uh, before I then looked to, looked for new roles.

I did some work outside of the games industry. I set up a research team at Parliament here in the UK. I set up a research team for a publisher. But I’ve always kept somewhat invested inside the game user research community. Throughout all that time when I was working outside of games, I was still doing mentoring.

I’d set up the IGDA program for… How to be a games user researcher and helping people learn what the role is like, and if they wanted to do it, partnering them with people in industry. And then I summed that all up in a book, uh, How to be a Games User Researcher. Since then, I think that’s pushed me much deeper back into games.

So that’s led to teams wanting support with establishing user research in their own company. Uh, the… From running [00:03:00] play tests and research. And so now I, I’m back doing a lot more work in the games industry than, than I had been for a few years. And that’s They pull you back in, right? Exactly. Just when I thought I was out, they pulled me back in.

Yeah. 

Vy: But it’s very interesting because typically people come from a lot of different backgrounds, right? And they tend to kind of, and, and this is maybe my observation and limited threat, but we tend to start with like web apps. testing for interaction design, like in my head, quite more simplistic challenges, but it sounds like you went head first into gaming research, right?

The immediately or. I 

Steve Bromley: think so. Well, obviously games are very complex and there’s a lot of things you could be understanding. However, also my own experience is that the maturity of user research and games can be less than outside of tech. So although games are very complex systems, and there’s a lot that you could be understanding, at least a decade ago when I first got started, a lot of focus was purely on usability.

Games come from that unique perspective of it’s both a business, [00:04:00] they’re making a product to sell. But also somewhat a creative medium that it’s art and people are making, putting themselves and their creativity into it. Because of that, a lot of the, the best practice that you find in tech and other types of user research about, well, let’s understand our users, let’s uncover user needs, let’s frame them as opportunities, and then let’s see what we can make that meets those needs, hadn’t really resonated and still doesn’t have a huge amount of cutthroat in game development.

Instead the start of software development, often of game development often thinks about what’s our vision and what, what’s our creative goal? And that pushes the role of user research into being evaluative and usability testing. And often those more simple studies that I think are user research outside might start their career doing usability testing, but eventually realized I should be doing discovery and I should be doing ethnography and I should be doing the, the broader skillsets that might be changing and, and I can see.

as is increasing [00:05:00] cross influence between tech and game user research, maybe that mindset will change, but at least traditionally it has been focused a lot on, on evaluative testing and usability. 

Vy: But I guess, so in that case, the challenges probably are quite similar that if let’s say you’re a researcher and you find that, okay, ethnography, let’s say is a way to go.

You kind of need to push for it, right? Like the, the, gaming. Would it be fair to say that the resistance then is a bit higher too? I wonder like what’s, you know, what’s the actual tactical challenge there for, uh, 

Steve Bromley: everyone? My perception is, is that the pushback is higher and that’s, and part of the challenge is, so just a couple seconds, can you hear my cat in the background?

I can hear a little footsteps. If that’s, if that’s too much, let me know. And I’ll go do it. No, it’s 

Vy: fine. It’s fine. I’m sure nobody would mind either. So nice. 

Steve Bromley: I perceive the, the challenge is different in the angles in which we have to take it to convince people can [00:06:00] be different. With traditional software development, optimizing

for. the success of the company or the success of, of, of this product, you can frame your, your argument for why we should be doing discovery research or why we should be doing something ethnographic around opportunities and it will help us land on the right product faster and things that are meaningful at that exact level where everyone wears suits to work.

And although that still exists in games and games is still a business. There is also that parallel track of half the people there are doing it because of passion and because of creativity and putting things into the world. And I think the immediate resistance you might get is, well, is this going to reduce my ability to be creative?

Are you going to tell me what I have to make and how it has to work? And that’s not why I work in games. If I wanted that, I would be working on other types of software. So as user researchers, we have to be very careful in how [00:07:00] we frame what we want to do around helping designers realise their vision.

And allay any fears that we’re going to change your vision or, or stop you from doing your vision. Make sure that they understand we’re trying to help them get the best possible version of their vision out into the world. I had a 

Vy: session and a chat with Michelle P just before this one. And she works as a community manager at Fatshark Games.

And she was very specific saying that it’s almost like a balancing act, but sometimes you might find something from the audience, but it’s not necessarily the type of game you want to make. And I feel like that’s where, like that balancing act, that’s where I’m like very intrigued to hear your thoughts.

What happens 

Steve Bromley: then? I, I’ve definitely seen that and I think it definitely occurs where Through scoping a research study with the team, they have very specific objectives and they often can be usability focused, for example, or at least about evaluating the experience. Take the players find this level fun.

Do they like this mechanic? Those types of things. [00:08:00] I think part of our duty as user researchers is look at the scope critically. So rather than just asking the questions directly related to The objectives and what we want to learn, taking it back a layer and trying to put it in context. So starting up a research session, for example, by asking, what are you playing currently?

Why are you playing those games? How do those games fit into your life? When did you play it? All those types of things that help us build up that broader picture of a user and who they are. And then whether teams want it or not, sometimes that can be the more important thing. So again, that comes back to how we report it.

When we’re going back to the team, often a report might look like we’ve learned all these answers to questions you had. Also, we learned this other thing that is, if anything more important, that if you, even if you executed your vision perfectly. No one would play it for these reasons. And so instead, we need to think about this and ideate on this part of it as well.

So your question was, how often does that happen? I think as researchers, we should be trying to encourage that to happen as often as [00:09:00] possible. Make sure that we are open to realizing that the, our assumptions about player behavior and how players act might be wrong. Make sure we are creating the opportunity to learn those things, to learn things about the user’s context, and then at least bring that back to teams so they can make a conscious decision about whether they They care about that or not, rather than unknowingly making a game that no one will play on launch.

Vy: Even if, I guess, the nature of playtests to me, as an outsider, sounds like it happens quite late, like you have to have something to give. to your players to try out and then see what is the life of a researcher is then, I guess, is it, is it a lot of that evaluative activities, as you mentioned before, or do you do something as I guess, ideas are shaping up as well.

And, and what was your experience like from all those years? 

Steve Bromley: Yeah. So prototyping does happen in games. They make gray box version to the game where there’s, there’s. It doesn’t look pretty, but some of the core mechanics exist [00:10:00] and players can do stuff in the game. So in theory, you can start to do some mixture of evaluative research and also provide inspiration for the new tangents for the game to go down quite early on when they’ve got that first prototype, or they might have what they call a vertical slice where they’ve made essentially a demo of the game.

However, My experience with the games industry, uh, it’s going to be a blessing and a curse. So, when teams say playtesting, that can mean a whole bunch of different things. And playtesting as a concept has existed in the games industry as long as games have been made. Like, all the way back to the 70s, Atari was sending people into the arcades to watch people play and see, see how people And because of that, the games industry has some preconceptions about how playtesting works and what playtesting looks like.

That type and exploring our, our virtual slice is often an internal only thing with no external users and it’s just us, us playing it together to [00:11:00] see how it feels. And real playtesting where you’re getting lots of outside players in is usually pushed for a latent development. So, uh, when they’re finalizing or polishing, they’re then going to look at difficulty and balance and that might be when they start to ramp up those type of studies.

I guess one of our roles as user researchers, especially with based on our experience in other industries and understanding how it works elsewhere is to take that ethos or that idea of playtesting exposed to teams. Actually, there’s a whole bunch of risks here. You shouldn’t be doing it just with internal users.

You shouldn’t only focus on it right at the end of development because you won’t get to act on your findings and then encourage them to run discrete but smaller tests throughout development. Which I think is one of the big challenges that we have inside the games industry. And it’s a 

Vy: perfect segue as well.

I have a quote, which I borrowed from the playtest survey report, and it says almost half of all teams run playtests with less than [00:12:00] 10 playtesters while others are running huge tests anywhere from 100 to 2000 play testers. And then I guess the theme drawn out of that was very interesting to me, which was saying that it also emphasized any amount of play testing with two to 2000 players has value.

Steve Bromley: Because the game industry has grown up with the habit of ad hoc guerrilla research run by game developers always. They haven’t had that idea of, well, let’s match the methods and let’s match the sample, let’s match the number of participants to our research objectives. So because of that, the, the approach they take do vary wildly.

And you do often find teams making decisions based on convenience. Like, well, we’ll put two people in front of this because we can find two people rather than doing proper scoping for how they approach it. 2000 

Vy: players sounds very large and I, and, and I guess from one side it’s not that big, but still a lot of the people who are listening might be kind of blown away.

Like, this is, this is a lot of like a [00:13:00] big sample. To do this, what sort of objective would you even try to address or look into if that type of sample? 

Steve Bromley: Yeah, great question. Um, so 2000 is not very large and obviously pushes you towards some methods like surveys or hands off methods because you as an individual can’t do that depth of quality of research, the type of things that you might that teams might be using a large sample sizes for.

includes difficulty and balance. So they might have, here’s a whole bunch of levels and we’re looking for where there is a statistically significant change in players perception of the difficulty or how often players fail. And with smaller studies, you will see some variation. You’ll see, hey, players rate level three much harder or players are failing more often on level seven, but not to the appropriate degree of statistical rigor.

And So when you do have those measurement questions, especially for live service games, for example, games that are already out in in the world and they have a lot of players [00:14:00] going through them, it’s reasonably practical for them to implement a survey or implement some sort of tracking to look for those types of.

Bumps or balance issues. So I think that might be what pushes people towards their larger sample sizes. To 

Vy: pull back slightly, what you mentioned before was, uh, I wanted to get back to was that balancing act as well between research and what you want to make. Like, how do you keep your biases at bay? I guess, because I imagine, you know, you’re part of this bigger team of designers, developers, researchers.

How do you kind of pull away? Like it’s much easier, I guess, in software development, I think, because. It’s easier to separate the jobs and kind of, you know, especially if you’re developing something which you don’t use yourself. But in gaming, like almost anyone can game, I guess, or anyone could appreciate.

And I’m sure you have your favorite titles, which could be very different from what your developer is making, right? 

Steve Bromley: Yeah. And I do see that as a common problem. I guess the first step is making sure the team have an agreed [00:15:00] vision on what game they think they’re making. Uh, making sure that they are defining, Hey, what is our vision?

What type of experience do we want players to be having? How do we want players to feel as they go through it? What bits do we want to be fun? And that is something that either a researcher or designer or, uh, a producer can do work to, to generate. They can understand what is the team’s vision and document that.

When it comes down to actually running a study, And as a researcher, I guess to make sure that, and this isn’t unique to games, that we have to be very careful about separating what is the problem, where are we seeing a deviation from the intended experience for players, and making sure that we’re capturing and communicating that.

But separating that from the discussion about what should we do about it, what is our recommendation for action. Some researchers and some research teams don’t make recommendations, they see that as the role of… That’s a design decision. Our role is to be an objective. Here’s where it’s deviating, team, and then your, this other role is the one [00:16:00] who’s going to come up with the answers.

Some, like myself, believe in such a hybrid approach where we’re responsible to make sure the team do agree on what they’re meant to do next. But, that idea doesn’t have to come from us, and so we might run a, run a workshop and encourage teams to ideate around different ideas and land on which one they want to do.

One of the more challenging elements to stay unbiased is if you are doing more of a fusion person who does research roles, so you are a designer who is testing your own game, and part of the day you’re doing design, and part of the day you’re working. You’re evaluating this, I guess my advice for there is being upfront with your thinking about what am I trying to to evaluate in the study?

What will the success metrics be? Like what does a fail look like versus what does success look like? And then trying to make that an objective discussion about or objective experiment about is this, is this game creating experience? We think it is as far as possible. Although with emotions that can often be very messy and games are meant to [00:17:00] create emotional 

Vy: experiences.

Has there ever been a case where it was a total flop or like close to, you know, no resonance? I guess it’s, I also presume that a lot of studios make those games because they have a good understanding of what experience look like, what resonates in the market, things of that nature. But is there ever a case where it would happen?

Uh, and maybe theoretically even to speculate, like, why would that happen? 

Steve Bromley: Yeah, I, I, there are cases, I’ll avoid making, mentioning a specific game. So I’ve seen it happen. For example, games are often based on IP, intellectual property, and so their license. And to get those kind of deals together and to make sure that they work, you often have to make a lot of upfront commitment about, here’s what the game’s going to be, here’s how it’s going to work, here’s how we’ll introduce your IP, and also that might also introduce limitations on what you can do.

You can do in the world that, that constrain the options available to you. Just like with all [00:18:00] technology, that means you’ve, you’ve suddenly become very waterfall. You’ve defined exactly what you’re going to make without having created any opportunities to see, Hey, does this work for players or will people enjoy that?

And I have seen that on, and some titles I’ve worked on lead to underwhelming launches and, uh, underwhelming products when they’ve had to say what they’re going to make, they’ve made it. And just created no opportunity for iteration. So to answer your question, I guess, a factor can be intellectual property and constraints in that kind of business relationship, I guess, as a second reason is just that trap that all software development falls into of trying to over define at the beginning what you’re making.

Without giving yourself enough opportunity to learn and pivot as you go down that journey. 

Vy: I guess in terms of parallels with software development, if you launch, if you invest a lot of time and as games do, let’s say, but imagine in the, in the old school software development waterfall model, where you would take something away and work on it for a couple of years, [00:19:00] you deliver a tool that could be disastrous for business.

Like that’s almost like it’s so hard to. shape some sort of bad product into something people would use versus kind of do it in a agile or close to agile iterative approach, but with games and with false, I guess, you know, disaster scenarios, like how would you pick it back up? Like what would be strategic thinking?

Steve Bromley: Yeah. Games is particularly vulnerable to it, at least traditionally, because although it’s changing these days with. the growth of mobile games and live service games where, or early access where a thing is out in the world and then they continue developing. Traditionally games were a boxed product that sells most of its copies over the launch month and so the first exposure and the first time it gets into the world also have, it has to be good at that point because that, it’s only going to get one shot.

And if your game launches and it’s unsuccessful, or doesn’t meet the quality bar that they’re [00:20:00] aiming for on launch, the whole thing’s a write off. So, games are very vulnerable to that. I guess it’s business model changes that has created new opportunities for game studios to, to employ agile methodology and to, to adapt to that.

I’ve mentioned a couple of times the idea of live service games, where a game like Fortnite, for example, has been out for half a decade or more. And keeps getting new content updates. And the idea is that it keeps you subscribed as a player and playing over time, rather than just being a 40 hour experience that you play on launch and then never play again.

And that creates the opportunity for, for iteration. You might also be familiar with the idea of early access, where teams are now starting to publicly sell their game when it’s in a beta period or when it’s earlier in development than traditional. Again, that creates more opportunities for iteration and feedback and seeing resonance.

Historically though, it has been difficult for games to claw back from a poor launch. There are examples, even with Fortnite for example. The [00:21:00] original thing that launched was a completely different game mode and didn’t have the degree of success that Fortnite has now. And it’s only when they released the Battle Royale mode that this started and it picked up.

So it was good for them that they had the opportunity to stick with the game, iterate game modes and find success. Similar to titles like No Man’s Sky, where they’re obviously under a lot of pressure to launch at a specific date. Uh, what they launched with didn’t meet the expectations of much of the audience.

However, that wasn’t the end of the team. They were given the financial freedom and time to stick with it and iterate and eventually deliver on what they promised. I guess it’s just business stops at being possible for many companies because it is a business. Publishers might also choose to cut their losses and leave, leave it after launch, especially if they don’t think there’s going to be any significant pickup after.

Rather than dedicate further resources to iteration. Yeah, 

Vy: that’s really interesting. And one thing that came to my mind was the Cyberpunk [00:22:00] 2077, which launched to, I guess, to very large audience and a lot of fans because the community there is vast, you know, from Witcher series and so forth. And, uh, I enjoyed that game, even if it was buggy as hell, but now I picked up Phantom Liberty DLC and, and the original things, I didn’t even know that they kept updating it.

And kind of chipping at it. And that, that was almost like that Phoenix story where the game, I don’t even know if it could be classified as failure, but it wasn’t a good reception. And I think it’s still kind of has that shadow of people kind of looking at it and saying, well, it wasn’t that good. They build it up and I guess it could be capital.

It could be a lot of backing. It could be again, very loyal fan base, a lot of those variables kind of feeding in. Like it, it truly impressed me because I was thinking that, you know, they’re going to be done. I enjoyed the game, but most people won’t. But then you look at the history of actual maker, the same or similar thing from what they read, you know, here and there in the community [00:23:00] happened to Witcher series too.

Like I think Witcher 3 Wild Hunt was also. Launched with so many bugs and probably maybe a year too early. Um, and it was kind of like a totally defeated in a way in that case, but, but that’s where it’s, you know, yet another example of, uh, kind of pulling, pulling yourself up, I 

Steve Bromley: guess. No, I don’t know. This is, I’m not saying this is good practice, but I guess what’s interesting with Cyberpunk and No Man’s Sky is because of the amount of marketing and the amount of attention on it.

Even though the actual product might not have expectations, it’s sold enough to fund them long enough to then iterate and make the thing they wanted to make. I probably wouldn’t recommend that as a, as a strategy, but I also understand that again, being a very hit driven business and having to hit deadlines.

For all those marketing spends and for big times like Christmas or Black Friday does push teams to sometimes having to release before they’re ready. So it’s good it worked out. But [00:24:00] you 

Vy: can’t, I guess, do that in software development. You know, even if you go with a small product or big product, it’s unless you’re a massive Big tech powerhouse.

It could be the end of it. That’s where I feel like massive difference there is to, to me, games are so uniquely positioned where they sort of tend to, at least the big names kind of take it away for a decade. And then come out with like this shiny title, if that makes sense. So it’s very different world.

Steve Bromley: Yeah. Like. In my opinion, I guess traditional software development feels safer because you, a lot of software is based on understanding user needs, understanding problems people have, and then positioning, creating a thing that solves that problem, it can create a lot more. upfront confidence that people will want your thing by the time you launch because you know you’re solving a real problem for people.

Because games aren’t so relevant for user needs in the same way, like I said they’re all primarily positioned as entertainment, that means there’s a lot less that you can do upfront to build confidence that there will be an audience for this, [00:25:00] that it will resonate with people. Those kind of conversations and research challenges are a lot harder.

So yeah, I can see how the approach of tech is much safer for development in many ways, which I guess leads to high A high failure rate with game studios overall. And I guess when 

Vy: you went, uh, just kind of digging more on those differences, uh, in, in terms of the worlds and things, was, was it the culture shock to you when you, you know, went from, I guess, in your early career at VAD, when you went from play tests to maybe, you know, let’s call it more generic tests, you know, for lack of a better term, like, like what, what, what was your kind of like, how did you experience that?

Steve Bromley: Yeah, I guess two things come to mind. One is I think immediately you miss the, the feeling of a game to use a research study because it is entertainment and it’s a field that people care passionately about. When you bring in a player to, to play a game from a series that they love and it’s an [00:26:00] unannounced game, so they, they didn’t know anything about it.

That’s probably the most exciting thing they’ve done that year. And so a lot of them, the actual moderation and the time with participants can be entertaining and fun and a nice environment to be in games. So I guess the actual resonance of the study and how the actual data collection bit goes can, can be different.

On a broader though, and on a research skills level, part of the reason why I wanted to look outside games and make sure that I wasn’t focused only on games was because of that over indexing of, on evaluative research and usability research. I was vaguely aware that. There’s other research methods out there and you should be doing generative research, we should be understanding user needs, and just had no exposure to that.

So I think that was a deep learning journey for me. Okay, what does study look like in that space? How do you run those studies? How do you make them relevant and compelling and how do you use that to inform product development? Again, we’ve alluded a bit to how for games user research [00:27:00] and game development in general, how important marketing is and how important it has to be a big, big launch.

And because of that, part of the culture is extreme secrecy that people don’t talk about the studies they’ve run. They What I’ve learned in using research for games, because there is a very active community out there who will take it and then it will run on the internet. And then it will ruin the marketing for a game.

That has some implications on both the mechanics of how we run research studies. So leaning heavily into ways to minimize leaks, such as if we’re doing in person testing, making sure we can, we remove NDAs, it might push you away from unmoderated or remote research because you have less control over.

where the build goes and what can happen to, to your game. And so that has some implication on what studies look like. Similarly, because the research objectives can often be more quant focused, they can be looking for difficulty, or they can be looking for the emotional response you get from a game, [00:28:00] and the length of content the games can be very long.

That has a difference in the shape of how these studies look as well. So at most publishers, I know at PlayStation we have this, but also Microsoft or Activision or other teams, they’ll have a gaming lab that has 20 or 40 or 80 pods set up and bring in members of the public to, to come and play the games in the lab setting at that mass scale.

That’s partly due to leaks and avoiding the chance that the game gets out there because we can’t do those types of studies remotely. And also, a way to answer those quant questions. That means that, yeah, running those studies can be different, because moderating a group of 80 of single players, and has a whole bunch of implications on how you would set up a research study, the actual operationalising of that research.

Strategic bottlenecks, I guess, are similar. When do you have a game in a good enough state that you can safely run those type of studies? How do you [00:29:00] convince teams that you’re not going to leak the game and it won’t end up on, on Reddit tomorrow are some of the barriers that we might face. 

Vy: Yeah. And I guess the incentive system probably is also has to adapt a bit, right?

Or even perhaps it might take a bit longer than, you know, doing something remote or something you can kind of easily source through, I don’t know, usertesting. com or, or communities or things of that nature is you presumably need to assess the candidates. Quite rigorously or segment them 

Steve Bromley: in a way.

Definitely. I, I have some concerns about those type of unmoderated tools in any context, even tech, just because I think we all have experienced people become professional testers and they’re on, they’re on usetting. com, but they’re also on all these other websites as well. And they’re doing it 24 seven and increasingly become professionalized or trained for what they want to hear.

So I guess I always have reservations about these platforms, but the games, I think it’s especially important. We need to make sure they play the right [00:30:00] games, they play the right genres, that we can trust them when they turn up and do a study. And sometimes it can be immediately obvious if, if they haven’t, or if they are, have lies to get through at the study.

Again, probably experience from playing games, a very common control method is dual sticks. We use both sticks to move. And one of the skills you learn early on in playing games is how to use one stick to move and one to look around. But that isn’t a natural behaviour for people. And so sometimes you’ve had people who have lied to get through screening or turned up inappropriately, and they just can’t get grips on the two thumb sticks because they’ll press both forward, run around looking at the floor all the time because they’ve pressed both sticks forward to move.

And those can be the immediate signs that actually there is a misrecruit here. This isn’t This is someone who is sneaking into tests, um, incorrectly and the type of things we should be looking out for. I think it 

Vy: happens to everyone. If you research enough, you’re going to encounter some of, uh, yeah, like, you [00:31:00] know, what, what I would label as like a tourist or someone who comes in and, you know, for, for all the wrong reasons and then doesn’t benefit anyone ultimately.

But, um, I’m sure people who listen, especially those who are experienced in research, we definitely recognize that. Um, one of the things which I really, really wanted to kind of dig. You know, and pick your brain on, um, what’s the future of research as a whole and not necessarily maybe play tests, but perhaps play this too.

I don’t know how you want to kind of help us out, but how would you think user research is going to shape up given AI emerging tech, you know, the LLM and everything in between, which is basically. Kind of escalating a lot of things. And the listeners I’m sure are going to be considering even should they even get into UXR from junior perspective, but also those who exist in the field.

Like I’m thinking about it almost daily. Um, so I wonder what are your perspectives? 

Steve Bromley: So I think some fundamentals will still be true. People will still need to see. Users using our software or understand users [00:32:00] in the world to inform product and design decisions. The role we’re filling will still be important and hopefully will continue to be recognized as important.

I’m dubious about AI’s role in being able to take over some of these roles. However, that doesn’t mean that I think everything is going to be okay. So I guess the obvious implication is users are going to talk, they’re going to think aloud as they go through a thing. And it’s reasonably simple to get chap TBT.

So analyzing the transcript of what people said replaces part of a user research roles, but not, not completely. Obviously you should also be observing and understanding. I’m dubious, though, whether a summary or a transcription of what users are saying is what researchers should be doing in the first place.

I think if all you’re doing is that kind of stenography and saying some users said this, your findings will be vague, it won’t be reactionable. I think those are the implications. These often indicate that you are a junior researcher and a thing you have to learn and get through. I haven’t seen strong evidence that I think [00:33:00] it can really replace the challenges of doing proper depth user research.

I’m sure the researchers who are listening will have encountered this. A lot of the challenge of being a user researcher isn’t actually the research or the analysis. It’s making people care. It’s getting a team in the right state, learning the right questions, making sure that we’re getting the right data to answer that, and then making it land at the other side, making sure that teams understand and are using what you’re learning.

I don’t see an AI replacing that role. However, I did say that I, I did have some concerns and AI can currently do, and what I can see it doing in the short term is only a shallow interpretation of a research role. I think the market for a shallow interpretation of research role will be huge. I think there’s a lot of teams who would like a checkbox of, Hey, we’ve done user research, or we’ve done usability testing, but don’t currently have that mindset of, oh, actually it’s good if we prove ourselves wrong.

And the point of this is to learn and to evolve. As opposed to just thinking it’s a thing we do because we have to do, or it’s a thing I can do to [00:34:00] prove my designs were right. And so I think there is going to be a huge market for AI tools, for AI user research to happen. And that will be a challenge for the industry to differentiate what do we think real user research looks like?

Where do we think the value is of doing that? And then making sure that that isn’t the same as what a tool can do. Yeah, 

Vy: but when you say, uh, shallow research, uh, 

Steve Bromley: how do you mean? Well, what’s possible for AI currently is listening to a bunch of things that people have said, and then aggregating that into themes and saying, players said that the game was, the controls were, were difficult to use.

And I, I think that’s okay, but often, and you see, again, see this in junior researchers, often doesn’t give enough granularity or specifics. For context, for a designer to actually take that and then do something as a result of it. You, if you’re going to change controls, would need to [00:35:00] know in what context were the controls not working, why did the controls not work for these players, what was the impact of that so we can prioritize it appropriately, and understand the issue in the round rather than just say, hey, players are saying it’s difficult.

AI can definitely do that, players are saying it’s a difficult thing, but then breaking that down into enough factors that someone can act on it. I think it’s beyond it currently. Yeah. 

Vy: Like to me, it’s also, you know, if we take like a specifically software development world, if the challenges are quite tied to a single product, I see a lot of applications and you know, I, I feel there’s going to be a lot of change in product design realm specifically, but what are your thoughts on like UX as a whole, how do you view.

UX evolving as a, as a set of disciplines or, you know, people coming together. 

Steve Bromley: I think I can only give a perspective from a, from a games development perspective. And I see in game development that UX has often had a challenge [00:36:00] to cut through as a, as a distinct discipline. I guess a classical understanding of what a user experience person is doing, assuming that we’re moving beyond, oh, they just make some wireframes or they just make some menus.

Would be, we’re meant to understand the holistic user experience, like what is happening to a user, how does a user feel, and then make sure we’re making conscious choices about, about that. That it all joins up and it all goes together to make the experience we want to make for users. A challenge I’ve seen inside games is that is a discipline that, to some extent, historically already existed.

But if you ask a game designer what they do, they would say, well, I’m creating an emotional experience for a user, or I’m putting these systems together to make it, make a distinct output from it. Perhaps not with the same framing exactly as UX about how users experience them. This should be the primary thing where we’re optimizing and same skill sets and same thinking was already existing.

Because of that conflict, UX has struggled to [00:37:00] cut through in, in games as a distinct discipline. And where it does exist, it often falls back to, uh, what you’ve described as UX UI. Someone who’s primarily designing an interface, designing menus, or describing, designing the heads up display in a game. And I guess their UX thing is just a mindset.

They’re trying to do that while thinking about psychology, while deciding, advocating for testing, but they’re not ultimately responsible for the end to end experience that they’re making. You asked about the future, and again, I can only talk about from a games perspective. I think that challenge will continue to exist.

I, I think there is a chance that UX as a distinct discipline, just be incorporated as a mindset into other, other skills. So the UX, UI designer will eventually realize, okay, now I am primarily UI designer, but I’m thinking about the user experience and then every other role will be doing the same. So the game designer continues to game design.

But with a, a [00:38:00] user experience mindset behind it, ideally your programmer is doing programming, but thinking about the user. So I think it can be framed more as a mindset within games, and distinct discipline might be the direction in which it’s going. Yeah, that’s 

Vy: very interesting. And I feel like it’s kind of already happening too.

Um, one of the examples I love to bring is the product manager. Uh, basically if you, if you view like a spider or, or radar diagram and you have like three axes. Usually one of the axes or one of the Venn diagram bubbles is going to be UX. That’s really telling because we had such a uprise in, let’s say, product management, which is such a new generalist role and UX being part of that, a big chunk means that now we have to kind of share it.

And I think the new product managers know, know it. Um, not every UXer, so to speak, I think appreciates that. And I’ve seen that, you know, in practical terms, where it’s a lot of kind of internal competition happening, user centricity as a whole, it’s going to become like a [00:39:00] super shared skillset, which anyone can apply probably better or worse.

But. If you think about it, even UXers do it better or worse, you know, there is like, you never get to a point where you’re just like a superstar UXer or researcher. So, like, you are always, it’s just a matter of challenges, I guess, kind of like overlapping. Yeah, and 

Steve Bromley: I guess a challenge will be. As it does become a shared skill set that everyone has the same mindset around, can businesses justify an individual role or will they want to justify an individual role about, yeah, you’re the person who really cares about this though, rather than just accepting that everyone does a bit of it.

I can see that. Being a challenge to the growth of the discipline. Yeah. 

Vy: And I guess another signal is, is merging of the titles or new titles and, you know, a flavor of UX, let’s say in, in marketing or sales can become customer experience. And there’s just so many other, other bits, but I wonder if like we could zoom out even further.

Um, Let’s say 10 years or so from now. I don’t, I don’t [00:40:00] know if you had any thoughts and I’m sure you haven’t, you know, maybe, maybe not, not this specificity, but like, how do you feel your job is going to look like? 

Steve Bromley: That fundamental need of putting people in front of the things we’re making and seeing if their experience is positive and working out where we should change.

I guess some of the logistics of where, how to do it. will get easier over time. You might invent, and I know there are some companies that already do things like this, you might invent safer ways of being able to run remote research for games that reduce the chance of leaks. And so your methods might change.

Recruitment is often difficult. You might find ways to speed up or automate recruitment so that your turnaround on studies can, can be quicker. So I think technology can speed up the process and create ability to do more. iterative testing than we currently are for an individual or for myself, I guess also that, that trend of.

As it becomes easier, there becomes less need for it to be a dedicated role, might be true. And so, [00:41:00] in helping teams run it themselves might be the direction that some user researchers go. Although, as we’ve already alluded to, that’s a slippery slope, that eventually ends up with the role not existing if everyone can just do it themselves.

But I also… I wonder if that might be a direction of travel for user research as well, as a role as well. I 

Vy: mean, it’s like, it’s going to escalate. I have no idea to what extent, I guess, and you know, we cannot predict the future. Like we shouldn’t even maybe try. But do you feel like there’s always, always going to be a need for a human counterpart, especially in gaming research?

Steve Bromley: And as you probably got, I’m somewhat dubious about the application of AI, but I might look silly in 10 years when you play this clip back to me. Um, partly because again, especially as, as I’ve gone up in seniority, the real challenges are people shaped challenges. They’re making people understand, making people care, making your colleagues engage and listen and, and take part and create more opportunities to do these type of things.

It’s less so on the execution of AI.[00:42:00] 

was excellent at the execution of the study and took that over. They would be low impact. They wouldn’t lead to better products and software and game development without solving the messy human challenges. And I think relationships and internal politics and dynamics between individuals and teams will continue to be a thing that needs humans to navigate.

We kind 

Vy: of covered the resistance or maybe a bit of cautiousness from people who want to get in, but they’re unsure, but also product teams out there who are, you know, trying to become a bit more user centered in gaming and also software development. Do you have any, 

Steve Bromley: any advice for both? So the tendency I see in game teams is to imagine that they need to do one big study and they need to learn everything in one study.

And I understand some of the pressures towards that, but it’s. There’s obviously an admin overhead of running a study, there is a, uh, you need to have the code, the game in a state that [00:43:00] can stand doing a test. And those factors push teams to both delay research and also then do one giant study when they get to the point of doing it.

Whereas we know from lots of these research studies and also looking at how it’s applied in general technology, you get in trouble by doing that. You end up learning stuff, but not having the ability to actually act on it because You’ve built dependent systems on top of it, it’s too late, your attention has moved on elsewhere, and so the actual impact of your study becomes very low.

The direction I hope to encourage teams to go in is doing smaller, tightly scoped studies, regularly having meetings to think about what decisions are we making currently, what’s the biggest risks to us, and how can understanding users help us de risk that, or help us prioritise our decisions, or help us decide what we need to do next.

And by doing that, you can come up with a lot smaller studies that are possible to execute on a much smaller time frame, rather than having to wait and run one big one. So although that is a very [00:44:00] common habit in games, I think that same lesson might apply to product teams trying to be more user centric.

Rather than thinking the user test has to be a big thing that we’ll do at the end of a big milestone and we’ll look at users then, instead just making sure you’re regularly having those conversations about what decisions are we making, which of these are risky, are any of them so risky that we should at least learn something about users to de risk that.

And then running perhaps a very small study. will just help you get better data at the right time to, to inform your product development. I think it is primarily about the, the cadence and the scoping of studies. So thinking they have to be big. Thinking that we should do that milestones rather than matching the cadence to decision making when are we making decisions that we should should test should be how teams think about when we should be running 

Vy: studies.

Maybe whoever is going to listen in a few months is not. I’m not even going to understand, but I’m [00:45:00] sure it’s still going to persist a bit. Like the market and layoffs are still happening. Like, how do you view the market itself? And do you have any advice for people who are struggling to find a new 

Steve Bromley: role?

So it is a difficult time. And even in good times, games can be a competitive industry to get into because it is a passion field. People work in the games industry because they, they like games and they really want to do it. Uh, there’s a lot of competition. Because we talk about some of their software development processes being very waterfall, studios do fail and they have layoffs, and so it’s not very stable.

And so because of that, it can be hard to get a reliable and stable job in games. However, people love it, people want to do it, and so if you knowingly are going into it and want to work in games, uh, some advice? Hiring managers are primarily looking for any experience of running research studies. So if you can show I’ve done interviews, I’ve done a usability test, I can do a heuristic review, that’s the thing [00:46:00] to, that should be prime on your CV.

But beyond that, game development believes its development process is special or unique, and the words they use aren’t the same as in other software development. The job titles aren’t the same as other types of software development, and they want to know that you have some sort of understanding of that.

That can be from reading some books. There’s a really good book called A Playful Production Process, which helps explain how games are actually made, and the different stages, and the different roles. And also from some sort of application. One of the things that I do when, when working with them to go and find a hobbyist developer, or a small indie studio, and then run a practice project with them.

Do a heuristic review, for example. And think about how you apply your UX and user research skills to the type of challenges that games have and the type of questions that game developers have. I think if you have any sort of things like that to show an interest in game development and anything actually [00:47:00] applying that, you will stand out as a very strong candidate when you’re looking for these type of roles.

And that can be a reason why often very experienced senior researchers from outside of games struggle getting through the interviews and in a game’s role, just because they can. aren’t showing that application of applying this to the game domain of game development. 

Vy: Yeah, that’s very fair. And I, I feel like also it’s such a, like, research is evidence, uh, based role.

So you kind of like, as a meta note, you kind of have to give the evidence that you can do the job, um, but it also applies to any other specialty as well. You know, even if you’re, let’s say, gonna apply for something like product manage. You’re naturally going to carry on the metrics, the measures, the success criteria, something which you achieved at the minimum as a line that, okay, I’ve been there, done that.

I feel like a lot of people can resonate with and something I push on as well for general UX flow, product designers in portfolios or not make a project. If you cannot find a project as [00:48:00] well, you know, just kind of push it, push the envelope a bit instead of, uh, doing what’s easy because yeah. It is required, definitely.

Steve Bromley: A nice thing in the games industry is because a lot of people do do it as hobbyists and are making games just off their own back. There is this audience of people who would like your support and are not doing it for commercial reasons. So you’re not going to be stealing. A job from an employed user researcher.

So there is that open opportunity to work on real games with real game developers through the hobbyist game development scene. If people are interested in a career in games, user research, or thinking about how to apply user research to their own game development. I write a lot about this. I’ve got a website gamesuserresearch.

com where, uh, has some guides on how to get started in the career or how to get started doing these kinds of play tests and user research studies. That I would direct people towards if, if they’re interested in 

Vy: taking this further. Absolutely. And I would recommend also for people to look into the [00:49:00] playtest survey report, which is something I looked at, even if I’m not working in, you know, a game specifically, but there are themes which are just immutable, I feel like from software development to, to everything in between, but this is amazing.

Thank you so much, Sif. Thank you. 

Steve Bromley: I’ve enjoyed our chat.