Don't give away the idea.
Emmett Shear, CEO of Twitch: The one thing you really don't want to do is ask them about a great idea for a feature. Ask them, "Are you excited about it?" Because the feedback you get from users if you tell them about a feature and ask them, "Is this feature good?" is often, "Oh yeah that's great." When you actually build it, you find out that while they thought it was a clever idea, no one actually cares to switch and get it. So the one question you can't ask is, "Is this feature actually good or not?"
Ask open-ended questions.
Emmett Shear, CEO of Twitch: You definitely want to Skype. You don't want to do interviews over email if you can avoid it, because interviews over email are non-interactive. The most interesting learnings come from the, “Interesting. Tell me more." The instant you hit this vein, they will say something that you didn't expect. And then you should drop into detective mode. Detective mode is, "Huh, that's interesting. Can you tell me more about that?" People don't like silence, so they'll keep talking to fill the void. The best part about doing an interview over Skype or doing it in person is that you have that interactive feedback. You can actually pull a lot more out of people. Email interviews are basically useless. In person or over Skype interviews are also easy to record. Make sure you ask them if it's ok to record. It's not polite to record people without their consent, but if they are willing to give you a user interview, they'll probably willing for you to record it as well. [How to Run a User Interview]
Ask the same question in different ways.
Emmett Shear, CEO of Twitch: Really, all these questions are asking the same thing, which is like, “What features matter to you? How do you use the service? Where are your pain points?” But you can’t just ask that once to someone. If you just ask someone that question, most people don’t spend enough time introspecting about the websites they use to give you valuable information the first time you ask. So, what you do is you ask that same question in eight different ways until you sort of stop getting new information. [TwitchTV: How to Pivot When the First Vision Falters]
Listen to their problems, not their solutions.
David Rusenko, CEO of Weebly: Listen to their problems, not their solutions. Customers will tell you the pain that they're facing, listen to that. When they talk about the solutions, their proposed solutions, you could just completely ignore that because they generally aren't that great.” [How to Find Product-Market Fit]
Emmett Shear, CEO of Twitch: The main thing you're trying to do when running this first set of interviews is not necessarily ask questions about optimizing user flow. Or questions about the specifics of any of that stuff. That can be distracting because users think they know what they want. You get the horseless carriage effect where you're asked for a faster horse instead of asked to design the actual solution to the problem. [How to Run a User Interview]
Talk to the target user, especially users of competing products and not just people who are easy to find.
Emmett Shear, CEO of Twitch: Another big mistake people make is talking to who is available rather than talking to who they need to talk to…. Because if you just talk to who's easy to talk to, you're not getting the best data…. We spent weeks digging for identifying information and figuring out who these people were so we could talk to them. This was a site that did not support messaging, so there was no obvious way to interact with them. We spent a bunch of time trying to network and find those users. [How to Run a User Interview]
Emmett Shear, CEO of Twitch: Given that we had very limited resources, we focused on the people using competing products. We knew that they were already interested in the behavior that we needed and they were willing to do it at all. Therefore all we had to do was convince them to switch, which is much easier to do than to create a new behavior.
Record user interviews.
How hard was it to get buy in from the rest of your company? You can say, "I'm in charge so you're doing what I say," but that's probably not the best way of doing it.
Emmett Shear, CEO of Twitch: That's a good question. If you just go to them and say, "I talked to the users. I figured it out. We have to build this," it's really hard because people don't trust you. There's something magic about showing them the interview though. I recommend recording interviews. It also stops you from taking notes in the middle, which is a little bit disruptive. It makes it hard for you to actually engage in the conversation. You can then play that recording for people. They don't have to be there for the entirety of all the interviews, but when you want to make a point about what what you should be building and why, you can play the interview back for the rest of the company. It's like magic, the influence it has o YB n people's thoughts, on what is the right thing to build.
User testing
David Rusenko, CEO of Weebly: Okay, UX testing sessions. I think these are the most important thing that you can do. It's really, really simple, but it can be really painful. Number one, get someone to use your product or service in front of you in person. Whether that's on your phone, whether it's sitting in front of a laptop or desktop, get them to use it. Number two, encourage them to give open and honest feedback. They will not want to give you open and honest feedback because it feels awkward telling someone that their baby's ugly, but you have to try your best to encourage open and honest feedback. Number three, ask them to perform a task. You are not allowed to touch the phone or keyboard. You are not allowed to do anything once you tell them to perform the task. Number four, and this is most difficult, do not say anything. It's really, really, really hard. You will go through extreme agony as they struggle to figure out how the hell to do something really basic, like sign up for your app. And you'll go through all this pain and it'll probably take them two to three minutes, and you'll get this visceral gut feeling. You are not allowed to touch or tell them to do anything until they successfully complete the task even when they ask for your help. It's really, really hard. You only need three to five testing sessions. I think you really only need three to be able to tell the most critical and important UX bugs, if you will, that you have.
Questions
Emmett Shear, CEO of Twitch: And I talked to all these people who produced that content and I sort of asked them, “What brought you to this? Why do you do this? What’s your motivation? Is it for fame, for money, for self-expression, to give back to the community? What’s your motivation?” That was really important. Understanding that is key to understanding what features matter to them. If they are for money, fundamentally your features better support money. And if they’re for self-expression, fundamentally you want a completely different set of features.
So, on top of, “What’s your motivation?” on top of, “Why broadcast yourself playing StarCraft? Why broadcast yourself playing ‘League of Legends?'” We asked them about the experience itself. So, it was a list of about seven or eight questions and they were things like, “Have you used Justin.tv before? If so, did you like it? What did you like about it? What did you not like about Justin.tv? Have you used any of our competitors before? What did you like about them? What was better? What didn’t you like as much? What did we do better?”
And then I think some of the most interesting questions I asked were, “If you were the CEO of Justin.tv, what would you do?” because that question, I found, really sparked people to suddenly take a step back and really think about what’s the most important thing. The other question I asked was, “If you could change just one thing-I’m going to give you the ability to change any one thing about the website. What would it be?”
Really, all these questions are asking the same thing, which is like, “What features matter to you? How do you use the service? Where are your pain points?” But you can’t just ask that once to someone. If you just ask someone that question, most people don’t spend enough time introspecting about the websites they use to give you valuable information the first time you ask. So, what you do is you ask that same question in eight different ways until you sort of stop getting new information.