Jim Morris: Designing and Running Customer Interviews that Work

Customer interviews are a critical part of the process of building great products that customers want to buy but they are rarely done well.

In this workshop session, Jim shows why customer interviews should be an integral part of your product development process and what you can learn from them. Jim demonstrates how to structure and run interviews that improve your product process before you conduct your own trial interviews in groups. You will learn why customer interviews should be a core and ongoing part of product management and how you can use the techniques and tools the best product teams use to build the products your customers will pay for.

Slides

Find out more about BoS

Get details about our next conference, subscribe to our newsletter, and watch more of the great BoS Talks you hear so much about.

Transcript

Mark Littlewood 

Okay, put your hands together and welcome Jim Morris.

Jim Morris 

Hey folks. You know, if you feel like it, you are more than welcome to turn on your video. The presence of cats or children or eating or drinking are all totally fine. For me, it’s really looking at all of you to see you know how, how the concepts are landing, and when I see people nodding their heads or shaking their heads or having questions, it helps a lot. So feel free to do that. Life is welcome here. Okay, so let’s get started.

Customer interviewing is something that I’ve done a lot of, and I didn’t necessarily think I was going to start out that way. So you know, for me, the. There we go. I started in computer science as a software engineer out of Stanford, and jumped into a startup that sold sporting goods. Back when Amazon only sold books. Grew that quite a bit, went to IPO, it was quite a road to travel, and it was amazing to learn all that. And did a lot of product just because there were no product managers at that time, and I was one of those business minded engineers who would talk to the people when they came over to the cubicles.

Then a couple of us started a company called Power Reviews, and it was a product reviews platform, and still is. And we went from zero clients to 1200 clients in seven years, and then we were purchased by our competitor, who also had 1200 clients, and I worked with them for a couple of years, and then we were sued by the federal government in the US as a monopoly, and so I am a convicted monopolist. So there you have it. And we got spun out and spent another year, kind of recreating the company from not quite from scratch, but almost after 10 years, I left that and became independent. And in that time frame, I did a startup posture. And I also teach at the UC Berkeley Institute for Engineering Leadership. I sort of teach grad students product management. So that’s my journey, and I really love product, and chose product when I went out of my own, because it is a place where we can make a lot of positive change.

So what do I do this? There’s a consumer type, consumer clients, digital health clients, about 50% of my business now, that’s a growing area, and then really tech, finance, energy, other industries. So we’re talking about customers, users. I’ve calculated that I talked to about 100 customers or users a year, not directly, but through my clients. I mean, I’m on the video often. They do the interviews. I coach them to do it. So it’s been about 700 of these interviews over the past seven years, and there are lots of trends that I see, and there are lots of ways that I’ve tried to make it easy for my teams. So I’ll share those with you today.

Why do we talk to users? Well, let’s talk about saving money. Here’s a case study of an entrepreneur I worked with, and when I met him, he’d already spent 200,000 US dollars on building a website, 50,000 in market research, and 150,000 on the site. So what that meant was he put a lot of his soul into it and his bank account, but he had no traffic. And so we set out to figure out what was going on, why he got had received no traffic. We actually created some fake prototypes, found what he thought were the target customers, and just started interviewing people. We didn’t even use this website because it wasn’t quite functional enough.

And after about 20 interviews, we had a great idea of where people are approximating his activity that he wanted to use technology for, and where they where they hung out online, and what these people really wanted. And so we on a no code website called Squarespace. I had him, his team, build this very basic website, and we started driving traffic to it, from Facebook groups and from next door. His idea is that when you want something done to your house, you need to find a plumber, electrician, handyman, but you may not know anybody in the neighborhood. So you want to meet your neighbors, and you want to get advice from your neighbors. There’s an online site to do that. And we started driving traffic. And with $150 and some sweat equity, he had a website up and running, driving 50 people a day, 100 people a day, 150 people a day. And so these 20 interviews really helped us learn about what we could build. Built something quickly, and then again, started doing surveying and interviewing about this site, iterated it into a second site where he felt more comfortable spending more money, you can see the design gets a little bit better.

And then he moves to pivot to more of a subscription service about who needs to find a plumber or electrician. Mark was just talking about jobs to be done. Just fix my clogged drain. I don’t need a plumber, and so it’s a subscription service to fix clogged drains and do things like that. So I didn’t necessarily know where this idea was going, but through interviewing, I was able to teach this entrepreneur and his company how to learn on a regular basis and not just think hard. But of course, as you can tell, success is more complicated than just launching a product. You’ve gotta either drive adoption, it’s usually leads into commercial goals. And so as you’re thinking about it’s so hard to build this product, this feature, this concept, this service, but really what you want to be obsessing about is, does anybody want it and will they adopt it? Well, of course, you can wait till after you build it, like this entrepreneur, and be quite depressed, or you can use product discovery techniques in advance, and that’s what we’ll talk about today, is customer interviewing, doing this advance, in advance of writing code. So you’re talking to a software engineer who’s telling you not to write code. So what’s another reason? Well, let’s stop wasting engineering time again. I feel sorry for all the engineering time that I wasted over the years. So I see this a lot with my clients. When I first meet them, build, build, build. You know, we have these cycles, but really it’s just build. What I want them to do is to really learn first, then build and then measure what happened. Let that lead into the next cycle of learning, building and measuring sounds simple, but not as common as you think.

So, a lot of these build measure, learn cycles, concepts came out of Lean Startup, and the idea is that in this well, who here has read this book, who here has heard of this book, but not yet read it? Okay? Perfect.

So one of the core stories is about how Eric Reese and his team wanted to build sort of a avatar Virtual Reality chat system, and decided they would kick it off by integrating with a bunch of other chat systems to have people kind of drag their friends to this new chat network. And so they got the great idea, and then they built six chat integrations over a period of six months, and they turned it live, and they got zero traffic.

My question for you in the crowd is whether you read the book or not, thinking of these like, how could you learn without writing code and that story I just told you, what could he have done differently? You can chat it, you can put it in. You can just unmute and tell me, but what could you have done differently besides writing all that code and spending all that time.

Mark Littlewood 

I think this might be a trick question. Is it interview customers?

Jim Morris 

Yes, but how would you. Well, yes, that’s one of the answers. Yes. Got a quiet crowd here, but yes, what make a paper prototype.

Audience Member 

You could have mocked up the experience somehow in the other in the other chat rooms. You could have just had a like a link, come and join this other chat, or something like you could have mocked up the service somehow, without doing any coding.

Jim Morris 

Yeah, you could mock up the service you, if you if you have an existing system, if you had, like, some sort of chat running with some people, he could put the button there and see if anybody wanted to click it. He could do interviews in advance with a system that looked like it and see if anybody would actually integrate it. And so you could use a paper prototype, or you could use the clickable prototype. Envision, Balsamiq, Figma.

The idea is to find your target customer and interview them and and get some preliminary information.

Now we don’t necessarily get proof, but we get evidence. And the idea is that instead of six months of writing code, you could maybe integrate to one shot system. You could maybe put a fake door test a little button inside of the existing application, or you could do a prototype interview. So many different ways to test it without risking so much money and time.

So, yeah, okay, so why did we talk to users? The third reason I hear is how compelling is the idea? This is one of the harder parts of customer interviews, which is, can you actually get them to give you a signal of demand? And Alberto Savoia, I saw him speak a couple years ago, and he wrote this book that not a lot of people have heard of, but it’s a great book for companies that want to really explore different prototypes, how to measure them. He has a whole system for thinking about this, but he really feels strongly about what’s called skin in the game, which is, can people give you some signal of demand?

Opinions are free. I’ll certainly give you lack of opinion for free, I mean, you could hire me as well, but love to get opinions on anything. But you can get a reaction from users, get an opinion, get a thoughtful response. In the game, cost something. Can you get the user to give you their real email address, not just a spam email address. Can you get a cell phone number, a home address with that entrepreneur trying to find plumbers we were looking to ask people for their home address so we could find the right list of plumbers and electricians near them.

Well, as it turns out, people would not give this to us, but through customer interviewing, we realized they would give us the intersection. So in San Francisco, I live at 35th and Balboa. It doesn’t tell you exactly where I live, but it gives enough information for the algorithm to find the right list of folks for you. So in discovery, you can learn what learn about privacy, you can learn all kinds of things, credit card information. What if you got them to pay a financial deposit? The larger deposit, the deposit, the higher the confidence. Can anybody think of a good story here about paying a financial deposit for just an idea? Anybody done that here? Anybody bought into a Kickstarter? Oh, anything else, yeah, what’d you buy, Mark?

Mark Littlewood 

A weather balloon, topically. I have a sneaking suspicion that was sent from China, in actual fact, but no, it was before drones. So it was a long time ago, and I got a weather balloon that you could send a camera up on, and we did a project at school and sent a camera up, rather than a child, unfortunately, which had been far more entertaining, and it wasn’t shot down.

Jim Morris 

Wow. I mean, if you, if you live west of mark in the jet stream, be be afraid.

Mark Littlewood 

Everyone’s west, west of me in the jet stream,

Jim Morris 

Exactly, exactly.

So, you know, Tesla did this in 2016 they said, Hey, we’re gonna make this car. It’s gonna be cheaper and, like, it doesn’t exist. But hey, you should sign up for it. And 325,000 people signed up for it, gave $1,000. No interest on this money just got there. I had two friends like you really signed up for this thing? Three years to deliver. But that represented about $14 billion in sales. And so that was a lot of confidence they could have to really commit to this project, and just want to remind you, when you’re out there and you have a new idea and you’re trying to get feedback on it, you can do more than just opinions. It’s not easy, and it’s different per product and per business model, but it does give you more confidence, right levels of confidence in user interviewing.

So who? Who should we interview? Okay, oh, someone that you don’t know that fits the description of your target customer. Okay, by the way, I’m going to call this a relevant stranger, my new term for it, relevant stranger, not your friends, not your coworkers, not your family. Someone, if it’s an existing product, you could be thinking about someone who’s recently used the feature, product or service you built. Depends on if it’s talking to cancer patient the last year or two, probably works, because it’s a very impactful event that people don’t forget. If it is buying a fashion case for an iPhone, you might want to talk to somebody who’s done it in the last 60 days, because the last 60 days, because it’s not that impactful of a purchase, and you’re not going to really recall that memory.

If you don’t have a product or feature yet, you might want to find somebody who might use the product feature or service you would build. Okay, it’s okay to interview users again if they have constructive feedback. Now, the one situation you can’t do that is in first time experiences, because that confusion of onboarding and that first time experience is hard to unknow, and it’s okay to create like a user group, as long as they’re aligned to your vision. A lot of folks who have existing customers have a couple loud customers who just want you to build something as if you were an outsourced developer for them. So want to make sure that you’re focused on people that are aligned to your vision, and getting feedback from them and making a partnership.

And it sort of looks like this these days. I used to go to transit stations and shopping malls to interview people, and now we do it online. People do it from their cars. There’s a guy who held his child during the entire interview. Again, just a sample of the 700 interviews, okay.

Now, when do we talk to folks? This is classic double diamond of product development. Who’s seen this type of diagram before? Okay, couple of you all right. Sometimes it’s three diamonds, sometimes it’s four. I’m a consultant, but consultancies will definitely give you four diamonds, five diamonds, or six diamonds, because there’s you know things to learn in all all the diamonds. My goal in life is to make this simplified and just enough process so that you can follow it, but then really understand why we do it.

Then you go through design activities, brainstorming, competitive research, to expand your solution. And I say solution experiment, because you don’t want to jump straight to a solution right. When we do customer interviewing, we’re going to have multiple options I’ll show you in a minute. And we use customer interviewing to really narrow these solutions down to a validated solution. So we do want external validation for our solutions. Most of us want external validation, right?

So, but what’s traditional product development? Well, forget about the user, and hey, there’s this roadmap item, and you got to get it done. Forget about experiments. We’re not going to expand so far in our solutions. We’re going to cut off that expansion. We’re just going to build maybe the first thing that comes to mind, and this quote at the end, just get it done, is from many of my clients, many of my product managers, are under the pressure to just get it done. And this gets in the way of actual innovation, right? Speed is not innovation.

So let’s just explain it here. We’re going to start on the left, and we’re going to expand our set of problems so, and I’ll tell you how this is relevant to customer interviews in a second. So you want to really get that like, what are all the universal problems? Since Mark’s like throwing things in the chat here, Theresa Torres is a great technique, and talks on opportunity solution trees. This is a great place to sort of expand your problem set. As you’re contracting the problem set, you actually, you know you have to find something to do, right? So you’re going to focus in on a problem. Many of us get a solution idea from our executives, our stakeholders, our clients, always reverse engineer the problem. Okay?

Okay? So when do you interview people? So I’ve got you, I’ve explained the diagram. You’re not fully functional product managers. Did you know when to do what? Well, here in the beginning, you’re gathering and understanding problems. I’ll show you a UX a user experience map, which is a technique I use when I do this. This is one of the more common places that people interview people, because it’s easy just sort of talk to them, ask them general questions and get some feedback. This one is not as common, where you actually go back to people and you verify a top problem and success metric. And I’ll show you a tool you can use for that. And the least common is user testing your solutions. And this is sort of the product discovery value of death, which is this diagram. Physical diagrams, people.

So you do a lot of research up front, you feel confident, and then you just start building something, and you come back to people like, three months from now, really bad idea. You want to test and iterate your solutions as you go. So again, gathering and understanding problems, verifying the problem in the success metric, and then testing solutions.

I, you know, there’s also usability testing. I’m not going to cover that much of usability testing today. It’s very well known. It’s the very end of the process. Just to make sure you haven’t confused somebody. It’s typically with very highly polished, pixel perfect, high, you know, prototype, the mock ups. This will be a lot of the customer interviewing before that moment.

Gathering Problems

Okay, so let’s talk about this phase of gathering problems. So we’re going to interview experts. Well, what’s a good way to do this? Well, your team needs to get a little bit smarter at this product area. So if you’re a new team, or if you’re trying to delegate, like some of you heard some of you say, you know you’re getting out of pm because other people are going to jump into it, or you don’t have a PM, your team still needs to have a context of what, where the product fits in the world.

So let’s use that plumber example. I’m a homeowner on the left, there’s a trigger of a clogged drain, a starting point. I text a friend, I’m going to get some suggestions for each suggestion. I’m going to go through this loop of finding out information. After I evaluate everybody, I’ll choose somebody, and then my hopefully at the end, my clogged drain is cleared. Okay.

So that’s the existing process, and then you use this to identify problems and opportunities. How might we ask our neighbors for recommendations? And you’re going to brainstorm several of these, and they’re all over the map, because at some point you might make a decision in different parts of your journey to do something different or to make a solution. So how might we ask our neighbors for recommendations was the core idea that fit right in this part of the user experience map? This contact is super important, and we interview experts to validate our intuition and to learn more problems and opportunities. So ask the expert to tell everything you know about the challenge at hand. Where is the map wrong or needs additions? Can you explain knowledge about how you make that decision? How did you choose between those plumbers? Ask why? So, yeah, the five whys works, although after three or four wise people get pretty annoyed and then tell me more about it.

Lot of folks try to talk during interviews. Really you want to listen.

This is if you want to read more on this technique. Get the sprint book. It’s a great book. Think about the sprint book is like that recipe book at the at the at the bookstore from one of your favorite cooks and chefs. And it’s it has all the recipes laid out, and you can think of folks like me as sort of the chef hanging over your shoulder helping you, tweaking these things so that actually works.

There are other touch points, Win Loss Analysis. So, and it’s great to have product people do these interviews, because when you when a salesperson loses a deal, it’s not that common that the the not client wants to talk to the salesperson. Okay? So you as a product team can get on the line and talk to people about why they didn’t or did choose your product.

Go visit people.

Go see what it is, especially if you’re not familiar with the domain. Get to the see how the electronic medical record is used. See how people use the Admin Control Center for your facility. These are common problems my clients have. Read the reviews, that’s not an interview, but it’s getting data. Get it on customer call, not technically an interview. You’re just listening in, but sometimes you do ask questions. Sales, customer success and support are meeting with customers on a regular basis. Tap into that.

Go on a client listening tour.

I did this. I met two clients a week for a year, 50 clients out of my 1200 clients, was an amazing, sorry, 100 clients can’t do Matthew speaking, was an amazing year to really get to Reno my client that was after this sort of acquisition and then divestiture, and they didn’t seem to mind talking to a convicted monopolist but you know. The learnings from those interviews quite important and there’s a case study later about that. We were there physically. We were there in zoom, and we had customer event in different cities, Boston, San Francisco, things like that.

You can send out surveys. If you pay for survey responses, you probably expect low quality. So surveying your existing customers is pretty good, but you should do qualitative research to understand what a good survey would look like and what the answer should be. Don’t just send out quantitative surveys as your first thing. This is really too easy, and the data is not that great.

Okay, so we talked about gathering problems another touch point, an important touch point, is finding your problem and metric and reflecting that back on your customer so we can interview business buyers and users. So who here is like you consider yourself B2B, where the business buyer is not quite the user, almost like a B2B to B, or B2B2C, okay – Liz, Amy, Steve, Mark, yeah.

So the business buyer, the target buyer and the target customer might not be the same person. One of my clients has a channel set up where there’s an installer, there’s a facility owner, there’s a tenant inside that facility. The tenant has employees, and the employees have guests that come. So imagine just a very simple office building, and you have the access control system for that office building. You have sort of four or five layers of indirection, so you need to know in that ecosystem who to talk to, and when.

This cautionary tale about finding a top problem about who to talk to is pretty important. During that listening tour,were meeting two people a week, we were shopping around this roadmap. It wasn’t full of like colored rectangles, and actually had what we thought were great ideas, but I imagine that when my client thought they just saw a bunch of colored rectangles, because what was impacting them was the growth in mobile traffic and the fact that their websites were awful on mobile, not responsive. They didn’t have apps, and people were browsing but not buying, and they were they were incredibly worried. And this was the year that mobile traffic went about 50% – this year, 2014 2015.

We listened to this our software for for reviews was not compatible, so during these interviews, we learned to pivot, and we threw away this roadmap, and we did a total focus on mobile compatibility. And we did six months didn’t build really needed features, and came out with APIs, mobile responsive platform. Ours was so good that our clients started asking us how we did it, and we became like unofficial, informal advice givers about how to, like turn on a mobile experience. That’s amazing. So again, when you’re interviewing, you may learn things that you are not intending to learn. You may learn things you do not want to learn. You may have to hear it many times, but if you do it enough, you will be dragged kicking and screaming into the world of everybody’s reality, not just what’s in your brain. That’ll build your product until you validate a problem in the success metric.

Opportunity Assessment

And I use this thing called an opportunity assessment. This will look and feel like a canvas. That’s fine. I’m not dogmatic about the actual device, except I really like simple, easy, lightweight devices like this. But you’ll, you’ll see it as like a canvas type operation, canvas type thing.

So here the business objective was mobile compatibility. The key result was, we decided a great key result for this project was product review completion on mobile goes from 10% to 50%, just get it much higher. We did, we did quite a bit of progress there when we actually launched it. E-commerce Vice President was one of our common buyers. We also had it. We also had marketing. Our customer was really the website consumer. They were the ones like actually submitting the reviews, and had to work for them. And the problem was, it’s just too difficult to write a review on my mobile device. I had to squeeze it, move it, type it, and then field would go away and I couldn’t find it. And of course, the constraint for this project is about mobile devices. And with this opportunity assessment, my team was allowed to find multiple solutions to this problem. So a good problem statement, a good opportunity assessment, necessarily allows for multiple solutions.

Okay, so again, target buyers, target customers.

Don’t ignore your end users, right? They’re out there. Make sure that you poke through and get to those end users.

So there’s product marketing and there’s Product Management. Try not to mistake the two, okay?

Because that renewal is going to be more locked in if you’re delivering value for the person using your software right?Now, of course, if you’re doing monetary value for your client’s business, the business buyer will be super happy, but if nobody inside that business adopts that, that concept you’ve given them, you won’t deliver that company value. Okay, you can test each of these concepts when you’re doing an interview, similar as that user experience map. So this was inspired through the book. Inspired Marty Kagan, again, this the PDF here has Miro board with some slides and templates. Who here has used Miro – Kirk, Mark, Liz, Lucy, awesome. Ian, yeah. Mira was fantastic. It was transformative when I picked it up a couple of years ago during the pandemic, and just it’s been amazing to run remote sessions with it.

So again, we’ve talked to different areas to do interviews. This last area is where we spend the most time, and we do some practice with just testing solutions. So like you saw that physical board, I put up the parts discovery valley of death. This is where people somehow drop off in trying to get customer feedback. And it’s the most dangerous area, because you do waste engineering time, which you could spend on other things. You do lose money, and you don’t know whether actually people want it.

So here’s another diagram called the stock tooth diagram, or brisk and again, you see that little learn bubble. If you green learn. Yes?

Audience Member 

I have a question about that last slide And so, yeah.

Jim Morris 

Alright.

Audience Member 

So, no, no, actually, the one with the valley of death.

Jim Morris 

Yeah.

Audience Member 

So you’ve got, like, the solution that you want to show to your customer before you go and actually build it, but you’re not getting anybody to respond. Should you take that as a signal that is not worth building if nobody wants to engage with you to go through that? Here’s the temptation would be to jump to building it, and then you find out you could have learned a step earlier that it was not the value that everybody said it was. Right?

Jim Morris 

Now, if I understand you correctly, it’s sort of perception versus reality, meaning they won’t even talk to you, because the perception of it is they don’t want it, all right?

Audience Member 

Well, it’s kind of getting back to the original concept of having some skin in the game. If they’re not willing to spend some time reviewing a solution, they’re definitely not going to buy it. You know, that’s that I’m oversimplifying it, but I’m wondering if that’s a signal of that lack of value of the the actual solution.

Jim Morris 

It depends. I go back to the analogy of the horse and buggy and the car, and the people with the horse and buggy wanted a faster horse, a more comfortable buggy, maybe a roof, you know, just they would keep iterating on this, on this dead end concept. And if you said, Hey, try this car thing out, they’d be like, What are you talking about? That’s stupid.

And so consumer customers are really bad at envisioning future solutions. But if you were to put them in a car and they were to experience it, that is a transformative moment. And we think about solution testing, we’re thinking of trying to create that transformative moment so they can get over that hump. And so I would make them use the product, I would pay them, I would do whatever I can to get some facsimile of the product, or make them watch an explainer video. Maybe an animated video is a great way to show the feature, because it’s cheap and animation has no laws of physics. Some way to show them the feature and once you feel that they understand that feature, then you ask that question, how disappointed would you be if you didn’t have this feature? But if they don’t understand your solution, I then, no, I don’t really care about their opinion yet.

Audience Member 

Thanks, Jim. Sorry to interrupt. Didn’t want to please interrupt.

Jim Morris 

Okay, please interrupt. No, it’s all good.

Audience Member 

Hey, Jim, I had a question. Yeah, what signals should we look for when we’re interviewing customers? People say all sorts of things that would lead you to think that they’re interested. But what are some of those signals that we should be on the lookout for?

Jim Morris 

Initially, you’re looking for an emotional response, Oh, wow.

Okay, so that’s one very important signal, which is when you craft your experiment and when you start your interview, you don’t want to over talk it because you want to have somebody really you want to wait for that moment of value to see if it happens, the aha moment. I would say people who over talk it means they might be trying to justify liking it. So for those of us in America that’s very common, people want to tell us things, to be nice to us. So you have to be aware of that if, when I did the product reviews company, we had 1200 clients all around the world, I did average rating by country one time, and then just for fun, and the French were the lowest average rating for rating products. And I’ve asked French people about this time and time again, and my impression is verified by them, is that they’re really honest. And we’ll just tell you, yeah, this thing sucks. So I thought, I suppose folks in Europe have a much better feeling for this, but we in America, like we get these fake positive responses all the time. So my So Mark, Matt, I would be careful about people who overtalk it, who seem nice to you. The longer you talk something out, the less you can trust that conversation, because you might be steering the conversation towards liking it. Not always. Those are a couple things that come to mind. I mean, I’m going to tell you more about it’s more about the experiment you set up than about being an expert interviewer. That’s the like the secret here.

Audience Member 

Yeah, I like what you said about not over talking it from the perspective of the person who’s walking them through the prototype, walking the customer through because you want that organic aha moment and that emotional response. That’s a unique sort of piece of advice that I really don’t hear. So thank you for that.

Jim Morris 

Sure, and then not to pick on you, but you’ll never want to walk anybody through anything. Yeah. In those days, you would sort of think those are over, and what you want to do is give that user control through a clickable prototype. Even Balsamiq has clickable prototypes that users can operate. Not the prettiest thing, but it does work. And I know we were talking Tim was talking about Figma and High Fidelity. What I would say is I take screenshots, throw them into Figma, and just make them clickable, and so you can do these quick, low fidelity prototypes, and you’re going to experience one that I made personally.

Audience Member 

Put them in driver’s seat, let them drive.

Audience Member 

Yes.

Audience Member 

Shame on the subject of signals is this going back to earlier when you’re talking about different like, sort of customer touch points to gather, basically to interview. And one of the ones that kind of struck a nerve with me, because I feel like I’ve made I’ve just wasted so much time and money chasing this was you talked about, kind of like the, I think you called it the win loss analysis. Mostly it’s loss, right? And so you asked, you know, the customer, you know, why didn’t you buy? And I, this is my guess, but I feel like they feel like they have to criticize the product in some way when it may be something else, maybe something totally else in their life. And so they’ll say, oh, it’s lacking this thing, this feature, or this report, or whatever. And I would take that at face value, go back to the product team, build that feature, and then come back to the customer and say, Oh, now we have it. You ready to buy? And of course, they don’t buy like and I can’t remember how many times I’ve done that. Yes, I’m not that. I’m not very smart, and so I’ve kind of grown to distrust feedback at that stage. And I’m wondering, maybe, instead of that, maybe I should, I’m asking the wrong questions. Are going about it the wrong way. I don’t know. I just, I’ve, I’d love your thoughts on that, because you know you are in a conversation. They you know, they’re they’ve already, they’re willing to spend time with you and evaluate the product, so they’ve already invested a lot. So it seems like an ideal person to talk to, but, yeah, I don’t know. I feel like I’ve failed so many times in that area.

Jim Morris 

Yeah, I’m with you there, just by the way. I think, though there’s no magic recipe here. My sense is you want to hear something from multiple people. You want to draw some diagram where these ideas go back to problems or opportunities that you think are relevant for these companies. And so anytime someone gives you a solution idea, it’s really important for you to reverse engineer the problem or opportunity. And that will give you a sense of the scale of importance to them. And my sort of you know, scale of importance is okay, well, if someone used with something with someone use this. What’s the use case, and maybe tell me why you’d use this. What’s the problem? What’s the frequency you’d use this? Is it a daily thing, like, I’m checking my Facebook feed? Is it a weekly thing? Is it a quarterly thing? Is it a couple times a year thing, like, I’m renting an Airbnb? That frequency is going to give me a sense of whether there’s a lot of money in automating this, because if it’s only every quarter, they’re probably fine just by pulling the report by hand. And then I want to know, I think the press release technique, which is getting more, I would say, more importance these days, is talk about the feature of when this thing exists. What would it do for that company? Would it drive revenue? Would it save money? Would it make things make people happier? Try to take them to a future state where the thing exists and have them tell you what the impact on their business would be. So I think, I mean, those are some of the techniques, you know, multiple people saying the same thing, really finding another problem, learning about the frequency of it, and then trying to tie it to some metric, and getting them to say, Oh yes, this would drive this amount of money, or it would drive reduction in gross margin, or increasing gross margin, you know, reducing cost. So problems, metrics, sheer quantity of folks, those are things that come to mind.

Audience Member 

Jim, can I just jump in? I’m sure if we’re kind of interrupting you too much now, but it was really interesting. We have to say particularly that I. Uh, talking too much when answering questions, trying to almost negotiate with with the kind of person before they come up with an answer. I was just thinking, having just read, um, thinking fast and slow. Again, which is great. And so a kind of psychological, kind of context that we need to be aware of. And you mentioned the cultural context with the French versus American telling you what they want to hear, being overtly honest, emotionally constrained or whatever. Is there something that? Is that, something you need to be aware of? Is there a bit of digging to get below that surface? Do you think?

Jim Morris 

You know, it’s, it’s setting up the interview. So you’ll see that there’s a setup part of the interview that’s short, but meant to put the user at ease, create the environment for them to give you feedback. But you know, really a lot of us now believe you want to create Teresa Torres will say, create a compare and contrast moment, as opposed to a whether or not moment. And so as you get people to and this is why, when we do prototype experiments, I want people to stretch the idea set, the ideas that may they may not want to build. So that’s why it’s a solution experiment. Because my I have to teach my teams, the things you’re showing people are not necessarily what you’re going to build. It’s it’s like a test for the person that they don’t know they’re taking. And if you stretch the the concept, the ideas, some ideas will sort of pop out is, Oh, I like this and I don’t like this. And you can get people to say, I like and I don’t like that’s part of the win. If people are just giving you pleasantries, it could be they’re not your target customer. When I do recruiting through, when I recruit people through online services, there are professional testers here, and I have to really ask a couple questions and decide, like, are they just saying really generic things to me? Maybe they’re not my right they’re not the right user, if they’re trying to be nice to me. Yeah, would you use this? When would you use this? Why would you use this? Absolutely, you might have to dig.

Stay away from those yes, no questions. Um, and get into their everyday life, not like their personal life, but when they’re in an interview, their business life is on full display. What is that moment you’d use this and why? You know, what does it give you as a business? Um, and you might start to feel a lot of nice to have in the conversation. And nice to have is terrifying, just, just absolutely terrifying, because that means they’re just not going to use it. Yeah, the Mom Test, exactly. Just looking into the chat here the Culture Map. Someone’s telling me about the Culture Map right now. My friend is reading it. Yes, if you are in international business, the concepts of low context and high context, this is why Americans are such adults, because we are low context people. You need to spell it out for us. If you say something indirectly, we just won’t get it. And then when we talk to other people, we come across as British because we’re so forward.

Yeah, interesting. Thanks, Andrew, any other questions? Yeah, we’re close to the end of this phase, and we’ll move into the practicing phase in a second. All right.

Mark Littlewood 

Right. Press on some excellent, excellent questions.

Jim Morris 

Okay, I. all right, I did that and not hit everybody. Of course, that’s fascinating. All right, we’ll try something else. All right, you know, I found out what zoom was doing all this time. Instead of improving this thing that we’re using, they’ve been building slack, and they’ve been building email, and someone convinced them to reinvent 2015, I’m blown away. I’m incredibly sad, and now I know why they’re not doing well as a company. They’ve totally lost their way. Okay, good. I can see what you’re seeing. Perfect. This is just another way to look at.

Audience Member 

Opinion, okay, you’ve still got that bar across the top that says you’re sharing your screen, you know? Yeah, it’s still, like, on the top of the thing we’re seeing in the little bar, yes, that thing

Jim Morris 

Perfect.

Audience Member 

Thank you.

Jim Morris 

Yeah, thank you. And they’re starting to have some of these light, lightweight bugs that you didn’t used to have, right where, where their own pop ups are getting in the way of their own sharing.

Okay, so here the waterfall method, delayed customer validation. If you don’t talk to customers, you can just imagine there’s like a little bank account of risk building up, in addition to the other bank accounts of risk. But, and this is not a good bank account to fill up, and you’re just sort of hoping and waiting, and it’s like a pre revenue startup, right? The valuation is the sky when you haven’t put a line in the sand about how much money you’re serving for product will make. But we on the inside, need to know whether something is valuable and what to spend time on. So you want to construct your development process. That’s what you can learn, build, measure, okay, on a regular basis, and that does this thing where it reduces the total risk. Okay?

So making a test is actually more powerful than being a great interviewer. So the design prototype interview, the solution test, quick on the writing code. Remember that cautionary tale about the entrepreneurs spent $200,000, $50,000 on market research, right? Which is, for those of us who do customer interviewing, market research is really not the same thing, and not even close.

So we’ll talk about an example of improving Zoom breakout so you’ll see this again and again. I teach a class at Berkeley. There’s 35 students that go into 10 or 11 breakout rooms, and I’ve got 10 minutes, and I’ve got to figure out which one to visit, who’s quiet, who’s got the videos on how do I know which one I need to go visit to, sort of, like, stimulate conversation and get them going? So let’s say that you work for Zoom and you’re gonna, like, work on this problem. They have not worked on this problem, but let’s just say they’re going to how can we use rapid discovery to figure out which ideas could move the needle here, which ideas to Joe to your thing? How do I test perception quickly? Let’s say you got 10 ideas on your backlog. Well, I can Zoom on my phone. I don’t really use them on my phone, but they can just send me push messages like this.

Zoom now offers advanced features to manage the success and help the breakout rooms. So of the millions of people on zoom on their phone, like, oh, who swipes this? Who ignores this? Well, let’s try another one. Zoom now offers to be able to listen into a breakout room without having to enter it. Now, for those of us who run breakout rooms, like, Oh, interesting, and for those who don’t, they’re like this creepy. Some has video turn on and off for breakout rooms without having to enter them. Again. They can also send this push message to people who have created breakout rooms before, because in their vast database, they have this information, right? They don’t have to just send this to strangers.

And again, you get the idea like you have new ideas, and there’s simple ways. I call it. This is called a micro prototype. I just went through four ideas, if it’s sent to people who make breakout rooms, this is probably enough to understand the feature. And what’s the interest level, and maybe I start doing more customer interviewing and discovery on this interest level. Okay, now let’s say I just want to bring in people and do customer interviewing, and I just want to show these messages to them. I’m not actually doing this in a real push message. Okay, so here’s the video example. This is not about breakout rooms, because I don’t have a good video on that, but I do have a good video on improving classroom technology. I went to a classroom here in San Francisco with my this team I worked with. This is an engineer Byron who’s in who’s interviewing a teacher there on the left.

And there are two ideas you’re going to see. I want you to think about which idea user, which the idea you think user would actually use. Note that the user has been controlled the prototype. The teacher is holding the prototype, and the prototype is quite simple, and you’re going to tell me with thumbs up if you can hear the audio.

Video 

Well, we just walk you through a couple scenarios. There’s two new apps or features we’re gonna ask you to go through, and then, like I said, give us your feedback. We are not testing you. We are testing the experience with the app.

Video 

Okay

Video 

So there’s nothing, but just the app that we’re interested in.

Video 

Got it.

Video 

So let’s say it’s after school, just like this, school’s out and. If you you’re using dojo and you’ve gotten this message, or you’re just looking at your phone, go and read that.

Video 

Try setting up an exit ticket for tomorrow and toolkit slide to reply.

Video 

Go ahead and test okay. So do you know what Exit Tickets are? I do. Okay, so let’s say that it’s four o’clock. You’re ready to get your you’re done with all the immediate stuff, and you’re thinking about, you know, maybe, maybe planning and whatnot. So then you get this message from Dojo. Go ahead and read that and tell me what you

Video 

Try out class pals, a new way to connect your students with other classrooms around the world. Yes, I would do that. I would love to do that. I actually am in the process of doing that with Skype with another classroom for social studies, yes, but if it’s already on Class Dojo, I would definitely use this.

Video 

Why is it interesting to you?

Video 

Because I want my kids to realize that it’s not just them and their little San Francisco bubble and there’s a lot of things happening in the world around them, and to see a different first grade classroom with different first grade kids, I think it’s a way to really awesomely connect the curriculum. Because of first grade, they’re learning about the continents and we’re learning about countries and oceans, and be able to see that maybe it’s a different country.

Jim Morris 

So which idea do you think she preferred on first glance? Number two says Liz. Number two says Joe. Yeah. I mean, it was an emotional reaction, which is the engine too. It’s, it’s, and that’s the you could tell that first one, she knew what an exit ticket was, if you look up exit tickets. My wife’s a teacher here in San Francisco in a public school, and she knows I’ve asked her about it, and it’s a thing, what’s another what’s another reason, besides the emotional response that would give you confidence to do more discovery on that class palette idea.

Audience Member 

So she was already looking to solve this problem. She was already trying to put together a solution through Skype. So it was kind of almost like she was cobbling together a solution.

Jim Morris 

Yeah, replacement technology is one of the most competent areas that you can enter. If you can do something faster and easier and take this best of breed stuff and really make an integrated solution, that’s a pretty good area to be in.

So again, short ways to test things in customer interviews. And when she started talking, then I just said, Well, why is that interesting to you, right? And then she kept talking, and she gave me more reasons to feel confident about what she was doing. So anyways, this is a very simple idea. And so again, I like this. That’s interesting. I’d use this. You know, I might even say they’re lying to you. It’s like, meh, feedback. What do I want to hear? Ooh, oh, wow. Is this available? So the math question, long time ago, this is one of the first things I look for, which means that you, when you’re doing interviews, you have to really be paying attention to the user, not so much your script and your notes and your prototype, but you have to put attention to those too.

Jim Morris 

Okay, so we did Q and A, Mark. I think we’ll probably just go through the next level where we start practicing.

Mark Littlewood 

Yeah.

Jim Morris 

Okay, so instead of one solution, we’re going to test multiple flows. This is just a setup for our interviews. It’s a compare and contrast moment. Like I said, it stretches the imagination of your team. Let’s take an E commerce experience. Homepage, Search, Filter, select Add to Cart, Check out, Billing, Shipping, Credit card. Let’s just pick Add to Cart. I have clients who want to who want to redesign this whole process. This is so much software. It’s like six months of software, and they’re going to do three or four month design, and it’s all these big bang projects that seem great, honestly, this a big danger risk zone, right? You want to pick something small, innovate it, get it live.

Now, let’s pick three different concepts for add to cart, and we’ll create an experience around it so that it feels like a real experience, like they’ve selected a product, they’ve done the add to cart. They’ve done the add to cart, and there’s a checkout experience here, like just a screen. But really, we’re focusing on this innovation by creating multiple versions of one thing. If you have this huge process, you can’t possibly do multiple versions of each of these screens. You will barely get one version out, right whether or not decisions one version is not a good idea. In fact, that’s where you get that confirmation bias. Putting one thing up to people.

Being open minded is creating an experiment and actually being innovation oriented. You can see that like you’re going to get some innovation here, because your team’s going to have to come up with at least three ideas, or two ideas, or five or seven. And that your designer will have to build the idea of the data scientist, not just his or her own design. And so to some degree, the designer’s got to be open minded to all of us creating user experiences in a cross functional team. So how do you know where to start? I might spend a lot of time on this. You need to take a slice of your service or product, okay, you want to pick a valuable slice. That little green part at the top is value. Some people call this an MVP. To me, that’s a four letter word. So I use the term vertical slice, where you’re trying to pick a piece of the product that you might build, and you do a test on this. You might call it MVB test. So we’re back to improving zoom breakout rooms. I’m going to do a bad interview. Who would like to be my bad interview? User, tester.

Audience Member 

Emmy..

Mark Littlewood 

Oh, well, voluntary,

Jim Morris 

Perfect. Okay, so here we go. Emmy, have you ever created Zoom breakout rooms?

Audience Member 

Yes.

Jim Morris 

Okay, all right, all right. So I’m going to show you a couple examples of maybe better ways to manage Aoom breakout rooms. What do you see here on the screen?

Audience Member 

Some rooms with people in and information about their talk factor, and can see if their mics and the cameras are on and an option to enter the room.

Jim Morris 

Yeah,  what do you think about it?

Audience Member 

I think there’s a lot of information there to take in across three different rooms. But I think if you got familiar with the screens, you’d probably end up quite quickly being able to interpret it. But I don’t know if the purpose of the thing, this thing is to know which room to go into and help then I’m not sure which one to go in, still.

Jim Morris 

Okay, let me show you another example. Take a look at this one. What do you see here?

Audience Member 

People’s faces or knots, three rooms and color signals, but I don’t know what the color signals mean, but they’re red, white and green, so I think green probably means something good.

Jim Morris 

I love the heckling. This is good. Keep going. All right, okay, here’s another example. What do you hear? What do you think of this screen?

Audience Member 

So, game, three rooms, colors, red, white and green. It’s clearer. Like the indicator to me is the green looks fine, but then there’s a black hand that don’t know what that means, and a yellow hand. It’s a bit confusing. Although there’s more information about who’s talking most and the level of a conversation in terms of age and I and system is giving a recommendation, yeah.

Jim Morris 

Okay, which of these do you think you would prefer to use? You can tell me to stop or go back.

Audience Member 

I don’t know if I would want all of any of them, really. I think a combination of parts of them, the best parts, I think the last one, the system recommendation, is kind of what I’d probably use as a kind of trigger to go in or not, and you’ve got options there, so you can either enter or listen or chat, which is quite nice, but it still has got quite a lot of information on the front screen.

Jim Morris 

Yep. Okay. Thank you. Appreciate that. Everybody give a round of applause for Emmy. That’s a little bit short and but I will give you some some tips that I was doing, that that you could think about avoiding. Don’t talk so much. I didn’t necessarily talk that much. It’s hard for me to talk a lot during interviews these days, but don’t want to narrate the on screen options. I didn’t necessarily do this, but I see my clients do it all the time. Don’t fill the silence. Again, I could do a better job filling the silence here. You want to wait, wait, wait, don’t say this. What do you see? So I did do this. Describe the screen. What do you think about this screen? Please do not say these things. The reason is, you’ve got eyes too. All of your help you here have eyes. You can see what that person can see. The fact you might have had a hand in making this thing, so you actually exactly know what they’re seeing, describing it. So again, we got a lot of like, design feedback from Emmy, like, oh, it’s kind of busy and this and that. Like, okay, then at the end, I wasn’t quite sure what was what she really preferred. And then this is question called what’s missing here, which I didn’t ask. I forgot to ask, but that’s a really hard question for users to give, because, again, users are bad at solution. It’s your job to fill the spectrum and to give optionality. Once you give enough optionality, you might be able to get new ideas. But what’s missing is quite difficult. It requires software. It’s not too hard. So I think in electronic health record, we’re, I have a client, we’re talking to a certain specialty, and yes, they know what’s missing, because there’s required elements that they’re looking for. But most people in an optional software they don’t know, don’t ask users to read an email, right? I see this a lot. Hey, read this and tell me what you think. Do you know what’s a great way to test an email or a piece of text. Who has the who’s done this before?

Audience Member 

I haven’t I haven’t done it necessarily, but this putting some kind of call to action in and seeing if people do it might be one thing.

Jim Morris 

Yep, call to action. There’s something called a five second test, where you show the email or text on the screen for five seconds, and then you take it away, and you ask the user what they understood, and you don’t necessarily, you don’t necessarily tell them it’s only going to be five seconds. And I’ve done this with with systems, and turns out, people can understand quite a bit in five seconds. Okay, so I’m gonna do a good interview. Who’d like to be my, my good well, maybe we’ll go back to Emmy. Want to be in the good interview. Let’s go for it. All right. Everybody can open the script. You can see this here, Emmy, I’m going to have you open this link.

Jim Morris 

I got that right. I did perfect. You should see a zoom prototype. I in fact, what I’m going to have you do is go and load that link Emmy, and then put it on your screen and show us.

I’m going to do this briefly, and then we’re going to go into breakout rooms where you’re going to practice this. Okay? And so, mark or Kirk, if you want to set the breakout rooms, it’ll just be two to a room, and it can be random, and it it’s possible that folks that aren’t on the video might be a stepped away. And so we’ll just monitor the rooms to make sure that if people come back, that we can pair them up.

Great. Thank you for sharing your screen.

Those of you, you can open up this script and you can follow along. I’m going to do this by heart, because we’re going to start.

Jim Morris 

Emmy, thank you for taking the time. I’m Jim. I’ll be leading you through the interview. Have you done any user test before?

Audience Member 

No.

Jim Morris 

Great. Well, thank you for for volunteering, and it should be easy. And, yeah, it’s going to be fun. So I wanted to encourage you to be honest. This is a great time in our project, our project timeline to get feedback, so there are no right or wrong answers. So we are just looking for your honest feedback on our product. You’ll note that there are a bunch of colleagues on the line. They may unmute to ask you a question, or they may ask me to ask you a question. And is it okay if we record discussion?

Audience Member 

Yeah, sure.

Jim Morris 

Great. Thank you. Just see the script about sharing the screen. We’ve already done that. Tell me a story, a brief story about the last time you opened up and had to set up Zoom Breakout rooms.

Audience Member 

I don’t do it very often. I’m may use Teams a bit more frequently, but yeah, so in training courses and things like that, sometimes when you go into the training course, you don’t know how many people are going to join. So you don’t know how many rooms you’re going to end up with. And it’s a little bit but bit of an unknown.

Jim Morris 

Yeah, great thank you for sharing. So now what I’d like to do is is go and click an option one, and what I’d like to do is, here’s the scenario where you leading a breakout like a session, like you were talking about, and you’re going to open breakout rooms, and you’d like to enter the room. Do you think needs you the most to create a healthy discussion? So go ahead and do that. Open the breakout rooms.

Audience Member 

I clicked on option one. I don’t know where I’d click actually, where? Oh, then the rooms are down there. Okay. I

Audience Member 

I don’t know what I do here.

Jim Morris 

Go ahead and open the rooms.

Audience Member 

Okay, right? I’m here. This is familiar

Jim Morris 

Looking here. Which of these rooms? What would you do next?

Audience Member 

In, I’d probably, if I was just trying to keep things moving, looking at this, I would probably go to the room with all the red dots.

Jim Morris 

Why is that?

Audience Member 

Um, because the red stands out as sort of quite a strong signal that that room needs help, um visually, but then I don’t know, because then looking at it in a bit more detail, they’ve all got their cameras on and their mics on, where it says room two, where two of the people have no mic and no camera. That’s probably the room I should go in.

Jim Morris 

Yeah, So thinking back to when you were a session, that would be the room you would enter. You had this information.

Audience Member 

I think. So, yeah, Daria is a bit quiet, and it’s also got a camera off in room one, but there’s, yeah, that’s probably the room room two,

Jim Morris 

Right. Okay, Type R on the keyboard, and that’ll take you back to the beginning.

Audience Member 

All right.

Jim Morris 

And then I would run Emmy through the rest of these. Let’s go and give it a clap for Emmy. Thank you so much. You can unshare, and that’s just a taste of it. What were the differences that you saw?

Audience Member 

She had a she had a lot more context in the second interview about where what she was looking it at fit into the into the bigger picture. She wasn’t just being asked to like, look at these screens without understanding what exactly they were for.

Jim Morris 

Yeah, so be careful of dropping people in the screens, even if they’re the right the relevant customer. Yep. What else?

Audience Member 

Is it she had more control over it as well. You were asking her to use it. You weren’t just, like I said, dropping her in. So she had to work her way around it as well, and giving her that option as well.

Jim Morris 

Yeah, that helps her really derive that con that context. You know how, when she struggles to find the break the open rooms button and then she opens it, she knows she’s opening rooms?

Mark Littlewood 

Yeah, she did a very good job, actually, of describing her thought processes far better than I would have, because I would just be stabbing at things and not articulating it. So it’s good interviewee.

Audience Member 

True.

Audience Member 

The thing that struck me a little bit was that, although you’re obviously doing ABC testing on the on the the breakout room descriptions, it also potentially surfaced the fact that the Breakout Room button is hard to find.

Jim Morris 

It is hard to find. I agree, Nigel. What else? Fix that?

Audience Member 

She was able to discover it in her own time. So she was able to kind of look at something then think again. We didn’t just jump in with her first answer and go, Oh, good, thanks. Thanks for telling me that. Here’s the next question. She had time to discover.

Jim Morris 

Yeah, time to discover. I agree. What else?

Audience Member 

It felt like. There was a few different things that were being tested and it could be tested at the same time. There was the color of the buttons. There was the video on, video off. And again, it was interesting that she could look at it and spend time, because the things that jumped out of it at first was the kind of traffic light colors, but then then lay to she kind of clocked the video and the sound thing, but they weren’t as prominent, but sort of having the scope of lots of quite a few different features, but with a specific problem, kind of enabled us to understand how useful those features are with a specific problem in mind.

Jim Morris 

Yeah, we we know the design is awful looking. But in that first interview, the bad interview, we didn’t necessarily understand her thought process about which room she would choose, and we thought to ourselves, oh, the design’s gotta be better in order to get better feedback. But when I said, What would you do? And I gave her the pressure of of actually making a choice, she started to like, her brain started to squeeze, and she thought, Oh, I’ll go to the red light, like you said. And then she’s like, well, actually the quiet rooms are the ones that bother me. And so there isn’t actually a right answer to this, to this prototype, which is why it’s a good experiment, because people do have different reactions. But I have the same reaction as Emmy. I would, I would go to the quiet rooms, those terrify me. But yeah, as you go through the prototype and you ask your questions, there are different there are different hypotheses that I’m making about talking or not talking, video or not video, and I use different design elements to bring them out. But really I’m the design elements are just to get into Emmy’s head. Yeah,

Audience Member 

yeah. I think the traffic lights interesting because it’s so visually prominent, then red means bad, but, but Emmy decided that actually all red in Einstein wasn’t bad. They were all talking. Some of them were too loud and some of them were a bit raw, but they were all talking, and that wasn’t bad compared to white, which was actually really bad.

Jim Morris 

Yeah, yeah. So the make sure I’m sharing this perfect okay.

Jim Morris 

So user control to prototype user talks, or their setup screens to give context. We asked them to tell us a story, someone to ground them in a real story. If she can’t tell me that story, then we’re actually going to end the interview, because it authenticates that relevant part. Remember, relevant strangers, not friends, not family, not co workers, but they need to be relevant to us. It’s like never opened a breakout room. Their feedback on Breakout Room health is going to be useless. Okay? That’s the difference between value testing and usability testing, right? You need that authentication.

And then I at the end of the interview, you’ll see in the script, there’s a survey question about which of these would you be disappointed if you’ve never had access to and I think what I did after was, which room would you go into, and why? And I kept going at which room, and so that’s where, when you’re done with the interviews, if you’ve recorded that for five users, you have pretty good sense. And then skin in the game. I didn’t offer this because I kind of cut it off, but at the end of all of them, I would say, What if you got this at, you know, a discount of 50% and you can sign up right now, you know, get it for $20 extra a year instead of $40 then the script. Let’s do that, and it’s hypothesis focused. I don’t have a set list of questions. I do have a set script in the beginning, but then I don’t have a set set of questions, because I’m really looking for the answer to video, not video, red light, not red light. Like which room we should go into. I don’t know exactly where the interview is going but I know that I set up a good experiment. So as you’re going through this, you can remind people to think out loud. You can remind them of the prompt. What would you do next? Which room would you enter? Can Repeat these phrases safely. What would you do next? If they ask you a question, ask them the question back, well, what do you think it should do? And then if they start talking about other folks, like Amy’s like, Oh, I’ve got a friend. And I bet this friend would be really into this. Forget about it. Just get the friend’s email and interview the friend. Okay, so when you’re stuck, think of these areas. Okay, so follow the script. Sort of you’ll see the script here. Hopefully you’ve all opened it. These are the part. This is the prototype I made through screenshot in figma. Personally, I made this so you can do this too. Okay, I use a tool called Snagit. At the end of the deck, there’s a list of my tools. You collect a story prompt the user, and you’ve got these hypotheses. And then he can guide the user through options two and three. So what I like to do here is get you into actually interviewing. When you get into the room, you’re going to play rock paper scissors, and the winner becomes the interviewer. Just do it once, and you leave the script, and the other person can sort of share the screen, and then you can act like the user tester. We probably have room least for one interview. It’s up to Marco. They may want to switch and do two, but you’ll get a really good sense of it when you when you lead this, how much time will be open?

Mark Littlewood 

How much time for each interview?

Jim Morris 

Say, probably about 10 minutes to get through a couple options. And we got about 12 left.

Mark Littlewood 

Yeah, we’ll probably do one, because I’m sure there’s going to be lots of things to kind of wrap up on, but we’ll see that. We’ll see how we go. We’ll do that. Great.

Kirk Baillie 

How good. I will send you all into rooms. Good luck.

Jim Morris 

Awesome. Kurt, thank you.

Kirk Baillie 

I can see someone’s got someone left in the middle of doing that that’s typical. Let me just fix that room five, and I’ll bring Mark out of it.

Jim Morris 

And I can go into a room if there’s nobody there.

Kirk Baillie 

I No, it’s fine. I’m just, there’s one of them with three in it, because Mark is in there, because he was filling again. Oh, he’s not needed, but he he was, he’s refusing to come back.

Jim Morris 

Um, well, send me in. That’s fine.

Kirk Baillie 

I’ll add you into that one then.

Jim Morris 

So everybody I can chat you so I can leave the room.

Kirk Baillie 

You should just be able to hit you should be able to hop out and say, Leave room. Okay, beat the bottom. If not, you can raise your hand. I think. Oh, wait, hold up Mark. Mark is back, so we’re back then to two interview room.

Jim Morris 

Here we can, kind of, we could pop around just to make sure people got the script.

Mark Littlewood 

Yeah, can, can we send a message to all of the breakouts and just make sure they’ve got a link to the script in it? Because I think we were slightly struggling to

Kirk Baillie 

Yes, if you could remind me what it is.

Kirk Baillie 

Thank you. So there’s a bunch of people screen sharing, which is nice to see.

Jim Morris 

Hopefully you can see that.

Kirk Baillie 

Yeah, yeah. See in all sorts these.

Mark Littlewood 

How do you tell other people? That is..

Kirk Baillie 

The little green screen shading symbol.

Mark Littlewood 

Are you a co host? Jim, can you see this stuff?

Jim Morris 

I’m not. I can’t see it.

Let me change that, and you can have a look. Make a ghost.

Jim Morris 

I Okay, oh, look at that. If they knew, wow, they actually are doing something.

Mark Littlewood 

They are doing things. So they’re doing a lot of things that I think, I think maybe you’re I get why people are so frustrated with Zoom. And there’s a load of things that appear on my screen on the right hand side the app, sort of just like whatever. But they, they have done some things quietly. But, yeah, I mean, they struggle. I mean they’re they’re struggling. They IPO and their stock price is absolutely enormous, and they’re never really going to be able to justify it, unfortunately, um, because they, I mean, it’s that classic, classic kind of thing, of perfect storm of ipoing at the time when there was real appetite for things, and then the pandemic hit, and it was the only game in town. And so it’s kind of exploded, very difficult to you can’t have exponential growth forever. Apparently, it’s a weird old thing.

Kirk Baillie 

Yeah, yeah, I can see that one room. No one is screen sharing.

Jim Morris 

I’ll go in there and check it out.

Kirk Baillie 

Yeah, maybe worth dropping in.

Mark Littlewood 

That’s interesting. Just you and me in here.

Kirk Baillie 

Correct. And the whole of YouTube, yeah.

Jim Morris 

So I did run over in the intro part so they didn’t get the whole thing, yeah.

Mark Littlewood 

Which is, it’s a shame, but the intro was interesting, so I’ll need to finish this. But yeah, I think it’d be good to hear what people think and maybe giving a few pointers to change or improve and then wrap up. Yep, yeah,

Audience Member 

That was fun.

Jim Morris 

What is, what is, what is it? Did it help you? Have you done interviews before? Liz, like, yeah, tell me about that experience of maybe not doing it now doing it.

Audience Member 

Um, so I do try to, like, follow kind of best practice and stuff when I am doing interviews. But I liked that there was. What I liked here is options, because when I’ve interviewed people, I’ve only ever really showed them one option. So I’ve kind of used these interviewing techniques about letting them talk and speaking out loud and all that. But I haven’t, sort of, is the contract, the contrast, where someone goes, Oh yeah, I really like this, you know, because especially when they’re comparing to something else, like, I felt that, yeah. And also I felt able to like, say, Well, this feature, I like, like that, other option, but with this feature from option three kind of thing, but it’s, yeah, it’s just a really playful thing to do, like I was the participant. I just found it fun to I really, I really like being a participant in user experience studies. Anyway.

Jim Morris 

All right, who else? What other thoughts people have was like.

Audience Member 

I think I wasn’t able to keep being an interviewer. I had to join in as the interviewee as well sometimes and my feedback, which I did try not to do, it was a bit of a technical issue with our side of everybody else had this, but we lost the chat, and therefore lost the link to the to the examples. Unfortunately, I still had it on my screen to able to do that, but I think we lost, I lost the script as well. So I was very much winging it.

Audience Member 

That’s really that’s really that’s going back to what Nigel was saying. It’s very it’s very challenging not to where they’re telling you their story, to kind of join and go, Yeah, I’ve got another story just as good as that one. I found out that’s probably just me. It’s very difficult to pull back on that one and to to take advantage of the silences, I think you were saying, Jim, to allow the interviewee to talk. Sorry, Patrick.

Audience Member 

Oh, no problem at all. I thought it was your greatest interviewer.

Audience Member 

Oh, he’s very kind. He was very good as an interviewee, excellent.

Mark Littlewood 

They’ve basically just been plotting a mutual admiration. Exactly, interviews at all.

Audience Member 

It’s about two o’clock actually. Yeah, it’s been really good. I really enjoyed that bit stressful. But yeah, it’s good.

Audience Member 

I think the other thing, as well as like, trying to, like, be in listening mode, not talking, because it’s not about you, the thing that’s the thing, that other thing I find really hard is when you’ve when it’s your thing, and you want it to work, and it’s not working, and it’s like, oh, okay, yeah, that’s really interesting. Folks, yeah. What would you do next? Whereas inside, you’re like, No, it’s not work.

Audience Member 

Yeah, yeah, yeah.

Audience Member 

I was, I was with Laura, and she’s a UX designer, and I thought the whole thing highlighted the importance of having a script which was very useful, keep you on track and make sure that, make sure you you don’t deviate too much. But one of the things she said she would sorry for speaking for you, Laura, but said that she always adds in when she’s the interviewer is she tends to put in a qualifying statement saying, I didn’t build this. Don’t worry about my opinion. You say whatever you want. It won’t bother me, which I completely agree with.

Mark Littlewood 

Even it’s great feedback.

Audience Member 

I always, even if you’ve

Jim Morris 

built that, yeah, especially if you’ve built it,

Audience Member 

I always say, Yeah, you’re not lucky, not like I hurt my feelings. Yeah.

Jim Morris 

That’s great. Laura, nice work.

Mark Littlewood 

So true one, I mean, one thing I picked up on the conversations coming back was that comment about it being fun. And I, I’ve noticed the people, when they’re doing interviews, it’s like they it’s just like this huge mountain they have to climb, and they they’re convinced that if they speak to their customers, the entire company is going to collapse and all their customers are going to die. Or they something worse than that. I mean, I just don’t quite get that thing of, well, we can’t talk to our customers because we’re just really going to annoy them or whatever. And then people say, Oh, we’ve talked to our customers and but we know what they want, which generally means they haven’t spoken to their customers, and that they really need to be doing it a lot more. So there’s a whole load of kind of excuse making. And then it’s it feels like it’s a real bind and a drudge to do so delighted with the you know, it felt quite playful and fun, because they should be, and people love talking about stuff, and it makes people feel really important to be given something and given free reign to ask their opinion. And if you set those things up in the right way, it it’s energizing and it’s successful.

Audience Member 

I think it also like builds empathy, like every time I speak to a user or a customer, I just feel such affection towards them and empathy that then it really drives it’s very motivating to then for driving you to do good work. Because, you know, like, who’s going to be using it and like, you can imagine the sad little face if you don’t do it very well.

Jim Morris 

Well, you know, I was only one degree of separation from customers for a lot of my career at this power abuse company as CTO, I would get feedback about customers needs through all channels, CEO, head of sales, head of marketing, customer success, tech support, implementation of folks, they would come talk to me about this, and to me, this input was actually annoying, and I really learned to dislike my customers and I spent about eight years I didn’t do all these practices most of my career. I did them towards the end of my career, because when I was exposed to them, I realized there was a different way to do it. And once I started talking directly to customers on that listening tour, to folks a week, doing interviews with people who use reviews to make choices online. It changed my perspective, and I learned to love my customers. You know, I wasn’t sure that I could build everything they wanted, but I knew why they wanted it, and I respected their their asks.

Audience Member 

It’s a really good point. I think back to what Liz was saying, but with adapting our product management much more now, but we would, there’d be a small window of opportunity when our customers could talk to us, we would talk to them, we’d get their feedback, and then we would shut that window and that beer, I think, and I enjoy it. I enjoy talking to customers. I enjoy talking generally. That’s just Yeah. So being able to, I think a company’s turning itself, working for Cambridge University Press and assessment, a couple of other colleagues here to be encouraged to go back out and have continuous conversations as we develop, as we move forward. I think that’s really exciting. That’s really good. That’s a real that’s a real positive movement, I think, from the company, because it’s enjoyable. It’s fun, like you said. Mark Liz, we do get the empathy when we talk to them, which is necessary when we develop products. Absolutely great.

Mark Littlewood 

Amazing, Jim, anything you want to because I’m conscious we’ve kind of run over and, like all of these things, I could talk till the cows come. And, yeah, there’s, there’s never enough time.

Jim Morris 

So there’s more links in the PDF, you know, that you can look at in my presentation, and then just the two things I always wrap up with is, do you need a professional UX researcher to do this? The answer is, no, I coach my teams. 95% of my teams of the 150 teams I’ve worked with do not have UX researchers. They have people like us who are well intentioned, who learn a skill, and as long as you’re interested in someone’s feedback, they’ll feel that. And set up a good experiment so that you get the good feedback. And then you can use my script. You have the script here, feel free to copy it and use it. Okay? And then how many people should you talk to? A week? I love you to talk to three people. Don’t all. You don’t always have to do a solution test interview. You can just sit in on a tech support call. You can sit in on a sales call, you can develop the empathy in small chunk. And you just get mechanism inside your company where your product managers, your heads of engineering, your designers, you’re just hitting in on calls on a regular basis. Best of luck. You’re welcome to reach out for me, to me if you have any questions. I. Them on LinkedIn and various platforms. You can see all the graphics that I make on Instagram. So if you’d like to follow graphical things and you don’t want to read, could do that? Been great to hang out with all of you. Thanks for having me.

Mark Littlewood 

Amazing. Thanks so much, Jim. And really appreciate your your early start this morning, and I’m very glad that the sun has made it all the way around the world. To to you, we did see it over here today, which is unusual. Brilliant. Thanks so much, Jim.

Jim Morris

Founder, Product Discovery Group

Jim is based in San Francisco and learned how to build and scale successful products as an engineer, product manager and leader at startups where he’s turned ideas into successful products many times. Starting before the first dotcom boom, he designed and product managed one of the first ecommerce systems at Fogdog.com before a $450 million IPO. He also co-founded PowerReviews which grew 1,200+ clients and a $168 million exit.

Jim graduated from Stanford University with a BS in Computer Science and is now coaches Product Management leaders and teams in startups, tech & Fortune 100 companies including Kaiser Permanente, PagerDuty, VSP Global, Dictionary.com and Hallmark. You can find him at ProductDiscoveryGroup

Next Events

BoS Europe 2025 🇬🇧

🗓️ 31 March – 1 April 2025
📍 Cambridge, UK

7 DAYS LEFT to grab Early Bird Tickets

Spend time with other smart people in a supportive community of SaaS & software entrepreneurs who want to build great products and companies.

Lock in Early Bird Pricing

BoS USA 2025 🇺🇸

🗓️ To be announced soon
📍 Raleigh, NC

Learn how great software companies are built at an extraordinary conference run since 2007 to help you build long term, profitable, sustainable businesses.

More Details