Rahul Vohra on The Product-Market Fit Engine

About ten minutes in to this talk by Rahul Vohra, CEO and Founder of Superhuman, I tweeted…

I’m getting a feeling I might be listening to a genius. #BoS2018

— (@MarkLittlewood) October 3, 2018

This will change the way that a lot of people think about product market fit, BS metrics, understanding the needs of the people that really matter.

Find Rahul’s talk video, hangout, slides, transcript, Rahul’s blog about this talk, and more from Rahul below.

Find out more about BoS

Get details about our next conference, subscribe to our newsletter, and watch more of the great BoS Talks you hear so much about.

Video

Hangout

Slides

Transcript, The Product Market Fit Engine

Rahul Vohra: Good morning everyone. How are you? So I’m going to talk about Product-Market Fit. I couldn’t imagine a better backdrop for that. Then this evolution of catapult s and trebuchets… though I should point out that a trebuchet is a very specific thing. I was. Yeah, and I forced our team to build a trebuchet – in retrospect, that was a very bad idea. Anyway, my name is Rahul. I’m the founder and CEO of a company called Superhuman, and we make the fastest email experience in the world. Our customers get through their inbox twice as fast as before and many of them see inbox zero for the first time in years. This is the story of how we’ve built a Product-Market Fit Engine. So Product-Market Fit is the number one reason why companies succeed and the lack of Product-Market Fit is the number one reason why companies fail, but what really is Product-Market Fit? Paul Graham, the founder of YC, would say it’s when you made something that people want. Sam Altman, the president of YC would say it’s when uses love your product so much that they spontaneously without you asking them, go and tell other people to use it, but it’s perhaps Marc Andreessen who has the most compelling, most vivid definition that I’ve come across. He says, Product-Market Fit… You can always feel it when it isn’t happening. Your customers aren’t quite getting value. Usage isn’t quite growing. Word of mouth isn’t spreading quite fast enough. Press reviews are kind of blah, sales cycles take too damn long, but you can almost always feel it when it is happening. Customers are buying as fast as you can add servers. You’re constantly hiring for sales and support. Reporters are constantly calling you. Investment bankers are staking out your house. You just got Company of the Year award from Harvard Business School. It’s certainly a vivid definition and one that I was staring at through tears in the summer of 2017. It seems so subjective, so unactionable. What do you do if by this definition you don’t have Product-Market Fit? Indeed, can you measure product market fit?

If You Can Measure, You Can Optimize

If you could, then maybe you could optimize it and maybe just maybe you could systematically, perhaps even numerically increase it. Well, spoiler alert… it turns out you can do all of those things. You can measure it. You can systematically increase it. You can numerically work your way to Product-Market Fit, but before I share how, let’s wind the clock back eight years in 2010, I started this company, Rapportive. We were the first Gmail plugin to scale to millions of users on the right hand side of Gmail we showed you what people look like, where they were based, what they did, their recent tweets, links to their social media profiles. We grew rapidly and less than two years later, we were acquired by LinkedIn and in those four years I developed a very intimate view of the email space, Gmail, which was once really fast and very clean, I could see becoming worse every single year, the UI becoming more cluttered, the code leaking more memory, requiring more CPU, slowing down your machine and inexplicably still not working offline. And then on top of that, people were installing these plugins like ours Rapportive, but also Mixmax, Boomerang, Clearbit, Yesware, you name it, they had it and these plugins took each of those problems, clutter, performance, memory, offline, and made every single one dramatically worse. It was time for change, so we imagined an email experience that was blazingly fast where search is instantaneous, where everything happened in 100 milliseconds or less. An email experience where you could do everything from the keyboard. You never had to touch your mouse. You could fly through your inbox. An email experience where offline just worked so you could be productive from anywhere. An email experience with all of the best Gmail plugins built in natively and which was still somehow subtle, minimal and visually gorgeous. And so in the summer of 2015, we opened an office (it’s very fancy as you can see) and we started to write code. And in the summer of 2016 we were still writing code and in the summer of 2017 we were still writing code, although there are quite a few more of us at that point. I felt this incredible intense pressure from the team and also from within myself to launch what we had built. After all Rapportive, the previous company, started, grew, scaled, and was acquired in less time than we had been writing code. We were two years in and we still have not launched. But no matter how much pressure I felt, I knew that deep down inside if we launched it would go really very badly. It wouldn’t be the Marc Andreessen story. I did not believe that we had Product-Market Fit, but I couldn’t just say that to the team. These are hyper ambitious, super intelligent engineers. They’ve poured their hearts and souls into building this product. That’s not the message they wanted to hear, and so I started my search for the holy grail for a way to define Product-Market Fit, for a way to measure Product-Market Fit, and for a methodology to increase Product-Market Fit. I searched high and low. I read everything I could find. I spoke to all the experts, and I found this guy, Sean Ellis.

Sean Ellis & The Engine

Who here has heard the term growth hacker? Let’s raise our hands if we have. Okay. That’s pretty much everybody by this points shown, invented that term. He ran early growth at Dropbox, LogMeIn, Eventbrite and many other great companies too. As vivid and as compelling as the Marc Andreessen definition of Product-Market Fit is, and it’s an accurate definition, it’s a lagging indicator. By the time that investment bankers are staking out your house (and isn’t that what all of us want?), you’ve already won. You have Product-Market Fit. Sean found a leading indicator, one that is benchmarked and one that is predictive. You simply do this, ask your users the following question: how would you feel if you could no longer use our product? And you let them say very disappointed, or somewhat disappointed or not disappointed at all, and then you count the percentage of people that said very disappointed. This is a very simple metric. And what Sean found, benchmarked on over 150 companies, is that those companies where the users who would be very disappointed without your products represented less than 40 percent of respondents – those companies almost always struggled to grow and struggled to get any kind of meaningful traction. But the companies where the percentage of respondents who were very disappointed was more than 40 percent. Well, those companies grew easily. Those companies found traction easy, and many of those companies went on to achieve great success.

This is an example of that question being answered by 731 Slack users. 51% of those users would be very disappointed without Slack. 51 is greater than 40. Therefore, Slack has Product-Market Fit. Okay, so today that may seem abundantly obvious, but the purpose of this example is to show you just how hard it is to beat this benchmark. It’s not easy. This metric is more objective than a feeling. This metric is more predictive than say, NPS, and this metric is not only the best way that you can find to measure Product-Market Fit, it even lets you build a Product-Market Fit Engine, and this Engine gives you a way to systematically, even numerically increase your Product-Market Fit. It will even write your roadmap for you as we shall shortly see.

The engine has 5 stages. Survey your users; Segment them; Analyze what they say, why do they love it, what holds them back; Decide what changes are going to make; and then implement those changes and track. And we’re going to dive into each one. I’m going to give you the blueprints for how you can do this on your own products at your own companies.

Step 1: Survey

Step 1: Survey. So these are the 4 questions that you want to ask all of your users. How would you feel if you could no longer use the product that we just discussed? Number 2, what type of people do you think would most benefit from your product? Number 3, what is the main benefit of the product to you? And number 4, how can we improve the product for you? This is a very short survey. The 3 latter questions, they should have open ended answers. You want to get ideally long text responses from people and each of these questions is very important. They come into play at different points in the Product-Market Fit Engine. Once you have these questions, you want to send the survey out to all of your users. You want to give your a users an opportunity to try the core experience of your product. They should at least have used your products twice. For Superhuman, we send it about 21 days into the journey of a user and these are the results that we got in April of 2017, A mere 22% percent of our users would have been very disappointed without Superhuman. We were clearly nowhere near Product-Market Fit and that may seem sad, but now at least I had the language and the tools to explain that to my team.

Step 2: Segment

So what can we do to increase the percentage of users who would be very disappointed without the product? What can we do to increase that very disappointed segment of this pie? Well, that’s step 2 and it’s all about segmenting. And this part’s kind of like magic. It goes real fast. So we want to understand who are the people who really, truly deep down love our product and the way that I like to do this is using a concept called the High Expectation Customer. This is a concept I found from Julie Supan who ran early marketing at AirBnB, Dropbox and many other great companies too. The High Expectation Customer is the most discerning person in your target market. They will enjoy your product for its greatest benefits and importantly they will help spread the word. Now other people who might not be quite as high expectation will aspire to be like this High Expectation Customer because they see them as clever or as judicious or as insightful. You want to create as rich a profile as you can have your High Expectation Customer, but let me give you some examples. Let’s think about Dropbox. The Dropbox HXC wants to simplify their life. They’re very trusting, the technically sophisticated, and they really just wanted to get time back in the day. At the end of the day, they want to know that a service, Dropbox, has their back when it comes to their life’s work. I’m a Dropbox HXC and I’m sure many of us in this room here today as well. Here’s another example. AirBnB, the AirBnB, HXC does not simply want to travel. They want to go somewhere and experience it like they belong. They want to go to Paris and live like a local. They’re energized by that kind of experience. AirBnB’s great successes came from in the early days, focusing on high expectation customers like these that then drove tastemaking in the rest of the market.

So here’s the Insight. You want to go back to the survey results. You want to take all of the respondents who said they’d be very disappointed without your product and you want to analyze in detail their responses to question number 2. Question number 2 was, what type of people do you think would most benefit from your product? And the cool part is that people who love your product will almost always describe themselves in the words that matter most to them. Notice we didn’t say, can you describe who you are? We didn’t say, what do you do? We didn’t say anything like that. We just said, who is this best for? And correlated it with who most loves the product. Try it. You’ll see. They almost always describe themselves and you’ll get it in the most authentic words possible. You can then take all of that data and build a rich profile. I’d like you to meet Nicole. Nicole is the Superhuman High Expectation Customer. She’s a hardworking professional. She deals with many people, she might be an executive or founder and investor, a manager, or many other job roles. She works really long and actually often into the weekend, she considers herself busy and she wishes she had more time. She does feel productive though, and she is self aware enough to realize that she could be better and occasionally she’ll have time to investigate ways to improve. Now she spends much of her days in her work inbox. On a typical day, she’s going to receive and read at least 100 emails, perhaps 200, maybe even more, and on a typical day she’ll send 15, 40 if things are getting really crazy, maybe 80 emails a day. Critically, she considers it part of her job to be responsive and she prides herself on being so because she knows that when she isn’t responsive, it will either block her team or it will cause her to miss opportunities or she’ll damage her own reputation. She aims to get to inbox zero, but she doesn’t get there very often, perhaps two or three times a week and very occasionally, perhaps once a year she’ll say to hell with this and just wipe her inbox clean. Declaring email inbox bankruptcy. Now she generally has a growth mindset. She is very open minded about new products and keeps up to date with technology. However, when it comes to email, she probably does have a fixed mindset. Gmail is what it’s always been and that’s what it’s always going to be. While she is open, in theory to the idea of new clients, she’s sceptical that one could make her go faster. This is the level of detail, if not more detailed that I encourage you to design your HXC for. Now once you have this persona, we come back to the survey results and we tag each one with who they are. Here you can see the very disappointed crowd, the people who love the product the most up in the top left. Down at the bottom, you have all the people who are feeling ‘meh’ about the product and in the top right you have all the people like, I really don’t like this thing, it’s not useful to me at all, and here’s the magic. You take your HXC, you take the persona profiles that you just assigned to these very disappointed users and you use that to narrow the field. In this case, we’re deliberately ignoring everyone who isn’t these people – we’re deliberately ignoring sales, customer success, engineering, data science, any other role. I’m focusing only on the roles that most correlated with the people who really love our product. And the impact of that is huge. It takes just a few minutes and in those few minutes we added 10 percent to our Product-Market Fit score. We jumped from 22 to 32. Now we’re not quite at 40 percent. Yes, but that’s really high ROI from just a few minutes of analysis.

Step 3: Analyze

Speaking of analysis, that is the next step of the funnel. So we’ve done a survey, we’ve done our segmentation, we figured out our High Expectation Customer, we know the profile of the person who really loves what your product is. Now we need to answer two more questions. Those people who really love your product, why? What is it about your products that they love and just as importantly, the people who don’t love your product, what’s holding them back? So we’re going to do two different analyses. You want to go to question number 3. What is the main benefit of the products to you, and again, you want to scope this so only read the answers for the people who said they’d be very disappointed without your products. Here’s some Superhuman examples: processing email is much faster, I get through my inbox in half the time, the app is crazy fast, the shortcuts make me an actual Superhuman, I wish that were true, faster responsiveness, navigation, Superhuman is so much faster than using Gmail, more efficient with my time, I can work through my incoming email more quickly, speed, aesthetics, everything from the keyboard, speed, and a great set of shortcuts. Read through these, load it all into your head and the easiest way to get a grip on what people are saying, it’s just to throw it into a word cloud. We print this out and we stick it really big on our wall. You can see as clear as day that the people who most love Superhuman, love it for its speed, its keyboard shortcuts, and the sense of flow that you have when you’re in the product. So that’s why people really, really love your product. You now need to go back to this pie and figure out how on earth are we going to increase the size of the somewhat, sorry, increase the size of the very disappointed crowd. And remember the very disappointed crowd are the people who’d be very disappointed without your products. They are the people who love you. Now as painful as it is for me to say this, do not pay any attention to the people who say they would be not disappointed without your product. They may actually be your loudest users and they may be your most demanding and they may ask for all kinds of things, but guess what? At the end of the day, that’s all just distraction. You can build all the things they want and they would probably still be not disappointed without your product, so just discard those responses immediately, but the other crowd, the somewhat disappointed crowd, we can do real work there. Again, I can’t stress this enough. Do not just go and directly act on their feedback. You will end up with a mess of a product and you won’t increase your Product-Market Fit Score. I can guarantee it. Here’s what you do. Instead, we need to figure out which of these people to pay attention to, which of these people, if you built that thing, if you service their requests could convert and become fanatical and would be very disappointed without your product. So we segment again. We take the main benefit that we discovered in the previous step, which was speed primarily and keyboard shortcuts secondarily for us, and we use that to segment the somewhat disappointed crowd. Look at their surveys, see how they answered that question, how many of them were also talking about speed? Those are the people that you should focus on, and in our case that was two thirds of this segment, the other one third, as painful as it is, do not work on their requests. You’ll just waste your time. But the two thirds who agree with the people who really love your product about what makes it special, you work on their requests and you can convert them into being very disappointed.

Here’s what they asked for. So this is the analysis of the 4th and final question. How can we improve the product for you? Once again, go through all the survey results, read them all. I recommend organizing a group meeting, get all your product leaders, your engineers in one room, reads through the mall, and then summarize it like this and put it up on your wall next to the other circle. These other things that are holding people back. People who had this close from falling in love with your product. For us it was a mobile app. Okay, that makes sense. That’s not very interesting, but the other things beneath it are much less obvious and much more interesting, better integrations, better handling of attachments, calendaring, better search, read receipts, unified inbox, better handling of unread. These are things that a typical product management process wouldn’t find. Typical product management process is going to be looking at what the market is going and looking at what the competition is doing at maybe all of the feedback from your users triaged. This is a very specific way to focus on the feedback that is most likely to increase your Product-Market Fit.

Step 4: Implement

And that brings us onto the next stage which is to implement. So the whole purpose of this Product-Market Fit Engine is not just a way to measure Product-Market Fit, but to increase it. So here’s what we do. We spend 50% of our resources doubling down. We’re building more of the stuff that the very disappointed users, the people who really love our product, we’re building more of the stuff that they like. In our case that means more speed, more shortcuts, more efficiency, more beautiful things. And just as importantly, we spend the other half of our time, money, and attention systematically addressing the objections, but only of the people who have somewhat disappointed without the product who happened to agree with the people who love the product on what the main benefit of the product is. And you just work down this list. 50/50.

Step 5: Track

The final stage of this is to track. This framework will work. I can guarantee it. But it is not a silver bullet and things changed and your users change and the market changes. So as you’re working on these pieces of feedback on doubling down on what the very disappointed users like and systematically addressing what holds back the somewhat disappointed users, you should constantly be surveying, send that survey out to every single user who signs up, builds this Product-Market Fit Score into the dashboard of your company. In our company it’s one of the very few numbers that we actually run the organization with. Make it very large, make it visible, report on it weekly, monthly, quarterly, and I would recommend doing that VD/SD roadmapping exercise on a quarterly basis. Here are the results from Superhuman. So in that quarter, Q2 of 2017, as you recall, after our segmenting Ninjitsu, we jumped from 22% to 33%. A quarter later we were at 47%, we raced past the 40% benchmark. But there’s a law with all metrics is that they all go down over time you’ll lead quality will go down over time. Excitement about your company will go down over time. And so you have to keep on pushing. A quarter later we were at 56% and a quarter after that, we were at 58% and you can see this beautiful asymptote beginning to happen as we saturate that 2/3 of somewhat disappointed uses and start converting them over.

This Product-Market Fit Engine really does work and I’m shocked that more people don’t know about the underlying metric and haven’t built it into a way to operationalize how we think about building product. It gives us a way not just to define Product-Market Fit, but a way to measure it, and it gives us a methodology for increasing it. I really hope that some of you, when you go home, take this and get to try operationalizing it inside your own companies because it’s been transformative to our own company and if you do, please let me know because I would very much love to help you along. I’m still learning this stuff as we go. Now I’m going to break here early for questions. We have about 20, 25 minutes left because this is a very nuanced topic and there’s a lot more detail behind that, failed approaches and so on that I’d love to share with you, but I thought you guys instead could ask me what’s on your minds, and so with that we’ll go over to questions. Yes.

Q&A

Audience Member: Great. Uh, thanks very interesting. I wonder, uh, I see a lot of products where the segment of people who would be very disappointed is too small to sustain the thing we’re doing. What do we do when that’s not a big enough slicing and growing? It’s not going to be sufficient.

Rahul Vohra: The rule of thumb that I have is if your product is in the 15 to 20 percent very disappointed range or less, then you almost certainly have entirely the wrong product or entirely the wrong market. And I would not advise using this approach if you’re in that 15 to 20 percent range. We were actually close. We were only 22 percent, but I had just enough conviction in what we were building and the segmenting exercise added another 10 percent. So actually I should rephrase the answer. Do the segmenting exercise first. If you’re still only 15 to 20 percent, it’s time to dramatically change the product or the market.

Audience Member: The four questions are great. How did you determine what role the different people played? They were a developer. They were in sales, they were this. Did you have some additional questions you asked?

Rahul Vohra: Yeah, let’s head back to the four questions. It is the second question. It’s actually one of the most important ones. Okay. There you go. So notice the second question is what type of person do you think would most benefit from this product? Now it’s very important in this methodology that you don’t just ask who are you and what do you do? Because you want the words of the user. There’s a whole separate talk I can do on positioning… In order to market our products that has product market fit well there is no better set of words than what came out of a user’s mind and that’s what this question is designed to elicit. So that’s how we answer. What role are you, not just what role are you, but how would you describe what role you are? Um, it won’t give you everything, especially for the somewhat disappointed users, not disappointed users, and for them, I would recommend just taking all your emails, throwing it into an email lookup service like ClearBit or FullContact or building your own version of that.

Audience Member: A very nice presentation. I like it very much. I’m always looking for ways to get information for our, from our customers. And my question is how you get your customers to answer this.

Rahul Vohra: Yeah. I just go and visit them one by one. No, not really. We send an email. We send an email 21 days, roughly speaking, after they’ve started to use the product. You’re probably wondering, well, does this work? If not everybody answers, and clearly not everybody’s going to answer this, but that’s why the 40 percent benchmark comes in. That number has nonresponse baked into it. That’s why I said you shouldn’t Nag. You shouldn’t be like, hey, fill out the survey. Hey, fill out the survey. Hey, fill out. No, because that will skew your numbers, just send the email and include a link. We, we obviously don’t send this piece of paper, we have a link to a typeform and that’s all you need to do.

Audience Member: Hi Rahul. So this is actually a piggyback now and what Ricardo just asked, do you use resend this question to users who have been using this for a longer period of time? Uh, and the reason I asked this because our user base tends to be fairly static and getting a good response set, I think overtime might be a challenge for us. So, and doing it quarterly might fatigue people to see that same message constantly. So do you have any advice for a company like ours?

Rahul Vohra: Sure. Can you give us a rough idea? Order of magnitude of the use of set size? Yeah, 10,000. Okay. Um, so first of all I would definitely not multiply survey, you will absolutely get skewed results if you ask the same user this questionnaire multiple times. Uh, and secondly, it’s really interesting, but if you do a quick analysis on how many responses you need in order to start getting statistically significant results, it’s really not that many, even at around 50 responses, you’ll obviously have to survey more people than 50 to get 50 responses, but even with 50 responses, you start to get statistically significant results from this. Uh, so with, with the size of the user base you have, I suspect you’ll be able to get at least a directionally correct idea of how you’re doing. Okay.

Audience Member: Rahul, thanks very much for the great talk. I hope this is recorded as I’ll make our marketing guy watch it several times. On the same lines, are there any concerns about the sincerity of the answers? A customer just wants to make you feel better and answers, uh, in a skewed way. And a second question is, do you think about any kind of smart way to integrate that polling into the product itself? Can you share your thoughts on that?

Rahul Vohra: I think that’s the genius of this question. Uh, and, and Sean did a great job in coming up with it. It’s very different to a 10 point scale and I think Jared did a good job of systematically taken apart. It’s very different to net promoter when you ask people, do you like me? Then they’ll be like, yes, I guess I like you go away now please or something. Those questions are very leading in my opinion, but when you turn the question around and you ask it in this negative way and you say, how would you feel if this thing worked here? And then you limit the response size from 11 or 10 down to just three possible answers and everyone can, everyone can intuit the difference between very disappointed in somewhat disappointed. You actually remove almost all of that bias. Now I would say if the, if the person you’re asking actually personally knows you, yeah, they’re way more likely to say very disappointed. And we had that for perhaps the first 50 survey results that we got back in. And so for the purpose of the numbers that I showed on the screen, we just completely ignored those surveys. We ran this whole thing for a quarter or so before we started to pay attention to it. Uh, the second question was other smart ways of asking people this question. Uh, yeah. I mean there’s any myriad number of ways we make an email product and so we like to keep things email native. Uh, but I could imagine having the intercom operator asked this. I could imagine having a text message if you have a mobile app, ask this or an in app notification or really any other way that you can reasonably interrupt a user.

Audience Member: Hi there. Thank you so much for your talk. Uh, I’m curious as to you how you would apply this method when I know we’re at a software conference, but if you don’t have a product but rather a service. So for example, a use case would be a tech agency that builds iOS apps or uh, maybe, maybe, um, provides like content marketing services. How would you go about finding a Product-Market Fit in that case?

Rahul Vohra: That’s a good question. Let me think about that for a second. Okay. I have a question for you. This, you know, this survey is really designed to get at a few different things, its dependency on the product. It could be uniqueness in the marketplace, differentiation and also the emotional bond that people have with what it is you’re providing. So if you were to describe in a sentence or two, how is your agency differentiated in the market, then I might be able to answer your question about it.

Audience Member: So, uh, I work for several different agencies so it would really depend on which one I’m working for. Um, but I do think that, um, the, the commonality between the different clients I tend to work for is that they are, are exceptional when it comes to providing user experience. So, so that’s what I can answer you, but it is case by case. So.

Rahul Vohra: Okay. So in this case I would say that actually you would probably be better off with another metric that is designed to get to the things I would suspect you care about. I would suspect you care about the willingness to recommend, for example, uh, and that measures a different thing to what this meshes. This was really benchmarked, if you’re going to apply the 40 percent number against high growth, usually venture backed companies that are creating a product that is designed to play a meaningful part in someone’s life. For a consultancy or an agency where almost all new work is coming in through referral, then I would ask a question that is specifically designed for that.

Mark Littlewood: That’s amazing. Now Tyler, first time at BoS. This guy is pretty, pretty awesome. I had a little interview with him yesterday. I’ve got a new phone so I’m not very good with it. And um, I asked him what he felt about BoS and he gave me like a little 30 second answer and at the end of it I had a fantastic picture. So I then worked out where the video button was. And then off the cuff, gave another 30 second answer, which was precisely the same thing he’d said. And I don’t think I’ve ever had someone just extemporize like that so precisely. So, Tyler.

Audience Member: Thanks Mark. Thanks for the great talk. I really loved it. It seems to work or it seems to be a very applicable process for a situation where you’re selling to someone who’s making the purchasing decision and there the majority of the users, uh, what are your thoughts on a situation where, uh, the, you’re selling to someone but the majority of the users, for them it’s a compulsory, like in more of a B2B scenario where you sort of sell to smaller amounts of purchasing agents and then they sort of distribute the software, the tool out to a majority of users who don’t really drive that, that purchasing decision as much.

Rahul Vohra: I would, I would actually say a very similar thing to what I just said, which is we need to be clear about what objective we’re trying to drive for the business. I don’t know the particulars of your business and so it might be the case that if you can successfully sell lots of your first layer, the next layer takes care of itself. That sounds pretty cool if so. It might be the case that that first layer really needs to see true love and evangelism in that second layer in order to keep on working with you over the longterm. If it’s the former, then I would use this, but on that first layer who then resell you? If it’s the latter, if you really care about making huge impact in the lives of the end users and that’s what’s going to make your company successful, then I would survey them. I suspect that in practice it’s going to be a combination of both, perhaps short term, first layer, long term, second layer, uh, in which case there’s your answer. In the short term, I would survey the people who resell and in the long term I would survey the end users.

Audience Member: Thank you so much. That was fantastic. I was wondering if you could talk a little bit about the gap between when you said you’d been working on this for two years, um, you didn’t feel like you had Product-Market Fit, but you didn’t know and you felt like you needed something more tangible to tell the team and then this chart because you have to start getting early users and so I’m trying to figure out how you got those users, that gap between what you believed you needed to provide to satisfy a certain market, what you had at two years and then how you got users on board and really tried to separate that feedback from early adopters who were willing to take less versus who you thought your target customer ultimately was going to be.

Rahul Vohra: Okay, great question. Great many questions. Alright, let’s, uh, okay. So I think one of the first questions was how did we have users? Maybe there was a gap at that time. People that we couldn’t serve. I couldn’t ask when we started this process, if I recall correctly, we had around 100 paying customers and our product costs $30 a month. So it’s actually not very, not very many customers at all, but if you recall my previous answer, you start to get statistically significant responses when you have a results rather when you have around 40 or 50 responses. And so it was enough to sort of get a directionally correct idea of, of where we were. It was clearly enough to know that we did not have product market fit and that was the first very difficult thing that I have to internalize. And then I had to figure out how to convey to a team without just saying we’re screwed, yay. Instead you want to say it’s not going well, but there is a plan. We just have to do this thing. The other question you asked was the gap between what I thought we had to build and what we needed to build in order to be successful. So I think we got very lucky in the sense that if you look at what the VD users light, remember the VD users are the ones who really, really love your product. The stuff. This is exactly what I wrote down two years prior saying, I believe this doesn’t exist and I think I just got lucky having been an insider on the email space for many years. I was able to see Gmail decaying, I think faster than other people noticed it decaying and so I just wrote this down and got that one correct. What I didn’t foresee because no one can see everything about your users (they’re a surprising bunch) is the importance of this stuff and that’s where the survey becomes really, really valuable and I’ve been on previous engineering teams where we did not do this and the default mode of operation for that team is this is the normal level of sophistication I see is you get a whole bunch of feedback in. You sort of score it by how severe it seems by how many times it’s been asked for, does this client pay you a lot and then just go work on that list and that’s how you end up wasting resources and wasting cycles and building an undifferentiated product that kind of does a little bit of everything. What this lets you do, if you’re leading a team, is really focus on the things that are going to increase the size of your very disappointed segment, the people who really love your product. And so that’s how I bridged that gap.

Audience Member: Hey, digging into a similar point to as was said earlier, I can see how this would work for kind of high volume, kind of B2C business, but if you’ve got a more complicated buyer group, uh, so you’re selling into an organization, you’ve got end user value propositions, organizational value propositions, you’ve got a sales op, things you do for sales objections which are never part of the end user or emotional jobs to be done for the purchaser rather than the end user. Have you got any thoughts about how you mix all that in if you’re selling this to a, a complex group of buyers rather than just the end user?

Rahul Vohra: Uh, yes. So when this survey was initially being developed, uh, it was initially created for consumer companies and B2B SaaS companies, not the far end of the spectrum of complex enterprise styles. Even in the constellation that you outlined, I would imagine that like with any enterprise sale there is a most important person, there is a most important constituent. Not everyone will, uh, you can’t stack rank everyone into one single line, but there’s probably someone that you care the most about because if you delight them then the energy will start spreading across the organization.

Audience Member: I guess my worry is that it’s not necessarily the end user, the end user is necessary but not sufficient for the purchase in a lot of cases of enterprise software.

Rahul Vohra: Yes. And so if it’s actually somebody else, if it’s for example, the person who runs information technology, then I see no reason why you couldn’t use this survey for them in a directional sense. I just, in that case, I would not obsess over the 40 percent benchmark. I would just use the methodology for increasing the number, the 40 percent benchmark one really make sense. If you’re talking about a high growth venture backed product company.

Audience Member: For the number four implement phase, you said that 50 percent, you double down on one group and 50 percent you double down on another group. The process of the double downing it can understand is based off of taking the words from those groups and trying to decide on the appropriate action for you and your group to take. Can you describe what that process looks like? How do you take these words and translate them into actions that therefore increase your, um, your very disappointed group percentage.

Rahul Vohra: Okay. So there are two different processes for doubling down for the VD group and the SD group. Which one are you more interested in?

Audience Member: Probably the somewhat disappointed that’s the goal, right? To increase them to the very disappointed?

Rahul Vohra: Okay. So for these somewhat disappointed crowd, the part that I haven’t described is building a world class customer success product management engine. One of the things that most companies actually don’t do a particularly great job of doing is diligently tagging and systematizing all of the feedback you’re receiving on all of these different things, so I skipped the slide, but you can imagine these are all nice things that people are saying. You can imagine exactly the same set of things that aren’t quite so nice and we have individually logged and triaged, I think at this point, 12 or 13,000 different pieces, different sentences in the words of the user related to that red word cloud and these go into our CRM, we happen to use AirTable – I made a very clear early on that we need the actual words of the user. And then we rank the the things that we want to work on by the size of the word in this list and when it’s time to work on calendaring for example, we go into that CRM and I would ask the product manager (in this case actually for calendaring, it was me) to load the hundreds if not thousands of pieces of feedback that we’ve received on that feature into the head all at once, literally just sit down and read everything for hours until it’s all in your head. And distill it down into use cases, jobs to be done. Whatever framework you really like, it doesn’t matter, uh, and write your product spec when you are in that mind frame because if you do it that way, you’ll end up with a much more nuanced solution than what many companies do, which is, oh hey, it seems like we need to build the calendaring thing. Let’s go and talk to 50 people about that calendaring problems and see and see what they feel. Because when you’re having that face to face, you know, people are going to be polite and that they weren’t quite tell you the things that they need. When you have thousands of pieces of feedback in the actual words of the user with the distance of a screen and email, you’ll get real feedback. So that’s how you work on the SD stuff.

Audience Member: Hi, great, great talk. Um, so, um, I’ve got two questions because, uh, you are constantly monitoring and course correcting. So that course correction was happening with new user feedback. So you make the corrections and as the new users are coming in, you’re seeing whether you’re getting slightly better. I just wanted to confirm what that is and if you have an existing product or service that you think that you could apply these things to, to what extent do you think that you could do this in a nontraditional… so you made the case very strongly that this is a particular technique that works for a VC backed high growth consumer (mainly) facing a software product, but if you could see parallels of the principles that you could apply to an existing product or service. Could you talk about the ways in which you might manage how you improve that number in the same way?

Rahul Vohra: Your right, we are tracking this over time and we look at the number of weekly that rolls up into a monthly aggregates and it rolls up into a, a quarterly aggregates. And the reason that it’s important to do the rolls roll ups, at least for high velocity, low price point business like ours, you’re going to get streams of random traffic coming from different sources at different times. So recently, uh, the email spaces and kind of a meltdown. So Newton unfortunately had to shut down because the churn was too high and uh, Astro decided to shut down because they were required by slack and Google’s pulled the plug on Inbox and that will formally be shutting down in Q1 of next year and it’s already haemorrhaging users. And so we’ve just had like this fire hose of not particularly well qualified users or less qualified than they used to be. And so if you don’t do this multiple sort of fractal level of roll up, you’re going to get odd skews at different frequencies. And that’s why it’s important to do that. Um, to your second question of can you apply this in a, you know, in a non high growth venture backed setting, I see no reason why that wouldn’t. I just wouldn’t particularly care about being above or below 40.

Audience Member: So I had a question for you about, where in the customer journey that you start asking the, the, the four questions that you listed and I know that you had said that you ask about three weeks after they start using the product. And I would imagine that when you start asking that question depends a lot on when they get value. So if somebody signs up and they don’t really see the value of your product until 14 days out or 30 days or 45, it should seem like it would push that out. So I’m just curious for context, how quickly do your users see value and like if it’s day one that obviously it seems like it would be about 21 days after they start seeing value. I’m just curious where that line roughly is.

Rahul Vohra: Uh, so very roughly speaking, if you were to ask Sean, he would say about two weeks on average for a average consumer or B2B SaaS product that this was designed for and that the person should have used your product at least twice. The most important thing though is they’ve experienced the core of what makes your product hopefully great. That could be as short as four days. It could be as short as one day, maybe if you have a transactional one-time use business where you’re expecting someone like Uber to come back multiple times over the next few days. For us, we have a relatively complex product in the email space, and it takes time for people to understand what makes it special. And that’s why we wait three weeks.

Find out more about BoS

Get details about our next conference, subscribe to our newsletter, and watch more of the great BoS Talks you hear so much about.

Rahul Vohra

Rahul Vohra is the CEO & Founder of Superhuman, the world’s fastest email experience.

Superhuman is an advanced email application for professionals and power users. Previously, Rahul was the CEO & Co-Founder of Rapportive, a gmail plug-in that displays your contacts’ social media information inside your email application. After selling Rapportive to LinkedIn where he worked for two years, (and even shared some great advice for entrepreneurs who want to thrive in a big company post-acquisition), Rahul has worked with several startups in both investment and advisory capacities.

More From Rahul.