Matt Lerner: How to Focus Teams on the Most Impactful Work

You may have noticed, diligence and hard work do not always lead to success, and doing more, faster is often a bad idea. What is it that causes an organisation to miss the important obvious opportunities while, technically, working hard and doing their jobs really well?

Matt explains how ‘sensible’ incentive structures and widely accepted organisational practices often lead to bad outcomes, and suggests a better approach. He challenges you to rethink the way strategy is developed and communicated; the incentive structures you put in place; and the outcomes your culture celebrates, so your team can achieve the success you seek, even in today’s resource-constrained environment.

Slides

BoSEU23 | Matt Lerner | How to Focus Teams on the Most Impactful Work from Business of Software Conference

Transcript

Matt Lerner 
Good morning. Thank you, Mark. It’s an honour to be on the BoS stage in person. I mean, you’ve seen the other speakers here and past events. So I am thrilled to be here.

I’m going to tell you a story about a fascinating story about funnel conversion optimization. Yeah. Which is actually a story about Stanford University, which is actually a story about my own neuroses, which is actually a story about the misuse of statistics, which is actually a story about an incredible amount of untapped potential. So let’s start with potential. You remember, like, there was this idea meme going around that we only use 10% of our brains. And if we could unlock the other 90% we’d all be like super genius, Elon Musk types, Tony Stark.

Remember that neuroscientists told us that’s actually bunk, and just not at all true. But I do think it is possible that this is true at an organisational level, that we run our companies and our organisations in such a way that 90% of the potential brain power in these teams, goes unutilized. And this is an idea I’ve been playing with more and more lately. So let me tell you how I got there.

My Story
So this started, actually, let me quickly introduce myself. It’s me symbolische. I like blueish pattern shirts. Obviously, Americans spent most of my career in Silicon Valley, two startups that fail that you’ve never heard of, one startup was bought by Macromedia for about 40 million. So then I just quit and travelled for a while it done 40. Most of that 40 million was not mine, I was just ahead of growth, just to be clear, but it was enough to take some time off. I came back and I joined the early growth team at PayPal. And I was there for 11 years in various general management roles, moved to the UK for my sins, I then became a VC. And the neat thing about going from being you know, like T shaped like being a mile deep on PayPal, to being a VC and I probably looked at close to 1000 pitches in four years, invested in 35 companies sat on boards were closely is it gave me this incredible ability to pattern match, like which of these things were idiosyncrasies and quirks of PayPal, in which of these things were actually a pattern. And I’m going to tell you about one pattern, I noticed that I just keep coming back to.

So pretty early when I joined, they were launching card processing API’s. You remember like PayPal one Dotto, they redirect you to a PayPal website to pay. And of course, nobody liked that. So in 2004, they built an AI launched card processing API’s sign up, you pay $30 a month, you integrate these API’s. And you know, we do the offering everything in the background. And the problem was, we launched it and even though it cost $30 a month, almost 90% of the people who signed up for this product never actually processed $1. Not even a single test transaction through the system. So we had these technical meetings and studied the problem. And we found these obvious fixes, like email them links to the documentation, give them a phone number, they can call it they have questions and we fixed all the easy stuff. And then we found a bunch of hard stuff. Sorry, this is easy stuff. Then we found hard stuff like we weren’t integrated with enough e commerce platforms, the sort of progenitors of Shopify are API’s. SDKs only worked with a patchy there were other web servers. Our vetting team was declining, good merchants for bad reasons. So we had a false positive problem on the risk side.

Anyways, we fixed all these. And the number just kept ratcheting up until at some point, we hit a wall. And everything else we tried. None of it worked. And this went on. As you can see, the X axis is a little inadequate on this graph, but it goes on for like five or six months and nothing working. And eventually I did what I probably should have done five or six months earlier, step back, think about the problem. Go and talk to customers. So in a big company, the way you talk to customers is you fly out to the call centre and I sat down next to a customer service rep and I put on a headset and she started calling people I gave her a list of questions who had signed up for the product and not yet activated, just to find out what’s going on. And it didn’t take long I mean I was I booked like to be there for the week, but it’s really about third or fourth call and the lady answered the phone. ‘Yeah, Pay Pal. Great. Hi.’ ‘Oh, yeah, I love PayPal.’ ‘Yeah, it’s up on my website.’ ‘Yeah.’ And we look and it’s on our website. And we said, ‘well, how come? How come we haven’t had any transactions yet?’ And she said, ‘Oh, we haven’t had any customers yet.’ So okay, ‘well, how long has your website been live?’ And she said, ‘four months’. And I realised, this lady doesn’t have a business. I think, people who buy like how to start an E commerce business for dummies and never actually even buy a domain name, right? Like, I realised all of our customers like, are not created equal. A lot of them probably don’t have a business. And we can’t market our way out of that, right? We’re just gonna have to treat that as not addressable.

So I got back to my hotel room that night, and I emailed Igor. Igor is the data analyst on my team. Now, he actually wasn’t on my team. He’s within Pay Pal. He’s a genius. And I said, Igor, you know, because I dislike you guys know, like, the 80/20 rule. And the purrito distribution. There we go. Were like 20% of our customers gives you 80% of your revenue. So I thought maybe something like that was going up. So I emailed Igor, and I said, ‘Igor, what percentage of our revenue comes from our top 10% most valuable customers?’ And he replied, right away, he must have been working late. And he said, ‘102%’. I have had somewhere I have these prizes. These notebooks for anyone who can tell me how this is possible. Because Igor has an advanced degree in Aeronautical Engineering from Stanford, he did not do a this is not a math error. How is this possible?

Audience Member 
That costing you money?

Matt Lerner 
It’s not cancellations.

Audience Member 
Worst customers costing you money?

Matt Lerner 
That’s right, you get a notebook. Congratulations, America. So our worst customers, the worst sign ups were fraudsters. And they would pull about 4% of the value out of the system. So the net distribution of value in the system was plus 102. Negative actually, it was like it netted out, it was like 98 to minus four at the bottom. So we didn’t need to activate 40% of our customers, we needed to activate the right 10%. So feeling, so I emailed Igor explaining all this to him. He said no problem, I’m going to build a predictive model. Whenever a new business signs up for a PayPal account, we will score them on their propensity to be a high value customer. And that wasn’t even very hard, like, you know, is it a Hotmail address? Or did they have their own domain name? Is it? Is it Mike at blah, blah, blah, cars.com? Or is it sales at blah, blah, blah. So he built a simple propensity model, and scored everybody.

And then we started calling the top 10% customers. And someone on my team and Laurie developed an outreach plan, just, you know, triage call, are they on the right product? Do they have the tools they need to integrate, get them through all the risk and compliance hurdles and just get them off and running and give them like a phone number, they can call if they have trouble. So we started doing that. And we ran it for six months. And after six months, we looked at the data. And these numbers are, by the way, are all fictitious but directionally accurate. And the people we called were worth about 10x, more than the average customer. Got another notebook for someone who could tell me Is this good?

Audience Member 
No.

Matt Lerner 
Who said no? Why is this not good?

Audience Member 
Because you’ve got a huge raft of customers that are not taking advantage of it. You’ve just proved you’ve got a small group, who by giving them the love of performing, so you’re not doing your job, right? Because you need to be applying that to lots more.

Matt Lerner 
So Igor’smodel might have been wrong, we might have been missing other high value customers, and we should have been calling on to, we couldn’t afford to call everybody.

Audience Member 
Yeah. You’re not pulling out. All you’re doing is proving that by activating these customers that they succeed. It’s not telling you which one was which other ones might have succeeded.

Matt Lerner 
It’s not even telling us that calling them cause them to succeed. That’s the problem. Anyone else?

Audience Member 
For example. Cool. All right.

Matt Lerner 
Give it, give them the process cards. You get process codes now. All right. So that’s the problem, right? Equals model was gonna find high value customers. Maybe this just tells us ego’s model found high value customers and calling them made no difference at all. So in fact, what we did is we called half the customers chosen at random And we ignored the other half. And the ones we called ended up being worth 40% More than the ones we didn’t call. Now in a company that is signing up 10,000 merchants a week, and processing over a billion dollars, the one processing over 100 billion a year, earning over a billion a year in revenue, increasing new signup cohorts by 40% is an astoundingly big change, like we were trying to get, you know, 1% here, 2% there. So this was just an absolutely massive outcome for us.

Funnel Optimization
So, this takes me this is the funnel optimization part of the story. And this brings us to Stanford. So in my neuroses, so I should tell you, I’ve always kind of struggled with this imposter syndrome, I’m sure a lot of you can identify with that. And then getting to PayPal really didn’t help because PayPal was, you know, a Stanford shop. And in the early days, they really just had their pick of who they would hire. And so I was on a team with two other people. One of them, went to Stanford undergrad and Harvard Business School, the other one went to Harvard undergrad and Stanford Business School. I went to a school called Miami University, which for reasons I will not go into is in Ohio, and it’s a lovely institution. But, but it’s not Harvard. And it’s not Stanford is not even the same planet. But a few years later, I managed I started getting invited to guest lecturer at Stanford, they wanted, you know, Pay Pal people and the students liked my talk. So they kept inviting me back. So I was up in Palo Alto giving this exact same talk only with different formatting the slides. And I mean, I’ve now lecturer at Imperial at Berkeley at a few different places, and Stanford Business School students have a level of confidence that is unmatched, even by their own abilities.

And I love the Q&A, because they really ask hard questions, and they really challenged me. And that would push my thinking, but I always still, in the back of my mind thought, I’m gonna get busted one of these days are gonna ask the question, and everyone’s gonna realise the emperor has no clubs. And that did happen.

The Question That Really Got Me Thinking
This lady put her hand up. And she said, ‘Matt, what you’re talking about is concentration of revenue and your customer base. This is not crazy. Like every airline knows this. Every online gaming site knows this. How did it take you guys two years to figure out your revenue concentration in your customer base.?’ So I didn’t show it. I mumbled something about in the fog of war, blah, blah, blah. But everything’s obvious in hindsight, and finished the talk and it went fine. And they even invited me back. But I carried the question with me, because it’s a great question, right? Like, how do we miss this? And I was talking about it. Does anyone here know most cited? Couple of you, maybe. So I was talking about my friend Moe, and he’s a genius. And Sally, we should talk about that later. And he said, ‘No, Matt, he said, You’re not the imposter. You’re not the idiot in the whole company full of Stanford people. You’re the one who figured this out.’ I was like, ‘Oh, great, thank you.’ And then he said, but not ‘she did ask exactly the right question. How can you take a company chock full of, frankly, probably the greatest tech business talent of our generation, and they not see something that is in hindsight, as obvious as the nose on your face?’ I thought, ‘that is a really good question.’

So I just started going back through it in my head and puzzling with this. Why did I not see this sooner? What would happen is every quarter OKRs, my boss who would sit down with me and he’d say ‘Matt, what’s the activation rate?’ ‘What’s the the activation rate target?’ So this is back to the untapped potential piece of the story. ‘What’s your activation rate target?’ I’d say ‘33% 34%.’ And he’d say it was 34% last quarter, what we need you to raise the target. And like I’m not hitting the targets and like, and I remember thinking something about this doesn’t feel right. I couldn’t figure out what it was. But something didn’t like, why are we trying to activate all these merchants? And I tried to say that to him, and he just say, Matt, you’re sandbagging. I need a target. We got to make this number go up. I thought, Okay. Why is my boss pushing me? Why was my boss pushing me on this target? I’m willing to indulge a conversation. And I realised it’s because his boss was pushing him, the head of, you know, Pay Pal, North America or whatever she was, was pushing him. And I thought, why is she pushing him? I thought, because she probably has a spreadsheet from the finance team that looks something like this. Obviously, I made this up. But I’m sure the formula for calculating our revenue forecast was something like number of activated merchants times the average revenue per merchant, because that makes sense for a model.

But the problem is, in this case, average was stupid, right? It’s a nice idea, but it just didn’t make sense because if you’ve got a normal distribution of value than your average is an extremely helpful, meaningful number. But if you have a highly skewed distribution, then what you’ve actually got is a small number of merchants who are worth a tonne of money. And then a shit tonne of merchants who are worth, fuck, all right. And your average merchant is is vanishingly small number of merchants who happen to be worth this arbitrary little amount that looks nothing like your customer base. So that’s the misuse of statistics part of the story.

But then the question is, Okay, fair enough. But how did this error propagate? Again, why did a company full of, frankly, some of the greatest business minds of our generation, not spot this? And this takes us back to Stanford, and every great uni right, so these were people who got into Stanford, and then they graduated, actually my boss’s boss, after she came out of Harvard Business School, she started a company and she IPO that company. And then after she exited that business, she came in ran a division of paper, like these are not dumb people, these are the highest of high achievers. And the way they got there is by doing all the things and doing them better than anyone else. Like if I said, How do you get into Stanford, and we couldn’t all have gotten into Stanford. But we all know how to get into Stanford, right?

You get perfect grades, you do perfect on your aptitude and standardised tests, you choose some extracurricular activities, and you’re one of the best people in the world at them. And then it helps if you had like a personal tragedy in your life, where you come from some kind of interesting background, and then you might have a chance to get into Stanford, it’s not a secret is not a mystery at all. But it means do knowing all the things and doing them better than anyone else. And then you graduate from Stanford, you get a job at Goldman Sachs, or McKinsey or Silicon Valley Bank. Just kidding. And again, you get promoted and rewarded by doing all the things whatever you’re in finance, you’re in marketing as a list of things you do to do marketing or to be a good Bain consulting, Engagement Manager, you do them all, you do them better anyone else.

In short, this is a process of optimization. Sorry, optimization. And this looks a lot like, you know, any mature business, any mature sport, where like everyone knows all the things you have to do when Team GB won the Tour de France, what was your secret, they famously said, we did 100 things each 1% Better, every little detail, you know, there’s legal limits about what you can and can’t have on a bike or in your bloodstream. And they got as close to the limits as I could and they push and optimise every tiny little thing. Optimization is the way you win. In an established mature business, where you’re trying to do something, if you’re another car dealership, then you play the optimization game to outsell the other cars. But if you’re doing something completely new, like selling cars on the internet, then that’s a different process. Because you have a lot of uncertainty.

So in a mature business, in a mature sport in a, you know, mature domain, this is stuff you know coming in, they hired you because you’re smart, you know most of the things you need to know. And when you hit a wall, and there’s something you don’t know, you go find the person who knows in your company, you hire consultants, you go to the Business of Software conference, and you learn a bunch of great frameworks. But the knowledge is all out there. And it’s just a question of asking enough questions and finding it. But if you’re doing something no one’s ever done before. If you’re building something kind of from the ground up, then the stuff the organisation knows, is only still a small piece of the puzzle. The bigger piece of the puzzle is all this stuff you need to figure out. So your success is much less a game of optimization, and much more a game of discovery. You need to turn your organisation from an organisation of optimization into an instrument of discovery. Now, the reason this is a problem is because most people don’t realise what they don’t know. So Epictetus famously said, it’s impossible for a man to learn what he thinks he already knows, our brains are blocked, because we make those spreadsheets and we assume the spreadsheet model is the business, we assume the map represents the territory. And we’re not even aware of the things we don’t know. So the first thing you have to do, to move from this sort of an instrument of optimization to an instrument of discovery is simply acknowledging that you don’t have all the answers. And that’s like, a huge step. That’s absolutely the hardest part.

Now, how do you do the rest of this transformation from being an instrument of dis optimization to an instrument of discovery? So I go back and look at what are all the things I did in this story, from optimization to discovery and an instrument of discovery. I mean, an organisation that is only limited by the laws of physics and the laws of the land in which you’re operating and you have millions of degrees of freedom of places you can take it so you don’t have to do 100 tiny things. You just have have to find the one or two big things. Because if you look at any successful startup and look backwards, you’ll see that 90% of their growth came from, like 10% of the stuff they did at PayPal, it was, it was just down to about five things that drove all of our growth. I mean, early getting on eBay, getting pre integrated with all the shopping carts and hosts having this simple copy and paste HTML checkout. So just you know, a lot of developer outreach in the beginning was just a few things. We did lots of other stuff, we had all this money, we had to reinvest it in growing the business, we changed the logo twice, we bought all these ads that never made any difference. We built tonnes of products that nobody ever used, all this stuff that if we’re honest with ourselves didn’t make a difference. Good news, if you’re a startup, you’ve got to find the 10% of stuff, that’s actually going to make a difference. And that is a process of discovery.

So how do you actually do discovery? So I went back, and I thought about my experience at PayPal. So what did I do, things weren’t working, I said, ‘Wait a second, we need to think about this problem.’ And then I went, and I listened to customers. And then I came up with an idea. And then we played with data. And I got some support for that idea. And then we literally designed and ran a controlled experiment. So I was literally doing science, you have to do science. Now maybe it’s simple. Maybe you just hire a bunch of scientists into your company and have them work on the confusing bits. But I’ve seen people tried to do that. And that doesn’t work, either. Because the ethos of the organisation will overpower the ethos, the mindset of any individual. Or to put it in very simple terms. If you hire a bunch of scientists onto a baseball team, they will not do science. They will play baseball, they’ll probably play terribly. But they won’t do science, because it’s a baseball team. So you have to, it helps out some scientists, we have to shift the entire ethos of the organisation. And this is not easy. So how do you shift the ethos of the organisation? Well, as it happens, my partner, my co founder, Nopadon and I run a business that does exactly this, we work with about 60 startups a year. And we help them shift from optimization, to discovery and find their big growth levers find like the few things that are going to have the biggest impact. And so we’ve had a lot of time to watch and study this and think like, what’s the difference between the winners and the losers, the people who are successful and the people who aren’t. And what we do, it’s not magic is no dark art, like this is all of it right here. So the first step is the measurement piece is we map out that we work with them to map out their growth model map out their business in terms of user flows and conversion rates. And if any of you have a background in systems theory, this is it. It says input process, output buffers, constraints, feedback loops, find those. And when you understand the data, you’ll look for feedback loops, leverage, and you’ll find a rate limiting step.

I’m sure a lot of you are familiar with the theory of constraints. It doesn’t just apply to factories, same thing can happen in an online business, there’s a bottleneck in there somewhere. And if you apply resources to the bottleneck, it makes the entire business grow faster. If you apply resources somewhere else in the system, it’s less efficient, it’s or even entirely wasteful.

So the first step is just understanding what is your rate limiting step. Second thing we do is discover it’s jobs to be done interviews, literally, we just teach them train them. Thank you, Bob, how to do jobs, interviews, send them out in the field, they start interviewing recent signups new customers, mapping out the four forces, alternative solutions. And if you do that with a sense of what is your rate limiting step, you just magically start getting all these ideas. So then the next thing you do is generate hypotheses, we teach them to do science, generate hypotheses, make predictions, run experiments. Now, what’s the difference between running an experiment and like, hey, let’s just go do it. JFDI, I have an idea, let’s do it. The difference is only about five minutes. The difference is the five minutes it takes to write a well formed hypothesis, share it with your team and make a prediction. But that is such a valuable step because writing a hypothesis really clarifies your thinking, I believe this, and therefore I predict if I do this, then this number will move in this direction by this much. And just even saying that often you realise wait, actually, this isn’t the best way to move that number or like you come up with something else. And then making a public prediction, eliminates hindsight bias, because everything’s obvious in hindsight. And this shows us here’s what I was thinking before I did this experiment.

Therefore, since that didn’t happen, I need to go back and re examine my thinking, because that’s the goal of experimentation is to get a little smarter is to do discovery. So that hypothesis, writing step is critical. Anyway, so we’ve worked with a bunch of companies to do this. And to be honest, like teaching them and doing all of this, you could do this and what do you think three weeks, four weeks? It’s not that hard. But the programme is 10 weeks long and we I have to make the programme shorter because then we can do more companies in a year and we can make more money. But we can never get it shorter than 10 weeks. Because of that middle bit, the mindset and the leadership habits. It’s one thing to follow the process like a cargo cult, and take all the steps. But what we’re trying to do here is drive a mindset shift to people who realise when they’ve reached the limits of their knowledge, people are comfortable saying, I don’t know, let’s try this. Oops, that didn’t work. I’m glad it didn’t kill the company, what can we learn? How can we be smarter next time. And if you’ve hired a bunch of brilliant people who went to great schools and had great jobs before, you’ve hired a bunch of optimizers, and they’re not comfortable talking about the things they don’t know, they’re not comfortable saying, Here’s a list of 100 things we should probably do. But I’m going to do six of them. And I’m not going to do the other 94. And that’s not going to be great. They’re not comfortable doing that. They’re not comfortable saying, hey, I really believed in this thing, and I tried it, and it was a disaster. So just getting people to that comfort level. So how do you do that? Well, the first one we talked about, it’s Epictetus, right, it’s just being able to acknowledge this is not an optimization game. This is a discovery game, this is a search for some big levers that are going to make a huge impact on the business. And those are not going to be obvious. And they’re going to be in our customers heads. And they’re going to be in our data. And we can make some theories and test them. The second thing, super powerful. Anyone want to guess? How do you change the fastest way to change the approach and mindset of an entire organisation? One word, trust, it’s a good word, but the stuff that I’m looking for trust is necessary, just say nudge, close push. Sorry, emergency, you could declare an emergency it’s happened. Incentives, you got it. There’s your notebook.

Change the incentives of the organisation. So how do you do that, and still keep your incentives aligned with business outcomes. So let’s work backwards, right? We talked about Bill talked about sales teams, very simple creature, you have a revenue target. If you hit your revenue target, you get money. That’s great if you’ve got a mature product, and we all know how to sell it, and it’s just a question of following the steps and doing the things. Revenue is perfect. But if you take an early stage startup that is or a piece of a new piece of the business summary, I acknowledge and say go make money. Brilliant, people will find 100 different ways to make money, your salespeople are going to sell all these custom services that are going to tie up your engineers, your product marketers are going to like raise the price and do bundling strategies that piss your customers off, but increase the take, right? Like there’s a million ways to make money that are not going to lead to breakout growth. So then the next thing to think about is, okay, ultimately, we’re going to make money by delivering value to customers. So let’s figure out how to increase value delivered to customers. And I’m sure that your best OKRs have at the root, some sort of customer engagement, customer outcome metrics that are going to hopefully lead to money, if you can figure out how to delight a lot of customers. monetizing that, while not trivial, is relatively easy thing. But early on, when you’re not sure who your customer is, why they’re gonna buy your product, where are you gonna find them? What is their desired outcome? What are they struggling with, you have this in between step, which is learning.

See, ultimately, you want to incentivize learning. And your success in a domain of high uncertainty in a discovery challenge, is going to be mediated by the pace and quality of your learning. So that’s the shift in incentives. But still, you can tell people, you know, I want you to be running three experiments a week, I want you to tell me what you learned, then they’re going to choose these tiny little incremental experiments, because they don’t want to screw up. They don’t want to get in trouble when the CEO sees him in the elevator and says, What’s your goals? They didn’t want to say? Or how are you doing on your goals, they don’t want to say, Well, I’ve tried 26 things in the last few months, and none of them worked. So the next thing is then getting people comfortable, because most of your experiments should fail. Because if every experiment you try is working, you’re not pushing, you’re not being bold enough, you’re not taking risks, and you’re not learning. So how do you get people to get comfortable taking a punt and making a mistake? So this is the hardest part. And as a leader, you need to model the behaviour. And I’ve spent a lot of time thinking about these adjectives. And I think this is it. And very much you know, these are opposition’s. So you’re not just you have to be curious. And part of a big part of curiosity is saying, I don’t know, and saying having humility to say I thought this and I tried this and it didn’t work. And you know, every entrepreneur has to bring or fake confidence because that’s how you raise money. That’s how you hire people and get them to follow you and get them with their pension plan over to your scheme, and whenever you get them to depend base our car payment on your business’s success, you have to have that confidence. But you’ll also have to be able to suppress that confidence to say, I don’t know all the answers here, I need you to figure that out. So part of it is being able to just replace that confidence with talking openly about your own mistakes and about the things you don’t know. Because that creates space for people who have never publicly talked about a mistake to their boss before, to be willing to do that.

Now I’ve crossed out diligent, I bet most people want to hire diligent employees, I bet diligent sounds like a good thing. But in your early stage, startup, diligent people will carefully go through a list of all the things that could potentially go wrong. And then they’ll engineer the campaign or the product or whatever sales and marketing that they’re about to do. They’ll spend weeks fixing it, so there’s no typos. And you know, everything’s properly done, so that they don’t have any of those mistakes. The problem is, your experiments probably going to fail. And it’s going to fail for a reason that’s not actually on that list. So better not waste the time trying to think of all the things that could go wrong. Better to just chuck it over the wall, get it in front of customers, and spend your time figuring out what did go wrong, because that’s the point at which you’re learning. So diligence, great in BAU business, as usual, great, an established piece of the business, but at the edge of knowledge in an area like growth in a high uncertainty domain. Diligence slows us down. Another thing that slows us down is strategy is theory. I work with very intelligent teams, top marks, former consultants, and they’ll spend weeks debating, I wonder if our customers want this or our customers want that. And I lose my patience about 10 minutes and I say, Guys, you’re debating a topic. That is not an opinion question. There is an answer to this question. It is not in the room. If you got one week to go out somehow interviews, experiment, something, validate this bottom out the answer to this question, how are you going to do it?

So teams that get stuck in this overthinking loop, those are the least successful companies of all the ones I work with the last pieces industrious already talked about in any successful startup 90% of their growth came from 10% of the stuff they did. So you need to find people who don’t try to do all the things. Some people have a very strong bias for action, I admire that I’m not one of those people, but they’re very important. And they’re gonna just go do everything. And they don’t necessarily think in terms of the opportunity costs. If I spend my time doing things that are fives and sixes and sevens in terms of impact, that comes at the expense of doing the 10s. So really challenging people. And just saying like, so as a leader, the way to model this behaviour, and Sally talks about this is asking people, which work should we do? And then don’t override them don’t say, Well, I don’t think we should do that. But just push them and spend time tested our math and make sure you know, ask them if this works, how big can it be? Run me through the numbers. If this works, how big can it be? That’s going to help you find the big stuff. So to quickly summarise, if you’re in a domain of high uncertainty, either a totally new business or a piece of the business that isn’t working isn’t growing the way you want, you need to change from an instrument of optimization to an instrument of discovery. The hardest, the simple part of that is doing jobs, interviews and running experiments. The hard part is mindset change three pieces of that first, just acknowledging you don’t have the answers. And that it is a search is huge, then thing to establishing the right incentive structure to accelerate the pace and quality of learning. And then number three, is to model the behaviour. That’s the talk. I don’t know Is anyone here, this is the tiny little sales pitch is anybody here subscribed already to my email newsletter? Okay, for those of you who aren’t, it’s two minutes a week, I’m trying to write the most engaging newsletter in startup growth. And you’re welcome to join the list if you want.

Now, what I’m going to do is I’m going to invite Nopadon, my co founder up here, and we’re happy to take any questions that people have about this. And I just want to also introduce Nopadon because he’s giving a talk tomorrow morning at 11:30 and he’s going to be doing Website Teardowns. So and he’s going to be happy to take victims from the audience and he’s going to do it through a jobs-to-be-done lens. So welcome.

Mark Littlewood 
Just for the books now give it to him because actually he gave the right answer right just like

Matt Lerner 
Are there any questions?

Audience Member 
Hi, where do you sort of draw the line? I guess it on two axes one is that been decided serving and not to strategic theoretical versus being judicious where you do need to kind of work out the trade off and think about where you’re spending your time? And then the second question is, when do you start moving towards optimization that there are those five things that really makes a difference? But then at some point, it’s how do you know when do you start optimising on those five decisions versus trying to look for the next big thing?

Matt Lerner 
Okay, so in terms of the first one, I find when I work with companies, there’s usually a deeply ingrained culture that is overthinking or under thinking. If you find yourself having the same conversations again and again, week after week, you’re probably overthinking it. Pencils down. If you find that you’re doing lots and lots of things all the time and none of them seem to be really working. Then maybe take a step back and think harder about just it’s, you know, again, a quick math exercise back of the envelope. If this works, how big can it be? Now your second question, remind me.

Audience Member 
When do you start optimising?

Matt Lerner 
Optimise versus so if it’s an early stage startup, and nothing’s particularly working, it’s squarely a game of discovery. If well, here’s what I’d say. If you’re not sure. Your discovery. When something starts working in a high growth business, it starts working like you know, what is Who is it that says, you have product market fit when people use your product, even when it breaks a lot? Gil, I think said that, like when he’s working, you know, but I think there’s another level to that question, which is, for each department in your company, how do you know if they need to be doing discovery or optimization? Should every customer service rep who picks up the phone be optimising that process?

And the answer is no, but some of them should. Because in a successful startup, growth almost never really comes from marketing. It’s it’s a team sport, we say, if you look at like Spotify, it wasn’t that hard to convince people that streaming music was better than like buying DVDs or buying paying 99 cents for iTunes. The hard part was getting into America, and getting past the record industry. So the lawyers and the lobbyists were the growth hackers, transfer wise, had a compliance team, they had to KYC everybody just like everybody else. But they made that a quick and painless process, which helped them crush all the banks. So in their case, their compliance team were the growth hackers, they still had to sort of Bau the compliance checks. But they were doing it with an eye towards the Northstar metric, they were doing it with an eye towards how can we onboard as many customers as possible. So even in a mature established part of the business, you know, your rate limiting step will tell you is this somehow bottlenecking our growth? And if so, you know, you need to have some people in there who have an optimization mindset and can experiment with it. Great questions.

Thank you. Did you have anything to say. Hi, Ryan.

Audience Member 
How do I ask this? The optimization kind of mindset is a kind of default that people very often kind of come into some work activity with this in their mindset, right? And it’s like something usually has to happen to shock us out of that, or wake us up from that. And so how do you kick people out of of the optimization thinking to actually stare into the unknown, and start reaching for all those totally different kinds of tools?

Matt Lerner 
Do you want, I can take it if you want. So we force it, when people come into our 10 week programme, we very quickly present data and ask them questions that they obviously don’t know the answers to, we have to make predictions, those predictions are wrong. We just confront them with the gaps in their knowledge. We structured in that way. And then we just, it’s hard, because then suddenly, it’s very scary, you know, and their bosses on the same zoom call. And then so we spent a lot of time just working with everyone talking about our own mistakes, to make sure like this is a safe space. This is a feature, not a bug. This is step one, how do we go from here?

Audience Member 
There’s a big social element to it. There’s a there’s a social element of not wanting to be wrong or say the wrong thing. And like the boss is there. That’s really interesting. Exactly.

Nopadon Wongpakdee 
That the culture, sometimes it’s cultural issue, right? And so we have a team now based in Germany, and they’re, they’re comfortable space is complexity. They’re smart. They love to show that they’re smart, they love to show their teammates that they’re smart, and they end up overcomplicating everything. Right? And so, what you know It’s like, in the cohort, like someone will ask a question, I don’t even have to know who it is. I’m like, You’re from that company, aren’t you? And so I said, you don’t have a problem with this experiment, you have a cultural problem where you love to make things complicated, for whatever reason, right? And now they’re beginning to sort of unwind that and, you know, not show off, I guess, is what is what’s happening. So, deeps deep rooted cultural issues within the company, as well.

Matt Lerner 
Oh, and one last trick. We only work with 60 companies a year, and Nopadon interviews every applicant. And the main thing he’s looking for in the interviewer is defensiveness. And we only choose to work with companies where we think that they’re really open to learning.

Audience Member 
Well, so So my question, or my, the thing I struggle with, with most teams, is getting them to frame the unknowns? How do you have a Do you have a way in which to kind of how do you get the unknowns on the table? How do you actually prioritise the unknowns of what to solve? Because at some point, once they see that they don’t know, it’s like, holy shit, you know, it’s just a big, it’s a lot of them. So how do you? How do you get teams to actually kind of start to prioritise those unknowns to ask and what to go after? I know that Mike,

Mark Littlewood 
Can I just interrupt before you answer if anyone else got questions kind of wave at me, because one of the characteristics of high functioning Highly successful people is that they will ask questions all the time. I can see that just bouncing between Bob and Ryan.

Audience Member 
Five, and he’s had another two. So we’re. sorry.

Nopadon Wongpakdee 
So Bob, so like, for instance, someone, any person can come to you and say, Oh, your SEO is a mess. Yeah. Right. And so you’ve done this long enough, you know, that spaces that are important, ask the question that, you know, they don’t know the answer to. Right. So for instance, Oh, that’s interesting that like your nurture stream, is that because you’re tackling this type of inertia? Right? And they’re like, Sure.

Audience Member 
So it’s contrast, you’re using contrast to say, this or this, and then actually doing that helps them actually uncover that unknown even more deeper.

Nopadon Wongpakdee 
Exactly.

Audience Member 
It’s unpacking,

Matt Lerner 
We have a much simpler business, because we’re working with companies who have a product that people like, and are struggling to grow. So the fact that they have product market fit with the customers, they found already usually narrows the scope of the problem. And then we map out their growth model and say, Okay, well, your rate limiting step, you’re getting lots of signups and no one’s starting to use the product, or no one knows they’re looking for your product, so you’re not getting any traffic, and we’ve even narrowed the step that much. So we can then there’s kind of a pretty standard set of questions within there. Like, you know, what are your prospects struggling with? Now? Where are they looking for solutions? You know, what, you know, these questions better than anyone in this room? What alternatives are they considering? And so, you know, we just figure out which of those you know, once we ask them all those questions, they don’t know the answers to them. That narrows it down pretty quickly. Got it? Thank you.

Mark Littlewood 
Can I borrow your? Max, sorry?

Audience Member 
Really entitled, oh, thank you. I really love the adjectives around the mindset that you brought up. But coming back kind of to this, I wonder, do you see is a use? Could there be a cycle between optimization and discovery? And does that correspond to certain stages of growth or company maturity? And once you have put the team into the mindset of discovery, what happens to them? When the company goes into a mode of optimization? Do they, you know, need to be relocated? Replace? Can they be replaced? Do they need to be replaced? What happens then?

Matt Lerner 
You’re a little bit beyond where I’ve taken this. But what I what I can reliably say is, I have not yet seen a team who can just switch back and forth between optimization and discovery mode. Getting out of optimization mode into discovery mode is quite difficult. But when you have a growth team in a company that is running in optimization mode, it’s actually fairly straightforward to just point them at a different part of the business. So Milan Harris, who’s the head of growth at TransferWise, he told me a story about you know, he got a call from the head of compliance, saying, Can you turn off the ads? We can’t onboard KYC customers fast enough. And he’s like, Well, that’s horrible. So instead, he turned off the ads. And then he had his growth team, go and help the compliance team, create welcome sequences, automate some of their onboarding, build models, and use their their sort of discovery approach to improve the throughput of the onboarding problem. So I haven’t thought this through exhaustively but to me, that’s Seems like the model. And then as a senior leader, just having enough understanding over the business to understand where your growth is limited by your knowledge and your understanding and your approach, and then moving the tools moving the right people over there.

Mark Littlewood 
Thank you, Sharon. Nothing because MIT next.

Audience Member 
Thank you, Matt, what a great talk. Your every message resonates me with me so much. Halfway through, I was so excited. I’m thinking oh, we we got to MIT you to help us there knock down says you choose who you want to work with you pass your test. So I work for a large organisation, the University Press and assessment, we do to feel like 2 million things you want to go and even we probably don’t have time to think. So in this kind of situation. Is there any possibility you can focus like sort of one area? Where do you start? So if we do want you to help us and we do want to pass a test, we sort of focus on one area instead of the whole organisation? It’s just, you know, some thoughts on that.

Matt Lerner 
So as your question like, could we are how we engage with you? Or is it we’re overwhelmed with the scope of this problem? How do we even get to, are we doing a lot?

Audience Member 
We try to identify a few, you know, new cash cows, for example, identify that 10%, we are doing 100% at the moment, but helping us to identify that person, I thought that’s your business. Yeah.

Matt Lerner 
So the way you would approach that with a company is the first thing we would do is map out the growth model, just literally, it starts with, Okay, we have these different prospective customers, this person will wake up in the morning and think, oh, no, I need to figure out a way to do X, and they’ll go here, and eventually they’ll come in contact with you. And their first experience would be this. And their second experience will be this. And at this point, they’re a marketing qualified lead. And at this point, they’re our customer. At this point, they turn and just literally map that out those journeys, and put numbers next to them. And often when you do that, between us like we’ve seen enough benchmarks we know but often when you do that, you can just tell for yourself. This doesn’t make sense. The only reason anyone would ever come to our weird little website is if they wanted the thing we’re selling because, you know, we have no SEO? Like, you’ve got to hear about it. Why are only 2% of people signing up? Okay, well, that’s clearly the bottleneck, you know, our website, it’s the bottleneck.

So or whatever, lots of people are signing up. Lots of them are using the free trial, none of them are paying why that makes no sense why. So you sort of start by just like, where’s the rate limiting step? And then you sort of work backwards from there. And most people, the mistake then is to brainstorm, we have an activation problem. Let’s send nudges, let’s send push notifications, let’s send reminders. Let’s remove fields from the forum nice. People start brainstorming a list of generic tactics at that point, instead, because doo doo doo optimise, instead of saying, why aren’t they? And then interviewing people jobs to be done? who just got past that point? And talk about the experience? We don’t even talk about the product, we just talk about what got you here? And tell me about that. And that will usually give us insights into why it’s breaking down for other people. Sure.

Audience Member 
Hey, Matt. So we did your course last year, and you were brutal in all the best ways. So thank you. No, thank you. We’re still picking up the pieces. So something I’d like to go back to is you were saying earlier with the Stanford thing, and the problem that actually you’d had all these smart people, and they essentially stopped being smart. So I wonder to what extent you thought your job is more to be the consultants who come in and tell us what we already know. And get us to do what we should have been doing on the lawn, as opposed to the people who come in and say, Hey, this is a new idea. So I just curious to know how you feel on that sort of scale. And whether most companies are fundamentally know what they should be doing, but need a good kick up the backside to do it. And that’s your job or whether it’s about we’re all doing the wrong things.

Mark Littlewood 
I literally got no mic. They got no right to reply here. This is good.

Nopadon Wongpakdee 
So I worked with his team. And he has a very smart team. And a team that loves to build, build, build, build, like default behaviour is like let’s build something. And so my big anxiety sort of not criticism, but an anxiety that I had with the team when I was leaving, is we were doing you know, Matt was talking about writing a hypothesis putting it in a backlog scoring the backlog. And his team had created this most amazing system for putting an idea in prior are typing the idea and it would shoot off and put things into Slack and everybody could see and this amazing system. But then when I would look at the ideas in the system, there were fives and sixes and sevens at best. And so there’s like a Buddhist analogy that was used on me once I was believing, and I’m like a Buddhist snob, which is like, how does that happen? And I said to this month, once I was like, you know, these guys, they do slightly different prayers, and isn’t that strange, and he said, You know what you’ve done. He’s like, you’ve taken the most utmost care to peel a banana did not leave any skin on the banana. And then what you’ve done is you’ve eaten appeal. And I said that tomorrow, I said, you have this beautiful system of ABA, but you’re not paying attention to the idea. In this system, you’re eating the peel. And so was, you know, so it’s like, keeping companies who have a tendency and bias to keep them in check, basically. And so part of the question was, like, tell you things that you already know. So I think you guys love the bill. But I think you need to push with the big, big, scary, hairy ideas that you’re that you’re afraid of.

Audience Member 
The other brilliant thing you did was you told my team, many of whom I’ve employed as a first time, as you said, Mark’s a bit of a shit umbrella. So he’s protecting you from all an awful lot of ship, and you’re in a really good deal this time. So you need to be a bit less complacent and like, get on with some work, or you have to go work for a really ship company. There’s a lot of them out there. So thank you for telling them that they’ve been working really hard.

Mark Littlewood 
You got time, one more, Bob? Yes. Okay. Two more Bob.

Matt Lerner 
But if I can just give one less, an allegorical and more mundane answer to your question. 10% of the job is figuring out the things you need to do 90% of the job is shifting the mindset of the organisation to operate in that new way.

Audience Member 
So as I look at that list, right, what’s very interesting to me is I feel like, I’m on the Discover side, but my job is to be curious to create confidence. It’s to be impatient to create diligence. It’s very empirical to build the theory. And so to me, the thing is, as a really good builder, I actually have to have both modes. But I can’t start with optimise I have to start with Discover. So that thought of just kind of like how these relate because one, confidence doesn’t cause you to be curious, but curious is cause you be confident. And so it’s the causation that you have to follow. Thoughts. Can I use that? Thank you.

Mark Littlewood 
Any thoughts?

Matt Lerner 
Oh, no, it’s yes. I’m gonna take that completely. It’s, you’re absolutely right. It’s brilliant.

Mark Littlewood 
Final, what then got? The problem with a tech conference, is that you got microphone and make it work. How do I break it? It’s it’s switched on, if you were normal people wouldn’t do this.

Audience Member
Thank you for a great, great talk. My question is around maybe a pattern recognition that you might have done by working with, you know, a cohort of 60 companies each time, is there a time during which the companies switch from that exploration mode to exploitation? Maybe in terms of stage or maturity, or maybe the industry they’re operating in? So that you’re seeing Aha, maybe now they’re a series B Company, they stopped, you know, exploring?

Matt Lerner 
Yeah, absolutely. With luck, companies will hit those thresholds. It’s usually pretty obvious when something starts working, then you’re doubling down on it, but it’s still it’s constant optimization, you know, the things that we’re bringing, you know, whatever, Facebook signups like crazy in 2010, they’re not doing those things anymore. So this becomes an element of Bau and investing. But you still get always be kind of innovating at the edge edges of this. So one of the things I love about BoS is that all the speakers everyone sits around and talks to each other. So I wanted to also introduce marching xaba, who’s a coach on our team and Nopadon and I were all going to be around for the next few days. So if people have other questions? Want to chat? Want to get specific? We’re just happy to thank you again.

---

---

Next Events