The science of lean startups | Eric Ries | BoS USA 2010

Eric Ries had a busy weekend as he launches his book, The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. If you are reading this blog and you haven’t bought it already something is very, very wrong.

This is Eric’s talk at the Business of Software 2010. This video is not a substitute for reading the book which is brilliant, not just for the content it contains, but for the incredibly innovative way in which it has been launched. Eris is giving away great stuff to people who buy the book, Stuff that the people that are likely to read it will really like. If that works, it will hit the Amazon and other best seller lists and this will mean it will be bought by people who woldnt normally know about it.

It will be a huge surprise is this is not one of the most successful book launches in history for a relatively unknown author (in comparison to say, Dan Brown, best-selling drivel-churning novelist). He deserves the success. You deserve to read The Lean Startup.

Video

http://businessofsoftware.wistia.com/medias/6y3g3fyhk7?embedType=async&videoFoam=true&videoWidth=640

Notes

Notes from Eric’s talk here.

Transcript

Joel Spolsky: According to this our next speaker is Eric Ries. If you’ve heard venture capitalists use the word “pivot” way too often in their conversation, like when they decide to send back an entree that they ordered, and it comes out wrong and they decide to pivot and get a different entree altogether. The person you have to thank for that is Eric Ries, his concept is the Lean Startup. He’s done about three pretty Lean Startups. The one you probably know is IMVU 3D Avatar world gaming kind of situation. He wrote Java books when he was in high school. Please welcome Eric Ries. [Applause]

Eric Ries: Thank you. Thank you all very much. I’m very excited to be here. I’ve got to tell you years ago when I was sitting in a cubicle in Microsoft and reading Joel on software I was like, “Wow, Joel, is like a major celebrity in my world.” So to get the email from Joel to say, “Come speak at my conference.” It was like, for me anyway, getting an email from Lady Gaga asking me to come perform onstage with her. I was like… Well OK, maybe it’s not exactly the same, but you get the idea, it was very exciting. So I’m very pleased to be here.

I want to just set up some quick ground rules. The most important one is that I do not want your undivided attention. So, who has a mobile phone? That’s just a question to see who’s paying attention. Not that many people, OK. Please take it out of your pocket, hold it up, thank you. Turn it on, no phones off, please. Laptops on, power on, get online, if you’re not online you’re basically not alive. So get online. And all I ask, if your attention starts to wane, whatever happens #leanstartup is the hashtag so tweet amongst yourselves.

But as you see I really welcome feedback, so if I say something that you think is worth tweeting I appreciate hearing about it. And, of course, if I say something that is too dumb for words I appreciate hearing about that too. Again #leanstartup hashtag. I have been on this journey to promote this thing called a Lean Startup for the past two years. It means I can’t get this thing on. And I wrote this blog “Startup Lessons Learned” and I never thought that I would do this for a living. I used to be an engineer and that’s a job I really understood well. This I understand less well. But what keeps me going, the reason I get excited about doing this is that I have been imploring audiences, whoever would listen basically to stop wasting people’s time. It’s actually that simple.

Most of the products that we make are never used by anybody. If you think about that, that’s got to be a preventable condition. There has to be a way for us to stop living like that. To stop pouring people’s time and talent into activities that go nowhere.

That’s what I think, by changing the way that we go about building new products, new companies I think that we can change that outcome. Everyone knows that most startups fail. Most of my startups have failed. I know if you’re trying to become a professional expert that’s actually not a good way to start “Hi everybody, most of my adventures have failed.” I have built plenty of products that nobody used, I’m quite embarrassed to say. But we have to start getting serious about the fact that this is an epidemic in our industry.

I brought a demonstration. Anyone remember Web 2.0? So at the height of Web 2.0, this is from 2006, a graphic designer put together this chart. To encapsulate the incredible wave of innovation that Web 2.0 was going to unleash. Then just three years later another graphic designer was feeling a little disillusioned and put together this graphic. Here’s our three years report card on Web 2.0.

You can see already something like half the companies are covered in blood red X’s because they are no longer with us. Which is pretty embarrassing. The green circles, I don’t know if you can see, mark those companies that had an exit. So those are the companies that I guess, by somebody’s standard, were successful. And then the unmarked companies are the ones that are still alive. Of course, many of the companies still alive are just sitting in the land of the living dead. So consuming resources like a zombie, but no one could quite put the bullet through the head to take them down. That’s not a good use of people’s time. And the green circles, sure many of them, somebody made money, and I’m all for people making money, but what about asking the question “Where they successful by this higher standard?” Namely did they actually succeed in living up to the raw amount of time, talent, potential, creativity, and energy that was poured into them by their founders and employees. I think by that even higher standard very few of these companies can be rated a success. And I think that’s depressing.

I think we can do something about it. To do something about it we’ve got to figure out who to blame. Who’s fault is it that these companies have failed? Who’s fault is it that my company’s failed? Certainly not mine.

So, I like to blame this guy. Here’s Frederick Winslow Taylor. He is the father of scientific management. It might seem a little odd to blame failures in our industry on a guy who died before the invention of the digital computer, and I guess it’s kind of unfair. But, I’m alive. The thing is, and it’s funny to use the word “blame” to talk about Taylor’s impact because in fact the incredible material abundance that we enjoyed in the 20th century is directly attributable to this man’s ideas. He had the very simple insight which is we should use science to study work in order to make it more efficient. Which is an idea that’s so stunningly obvious, it’s hard to remember that it had to be invented. In fact most of us who have heard of Frederick Winslow Taylor have mostly heard about his discredited ideas like time and motion studies, and the idea that workers should be treated like automatons. Because his true contributions, all of them, we now take so for granted we cannot imagine that they had to be invented, that we should study work. In 1899 when what we now call “Blue collar work” was done it was done by skilled craftsmen. The idea was that it required years of work by a craftsman to become a master craftsman. If you were to suggest to somebody that a business owner, or a manager could somehow tell that craftsman what to do, to know that person’s job better than they do, you’d be considered an idiot. Because you haven’t trained for all the years of doing the work of a craftsman.

Is this sound, to anybody, familiar? Just a little bit? Because it turned out that within only a few years Taylor and his colleagues when studying even ancient crafts like brick laying, gardening, shoveling were able to use science to discover more efficent ways to do tasks that everyone thought they already knew how to do best. Thus was born the idea of white collar work, or what we call “Management.”

You’ve got to remember Taylor and his peers were engineers. The very first managers were engineers. What we call “Management,” they called “Shop management.” They were trying to manage machine shops. Machine shops were the world’s first self replicating technology with a machine shop you can build the tools to build another machine shop. And every time you do that you have the opportunity to get more efficient. So that idea that that should be our goal to become more efficient to divide work into tasks, that those tasks should be done by specialists and managers are specialists to themselves to efficientialize that was all worked out by engineers 100 years. It’s almost the 100th anniversary of this utterance of Taylor’s “In the past the man was first, in the future the system will be first.”

Taylor was the first person to conceive of work as a system and try to figure out ways to optimize not just the individual work that people do, but the whole system, the whole productivity of the whole system. An idea that we take completely for granted today. And yet like any paradigm, this was the first paradigm of management the world has ever seen. And like any paradigm it’s very success has sewn the seeds of its demise. So generations of people have handed down this paradigm from one to the next.

When I received it, it looked like this. This is the waterfall methodology of product development. People are laughing because any time you hear a talking head use the word “waterfall” to describe this, you know they’re about to wail on it. It’s like an unwritten rule. People who actually like to do it this way would never call it “Waterfall,” it’s a pejorative term. But the thing you have to understand, when I was trained in engineering at school, at Microsoft, in the venture back startups that I worked at. I was taught this as the manufacturing metaphor of software development. You can imagine, incidentally how pissed I was when I found out they don’t even use it in manufacturing any more. It’s not clear to me what our excuse is.

But as much fun as it is to me to beat up on waterfall, it’s important to understand that this is a system that works really well in a specific context. And, of course, it’s the context that Fred Taylor would know very well. Situations where the problem is known, and efficiency is also known. So when we’re building something, like a new machine shop, very similar to something we have built before, or maybe like a video game sequel, this can work. In fact it’s very efficient to break the work into tasks and have people specialize in each of the individual tasks. When I did this in one of my startups, a startup that I joined, we were trying to build this really cool virtual world, so we had this huge vision change the world kind of situation. We really wanted to not build something cheap, to build the flip, but really change the world. So what we did is we raised something like $50,000,000 and spent about five years in stealth R&D building up a product. Those of you who have been around the business long enough should already be able to predict the outcome from those two simple statements. [Laughter]

Eric Ries: Because it’s almost inevitable that we did what I call “Achieving failure.” We successfully implemented the plan, we did it with high efficiency, but it turned out there weren’t customers to support our business at the end. After our huge high profile, and then crash the company eventually became a defense contractor. Strange kind of failure to have for a consumer products company, but anyway. It’s kind of depressing.

And, yet, here’s the thing you don’t understand. If you’ve never been through this kind of reality distortion feel experience the whole time we were building, for almost five years we were getting constant positive reinforcement that we were on track, we were being disciplined, we were metrics driven, we were advancing to the next stage of the waterfall constantly. And every time we advanced we would celebrate.

That’s the unit of progress in waterfall development is that we advance to the next stage.

The assumption is that somewhere out there there is an all wise manager who, like Fred Taylor would have, studied the problem in great detail, worked out what needs to be done and our job is to execute with high fidelity.

Anybody ever live through that before? Anyone ever achieved failure? Good, thank you for being honest. So, I don’t think it has to be this way, and I think the Lean Startup is fundamentally about taking the practice of entrepreneurship and putting it on a more sound, and more scientific footing. In order to do that we have to start with the definition. Because I think know a startup when we see it, but that mythology we have in our heads of the two guys in a garage eating Ramen noodles inventing the future is actually very misleading. What makes an entrepreneur is not what kind of noodles you eat, but it’s the kind of context that you operate in.

So here’s my definition of a startup. A human institution designed to create something new under conditions of extreme uncertainity. You guys have put together a great lineup for today, so you’ve already really heard a lot about the importance of the human part of entrepreneurship. And, of course, the importance of disruptive innovation in creating something that is truly different than what came before.

I want to focus today on the extreme uncertainity, but also just incidentally notice that this definition has nothing to do with the size of the company, sector of the economy or industry. Absolutely nothing. It doesn’t matter if you’re two guys in a garage or 200 people in a new division. If you’re job is to create something new, under conditions of extreme uncertainty, then you are an entrepreneur whether that’s what you think you signed up for or not. This is just a fancy way of saying that a startup is an experiment. When we call the Lean Startup “Innovation through experimentation” this is what we mean. It asks us to reconceive everything we do as entrepreneurs. Everything we do. Every new feature we build, every new marketing campaign that we create. Everything as an experiment designed to teach us what we don’t know. Because of that, because it is not just an experiment in product development, it’s an experiment in institution building that means another counter intuitive truth. That the experiment is not just in “Can something be built,” but “Should it be built?” In particular “Can we build a sustainable organization around a serious of products and services?”

Which means that entrepreneurship is management and if there’s a more bizarre juxtaposition of two words in the English language, I don’t know what it is.

Because entrepreneurship is our day, in our time and place is a hot and sexy thing that cool people do, it gets them on the cover of magazines. And management is probably the most boring activity that people do. Guys in grey suits in giant bureaucracies. So putting the two ideas together may seem kind of strange. But I believe the time has come for a new management paradigm. Not better, not worse than the old 20th century general management that was started by Fred Taylor. But simply different to geared simply to the conditions of extreme uncertainty. Is that making sense so far?

The pivot. I’m sorry. [laughter]. You’ve already heard that this is, by far, the most overused piece of jargon of the year 2010. I never had any idea of what was being unleashed when I started using this word to describe something in startups. I knew it was getting out of hand when I saw this in “The New Yorker.” [laughter] Can you read this caption? “I’m not leaving you, I’m pivoting to another man.” And I thought “Oh, lord, what have I done.” But there’s absolutely no way to understand entrepreneurship without understanding this fact. That the great entrepreneurs when they encounter difficult do not give up hope. They don’t just go home. Neither do they preserver the plane right into the ground. Instead they take this combination of commitment and iteration and we use the metaphor of the pivot from basketball; keeping one foot firmly rooted in what they have learned, while changing systematically one other thing at a time about their business. If you look at the successful entrepreneurs and compares them to the failed entrepreneurs and ask yourself “What’s the same, and what’s different.”

One of the things that’s the same, surprisingly is the initial quality of their idea. In general, very poor. I would go so far as to say in general crazy. What’s different is that the successful entrepreneurs were able to find the zig-zaggy path which only in retrospect seems obvious. Unfortunately there’s this mythological and industrial complex out there who’s designed to sell you the idea that startup founders are heroic, their ideas are brilliant. And they were always in the right place at the right time and had a very linear path that benefits everybody, myself included. Except that happens to be wildly inaccurate. So, sorry about that.

The premise of the Lean Startup, therefore, is also correspondingly simple. If we can reduce the time between pivots, we can increase our odds of success before we run out of money. Because what matters is not how much runway, do you have left. It doesn’t matter how much money you have left. It only matters how many pivots do you have left. And yes, you can get more pivots by raising more money. But you can also get more pivots by reducing the time it takes to learn from each pivot. If that’s starting to sound familiar it might be because the Fred Taylor management paradigm is not the current reigning paradigm.

There’s all this talk of iteration, and pivoting, and doing things in a more efficient way, a more value creating way is owed primarily to these two guys. This is Deming, and a guy named Taichi Ohno who’s the father of the Toyota production system. This current management paradigm which has produced almost everything you see in this room from the chairs, to your clothes, to the laptops you’re typing on, originated in Japan in the postwar period and has come to totally dominate the manufacture of physical goods. So we owe these guys a great debt.

When this Lean revolution was translated into software it came to look like this. Here’s our own management guru from software, Kent Beck, I put it in sepia tone, so it would fit in. [Laughter] He is still alive, though, I can’t… And, of curse, not just Kent Beck many members of the Agile alliance made this possible. Extreme programming just happen to be my favorite. The idea here is if we built the product more irritativly with the customer actually a joint participant in the process, so not just a teddy bear sitting next to us “Oh, I love that idea from Darmesh.” But actually physically sitting with the engineers, then whenever we want and we have a question about “What are we supposed to build?” We can turn to the customer and we can say “Excuse me, can you give me an authortative answer about what this feature should look like, what this marketing campaign should be?” Or any other question we have.

The thing that’s important to understand about Agile is it has its roots exclusively inside the IT organizations of big companies. For it contacts where the problem is know, its the solution that’s unknown. So when the Chrysler Corporation needs a new payroll system nobody has to ask “Why is payroll important? Why would a company want payroll?” We know. And what it’s supposed to do, we know. The issue is how can we port it to this different architecture, how can we build a system that does it more efficently. So the unit of progress of agile is fundamentally a line of working code. Which is really a big deal. It means getting rid of all of the other extraneous factors like meetings, and documents that go stale, and those specification documents that nobody reads, and all that nonsense and moving it from development and really focusing on the production of stuff. The problem is that as entrepreneurs we don’t live in this world, either. The Lean revolution began with a simple question, “How can we tell the difference between value creating activities and wasteful activities?” Which I guess it must have seemed kind of obvious to the people who were asking it for the first time, but it actually was a big deal. Instead of just asking “How can we do whatever it is that we’ve done before most efficiently?” Which is basically Fred Taylor’s insight. The question was “Is everything that we currently do actually necessary to produce value for customers?” It turns out that there are a lot of things that we were doing in manufacturing and then throughout our industrial economy that turned out to have no value. Like moving batches of stuff around.

Customers don’t care if you’ve efficiently transported parts of a car fast. They just want the car. So focusing on the cycle time how long it takes you to go from idea to completion is a big deal. But here’s the problem Deming made famous the idea that the customer is the most important member of the production line. The idea is that only things that count in the eye of a customer should count to us. But, what if you don’t know who the customer is?

We talked about extreme uncertainty, the biggest uncertainty that most startups face is who their customer will be. If you don’t know, then how do you know what they’ll value. How do you know who you’ll sit next to the engineer so they can turn and get an authoritative answer. In entrepreneurship we can’t do that. We live in this world of the unknown problem, and the unknown solution.

Another management guru, a mentor of mine named Steve Blank, also still alive. Hi, Steven! But in sepia tone, it’s important. Created this thing called “Customer development.” Which is a parallel process to Agile development for concurrently asking the question “Who’s the customer? What problem are we trying to solve for them?” In line with building intuitively a product that serves our current hypothesis about who those customers are going to be. This changes the unit of progress from that working code to what I call “Validated learning.”

I’ll illustrate with a story from something you just heard about called IMVU. When we were starting, we call it IMVU, just get used to it. When we started IMVU we wanted to bring the power of virtual worlds and 3D messaging to the masses. At that time, this is circa 2004, so the hot new technology for social networking was instant messaging. Remember that? So we wanted to create an instant messaging company that would be more intimate, more expressively communicative. So our theory was as follows; we sat at the white board and we though hard about the strategy for this company.

We reasoned as follows; everybody knows that IM is a networks affects business, the value of the network increases proportionally to the square of the number of participants. That’s Metcalfe’s Law. Since everyone already had an IM client, they would be high switching costs to create a new one because you have to bring all your friends with you, and therefore IM has a high barrier of entry to industry and therefore we could not create a standalone IM network. Does that sound reasonable?

When I give that talk in front of MBA audiences, as I will later this week, I usually get a lot of head nodding. Like “That sounds pretty smart.” And then it gets even better when uncertainty brilliant strategy to avoid that barrier to entry was we create an instant messaging addon that would inter operate with all of the existing IM networks. So you wouldn’t have to create a new buddy list, you wouldn’t have to learn new software. You just poof turn your instant messaging into 3D instant messaging uncertainty sound clever? Yeah. I get some chuckles because some people know the antic.

For reasons we don’t really have time to get into in detail, everything I just told you in my analysis is 100 percent wrong.

Network effects incorrect, switching effects incorrect, barriers to entry incorrect, therefor IM strategy woefully incorrect. I want you to just think about me for a second, just sympathize with me. I was a primary technical cofounder of this company and we spent something like six months developing this IM addon software. I mean crazy hours, the true startup thing. It probably took us another three or four months to realize that it was basically completely flawed and we wound up having to throw all of that code away. I was personally depressed. Why? Because I was following Agile which told me that I could drive the waste entrepreneurship process. And yet I had just committed the biggest waste of all in product development which is building something that absolutely nobody wants.

I thought “Gosh, would the company have been just as well off if I had spent the last nine months of my life on a beach somewhere drinking Mai Tais as it was with me writing this code, which after all had got thrown away. Anyone have an answer to that question? What’s that? Yeah, somebody else over here. Anybody want to throw it out?

No? No, why not? Because I learned something.

That’s always the answer to this question. Anyone who’s been in any kind of management position, or any kind of managerial organization will know “What do you do when you fail to execute?” You claim to have learned something. [Laughter]

It is the last refuge of all execution failures, it’s what you hide behind. And is it any wonder that in management circles, people who actually practice management, learning is the kiss of death. You never want to say that word, it has a really bad connotation. But that’s exactly what I did, I said “If I hadn’t built all this code, we wouldn’t have learned this important thing about customers and therefore it was worthwhile.” And that’s how I made myself feel better.

Until I had the following dark thought. The point as follows; if the goal of the last nine months was to learn this important thing about customers, why did it take nine months?

Why wasn’t I asking myself along the way “Could I learn the same amount in a shorter amount of time?” For example did I have to support 12 different IM networks for interoperability? Would I have learned the same thing, namely that no customers want IM interoperability with only six networks? [Laughter] Yes.

With three? What about with only one? I guess so.

But then, this is the really dark though, “Did I actually need any code at all?” What if I just created a single webpage that offered people the opportunity to download 3D instant messaging addon software to add avatars to their IM networks. Would anybody have clicked the download button? The answer is basically no. [Laughter]

So if I couldn’t even get people to download my software why did I have to write it? Listen, maybe that was the right thing to do, maybe not. But the fact that I could even ask that question was horribly, horribly depressing. Because my job was an engineer. My job was to make code. A working line of code was the way I evaluated progress, what I realized was that’s just fundamentally wrong. Along the way I needed to be asking “How can I learn what I need to learn about customers?” When I talk about reconceiving every product development feature as an experiment, that’s what we need to do. In other words as engineers we’re generally trained, given a specification of a product how can we achieve that specification at the lowest cost. It makes us feel really good when we figure out a way to get 80 percent of the value for only 20 percent of the cost. But, if we’re building something that nobody wants who care if we do it on time and on budget? Who cares if it’s high or low quality? It doesn’t matter.

Everything we’re doing with lines of code is wrong. Instead we have to as “How can we learn what we need to learn at the lowest cost?” That’s a really different way of thinking about product development. Here’s the basic scheme. A software company is absolutely nothing but a catalyst that turns ideas into code. Everything else is a side effect. When customers interact with that code we can chose to measure it; which generates data qualitative, and quantitative which if we want we can chose to learn influencing our next set of ideas. This is like the Fisher Price model of entrepreneurship. Except this is a very powerful diagram, this feedback loop gives us a clear heuristic that we can use to evaluate any proposed process or architecture change in a startup. Namely does it minimize the total time for this feedback loop, or does it suboptimize by making us really good, Fred Taylor style, at one of the individual activities.

When I was an engineering manager I used to say stuff like “Customers don’t care what kind of reports we have. They only care if we have a good product. So why are we wasting all this time on data warehousing and measuring all this crap that no one ever looks at, anyway? Let’s just code faster.”

The problem with that argument is it is true, if you stop measuring you can build new features faster. In the same way that if you close your eyes and just floor the accelerator you can get the car going at maximum speed. But, of course, our goal is not to get the car in motion as fast as possible, our goal is to get to some kind of destination.

And similarly if there’s any data warehousing people in the room, you might have occasionally found yourself uttering the idea that if measuring some things is good measuring more things is better and therefore we should have 10,000 graphs. Of course nobody learns from 10,000 graphs, you only learn from the fewest number of key ones which teaches what we need to learn.

Then, of course, I affectionately call it the “MBA fallacy.” Although many of my best friends are MBAs, as they say. The idea that if you iterate at the white board, that is the fastest way to learn. That’s how we came up with the IM addon strategy in the first place. Except iterating at the white board is a magic fact free zone. Where all you’re doing is getting high on your own supply. So not necessarily a good idea. So what we need to do is figure out “How do we get through this loop as quickly as possible?” I brought with me some slide.

I just, very quickly, want to walk through a couple of different specific techniques from the Lean Startup model.

First of all do uncertainty tactics with the principles. I spent all this time trying to walk through the principles of the Lean Startup so that you can evaluate for yourselves whether these tactics actually may or may not work. My contention, my thesis for you to prove or disprove in your own mind is that each of these techniques over uncertainty stage of the feedback loop, but have the net effect of minimizing total time.

Probably the most controversial in the Lean Startup arsenal is called “Continuous deployment.”

At IMVU we’re famous for having put software into production about 50 times per day on average. So some days more, some days less. I like to look around at that point to see if there’s an engineer who’s got that look on his face, like “That does not seem like a good idea.” [Laughter] Who- “Everything was sounding good until you went- wait, wait, hold on, what?” Because think of all the things that could go wrong. Someone could just take the site down if they wanted to. They could regress an old bug. My personal favorite, remove the checkout button from your ecommerce flow. Turning your business back into a hobby.[Laughter]

And those things can happen, but the discipline of continuos deployment makes it very unlikely.

We engage in prevention of those activities and the plus side is that we can put software into production as fast as we can write it. Take about 20 minutes between the time is written, checked into the trunk, there’s no branches and the time it’s live in production. So that allows us to do the features that take more time to prioritize than they do to build. And just run experiments constantly. The reason you have to understand the theory of the Lean Startup is this is just going to sound nuts if you evaluate it Fred Taylor style. Does this make everyone in the organization individually more efficient? No because they’re shipping, they’re deploying software all the time. It would be a lot more efficient to batch it up and do it all one month at a time, or three months at a time, or God forbid, put the year that the software was written in the name of the software. [laughter]

Oh, lord, oh, dear. What kind of iteration cycle are we going to have then. And, of course, what are the odds that by the time we realize we’ve made a horrible mistake in product development what are the odds that we’ll actually remember what we were thinking at the time?

If the feedback is extremly fast, we’re going to learn really fast. If the feedback is extremely slow, we’re going to run up against the limits of human memory.

So here are the very basic continuos deployment principles, and I promise we’ll talk specifics, just very, very quickly.

First we have a commitment to having every problem under the sun exactly one time. Our motto at IMVU was “Make any mistake you want, if you make the same mistake twice, you’re fired.” Very simple. So we would have people, for example, deploy a production on their very first day of employment. We would hire a new engineer and if by the end of the day they hadn’t shipped to production, we would be like “Boy, something is seriously wrong.” To have it on day two, OK, but to have it on day three major, major crisis. Almost always happened on day one.

You can imagine for people trained in a large company that would be a little bit scary. They would have a little bit of resistance to doing that, they would always ask questions like “Wait a minute, what if I accidentally take the site down?” And our point of view was “If it’s so easy to take our site down that someone can do it on their first day, shame on us for making it too easy.”

Second is that we have this commitment to get to the root cause of things and fix them. So that they never happen again. That requires taking time and space for that to happen. So we took an idea directly from the Toyota production system. Everyone knows the famous hand on cord that any employee can pull to stop the production line if they see a quality problem.

We took the same approach. Whenever anything happens we immediately stop work and create this space before we start working again to fix that problem. Our assumption is that if our process is developing problems, if it’s shipping code that doesn’t work, taking the site down, or whatever it’s doing it’s not the individual person’s fault, it’s the system’s fault. So continuing to operate the system while it’s malfunctioning is fundamentally crazy.

The other thing that can really kill you, especially in a startup is trying to prevent every possible problem. If you want to kill someone’s project, this is especially true in big organizations, just sit in the product planning meeting and start mentioning pointer cases of things that might go wrong. That thing might cause an asteroid to fall on our customers. It might cause their house to catch on fire, it might cause their computer to… It’s like, you can start to “OK, well how are we going to prevent that from happening?”  You start to plus up the spec, because you’ve got to take into account all these pointer cases, pretty soon it’s so big it’s pretty easy to axe and do your much smaller project instead.

Startups do that to themselves, they don’t need some extra person to do it, although God forbid if they have VCs involved, then it can really get out of control. So instead of even going into that argument. What we do is say “We want to make sure we invest, instead of preventing specific problems and making our whole system more agile so we’ll be able to respond more quickly if it happens. And because we’ll never have every problem a second time, then we’ll be able to invest in the dementia at the time.” Does that make sense?

Here’s specifically what it means, if you want to do continuos deployment you obviously have to be able to deploy the software quickly. As I mentioned it take about 20 minutes on average at IMVU. Most of that time is not copying authoritative but really trying to certify the change to make sure it’s actually good, not harmful. Then if it is harmful you have to be able to revert it quickly. So I’m sure somebody in this audience, while I was talking about the business to hobby bug, removing the checkout button from the ecommerce flow would be like “Come on, that’s not a very good example. Anyone with a halfway decent continious intergration server and some functional testing could stop that.”

So let me make the problem a little harder. My personal favorite, instead of removing the button, as our hilarious April Fool’s prank, come on back to my desk, efficiently lack in time to IMVU. And we’ll just make the button white on a white background. So it’s still there, it’s just no human beings can see it. Then we’ll check that in. If we actually did that experiment, and at IMVU, of course, we actually did stuff like that all the time, not intentionally, but by accident. But what happened is within 20 minutes I would get an email from what we called the “Cluster Immune System.” “Dear Eric, this is your Cluster calling. I see that you tried to check in change number 4731, that was a catastrophically bad idea, so it’s been automatically reverted. And we’ve actually shut down the lines so that nobody else can check anything in, until someone gets to the bottom of what the heck happened.” Oops, so much for my April Fool’s joke. Because the Cluster Immune efficiently monitoring not just does the software work as intended, but is it producing the intended business results for the company. And if that’s suddenly going – When we’ve deployed at 20 percent of machines in the cluster, and revenue is down 20 percent that’s not a very good indicator. When we’re going to 40 percent of the machines and now revenue is down 40 percent we need to immediatly revert and that’s what happens automatically. In order for that to work, of course, you have to work in small batches.

Just to give you a sense of what I mean if we had an individual engineer writing software by themselves for three days we would consider that a large batch. God forbid you have a team of five working on a branch for two weeks, we would consider that a ridiculously large batch.

Because most of the work uncertainty creating a new feature is not actually the feature itself, it’s all of the integration between that feature and all of the uncertainties, and APIs and everything else. All of those changes that are side effect free. We’re just widening the API a little bit and putting the default to be the old thing it was before. That won’t cause any changes, right? Right? If that’s true, let’s just deploy it right now and make sure. What’s the harm? You’re the one who said it was side effect free. Why not put it into production, it’ll be the acid test. Of course if we make a mistake Cluster Immune System will bail us out.

We used a lot of open sources software to make this happen. Everyone in the company had a sandbox, when I say everyone, I mean not just the engineers, but everyone in the company. If you were in marketing, QA, operations, anybody who wants a sandbox, you can have one. In fact if you could figure out how the sandbox worked, it was relativly simple, you would would start to realize “Hey, wait a minute, this template of HTML looks a lot like the thing I see in my browser and there’s a typo here, and a typo there. Can I just change this typo and make it go away?” Our point of view was “Sure!” If our system was so easy to take down that a marketing person could do it, shame on us.

So if you want to teach yourself to code and check in changes God bless you.

As long as you follow the process that we’ve developed.

We, of course, ran continuous integration. We used BillBot, I would say about 10 percent of the computational power of our cluster was devoted just to running these tests because we had to get them to run every 20 minutes. We did the incremental deploy that Cluster Immune System idea. We also did alerting and predicting monitoring so the idea was if we ever had a problem that we didn’t catch with our immune system we’d want to get somebody out of bed immediately with their pager going off. Every time a defect would make it through these layers of defense we would ask “How could we strengthen the layers of defense a little bit so it’s less that that would happen later?” So if someone got paged in the middle of the night, why didn’t we catch it in the step before? Why didn’t we catch it in continuous integration? Et cetera, et cetera, et cetera. Does that make sense?

Let me talk about another buzz word that probably some people here are sick of hearing about already called “Minimum Viable Product.”

This is not quite as bad as pivot, but it’s getting there. I’ll just go through this relatively quickly because hopefully it’s self evident by now. Let me just remind us, why did we build products in the first place. As I said they’re experiments to teach us things, but what do they want to teach us? How to delight customers, how to get them to signup, how to make money from them. But more importantly how to realize a big vision. You do not need any of these techniques to build something small. Just build it, see what happens is fine.

When we try to do something big, though, the number of assumptions starts to really pile up, and that’s what we’re trying to prevent with Minimum Viable Product. Most people I know fall into one of two camps. One I call “Maximize chances of success,” that’s basically like “We only get one swing at the ball, so let’s make it as good as possible.” That’s what leads to features creep or scope creep. The opposite, which is release early, release often. Also has its problems like as soon as you have three customers, you already have five opinions about what you should do next. Now what? So you can’t just listen to customers, we have to have a vision and test that vision against what customers will accept.

So here’s Minimum Viable Product. The minimal number of features needed to begin that feedback loop of learning and discovery. Because I know we have a lot of engineers in the room I can confidently say much more minimum than you think. So if you’re ever actually in a situation where you’re not sure, just as a thought experiment try cutting it in half, cutting it in half again, and cutting it in half one more time and asking yourself “What would happen if I shift that one-eighth what I was thinking of shifting?” Would the world really come to an end? Really? Because the opposite of customers loving your product is not them hating it and telling all their friends, the opposite is indifference. But if people are indifferent what’s the harm in doing it? Maybe they won’t be indifferent, right? I think one of the secret fears in maybe they’ll actually like the crap that I put out there. What would that say about my internal value of quality and all that other stuff?

It’s actually necessary to do Minimum Viable Product because of the technology life cycle adoption curve. If you’re a startup the only people that will talk to you are early adopters. So if you try to sell to mainstream customers, as we did at that virtual world company I mentioned you’re going to fail anyway. Because they don’t want to buy from a startup. Therefore any extra work you do to meet the standards of mainstream customers is by definition a form of waste. Since you’re going to have to sell the early adopters anyway, let them fill in the gaps and help give you the feedback you need to learn.

I can’t stress this enough MVP is for big vision products. If you’re doing something small, if your plan is to ship something and see what happens I guarantee you that you will suceed. You will succeed in seeing what happens. Congratulations. But if you don’t fail you can’t learn, so what we need to do is start to think like scientists. Make specific predictions about what’s supposed to happen when people engage with the MVP, including sometimes the prediction they’ll absolutely hate it. And then see is that really actually true.

When we did IMVU and we put out that first version of the product it was so crappy. I mean it would crash your computer more often than it would give you a delightful 3D avatar experience. I was personally embarrassed to have my name one it, except… I, first was really relieved when it turned out nobody used it. Then I was like “Wait, hold on! If no one’s going to use the product why did I just spend the last nine months of my life arguing with my cofounders every day,” I can’t tell you how many hours we invested arguing about “does it have to have this feature or that feature? Fix this bug or that bug.” All this institutional energy was all wasted.

You can read about these techniques online, I’m not going to go into any detail here.

The basic concept of smoke testing landing pages as I talked about creating the page with just a download button to and see does anybody click it. We call the technique “SEM on Five Dollars a Day.” You can read about that on my blog. I’ll talk in a moment about brining split testing into the product, that’s very important. All of the design thinking techniques; prototyping, customer discovery, removing features like they did with cut and paste on the iPhone, all of those techniques are really valueable, and it’s been written about quite a bit online so I don’t feel the need to go into it right now. Happy to take questions about it in a moment.

If it’s so obvious why don’t people do it? I think there are three basic fears.

The fear of the false negative. To maintain the reality distortion field requires keeping data at bay, because as long as there’s no data you can convince people that anything is going to happen. But as soon as there’s even one data point that starts to call into question “is it really going to happen?” So people sometimes feel like “If I put something small out there and people don’t like it, do I have to pivot at that point? Do I have to give up?” And that’s just not entrepreneurship. Our goal is to persevere until it’s completely obvious that the vision is wrong, and then pivot.

Second the visionary complex that customers don’t know what they want, which is a true statement, but that doesn’t absolve us from talking to them anyway. Just because customers don’t know what they want doesn’t mean that they don’t have useful information that we need. After all our goal is to change customer behavior. When people say “Change the world,” they don’t usually mean actually… carving your face into the surface of the moon or something like that. Unless it’s an episode of “G.I. Joe” I haven’t heard about. What they’re talking about is changing customer behavior in a beneficial way. In order to discover if that’s possible we have to test with customers. But not necessarily obey them, as Darmesh said. And then my favorite is too busy to learn. I sometimes get the question “How could you guys afford to waste time in the early days building metrics into your product. Weren’t you too busy building the features?” That’s just going back to the idea that you could close your eyes and slam on the accelerator and hope for the best. What you have to do is actually create the space to get through that bill measure learned feedback.

This is another technique I took directly from Toyota production system, it’s called “Five Whys.” I know Joel has written about it, so you may have heard of this already, unlike most audiences. So I’ll just address it briefly. The idea is when anything goes wrong in our process we want to ask why five times in order to get to the root cause. So, for example, a server failed. “Why?” Well, someone called this obscure API that caused the CPU load to go to 100. “OK, why?” Well they’re a new employee and they didn’t realize that if you use that obscure API, then it will have this side effect. “Why’s that?” Well they were never properly trained in how to use our company’s APIs – or alternatively our APIs are so complicated nobody could ever be trained to use them. It’s the same. But then you say “OK, why’s that?” Well their manager doesn’t believe in training. “Huh, that’s odd.”

You started out with a technical problem, a server fault, but now we’ve got this managerial problem, someone who doesn’t believe in training. The five whys approach is to make an incremental investment at each of the five layers of analysis. So we’ll bring the server back up, we’ll obviously fix that API, we’ll teach that engineer what they need to know, and we’ll have a conversation with that person’s manager to say “Hey, you really should train new engineers.”

But if you ever actually had that conversation with a real engineering manager you might be expecting it to go something like this; When someone used to say stuff like that to me I would say “You know what, you’re so right, we really should be training our engineers. I will take me approximately eight weeks to develop a new comprehensive engineering training program. So if you have somebody who can do all the other stuff I was planning to do for the next eight weeks I would be happy to do it.” Which is manager speak for “No way in Hell.”

So here’s what we’re going to do, instead of having that completely fruitless argument, we’re going to say “That’s fine, that’s fine. Since this was a minor problem we’re going to make a proportional investment in prevention.” Let’s say in this case proportional will mean one hour’s work. “So great, you’ve got an eight week plan, perfect, just do the first hour of it. And then stop.” “Oh, I couldn’t get anything done in the first hour.” “Really, nothing at all? What were you going to do first? Before you spent 100 hours you must have done something.”

“Well, I guess I would start by creating a wiki, and creating a page where the training program is going to go.” “Great, let’s just do that and stop.” That doesn’t seem very valuable, right. Except that the next time we have a problem caused by a training failure we’re going to have that same argument, with maybe the same or a different manager and they’re going to say “I couldn’t get anything done in an hour.” And you say “Sure you could. Dude, what’s the first thing?” “Well, I created a Wiki page.” “Ahh, someone’s already done that.” [Laughter] “So, why don’t you go ahead and do the second hour.”

Notice what’s happening here. The amount of time we spend in working on engineering training is directly proportional to the number of problems that engineering training is actually causing. So we have a natural feedback loop between the issues we’re having and the prevention we’re taking.

Is there a question over here? Is that what I saw?

Audience Member: I’ve been working a while with efficiently on GDT, and one of the issues with GDT is it takes a while to trust the system.

Eric Ries: Yes.

Audience Member: Did it take you a while to trust the system that you will actually come back to that the second time.

Eric Ries: We’re still working on it. [Laughter] This is always a perennial issue. And it really requires intense management support, so don’t just try this at home, unless you’ve actually got your team bought into the idea and here’s the issue. Five whys can go wrong in two ways. One is you can do the analysis and then convince yourself that you don’t have to do any prevention, which is always wrong. So our goal was you can use any amount of judgement to decide how much you should do. The only answers that are not allowed are zero, and stop everything and work on it forever. Because the other thing you can do, of course, since we’re all engineers we can say “I have designed the perfect solution that can fix not just this problem, but every other possible problem from now until all eternity. It will take me only one month.” [Laughter]

Always, always, always takes only one month. I’ve never heard it take six months, even though, of course, if you actually go down that road you’ll never get anywhere. So either of those extremes is no good. What you need to have, we actually were to appoint to each area of our business a five whys master who’s job was to run those meetings. I’ve written a lot about this online, so if people are interested in the specific techniques, how to use them in a startup context you can get them online, but that’s the basic idea. The reason five whys is so powerful is that it acts as a natural speed regulator. I spent so much time talking about how speed wins, sometimes people feel like that means “Oh, I should go as fast as possible.” But that’s not true. We want to go as fast as we can reliably execute this model. We’re going through the measure of learned feedback loop. The fallacy of the older engineering paradigms was “Time, quality, money pick two.” Anybody ever trained in “Time, quality, money pick two?” That’s how I was trained as an engineer. It’s an old engineering maximum. It turns out to be false, you cannot trade quality for time because quality problems eventually slow you down. So what you want to do is figure out how fast can we actually operate at a sustained level.

The nice thing about five whys is the more mistakes we’re making the more time we will have to spend in prevention, which slows us down, which allows us to make fewer mistakes. As our investments in prevention pay off we makes less mistakes, we can speed up. So we’re naturally regulating to find our optimum speed. Does that make sense?

Let me talk about one last technique, then I’ll stop for questions.

This is rapid split testing. People who work in direct marketing have known about split testing for years, and I feel like it’s starting to become more and more known in product development. But I think it should be considered a core competency right up there with knowing how to use a debugger. So what I think we need to do is be split testing all the time, so just a completely obvious reflex and there’s no excuse not to do it.

Here’s how we set it up at IMVU, we had a very simple API, so that creating a new split test was always exactly one line of code. If it’s two lines of code someone can come up with an excuse not to do it. But one line, come on, one extra line, an if clause, come on, how hard is that. And when you’re changing a product, it’s even better because you always have a natural split test. You have the way it works now, and the way you propose it to work. So let’s temporarily allow a little code duplication and just have two paths and see which one is better for customers. To do split testing the hard part is actually not the test itself, but it’s understanding the analytics behind it.

You have to really understand the three A’s of metrics.

They have to be actionable, accessible, and auditable. I’ve written a lot about this, so I won’t go into too much detail. The basic idea here is if you show someone a report, the most valuable thing you can do with an analytical report is kill someone’s pet project, that’s where you get the savings. But, of course, people don’t want their pet project to be killed, what they want to do is say “Those numbers aren’t accurate.” So any time you’re dealing with what we call “Vanity metrics,” like the number of hits you get in a month, or your gross revenue, or any kind of thing that’s subject to external fluctuations people will find all manner of excuses to weasel out of having to pay attention to it.

Instead what we want to do is have every report we have be completely obvious what’s going on. And this last one is the one I’ll harp on, and be auditable. Meaning if someone was skeptical and said “This report said nobody used feature X,” or “100 people used feature X. I don’t believe you.” You could say “That’s great, let’s go call those 100 people and ask them if they used it.” “What? Oh well, we’ll just…” The auditing is we can take the number 100 and remember that metrics are people, too. So let’s discover who are the 100 people that generated that specific number. Last thing is measure the macro. And what this is is that in marketing we often do these little split tests like change the color of a button and see if it improves the click through rate. I’m all for those tests, because the curse of product development is sometimes small things make a big difference, sometimes big things make no difference at all and it’s hard to tell which is which. So it’s fine to test something small, but it’s not fine to measure something small. Nobody cares about the click through rate of the button. If you’re changing the color of something it’s because you think it will change customer behavior in a material way. Otherwise, it’s too small for a startup to be worried about.

So what I urge you is for every test only use your key high level business metrics, the one you really care about and use the same set every time. So these numbers are all made up, but this can give you a flavor for what that report looks like. What’s amazing is how many times we have the impulse that if we make a change that makes things look better, but it changes customer behavior not at all, we want to ship it anyway. Because it didn’t harm anything, and it still looks better. If you have the discipline to say “No, we will never ship something unless we can prove that it actually made customers’ behavior better,” what you’ll discover is that over time instead of your product getting prettier it will get more effective and I think that’s what we really want. So that’s split testing, that’s kind of a sampling, I would say, of techniques for a Lean Startup. There’s, of curse, much, much more. I write about this in my blog, and you can learn a lot more.

I have a book coming out next year, so I’m sure you all want to buy hundreds of copies of it. Please let me know if I can help you order in bulk. No problem. Heres’s a bunch of the techniques again. What I believe about each of these techniques is that they operate at one specific stage of the feedback loop, but have the net effect of optimizing total time. And that if we’re willing to change some of these more sacred beliefs about whether we’re an entrepreneur, what is it that we’re trying to do,a bout whether our goal is to execute or to learn. Then we can actually increase the odds of our startups being successful. In other words we can stop wasting people’s time. Thank you all very much. [Applause]

Eric Ries: Do we have time for some questions? Is that a yes? I’ll take some questions. Yes?

Audience Member: How would you do fastest iteration on the App store, newly release earliest thing? [Laughter]

Eric Ries: Oh, how would you do rapid iteration on the App store? Do it on Android and then port it to iPhone. [Laughter]

Seriously, remember our goal is not to make money, our goal is to learn. So if the learning is equivalent on the two platforms, always chose the platform with the highest agility. That said, it’s important to understand that continuous deployment doesn’t mean every 20 minutes, it means as fast as you can. So, for teams that are operating, doing a release only once a month, which I’m sure many of you are. Let me see- a quick show of hands, who does releases once a month, just curious? OK, so all of you can operate on the App store, no problem. It only becomes a problem as you start to crank down the speed of iteration. Does that help? Questions? Yeah.

Audience Member: I like the methodolgy and concept that you’re driving at, but in our particular industry if we put out software with mistakes people could get hurt, because we’re in an electrician’s office. So how do we take advantage of this methodology without killing our clients? [Laughter]

Eric Ries: What a very nice way of saying “If I do what you’re saying I will accidentally kill people.”

Audience Member: Correct.

Eric Ries: “And is that OK?” [Laughter] No. That’s not fair, the question was how to do it in a situation where the software is truly mission critical, and that’s actually a really important question.

There’s two parts to the answer.

The first is there’s an assumption in your question that operating under a fast iteration continuous deployment option will lead to lower quality software over time. That’s actually not true. I think actually- Because it prevents us from making mistakes allows us to discover our mistakes much more quickly. It’s just that the kind of test we’ll do before something goes into production will be different. So I’ll give you the example of a company called KaChing. When they make a mistake nobody dies, but they’re a mutual fund company, so they actually manage your money for you. They actually manage your money for you, like they actually direct your mutual funds and make trades on your behalf in your brokerage account. So when they make a mistake people lose millions of dollars. And they’re SEC regulated, so they’re under a kind of corporate death sentence if they were to make mistakes, but they found that by practicing continuous deployment that just never happens.

And in fact it happens a lot less than it does in organizations that rely on a manual QA staff. The problem with manual QA is that at first it seems like a really good deal because you have human beings actually testing everything that could go wrong. But because the worst part of software is action at a distance, a change over here can cause a change in behavior over there that seems unrelated you have to check everything every time. The number of things to check is a function of how much software has ever been written, which is a function of how many employees you have had times how long they have been writing code. Which basically grows polynomially with time. Very few people can afford an organization where the QA department also grows polynomially with time. Therefore at some point they have to start making tradeoffs and say “Well, we’ll only check the stuff that’s releated to the change.” That’s when you know you’re in trouble. Because, of course, if you knew ahead of time what was related to what you wouldn’t need a QA department at all. So that’s my answer. Right here.

Audience Member: I’m in a startup in Ireland. And basically customer discovery what would be your basic key elements to get the best customers. Build the features, for this ideal customer, find the customer that actually be there or just kind of a balance between the two.

Eric Ries: I don’t understand your question. Come again. I got that you’re from Ireland. That’s good. [Laughter]

I’ll be in Ireland in a couple weeks, we can discuss.

Audience Member: Yeah, so basically customer discovery. Do you build the product and then call the customers, or have the features and then the customers will come. What’s the best way to…

Eric Ries: Oh, I see what’s the best way to do customer discovery. There’s no general purpose answer. Every industry the dynamics of how many customers you have are really different. I mean think about it, you’re trying to build the next Google, you’re going to have millions of customers. But if you’re trying to build some highly specialize enterprise software you might have a total universe of 100 potential customers or anything in between. so the specific tactics to get to know those customers are going to be different. But the general approach is simply, it’s not either or, in parallel do both activities. Customer development, and product development.

So some part of your time, or you organization, or even if it’s just one of you split your mind, split your days, alternate between those activities. One having conversations with potential customers, trying to get inside their head. Understand who they are, understand what their problems are. On the other days actually try building out the solution. You can’t do one without the other. If you just work on the solution, if you only do product development, you’ll eventually achieve failure. But if extremely do customer development, you’re only spending your time with customers, I guarantee you, you will eventually start trying to build teleportation. [Laughter] Because it goes like like “What do people really want?” If you start working really up their hierarchy of needs, what would be great would be immortality, free energy, and teleportation. And so, you have to have both activities in order for one to balance out the other. Do we have time for some more? Yeah, OK. Yeah?

Audience Member: As a person who just came from big organizations. I always see that you end up with one department who builds or measures, and one who learns. And then they get disconnected. Do you have any ideas how to avoid this?

Eric Ries: Yeah, yeah, I do. The question is how to scale it. The true answer is we don’t know for sure that it scales yet. I mean IMVU is probably 120 employee company now, so it’s gone to a reasonable scale. Every time IMVU has doubled in size there was somebody who said “Well, sure it worked at size X, but it’ll never work at size 2X.” And I heard that when we were five people, 10 people, 20 people. So so far we’ve scaled up OK. And people are starting to build larger organizations with these ideas. But, the true answer is we don’t know. We know some things that definitely don’t work. And the first thing is what Fred Taylor called “Functional formanship,” or what we call “Departments.” If you have a department that builds, a department that measures, and a department that learns, I guarantee you will do none of the three activities particularly well. Or, maybe I should say you’ll do each of the three activities exceptionally well. Except that the whole system will accomplish none of your goals. So the issue as you scale up is you can’t just have one problem team, one solution team anymore, you have to fragment the company into multiple small modules. And the question is…

My experience is entrepreneurs face a fundamental choice. Either the can try to recreate for themselves the experience they had when the company was small, when they could all sit in a room together and make all the key decisions, take in all they key inputs and iterate quickly. I call that the hive mind approach. So what happens is they still stay in that room, but now the company grows around them. And everyone in the company basically becomes an extension of the hive mind, a tentacle goes out, touches the world, get more information. But the fast iteration is maintained for the founders.

I’ve done it both ways, that’s more fun, as a founder it’s great. You really have this feeling of power, that you’re affecting the world. The problem is that eventually your inputs become in no way correlated to reality, and so, you’re getting high on your own supply again. The other alternative is instead of creating that experience for yourself make everyone in your company a founder of their own small team. And really decentralize the decision making. And then as a founder, your job is to really focus on the system that empowers those people and make sure they’re acting in a coordinated fashion. In other words, you become a manager. But not the kind of general manager that we like to caricature, an entrepreneurial manger.

That, to me, is the essence of entrepreneur management. It’s making other people into founders. It’s not as much fun, but it does work. So I think one’s better than the other.

Yes, anybody else? Anyone in the back? I feel like people in the back have yet to ask questions. Yes?

Audience Member: How much initial effort went into building the heath check automation you have to make sure you don’t screw stuff up and what percentage, would you say, of ongoing technical effort goes towards keeping that up to date in order to…

Eric Ries: Those are really good questions.

The first is how much time initially went into build the Continuous Immune System Integration zero hours. The very first version of our Cluster Immune System was rSync gel script that was one line long that would just push from one directory onto our one server. And we just followed the Five Whys approach. We didn’t know it was called Five Whys at the time, we just thought of it as making every mistake only once, making it incrementally more sophisticated over time. And that’s a much better approach than trying to build something sophisticated in the beginning. Even if you’ve built your a perfect replica of the system we have at IMVU, then you would have a system that is perfectly well adapted to my company. If you follow the approach that I recommend here you will wind up with a system that is perfectly well adapted to your company, it’s much better.

Second part of your question was what’s the ongoing effort required to maintain. And it is significant, I don’t want to kid about this. I would say between 20 and 40 percent of the company’s efforts are spent in what we general consider engineering infrastructure overhead. That doesn’t bother me for two reasons. The first is, I don’t think many organizations have a very honest accounting of how they invest their time. I think the traditional waterfall model actually hides problems. My favorite is the guy who has to be the integration monkey. Anybody ever have that job? I have. It’s usually the most junior person on the team who stays up until 2:00 AM merging branches together. We generally feel like that person’s time’s not very valuable so we don’t mind. Of course that means that some of the most critical decisions that affect the company’s performance are made at 2:00 AM by some guy who doesn’t know what the hell’s going on. [Laughter]

I’m happy to trade those invisible costs to the more visible costs of continuous deployment.

A second thing I’ve noticed, especially now that I’m not involved in the company day to day it’s easier for me to see that when the company makes a conscious effort to cut back on the investment in this kind of infrastructure they inevitably have a temporary boost in productivity and then an immediate decline. Because, of course, as soon as you take the feedback loop out of operation things go haywire. So that gives them confidence that we’re actually investing the time really well. Ultimately the real test is “Do we deliver for our customers faster than our competitors?” And I think the answer is yes. Yeah?

Audience Member: Just a question about your background. You refer a lot to the Toyota system. Where did that education come from? Where did you get exposed to that?

Eric Ries: I read a book called “Lean Thinking.” The question is what is my educational background, how did I learn about Toyota production system and stuff like that. I have no formal relevant training, I do not have an MBA, maybe I should have disclaimed that at the front. I haven’t actually worked on a factory floor, in fact I know nothing about how cars are assembled. Occasionally I give a talk where there’s actually someone who does know about that, and they’re like “Hold on, hold on, that’s not right.” I always worry about that.

The truth is when I was at IMVU, I had this intuition, what I would consider the very beginning, the seed of what eventually became a Lean Startup continuous the benefit of Steven Blank and the customer development model to build up, and Kent Beck, and the Agile development mode. I had those antecedents to build on. But my attempts to implement them just weren’t working, and I couldn’t understand what the problem was. For those who understand the Thomas Kuhn, and the study of the Scientific paradigm, it really was that kind of experience.

Where a paradigm that had made people productive in the past ceases to make them productive in the present. For a long time, I would say probably a decade, I spent time trying to tinker with the current paradigm to make it work for me. And only out of real desperation did I start to be like “Well, where did this paradigm come from?” And I tell you about the manufacturing metaphor, I was like “Oh, I should probably learn something about manufacturing,” because that’s what my whole paradigm is based. And then when I started to read about it I was stunned to discover that they don’t use it anymore. I was pretty upset. [Laughter]

But, I think if you go out, and you decide you want to teach yourself an MBA, you want to read all these books and get educated, I don’t think it will be a very valuable exercise. What worked for me was when I had specific problems trying to reach out and understand the theories that might help me predict those problems and figure out solutions. Do we have time to squeeze one more in? All right, one more we’ll take- make it a good one. Who’s got a really good question? This guy, right here.

Audience Member: The slide just before this where you listed tons of techniques, how many of those are in your book? Sell us on your book.

Eric Ries: Ah, yes, thank you very much. [Laughter]

You mean this slide. This slide here. Every single one will be in my book, I guarantee it. You will become a guru. Now, first of all, the book will not be available until next Fall, so the publishing industry doesn’t exactly work on rapid iteration. [Laughter] However, that doesn’t mean that you can’t preorder the book today. [Laughter]

Because I do believe in rapid iteration, in fact those who preorder now will, of course, get to be part of crafting the content of the book itself. So if you have an opinion about what should be in the book I invite you to join me in that process. My guess is something like 10 percent of you are early adopters, you are all welcome. It just so happens there is a huge sale on this thing called the “Ultimate Lean Startup Bundle” being sold on AppSumo, just happened to have launched today.

So it was a really good question. I have a ringer right here in the audience. AppSumo is like a Groupon for online apps, and included in the thousands of dollars worth of stuff that you’ll get for only $40 will be a copy of my book, when it eventually comes out and tons of other stuff for the meantime. So, thank you for that. That’s just my favorite kind of question. I am on Twitter, this is my blog “Startup Lessons Learned,” you can drop me an email any time, I’m at your disposal. And I thank you all very much. [Applause]